Gas station without pumps

2013 March 26

Petition for human readers

Filed under: Uncategorized — gasstationwithoutpumps @ 15:32
Tags: , , ,

I just found out about a petition against the machine scoring of essays at Human Readers (thanks to a blog post on the AAUP blog):

We call for schools, colleges, and educational assessment programs to stop using computer scoring of student essays written during high-stakes tests.

Every year hundreds of thousands of students write essays for large-scale standardized tests. The scores are used in life-changing decisions. Students are accepted into, placed within, and rejected from educational programs. Graduates are hired or not hired. Teachers are qualified, evaluated, promoted, and fired. Learning institutions are compared, accredited, and punished. Yet in a major disservice to all involved, more and more of these essays are scored not by human readers but by machines.

Let’s face the realities of automatic essay scoring. Computers cannot “read.” They cannot measure the essentials of effective written communication: accuracy, reasoning, adequacy of evidence, good sense, ethical stance, convincing argument, meaningful organization, clarity, and veracity, among others. Independent and industry studies show that by its nature computerized essay rating is

  • trivial, rating essays only on surface features such as word size, topic vocabulary, and essay length
  • reductive, handling extended prose written only at a grade-school level
  • inaccurate, missing much error in student writing and finding much error where it does not exist
  • undiagnostic, correlating hardly at all with subsequent writing performance
  • unfair, discriminating against minority groups and second-language writers
  • secretive, with testing companies blocking independent research into their products

The basic premise of the petition is good: computers can’t score those aspects of writing that people actually care about, and so should not be used for scoring any essays that matter. Of course, some of the alternatives are just about as bad (like the MOOCs that use peer grading by incompetent “peers”), but I signed the petition anyway.

The bottom line is that assessment of writing is difficult, and so is expensive. If you can’t afford to pay for proper evaluation by well-trained human readers, then you can’t afford to use essays as part of an assessment. There are no shortcuts.


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Create a free website or blog at

%d bloggers like this: