Second Language Writers and The Machine Scoring of Essays

Deborah CrusanWright State University

Agenda

• Define • Discuss• Consider

But first, we must remember

• "For there is nothing either good or bad, but thinking makes it so."

~ William Shakespeare (1564-1616), Hamlet, II.ii

So . . . What is machine scoring

• Computer technology that evaluates and scores written prose (Shermis & Burstein, 2003)

• web-based • scores and responds in seconds.

Some Machine Scoring Software

Company Scoring Engine Technique Instructional Program

Scoring

ETS e-rater ® Natural Language Processing

Criterion Single holistic score

Pearson Intelligent Essay Assessor (IEA)

Latent Semantic Analysis

WriteToLearn Holistic and component scoring

Vantage Learning

Table adapted from Dikli (2006) Warschauer & Ware (2006)

IntelliMetric ™ Artificial Intelligence

MY Access! (and others)

Holistic and component scoring

How does it work?

Artificial Intelligence (AI)

Latent Semantic Analysis (LSA)

Natural Language Processing (NLP)

Recognized Machine Scoring Platforms

https://www.ets.org/criterion

Recognized Machine Scoring Platforms

Recognized Machine Scoring Platforms

Positive features

• Objective

• Consistent

• Broad in scope• High stakes testing environment

The trouble with humans

DESKILLING OF TEACHERS

AUDIENCE

INADEQUATE DESCRIPTION OF ELL ERRORS

BORROWING

SOCIAL ASPECTS OF WRITING

ONE DRAFT WONDERS

For a complete reference list, contact deborah.crusan@wright.edu