a look inside: scoring the ged language arts, writing essay · 2011-05-31 · language arts,...

A Look Inside: Scoring the GED Language Arts, Writing Essay

March 2011March, 2011

Marjorie WineAssociate Director, Assessment Services

Introduction

This presentation will walk you through common questions related to scoring the GED Language Arts Writing Essayrelated to scoring the GED Language Arts, Writing Essay, including:

What can we learn about scoring the GED Writing Essay?– What can we learn about scoring the GED Writing Essay?

– What does “holistic scoring” really mean?

– How do essay readers apply the GED Essay Scoring Rubric? Do they use it differently than teachers do in their classrooms?

– Is there any one thing that can make a test-taker’s response ineligible for the full range of scores?

In this presentation:

• A glossary of scoring terminologyThe Essay Prompt• The Essay Prompt

• The Language Arts Writing Test• Responding to the Prompt• Scoring the Essay• The 2002 Series GED Test Essay Scoring Rubric• A deeper look into score points and dimensions of writingA deeper look into score points and dimensions of writing• Sample Prompt Alpha• Annotations for responses in the Sample Prompt Alpha

Guide SetGuide Set


Glossary of Scoring Terminology

To help you understand the following discussion, please review the scoring terminology as it is used here:review the scoring terminology as it is used here:

• PromptA stimulus or topic designed to generate a written response from examinees.A stimulus or topic designed to generate a written response from examinees.

• ResponseAny writing generated by an examinee in response to a prompt.

• RangefindingThe process by which the range of score points is determined for each individual prompt. For each prompt, the Writing Advisory Committee reviews roughly 100prompt. For each prompt, the Writing Advisory Committee reviews roughly 100 responses, comparing each response to a Guide Set (see Training Prompt Alpha Guide Set) that is created by the GEDTS Language Arts Test Specialists. Based on the Guide Set, in conjunction with the Rubric, the WAC will define the range of each scorepoint (e.g., a high 1 vs. a low 2 or a typical 3 vs. a high 3). Th fi di i i t d d t d t i T S hThe rangefinding process is intended to determine True Scores on each response in the 100-response sample.



• True ScoreThe consensus score given to a response as determined by the WACThe consensus score given to a response, as determined by the WAC during rangefinding. If consensus on an individual response cannot be reached, the paper will be marked “Do Not Use” and will not be used in reader training.

• RubricA scale developed to provide holistic scoring guidelines to rangefinders andA scale developed to provide holistic scoring guidelines to rangefinders and response readers. The rubric should never be used alone to score a response, however. The rubric can only be used only in conjunction with the Guide Set.



• Training PapersSeveral sets of responses that have already been given True ScoresSeveral sets of responses that have already been given True Scores. These sets are used for reader training. Readers will have access to all of their training papers throughout live scoring. They will be encouraged to compare the responses to the training papers in order to determine the correct score. Because it is easier to compare a one response to another response, the readers are likely to have better agreement rates if they use their training papers to help them score, alongside the rubric, rather than using the rubric aloneusing the rubric alone.



• Guide SetA selection chosen from the previously rangefound responses ofA selection, chosen from the previously rangefound responses, of representative responses that reflect the range of all score points. The Guide Set will include 4-12 responses, with 1-3 responses per score point, demonstrating the range of each. Papers in the Guide Set are sometimes referred to as Anchor Papers, as they “anchor” scorers to the score points as described in the rubric.


The Essay Prompt: A Definition

The GED Essay Topic is really a promptprompt— i.e. it is a group of sentences that provides thesentences that provides the examinee with a starting point for writing a response.

No specialized knowledge is needed to respond. In order to provide guidance each prompt contains a sentence similar to the following:guidance, each prompt contains a sentence similar to the following:

“In your essay, use your personal observations, experience, and knowledge to support your essay.”


The Essay Prompt: The Vetting Processg

• Presents the test-taker with a single topicsingle topic.

• Is based on general knowledge.• Is chosen to be interesting to both g

writer and reader.• Is the same rhetorical type and

format as all other operationalformat as all other operational topics—expository.

• Has been field tested.• Has been analyzed by

psychometricians.


The Essay Prompt & Scoring

A response that shows any evidence that the p yexaminee has read the prompt designated in his or her answer document will not be deemed non-scorable and is eligible for the gfull range of scores.


The Essay Prompt & Scoring

• Examinees need not answer each and every word of a t i d f th i t iprompt in order for their essays to receive scores.

• The point of the prompt is to get examinees writing, not to assess how well they read the prompt. – Reading comprehension skills are tested elsewhere (i.e.,in the

Language Arts, Reading Test).

• While the ability to communicate ideas effectively is a key factor in the score a response receives, effectiveness at addressing the prompt is not one of the dimensions uponaddressing the prompt is not one of the dimensions upon which they are scored.


Language Arts, Writing Test, Part I

Test-takers must begin the test with the multiple-choice items. They will have 75 mins to work through the same processes they will usewill have 75 mins. to work through the same processes they will use in their essays to make corrections, revision, and construction shifts:

Surface Editing– Surface Editing– Organizing– Revising


Language Arts, Writing Test, Part II

After they complete the 75 minutes allotted for the multiple-choice section the test takers are directed to begin Part II The Essaysection, the test-takers are directed to begin Part II, The Essay.

They will have 45 minutes to:– Plan– Draft– Revise

their responses to the prompt.


How Readers Score Responses to the Prompt

• Readers acknowledge that an examinee’s response may take into account multiple definitions of wordsaccount multiple definitions of words.

• The reader understands that the writer had only 45 minutes to plan, d ft d i thdraft, and revise the response.

• …Therefore, the test-taker’s response is not a finished product, but a “good first draft.”


Some Considerations for Responding to the Prompt

• Occasionally, examinees misread or misinterpret the wording of the prompt Any response that displays that the examinee is in factprompt. Any response that displays that the examinee is, in fact, responding to the prompt designated on his/her answer document will be considered a valid attempt to respond to the prompt. Such misinterpretations will not be marked “off-topic.”misinterpretations will not be marked off topic.

• Although some prompts state, “Identify one opinion, problem, incident etc ” a response in which an examinee develops severalincident, etc., a response in which an examinee develops several opinions will be considered a valid response.

R l t dl i hi h i d t l• Relatedly, a response in which an examinee responds to only one or two sentences in a three-sentence prompt will also be considered valid.


Some Considerations for Responding to the Prompt

H h f th i ’ i t t bHowever, each of the examinee’s points must be developed with specific and relevant detail so that the reader can understand and follow his/her ideas.


What is holistic scoring?

“Holistic Scoring is a method of evaluating writing in which a reader judges a piece of writing for its overall effectiveness. No one

factor—neither grammar, organization, spelling, etc.—is weighted more than any other in determining the overall effectiveness of the

piece of writing (Conlan1976).”


The GED Essay Scoring

• Two certified GED® essay readers read and score each examinee’s responseexaminee s response.

• The two scores are averaged with the possibility of .5-point scores (e g a response that is scored as a 2 and a 3 by twoscores (e.g., a response that is scored as a 2 and a 3 by two readers respectively receives a final score of 2.5).

Th Chi f R d l i i• The Chief Reader resolves scores on responses receiving two non-adjacent scores (e.g., one reader gives the response a 2 and another gives it a 4).


How Readers Score – The Holistic Approach

• The GED Testing Service-trained reader reads the response to get an overall impression of an examinee’s response, not to tally specific errors in

t t t d h isentence structure, usage, grammar, and mechanics.

• The examinee must “build the bridges of communication” to the reader effective word choice, sentence, and paragraphing.

• The reader’s job is not to “build bridges” to the writer by making inferences or judgments about the writer’s intent. The reader must score the response that is in front of him/her.

This means that…• The reader’s impression is based solely on the examinee’s words on the

page(s).page(s).• The reader reads only the two pages of the answer sheet—no

additions.


The 2002 GED Test Series Essay Scoring Rubric


Characteristics of Each Score Point

Each score has a one-word descriptor:1-Score is INADEQUATE2 S i MARGINAL2-Score is MARGINAL3-Score is ADEQUATE4-Score is EFFECTIVE

…and a brief description of the overall impression responses at each score-point give: 1-Score: Reader has difficulty identifying or following the writer’s ideas.2 S R d i ll h diffi lt d t di f ll i th2-Score: Reader occasionally has difficulty understanding or following the writer’s ideas. 3-Score: Reader understands the writer’s ideas. 4-Score: Reader understands and easily follows the writer’s expression of idideas.

*Keep in mind that each score point describes a range of responses and can represent quite a wide variety of “writer-ly” approaches.


Classroom Scoring vs. Standardized Scoring

Scores *Approximate

“It doesn’t range of traditional letter grades

have to be an A+ to get a

gradesNon-scorable

Invalid response (blank, off-topic, non-English, illegible etc )A to get a

4!”illegible, etc.)

1 Not passing2 D- to C-3 C to B4 B+ to A+

*The ranges at each GED Essay score point do not g y pmap to traditional letter grades exactly.


The Five Dimensions

The Scoring Rubric also breaks down each score level into five dimensions that contribute to the reader’s overallinto five dimensions that contribute to the reader s overall impression. *Readers are trained to ensure that no one dimension is weighted more heavily than another.

• Response to the Prompt: How well does the examinee establish a main idea and maintain focus throughout the response?

• Organization: How effectively does the examinee order sentences and group g y g pideas into paragraphs to create a cohesive whole?

• Development and Details: Do the details and examples the examinee chooses to include support idea or further an argument?

• Conventions of Edited American English: How well does the examinee gcontrol sentence structure and demonstrate mastery of English language conventions?

• Word Choice: How varied, precise and appropriate are the words the examinee has chosen to include in the response?


The 4-point Response: Effective

Reader understands and easily follows the writer’s expression of ideas.

Response: Presents a clearly focused main ideathat shows evidence of addressing th tthe prompt.

Organization: Establishes a clear and logical organization.

Development: Achieves coherent development withDevelopment: Achieves coherent development with specific and relevant details and examples.

Conventions: Consistently controls sentenceConventions: Consistently controls sentence structure and the conventions of Edited American English (EAE).

Word Choice: Exhibits varied and precise word pchoice.


The 3-point Response: Adequate

Reader understands the writer’s ideas.

Response: Establishes a main idea that shows evidence of addressing the prompt.evidence of addressing the prompt.

Organization: Uses an identifiable organizational plan.

Development: Has focused but occasionally uneven p ydevelopment; incorporates some specific details.

Conventions: Generally controls sentence structureand the conventions of EAE.

Word Choice: Exhibits appropriate word choice.


The 2-point Response: Marginal

Reader occasionally has difficulty understanding or following the writer’s ideas.

Response: Shows evidence of addressing the prompt but the focus may shift or deviate from a central idea.

Organization: Shows some evidence of an organizational plan.

Development: Has some development but lacks specific details; may be limited to listing, repetitions, or generalizations.

Conventions: May demonstrate inconsistent controlof sentence structure and theof sentence structure and the conventions of EAE.

Word Choice: Exhibits a narrow range of word choice, often including inappropriate or vagueoften including inappropriate or vague selections.


The 1-point Response: Inadequate

Reader has difficulty identifying or following the writer’s ideas.

Response: Attempts to address the prompt but with little or no success in establishing a

ffocus.Organization: Fails to organize ideas.Development: Demonstrates little or no development;

usually lacks details orusually lacks details or examples or presents irrelevant information.Conventions: May exhibit minimal or no control ofConventions: May exhibit minimal or no control of

sentence structure and the conventions of EAE.Word Choice: Exhibits vague, general and/or g , g

inappropriate words choice


The Guide Set: Sample Topic Alpha

In this section:• Sample Prompt Alpha• Examples of test-taker responses written to Sample

Prompt Alphap p• Scoring annotations about each response


Sample Prompt Alpha


Score point 1


Response 1


Response 1 AnnotationTrue Score: 1

Alth h thi d t t th t it it d th t it d Although this response demonstrates that its writer read the prompt, it does not sustain focus for its duration. Rather, it jumps from one idea (“…I would want someone to aknowlege [sic] me…”) to another (“…discrimination is another…”).

The writing is not sustained long enough to establish much of an organizational structure or a grouping of ideas.

Included details are minimal and the writing sample contains several mechanical and spelling errors.

Although the writer has included a few appropriately chosen words (e.g., g pp p y ( g“aknowlege [sic],” “discrimination”), most of the response is composed of common, general words and, again, the writing is not sustained long enough to display the writer’s command of written English language.

This inadequate response received a score of 1.


Response 2


Response 2 Annotation

True Score: 1

This response clearly misinterprets the prompt, but because it demonstratesThis response clearly misinterprets the prompt, but because it demonstrates that the writer has, in fact, read the prompt and sustains the focus of the response based on that misinterpretation, it still received a score.

The strength of this response lies in its appropriate grouping of ideas andThe strength of this response lies in its appropriate grouping of ideas and well-paragraphed organizational structure.

However, each idea is minimally developed and contains only vague allusions to examples (e.g., “I’ve worked as a waitress for a restaurant...” andallusions to examples (e.g., I ve worked as a waitress for a restaurant... and “I’ve went [sic] to all kinds of businesses…”).

Although the response contains a few mistakes in conventions, these mistakes do not impede the reader’s overall understandingmistakes do not impede the reader s overall understanding.

The word choice is repetitive and lacks precision.

This inadequate response received a score of 1This inadequate response received a score of 1.


Score point 2


Response 3



True Score: 2

Although this writer does not sustain the response for very long, each sentence builds logically from the one before it, adding new ideas and information as it proceeds in order to make a brief but coherent argument.

It contains both an introductory sentence and a summative concluding sentence that provide a skeletal organizational structure.

Although it contains a few mechanical errors, they do not impede meaning.

The word choice is appropriate and occasionally quite precise (e.g., “outragous [sic]” and “appreciative”) the sample is not sustained longoutragous [sic] and appreciative ), the sample is not sustained long enough to demonstrate the writer’s fluency.

This marginal response received a score of 2.


Response 4



True Score: 2Thi t i it i id f it d ti ith littl This response sustains its main idea for its duration with little or no irrelevant information.

Each sentence brings in new information, building upon the idea(s)Each sentence brings in new information, building upon the idea(s) expressed in the last. The grouping of ideas is mostly appropriate.

Although this response contains some errors in grammatical conventions they are not so severe as to prevent the reader fromconventions, they are not so severe as to prevent the reader from understanding the writer’s ideas.

This response contains a mixture of general, repetitive terminologyThis response contains a mixture of general, repetitive terminology (e.g.,“…better communication leads to bigger and better things.”) and more specific word choice (e.g., “…dental insurance its [sic] a win win situation…”).



Response 5



In this response we see a clear thesis statement The writer maintains acute In this response, we see a clear thesis statement. The writer maintains acute focus on this thesis for the duration of the essay, thereby showing elements of higher-scoring responses.

The response makes good if rigid use of the “five paragraph essay” format The response makes good, if rigid, use of the five-paragraph essay format in order to organize ideas into paragraphs.

Although each paragraph develops unique ideas with relevant details, they are communicated using a mixture of general and specific language withare communicated using a mixture of general and specific language, with little variance in sentence structure (e.g., “This is…,” “That is…” of “Those are…” begin 8 different sentences).

There are relati el fe errors in spelling or mechanics thro gho t this There are relatively few errors in spelling or mechanics throughout this response, but it lacks the precision necessary for a higher score.



Score point 3


Response 6



One strength of this response is its clarity and sustained train of thought It does One strength of this response is its clarity and sustained train of thought. It does not deviate from its central premise at all.

Another strength is the correct application and precision of its word choice (e.g., “ it l t iti ki i t ” “i it bl d fi t ” “ f d tit d“vital to a positive working environment,” “irritable + defiant,” “newfound gratitude and respect,” etc.), particularly in light of a small number of errors in conventions that we see here.

However, although the details included are relevant, there is only one substantively developed point (e.g., paragraph 2). This writer’s skill in elaboration is sufficient, but not advanced.

Relatedly, even though we can see a clear introduction, body paragraph and conclusion, this response lacks enough development for this organizational structure to be employed to its full benefit.

This adequate response received a score of 3.


Response 7 (page 1)


Response 7 (page 2)



True Score: 3

This response is well organized with an introduction, a conclusion and two body paragraphs that each include some detail and elaboration upon each of the writer’s two main ideas.

Although most of the information included maintains focus on the response’s thesis statement (“One of these benefits that I would like to recieve[sic] from an employer would be health insurance.”), it expresses its id i i t f ifi ( “ d th t l f ”) dideas in a mixture of specific (e.g., “…spared the struggles of…”) and more general, repetitive language (e.g., “…to keep a good relationship between employer…”).

The grammar usage and spelling are generally correct throughout the response.

This adequate response received a score of 3This adequate response received a score of 3.


Response 8 (page 1)


Response 8 (page 2)



True Score: 3

This response brings in four key, focused ideas and organizes them effectively in service to its one main idea: “…there is more to a job than just the salary they receive.”

Although each of the four body paragraphs contain some elaboration supporting the paragraphs’ main ideas, the elaboration is general and does not contain specific examples or details.

Similarly, the word choice throughout this response is clear and easily understandable but is occasionally repetitive and lacks some precision.

Any usage or spelling errors are negligible and do not impede understanding.

This adequate response received a score of 3This adequate response received a score of 3.


Score point 4


Response 9 (page 1)


Response 9 (page 2)



Th i ti l t t f thi th fi h f t t The organizational strategy of this response uses the five-paragraph-essay format to good effect. Each body paragraph contains well-grouped ideas that progress from sentence to sentence, and the introductory and concluding paragraphs create an effective frame to the response.

Each sentence brings in new information that builds on the sentence previous. Each paragraph contains adequate elaboration and some specific detail.

Although the language is not consistently exact throughout the response, it contains some specific and effective phrasings (e.g., “not likely a computer will ever deliver a baby,” “excellent incentive,” etc.).

Also, sentence structure is varied throughout.

Grammatical errors are few and do not hinder understanding.Grammatical errors are few and do not hinder understanding.This effective response received a score of 4.


Response 10 (page 1)


Response 10 (page 2)



This response maintains clear focus on its central message and awareness of its di f it d ti ti lli taudience for its duration, creating a compelling argument.

Also, this response provides a strong example of how a writer can deviate from the five-paragraph-essay format without compromising organizational structurethe five-paragraph-essay format without compromising organizational structure. Rather, this writer’s effective argument brings in new details and ideas with every sentence in order to build to its central point.

Fluency of language is another strength of this response. Although the chosen vocabulary isn’t “flashy,” it is correctly applied and mostly precise. The sentences are fluid and lead logically into each other.

Errors in conventions are negligible and do not impair the reader’s understanding.

This effective response received a score of 4.This effective response received a score of 4.


Conclusion and Contact Information

Conclusion and Contact Information• Discussion and QuestionsDiscussion and Questions

• Contact for further information:– Marjorie Wine– Associate Director, Assessment Services

marjorie wine@ace nche edu– [email protected] – Office: (202)939-9474– Mobile: (202)870-1469


a look inside: scoring the ged language arts, writing essay · 2011-05-31 · language arts,...

Documents