Setting Cut Scores with WritePlacer®

Deanna L. Morgan

Edward Hardin

© 2009 The College Board. College Board, ACCUPLACER, Advanced Placement Program, AP, College-Level Examination Program, SAT, WritePlacer and the acorn logo are registered trademarks of the College Board. ACES, Admitted Class Evaluation Service and inspiring minds are trademarks owned by the College Board. PSAT/NMSQT is a registered trademark of the College Board and National Merit Scholarship Corporation. All other products and services may be trademarks of their respective owners. Visit the College Board on the Web: www.collegeboard.com

Deanna L. Morgan is an associate research scientist at the College Board.

Edward Hardin is an assessment manager for test development at the College Board.

The College Board is a not-for-profit membership association whose mission is to connect students to college success and opportunity. Founded in 1900, the College Board is composed of more than 5,600 schools, colleges, universities and other educational organizations. Each year, the College Board serves seven million students and their parents, 23,000 high schools, and 3,800 colleges through major programs and services in college readiness, college admissions, guidance, assessment, financial aid, enrollment, and teaching and learning. Among its best-known programs are the SAT®, the PSAT/NMSQT® and the Advanced Placement Program® (AP®). The College Board is committed to the principles of excellence and equity, and that commitment is embodied in all of its programs, services, activities and concerns.For further information, visit www.collegeboard.com.

ContentsIntroduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

Standard Setting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Who Participates in a Standard Setting Study?. . . . . . . . . . . . . . . . . . . . . . . . . . 3

The Facilitator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Subject-Matter Experts (SMEs) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Overview of the Standard Setting Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

The Authoritative Body . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

The Just Minimally Competent Examinee . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Performance Level Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

The Modified Body of Work Method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Training the SMEs on the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Judgment and Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Consequence or Impact Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

To Share or Not to Share Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Evaluation and Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Setting the Cut Score: Step by Step. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Step 1: Make Policy Decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Step 2: Reserve Facilities/Invite SMEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17

Step 3: Prepare Materials in Advance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

Step 4: Assign SMEs to Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Step 5: Introduction to the Purpose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

Step 6: Defining the Just Minimally Competent Examinee. . . . . . . . . . . . . . . . . . . . 20

Step 7: Training the SMEs on the Standard Setting Task. . . . . . . . . . . . . . . . . . . . . 21

Step 8: Round 1 of Range Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Step 9: Round 1 Small Group Discussion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

Step 10: Round 2 of Range Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Step 11: Round 2 Small Group Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Step 12: Round 2 Large Group Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24

Step 13: Round 2 Presentation of Impact Data (If Applicable)

and Preparing for Pinpointing Round . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

Step 14: Round 3 of Ratings—Pinpointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Step 15: Evaluating the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Step 16: Computing the Final Cut Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Step 17: Documenting the Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

Step 18. Provide Results to the Authoritative Body . . . . . . . . . . . . . . . . . . . . . . . . . . 31

Logistical Details for Consideration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Printed Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Support Staff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33

Data Spreadsheet. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Calculating the Cut Score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

Calculating the Standard Error of Judgment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

Appendix A: Information on WritePlacer® . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

Appendix B: A Sample Schedule for a Standard Setting Using a Modified Body of Work Process to Set One Cut Score . . . . . . . 41

Appendix C: A Sample Schedule for a Standard Setting Using a Modified Body of Work Process to Set Two Cut Scores . . . . . . 43

Appendix D: WritePlacer Sample Prompt for the Essays Provided in This Document . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Appendix E: WritePlacer Scoring Rubric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

Appendix F: WritePlacer Essay Training Set . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49

Appendix G: Training Record Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

Appendix H: Training Evaluation Form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54

Appendix I: Range-Finding Essay Set. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Appendix J: Range-Finding Round 1 Evaluation Form . . . . . . . . . . . . . . . . . . . 71

Appendix K: Modified Body of Work Essay Grouping Record— Range-Finding Round 1 and Round 2 . . . . . . . . . . . . . . . . . . . . . . 72

Appendix L: Official Score for Each Essay in the Provided Essay Set . . . . . 73

Appendix M: Pinpointing Essays for Round 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

Appendix N: Modified Body of Work Essay Grouping Record— Pinpointing Round 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

Appendix O: Final Evaluation Form for the WritePlacer Standard Setting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81

Appendix P: Dictionary of Terms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

Tables1. Example of a Simple Display of Impact Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2. Example of Spreadsheet Entry for One Group of SMEs . . . . . . . . . . . . . . . . . . . . 23

3. Example of Results Feedback to Panelists After Each Round. . . . . . . . . . . . . . . 23

4. Example of Round 2 Spreadsheet Entry for One Group of SMEs. . . . . . . . . . . . . 24

5. Summary of the Essay Groupings for the Total Group After

Round 2 of Range Finding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

6. Summary of Essay Placements for Total Group After Round 2 . . . . . . . . . . . . . . 26

7. Computing Temporary Cut Scores and Identifying Pinpointing Ranges . . . . . . 27

8. Score Table Sorted by Score, with Median Illustration . . . . . . . . . . . . . . . . . . . . 28

9. Probability the Examinee Will Be Successful in the Entry-Level Course . . . . . . 31

10. Example of SEM and SEJ Data Provided to Authoritative Body . . . . . . . . . . . . . 32

11. Example of Computing the Individual Cut Score for Each SME . . . . . . . . . . . . . 36

Figures1. Diagram of hypothetical placement decisions using two cut scores . . . . . . . . . 2

2. Translating the definition of just minimally competent examinee

to a cut score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3. A math example of performance level descriptors for use in a

standard setting study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

4. Example of finding the median score . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

1

IntroductionIn February 2009, the College Board ACCUPLACER® program introduced new scoring rubrics and new prompts for the WritePlacer® and WritePlacer ESL tests. The College Board recognizes that institutions1 may be eager to begin using the new writing scores in their placement decisions. This document includes materials that will facilitate the process to set preliminary cut scores using a nonoperational essay prompt and sample essays.Although a number of institutions have used the former WritePlacer and WritePlacer ESL tests in their placement procedures, the former rubrics and/or prompts will not be available after the transition to the new i3 platform. The new rubrics and prompts are different than before, and score points do not necessarily reflect the same level of performance as seen on the previous rubrics and prompts. The new rubrics on the i3 platform include a score scale ranging from 1 to 8 rather than the previous 2 to 12. The prompts are new to the ACCUPLACER program, having been retired from use with the SAT® Program. The new prompts, which are designed to stimulate critical thinking about an issue of significance, and which ask students to develop and support their own point of view on the issue, more closely resemble the tasks students are asked to perform in college-level writing classes. The previous prompts, by comparison, were less academically focused, typically asking students to imagine a day spent with a famous person or describe a place that was special to them. In keeping with this change in the prompts, the new rubric includes a critical thinking component that is absent from the previous rubric. Institutions are cautioned against using their existing cut scores for WritePlacer and WritePlacer ESL, with students testing on the new i3 platform due to a lack of comparability between scores from these two different rubrics.This document provides guidelines for establishing cut scores2 through standard setting on the WritePlacer and WritePlacer ESL tests. It gives guidance to institutions in using writing scores in their admissions and placement procedures before they are able to collect sufficient data on their own students. The College Board urges colleges to collect sample essays from their own students on the essay prompt of their choice to obtain the most accurate cut scores for their institution. It is strongly recommended that institutions collect data and after one year perform a validity study through the College Board’s free Admitted Class Evaluation Service™ (ACES™) to evaluate the functioning of their placement scores.Materials for use for setting initial cut scores with the WritePlacer test are provided in this document. Although no essays are provided for WritePlacer ESL, the process would be the same once the college has collected essays from their applicants for use in setting cut scores. Many of the other forms and supplemental materials could still be used with little or no modification.This document is organized into two sections. The first section provides general information on standard setting and the rationale behind some of the procedures and practices involved. The second session is a step-by-step guide to the standard setting process and provides more specific details about exactly how to conduct the standard setting study.

1. In this document, the word “institution” is used to refer to the broad assortment of higher education entities, e.g., colleges, universities, two-year/four-year, technical schools, etc.2. Italicized words appear in the dictionary in Appendix P.

2

Standard SettingStandard setting process is the name given to a set of methodologies that may be used to establish a cut score to separate examinees into adjacent groups or categories. Frequently, it is necessary to identify a single point on a test score scale where examinees below the point are considered to be members of one group, and examinees at or above the point are considered to be members of another group. An example of this may be the identification of a test score such that examinees scoring at or above the score are considered to have sufficient proficiency to be placed into the entry-level English course, and examinees scoring below the score are considered to lack proficiency for the entry-level course and need to be placed in a remedial English course. The set of knowledge or skills required to show proficiency in a course can be referred to as performance standards. The point on the score scale used to define or implement the performance standard in an objective manner is referred to as the cut score (Kane, 1994; Cizek & Bunch, 2007).It is important to note that the College Board, in agreement with the Standards for Educational and Psychological Testing (AERA/APA/NCME, 1999), recommends that decisions be made based on multiple sources of information and not a single test score. The standards state that “a decision or characterization that will have major impact on a student should not be made on the basis of a single test score” (p. 146).Some decisions, such as course placement, may need multiple cut scores. It may be desirable to separate students into multiple groups that are aligned to course placement. An example may be the use of two cut scores to separate students into those who should begin in the entry-level course, those who should begin in a remedial course, and those students who may be successful in a more advanced course (see Figure 1).Many standard setting methods exist to facilitate the identification of a test score to be used as a cut score. Cizek and Bunch (2007), Cizek (2001), Morgan and Michaelides (2005), and Hansche (1998) provide procedures and explanations for standard setting methods beyond the one elaborated in this document. Each standard setting method has both advantages and disadvantages that are specific to its use and methodology that should be carefully considered before choosing a method for use. This document elaborates on the Body of Work method (Kahl, Crockett, DePascale, & Rindfleisch, 1994, 1995; Kingston, Kahl, Sweeney, & Bay, 2001) due to the relative ease of use, the abundance of research on the method, the large precedence for its use, and its appropriateness for use with writing samples. The step-by-step guide will actually be a Modified Body of Work method due to the recognition that it may be outside the scope of some institutions to carry out a full study using logistic regression, as is typical for the complete Body of Work method. This document provides an alternative method to calculate cut scores to make the process more accessible to institutions.

106

105

59

58

Expected grade below C in the

entry-level course

Expected grade of C or better in the advanced course

20 120

Expected grade of C or better in the entry-

level course

Students may be successful inthe entry-level course

Students may bene�tfrom remediation

Students may be successful inan advanced course

Figure 1. Diagram of hypothetical placement decisions using two cut scores.

3

Who Participates in a Standard Setting Study?A standard setting study is one part of a more comprehensive standard setting process. This part, the study, has two key roles that must be filled for the standard setting process to proceed with minimum bias in the judgments of the subject-matter experts (SMEs) and maximum validity for the recommended cut score. The first role is that of an external facilitator to oversee the standard setting process and ensure that appropriate procedures are followed and documentation is maintained. The second role is that of the SMEs who, after sufficient and appropriate training, provide the judgments used to form the recommended cut score. Note that the outcome of a standard setting study is a recommendation that will be used by the Authoritative Body (described below), which has the responsibility of considering many pieces of information, including the recommendation, to establish the final standard.

The FacilitatorThe facilitator is a person with specific skills who is outside the process and does not have an immediate stake in the outcome of the standard setting. The facilitator may be, for example, the director of institutional testing at the institution, a faculty member in the department of Educational Psychology, or another individual familiar with the standard setting process who has the ability to train the SMEs to perform an unfamiliar process, elicit full participation from each participant, and ensure no one participant or set of participants dominates the process. The facilitator must not have an immediate stake in the outcome of the standard setting to make sure that the panel’s recommendation is, and appears, independent of the facilitator or Authoritative Body. The facilitator could be someone from the admissions or placement office. However, this may have the appearance of bias or lead to real bias in the process and results.The facilitator’s role is to ensure that the procedures and tasks that take place during the standard setting are performed as intended to maintain the validity of the process. This includes not only training the SMEs on the standard setting tasks they will be performing but also monitoring large and small group discussions and providing the SMEs with information that may need to be considered in the standard setting process. The facilitator should be familiar with standard setting and its requirements, and especially familiar with the specific standard setting method that will be used. The Modified Body of Work method is described fully in this document, but other methods are available. In addition to the facilitator’s not having a stake in the outcome of the standard setting, the facilitator should not be in a position of authority over the SMEs. The standard setting process is most likely not a process with which the SMEs are familiar. The process entails some training and involves a question-and-answer period where the participants need to feel free to express themselves. In addition, the process requires and encourages a large amount of discussion, and the SMEs must feel free to express their opinions. The facilitator should not provide any judgments during the process or interject any personal opinions that may influence the judgments of the SMEs.

Subject-Matter Experts (SMEs)The SMEs should be knowledgeable about the examinee population and the skills and knowledge that are required of students in relation to the decisions being made. Given that the purpose of the standard setting is to determine course placement in a specific content area, the SMEs should be experts in the content area under consideration.Ideally, the SMEs will be faculty members of the institution(s) that will use the resulting cut scores. The SMEs should be representative of the college or institution for which the placement decisions are being made. Representation should be considered in terms of gender, race/ethnicity, tenure (both new staff and veteran staff), and, in cases where the cut score may be intended for multiple campuses or locations, geographical location

4

and campus size. For example, if the cut score will be used systemwide, then representatives from around the system and not just the main campus should be included, and representatives from both two-year and four-year institutions are suggested. The more representative the panel of SMEs, the more generalizable and valid the results will be. The panel of SMEs should consist primarily of faculty currently teaching in the subject area. The standard setting study involves a period of training, and the rest of the process should not be undertaken until all participants have had their questions answered and have established a level of understanding sufficient to perform the duties with confidence and competence. As the process encourages discussion and interactions, the panel should have at least 10 members for the purposes of representation but no more than 20 to allow every panelist to participate and contribute.

5

Overview of the Standard Setting ProcessThe following sections will present information about the process of standard setting. Standard setting has a strong tie to policy, and many decisions will need to be considered before the start of the standard setting process. These policy decisions (e.g., how many cut scores are needed, whether to provide impact data, etc.) must be made before beginning the standard setting process, to avoid the risk of making a decision based solely on the outcome of the process and therefore introducing bias into the results. Following the overview, a step-by-step guide is presented to aid users in conducting a standard setting at their institution.

The Authoritative BodyThe SMEs convened for the standard setting are content experts and serve to provide a recommendation based primarily on content for the placement of the cut score. The final cut score that will be adopted for use must be approved by someone with the authority to make policy decisions. The Authoritative Body makes many decisions through the course of the standard setting process, including which method to use, what the makeup of the panel will be, who will facilitate, what data will or will not be provided to the panel, the criteria for making final decisions, and ultimately the final cut score(s). Though the Authoritative Body makes several decisions related to the standard setting, it should not participate in the actual standard setting session. The Authoritative Body may be a single person but generally is a small group of people. The identity of the Authoritative Body will differ based on policy and procedure at each institution. Some possibilities for Authoritative Body members include the director of admissions, the president of the institution, the board of regents, the vice president of academic affairs, or some combination of those listed. Each institution will have to determine what person or group would best fit its situation. The key point in deciding the identity of the Authoritative Body is ensuring that the person(s) have the authority and knowledge to make the final decision on the location of the cut score(s).

The Just Minimally Competent ExamineeThe objective of the standard setting process is to identify the point on the score scale that separates examinees who meet the specified qualifications from those who do not. In the case of WritePlacer®, this may be conceptualized as those examinees who exhibit sufficient proficiency in writing to be placed into the entry-level English course, and those who do not exhibit sufficient proficiency to be placed into the entry-level English course and therefore may need remediation. This may be shortened to think of those who meet or exceed expectations and those who do not meet expectations. Generally, each group (those who meet or exceed and those who do not meet) contains examinees that belong in that specific group, for example, very low scorers or very high scorers. However, each group will also contain a number of examinees who either exhibit just enough proficiency to be placed in the entry-level course or who lack proficiency, but just barely, to keep them from being placed in the entry-level course. Identifying the point on the score scale that signifies just enough proficiency for placement in the entry-level course is the task of the SMEs. Students who exhibit just enough proficiency for placement in the entry-level course are known as just minimally competent examinees. When setting a cut score, the SMEs should make all decisions with the just minimally competent examinee in mind. Part of the training and discussion entails developing and understanding the definition and description of the just minimally competent examinees, described below.Placement decisions often require that multiple cut scores be determined. The number of cut scores needed is a policy decision. In the example given earlier in Figure 1, students are being placed into one of three courses: remedial, entry level, or advanced. For this example, two cut scores are needed. The lower cut score separates students who are just minimally qualified for the entry-level course from those who do not qualify for the entry-level course and may need remediation. The higher cut score separates students who are just minimally qualified for the advanced course from those who do not qualify for the advanced course. Figure 2 provides a visual of the

6

ability continuum for writing and the location on the continuum of the just minimally competent examinee. The next section discusses the definition of students who belong in each category.

The ability continuum is not a number line but represents verbal descriptions of the writing skills students are capable of demonstrating as they progress from limited or no skills to highly proficient. Once the descrip-

tion has been derived, the process of translating the descriptions and performance level descriptors associated with the just minimally competent examinee to a test score can begin. This example is for one cut score but

can be adapted to the multiple cut score process. (Melican, personal communication, 2009).

Ability Continuum for Writing

Candidates far away from having

requisite skills. Of little or no

interest in study.

Slightly below minimal

competence.

Just minimally competent.

This is a group in need of definition of study.

Candidates with more than enough

requisite skills. Of little or no

interest in study.

Figure 2. Translating the definition of just minimally competent examinee to a cut score.

SMEs bring a diverse set of experiences with students and courses and opinions into the standard setting process. While this diversity increases the generalizability of the standard setting results, it may also introduce a variation in initial definitions of the just minimally competent examinee. Consider, for instance, a group of faculty members teaching the same course at a college or university. The course may be the same and may use the same curriculum and materials, but it is not uncommon for the requirements necessary to earn a grade of A from one professor to differ slightly or dramatically from the requirements necessary to earn a grade of A from another professor. Therefore, it is likely that when asked to think of the just minimally competent examinee, each SME will picture this hypothetical person differently. As a result, it is recommended that before the task of standard setting, the SMEs create a set of performance level descriptors that will guide the process and serve to calibrate the SMEs before the standard setting task.

Performance Level DescriptorsThe type of performance level descriptors needed is determined by the purpose each institution has identified for the cut scores. The facilitator should begin this task by giving SMEs a brief introduction of the goal of the standard setting process, an overview of the test (sample essay question, overview of the scoring rubric used for the essay, etc.), the intended use of the resultant cut score(s), and an overview of the approval/adoption process that will be conducted once the SMEs have made their recommendation.3 Once the SMEs have been informed of their role in the process and the goal of the session, the facilitator should ask the SMEs to verbalize the characteristics in terms of knowledge and skills in writing that a just minimally competent examinee would be expected to exhibit. It is often helpful to conduct this as a brainstorming session by recording all ideas until few additions are forthcoming and then going through the list one idea at time, asking the SMEs to consider how relevant it is to a just minimally competent examinee.In the case of placement decisions, multiple cut scores may be needed to separate students into groups for each course level under consideration. Therefore, multiple performance level descriptors will be necessary. Begin by considering the situation where all entering students are placed into the entry-level course. Some of these students will do very well and may have possessed sufficient knowledge and skills that they could have taken a more advanced course rather than the entry-level course and still been successful. Other students will be 3. A complete list of the training steps is included in a later section.

7

successful in the entry-level course but probably would not have been successful had they been placed into a more advanced course. The third group of students is composed of those students who were unsuccessful in the entry-level course because they did not possess sufficient knowledge and skills to succeed and would have benefited from remediation in a lower-level course. The institution may be interested in identifying the cut scores that will separate students into these three groups to allow differentiated instruction where needed.The creation of performance level descriptors for placement decisions must begin with a policy decision for how the institution defines success in a course. Typically, success is defined in terms of the grade achieved in the course. For the scenario described above and illustrated in Figure 1, two cut scores will be needed: the cut score representative of the just minimally competent student who is successful in the entry-level course, and the cut score representing the just minimally competent student who is successful in the advanced course.How should success be defined in each of these cases? Would a student receiving a grade of C in the entry-level course be considered successful, or must the student receive a grade of B? The choice of using a grade of C or a grade of B is a policy decision. At some institutions, the decision may be to place the cut score using a grade of A or a grade below C to define success. However, typically a grade of B or a grade of C is chosen to define success. Consider the case where the definition of success is a grade of C or better in the entry-level course. This would signify that students earning scores that are similar to those earned by students who later took the course and were successful at the C level or higher would be placed into the entry-level course, and students earning scores similar to those of students who later took the course but were not successful (defined as earning a grade below C) would be placed into a remedial level course. To help the SMEs clarify the difference in students who are just minimally competent for placement into the entry-level course and those who lack competence for placement into the entry-level course, the SMEs should be asked to develop a performance level descriptor for each group of students: what skills students who are successful (final grade of C or higher) in the course possess when they enter the course and what they know, versus what skills are possessed at entry by students who are unsuccessful (final grade below C) in the course and what the unsuccessful students would or would not be expected to know.The second cut score would also require that a policy decision be made as to the level of success needed to be placed into an advanced course. The level of success should be in terms of performance in the advanced class. Typically, this cut score would also be defined as the score that identifies a student who is just minimally competent to earn a grade of C in the advanced course. Institutions may have policy reasons for using a grade other than C to define success in the advanced course. It is important that the decision for the level of success be defined and communicated to the SMEs so that they can create appropriate performance level descriptors for that level of success.

The set of performance level descriptors should:Describe what students at each level should reasonably know and be able to do.••

Relate directly to the content standards, course prerequisites, and course requirements.••

Distinguish clearly from one level (remedial course) to the next (entry-level course).••

Be written in positive terms.••

Be written in clear and concise language without using nonmeasurable qualifiers such as often, seldom, ••thorough, frequently, limited, etc.

Focus on achievement.••

The performance level descriptors that are derived by the SMEs will be a major part of the documentation provided to the Authoritative Body and in the final report.Figure 3 provides an example of a set of performance level descriptors that may be appropriate for use in setting cut scores for a mathematics test. Mathematics was chosen so as not to influence any institution’s deliberations

8

about the specific characteristics of student writing appropriate at each performance level for their specific situation. Notice that while considerable similarity exists between the performance level descriptors at each grade level, the level of complexity or abstraction varies. For this reason, an exact one-to-one correspondence between the performance level descriptors at each level is not necessary. Five statements may be needed to accurately describe the performance at one grade level, while four or even six statements may be more appropriate at another grade level. Of primary importance is developing a set of performance level descriptors that the SMEs are comfortable with and believe they can work with during the standard setting process. For additional information on writing performance level descriptors and to see other examples that have been used in standard setting studies, see Perie (2008), Cizek and Bunch (2007), Hambleton (2001), or Hansche (1998).Once the performance level descriptors have been composed, containing only the knowledge or skills that are considered essential for an examinee to be considered just minimally competent, ask the panelists to indicate whether they feel the performance levels are adequately defined to allow them to differentiate between levels during the standard setting task. Once a general consensus is reached, the list should be posted in a spot visible to all SMEs, or copied and distributed to each SME for reference during the standard setting process. A copy should also be maintained in a folder or binder for reference and documentation in the event that anyone should ever question the institution’s use of a specific cut score. The final set of performance level descriptors provides the meaning in words of the numeric cut score that will be set, and adds to the validity of the standard setting process and the resultant cut score (Perie, 2008; Hambleton, 2001).After generating definitions and training in a large group, we suggest breaking into smaller groups of five to seven SMEs for the next steps while the panelists begin making their judgments. In a full panel of 20 members, this would create three to four subpanels. This provides an opportunity for cross validation of the recommended cut score between the groups and may encourage active participation by more panelists who may feel less inhibited in a small group setting.

9

The Modified Body of Work MethodIn the Body of Work method (Kahl, Crockett, DePascale, & Rindfleisch, 1994, 1995; Kingston, Kahl, Sweeney, & Bay, 2001), panelists examine complete sets of student work, including responses to both dichotomously (e.g., multiple-choice items that are scored correct or incorrect) and polytomously scored items (e.g., essay questions that are scored in more than two categories). Panelists review each sample of student work and sort it into a performance category based upon its match to the performance level descriptors (PLDs). The process proceeds in two stages: range-finding and pinpointing.

Range-Finding

Minimally Competent A-Level StudentsThey have a strong conceptual understanding and mathematical ability in algebra and geometry.•They have the ability to solve and model problems in real-world and contextual situations effectively.•They can synthesize concepts, processes, and procedures to solve complex and nonroutine problems in contextual and •real-world settings.They are able to make connections between abstractions and concrete situations.•They are able to use several components of their understanding at the same time. They can break down and keep track of the •individual components that build toward the final answer. They will give the completed answer in the proper form if there is one.They can identify and apply efficient or insightful methods of solution.•

Minimally Competent B-Level StudentsThey are able to work at an abstract level of understanding.•They can synthesize concepts, processes, and procedures, but usually only within the realm of routine problems.•They can work with and interpret algebraic and geometric models and have the ability to construct a model for a real-world •or contextual situation.They have enough conceptual understanding to solve new problems that aren’t complex; that is, they can solve new problems •that don’t have many constituent parts.They are likely to make a careless mistake along the way if the problem involves many parts, and they may make a few •mistakes when the level of abstraction in increased.Although they know how to solve a problem, they do not always apply efficient or insightful methods of solution.•

Minimally Competent C-Level StudentsThey can, and do, learn the fundamental definitions and theorems, maybe from memorization or by repetition of exercises, but •haven’t made the connections to the understanding, the analysis, of the concept.They demonstrate proficiency with simple procedures and algorithms.•They are memorizers. They can usually only solve routine problems.•Most modeling by C students has probably been shown to them by someone else. They would have difficulty creating •their own models.They have difficulty with multistep problems because they lack the necessary skills to work through the problems without •making a mistake. A weak background in algebra can cause problems even in those students who have a good understanding of new concepts. Many times students are simply unable to recognize the constituent parts.

Minimally Competent D-Level StudentsTheir use of processes is at a concrete numerical skill level.•They experience difficulty with even simple abstractions, procedures, and algorithms.•They can only work problems they have seen others work several times before, and have practiced solving similar problems.•They do not fully understand the definitions of basic concepts.•They do not detect errors in their solutions.•

Figure 3. A math example of performance level descriptors for use in a standard setting study.

10

The score scale for WritePlacer extends from 1 to 8, but it is unlikely that the cut scores will be set at very high or very low points on the scale. The first stage, therefore, is to find the area on the scale where the cut scores are likely to occur. Once this more narrow range has been identified, the final cut score(s) are identified in the pinpointing round. A small sample of student performance samples across the range of possible scores is used as a range-finding activity to narrow down the approximate locations for where the cut scores should be placed. Discussion follows each round of the range-finding stage, and panelists have the opportunity to revisit their range-finding decisions. It is important that panelists agree on the range under consideration for the placement of the cut score before the next stage, pinpointing, begins.

Pinpointing

Using the more narrowly defined range, sample student booklets are chosen to represent every score point between the lowest possible score in this range and the highest possible score in this range. Although several approaches may be used to select the next round of essays, most users provide an equal number of essays to judge at each score point in the defined range. Approximately four samples are chosen for each score point in the range. Panelists are then asked to work on one cut score at a time and sort student essays into one of the two performance categories surrounding the cut score. If multiple cut scores are needed and the score scale range is relatively small, such that the identified range for two adjacent cut scores overlap, then the panelist may find it less confusing to work on both cut scores in the range simultaneously by sorting the essays into multiple piles. The final cut score recommendations presented to the Authoritative Body will be computed as the median score separating the adjacent groups of essays.An advantage of the Body of Work method is the relatively simple task of assigning student booklets to performance groups and the fact that panelists are working with real student responses. Evaluating student performance is a task most classroom instructors have experience with and feel comfortable doing, so the task is less abstract than that associated with other standard setting methods. However, sufficient time should be allowed due to the amount of preparation required, the number of essays the SMEs will need to read, and the large quantities of student work available from which sufficient numbers of test-taker examples at every score point under consideration will be extracted and provided to the SMEs. However, this is a solid method for tests that are primarily performance based.

Training the SMEs on the ProcessThe SMEs should be trained on the goals of the standard setting and the intended use of the cut score to be recommended. It is also critical that the SMEs be thoroughly trained and have sufficient practice with the process to be used in the standard setting. The SMEs should be given an overview of the process, which includes the tasks they will be performing and the steps that will occur after their recommendation but before the cut score will be considered final. To avoid any potential feelings of having wasted their time or being deceived about the outcome or goal of the process, it is important that the SMEs be made aware if their recommendation may be adjusted by an Authoritative Body. Often a simple explanation of why their recommendation could be adjusted—that focuses on the ability of the Authoritative Body to take into consideration additional information and impact data—can minimize such feelings or complaints. This is also a good time to ensure that the SMEs know that the materials used in the standard setting and the discussions that take place during the standard setting are considered confidential and that no materials should leave the room. This is less important using the sample prompt and essays enclosed in this document (see Appendix I) because they have been fully disclosed and are not used operationally, but reasons may still exist that would preclude any information from being divulged about the session. If using an existing prompt on the i3 system, and student essays written to and scored for that prompt, it is imperative to keep the essays in a secure location to avoid the prompt and responses from becoming part of the public domain, and therefore no longer useful for testing.

11

The training of the SMEs on the process should include an opportunity for the SMEs to take the exam and to practice using the process. Taking the exam gives the SMEs, who are more accustomed to giving exams than taking them, the opportunity to step into the examinees’ shoes and see what it feels like to take the exam. Essay topics may look much more accessible at first glance than when trying to write an actual response without preparation. Administering the exam to the SMEs familiarizes them with the essay prompt and instructions and provides them with a less biased view of the true difficulty of the essay writing task. If possible, the SMEs should take the test under the same time constraints and conditions as the examinees. As the SMEs complete the exam, the facilitator should distribute the scoring rubric and allow them to assess their own work. Scores should not be collected, but it is a good idea to give the SMEs the opportunity to discuss any points that they felt were overly difficult or questionable. It should be noted that the scoring rubric is not under discussion. If an SME has a suggestion concerning the rubrics, then the SME should be encouraged to provide the comments to the facilitator another time. The rubrics should be used as is for the rest of the standard setting process. Their comments will be provided to the ACCUPLACER® program for evaluation and discussion.The SMEs need the opportunity to practice using the Body of Work method before performing the task operationally. The task may seem simple when explained by the facilitator, but questions often arise when the SMEs try to implement the process. These questions must be addressed before the start of the operational standard setting. To provide the practice opportunity, compile a small set of student essay responses that are similar to those being used in the operational standard setting but not part of the standard setting set. A practice set of essays to be used in the training is provided in Appendix F. These essays should be copied so that only one appears on each page, and the scoring rubric should be provided to the SMEs if it has not been provided previously. Ask the SMEs to separate the essays into two piles representing entry level and remedial using the process discussed in the training. Then ask the SMEs to record the grouping for each essay using the training record form in Appendix G. The goal of the standard setting may be to set multiple cut scores, but this task is to simply allow SMEs to practice the process before moving on to the operational standard setting task.When all SMEs have completed the task, ask for three to five volunteers to share their essay groupings and explain their reasoning. This will allow the facilitator to get an idea of whether the SMEs are providing the ratings as instructed and will also allow the group of SMEs to get an indication of how their ratings may or may not be similar to those of the group. The practice opportunity and the small debriefing, as the SME volunteers share their groupings, should identify any questions or areas of confusion with the process that need to be resolved before the operational standard setting. At this point, the SMEs should complete an evaluation form (Appendix H) to allow the facilitator to check on their level of understanding and comfort with the task and to provide another piece of documentation as to the validity of the standard setting process and resulting cut score (Kane, 1994). If a panelist indicates that he or she is not ready, then it is important to talk with that panelist to identify the area of concern and attempt to alleviate it. It is certainly possible, if not likely, that others in the group may have similar concerns that were not voiced. Therefore, it is suggested that the information be shared with the larger group as well. Another evaluation form should be provided so that everyone indicates that they are ready to move forward before continuing with the process.

Judgment and DiscussionWhen the performance level descriptors have been written and distributed, training is complete, and all SMEs indicate readiness to proceed to the standard setting task, the actual process of making judgments about the essays and the correspondence of essay performance to the performance level descriptors begins. The actual standard setting task is an iterative process alternating between independent judgment and group discussion. The task typically includes three rounds, with independent judgment and group discussion occurring within the first two rounds and the final round including independent judgment only. Some standard setting studies have used only two rounds or as many as four or five. However, it is the opinion of this author that three rounds is optimal unless SME evaluation forms indicate a lack of understanding and the need for retraining on the process to occur.

12

In the case of retraining, it is not uncommon to add an additional round of judgment and discussion to provide ample opportunity for the panelists to make judgments with the new and hopefully improved understanding of the task. An experienced facilitator will be able to make the call on whether an additional standard setting round should be performed. Zieky, Petrie, and Livingston (2008) offer additional discussion on deciding how many rounds of iteration should occur.The first iteration should begin with asking all SMEs to evaluate the essay sets and sort the essays into the performance level group that they feel the essay best represents. The sorting task should be completed independently by each SME without discussion or the presence of any information related to the score of an essay or the group placement by any other SME. This will be the first time the SME has closely read and evaluated the essay set, and it is important that it be as unbiased by the opinions of others as possible. When all SMEs have completed the sorting task and submitted their placements, it is a good time to schedule a break to allow time for the data to be entered. Scheduling this first round of judgment to occur at the end of the first day is ideal because it allows the data to be entered (see Table 2 later in this document) and checked for accuracy overnight, and feedback on the ratings to be prepared for the discussion portion of this first iteration.Discussion for the first round may occur in small groups, if being used, or in the larger group. At the beginning of the discussion, the summary of results from the first round of judgments should be shared. See Table 3 later in this document for an example of how the results may be displayed. The results summary provides a starting point for discussion and helps to focus the discussion on the strengths and weaknesses identified in the essays reviewed, as well as on the rationale of SMEs who placed the essay into one group versus another. It is helpful to identify essays with the most discrepant placements to start the discussion, and to always refer SMEs back to the performance level descriptors as they discuss group placements. For example, if essay #237 was placed into the lowest group by a portion of SMEs, the middle group by another portion of SMEs, and the highest group by a third portion, it would be instructive to hear a rationale from someone placing the essay into each group so that the entire group of SMEs is able to hear the reasoning behind the placements and possibly identify areas of strength or weakness in the essay that may have been viewed from a different perspective and therefore resulted in a different placement. It is important to have many different perspectives discussed to ensure careful consideration of each essay. This discussion will clarify how the essays relate to the performance level descriptors. Discussion should be mediated in such a way as to allow all the SMEs the freedom to express their ideas and opinions.The second iteration begins with another round of the sorting task and should again be completed independently. At this point, the SMEs have the option to keep each essay in the group into which it was originally placed during the sorting task of the first round, or to change the group placement based on additional review and consideration of the perspectives shared in the first round of discussion. Be certain that SMEs understand that although they may change the group placement for any and all essays, they are not required to do so.The second round of discussion would begin in the same manner as the first round, with participants sharing a summary of the results from the first round of sorting and giving the SMEs the opportunity to discuss any discrepant essay placements or essays that they feel are particularly difficult to classify. If working in small groups, it may be preferable that the discussion occur in two phases. Phase 1 would be the small group discussion and phase 2 would bring all the small groups back together to share in a large group discussion before beginning the third iteration of the sorting task. Following the round 2 discussion, if it had been decided by the Authoritative Body to provide impact data to the SMEs, impact data would be provided. The next section discusses the use of impact data.The third round of iteration comprises only the sorting task and is the final opportunity for SMEs, based on discussion and impact data (if any was provided), to adjust their placement of essays into groups. Again, the SMEs may change any and all essay groupings or they may choose to make no changes if they feel confident of their prior placements. It is based on this third and final round of the sorting task that the final cut score recommended to the Authoritative Body will be computed. This third round may be followed by a short debriefing discussion for feedback and evaluation, but for all intents and purposes, the standard setting process is complete once the SMEs submit the ratings record from this round of placements.

13

Consequence or Impact DataOnce the final round of standard setting is complete, there will be a recommended cut score or cut scores. It is suggested that if data will be provided to the SMEs on the impact of the recommended cut score, they should be provided before the last round of the sorting task. Before the provision of the impact data, the standard setting session will have focused on student performance and skills and how much skill a just minimally competent examinee would be able to demonstrate. The introduction of impact data to the process serves to quantify the results of the standard setting process in terms of the effect that the recommended cut score would have on examinees. It is natural in this context for the SMEs to be concerned if the failure or placement rates look very different from previous expectations. However, it is important that the SMEs remember all the work that has gone into the identification of a cut score to that point and to consider what any major changes in their ratings would mean in terms of the skills that students would be expected to possess and demonstrate. Are the SMEs able to justify accepting a lower level of performance to increase the passing or placement rate? Are the SMEs able to justify expecting a higher level of performance to lower the passing or placement rate? The facilitator is responsible for reminding the SMEs that this is only one more piece of information and should be considered along with the earlier performance and skill discussions. The SMEs should also be reminded that they will have the opportunity to revise their ratings one more time (in round 3) before the completion of the standard setting, if they feel the need to do so.Impact data may take several forms, and it is a policy decision as to which form it will take, if presented. The ability to present impact data depends on the availability of performance data. The simplest form of impact data is the total number of students plus the percentage of examinees that would be placed into each course, based on the panel’s recommended cut score following the most recent round of standard setting (typically round 2). For simplicity only, assume that the recommended round 2 cut score for placement into the entry-level course corresponds to a score of 6 on the WritePlacer test.4 The percentage of students at the institution(s) scoring 6 or higher would be presented to the SMEs along with the corresponding percentage of students scoring below 6 as an indication of the impact of the cut score staying in the current location. This may also be broken down and reported by relevant subgroups to the SMEs, e.g., gender, race/ethnicity, disability, English as a second language, etc. (see Table 1). Based on this table of hypothetical data, if the cut score based on the round 2 essay placements were implemented, it would be expected that 67 percent of examinees would place into the entry-level course and 33 percent would place into the remedial course. It is a policy decision to determine if the impact data are presented and to determine how much information to present, and for which subgroups, if any.

Table 1Example of a Simple Display of Impact Data

Group Percentages

Total Caucasian African American Asian Hispanic Male Female

Number of Examinees 6,000 2,370 2,130 623 877 2,353 3,647

Entry-Level Course 67% 71% 65% 82% 63% 58% 73%

Remedial Course 33% 29% 35% 18% 37% 42% 27%

Total 100% 100% 100% 100% 100% 100% 100%

4. The number 6 is a random number chosen for explaining the process and should not be taken as a default value or as a recommendation of the College Board. The cut scores each institution will derive will be idiosyncratic.

14

If the intent of setting a cut score on WritePlacer is to establish a score to be used in placement decisions, it is recommended that data be collected for a group of students representative of the institution’s student population that contains both the WritePlacer i3 score on the new rubric and prompt and the score on an alternate measure of writing (such as the WritePlacer on the 2.0 platform using the prior rubric and set of prompts, if this is already in use at the institution for each student). These data will allow the SMEs the opportunity to determine if the group of students placed using the cut score under consideration for WritePlacer will be similar in number and demographics to the group of students who have been placed into courses in the past using the alternate measure of writing proficiency. Although this information is important to aid the SMEs in evaluating the validity of using the cut score under consideration for the WritePlacer for their institution, this should be presented to the SMEs only after the second round of the standard setting process so the impact is tempered by the confidence of the SMEs in their evaluations so far.It should be noted that the use of impact data in standard setting is controversial. Whether or not to use impact data is a policy decision and should be made locally at each institution before the start of the standard setting session. Many experts feel strongly that impact data should be shared with the SMEs to put their ratings into context (Zieky, Perie, & Livingston, 2008). Other experts feel strongly that the introduction of the impact data will bias the results and should be avoided (Reckase, 2001). Empirical data as to the amount of impact these data have on the subsequent actions of panelists are sparse, and most evidence is anecdotal in nature. While this document will discuss the presentation of impact data for the information of those institutions that may choose to provide impact data, readers should be cautioned that this decision should be made locally by the Authoritative Body after careful consideration of the issue. The inclusion of the procedures to use impact data does not constitute a recommendation to do so.

To Share or Not to Share ResultsAt the completion of the standard setting session, the SMEs will have developed a vested interest in what the outcome of the session will be. During the session, the SMEs were provided feedback as to the location of the recommended cut score. It is common that some of the SMEs will also feel strongly about knowing the location of the final cut score that comes out of the round 3 ratings and will be presented to the Authoritative Body. Whether or not to share the location of the round 3 cut score is a policy decision. It is customary that the materials used and the discussions that occur during a standard setting are considered confidential for reasons of test security. However, this is less of a concern when using the materials provided in this manual since all materials have previously been released to the public and are not currently operational on the i3 system. For schools using an existing, operational WritePlacer prompt and real student essays, security of the materials and confidentiality of the discussions should be strictly maintained to minimize the risk of compromising test scores through release of proprietary information. However, when a recommended cut score is going to be reviewed by an Authoritative Body before being adopted for use and the possibility exists that the cut score could be modified by the Authoritative Body, it is sometimes preferable that the SMEs not be aware of the final outcome. This can help to prevent any potential bad feelings on behalf of the SMEs when they know the cut score was modified and by how much.It is important that the SMEs be made aware of the potential for modification when the cut score goes to the Authoritative Body, no matter what the decision is on whether the round 3 cut score will be shared, but it is essential if the round 3 cut score is to be shared. It is also important to inform the SMEs that the Authoritative Body will have additional information for use in its decision of whether to adopt the cut score as it is, or to make modifications, and will not just arbitrarily decide to change the cut score.

Evaluation and DocumentationThe SMEs should be asked to complete an evaluation form (Appendixes H, J, and O) for the standard setting session after the training and/or after round 1 of ratings, as well as at the conclusion of round 3. The first rating provides feedback on the panelists’ confidence with moving forward with the standard setting task and provides

15

an opportunity for retraining if indicated by the evaluations. The final evaluation provides feedback to the facilitator on how the process may be improved in the future and provides evidence of how the SMEs view the standard setting session and the resulting cut scores. The evaluation results should be summarized and included in the materials to be presented to the Authoritative Body. Strong ratings of confidence in the methods used and the resulting cut score provide support for keeping the recommended cut score as it came out of the standard setting session. Low ratings of confidence and understanding in the methods provide support for modifying the cut score from the standard setting session, or giving more weight to additional information being considered along with the cut score recommendation. Of course, if low ratings of confidence and understanding are noted during the process, there should be an immediate effort to review the process again and retrain panelists. This may also necessitate adding an additional round of judgment depending on where in the process these ratings are noticed. The decision to retrain or add a round of ratings should be well documented. A knowledgeable facilitator will be able to decide whether additional training and rounds of rating are needed. The original evaluation forms should be kept as documentation.Documentation is critical to being able to defend the standard setting process and the resulting cut score if a legal challenge is made. Documentation should include at least one complete set of all materials and handouts that the SMEs used or had available during the standard setting session. The names and contact information for the SMEs, along with a summary of their demographic characteristics, should also be retained for documentation. The completed evaluation and item rating forms should be kept. The completed rating forms can be used to go back and verify that calculations were done correctly. The forms also can be used to determine the variation or similarity of the SMEs’ responses. It is good practice to summarize the activities and outcomes of the standard setting in a final technical report as part of the documentation. The technical report should include a thorough description of the procedures and time lines; a copy of the individual group results shared after each round and the large group results shared after round 2; a summary of any impact data provided; a summary of the evaluation form results after the training and at the conclusion of the session; a summary of any information provided to the Authoritative Body; and the outcome of the Authoritative Body with appropriate rationale.

16

Setting the Cut Score: Step by StepThe following sections are intended to provide a step-by-step guide to conducting a Modified Body of Work standard setting method. Where necessary, additional detail will be provided, but whenever possible, the reader will be referred back to previous areas of this document for specific details. Examples of the agendas for both a one cut and a two cut standard setting session are provided in Appendixes B and C to give an idea of the amount of time that is needed for the standard setting process. In this step-by-step section of the paper, the following steps will be further discussed:

Make Policy Decisions1.

Reserve Facilities/Invite SMEs2.

Prepare Materials in Advance3.

Assign SMEs to Groups4.

Introduction to the Purpose5.

Defining the Just Minimally Competent Examinee6.

Training the SMEs on the Standard Setting Task7.

Round 1 of Range Finding8.

Round 1 Small Group Discussion9.

Round 2 of Range Finding10.

Round 2 Small Group Discussion11.

Round 2 Large Group Discussion12.

Round 2 Presentation of Impact Data and Preparing for Pinpointing Round13.

Round 3 of Ratings—Pinpointing14.

Evaluating the Process15.

Comparing the Final Cut Score16.

Documenting the Process17.

Provide Results to the Authoritative Body18.

Step 1: Make Policy Decisions

The purpose of this section is to outline the decisions and steps that the Authoritative Body should undertake before the method has been determined or the panel has been convened. The Authoritative Body should carefully consider the implications and make final decisions on the following issues before the start of the standard setting session:

What is the purpose of the standard setting session?••

How many cut scores will be needed?••

17

Who will be the facilitator (faculty member, testing director, outside consultant)?••

What are the qualifications required of the SMEs?••

What level of proficiency should SMEs keep in mind when creating the performance level descriptors ••and setting cut scores (grade of B, grade of C, other)?

Will the performance level descriptors be considered confidential or will SMEs be able to take this ••information with them when they leave the session?

Will impact data be presented?••

How much impact data will be presented and for which subgroups?•▶

How will the results of the postimpact data recommendations from the panel be presented to and •▶used by the decision-makers?

Will the final round 3 cut score be shared with the SMEs?••

These decisions should be clearly documented and communicated to the facilitator before the start of the standard setting session.

Step 2: Reserve Facilities/Invite SMEs

It is important that the SMEs have a large space in which they can spread out and that will be reasonably comfortable for the time needed to complete the standard setting session. A large room with tables and chairs that can be moved and grouped as needed is ideal. If the cut score will be used systemwide across multiple campuses, arrangements may need to be made for lodging of out-of-town participants. It is also a good idea to make arrangements for snacks or meals that will be provided. This is typically the first step in preparing for a standard setting because confirming the availability of suitable space on a particular set of dates must be accomplished to enable sufficient information to be passed on to the SMEs when trying to obtain a commitment from them to be available.

Identify the number of participants and the number of small groups that will meet.••

Allow adequate work space for each panelist to use during the independent work periods.••

Allow space for observers in an unobtrusive location.••

Determine if additional rooms are needed for the small group meetings or if the small groups can meet ••in the large room without creating a noise issue.

One of the most important steps in the standard setting is identifying, inviting, and obtaining a commitment from the SMEs. Typically between 15 and 30 SMEs will be needed, and as the standard setting time approaches, attrition will often occur. Therefore, it is wise to invite the maximum number of SMEs or to include a few extras to ensure that at least 15 are available on the actual dates of the standard setting.The Authoritative Body should determine the criteria required to ensure a representative sample of SMEs. Geographic location (e.g., campus), disciplines, teachers of introductory classes or advanced classes, years of experience, gender, and ethnicity are possible criteria. The knowledge that is necessary should be identified. A letter to deans and department heads may be sent soliciting nominations, stating the intentions of the standard setting, and listing the knowledge required.If the cut score will be applicable only to your campus, then it may be necessary to include all SMEs on the campus to reach the minimum number. If the cut score will be applicable systemwide or at multiple campuses,

18

then recommendations for representatives from each campus should be solicited from administration at each campus, e.g., the relevant dean of the college for that content area. Soliciting recommendations allows you to indicate on the invitation that each SME was chosen for the honor of participation, which sometimes helps to obtain the ultimate commitment from the SME. Be sure to provide ample notice between the invitation and the date of the standard setting, and always ask for a reply confirming acceptance to assist you in determining if additional SMEs need to be invited.

Determine the number needed for the study and a reasonable oversampling rate to allow for attrition.••

Determine target demographic makeup of panel.••

Solicit recommendations from relevant groups and stakeholders.••

Send invitations over the signature of the Authoritative Body explaining the nature of the study and the ••dates. Request a quick reply.

Follow up with nonrespondents.••

Follow up with all respondents and thank them all. For positive respondents, remind them of dates ••and send reminders at reasonable points, generally with additional information about location, accommodations, information about WritePlacer, etc.

Step 3: Prepare Materials in Advance

The purpose of this section is to demonstrate the extreme need for all materials to be defined, prepared, and reviewed long before the panel is convened. The amount of paper and the need for security and efficient handling of the materials place an important burden on the facilitator during the meeting, so all efforts to facilitate the logistics of the study should be made. During preparation, make changes in the logistics of the paper flow to ensure that everything is available on time and everything is accounted for at each point in the process.It is important that all materials and data be collected before the start of the standard setting process. Sample rating forms and evaluation forms are included in the appendixes. Feel free to customize the forms to meet your needs. The essay sets should be used as they are presented in the manual and not edited in any way. That may result in a change to the score that would have been assigned. The materials include:

A copy for each SME of the set of student essays to be evaluated (Appendix I), prepared so that only one ••essay appears on each page. Essays should also appear in random order so they are not grouped by the actual score the essay obtained (i.e., should not be all the scores of 1, followed by all the scores of 2, etc.).

A copy of the prompt as it appeared to the examinees. (This will be a straight copy of the form provided ••in Appendix D for the sample prompt used in this manual. If you have collected and are using your own essay prompt and essays, then consider having the panelists take the essay online under the same conditions as the examinees). Reminder: The SMEs will take the exam as if they are test-takers, so the scoring rubric must not be available until this task is completed. The SMEs will score their own exams, and only they will know the results. If computers are not available, then panelists may complete the essay by hand and should be allowed the same amount of time provided to an examinee. Another option is to set up an ACCUPLACER site for the standard setting that will administer the prompt being used for standard setting, and have the SMEs complete the essay online as part of the training on the first day of the standard setting. Alternatively, the panelists could be required to complete the essay at home just before coming to the standard setting meeting.

The record forms (see Appendixes K and N) will be needed for each SME to record the group into which ••they placed each essay.

19

The separate set of essays that are supplied in Appendix F for the purpose of training will be needed for ••each panelist. These should be prepared one to a page with the scoring rubric provided as part of the set (see Appendix E).

Impact data, if the decision has been made to provide this information. The data should include a ••frequency distribution of student scores on the WritePlacer exam.

The Excel spreadsheets to be used to enter the panelists’ ratings. The formulas should be entered and the ••spreadsheets tested with dummy data before the study occurs.

Compile the materials into tabbed, numbered binders (or other packet of your choice) in the order listed below, giving consideration to the details covered in the “Logical Details for Consideration” section of this document.

. Agenda (see Appendixes B and C for examples). A

. Information on the purpose of the standard setting.B

. Information on the test (see Appendix A).C

. Training materials (overheads, handouts).D

. WritePlacer prompt and blank paper to write the essay (see Appendix D), if computer administration of E the prompt is not possible.

. WritePlacer rubric (see Appendix E).F

. Training set of essays (see Appendix F and copy one essay to a page).G

. Training record form (see Appendix G).H

. Training evaluation form (see Appendix H).I

. Range-finding set of essays with five to seven essays per score point as determined by the i3 system (or as J designated by the provided guide for the sample set in the manual—see Appendix I for sample set).

. Round 1 range-finding evaluation form (see Appendix J).K

. Range-finding rating form (see Appendix K)—the same form can be used for both rounds of range L finding by instructing SMEs to cross out the rating for any essay whose grouping they wish to change. Then write the new grouping beside the crossed-out one on the blank. A similar form will need to be created on-site for pinpointing once the number of essays that will be included in the pinpointing round and the essay ID numbers that will be included have been determined.

. Guide to the official score each essay obtained on the i3 electronic scoring system (see Appendix L for M the sample set). When SMEs use essays that were collected on a different prompt, they should collect the score each response received.

. Set of pinpointing essays for round 3 on the sample prompt (see Appendix M).N

. Pinpointing rating form (see Appendix N).O

. Final evaluation form (see Appendix O).P

20

Step 4: Assign SMEs to Groups

Dividing the SMEs into small groups for most discussion serves to provide multiple comparison groups for validation purposes and encourages all SMEs to take part in discussions rather than having a limited number of SMEs dominate.After the SMEs have signed into the standard setting study, assign the SMEs to groups of five to seven people. They will join each time there is a need. The standard setting process will focus on independent input from the individual, and interaction will occur in both large and small groups. Training and some discussion will occur in the large group, most discussion will occur in the small groups, and the individual ratings of essays should occur independently by each SME.Each SME should be assigned a unique identification number to be used to identify all their materials. The ID number should appear on their rating form along with a number identifying to which group each SME belongs. It is advisable that all the materials the SMEs use in the standard setting process also carry their unique ID number to help ensure that all materials have been returned. Requiring that the SMEs use the same assigned number on all their materials will also help the facilitator to identify which SMEs have not returned their materials. More tips for handling materials and SMEs can be found in the “Logistical Details for Consideration” section.

Step 5: Introduction to the Purpose

The panelists will inevitably have many questions that they will bring to the study. Why are we here? Why was I and why were my colleagues chosen, and how? Do we have the skills necessary since we have never performed a study before? The standard setting should begin with an introduction to the purpose of the standard setting and a broad overview of the work to be completed during the task. This is also the appropriate time to address any security, confidentiality, and logistical concerns. Representatives from the Authoritative Body may want to use a few moments of this time to express their gratitude for the participation and expertise of the SMEs. However, any representatives from the Authoritative Body should leave promptly after offering their gratitude to avoid any undue influence on the process. Representatives from the Authoritative Body may check in occasionally to see how the session is progressing but should not speak to the SMEs or in any way offer an opinion that may influence the process. Any visits by the Authoritative Body should be kept to a minimum.The introduction should include basic information on the WritePlacer test (see Appendix A) and an explanation for how the cut scores will be used. If the use is for placement purposes, then there should be a brief description of the courses involved, along with any prerequisites or other information that would be relevant for consideration.

Step 6: Defining the Just Minimally Competent Examinee

The definition of the just minimally competent examinee is one of the first tasks in the standard setting process. This task sets the stage for the remainder of the tasks in the process and gives the SMEs an initial opportunity to become familiar with each other. Every effort should be made to encourage participation from all SMEs. It is a good strategy to introduce the concept to the large group of SMEs and then ask the small groups to work independently on the brainstorming activity to produce a list of characteristics of the just minimally competent examinee (see the previous sections in this document, “The Just Minimally Competent Examinee” and

“Performance Level Descriptors,” on pages 5 and 6, respectively). Once the small groups have each developed a list, these lists should be shared with the large group and decisions made collaboratively on the final list of characteristics for use in the set of performance level descriptors. Once a final list is developed, it should be made readily visible or copied and distributed to the SMEs for reference in the remainder of the standard setting session.

21

Step 7: Training the SMEs on the Standard Setting Task

The intention of this task is to have each SME understand the definition of the just minimally competent examinee; the factors that affect the evaluation of student performance and interpretation of the rubric; and the standard setting method and process. The test-takers will be administered one of the prompts, and their responses will be scored according to the existing rubrics. This fact is important, because as the SMEs make their judgments about the performance of real test-takers, they must make sure to use the same rubrics that are used to rate the test-takers. The SMEs may not use scoring rubrics of their own.Once the performance level descriptors are finalized, training should begin on the scoring rubric for the WritePlacer exam and on the specifics of the standard setting tasks. Begin with the SMEs taking the examination under the same time constraints and conditions as those used in an operational administration (see “Training the SMEs on the Process” on page 10). As the SMEs finish the test, hand them copies of the scoring rubric to use in checking their work. When everyone is finished, or when the allotted time has concluded, allow the SMEs to discuss any parts of the prompt or essay writing process that they may have found to be overly difficult or confusing. It is important that the SMEs understand, and they may need to be reminded, that the purpose of the study is to establish a recommendation for a cut score(s) using the existing test and rubrics, not to change or edit the test or rubrics. However, do allow panelists an opportunity to submit feedback on the test and rubrics in writing to be passed on to the ACCUPLACER program for future consideration.When all SMEs are familiar with the test, begin the training on the method. The facilitator should explain the standard setting task very carefully and be prepared to repeat the explanation often. Refer the SMEs back to the definition of the just minimally competent examinee and remind them that any decisions they make should be made with this group of examinees in mind.The facilitator must be prepared to address the most common concern expressed by SMEs. The SMEs will immediately observe that the standard setting process is arbitrary. In fact, it is a matter of professional judgment that is performed in an objective and replicable manner with training, practice, and discussion. The size of the panel reflects the Authoritative Body’s understanding that SMEs will have different experiences and understanding of just minimal competence because of these experiences. Their individual results are important, but the collective knowledge is even more so. This is a good time to emphasize how their results will be used by the Authoritative Body.The SMEs should begin by rating each essay according to the scoring rubric. When the panelists have scored the essays in their set and all questions from the SMEs have been addressed, a practice opportunity should be provided. Using the training set of essays (see Appendix F), ask the SMEs to independently place each essay into one of two classes: entry-level or remedial. When this has been completed, ask for volunteers to share the essays they placed into each class. If discrepancies exist between the groupings, ask for volunteers to explain their rationale for the way they grouped the essays, especially for essays that were grouped differently from how the majority of panelists had grouped them. The rationales should focus on what skills the just minimally competent examinee demonstrated with the essay. What skill or lack of skill is related to producing the given response? Others in the group may also want to join the discussion. The facilitator should allow enough discussion to address any misunderstandings or concerns and remind the SMEs that this was practice and to please ask any questions they may have about the process. The facilitator should be careful not to express agreement or disagreement with any of the ratings provided. All facilitator comments should be on the process and not evaluate the actual groupings. When all questions are answered, the training evaluation form should be distributed (see Appendix H). When the SMEs have completed their forms, the facilitator should quickly review the forms to determine if more training on the process is necessary.

22

Step 8: Round 1 of Range Finding

When training is complete, the actual task of evaluating essays and identifying the cut score will begin. The SMEs should have available to them the following materials:

A copy of the full numbered set of range-finding essays ordered randomly (not by score) with a copy of ••the scoring rubric.

A record form (see Appendix K) to record their placements for each essay.••

A copy in hand or posted in sight of the performance level descriptors of the just minimally competent ••examinee at each cut score that were developed earlier in the session.

The SMEs are expected to independently place each essay into a group and to work at their own pace. Instruct the SMEs to turn in their rating forms once they are finished so that data entry may begin. When finished, the SMEs should either sit quietly while the others finish, or they may leave the room. It is ideal to time the session so that lunch, a snack break, or the end of the day occurs at the time the SMEs are expected to finish to allow time for data entry of results and preparation for the next round. Data entry typically takes longest for round 1 because all essay groupings must be entered for each panelist. In subsequent rounds, the essay placements can be copied and pasted into the spreadsheet for that round, and only the items whose ratings were changed will need to be edited.

Step 9: Round 1 Small Group Discussion

When all essay assignments have been entered into a spreadsheet by each SME’s ID number and group number (see Table 2), summarize the number of panelists placing each essay into each category and share with the group (see Table 3). If this discussion is performed in one large group of participants, the summary for the entire panel may be displayed on an overhead projector or computer display screen so that everyone can reference the results during the discussion. Alternatively, for small group or large group discussions, the results may be summarized by group and distributed separately to each group so that SMEs receive only the results for their group members. The essay record sheet for each panelist should be redistributed to each SME so they can reference how they placed each essay during round 1. Each group, whether one large or multiple small, should review the essay placements and discuss the essays with the lowest agreement. For example, if 40 percent of panelists place the essay response in the lower category and 60 percent place the essay response in the higher category, there will be disagreement about this response. The facilitator will encourage panelists to share their rationale for placing the essay into one group as opposed to another. It is important that the groups be allowed sufficient time to compare results within the group and to discuss the ratings and their rationales. No discussion of results should occur between groups at this time. Of course, this instruction can be ignored should the decision be made to only work with the large group.The purpose of this discussion is to ensure that the SMEs apply the rubrics and the PLDs appropriately in their view of each essay response. Sometimes an SME will not realize that a particular phrasing of the prompt would be difficult for some students to react to, or may find that one student may believe an aspect of writing is highly important, and another student may de-emphasize this aspect and decide to be more lenient, and for that reason place the essay into a different category. The reverse is possible as well. The facilitator should emphasize that the panelists have an opportunity but not a requirement to change their ratings for any essay or essays if they believe that there is cause to do so. There should be no effort to lobby an SME to change the responses. If an SME appears to have developed an agenda or has decided to change the definitions of the just minimally competent examinee, the facilitator may want to remind the entire group of SMEs of the performance level definitions that were developed earlier in the process and to use those as a guide to thinking about student performance and skills.

23

Table 2 Example of Spreadsheet Entry for One Group of SMEs

Round 1 Ratings - Group 1

SMEs in Group 1

Essay ID SME1

SME2

SME3

SME4

SME5

1 R R R R R

2 E E A E A

3 A A A E E

4 E R R A E

5 R E E R R

6 A E E E E

. . . . . .

. . . . . .

. . . . . .

38 R R R E R

39 E E E A E

40 E R R R R

R—Remedial E—Entry Level A—Advanced

Step 10: Round 2 of Range Finding

When all groups believe they have had sufficient opportunity to discuss and compare results from round 1, the SMEs are given the opportunity to revise their ratings from the first round. Typically this second round of ratings will not take as long as the first round, but the SMEs should still assign the ratings independently and work at their own pace. As mentioned above, before the start of the round, advise the SMEs that they should feel free to change any and all ratings from round 1, to keep the same ratings they assigned in round 1, or to revise some ratings while keeping others the same. The decision of which essay ratings and how many, if any, should be at the sole discretion of each SME. When the ratings are completed, the SMEs should again hand in the rating forms for data entry. See Table 4 for an example of when panelists finish round 2.

Table 3 Example of Results Feedback to Panelists After Each Round

Essay ID Number

Number of Panelists1 Classifying Each Essay into Each Group

RemedialCourse

Entry-Level Course

AdvancedCourse

11 25

22 20 5

5 7 18

14 2 22 1

9 1 23 1

35 15 10

2 8 17

28 2 23

Note: Assumes 25 panelists participating and two cut scores being set.

24

Step 11: Round 2 Small Group Discussion

When all the ratings have been entered into the spreadsheet for round 2, the essay summary table should be prepared for each group. If using one large group, this will be one table and you can move forward to Step 12: Round 2 Large Group Discussion. If using small groups, it will be important to prepare a summary for each group but to also prepare a summary for the total group to be shared following the small group discussions. The small groups should have sufficient time for discussion, and each group should be asked to write a short summary of the discussions that have occurred within the group to share with the others in the room during the large group discussion.

Step 12: Round 2 Large Group Discussion

At the completion of the small group discussions for round 2, the facilitator should address the large group and ask that each small group provide a brief summary of the discussions that occurred in the group. This should be the first time that the groups are aware of the ratings of groups outside their own. Keeping discussion in the small groups until just before the third and final round of ratings provides the opportunity to collect multiple cut score recommendations within the large group to be used as validation of the final recommended cut score.It is important to remember and to remind the SMEs that there is no known right choice and that the input of all groups is important. Being an outlier group does not mean that the group is wrong, but it is important that the reason for the discrepancy be shared and discussed in case one or more groups have missed an important perspective that may have resulted in the discrepancy. The facilitator must work hard to make sure that all groups are heard and that no one group dominates the discussion. The facilitator must also be careful not to insert his or her opinion into the discussion.

Table 4 Example of Round 2 Spreadsheet Entry for One Group of SMEs

Round 2 Ratings—Group 1

SMEs in Group 1

Essay ID SME 1 SME 2 SME 3 SME 4 SME 5

1 R R R R R

2 E E A A A

3 E A A E E

4 E R R A E

5 R E R R R

6 A E E E E

. . . . . .

. . . . . .

. . . . . .

38 R R R E R

39 E A E A E

40 E R R R E

R—Remedial E—Entry-Level A—Advanced

Note: Bold type indicates values that the SME changed in round 2.

25

Step 13: Round 2 Presentation of Impact Data (If Applicable) and Preparing for Pinpointing Round

Recall that before the standard setting, the decision of whether to present impact data should have been made along with the determination of how much data and for which subgroups. The impact data should be presented in reference to the temporary cut score after round 2 that is calculated on all SMEs. Using the examples provided in Table 5 through Table 8, the temporary cut score is computed by summarizing the data for the total group following round 2 (See Tables 5 and 6). Table 5 gives an example of a spreadsheet with as many rows as essays and as many columns as SMEs where the group into which each SME placed each essay is recorded by inserting the letter A for advanced, E for entry-level, or R for remedial into each cell. Table 6 displays the summary information for how many SMEs placed each essay into each group. For example, 25 SMEs placed essay #1 in the category representing the advanced course, while 5 SMES placed essay #4 in the category representing the remedial course and another 20 SMEs placed essay #4 in the category representing the entry-level course. For each essay, record the official score the essay received and the category or categories into which the essay was placed (See Table 7). For example, in Table 7, essay #3 has a score of 7 and SMEs placed it into both the entry-level and advanced categories, so both category columns contain an X for that essay ID number.

Table 5 Summary of the Essay Groupings for the Total Group After Round 2 of Range Finding

All SMEs

Essay ID SME 1 SME 2 … SME 24 SME 25

1 A A … A A

2 R R … R R

3 E A … E E

4 E R … E E

5 E E … A A

6 A E … E E

. . . … . .

. . . … . .

. . . … . .

38 R R … R R

39 R R … R R

40 E E … A E

R—Remedial E—Entry Level A—AdvancedNote: Assumes 25 panelists participating and two cut scores being set.

Sort the essay score and placement data by score from the lowest score (1) to the highest score (8) and note the minimum essay score that was placed into each category and the maximum essay score that was placed into each category (see Table 8). The minimum and maximum scores in each category will determine the score range that should be represented in the essay sets used during the pinpointing round. The lowest, or minimum, score placed in the entry-level category is a 3, and the highest, or maximum, score placed in the remedial category is a 5. Therefore, the cut score between remedial and entry-level is most likely between 3 and 5 in this made-up example. The set of essays that will be evaluated in the pinpointing round should contain multiple essays at score points 3, 4, and 5. Before pulling essays for pinpointing, share the range of scores from which essays will be pulled, and if panelists feel that the range is too narrow, then also pull a few essays at the adjacent scores of 2 and/or 6 depending on the panelist feedback. Approximately 3–5 essays at each score point should be sufficient for pinpointing, with 1–2 essays at the outer ranges of 2 and/or 6 if the decision is made to extend the range.

26

Table 6 Summary of Essay Placements for Total Group After Round 2

Number of Panelists Classifying Each Essay into Each Group

Essay ID Number Remedial Course Entry-Level Course Advanced Course

1 25

2 25

3 7 18

4 5 20

5 3 22

. . . .

. . . .

. . . .

38 25

39 25

40 14 11

Note: Assumes 25 panelists participating and two cut scores being set.

Because the exact range needed for pinpointing is unknown until midprocess, it is important to have extra essays available at all score points and access to a copier so that a set of essays for each panelist can be prepared in a timely manner. Ideally, this would be timed so that the break between round 2 of range finding and the pinpointing round occurs at the lunch break to allow extra preparation time. Repeat the process for each cut score so that a separate set of essays is available for each cut. Ideally, pinpointing would focus on only one cut score at a time, with separate essays for each round. If, however, you find that the pinpointing ranges for two cut scores overlap, then pulling a set of essays that will encompass both ranges and separating them into the three categories at the same time (as in range finding) is acceptable.Using the same information that was used to find the ranges (Table 8), find the median (center) score for the essays in all but the lowest category. This score should be used as the temporary cut score for providing impact data. In locating the median (middle) score, it is important to count starting with the lowest score (1) and count the number of X’s such that all X’s across both adjacent categories are accounted for before moving to the X’s in the next score point. Using Table 8 as an example, the count for the remedial and entry-level categories would begin with the three X’s representing score 1, then the six X’s representing score 2, then the five X’s representing score 3, the nine X’s representing score 4, the six X’s representing score 5, the seven X’s representing score 6, the five X’s representing score 7, and the one X representing score 8. This would give a total of 42 X’s, with the median falling between the 21st and 22nd X’s, which both represent a score of 4. Therefore, the median is 4 and the temporary cut score between remedial and entry level would be 4 as shown in Table 8, and the temporary cut score between entry-level and advanced would be located between the 17th and 18th X’s, resulting in a median of 6.5 See Figure 4 for additional information on finding the median.

5. The scores of 4 and 6 used in the example were calculated using data created for example only and should not be considered a recommendation for the placement of the cut score.

27

Table 7Computing Temporary Cut Scores and Identifying Pinpointing Ranges

Essay ID Score Remedial Entry-Level Advanced

1 8 X

2 2 X

3 7 X X

4 4 X X

5 7 X X

6 2 X

7 6 X X

8 5 X

9 1 X

10 2 X

11 5 X X

12 3 X

13 4 X

14 6 X

15 7 X X

16 8 X X

17 1 X

18 5 X

19 2 X

20 6 X

21 7 X X

22 5 X

23 3 X

24 4 X X

25 6 X

26 2 X

27 4 X

28 6 X

29 6 X

30 3 X X

31 1 X

32 8 X

33 2 X

34 6 X X

35 7 X X

36 4 X X

37 5 X

38 4 X

39 3 X

40 6 X X

28

Table 8 Score Table Sorted by Score, with Median Illustration

Essay ID Score Remedial Entry-Level Advanced

9 1 X

17 1 X

31 1 X

2 2 X

6 2 X

10 2 X

19 2 X

26 2 X

33 2 X

12 3 X

23 3 X

30 3 X X

39 3 X

4 4 X X

13 4 X

24 4 X X

27 4 X

36 4 X X

38 4 X

8 5 X

11 5 X X

18 5 X

22 5 X

37 5 X

7 6 X X

14 6 X

20 6 X

25 6 X

28 6 X

29 6 X

34 6 X X

40 6 X X

3 7 X X

5 7 X X

15 7 X X

21 7 X X

35 7 X X

1 8 X

16 8 X X

32 8 X

Min Score 1 3 6

Max Score 5 8 8

Median 4 6

21st

17th

22nd

18th

29

Depending on the decision made by the Authoritative Body, if this was a cut score to be used for placement, the impact data may include the number or percentage of examinees at the institution who scored between 4 and 6 and would be placed into the entry-level course, the number or percentage of examinees at the institution who scored below 4 and would be placed in a remedial course, and the number or percentage of examinees who scored 6 or above and would be placed in the advanced course. Additional information by subgroup may include the number of males and females in each group, and relevant ethnic or racial subgroups (see Table 1).

Step 14: Round 3 of Ratings—Pinpointing

Following the small and large group discussions of round 2 and the presentation of impact data, if any, the SMEs have one additional opportunity to revise their ratings from the previous rounds. This round will use a new but smaller set of essays focused around the range(s) identified in Table 8 as the likely location of the cut score. It is important to provide panelists with a set of essays that includes essays from the cut scores under consideration. This allows SMEs to really define where the cut score is without being distracted by essays that have already been determined to be outside the cut score range during the previous two rounds of range finding. The results of round 3 will produce the final cut score recommendation.When setting one cut score and therefore working with only two categories (for example, remedial and entry-level), it is easier for the panelists to focus on the task of really narrowing down how one category differs from the other and to which category essays belong. When setting multiple cut scores, selecting the pinpointing range can be complicated, especially with the small score range (1–8) found with WritePlacer. Ideally, the pinpointing ranges would not overlap, enabling the creation of two totally separate sets of essays for pinpointing. The essay sets for each cut score would be evaluated separately, one cut score at a time. Sometimes this is not possible, and it is necessary to continue working with both cut scores simultaneously, as was done in the range-finding round. The resultant range of essays used in the pinpointing round will at least be more targeted at specific cut scores than those used in the range-finding, which must encompass the entire score scale. Also, the fact that the set of essays are new to the panelists and not the same material they have seen previously will help them to think of the categories from a fresh perspective and better define the placement of the cut scores.

Step 15: Evaluating the Process

The validity of the interpretation of the cut score is related to how well the SMEs understood and applied the standard setting methodology. Their confidence in their training and their ability to make the judgments is an important part of the documentation. The evaluation of the cut score process and resultant performance standards should be designed into the cut score study. Evaluation should occur frequently at different stages of the process and ask panelists to give an indication of their level of understanding and confidence in the process, as well as to provide feedback on the type of information they found useful in their decision making. Evaluations serve two purposes: (1) to provide feedback on the clarity of the training and the level of the panelists’ understanding, and (2) to determine the panelists’ level of satisfaction with the process and final cut score, which is an important piece of evidence for establishing the validity of performance standards (Hambleton, 2001). Initial feedback should be given following the training session and reviewed before the next stage in the process so that any misunderstanding or confusion can be addressed before the panelists make any operational cut score recommendations.The frequency with which the evaluation occurs can vary. At a minimum, evaluation should occur following training (see Appendix H) and at the end of the cut score session (see Appendix O). It is recommended that evaluation also occur after the first round of cut score recommendations (see Appendix J), and sometimes it may be desirable to evaluate panelists’ understanding following each round of cut score recommendations, especially in high-stakes situations. Then, a final evaluation (see Appendix O) should be given at the end of the process to document panelists’ comfort level with the outcome (Hambleton, Jaeger, Plake, & Mills, 2000).

30

Step 16: Computing the Final Cut Score

Traditionally, the cut score with the Body of Work method is computed using logistic regression. The slope and intercept are computed and then a basic formula is applied to determine the probability that an essay will be placed into each of the two categories, which will be separated by the cut score. Typically the cut score will be identified as the place where the essay has a 0.50 probability of being placed into both categories. However, if the Authoritative Body has judged it to be more detrimental to place students who are not prepared into the upper category or vice versa, then a different probability may be used. The facilitator or a statistician in your local research or mathematics department can help you with these calculations and decisions.In the event that logistic regression is not considered a possibility, the final recommended cut score can be computed using the same method used in preparing for the pinpointing round. The steps for calculating the final cut score using the method described in the pinpointing round are delineated further on pages 34–35.

Step 17: Documenting the Process

Throughout the cut score process, consideration should be given to the type of documentation that should be maintained. In the event that the cut score recommendations are ever challenged, the cut score study documentation is the evidence of what occurred and of what the panelists recommended. Documentation includes the plan for the cut score study, any scripts used, the materials given to panelists, any slide show presentation given, panelists’ ratings, panelists’ evaluations of the process and the resultant cut scores, the impact data that were presented to the panelists, and data used to create any other materials used in the cut score session, such as score distributions and any item difficulty estimates that may have been used for item ordering. The documentation provides evidence to support the validity of the cut scores. Kane (1994) provides two guidelines for examining the validity of performance standards: (1) that the cut score corresponds to the specified performance standard, and (2) that the specified performance standard is reasonable given the purpose of the decision.It is standard procedure to create a technical report following the cut score session that describes the procedures and summarizes panelists’ ratings and evaluations, as well as a summary of panelists’ comments provided on the evaluation forms. The technical report should summarize the impact data, and provide the standard errors of judgment (SEJs) for each cut score and the standard errors of measurement (SEMs) for the test. It is good practice to provide the final cut score recommendations along with values representing +/- 2 SEJs and +/- 2 SEMs. Along with the resultant cuts, it is helpful to provide estimates of the percentages of students in each performance category based on the cut scores +/- 2 SEJs and +/- 2 SEMs for the total population and possibly for any subgroups of interest.

Standard Error of Judgment (SEJ)The standard error of judgment is a measure of the degree of variability among the judges’ ratings and typically will decrease after each round of iteration, though the magnitude of the decrease generally will become smaller with each round and typically reaches a plateau following the third round of ratings. Assume that following the third round of ratings the SEJ is 1.0. This should be interpreted to mean that if a second sample of SMEs from the same population were recruited and asked to rate the items, there is a 68 percent chance that the new recommended cut score would be expected to differ from the cut score recommended by the first group of SMEs by plus or minus 1.0 point or less, and there is a 95 percent chance that the new recommended cut score would be within plus or minus 2.0 points of the original recommended cut score.

Standard Error of Measurement (SEM)As a result of measurement error, two types of classification errors may occur. Based on their score(s), some examinees will be placed into the remedial course, though they could have been successful in the entry-level

31

course, and some examinees will be placed into the entry-level course when they would have been more successful in the remedial course. Error will always be present, but it is the goal to optimize placement into courses such that the error deemed most egregious is minimized.Therefore, the Authoritative Body may choose to raise or lower the recommended cut score to reduce one of these two types of classification errors: passing a test-taker who should fail or failing a test-taker who should pass. For example, the Authoritative Body may consider it more serious to minimize the number of examinees placed into the remedial course who could have been successful in the entry-level course than to place into the entry-level course a student who would be better served by the remedial course. They would then consider lowering the recommended cut score by some amount that has a rational basis, such as one standard error of measurement. Although this will minimize the number of examinees placed erroneously into the remedial course, it will increase the number of examinees placed erroneously into the entry-level course.Table 9 provides the approximate probabilities that a test-taker will be successful in the entry-level course. If the recommended cut score is 4, the probability that a student with a true score of 4 will be successful in the entry-level course is 50 percent. If the recommended cut score is raised one half a SEM, then the cut score is increased to 4.25, and the probability that a student with a true score of 4.25 will be successful in the entry-level course is reduced to 31 percent.

Table 9 Probability the Examinee Will Be Successful in the Entry-Level Course

Test-Taker’s True Score

Possible Recommended Credit-Granting Scores

SPS –2 SEM SPS –1 SEM

SPS –.5 SEM

Study Passing

Score (SPS)SPS +.5

SEM SPS +1

SEM SPS +2

SEM

3.0 3.5 3.25 4.0* 4.25 4.5 5.0

5.0 >99% >99% 99% 98% 93% 84% 50%

4.5 >99% 98% 93% 84% 69% 50% 16%

4.25 99% 93% 84% 69% 50% 31% 7%

4.0 98% 84% 69% 50% 31% 16% 2%

3.25 93% 69% 50% 31% 16% 7% 1%

3.5 84% 50% 31% 16% 7% 2% <1%

3.0 50% 16% 7% 2% 1% <1% <1%

(Adapted from College-Level Examination Program® Web-based standard setting report prepared by the Educational Testing Service.)

Step 18. Provide Results to the Authoritative Body

The results from the standard setting session should go to the Authoritative Body for review and final adoption. The materials for the Authoritative Body should include, but not be limited to, the following:

A complete set of the materials used in the standard setting session.••

The recommended cut score produced after each round with the standard error of judgment (SEJ) (see ••dictionary in Appendix P) and the standard error of measurement (SEM) (see dictionary in Appendix P) provided.

A list of the SMEs and a summary of their demographics.••

A summary of the results of the evaluation forms after the training, following round 1, and at the ••conclusion of the session.

32

A complete copy of the impact data provided to the SMEs, if any was provided, along with additional ••information by subgroups, if that was not part of the original presentation. Even if the decision was made not to provide impact data to the SMEs, it is still preferable to provide this information to the Authoritative Body.

The Authoritative Body should have as much information as possible to use in making decisions about the final cut score to be adopted. The cut score recommended by the standard setting session may be adopted as recommended by the committee. However, a variety of reasons may exist for why modification is necessary. It is not advisable to modify the recommended cut score by more than + 2 SEM or + 2 SEJ. When providing the recommended cut score to the Authoritative Body, the SEM and SEJ should be provided at a minimum. It is useful to compute what the recommended cut score would be at several possible points (as shown in Table 10) and to compile the results for the impact data using each of the resultant potential cut scores as the reference. This facilitates the review and discussion of the cut score and any possible modifications.

Table 10Example of SEM and SEJ Data Provided to Authoritative Body

-2 -1 Recommended Cut Score +1 +2

Standard Error of Measurement (SEM)

4.0 4.5 5.0 5.5 6.0

Standard Error of Judgment (SEJ)

3.0 4.0 5.0 6.0 7.0

Note: The SEM used in the example is 0.5, and the SEJ used in the example is 1.0.

33

Logistical Details for ConsiderationPreparation is the key to a successful standard setting. The more information you can prepare in advance, the better and more smoothly the standard setting will run. The following section provides basic advice on preparing for the standard setting session.

Printed MaterialsTo the extent possible, it is recommended that all printed materials for the SMEs be copied and placed into a large binder in the order in which they will be used. Tabbed dividers may be used to separate the materials into sections, such as agenda and introductory materials, training materials, test questions, rating forms, and evaluation forms. The use of a binder saves time during the process by reducing the need to pass out materials at each step. Any material that is considered confidential and should not leave the room may be copied on brightly colored paper to facilitate the identification of the materials. For example, the agenda and any handouts may be on plain white paper, but the essay prompt and student responses may be printed on goldenrod for easy identification. At the completion of the standard setting session, all materials not needed for documentation purposes should be disposed of in a secure manner.Each binder or packet of materials should be marked with a unique identification number, and the SME receiving the packet should be assigned the same unique identification number to assist in tracking materials as they are checked in or out for use. All essays should be printed one to a page and include a number identifying the essay. This identifying number will serve two purposes: It will allow the facilitator to determine the score the essay received on the i3 system, and it will make it easy to identify the essay during group discussion. However, remember that the numbered essays should be arranged randomly and not by score. Also, make sure that the score each essay received is not listed on the essay.

Support StaffIn addition to the facilitator, it is helpful to have other staff available during the standard setting session. Particularly useful for helping the process run smoothly is a data entry person. The data entry person has the responsibility of recording the group into which each essay was placed by each SME in a timely manner, and producing the summary results after each round for use in the discussion. It is also helpful to have a support person available to make copies and handle any details with catering, checking materials in and out for use in the standard setting, and overseeing any issues that may arise during the course of the standard setting so the facilitator is able to stay focused on the standard setting itself. The support person or the facilitator should have the responsibility of checking the data that has been entered at each round before providing feedback to the SMEs to ensure that the feedback provided is accurate.It is important to keep the number of people beyond the SMEs who are attending the standard setting to a minimum, or at least make sure they are properly informed about the importance of behaving a certain way during the standard setting. Anyone who provides information or in any way becomes part of the standard setting has the potential to bias the results. For this reason, nonparticipants should remain outside the meeting room whenever possible. In the event that the facilitator has agreed to the presence of nonparticipants in the meeting room to observe the process, it is imperative that these observers not speak without being asked to do so by the facilitator. Observers should sit in the back of the room as far away from the SMEs as possible so as not to interrupt, bias, or in any way intimidate the SMEs. Any conversations with the SMEs during breaks should focus on issues that are not part of the standard setting. Enforcing these rules can be very difficult, especially when the observer has a stake in the process and strong opinions about the outcome, but it is for that very reason that the rule must be enforced. Preparing everyone ahead of time with the behavioral expectations will help.

34

Data SpreadsheetA data spreadsheet is an essential part of any standard setting session. The spreadsheet serves as a record of the data collected during the session and can be invaluable for allowing the quick computation of results to be provided as feedback to the SMEs during the process. Essay groupings should be entered for each SME. Entering the essay groupings for all SMEs in the same small group together facilitates group level calculations of the mean and standard deviation of ratings for each round. If impact data is being provided, a spreadsheet that contains the impact data summarized by score point will ease the task of providing impact data to the SMEs for the cut score that is being considered.

Calculating the Cut ScoreThe cut score is calculated from the SME ratings using the following method that was described previously in the discussion of the pinpointing round. The same method should be used to compute temporary cut scores for each group following rounds 1 and 2.

For each SME, record their placement of each essay (see Table 5).1.

For each essay, tally the number of people placing the essay into each category (see Table 6).2.

Record the essay ID number, the official score of the essay, and the category(s) into which at least one 3. SME placed the essay (see Table 7).

Sort the spreadsheet created in step 3 by essay score from lowest to highest score (see Table 8).4.

Find the median (middle) essay score for the set of essays placed by at least one person into each category 5. by counting the total number of X’s in the two adjacent columns between which the cut score will be placed. Divide the total number of X’s by 2 and identify the middle score when the scores associated with the X’s are arranged from the lowest to the highest score—this will be the median. To avoid the appearance of bias, before the standard setting decide whether to go with the higher score or lower score in the case that an even number of X’s are in the two categories, and the median would fall between two different scores. For example, in Table 8 the first cut score will separate the remedial category and the entry-level category. By counting the number of X’s in the two categories, you will find that the total number of X’s is 42. Dividing 42 in half will result in a value of 21—or more specifically, the middle would be the score at 21.5 between the 21st and 22nd scores. If the scores associated with the X’s in the two categories are arranged from lowest (1) to highest (8), then the 21st and 22nd scores are both 4, so the median is 4, and 4 would be the recommended cut score for the entry-level category (see Figure 4 for more details on finding the median). If the 21st and 22nd scores were 4 and 5 respectively, the median would be 4.5, and then a rounding decision to round up to the nearest whole number or down to the lowest whole number should be made according to the agreed-upon procedure decided by the Authoritative Body before the beginning of the standard setting. It should be noted that the mean or average score between the two categories could also be used, but is subject to influence by extreme scores. Therefore, it is generally recommended that the median be used instead.

35

Scores from remedial and entry-level columns represented by X’s in Table 8 arranged from lowest to

highest score:

1, 1, 1, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 6,

6, 6, 6, 6, 6, 6, 7, 7, 7, 7, 7, 8

Number of total scores = 42Middle of range falls between 21st and 22nd score = (4+4)/2Median = 4

Figure 4. Example of finding the median score.

Calculating the Standard Error of JudgmentThe standard error of judgment (SEJ) is produced in the following manner and is an indicator of the variability of the recommendations provided by the panel of SMEs in the standard setting. Before the SEJ can be calculated, it will be necessary to compute the recommended cut score for each participant. This can quickly be accomplished by computing the minimum and maximum essay score that each participant assigned to each category. Using the lowest essay score that was placed in the upper category and the highest essay score that was placed in the lower category, take the mean of the two numbers and use this as the recommended cut score between the two categories for the individual. A decision should be made before standard setting about whether the results that produce a decimal will be rounded up or down. This is the same process used to find the range for pinpointing, except it is being calculated at the individual SME level rather than across the total group. This can be fairly quickly accomplished by adding an essay score column to Table 5 and sorting the table from lowest to highest by score. This will arrange the SME groupings such that the highest and lowest score associated with each grouping can easily be identified in each column for each SME (see Table 11): R (remedial), E (entry-level), or A (advanced).

Compute the standard deviation for the set of round 3 judgments for the SMEs.1.

Divide the standard deviation for the set of round 3 judgments for the SMEs by the square root of the 2. number of SMEs.

21st score 22nd score

Median (4 + 4)/2 = 4

36

ConclusionFollowing the procedures outlined in this document will result in a recommended cut score for the WritePlacer assessment. Though WritePlacer ESL essay sets are not provided in this document, the same process could be followed to set cut scores for WritePlacer ESL. Any process resulting in a cut score should begin with a careful study of the reason a cut score is needed and how that cut score will be used. Preparation is the key to the process running smoothly, and sufficient time should be allowed for the process to occur. Sample schedules are provided in Appendixes B and C.All cut scores should be validated through the collection of documentation and procedures that are internal to the process and through other means external to the process. A good way to validate the use of a cut score is to take advantage of the free Admitted Class Evaluation Service (ACES) offered by the College Board to users of the College Board tests. Additional information on ACES may be found at the following Web site: www.collegeboard.com/highered/apr/aces/aces.html.For additional information on setting cut scores on multiple-choice tests or tests that combine multiple-choice and constructed response items, see Cizek (2001), Morgan and Michaelides (2005), Morgan (2006, 2008), Cizek and Bunch (2007), and Zieky et al. (2008).

Table 11 Example of Computing the Individual Cut Score for Each SME

Essay ID ScoreSME

1SME

2SME

3

9 1 R R R

17 1 R R R

31 1 R R E

2 2 R R R

6 2 A E E

10 2 R E R

19 2 R E R

26 2 E R E

33 2 R R E

12 3 R R E

23 3 E E E

30 3 R E E

39 3 R R R

4 4 E R E

13 4 R E E

24 4 E E E

27 4 E E E

36 4 E E E

38 4 R R R

8 5 R E E

11 5 E E E

37

18 5 E E A

22 5 E E E

37 5 A E E

7 6 E A E

14 6 E E E

20 6 E E E

25 6 E E A

28 6 E E E

29 6 A A A

34 6 A E A

40 6 E A E

3 7 E A A

5 7 E E A

15 7 A A A

21 7 A A E

35 7 A E E

1 8 A A A

16 8 A A A

32 8 A A A

Remedial Range 1 – 5 1 – 4 1 – 4

Entry-Level Range 2 – 7 2 – 7 1 – 7

Advanced Range 2 – 8 6 – 8 5 – 8

Entry-Level Cut Score (2 + 5)/2 = 3.5 = 4* (2 + 4)/2 = 3 (1 + 4)/2 = 2.5 = 3*

Advanced-Level Cut Score (2 + 7)/2 = 4.5 = 5* (6 + 7)/2 = 6.5 = 7* (5 + 7)/2 = 6

*This example assumes a decision was made before standard setting to round up decimal results.

38

ReferencesAmerican Educational Research Association, American Psychological Association, National Council on

Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.

Cizek, G .J. (2001). Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Cizek, G. J., & Bunch, M. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.

Hambleton, R. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Hambleton, R. K., Jaeger, R. M., Plake, B. S., & Mills, C. N. (2000). Handbook for setting standards on performance assessments. Washington, D.C.: Council of Chief State School Officers.

Hansche, L. N. (1998). Handbook for the development of performance standards: Meeting the requirements of Title I. Bethesda, MD: U.S. Department of Education, Council of Chief State School Officers. Retrieved May 18, 2005, from http://www.ccsso.org/publications/details.cfm?PublicationID=131

Kahl, S. R., Crockett, T. J., DePascale, C. A., & Rindfleisch, S. I. (1994, June). Using actual student work to determine cutscores for proficiency levels: New methods for new tests. Paper presented at the National Conference on Large-Scale Assessment, Albuquerque, NM.

Kahl, S. R., Crockett, T. J., DePascale, C. A., & Rindfleisch, S.I. (1995, June). Setting standards for performance levels using the student-based constructed-response method. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.

Kane, M. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425–61.

Kingston, N. M., Kahl, S. R., Sweeney, K. P., & Bay, L. (2001). Setting performance standards using the body of work method. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Morgan, D. L. (2006). Setting local cut scores on the SAT Reasoning Test Writing Section (College Board Special Report). New York: The College Board.

Morgan, D. L. (2008). A guide to conducting standard setting for ACCUPLACER multiple choice tests using a modified Bookmark procedure. Paper presented at the 2008 ACCUPLACER National Conference in Ft. Lauderdale, FL.

Morgan, D. L., & Michaelides, M. P. (2005). Setting cut scores for college placement (College Board Research Rep. No. 2005-9). New York: The College Board.

Perie, M. (2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 27(4), 15–29.

Reckase, M. D. (2001). Innovative methods for helping standard-setting participants to perform their task: The role of feedback regarding consistency, accuracy, and impact. In G. J. Cizek (Ed.), Setting performance standards: Concepts, methods, and perspectives. Mahwah, NJ: Lawrence Erlbaum Associates.

Zieky, M. J., Perie, M., & Livingston, S. A. (2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: Educational Testing Service.

39

Appendix A: Information on WritePlacerWritePlacer and WritePlacer ESL are direct measures of on-demand student writing skills offered as part of the College Board’s ACCUPLACER program. Examinees are asked to provide a writing sample in response to a specific prompt. These assessments measure whether writing skills are at the level expected of a student entering college. They are used to assess the on-demand writing skills of entering college students to determine whether the student requires developmental instruction before taking college-level course work.WritePlacer and WritePlacer ESL are different than the other ACCUPLACER test titles in several ways. First, the tests are not adaptive. Second, rather than present multiple-choice items to examinees, examinees are required to write an essay. The writing skill areas that are evaluated on students’ WritePlacer essays are (a) purpose and focus, (b) organization and structure, (c) development and support, (d) sentence variety and styles, (e) mechanical conventions, and (f) critical thinking.WritePlacer ESL examinees are required to write an essay in response to a specific prompt. The rubric for scoring the WritePlacer ESL essays was developed and reviewed by ESL practitioners familiar with the writing of students for whom English is a second language. The writing skill areas that are evaluated on students’ WritePlacer ESL essays are (a) word use, (b) sentence use, (c) grammar, and (d) organization and development.

Development of WritePlacer PromptsNine WritePlacer prompts were introduced in 2009. A set of approximately 40 retired SAT prompts were reviewed for content and sensitivity by internal College Board content specialists and a committee of faculty members representing ACCUPLACER users. The ACCUPLACER committee also reviewed prompts for the level of accessibility to their students and rank ordered the prompts by appropriateness for the ACCUPLACER population. The prompts receiving the highest ratings were chosen for use in WritePlacer. Several hundred students responded to each prompt when the prompt was previously an SAT operational essay prompt, and this provided a rich database of student responses from which to pull essays for the calibration of the electronic scoring system. The SAT responses were rescored by expert scorers at Pearson Educational Measurement in 2008, according to the WritePlacer 8-point rubric that was developed by the ACCUPLACER faculty committee in 2006–2007 specifically for the WritePlacer program. At least two Pearson scorers rated each response. If the two scorers disagreed by more than one point, a third, master scorer reviewed the response and adjudicated the final score. A large number of essays representing each score point on the eight-point scale were entered into the electronic scoring system to calibrate the i3 scoring engine, and an additional unique set of essays for each score point of each prompt was used to validate the scoring.WritePlacer ESL prompts are not former SAT prompts and were developed separately using a committee of ACCUPLACER faculty with expertise in teaching students for whom English is a second language. The WritePlacer ESL committee revised the WritePlacer ESL rubric, and existing prompts from ACCUPLACER 2.0 were recalibrated with the new rubric on the i3 platform. In addition, three new prompts were added to the WritePlacer ESL test on the i3 platform. The process used to recalibrate the WritePlacer ESL prompts and develop the new prompts is consistent with that described for the new 8-point WritePlacer test.

WritePlacer ScoringWriting samples for WritePlacer are scored using modified holistic scoring, which is a procedure used to evaluate the overall quality of writing based on important features of writing. Holistic scoring is used to evaluate the overall effectiveness of the writing sample as evidenced by how well a piece of writing communicates a whole message. Each writing sample is evaluated based on its overall impression, not on the basis of the individual writing characteristics, listed previously, in isolation.

40

WritePlacer scores are reported on a scale from 1 to 8. A score of zero indicates that the response was off topic, in a language other than English, too short to score, or in some other way unscorable. A description of the qualities and characteristics of essays at each score point is provided in Appendix E.

41

Appendix B: A Sample Schedule for a Standard Setting Using a Modified Body of Work Process to Set One Cut ScoreDay 18–8:15 a.m. Opening remarks

8:15–9 a.m. Overview of goals and the Modified Body of Work procedure

9–10 a.m. Create definition of the just minimally competent examinee for each of the two categories

10–11:30 a.m. Take test and discuss process and scoring (may be preferable to take test at home before meeting and then discuss at the meeting)

11:30 a.m.–12:15 p.m. Review the Modified Body of Work procedure and practice using the method

12:15–12:30 p.m. Complete training evaluation

12:30–1:30 p.m. Lunch

1:30–2 p.m. Answer any last questions on the Modified Body of Work procedure

2–4 p.m. Read essays and group into two categories for round 1 range finding

4–4:15 p.m. Complete evaluation of round 1 range finding

4:15–4:30 p.m. Review tomorrow’s agenda and dismiss (remainder of afternoon will be needed to enter all ratings and compute results for the discussion)

6 p.m. Dinner as a group or on your own

Day 28–8:30 a.m. Continental breakfast

8:30–8:45 a.m. Review of goals and agenda

8:45–10 a.m. Provide feedback on round 1 essay groupings and facilitate small group discussions; provide retraining on method if needed as indicated by round 1 range-finding evaluation form responses

10–11:30 a.m. Make round 2 range-finding groupings into two categories

11:30 a.m.–12:30 p.m. Lunch

12:30–1:30 p.m. Provide feedback on round 2 range-finding placements and facilitate small group discussion

1:30–2:15 p.m. Provide impact data and facilitate large group discussion; pull pinpointing essays for the identified cut score range

42

2:15–3:30 p.m. Make round 3 pinpointing groupings into two categories and complete final evaluation

3:30–4 p.m. Final debriefing and dismissal

43

Appendix C: A Sample Schedule for a Standard Setting Using a Modified Body of Work Process to Set Two Cut ScoresDay 1

8–8:15 a.m. Opening remarks

8:15–9 a.m. Overview of goals and the Modified Body of Work procedure

9–10 a.m. Create definition of the just minimally competent examinee for each category (should be three categories for two cut scores)

10–11:30 a.m. Take test and discuss process and scoring (may be preferable to take test at home before meeting and then discuss at the meeting)

11:30 a.m.–12:15 p.m. Review the Modified Body of Work procedure and practice using the method

12:15–12:30 p.m. Complete training evaluation

12:30–1:30 p.m. Lunch

1:30–2 p.m. Answer any last questions on the Modified Body of Work procedure

2–4 p.m. Read essays and group into three categories for round 1 range finding

4–4:15 p.m. Complete evaluation of round 1 range finding

4:15–4:30 p.m. Review tomorrow’s agenda and dismiss (remainder of afternoon will be needed to enter all ratings and compute results for the discussion)

6 p.m. Dinner as a group or on your own

Day 2

8–8:30 a.m. Continental breakfast

8:30–8:45 a.m. Review of goals and agenda

8:45–10 a.m. Provide feedback on round 1 essay groupings and facilitate small group discussions; provide retraining on method if needed as indicated by round 1 range-finding evaluation form responses

10–11:30 a.m. Make round 2 range-finding groupings into three categories

11:30–2:30 p.m. Lunch

12:30–1:30 p.m. Provide feedback on round 2 range-finding placements and facilitate small group discussion

44

1:30–2:15 p.m. Provide impact data and facilitate large group discussion; pull pinpointing essays separately for both cut scores unless ranges overlap and only one set of essays will cover all three categories

2:15–3:30 p.m. Make round 3 pinpointing groupings into two categories for each cut score, or three categories if only one set of pinpointing essays is used, and complete final evaluation

3:30–4 p.m. Final debriefing and dismissal

45

Appendix D: WritePlacer Sample Prompt for the Essays Provided in This Document

An actor, when his cue came, was unable to move onto the stage. He said, “I can’t get in; the chair is in the way.” And the producer said, “Use the difficulty. If it’s a drama, pick the chair up and smash it. If it’s comedy, fall over it.” From this experience the actor concluded that in any situation in life that is negative, there is some-thing positive you can do with it.

Adapted from Lawrence Eisenberg, “Caine Scrutiny.”Can any obstacle or disadvantage be turned into something good?Directions: Plan and write a multiparagraph essay in which you develop your point of view on the above question. Support your position with reasoning and examples taken from your reading, studies, experience, or observations.

46

Appendix E: WritePlacer Scoring RubricScore of 8An essay in this category demonstrates clear and consistent mastery of on-demand essay writing with a few minor errors. A typical essay:

Effectively and insightfully develops a point of view on the issue••

Addresses an appropriate audience and demonstrates a clear purpose for writing in the thesis statement••

Demonstrates outstanding critical thinking, using effective examples, reasons, and other evidence to ••support its position

Is well organized and clearly focused, demonstrating clear coherence and smooth progression of ideas••

Exhibits skillful use of language, using a varied, accurate, and apt vocabulary••

Demonstrates varied and effective sentence structure••

Is free of major errors in grammar, spelling, and punctuation••

Score of 7An essay in this category demonstrates consistent mastery of on-demand essay writing, although it may have occasional minor errors. A typical essay:

Effectively and insightfully develops a point of view on the issue••

Addresses an appropriate audience and demonstrates a clear purpose for writing in the thesis statement••

Demonstrates outstanding critical thinking, using appropriate examples, reasons, and other evidence to ••support its position

Is well organized and focused, demonstrating clear coherence and smooth progression of ideas••

Exhibits appropriate language, using a varied and accurate vocabulary••

Demonstrates varied sentence structure••

Is practically free of errors in sentence structure, grammar, spelling, and punctuation••

Score of 6An essay in this category demonstrates reasonably consistent mastery of on-demand essay writing, although it may have occasional lapses in quality. A typical essay:

Effectively develops a point of view on the issue••

Addresses an appropriate audience and demonstrates a clear purpose for writing••

Demonstrates strong critical thinking, generally using appropriate examples, reasons, and other ••evidence to support its position

Is well organized and focused, demonstrating coherence and a logical progression of ideas••

Exhibits consistent control in the use of language••

47

Demonstrates adequate variety in sentence structure••

Is generally free of errors in sentence structure, grammar, usage, spelling, and punctuation••

Score of 5An essay in this category demonstrates adequate mastery of on-demand essay writing, although it will have lapses in quality. A typical essay:

Develops a viable point of view on the issue••

May stray from the audience and purpose but is able to refocus••

Demonstrates competent critical thinking, using adequate examples, reasons, and other evidence to ••support its position

Is generally organized and focused but could lack coherence and logical progression of ideas••

Exhibits adequate but inconsistent control of language••

Demonstrates some variety in sentence structure••

Contains some minor errors in sentence structure, grammar, spelling, and punctuation••

Score of 4An essay in this category demonstrates developing mastery of on-demand essay writing. A typical essay:

Develops a viable point of view on the issue••

May stray from audience and purpose••

Demonstrates some critical thinking but may do so inconsistently or use inadequate examples, reasons, ••or other evidence to support its position

May lack control of organization or focus, demonstrating some lapses in coherence or progression of ideas••

Contains some errors in sentence structure and use of grammatical conventions such as word choice, ••usage, spelling, and punctuation

Score of 3A response in this category demonstrates little mastery of on-demand essay writing; the response is flawed by one or more of the following weaknesses:

Presents a vague or limited point of view on the issue••

Demonstrates little awareness of audience••

Attempts to develop the main idea, though that attempt is inadequate••

Demonstrates weak critical thinking with little complexity of thought or with flawed reasoning••

Provides inappropriate or insufficient examples, reasons, or other evidence to support its position••

Is poorly organized and/or focused or demonstrates serious problems with coherence or progression of ideas••

Displays limited word choice and little sentence variety••

48

Contains many errors in mechanical conventions of usage, sentence structure, grammar, spelling, and ••punctuation

Score of 2A response in this category demonstrates very little mastery of on-demand essay writing; the response is flawed by many or most of the following weaknesses:

Presents a vague or limited point of view on the issue••

Demonstrates little awareness of audience••

Presents an unclear main idea••

Demonstrates weak critical thinking with little complexity of thought or with flawed reasoning••

Organizes ideas ineffectively, demonstrating a problematic progression of ideas••

Displays numerous errors in word choice, usage, and sentence structure••

Contains significant spelling, grammar, punctuation, and mechanical errors••

Score of 1A response in this category demonstrates no mastery of on-demand essay writing; the response is severely flawed by many or most of the following weaknesses:

Lacks a viable point of view on the issue••

Demonstrates no awareness of audience••

Fails to present a main idea••

Demonstrates flawed reasoning••

Demonstrates no complexity of thought••

Is disorganized and/or disjointed••

Displays fundamental errors in word choice, usage, and sentence structure••

Contains pervasive spelling, grammar, punctuation, and mechanical errors••

Score of 0No holistic score—this essay has not been given a holistic score or the six dimensional scores because it is either a blank page, incoherent, insufficient (too short to assess), in a foreign language, or off topic.

49

Appendix F: WritePlacer Essay Training Set#T1The average person faces numerous obstacles throughout his lifetime. At times, these challenges seem overbearing and impossible to pass, yet they are not. One must learn to look for an alternative way to solve a seemingly disadvantageous situation.Recently, my literature class read the classic book, “The Scarlet Letter” by Nathaniel Hawthorne. Hester Prynne faced the unbearable obstacle of living as a pariah of the community. She engaged in corrupt relations before marriage, and was forced to bear the scarlet letter in public there was no way to hide her crime. Although this obstacle seemed impossible to overcome, Hester continued with a life of persecution boldly, and raised a child. Pearl, her baby, brought some happiness and joy back into her life when times were troubling. Although she wore a symbol of her mishaps, Hester turned her despair into joy through raising her child and slowly re-entering society.Persecution and bigotry plague our modern world. Many religious and racial minorities face these obstacles every day. Last week, the whole country commemorated the death of a true hero, Martin Luther King Jr. This man faced constant threats to himself, his family, and his people. Most men in his position would hide in the face of danger, but did he? No. King devoted his life to transforming the disadvantageous lives of fellow minorities into lives of acceptance and prosperity. Attempts of the KKK and other white organizations constantly attacked Luther, yet he pursued his dream of a better life for his people. Martin Luther would be enthralled to hear that his desired goals were achieved; Black Americans now face less persecution; have equal jobs opportunities as white Americans, and possess the liberty that Luther could never experience. His obstacles paved the way for the acceptance of his decendants.An obstacle always presents two sides. One must learn to optimitically look for an advantageous solution to a problem rather than accepting its existence and dwelling in self pity and sorrow.

#T2Not every negative obstacle can be turned into something good. Obstacles can occur at any given time or momment and the person who encounters the obstacle could become seriously injured. For example, if one were to run over a few rocks while skating down a hill at high speeds, and that person were to go flying into the air and break an arm or leg when landing, this clearly could not be turned into something humorous, amusing, or entertaining. Not all obstacles in life are chairs blocking an actor while trying to get on stage, but more serious things. Of course there are little daily mishaps that can be converted but some things have greater magnitude than things like these. This is why I believe that not all things can be turned into something good.

#T3I think that any obstacle or disadvantage can be turned into something good. Sometimes you just need to adapt to the situation at hand when you are in a disadvantage. For example, when there is one minute left in a basketball game and your team is down by two you have to change that negative effect, get your team’s momentum up and think positively. You can adapt to an obstacle as the actor did. A chair was in his way to get to stage. So, instead of standing there he would smash the chair if it were a drama or trip on it if it was a comedy. This shows that from something negative a positive can happen. When you are delt with something negative in life, don’t avoid it, use it to try to make it a positiveFor example, lets say your in a comedy and your wife suddenly appears on stage, don’t panic and think the play is ruined. You can use her to make the crowd laugh until your wife goes back to the backstage. In conclusion, never think you can’t get past an obstacle. Think positively and you can change it towards your advantage.

50

#T4Obstacles and disadvantages are in everyones life, no ones is exempt from turmoil, some people find it difficult to cope with a little difficulty and because of it they break down. For those that are fortunate enough to find the brighter side of any situation, obstacles and disadvantages appear like minor speed bumps instead of road blocks.In addition to that, many people try to avoid problems when faced with them instead of confronting it. That in turn leads to even more problems. They let their tiny insignificant obstacle turn into something much greater than it has to be. Turning any disadvantage into an advantage is in the mind of the beholder. If that person just puts in their mind that they won’t let this interferance hold them back, then they’ll find that their is so much to gain. People who think that their problems will stop them in their tracks are pessimists, they see everything as negative. On the other hand people who are able to deal with any problem with stride are optimists. Studies have shown that optimists live a more long and stress free life. Pessimists of course live with fear and anguish and get more sicknesses than optimists. It all comes down to how the mind thinks. If people think that their illness or disadvantage is going to bring them down, then it will. The body will only do what the mind tell it to do.In conclusion, obstacles and disadvantages are easily overcome, thats if people can train their minds to think like that. If people don’t want their situation to bring them down, then they should always think positively, even in a bad situation, the more people think their problems are nothing, then their problems will appear a lot less.

#T5A disadvantage could be turned into many things that we don’t know of. There are many things to do with anyone you could either stay with, what you think as a disadvantage, or you could turn it into some cool advantages.To stay with a disadvantage could hurt your life forever. The SAT’s are a good example. You may think that the SAT’s are a waste of time. You don’t get to do anything, except sit for a full three hours and fourty- five minutes. Now you could hate how you spent three hours and fourty-five minutes doing nothing, or you can turn it around and make it something cool.You could turn this idea into something good now taking the SAT’s isn’t a waste of time, but the key to a good life. If you take the SAT’s you could get a high score and go to a good college. Then you could get a great job. When you have a great job you get a lot of money.What this concludes to, is, any disadvatage can be turned into something good. Here is another example. Rupunzel had the disadvantage of having really long hair. In the end her hair saved her life. So a disadvantage really can be turned into an advantage, or something good

#T6Many have encourntered obstacles and disadvantages in their lives. Countries, government officials and people have had obstacles they overcame by turning them into something good. After both world wars, there was a mass flourish of immigration to the United States. Immigrants from all of Europe many who were poor, disstressed & torn from families. The United States could of veiwed this as an obstacle, and not allowed their entrance. However, they welcomed them in, gave them opportunities, and many only helped the success of the country.In the earlier years of segregation, African Americans were never permitted in white schools or churches. After many court cases, and the help of Martin Luther King Jr., the country began to make laws that allowed them to join with the whites. Being that many were intelligent, they helped the country benefit in the business world, classroom...I have observed many, in time of a tragedy sickness or loss, build up their strength and veiw it for only the best. This helped them overcome it, even if they didn’t see the good right away.

51

Obstacles and disadvantages can almost always be turned to the good. One can see how America only benifitted from times that were or seemed difficult.

#T7I plan to use this opportunity to show how effectively I can develop and express ideas. I will, therefore, take care to develop my point of view, present my ideas, logically and clearly and use language precisely. An actor when his cue came, was unable to move onto the stage He said, “I can’t get in the chair is in the way.” And the producer said, “Use the difficulty. If it’s dramma, pick the chair up and smash it. If it’s a comedy fall over it.” From this experience the actor concluded that in any situation in life that is negative, there is something positive you can do with it. This passage was adopted from Lawrence Eisenberg, “Caine SCrubing”Can any obstacle or disadvantage be turned into something good? I will plan and write an essay in which I will develop my point of view on this issue. I will support my position with reasoning and examples taken from my reading, studies, experience, or observations.

#T8Any obstacle or disadvantage can be turned into something good. There is always a bright side in every bad situation. Bad situations may occur and allow you to feel terrible, but all you have to do is look at the bright side.Take me for example. My intermediate family was in a big struggle. We had many concerns on how life would be the next day. Some days there would be no water, some days there would be no heat, and some days there would be no lights. Life was a struggle being short on food and other resources. If seemed as if life couldn’t get any worse, but my family benefitted from these loses.Even though we didn’t have these certain resources and food, we realized that when these things were gone our family got together and kept each other warm. Our obstacles and bad situations brought our family together. Not having a car for transportation meant that we would spend more time together to become a happy family.Since we have been a happy family every thing has been better than before. My brother was able to get a full ride to the University all the way to his senior year, and I was able to spend the last couple of years in a $15,000 school. For some reason I do believe that there is something positive you can take out of something negative. All things happen for a reason, whether it is to make a play funnier or make a family stronger. Anyone can use disadvantages to have advantages.

#T9Up to the 1960’s, America has segregated between two groups, the white and the coloreds. The Afro-Americans had to go through the feeling of inferiority to the whites. They went through a miserable and difficult experience for years, but one man changed all that. Martin Luther King Jr. was a Baptist preacher who led the Bus Boycott and made the famous, “I have a dream...” speech. He helped American society revolutionize the way it thought about segregation. Thanks to this man, along with many other men and women, he turned this obstacle into a triumph for diversity.In the well-known book, Count of Monte Cristo, the main character, Edmond, is a young sailor who is about to marry the love of his life. This doesn’t happen, and instead, his enemies frame him and place him in a prison in an uncharted island. For years and years Edmond is in that prison. He finally escapes, finds a cave full of treasure chests, and gets his revenge. Those years in prison helped Edmond become stronger, wiser, more knowledgeable, and more agile. He not only got his revenge, but he married a woman who he fell in love with. Adversity helps individuals grow in character, and at times even physically in strength.A relationship between a guy and girl takes a lot of work mentally and emotionally. It’s like if they’re at sea in a boat. Sometimes the weather is stormy and at other times there’s not a single wave with the sun out. That

52

relationship has to go through drizzles, thunderstorms, and maybe even hurricanes, but once it’s all over, they both benefited from the hardships and get even closer than before. Hardships help us endure and persevere as time passes on.Walt Disney once had a very popular cartoon out that was a bunny. He made good business with it, but then a competitor stole his idea. Walt was distraught and didn’t know what to do, but then the grand idea of a mouse came into mind. Thus, Mickey Mouse was born! This mouse made Walt Disney famous and brought in business way more than before. It was a good thing that his bunny idea was stolen after all!Life is one big struggle of a rollercoaster. It has its ups and downs, twists and turns, but through all of the rough times, people can still learn from them. Characters can still become stronger and more resilient after trials and tribulations. “The struggles and hardships of today pay off for the triumphs and victories of tomorrow.”

#T10Yes, I firmly believe that any obstacle can be turned into something good. I see this throughout my world and my life. Being a football player, many of my life experiences have been on the field, or in the weight room. My coach always says that the most important thing is “how you respond to adversity.” He constantly challenges us not to be hindered by an obstacle, but to find the good in it and overcome that obstacle.Another place that I see this optimism is through my religion. In Judaism, there are many teachings, as in all major religions. One of these is about optimism and finding good even in the worst situations. In the Torah, Abraham was told that God would destroy Sodom and Gemorrah. It was common knowledge that these were two evil cities. However instead of seeing the bad, Abraham looked for the good. He begged God to save the cities. After some negotiation he agreed that God could destroy the cities if he could not find 10 righteous men. These cities were huge, but in all that bad all Abraham needed to do was find 10 righteous men and God would spare the cities.This optimism and positive outlook on life is also important for one to have as a healthy individual. Life is hard, there is little question there. Many bad things will happen, and many obstacles will appear in life. It is impossible to dwell in the negative of every situtation. Some way, some how one must find the positive through the negative. It is the only way to maintain a healthy mindset.Finally, this is a more productive way of thinking. When business deals go bad, do executives dwell on the negatives. Exactly the opposite, they hawkishly attack the positives so they can maximize their gains. This allows them to be more productive and achieve greater heights.I hold a firm belief that in any negative situation there is a positive. I see it throughout my life. From the sport that I play, to my religion, to everyday life and the business world. It is apparent to me that in every situation, no matter how bad, there is something positive that can come out of it.

53

Appendix G: Training Record FormFor each essay in the set, indicate the group into which you think the student exhibiting the skills and abilities demonstrated by that essay should be placed. For the purposes of this training, the goal will be to set one cut score to separate essays into two groups—either remedial (R) or entry level (E).Beside each essay ID number listed below, place either an R for remedial or E for entry level to show into which group you are placing the essay.

1. _________2. _________3. _________4. _________5. _________6. _________7. _________8. _________9. _________10. _________

54

Appendix H: Training Evaluation FormThe purpose of this evaluation form is to obtain your feedback about the training you have received so far on WritePlacer and the Modified Body of Work standard setting method. Your feedback will provide a basis for the facilitator to determine what information may need to be reviewed before the start of the actual standard setting process.Please complete the information below. Do not put your name or identification number on the form because we want your feedback to be anonymous.

Gender: _________Male_________Female

Race/Ethnicity:

_____Hispanic _____Asian _____African American _____White

_____Other (Please specify:___________________________)

Years of Experience as a Faculty Member:

_____1–5 years _____ 6–10 years _____11–15 years

_____16–20 years _____20–25 years _____25+ years

Affiliation:

_____ 2-year college _____ 4-year collegePlease read the following statements carefully. Place an X under one category (strongly agree, agree, disagree, or strongly disagree) to indicate the degree to which you agree with each statement.

Strongly Agree Agree Disagree

Strongly Disagree

1. I understand the purpose of this workshop.

2. The facilitator explained things clearly.

3. I feel comfortable with the task of evaluating each essay.

4. I feel comfortable with the task of assigning each essay to a category.

5. I understand the concept of the just minimally competent examinee.

6. I am ready to begin setting standards on the WritePlacer test.

Please add any comments you feel necessary to the back of this form.

55

Appendix I: Range-Finding Essay Set#1From my experiences obstacle or disadvantages can be turned into something good. The reason I say this because you can learn from past experiences that how you become wiser. An example of this when I went to take my driving test for the first time I was not well prepared. My mother kept telling me to study the drivers book but I said I didn’t need to. Well the day came to take the test and I failed with flying colors. Since I had a week to go back and retake the test I studied and studied until my eye turned red, but I past this time. This a good example of how obstacles can be turned into something good. Another example is when I tried out for the basketball team and didn’t make it. The coach came to me said “Jasper you have the ability to make the team. Just work on your freethrows and try again next year.” So I took his advice and practiced everyday that I had a chance to. Next year came by and I tried out again and this time I made the team. My shot improved a lot and so did my free throws, they called me the free-throw king because I made 80 percent of my attempts. In any event obstacles and disadvantages can be turned into something positive like the man in the story he could have tripped over the chair and made a joke or turned it into a comedy.

#2Yes, obstacles and disadvantages can be turned into something good because you will know how to over come obstacles and disadvantages in the future. You can help other people with obstacles and disadvantages because you have already been in that situation. You also can learn from your obstacles and disadvantages. It can also so help you in life and with

#3An obstacle or disadvantage may appear to be a setback, though either has the ability to be turned into something good. For instance, Rosa Parks participated in the Civil Rights Movement in the 1950s and 60s and was jailed for her involvement. Though her jailtime was initially a setback and and obstacle, she had to overcome, it eventually led to the Montomery Bus Boycott and the gradual desegregation of the United States. Similarly, the Dean of Students at my school has to overcome the obstacle of paraplegia each day, yet he describes himself as a stronger and more sensitive individual as a result. Rosa Parks and my Dean are clear examples of obstacles being turned into something good.In Montgomery, Alabama during the Civil Rights Movement, African Americans were required to sit in the back of public buses to make room for the white people. One day Rosa Parks refused to sit in the back of the bus and was sent to jail for her acts of civil disobedience. This action resonated with African Americans nationwide but particularly in Montgomery. The African American people staged a boycott and contributed to the desegregation which was a goal of their movement. Rosa Parks served as a symbol of what was possible if one would simply take a stand and turn their situation from unjust and degrading to one that made a profound difference in today’s society.The Dean of Student at my school, Mr. Barbee, was an English teacher, a track coach, and the Dean of Students before the roof of his barn collapsed on him a few years ago. He had always been a strong man who genuinely cared for his students. Who respected and loved him. After his accident, he lost the use of his legs and feet and his life drastically changed. He had become unable to care for himself and lost great amounts of self esteem. He ultimately decided he had to get his life back so he learned how to live independently through physical training courses, he learned to drive a car designed for handicapped people, and he returned to school. His experience which could have destroyed him and the goals he had actually allowed for him to build strength he never knew he had. Through his actions he made a severe disadvantage into a very positive life change.

56

Through their varying experiences, Rosa Parks and Mr. Barbee demonstrate that any obstacle or disadvantage can be turned into something good.

#4In life, we are often faced with obstacles, disadvantages and difficulties. At the time, that certain obstacle might seem like the worst thing to us, but sometimes, there is a great deal we can learn about ourselves, from the way we deal with the obstacle we can also ultimately became stronger, and wiser people, by overcoming challenges and diffuculties in our lives.Being diagnosed with a life-threatening illness is terrifying. It is the ultimate obstacle, often, people succumb to it. But sometimes, it can be fought, and in return we are given a second chance to life. Lance Armstrong was faced with cancer, and survived, and became stronger, and gave hope to many people. He is one of the best examples of how an obstacle can be something turned into something good.When faced with disadvantages, such as poverty, some people use it as a stepping stone to succeed, wanting a different life for themselves, and for their children.Obstacles are something we are faced with every day. By changing our outlook, and realizing that there might be opportunities in every diffuculty, we are wiser, and can really make the best of every situation, turning the lemons life gives us, into lemonade.

#5All obstacles put in a person’s way can turn into something good. In the excerpt we were given the actor was able to work with his obstacle, the chair, to improve his actiong performance. By accepting the challenge of having the chair in the way, the actor was able to portray, more realistically, the part he was playing. If people can view heir obstacles not so much of a problem but as a challenge or learning spot, they’d more likely see the positive side to it and a good outcome. Life will always throw you challenges and obstacles to face. It’s the attitude you take when you are faced with these challenges that decides whehter or not you truly succeed. Success is not only when you’ve won something or the turn out was great, but it’s also when you’ve grown as a person and learned something. You can’t learn anything new when your life is perfect, simple, and relatively uneventful. You need those moments of desperation and obstacles to challenge you and drive you to think and reason. So even in a situation with no obvious good ending, at least you’ve learned a new lesson in life and about yourself. Overcoming a problem, figuring out the solution to a tormenting query, is the best satisfaction a person can feel. It’s the obstacles in life that make you stronger. So when you’re faced with a negative situation or disadvantage you can always learn from it. Personal growth is an amazing thing and can easily be derived from what looks to be a terrible situation. For example a teenage driver gets into an accident and totals his car. This would seem to have no positive outcome, except that this event caused the teenage driver to learn to drive safer and follow the laws of driving, which could prevent many more accidents. All in all, the lessons learned from an obstacle can always be viewed as a good outcome.

#6Experience may often show us that in any situation in life that is negative, there is something positive you can do with it. Obstacles and disadvantages can most often be turned into something good. Even more so, sometimes the obstacles are required to bring about the benefit. If the disadvantage had never occurred, the positive results may have never ensued.An article I recently read expressed this idea very clearly. A karate coach had a student who was training to be a star. Week after week the karate student practiced his move, working ever so hard to perfect it. He labored intensely, yet after a while he realized that he had only mastered a single karate punch. Eager to excel, he inquired of the coach, if he could possibly learn more yet, the coach assured him that this was all he would need to know.

57

Sure enough, at the public Karate match, the student could not be opposed. He emerged a victor, using his one karate move. What was the secret behind his victory? The student was missing a left arm. The karate punch he had mastered used only his right arm. Furthermore, the only Known defense for this move was to grab the opponent’s left arm. The student’s biggest obstacle had become his greatest gain.Another example can be given to explain the above phenomenon. My father’s friend has a few children. One was born deaf. Surely, the family must have been heartbroken and bereaved. Having a special needs child can be difficult, but deafness is certainly exceedingly trying. At present, though, there is a thriving organization called Haazinu, which in Hebrew means “listen”. The deaf child’s father faced the issue with a remarkable approach. Instead of brooding over the obstacle, he turned it around, making it an immeasurable gain. Haazinu is an organization for the deaf. He sought to help his child in the best possible way, so he started an organization to help such children like his daughter. Imagine what would be if he hadn’t had a deaf child born to him? The whole organization would never have materialized. His daughter, now, is happily married to a special needs husband, an outstanding achievement!In general, much benefit is gained from obstacles if they are approached in the right attitude and frame of mind. The Karate student could have well given up. The deaf child’s parents could have sulked in depression. Yet, they did not. They took their obstacles and grew from them. They turned their obstacles into remarkable gains.

#7I believe that any obstacle or disadvantage can be turned into something good. For example the producer told the actor if it is drama use the chair, smash it turn it into emotion, which is drama. But of course if it is comedy use the chair, to fall over it. Such is life, use problems or situtations to make your life better. Or if it is drama that you feel, use your situation to show your feelings. I agree with the last sentence “from this experince the actor concluded that in any situation in life that is negative something positive can come from it, you can make it positive”. Whether it is drama or comedy. For drama maybe you’ll become a better person. For comedy maybe you’ll make some new friends. Any situation there can be endless solutions. From my personal experince after my parents were divorced I became a better person and made new friends.So in conclusion yes obstacles can become something good.

#8“Life is a stage,” says a famous poem. But is it really? On stage, if you miss a cue, someone will cover for you. If a prop is misplaced or a line forgotten, you can improvise. In the end, everything turns out okay and all the actors come out to take a bow. But real life is different. Missing a cue in real life can be fatal. Not always will someone be there to cover for you. Take 2 can’t happen in real life. You only have one take to do what you need to do, and go where you need to go. On stage, if something is negatively affecting a performance, it can be changed. The actors can almost always work around the obsticle, and if they can’t, then they start over. But life is not a show that can be rehearsed over and over again. Although there are many times where a bad situation can be turned into something good, something can always be unfixable. For example, people die every day. Why? Yes, of course there are those people who die of old age. But almost everyone else dies because of a difficulty—be it a disease, an accident, or even a murder. These people could not overcome their obstacles. For them, something bad did not turn out okay. How can a kidnapped person make good out of his situation? A person is bound, gagged, and left

#9I believe any obstacle or disadvantage can be turned into something good. It is all about what your attitude is and how you deal or react to things.

58

If you are an optimistic person than anything can look good, in the right lighting. Also, if you are not easily angered, than any situation can turn out to be a positive one.Keeping your cool, is also the key. Getting upset because something goes wrong, will not get you anywhere. There are many things in our lives that go wrong. We can’t get upset about the little things. We should only worry about the big things. And even those, we should take with a smile.Being positive in any situation will not only benefit you, but it will also help those around you. It will help them realize that any and every obstacle does not have to have a bad ending.So, in conclusion. I personally think that any obstacle or disadvantage can be turned into something good. With the right attitude and perserverance.

#10I believe that bad situations or any obstacle that are negative can be turned around and made better or looked at positively. Any time in your life there are going to be struggles and difficulties, yet it is how you deal with it that makes the difference.For example, if you are taking a test the you haven’t studied for and recieve a poor score, you learn from it and you know next time to study to recieve a better score.We learn all kinds of things through our mistakes, even as children. If you tell a child not to touch a hot stove and they do, not only do they learn that stoves are hot and can burn, but they learn to take advice and learn.Although you may not see it at the time, we know and learn the most through trial and error, through our mistakes. Going through obstacles also makes us stronger people.I know from my experience that it has helped me grow into a stronger person. I went through a crisis in my family, and although it was painful, I healed and learned that death is a part of life. I’ve matured alot from that experience and am now able to help others who go through a terrible experience such as that.Obstacles in life also teach you to appreciate the good. When you lose something or someone, it is then that you feel their true worth.In conclusion, I believe that obstacles and disadvantages enable you to learn, grow into a stronger person, appreciate everything in your life, and experience all that life has to offer. The good and the bad.

#11Life constantly harrasses modern society with a variety of problems and obstacles. But, negatives never come out without its positives. Events such as the Civil War, the lack of gay rights in America and messages that stem from Romeo and Juliet’s tragic ending all include something good even though all show obstacles of life. Therefore, society can gain good things through bad things.The Civil War broke out in 1861 and left a terrible tear on American culture. Even after horrible battles like in Gettysburg, Americans could still come together in the late 1800’s to form a more unified society. Even after slaughterings of American “brothers” between the North and South, our country still had the ability to transform itself to what it is today. Having not undergone the war (an obstacle the country had to overcome to continue to grow) our America would not be nation that is today—a nation that protects everyone’s freedom regardless of sex, race, etc.Moreover, Romeo and Juliet also acts as an excellent example of how disadvantages proves something good in the end. Shakespeare decided to end both the main characters lives at the end of the play but by doing so, he sends the good message of love. Love is the most beautiful thing in the world. Love connects society/people together by acknowledging that through death people can come together (the Capulets and Montagues) that is one of the most beautiful things ever.

59

Life will constantly be filled with problems but

#12When playing a sport or any game and your opponent has a great advantage over you, you must adapt. A possible solution would be to reverse the obstacles so that it helps you and hurts the other team. Sometimes when you are playing in front of an audience, you must make them see what they want to see. Another example is when you are giving a speech and you completely forget what to say. When that happens, you must continue to expand on your previous subject.

#13A question to end all questions. Can good come out of bad? Can a negative experience be turned into a positive one? The answer is yes, if you have the courage to face your fears.In the example the actor was unable to come onto the stage because of a chair. Really it was his fear of looking foolish or doing something wrong that held him back until he was given insight on how to use his obstacle to better his acting he was unable to perform as usual.In our own lives we face obstacles that will make us want to retreat. It may be a fear of getting hurt in a relationship or something as simple as a morning filled will “bad luck” that makes us want to curl up and retreat into the senselessness from whence we came.History is full of such obstacles. Take any great inventor or scientist. Everyone remembers them for their achievements, but in order to achieve greatness they had to overcome many obstacles. Edison made the light bulb, for which we are very thankful, but it took him so long he nearly gave up. He kept trying to come onstage, but there was a chair in his way and every time he thought he had conquered the chair there was a new one to face.Life is not easy. So many times I have faced things that I don’t think I can face. But I have, I have faced depression, suicidal thoughts, loss of friendship, and loss of love. But I’m still here, I faced my obstacles, my chairs and many times I couldn’t do it without help, without some advice on how I could use my chair for benefit.When we are not afraid to face our obstacles and use them to better ourselves or others we are taking negative experiences and turning them into positive ones. Our world, our lives would be so much different if we were not able to face the chairs of life and use them to our advantage. A little bit of courage and creativity goes a long way towards becoming a brilliant actor and person.

#14Life is what you make it. Time after time we are stuck with disadvantages and struggles, but it is how we receive it that makes a world of a difference. It is possible to turn obstacles into building blocks, to reach success.My most difficult school year was during the 8th and 9th grade. The school I was attending placed alot of stress on me—the homework was unbearable and the teachers seemed to care nothing for it. Many times as I went seeking help but would receive a cold shoulder or a slam in the face. Not only that, but I was not doing well socially with the rest of my peers. It was extremely difficult finding a way to fit in.I could have stayed and walloed in my misery but I decided to take the initiative and talk to my parents about the situation. Together we began building the blocks towards my success.That next year I left the school and we began home school. In one term I taught myself the subjects and this prepared me for what was coming next. My parents with the discipline I needed and I decided it was time to move on. I took a test to enter a very high academic—perhaps the highest on my Island—school to see if I would get in. I reached something beyond my wildest dreams and was accepted. Not only that, but the next year I was appointed head girl and am training to attend Harvard College next year.

60

My “painful struggle” through my early high-school years lead me to make the right decisions for which school I should attend to make the best of my future. Obstacles are only put to make us stronger, hurdles are placed so that we jump higher.

#15In life, there is good and bad situations. Some people can turn the bad situation into a good situation and they can deal with it in a good way. In the end they turn to be happy about their solution to their problem. There is some people that have a positive situation, they turne it to a really negative way for a silly reason. After, they will be mad and angry and they can’t fix the problem.An example that we see in our life that is turned to be good is that some students say we are not doing good in school. They start to study and work hard and they give extra time to study. At the end of the year, they do good and their happy about their grade. When a person don’t give up, he or she will get something good at the end. They have to see their problems in their life and face it so that they end up with something that they will be happy or agree with the solution. People should never give up in life because they still have time and they will experience more and more in life.Life is an experience you should know how to deal and work with the life. There is a positive things and a negative things in life but you have to know and take each step that is best for you.

#16A bad situation can be something good depending on what you make of it. We can all learn from our mistakes or from bad things that happen to us. Take for example a car accident. That can be one of the scarriest moments in your life. The first and only one I have ever been in happened a few years ago before I could drive. It was very scarry and we were only slightly injured. Even though it was a bad thing that our car had been totaled, Everyone (me and my siblings) weren’t hurt severly. Being in the accident also showed me how quick one can happen and that it can happen to anyone. It was a horrible experience but it helped me to be a more cautious driver when I did start driving.People can also turn a disadvantage into something good if they try hard enough. Look at someone who is really short. Many times people think if they or if someone else is short they can’t play sports, especially not basketball. This is not true. I know quite a few people who were told because they were short that they couldn’t play basketball. These people worked really hard and became very fast and concentrated on what they could use their height for. At least one of them is an awesome defensive player and the other is always in the spot to make the shot because they are so fast and know how to maneuver around other people. Both of these people could have given up at playing basketball when others told them they couldn’t do it. Instead they tried twice as hard and used their disadvantage of height to help them play on a totally different level.Everyone just need to remember that a situation or disadvantage is only what you make of it. If you let a situation be bad and don’t look for the positive side, then it is only going to be bad. If you let someone tell you that you have a disadvantage and don’t try to find a way to use it to your advantage, then it will always only be a disadvantage.

#17I do believe that any obstacle or disadvantage can be turned into something good. I believe this because people learn many things from their mistakes and that’s the only way. In the example that is provided, the actor is obviously confused. The director makes a good point on how to work around it, and in a case where they make use of the chair that’s in the way, which helps make it funny, exciting, or depressing. In a play or musical of any type, it adds meaning. Not only does an obstacle or disadvantage get turned into something good, but it can discover new things. Whenever any situation happens, such as in this case, I’d laugh if it’s funny, cry if it’s

61

dramatic, make a joke out of it, or act like it didn’t happen. Just keep in mind that the purpose of life is to learn from mistakes, which can turn into something good.

#18I have heard the expression “when life gives you lemons, make lemonade”, and believe this to be a very good way to look at life’s difficulties. There have been many times in my life where something has happened that I thought (at that moment in time) was impossable, but they have always been for the greater good. (If only as a good learning experience.)For example, last summer, I lived—alone—in Sagunto, Spain. When it was time to return home, I boarded a plane for Madrid, and flew to my appointed destination. Upon arrival however, I discovered that my plane had been delayed in the air, and that my next flight (to New York) had already departed without me. I couldn’t get another flight that day (in all actuality, probably not for many days to come) and I sat in the airport and cried. I realized however, that sitting and crying was, infact, not helping me toward my goal of returning to America in the slightest. And with this realisation, I got to my feet, and spoke to the man at the Iberia desk. By doing so, I recieved food and lodging for the night, and got a flight worked out for the following day. From this experience I learned a valuable lesson in independance, and also discovered how to have faith in my own ability. A lesson I may not have learned so well in any other way.In Jane Austen’s “Pride and Prejudice”, a certain Miss Elizabeth Bennet makes assumptions about, and learns to hate another caracter, namely Mr. Darcy. This situation seems very bleak, and depressing to her at the time, causing her much sadness and anger. It the end of this novel however, Miss Bennet and Mr. Darcy are married and it is most possible that she would not have been able to love this fictional caracter nearly to the extent she did, had she not hated him first.My last example is Ginny Owens, a now famous singer, who was born blind. Blindness may seem a great obstacle to one not afflicted by it, (and to many who are) yet she used this disadvantage to her advantage, and improved her voice, and ear for singing. She says she would not be nearly the singer she is today had it not been for her

“disability”.In closing, I believe that any obstacle can be turned into an advantage, simply because life has shown this to be true for me. I hope never to have the inability to see past a currently debilating situation, and into the good it could cause for my future.

#19In Lawrence Einsenburg’s “Caine Scrutiny”, it says “in any situation in life that is negative, there is something positive you can do with it.” This statement is true; my mother always told me that in order to succeed, you have to have a positive attitude.An example in literature is in “The Scarlet Letter.” When Hester Prynne was condemned to wear that scarlet letter “A” wherever she went for committing adultery, it was an enormous obstacle for her. She was outcasted by society and looked down upon by everyone. However, because of this letter, she became a stronger, smarter person. She would humble herself and help out society in little ways. Soon people came to know that scarlet letter “A” to stand for “able” and not “Adulturer.” She became a good role model to her daughter and a legend in that town.Another example of a positive coming out of a negative is in Marcelo Cigliuttis situation. He was the first man to marry another man legally. He fought for what he believed was right and finally, Argentina legalized gay marriages. Although at first, it was a great controversial issue, and a tough battle to fight, it was worth it because Argentina took the first step. Now, one by one, other Hispanic countries are allowing gay marriages.A personal experience of making a negative situation a positive one is my first rejection from a college. I carefully put my life into my 4 page application. I made sure everything at least met the requirements. I sent it in and

62

anxiously waited for University of Florida’s reply. “Regrettably, we cannot accept your application.” It was heartbreaking and life ruining. However, I was really torn between UF and FSU, so being rejected by UF actually made the decision for me, and made that a lot easier. Now I am excited to go to FSU and see that everything really does happen for a reason.Everyone can make a negative situation, and still suceed by having a positive mind and always looking for the good. “What doesn’t kill you, makes you stronger,” right?

#20I think that most of life’s obstacles or disadvantages can be turned into something good. The way you look at situations determines the outcome of a situation because if you look at the situation negatively you don’t have any hope that the outcome would be positive; but if you look at the situation positively you would have a better chance at being successful at what you do. The writer in the passage saw the chair in his way as an obstruction to his performance but the producer had a broader perspective saying that he could use the chair according to what the genre of the play was. For bad things to work out we need to stop being so narrow minded and broaden our perspective on life. If you were a drug addict most of your life and you got clean don’t just concentrate on the bad that happened in your life, use that experience to share with others showing them the hardship that goes with that type of lifestyle discouraging them not to do it. The outcome of a situation is determined by how you look at it.

#21Obstacles and disadvantages are in our lives every second of the day. Whether it is a big test in school in an important subject or a traffice jam keeping us from getting to work on time, we can always count on something going a way. However, even the biggest obstacles can be turned into something good if enough work is put into it.Small obstacles, like the big test in school, can be overcome with hard work and determination. If you spend enough time studying beforehand, and doing practice questions to build your confidence, chances are you will do very well. On the other hand, if you procrastinate and do not prepare for the test until the night before it, chances are you will do poorly.Large obstacles can be overcome with hard work and determination as well, but depending on the size of the obstalce, carefully planning, organization, and coordination are needed as well. For example, in World War II the Allied powers were fighting the Axis powers in battle after battle. The Allies needed to get into Western Europe through France and through a heavily fortified enemy. With careful planning, organization, and coordination, they launched the largest amphibious invasion in history and took the beaches of Normandy in what would be called D-day, and liberate France soon thereafter.In conclusion, any obstacle can be oversome into something good. Tests can be studied for, traffic jams can be overted by detours, and wars can be won against impossible odds. Since obstacles confront us everyday of our lives, we need to be able to overcome them. If we could not, then life would not be enjoyable.

#22If I when an actor’s name is called out to perform a drama act, in my point of view on this issue is that he should pick up a chair up and smash it. If it’s comedy, a person should do something funny nor stupied. For example falling over a chair is something that people will laught at. From this experience the actor concluded that in any situation in life that is negative, there is something positive you can do with it.When an actor’s name is called out, he or she always forgets to what he or she have to say. An obstacle or disadvantage could always turn into something good. A negative situation can always turn into positive. When your on the stage performing with thousands of people facing you, you turn red and always try to do it better.

63

#23Life is full of obstacles. They are an unavoidable facet of our existence. When we are met with obstacles, we can either choose to let our disadvantages consume us, or we can rise above them. It is said that “Whatever doesn’t kill you makes you stronger,” so any disadvantage can be turned into something good. A disability can connect a person to the people, a sports event can help ease racial tension, or even something as simple as a pause in a line during a comedy sketch can be turned into an advantage by giving humor to a world that sorely needs it.In the early 1980’s, “Saturday Night Live,” the live sketch-comedy show, was going through a dry spell. The original cast had moved on to bigger and better things. Lorne Michaels, the creator had been fired and wanted nothing more to do with the show, and the cast was viewed as being unable to live up to the original “Not Ready For Prime-Time Players.” All, of course, with one exception: the unknown Eddie Murphy, whose comedic timing and audience appeal were the only things keeping SNL from the brink of cancellation. His spot-on impressions of Buckwheat, an inner-city Santa, and Gumby as a hardened old showbiz veteran kept audiences roaring, and kept ratings coming in. Eddie was able to use any situation to his advantage. It was said that if a scene wasn’t working, the writers would put him in the scene because he could “fix” anything. On one night, Eddie flubbed a line. The audience began to laugh, so Eddie shouted “Shut up!,” which made them laugh even harder. Eddie managed to turn an embarrassing moment into one of the great comic moments of all time.The 1988 Calgary Winter Olympics saw a new team enter the sport of bobsledding: the Jamaican team. No one thought that a team from such a hot climate could ever compete in the Winter Olympics. The Jamaican athletes faced many hardships at those Olympics. Biased judges, fierce competition, and racial tension from all the other teams, especially the front runners, the Swiss team. However, the Jamaicans persevered. Though their sled malfunctioned and they won no medals, they taught everyone about perseverance and personal strength. They turned their seemingly hopeless situation into a learning and teaching opportunity and, at the next Olympics, returned as equals.President Franklin Roosevelt had a difficult life, to say the least. Stricken with polio as a child, he was left partially paralyzed and was confined to a wheelchair. However, he did not let a physical handicap stop him from pursuing a political career. He sent his wife Eleanor to be his legs, making appearances and helping out during the Great Depression. His fireside chats helped America out of the Depression, and his courage during World War II inspired our nation to greatness. Many even said that his physical handicap was an asset to his presidency, because people felt they could relate to him. He was “the people’s president”. FDR is a model for triumph in the face of adversity, and is proof that anything can be used to one’s advantage.No matter how big or how small, anything can be turned from an obstacle into an advantage. It is said that “when God closes a door, somewhere he opens a window.” Something as simple as saying “Shut up” or as complex as overcoming racial barriers or a physical handicap can be made into something good. “When life gives you lemons, make lemonade.” Or at least throw the lemons at someone you don’t like.

#24I believe that this statement is indeed true. There is a common saying that goes: “When the going gets tough, the tough get going.” Difficult situations seperate the good professionals from the great ones.For instance on this test. All college bound students are faced with a grueling, nearly four-hour test. The good students may study a little bit, or none at all and hope for the best. The great students will put lots of time into studying and try very hard to recieve a score that will be beneficial to their future.Another example comes from the fact that I am a soon-to-be driver. In the State of Ohio, all perspective drivers must complete 24 hours of Driver’s Education, eight hours of in-car instruction, and 50 hours of driving experience. This is quite a lot of time and energy that must be placed into earning a driver’s license! The weak individuals who shirk away from difficulties may try to wait until they are 18 (to avoid all the driver’s Ed.), but strangely enough, most teenagers choose to accept these challenges, and work hard to deserve their license. I say

64

“strangely” because it is the normal action for any one to avoid difficulties as opposed to facing them. This can be easily shown from the prompt itself. The actor sees an obstacle and decides he cannot complete his job. The director however, tells him that he must use the obstacles in order to get ahead in life.Another personal experience that supports the truth of the assertion comes from basketball practice. This year everyone decided to take the season more seriously and really work hard at having a great season. This means practice for two hours a day, five days a week. In those practices, running is our most common activity. The running is difficult, and there are two ways to approach the situation. Either a person can quit the team because it is too hard, or you can run as hard as possible knowing that every step you take makes you stronger I personally decided to run hard every time, and the result, our team has a six man rotation where every single person is able to run the court for all 32 minutes. Others on the team decided to jog some of the time, or not run at all, and they may have gained something in the short run, but I am now miles ahead of them in the long run.There is a quote that says that a person came upon two paths in a forrest. He took the less weathered one, and it made all the difference. I believe this relates to the prompt because most people choose the more weathered, or in this case easier, path, and end up losing in the long run. It is unfortunate that people can’t be more motivated, but until they become motivated, I will try to take advantage of their weakness. In life, its not always about how much skill a person possesses, but its also about how much work that person is willing to invest, especially during the hard times. The ones who can use the difficulty to their advantage will excel, whereas the ones who never enter the stage will forever be stuck behind the curtains.

#25This sentence is very good and right because you must belive thet have a good and positive, something also in a bed situation. I you think so, you succeed in your life. also in a bad situation, you must think a good and be a good. I have 3, examples from my life: 1 I basketball player I have alot of games and tugh games. I want a great player. I am alaways think good befor every game and also befor every practice. this thinking help me to secceed. If I lose a game I don’t cry I try to take the good things from the bad game, this is not help me if I cry all the time. 2. I had a important text befor 3 years I remember this text all the time, beacuse that I got a high score, because I learned alot before this text, this text all the time in my memory because the way was very difficulte, I thought good befor the text, and I secceded I all the time thought good and positive. 3. I didn’t secceded in the first, text driving. I didn’t cry, I tried to learn from my mistakes. In the second text driving I secceded beacuse I thought positive and I said to myself that I good driver and I will secceed. From this exampls I can to learn that I you think good and positive you secceed all the time. Also if you mistake or don’t secceed you must to take the good and the positive from your mistakes. I can to say to the end that if you thought good and positive and you learn to take all the positive things. you be enjoy and you will be fun and you succeed, all the time, you need to learn from the mistakes. Thare arnet perfect people! Bielive yourself!

#26The question is can any obstacle or disadvantage be turned into something good? Yes it can, because even in the story he couldn’t get in because of the chair and the guy tells him if it’s a drama “Smash it and if its a comedy fall over it. But when you think about it, if a girl was guilty and was pleading to say on how she was inocent and they catch her in a lie its practically the same thing. I mean that, because of what she did she is going to be punished. Like if you broke your watch or scoffed your sneakers, you get a new watch, and pair of sneakers or you get sneaker polish and clean it. But that’s not the point but you can change a negative into a positive someway. out of every negative their is one good positive.

65

#27Any obstacle or disadvantage can be turned into something good. In life we often meet obstacles & difficult situations which must be delt with. A persons choices of how to deal with them will detirmine an either positive or negative outcome. Any obstacle or disadvantage can be turned into something good.In the world of martial arts there are techniques for defending ones-self from the offensive tactics of an opponent. These techniques vary in purpose. Some merely stop the offensive movement while others use the energy and force of the opponenet and redirect it for the purpose of using it against the offensive opponent. In life these techniques can be used to deal with obstacles and disadvantages. A bad grade in a class, for example, can have a positive effect on the student if the student chooses to be motivated to excel above his current grade. Other examples could involve having a negatvie experience, and later, on in the life of the person, when a similar experience arrises they will have had the experience before and will be better suited to deal with it again.Even if a disadvantage of obstacle seems to have no obvious outcome for the seemingly unlucky in life, there is always the opportunity to improve ones character from such an obstacle of disadvantage. Wisdom may be gained as well as virtue. Hardship can have adverse or positive affect on an individual, and the affect all depends on the additude of the person.It is possible for an obstacle or disadvantage to have positive outcome and it all depends on the aditude and prespective of the person. It is a choice of the individual. Any obstacle or disadvantage can be turned into somethin good.

#28I live in a house that evry body in it came from acting. I remember my mom telling me this it you in find your self bad situation, don’t forget your smile wite “you”. I think she ment that what ever is the difficulty think always positive. For an example, I grow up in place that full wite bad poeple and one time some body try to convinse me to smoke. And smoking it very bad thing. So I started to tell joukes on people that canser and after 2 minutes I change the subject. Or that every time I am getting sick and fill not so good. I am trying to see comedy movies as much as I can. Because I have been told that comedy is the best cure. I think that as an actor on the stage you need to be always ready for something rong, and if you ready and prepard. It will be good and life for your self in you all life and not only there. This experience importent for your benfits, always a positive person and people will love you and get along wite you. This mark it the best.

#29Being able to contain a positive outlook in life is key. A person can somehow be able to turn a disadvantage to an advantage. It all depends on your critical thinking and how you decide to handle the situation. Having a good attitude will make a quite large impact, but being reluctant towards the situation won’t help. Also, on obstacle or disadvantage can come to handy. You can be able to take the situation, observe it, and learn from it. A disadvantage or obstacle would make you put more thought into the situation therefore helping you think. Being put in a situation will determine what type of person you are. For example, how you handle situations. Learning from your mistakes helps you remember it a lot better. Any type of disadvantage may help you in the future. In other words, I do think that any type of obstacle or disadvantage will be of great benefit to you later. It just depends on how you plan on dealing with the obstacle presented. You can take what’s handed to you and change it into something positive. In life you’ll come by many obstacles and the great part is you can choose how to deal with it. Either negatively or positively. It’ll catch you off guard when you notice how much a disadvantage has some how helped you. If you look at any type of disadvantage with a positive outlook you’ll learn to be grateful towards whatever disadvantage is presented. But if dealt with in a negative way you won’t accomplish much from it. In conclusion, being able to turn something to your disadvantage into a advantage is possible. It just depends how you handle it.

66

#30I am the daughter of Israeli immigrants. My parents arrived in Los Angeles without any distinct American background. They scarcly spoke English and I was raised speaking only Hebrew. Although English was not my first language and I struggled in school as a kid, my disadvantage became incredibly helpful.Upon entering high school I struggled in writing. My teachers, however, guided me in the right direction. In Hebrew, I excelled. In my sophmore year, my school, a Jewish Day School, offered an Arabic course. My background in Hebrew enabled me to take the course. I was the only one in the class. Today, I can now read, write & speak Arabic almost fluetnly. This gave me hope. My Hebrew opened the door for me to learn more languages. Arabic was not all.My Freshman year I took spanish, and I struggled in the beginning. The vocabulary, conjugortians, verb agreement, greatly confused me. After sometime I began to see the relation between English and Spanish. Miraculously, I became the top of my class. I continue yearning to learn more languages.This year I am a junior. I began to take French 1, in addition to Arabic 3 and Spanish 3 Honors. I am also in an A.P. English language course. My disadvantage of entering Kindergarden without any background in English motivated me to love learning languages. Someday I hope to learn Italian, Greek, Chinese, and other languages, in addition to majoring in English. This is my story.

#31There are many different ways to have an outlook on life. One may be pessimistic or optimistic. One may see the glass as half full or half empty. Generally when there is an obstacle in life or something doesn’t go as planned people have two choices; one: they can be upset and stop whatever it is they had planned because something was preventing the plan to be carried through or, two: one can take the path produced by the obstacle and have an exciting adventure while still producing an outcome. All things in life; obstacles and even disadvantages can be viewed for the good. Everything in the world has the potential to be good. As an everyday example: roadkill, an animal gone to waste? Not quite. If one were to take a closer look, that dead animal provides food and sustenance to insects and bugs and in some places vultures. Parental Divorce can be a great obstacle in someone’s life. As a teenager it would be very easy for me to just give up on my life as it is. Seeing the damage that can be caused, I may think, “what use is there?” Even a hardship like this has the potential to create some form of good. Instead of giving up my life I have taken notes and became observant to certain charachteristics that may make or break a relationship. With this outlook on the situation I was put in, I am not only continuing to make my life worth living in, but also the lives of people around me and future generations.Authors have a great skill in finding good in any obstacle or bad situation. If the charachter in a book were to come upon an obstacle and give up all the books in the world could be two pages long! The authors, though, plan the obstacles and when the charachters approach that fork-in-the-road, they usually find the correct way out, that gives the charachter a feeling of happiness; to have not given up, and the reader a feeling of happiness; to have more pages to read.In conclusion, it is possible for any obstacle or disadvantage to be turned into something good. All in all, it is not only possible but probably healthier too. Every person, no matter what is thrown at them, can swing the bat, maybe get a few strike-outs, but in the end they can grow and learn from mistakes and with practice and patience can hit a home run. To live one’s life to the fullest, no matter what one’s position in life, is trully pausible; the roadkill feeds the insects, the parent’s troubles strengthen their kids and the author’s charachters show light at the end of the tunnel for all the readers.

67

#32We can any obstacle or disadvantage turned into something good. Because the actor had his reason when. He said,

“I can’t get in, the chair is in the way.” And the producer said, “Use the difficulty. If it’s a drama, pick the chair up and smash it. If it’s a comedy fall over it.” From this experience the actor concluded that is any situation in life that is negative, there is something positive you can do with it.We need take from the line the beautifull momment and for ever remember it, to when came the bad times there is something positive you can do with it. We need obstacle and difficulty to get out the best from us.

#33Any obstacle or disadvantage can be transformed into a positive thing. For example, a handicapped person can be exempt from many things that he doesn’t want to perform. Another example is a tree that doesn’t grow any nuts or fruit can be considered a bothersome mass or a good place to rest in the shade. Something else that can turn obstacles to advantages is the usage of every resource. As stated in the paragraph, to enhance the film, the actor should smash the chair or trip over it. Either of these options would probably make the film more realistic, therefore turning the obstacle, the chair, into a positive enhancement. Also, something that might seem totally negative, like an accidental cut, might be good for you because now you have something to show off at school. Well, as the old saying goes, when life gives you lemons, make lemonade.

#34I know that everyone faces disadvantages in their life. Some not as bad as others. But, all obstacles can be turned into something good. You may not realize it yet, but if you think back, then you will see that all things work together for the good.One reason I believe this is because if you never come face to face with an obstacle or disadvantage then you will never learn from your mistakes. Many great people became famous because they were stuck in an obstocle and had to find a way out. It worked for their good as well as others around them.Another reason is because a disadvantage may not be too good for you, but for someone around you. Some one may see you in a disadvantage and they will be encouraged to get through their tough obstacles. If you had not experienced that disadvantage, then that other person would give up in their situation.These are just two of the many reasons that I believe any obstacle or disadvantage can be turned into something good. You could be helping yourself and someone else simultaneously.

#35There are many things in life that may seem to have a difficult or depressing outlook but if you try some of those things can be turned into something outstanding. For example there may be certain people you may run into on a day to day basis that may seem to carry themselves in a bad way but you have to try to get to know them because you never know that can turn out to be one of the best people in the world.Many people in our society look at things such as driving a car, to be a very hard obstacle because they feel that it’s something that is hard to do, but until they try they will never know. If we’d learn to look at the bright side of things instead of the negative then many things that we face in our life would turn out for the better. When you think you may have down all that is in your power to do to try to get the good out of something, believe me there is always someone who will be be able to help you. I know that there are things that may seem too hard or difficult but if you put your mind to it everything will come out for the better. I know there have been some times in my life when I have been in some difficult situations such as the time I lost money out of my wallet and I didn’t know

68

what I was going to do. but I stopped worrying about it and it wasn’t long before I had my money back. Whenever you come up on something in your life that you may think that there is no way of accomplishing

#36In my lifetime, I have seen many ways an obstacle or disadvantage can be turned into a good thing. Sometimes when something bad happens, it can prepare you for the future obstacles to come. For example, if a person gets a serious burn they would panic and not know what to do. If the same thing happend again, the person would be more prepared and have knowledge from the previous experience.If someone is in a wheelchair, about to open the door, would you open the door for them? If you are the person in that wheelchair you would be at a disadvantage to others. But this can be turned into a good thing. You now have someone to open the door for you and somebody who is at your mercy.Sometimes when things aren’t going your way, you have to improvise a little. For example, before a basketball game you see that the other team is much taller than your team. The coach might have only made up plays for a short team. The other team is tall, but slow. You use your speed to win the game. In the blink of an eye, you’ve changed a disadvantage into a good thing.The best comedians always turn an obstacle or disadvantage into a good thing. If the audiance is not laughing at his performance, then he has to change what he’s doing. This obstacle about the audience not laughing he could make a joke about, then making the obstacle into a good thing.

#37It is true that any obstacle or disadvantage can be turned into something positive. In a negative situation there is always a lesson that is learned, or a goal that is achieved. This idea is supported by aspects of “A Doll’s House” and the road to the Revolutionary War.

“A Dolls House”, a play by Heinrick Ibsen, proves that an unfortunate situation can have a fortunate outcome. The play tells the story of Nora Holmes and her quest for personal independence. Nora’s situation at the begining of the play seemed promising. Nora was married to Torvald, had two children, and had a stable financial situation. However beneath the facade, Nora’s life was not satisfactor to her. Torvard treated Nora as a child and did not respect her opinions as an adult. After a particularly harsh incident in which Torvard threatened her position in the family, Nora knew that she had to leave permanently. She either had to continue to endure the pain and emotional distress caused by her husband, or learn from this obstacle and use it to achieve her goal. When Nora finally left, she walked into her new life as an independent woman.The events that unfolded on the path to the Revolutionary War, support the thought mentioned above. At the start of the process, the colonies were a group of seperate entities under the rule of Britain. Each colony led an individual existence and grew and prospered under British rule. However, after a number of happy years, the British decided to tighten its hold on the colonies. They limited the freedoms of the colonists with acts such as the Sugar Act, Stamp Act, and Townsher Acts. The colonists were outraged at this sudden removal of freedoms which they felt were rightfully theirs. To overcome this obstacle, the colonists knew they would have to join together and unite against the common enemy when the colonies Declared Independance, they were no longer a group of separate entities, but one United whole that went on to become one of the Strongest Nations.Any obstacle or challenge can be looked at from a positive angle. The examples mentioned above prove that there is always something positive that can result from a negative situation. We should look at challenges not as unfortunate experiences, but as chances to improve ourselves and our lives.

69

#38In times of desperation, it is often difficult to see the positives in a situation. More often than not, our survival instinct demands that we obliterate any obstacles in our path, without heeding the potential consequences. However, I believe, using Emily Bronte’s Wuthering Heights and early American history, that it is indeed possible to use these apparent disadvantages as a means to improve yourself.The hero of Wuthering Heights, Heathcliff, was by any account, a man of humble origins. Not only was he an orphan without a last name, but he was also abused and tormented by other children—such as Hindley—of his new household. Constantly frustrated at every turn in life, Heathcliff as a boy could not marry Catherine—who he loved—partially because of his lowly social status. After Catherine left to marry her new husband, Heathcliff undertook a journey, in which he amasses a good deal of money and seemingly elevates his place in society.Although these changes are superficial, Heathcliff, used the adversity facing him as a boy as motivation to improve himself, to marry Catherine; thus his early obstacles were turned into something good (at least for him). From Emily Bronte’s Wuthering Heights, it is evident that obstacles can be transformed into motivation, a very positive emotion.In the 1770’s, America was under the rule of a tyrant in England. Legislation, such as those that were called the

“Intolerable Acts” that forbid such practices as forming a militia in Massachusettes, and the infamous “Stamp Act”, which was essentially a tax forced on Americans to gain revenue for the British Empire, were passed continuously against Great Britain’s colonies in America. The Quartering Act forced Americans to allow British soldiers to live in their homes, which resulted in many rapes and the situation was not good. However, the early American political leaders used these dire times to rally the American people. The country, incensed by British practices, joined those rebellious leaders, such as Patrick Henry, George Washington, and John Adams, to fight in the Revolutionary War against a corrupt monarchy. If the American people had not suffered through these indignities leading up to the Revolutionary War, the United States of America probably would not exist today.Clearly, when one is faced by obstacles at every turn, it is extremely difficult to try to twist negative experiences into positive ones. However, if one takes the model shown by Heathcliff of Wuthering Heights and America’s early political leaders, one can use these bad experiences to totally change one’s life for the better.

#39Very often, obstacles or difficulties encountered across one’s path (whether physical or emotional) can be discouraging, even devastating. However, good often springs from the bad. What one may percieve as a hindrance may in fact ultimately turn out both beneficial and positve. Hester Prynne from The Scarlet Letter committed a grievious sin which changed her life forever; her pains and difficult life, however, caused her to become a stronger, prouder, and independent woman and mother. Similarlly, when the Articles of Confederation proved to be an ineffetive solution to the government plans; many debated with difficulty how the Articles could be bettered only through conflict,arguments and compromises came about an efective solution to the problem-the US Constitution.Hester Prynne, the protagonist from The Scarlet Letter is presented to the reader, at first, as a beautiful, vivacious woman. After Hester commits adultery, however, the feelings of the town towards her turn malicious and hateful. Ostracized from her community, Hester is forced to rely not on her charm and beauty, but on her strength, personality, and persistence to survive. Thanks to her sin, Hester is compelled to become a stronger and more independent thinker; she also devotes much of her time to charity work, for she can now appreciate the ruins of society’s rejects.The Articles of Confederation proved, rather quickly, to be an ineffective man for a government of a budding; large nation. The Articles were unable to unite the States and failed to create a strong central government for the collective people. When the constitutional convention met in Philadelphia, many of the delegates were prepared only to amend the articles, not draft them anew. Throughout the ensuing months, the delegates argued debated,

70

and fought in order to reach a final compromise which created the US Constitution, whose government plans and policies the American government runs by today.Although obstacles are often present in almost every possible path, many times these obstacles prove to be beneficial and helpful in the long run. Without these problems to guide people in the right direction they may lose their way or else may not stop to think or consider before they act. Although obstacles are often painful and difficult to overcome, they force man to evolve and learn to survive. Without painful obstacles in the way, it may become extremely difficult to appreciate life and strive to improve it. Instead, man must learn to overcome his own obstacles and create a better life for himself, by himself.

#40When obstacles or disadvantages cannot be something good the result of the obstacle is a negative one. It could be applying to college with poor grades, being underage to gamble in Vegas, or shoe shopping with a swollen toe.When my friend was applying to college, she had a major disadvantage. Her school grades on her transcript was very low. She needed higher grades to get into college, therefore she was not accepted she had a disadvantage and her result was bad.Another personal experience that portrays my thesis is when I took a trip to gamble in Vegas. I wanted to gamble, but I was underage. My age was an obstacle for having a good time and enjoying myself. Because of the obstacle I had in my way, my result was negative.To further prove my stated idea, I have an additional example. I was in a desparate need for a new pair of dance shoes, to make my dance performance would be spectacular & when I went shopping for my new dance shoes, I had a broken toe all bandaged up on both feet. The shoes I bought that day fit my feet with the bandages. When it came to my performance the shoes were to big, and the shoes fell off, making my show horrible. If it were not for my obstacle, the broken and bandaged toes, I would of had a wonderful show. But because of my obstacle, my performance resulted a failure.An obstacle or disadvantage cannot turn into something good. Instead it will ruin a situation and have a negative outcome. Applying to college with bad grades, being underage when gambling, or shoe shopping with broken, bandaged toes, portray this idea.

71

Appendix J: Range-Finding Round 1 Evaluation FormThe purpose of this evaluation form is to obtain your feedback about the standard setting process that you have experienced so far on WritePlacer and the Modified Body of Work standard setting method. Your feedback will provide a basis for the facilitator to determine what information may need to be reviewed before continuing the standard setting process.Please complete the information below. Do not put your name or identification number on the form because we want your feedback to be anonymous.

Gender: _________Male_________Female

Race/Ethnicity:

_____Hispanic _____Asian _____African American _____White

_____Other (Please specify:___________________________)

Years of Experience as a Faculty Member:

_____1–5 years _____ 6–10 years _____11–15 years

_____16–20 years _____20–25 years _____25+ years

Affiliation:

_____ 2-year college _____ 4-year collegePlease read the following statements carefully. Place an X under one category (strongly agree, agree, disagree, or strongly disagree) to indicate the degree to which you agree with each statement.

Strongly Agree Agree Disagree

Strongly Disagree

1. I understand the purpose of this workshop.

2. The facilitator explained things clearly.

3. I feel comfortable with the task of evaluating each essay.

4. I feel comfortable with the task of assigning each essay to a category.

5. I am using the performance level descriptors that were developed earlier in this process to guide my thinking.

6. I am ready to begin round 2 of range finding on the WritePlacer test.

Please add any comments you feel necessary to the back of this form.

72

Appendix K: Modified Body of Work Essay Grouping Record—Range-Finding Round 1 and Round 2Rater’s Unique ID Number:__________ Group Number: _______For each essay in the set, indicate the group into which you think the student exhibiting the skills and abilities demonstrated by that essay should be placed. The goal of this process is to set one cut score to separate essays into two groups—either remedial (R) or entry level (E).Beside each essay ID number listed below, place either an R for remedial or E for entry level to show into which group you are placing the essay.(For round 2 of range finding, the same record sheet can be used by marking through the old category and replacing with the new category for any essays for which you wish to make a change.)

1. ________________________ 17. _______________________ 33. ____________________

2. ________________________ 18. _______________________ 34. ____________________

3. ________________________ 19. _______________________ 35. ____________________

4 _________________________ 20. _______________________ 36. ____________________

5. ________________________ 21. _______________________ 37. ____________________

6. ________________________ 22. _______________________ 38. ____________________

7. ________________________ 23. _______________________ 39. ____________________

8. ________________________ 24. _______________________ 40. ____________________

9. ________________________ 25. ____________________

10 ________________________ 26. ____________________

11. _______________________ 27. ____________________

12. _______________________ 28. ____________________

13. _______________________ 29. ____________________

14. _______________________ 30. ____________________

15. _______________________ 31. ____________________

16. _______________________ 32. ____________________

73

Appendix L: Official Score for Each Essay in the Provided Essay Set

Essay ID Score Essay ID Score

1 3 34 4

2 1 35 4

3 8 36 5

4 4 37 7

5 6 38 8

6 8 39 8

7 2 40 4

8 5 41 7

9 3 42 3

10 4 43 2

11 5 44 8

12 1 45 1

13 7 46 2

14 6 47 7

15 2 48 7

16 6 49 5

17 3 50 2

18 6 51 5

19 7 52 8

20 4 53 3

21 6 54 6

22 1 55 4

23 8 56 4

24 7 T1 7

25 2 T2 3

26 1 T3 3

27 5 T4 4

28 2 T5 3

29 3 T6 5

30 5 T7 1

31 7 T8 5

32 1 T9 7

33 2 T10 6

74

Appendix M: Pinpointing Essays for Round 3#41I believe that an obstacle or disadvantage can be turned into something good. Lance Armstrong, the most successful cyclist of all time, had to overcome cancer before achieving such greatness. Another possible example is Louis Armstrong. Louis was born to a single mother and lived a rough life. He was convicted of a crime and was sent to prison which is where he learned how to play the trumpet.Before he was a seven time Tour de France champion, Lance Armstrong had to overcome cancer, a very extreme obstacle to overcome. Lance was diagnosed with testicular cancer and by the time the doctors found it, it had already spread to his brain, lungs, and other intrinsic organs. After numerous surgeries, Lance was given a fifty percent chance to live. Armstrong did not give up. After months of treatment and another surgery, the doctors found absolutely no cancer in his body. Within a year, Lance was back on the bike. He went on to win seven Tour de Frances in a row and he remains one of the greatest cyclists, if not the greatest cyclist of all time.Louis Armstrong, one of the greatest Jazz trumpet players of all time, also had to overcome difficult obstacles in his life before achieving greatness. Louis was born to a single mother. They were poverty stricken, and were basically stuck with nothing. As a boy Louis was convicted of a crime and sent to prison. While serving his time, Louis learned how to play the trumpet from a fellow in-mate. Louis immediately fell in love with music. He formed a jazz group when he was released from prison and they started playing at local clubs and bars. On one fateful day, a record company representative heard the group and immediately wanted to sign them. If it weren’t for prison Louis might not have ever even picked up a trumpet.From these two examples, we can clearly see that obstacles or disadvantages can become something good. Both Louis and Lance overcame extremely difficult obstacles and became some of the best there ever was. They knew that anything was possible and were not discouraged by hardships.

#42Can any disadvantage turn into something good? Every person answers this question differently, and that is his way to live. Some are optimistic, some are pasimistic, either way, the way we answer this question is, in a way, our way of life. Some people can look on something that to me it is wonderful and find it terrible, and other can chear me up when something sad happens, and show me a nicer and chearful way of looking on it. A good example is what happend today, this morning. I came to take this test from my home which is a few drive from the test center for different reasons I came in late in about ten minutes, but instead of being stressed because of it, I was (and still am) quite calm, and actually in very happy and satisfied with my approach to this in compare to my father who took it very hard. I believe that the ability to look on the bright side, in both better and worse cases, is a recipie for a better and calmer life.

#43I do believe every obstacle or disadvantage could be turned into something good. If it’s something simple like trying to figure out how to get around an object that’s blocking you from something, like how the chair was blocking the actor in the excerpt adapted from Lawrence Eisenberg in “Craine Security”, you don’t really learn anything. If you are a poor person on the street, the advantage there is that you learn instincs, and street smarts. One day when that person isn’t poor anymore, they’re goin to know what the real world is like.

#44Many have said that life is a journey. As with any journey, there are obstacles on this journey we call life. What do we do when we come to these obstacles? How do we adjust our perfectly planned out-schedule, our thoroughly

75

thought-out goals, to adapt to these obstacles? As obstacles are an inevitability of life, we must learn to use them to our advantage. Regardless of the obstacle-big or small, serious or petty, it may be turned into something good when approached in the right manner.The short story “Goodman” by Etgar Keret, shows an hitman with a problem—he loves the woman whom he has been hired to kill. Keret portrays the hitman’s agony as he vacillates between feelings of loyalty to his love and to his duty. Surely, the hitman has encountered a significant obstacle; however, he does not let this obstacle cripple him. He does not let the obstacle destroy his psyche, and instead, uses it to his advantage. His obstacle causes him to do some self-examination, after which he discovers that he is unhappy with the person he has become. The hitman lets his love love, quits his job, and begins to transfort himself into a better human being. The obstacle he encountered broke the monotonous routine of everyday life, eventually resulting in change for the better.As a little girl in Poland, my Grandma’s life began as all children’s lives begin—crying, laughing, and running around. However, the year 1942 brought the greatest obstacle any person could face. This year, 3 years after the beginning of World War II marked the first year my Grandma spent in a sub-basement 5 feet wide by 10 feet long by 3 and a half feet high with 6 family members. Early in her life, my Grandma faced an obstacle which she would face for the next 2 years, an obstacle which seems insurmountable. Yet, she survived. Not only did she survive, she uses that obstacle, the adocaosty as motivation to make her life as good as it can possibly be, to, as she says “not let Hitler win”. My grandma encountered an obstacle, made it through to the other side, and now uses that obstacle as motivation to make her life good.Can life be difficult? Many patronized kids have heard from their loving parents “life is not fair.” Sadly, that is true. We all encounter obstacles on our journey of life, whether they are deserved or not is utterly irrelevant. One can sit and gripe, stall and complain about these obstacles, but then our journey will stall too. Rather we must use these obstacles to our advantage, as opportunities to change ourselves for the better. If that hitman was truly alive, he would every day think back to the obstacle which changed him. My grandma is reminded every day that she won, not Hitler, and she therefore continues to strive to make her life better. Difficulties are not impossibilities, and when used to our advantage, they can enrich our lives in a way nothing else can.

#45An obstacle or disadvantage sometimes can be turned into good and at times can be bad. you Just have to look at the information that you have and just go with it.For example this theme is expessing on how the actor should perform his act. The producer told him to use the difficulty, think about it.My point of view is that the actor in this theme doesn’t have enough self-esteem or confidence in what he does so thefore. he is confused on what he should do.Therefore, we understand through this theme obstacles or disadvantages can sometimes be good yet sometimes be bad. The example was about the Actor and it Shows us that even if life is hard and we experince problems that Shouldn’t stop anyone from accomplishing their goal in life. There is a positive Side in life.

#46As my favorite saying goes it’s not how many times we fall it’s how we rise after the fall defines who we are. I believe that the obstacles are the good chance to test ourselves and learn, therefore become stronger for the next hurdles. The example of Ben Carson, world renowned neurosurgeon will demonstrate this statement. Ben Carson who is a world renowned neurosurgeon had a terrible temper that he couldn’t control. His uncontrollable aspect nearly killed his friend while he’s a teenager who wanted to become a doctor. Through his prayers and his efforts he did not show any of signs of bad temper later on. Through this claced experience, he got to learn how to deal with problems and he didn’t repeat it. It’s the learning that we can acquire from being disadvantaged. Another example is Bob Hope, the greatest entertainer who put his poverty into humers that made people laugh.

76

#47Many people only see the negative side of an obstacle. Obstacles and failures, however, can in fact be valuable and educational experiences. Through literature and history we see how one can turn his or her obstacle into something good in order to achieve stable success.In The Scarlet Letter, Nathaniel Hawthorne clearly demonstrates how an obstacle can be turned into something good. At the beginning of the novel, Hester Prynne is publicly humiliated and isolated for an adulterous act she committed. She takes her punishment as an opportunity to redeem and elevate herself. She begins to care for the poor and provide for them food, clothing, and shelter. By the novel’s end, Hester becomes the most giving and selfless woman in the community. Instead of giving up on herself, Hester integrated her sin into her life to become a better person.Hawthorne was conveying the message that when an obstacle is put before us, we should not give up hope; we should use the obstacle as an opportunity to better ourselves.Not only do we the educational value of an obstacle in literature, but in history as well. After the United States became independent in 1776, much debate ensued over the writing of the constitution. Finally in 1777, the first American Constitution was completed, called the Articles of Confederation. The new Constitution, however, had many weaknesses. Because the Articles failed to give the National Government enough power a disorganized government resulted. The National Government could not control interstate trade nor make laws regarding taxes, states had final say in governmental decisions, The National Governments lack of authority created a weak alliance among the states.Apparently something had to be done. At the Constitutional Convention, delegates from the 13 states. rewrote the Constitution that fixed the problems of the Article of Confederation, The delegates learned from the Articles that in order to maintain stability, the National Government needed sufficient power. The new Constitution that was written was a success and it continues to be the basis of American government.An obstacle or disadvantage can evidently be turned into something good. Hester Prynne utilized her obstacle to better herself. The fathers of our nation turned learned from their obstacle to create a better Constitution that has made our country’s government stable and powerful. We, too, can overcome our obstacles in order to succeed.

#48Throughout history, horrible tragedies have occured. Since we cannot go back in time and change the events that have taken place, we must make the best of the given circumstances. After tragedies such as the Holocaust and September 11th are examples of how something bad was turned into something good.The Holocaust took place during World War II throughout Europe. Hitler organized the genocide of the Jewish people. Six million Jews were brutally murdered in the concentration camps. Nothing could replace the lives that were taken by the Nazis. The surviving Jews and Jews around the world realized a need for a Jewish homeland. The United Nations recognized this need. In 1948 a home for the Jews was established in the land of Israel. Many historians say that without the Holocaust, the United Nations would have never voted for the Jewish Homeland.September 11th will forever stand as a day of remembrance. No one will forever forget the largest terrorist attack on United States soil. In New York alone, 3000 people were killed. In the twin towers. Brave fireman and policemen ran into the towers to save as many lives as they could. The towers came tumbling down along with thousands inside. After this unforgetable day, the country banded together to fight the war on terrorism. Patriotism was on a new height. People started to appreciate the risks that the New York Police and fire departments take every day.After both the Holocaust and September 11th, people banded together to bring some good into the world.

77

#49Some negative situations in life can be turned into positive ones, but not all of them. For example, it would be extremely difficult to turn a sickness or a death into a positive situation.However, if the situation is simpler than those above, all that really counts is perseverance. If you try something & it doesn’t work out, just keep trying until it does come out right. For example, there is an old saying that says, “If you fall off your horse, get right back on.” The rider would turn the negative experience of falling into a positive one, by getting back onto the horse & fixing the error.Determination also helps to change situations. For instance, if you’re losing during a sporting event, you must keep yourself all the more determined to catch up to the leader, & eventually overtake them & win. That way, you can turn the negative feeling of falling behind into the positive feeling of victory.Not all negative situations can be turned around like that, but many can. The two main factors in making something a positive experience seem to be perseverance & determination. If you try your hardest, everything should work out just fine.

#50At some point during their lives, everyone is bound to be faced with an obstacle. In every situation, one can take this obstacle and turn it into something good.The key to turning an obstacle into something good is one’s attitude. With an optimistic attitude, anyone can turn their obstacle into something positive. However, there are those people who will choose to look at their obstacle in a negative light and with a negative attitude, one is bound to stumble over the obstacle.So in reality, any obstacle can be turned into something good. Whether or not this will happen is actually dependant on you.

#51Any obstacle or disadvantage can be turned into something good. Life is full of obstacles that differ greatly. In school, new material and projects might be hard to comprehend or finish, but getting through and learning help make students smarter and better organized. Harder obstacles help people mature. The older a person gets, the more they learn from their experiences. An example is moving to a new place for college. At the begining it seems to be very tough for someone to be able to balance a job and college at the same time. However, the longer the person can balance both, he or she can learn how to become proficient at both college and a job.An example from history is the civil rights movement in the United States. When Rosa Parks was mistreated for sitting in the “white” part of the bus, the African Americans used this as a unifying factor. Their peaceful boycott of the bus system and nonviolent demonstrations helped them earn equal rights.From my experience, I have had a friend who tried to kill herself. She survived and has grown since then. Her confidence in herself has increased so she seems like a normal girl. She is a lot happier and only thinks that her suicide attempt helped her realize how much she has to live for.

#52Any obstacle or disadvantage can be turned into something good. Any difficult situation may be used as a learning experience. My life has shown many example of ways to turn anything from small aggravations to life-threatening situations into a way to grow and learn, both from myself and the people around me.When I was fifteen, I was diagnosed with Fybromyalgia Syndrome. This is not life threatening, but is a constant interference. In my case, it includes Chronic Fatigue Syndrome and widespread pain. I was told by doctors that I should not participate in contact sports and needed to be extremely careful with my energy. This meant I had to

78

quit softball and Tae Kwon Do. Instead of dwelling on this loss, I began swimming. In no time, my knee pain was reduced and I began to enjoy this new activity. I also had to cut back on after school activities to save my sparse energy for schoolwork. This allowed me to concentrate on my grades and only two extracurriculars which are very important to me —singing and United Synagogue Youth. I have used these experiences, which could have broken my spirit, to grow and become dedicated to new and different things.A more serious example of growing from difficult experiences occured when I was thirteen. My father, my role model and the person I most look up to in the world, was diagnosed with cancer. He is the strongest person I know, and people constantly tell my family how lucky we are to have each other for support. My father has been on chemotherapy for nearly five years now and has spirit and my families spirit has never wavered. I worry about losing him; of course; but, because I know I might, I have used it as the opportunity to always do my best. I need to make sure, if the disease were to take him tomorrow, that my father would be proud of my accomplishments and could be confident in my future. I have been careful to make wise decisions and acheive great things while he is alive, and these choices have improved my life.It is important for people to remember that any situation, no matter how dark it seems, may be used to their advantage, and learned from. An experience is only as bad as your reaction to it. If you allow a difficulty to become a source of good in your life then you can overcome anything. It is important to remember that you can always make lemons into lemonade.

#53I believe that yes, any obstacle or disa disadvantage can be turned into something good. Even when things get in your way, there is usually a solution. Like the actor, you can use the things that get in your way to help you. The producer said to use the difficulty, and people can do that. An example would be my soccer team just lost a game and everybody is sad. We lost because the other team had better passing skills. Our coach isn’t sad, she gets the team together and we practice passing all week. We have a good chance of winning our next game. So because we practiced. We took our loss, figured out why we lost, and found a way to fix it. That’s how people can overcome any obstacle. By figuring out how to deal with their problems constructivly.

#54I truly feel and believe that any and every obstacle one may face throughout his life, can be turned into something, good. One’s life can be filled with many challenges and struggles. However, one should try to do all he could to take his struggles and view them in a positive light. By doing so, he will feel better about his present situation.When Grover Cleveland was elected President, there was much corruption in the government. He, himself, was an uneducated man, with little knowledge. However, this obstacle didn’t stop him from learning as much as he could to increase his knowledge, as well as overcome this struggle. Additionally, he had the responsible to rid and remove the corruption in the government, and therefore by using the education and knowledge he learned while President, did so successfully. President Cleveland did not accept the fact that he had little knowledge, but even when he was a middle-aged man, he worked and saught after as much knowledge as he could.Having attended elementary school, and presently attending high school, I have seen many students struggle with their studies. Students have trouble in math, science, reading, as well as many other subjects. However, the school doesn’t ignore this fact. Resource rooms are a part of the school. There a teacher is able to work one on one with a student. The material is taught in a slower and more exciting and interesting manner. Most students won’t simply accept the fact that they have more difficulty learning, but most students will make sure they are placed in a program which will enable them to learn as much as they could in a way that is beneficial for them.One must look at life with a positive attitude. In doing so, he is able to take any situation that life presents him with and view it in a positive, not negative, manner. One can turn to his elders for advice on how to overcome a particuler struggle. Viewing a situation in a positive manner, and seeing the light at the end of the tunnel, will

79

enable one to feel better about his present struggle. Additionally, his positivity can cause those around him, who may have an identical struggle or negative situation, to have positive attitudes and see the good in all the bad.

#55I can agree that any obstacle or disadvantage could be turned into something good, all that’s basically needed is an optimistic attitude towards it.For instance, I can bring up an experience of mine which would help support my opinion. Not too long ago, a flight of mine was delayed due too a heavy snowstorm. Even though it was vital that I reached my destination, I didn’t let it this unfortunate event ruin my mood. Rather, I sat down with a smile on my face and patientily waited for the next flight out. The following day I arrive at my destination, and informed that the meeting which I needed to attend was postponed until the next evening. From there I realized that there isn’t really a point for getting frustrated about everything and to just look at life in the most positive way possible.Therefore I can agree with the statment given by Lawrence Eisenberg that an obstacle or disadvantage could turn into something good. All that is required is just to make the best of it.

#56Most obstacles in life can be turned into something positive, though it may simply be more difficult in certain situations. Whether something is viewed as being positive or negative can be unique to each individual based on how they choose to respond. However, many situations in life can be a challenge when trying to see the positive, for example: a death of someone close.Everyone in the world is different and has a unique personality. Some may actually be more upbeat and positive while for others it may be a real challenge. Personalities can be defined by one’s experiences in life but, ultimately, it is the decision of the individual alone as to how they will respond. Some are more naturally inclined to see the positive while others grab a hold of whatever negative aspect there may be.Positive and negative situations are not defined by themselves but by whoever may be viewing them. Regardless of whether a person sees a situation as positive or negative, doesn’t change anything about the situation. It is however the individual chooses to see it. Some situations may be more dire than others but there is always something positive to be found. This is only found if someone is looking for it, though. People see what they choose to, often. Situations are only as bad as people make them out to be. Likewise, for how good a situation may be.

80

Appendix N: Modified Body of Work Essay Grouping Record—Pinpointing Round 3For each essay in the set, indicate the group into which you think the student exhibiting the skills and abilities demonstrated by that essay should be placed. The goal of this process is to set one cut score to separate essays into two groups—either remedial (R) or entry level (E).Beside each essay ID number listed below, place either an R for remedial or E for entry level to show into which group you are placing the essay.

41. ___________________ 49. _______________________

42. ___________________ 50. _______________________

43. ___________________ 51. _______________________

44. ___________________ 52. _______________________

45 ____________________ 53. _______________________

46. ___________________ 54. _______________________

47. ___________________ 55. _______________________

48. ___________________ 56. _______________________ .

81

Appendix O: Final Evaluation Form for the WritePlacer Standard SettingThe purpose of this evaluation form is to obtain your feedback about the training you received on WritePlacer and the Modified Body of Work standard setting method. Your feedback will provide a basis for evaluating the training, methods, and materials in the standard setting process and informing future standard setting meetings.Please complete the information below. Do not put your name or identification number on the form, as we want your feedback to be anonymous.

Gender: _________Male_________Female

Race/Ethnicity:

_____Hispanic _____Asian _____African American _____White

_____Other (Please specify:___________________________)

Years of Experience as a Faculty Member:

_____1–5 years _____ 6–10 years _____11–15 years

_____16–20 years _____20–25 years _____25+ years

Affiliation:

_____ 2-year college _____ 4-year collegePlease read the following statements carefully. Place an X under one category (strongly agree, agree, disagree, or strongly disagree) to indicate the degree to which you agree with each statement.

Strongly Agree Agree Disagree

Strongly Disagree

1. I understand the purpose of this workshop.

2. The facilitator explained things clearly.

3. I feel comfortable with the task of evaluating each essay.

4. I feel comfortable with the task of assigning each essay to a category.

5. I used the performance level descriptors to guide my thinking throughout the process.

6. I am comfortable with the recommended cut score.

7. I would be comfortable defending this process to my peers.

Please add any comments you feel necessary to the back of this form.

82

Appendix P: Dictionary of TermsAuthoritative Body: The group of individuals who will make policy decisions about the activities to take place in the standard setting study, review the recommended cut score along with additional information, and have the final authority in deciding the final cut score to be adopted for use. The Authoritative Body does not participate in the standard setting study.

Cut Score: The identified score at which anyone scoring at or above is considered to be a member of the higher group, while anyone scoring below the score is considered to be a member of the lower group.

Distribution: The number of subject-matter experts recommending each probability of a correct response from 0 to 100%.

Facilitator: A person with specific skills who is outside the process and does not have an immediate stake in the outcome of the standard setting. This person should be knowledgeable about the standard setting process and possess the ability to manage the group of SMEs so that training is effective and the discussion is on topic and not dominated by any one person or group of people.

Impact: The consequences or ramifications in terms of predicted numbers of examinees that will be assigned to each group if a specific cut score is used to separate examinees into groups or categories.

Just Minimally Competent Examinee: An examinee with sufficient knowledge and skills to score at the cut score and qualify for membership in the upper level, but just barely.

Performance Level Descriptors: A set of definitions that detail the knowledge or skill that members of each group or category should demonstrate. Each group or category should have distinct characteristics listed to enable the SMEs to easily distinguish one from the other. Performance level descriptors should be written in positive terms and be measurable, avoiding ambiguous terms.

Performance Standards: The set of knowledge or skills required to show proficiency at a given level, i.e., the knowledge and skills needed to be successful in the entry-level course at college.

Scoring Rubric: A set of guidelines used for scoring test questions that have a range of score points that the student may earn by demonstrating various levels of proficiency. The scoring guide will provide a description or list of characteristics for a response that would earn each score point. For the purposes of this document, the essay has a scoring guide (see Appendix E) that describes the characteristics indicative of a score of 1 through a score of 8.

Standard Error of Judgment (SEJ): If a different panel of the same size and constituency were to be convened and trained in exactly the same manner, the resultant recommendation would most likely be slightly different. In fact, should many panels be convened, there would be a distribution of recommendations with a mean and a standard deviation. The SEJ is an estimate of the standard deviation of a large number of replications of panel recommendations. The SEJ is frequently used to adjust the one (and generally only one) panel recommendation. The Authoritative Body may want to drop (or raise) the recommendation by one or two SEJs. For example, dropping the panel’s recommendation by one SEJ may be interpreted as indicating that it is unlikely that a panel of all possible judges would have set the standard below the final accepted standard.

Standard Error of Measurement (SEM): All tests are fallible, and examinees are not likely to earn exactly the same scores if they were to take a different version of the test, or tested the next day assuming no additional training/learning occurred overnight. Should an examinee retest many times (assuming no learning or fatigue), there would be a distribution of scores for the examinee. The mean of this hypothetical distribution of scores for an examinee is called the examinee’s true score—what the examinee would get if there were no error in

83

the measurement process. The standard error of measurement is an estimate of the standard deviation of this hypothetical distribution of scores. If the Authoritative Body believes the panel’s recommendation represents the best standard, they may still want to adjust the standard downward (or upward). An examinee whose true score is exactly at the recommended standard will fail 50 percent of the time because there is error in the measurement process. The Authoritative Body may decide that failing a qualified examinee is a worse error than passing an unqualified examinee. In this case, the Authoritative Body might lower the standard by one or two SEMs such that examinees at or slightly above the panel’s recommendation are not likely to fail due to errors of measurement. Obviously, the probability of examinees slightly below the panel’s recommendation also increases, so lowering the risk of failing a qualified examinee increases the risk of passing an unqualified examinee.

Standard Setting Process: A set of methodologies that may be used to establish a cut score to separate examinees into adjacent groups or categories. Typically the process consists of one or multiple standard setting studies to recommend a cut score, and an Authoritative Body that meets to make critical decisions before the studies and decides the final cut scores using the results of the studies and other information.

Standard Setting Study: One part of a more comprehensive standard setting process. The standard setting study collects recommendations of the cut score placement from a panel of subject-matter experts through an objective process supplemented with discussion and, at times, empirical data.

Subject-Matter Expert (SME): Faculty members with subject-matter expertise in the area for which a cut score will be set. SMEs are also expected to have knowledge of the performance and skills required for an examinee to succeed at the level in question. Every effort should be made to acquire a representative group of SMEs for the standard setting study.