where did your cut score come from?: a primer in standard setting

7/29/2019 Where Did Your Cut Score Come From?: A Primer in Standard Setting

http://slidepdf.com/reader/full/where-did-your-cut-score-come-from-a-primer-in-standard-setting 1/32

Click to edit Master subtitle style

March 27, 2012

Where Did Your Cut ScoreCome From?: A Primer in

Standard SettingDr. Susan GraciaDirector of AssessmentSimmons CollegeBoston, MA



March 27, 2012

Cut Score• A selected point on the score scale of

a test.•

The cut score separates students intovarious categories, such as a passingand failing.

•

The process of setting a cut score iscalled standard setting , which is amulti-stage, judgmental process.

22



March 27, 2012

Nomenclature• Achievement levels• Classification scores•

Criterion levels• Cut points• Cut scores• Cutscores• Cutting scores

• Cutoff scores• Mastery levels•

Passing scores• Performance levels• Performance standards• Standards (ETS, 2005)

33



March 27, 2012

Determine the Need for CutScores

• Why is there a need to set cutscores?

•

What benefits are expected from theuse of cut scores?• What decisions will be made on the

basis of the cut scores?• How are those decisions being made

now in the absence of cut scores?•

What reasons are there to believe 44



March 27, 2012

There is no “true” cut score• There’s no objective way to set cut

scores.•

Cut scores are constructed, notfound.• They are based on judgments about

people, student work, or test items.• Judges depend on subjective,

internalized norms about what

people can do. 55



March 27, 2012

Errors of Classification• Some people who deserve to pass

will fail.•

Some people who deserve to fail willpass.• Raising or lowering the cut score to

reduce one type of error will increasethe other type of error.• There’s no way to prove that a cut

score is correct. – “ ”

66



March 27, 2012

To Do for All Methods

1. Select judges.

2. Teach judges about cut scores.

3. Define “borderline” knowledge/skillsfor a particular task/assessment.

1. Borderline = Minimally competent

4. Train judges in the use of themethod.

5. Implement the cut score study.

6. Document the results. 77



March 27, 2012

Judges• Judges need to:

– Be qualified –

Know subject and population – Be representative and diverse – Be acceptable to stakeholders –

Be willing to follow procedures• Teach them about test purpose, cut

score purpose, consequences of passing and failing 88



March 27, 2012

Defining “Borderline”

1. Make sure judges understand whatthe assessment measures and howscores will be used.

2. Ask judges to describe in their ownwords a person whose knowledgeand skills would represent theborderline between acceptable andunacceptable levels of knowledge/skills the assessment

measures. 99



March 27, 2012

Examples of Borderlines• Worst reader who should get a

diploma / best reader who shouldnot

• Worst surgeon who should belicensed / best surgeon whoshould not

• Worst essay that deserves a score of 6 / best essay that deserves a 5

•

Highest bacteria count in safe water /lowest bacteria count in olluted1010



March 27, 2012

Judge Training• Overview of the standard setting

method: What is it? Why do it? Howis it done?

• Practice using the standard settingmethod.

•

Observe judges.• Correct errors.• Answer all questions.• Practice more until all judges are 1111



March 27, 2012

Selecting a Method• There’s no one best method.• Different methods will result in

different cut scores.• Some methods are better for certain

types of assessments and

assessment situations.

1212



March 27, 2012

3 Standard Setting MethodsMethods Based on

judgments about:Best

Modified Angoff Method

Test items Test that is only orprimarily multiplechoice

ContrastingGroups Method

People Small assessmentsituations wherefaculty havestrong knowledge

of students’abilities

Body of WorkMethod

Student work Performance tasks

1313



March 27, 2012

Modified Angoff Method• Judges examine each question on an

assessment and estimate theprobability that a borderline studentwould answer the question correctly.

– Or: Imagine a group of 100 borderlinestudents and estimate how many of

them would answer the questioncorrectly.• Probabilities will range from .00 to

1.00. 1414



March 27, 2012

Modified Angoff Method• Discuss probabilities. Aim for range

of 10-15 % points per question. Judges can change judgments basedon discussion.

• Sum each judge’s probabilities for allquestions to get a recommended cutscore for each judge.

• Average the judges’ recommendedcut scores to arrive at a average cutscore for minimum competency. 1515



March 27, 2012

Modified Angoff Example

1616

• Judge 1

Source: Livingston &



March 27, 2012

Modified Angoff Example Judge Recommended Cut Score

1 5.80

2 6.00

3 6.00

4 5.405 5.00

6 5.30

7 5.50

8 4.80

9 6.10

10 5.50

Average Cut Score forMinimum Competency

5.54

1717



March 27, 2012

Modified Angoff • Pros• Most researched

method• Stands up to

court challenges• Does not require

student data soit can be carriedout prior to testadministration

• Cons• A lot of dataentry

• Difficultcognitive task

• Does not workwell with heavilyopen-ended/performance-based tests

1818



March 27, 2012

Contrasting Groups Method• Faculty consider all they know about

a population of students.• Faculty predict individual students’

level of performance on anassessment (e.g., expectedpassing/failing, proficient/non-proficient) –without reference toscores.

• Obtain assessment scores.• Predictions are com ared to actual 1919



March 27, 2012

Contrasting Groups Example

2020



March 27, 2012

Another Example• Graph the scores of expected passing

and expected failing students in 2separate distributions.

• The point at which the 2 distributionsintersect is the cut score.

2121



March 27, 2012

Contrasting Groups• Pros• Uses real data,

not conjecture• Uses external

validation info inaddition to testscores

• Easy to explain

• Cons• Inconvenient toget judgments of people

• Relies on human judgmentwithout

examining actualtest performance

• Need scores

before cut is set2222



March 27, 2012

Body of Work Method• Pre-work:

– Performance level categories, as well asperformance level descriptors for eachperformance level, must be establishedand agreed upon.

– Scoring of a large sample of student

work must occur before standard-settingcan begin. – Select 40 to 50 intact samples of

student work to represent the range of student performance on an assessment.2323



March 27, 2012

Body of Work Method• Range-Finding phase:

– Identical sets of student work areprovided to each judge.

– Judges are asked to independentlycategorize the student work samplesbased on the performance level

descriptors, without any discussion. – This process reveals which work

samples (e.g., Graduation Portfolios)generate the most agreement and whichgenerate the most disagreement 2424



March 27, 2012

Examination of StudentWork Method

• Pinpointing phase: – Judges examine sets of work about

which they disagreed in the range-finding phase, along with additionalwork samples representing those samescore intervals.

–

Judges assign performance levels tothese work samples. – The minimum score for each

performance level is precisely"pinpointed" by determining the score 2525



March 27, 2012

Body of Work• Pros• Uses real data,

not conjecture• More intuitive;

panelists are notasked to imaginea hypothetical

minimallycompetentexaminee or to

estimate the

• Cons• Need scoresbefore cut is set

• Requires a lot of priorpreparation,volumes of

materials• Grueling work

2626



March 27, 2012

Setting the “Operational”Cut Score

• Only a legally authorizedentity (e.g., policy makers)

can authorize the use a cutscore.

•

Once authorized, a “study”cut score becomes an“operational” cut score.

2727



March 27, 2012

Complying with theStandards

• The Standards of the AmericanEducational Research Association,the American PsychologicalAssociation, and the National Councilon Measurement in Education specifythat the following be included in the

standard setting process: – Document how judges were selected,

their qualifications and training, theprocedures used, whether or not judges’ratings were independent, the level of 2828



March 27, 2012

Observe Cut Score Effects• Seek opinions on cut scores.• Find out what happened to people

who failed. – Is there evidence that any of them were

actually qualified?•

Is there evidence that any peoplewho passed are really unqualified?• What were the consequences of

misclassification errors? 2929



March 27, 2012

Comments? Questions?

3030



March 27, 2012

Discussion Questions• How could you use standard setting

in your setting?• Which standard setting approach

might you utilize? Why?• What challenges do you anticipate in

implementing this approach?• How might you address these

challenges?

3131



March 27 2012

References

3232

American Educational Research Association , American Psychologica l Association ,National Council on Measurement in Education. ( ).1111 Standards for Educational andPsychological Testing. Washington, DC: AmericanPsychological Association.

Cizek, G.J. ( ). An NCME instructional module on setting pass scores.1111 Educational Measurement: Issues and Practice , 11 ( ),1 11 - .11

Cizek, G.J., Bunch , M.B., Koons, H. (Winter ). Setting performance standards:1111 Contemporary methods. Educational Measurement: Issues and Practice , 11 - .11

Horn, C., Ramos, M., Blumer, I. & Madaus, G. (1111 ). Cut scores: Results may vary.National Board on Educational Testing and Public Policy Monographs,1 ( ).1 Chestnut Hill, MA: Boston College.

Livingston, S.A. & Zieky, M.J.. ( ).1111 Excerpts from Passing Scores: A Manual for Setting Standards of Performance on Education and Occupational Tests .Princeton,NJ: Educational Testing Service.

Measured measures: Technical considerations for developing a local assessment system . ( ). Augusta , ME: Maine Department of Education.1111 Pitoniak, M. ( ).1111 Considerations in Setting Performance Standards ( Cutscores).

Training session at 1111 National Council for Measurement in Educationconference, Montreal, Canada.

where did your cut score come from?: a primer in standard setting

Documents