1

New EnglandCommon Assessment Program

(NECAP)Setting Performance Standards

2

Purpose

• Provide data to establish the following cut scores for Math, Grades 3-8:– Proficient/Proficient with Distinction– Partially Proficient/Proficient– Substantially Below Proficient/Partially

Proficient

3

What is Standard Setting?

• Set of activities that result in the determination of threshold or cut scores on an assessment

• We are trying to answer the question:– How much is enough?

4

What is Standard Setting

• Data collection phase• Policy/Decision making phase

5

Many Standard Setting Methods

• Angoff• Body of Work• Bookmark

6

Choice of Method is Based on Many Factors

• Prior usage/history• Recommendation/requirement by some

policy making authority• Type of assessment

7

Choice of Method is Based on Many Factors

• Weighing all these factors, it was determined that the Bookmark Method would be used for NECAP Math

8

The Bookmark Procedure

• Well established procedure that has been successfully used on many assessments

• Has produced defensible results• Appropriate for assessments that consist

primarily of multiple-choice items but also include some constructed-response items

9

Standard Setting vs. Standards Validation

• Standard setting– Generally three rounds:

• Round 1: individual ratings• Round 2: table group discussion of Round 1 results,

followed by second round ratings• Round 3: whole room discussion of Round 2

results, followed by final ratings

10

Standard Setting vs. Standards Validation

• Standards validation– Starting cut points are provided to panelists at the

beginning of the process– Two rounds:

• Round 1: table group discussion of starting cut points, followed by first round ratings

• Round 2: whole group discussion of Round 1 results, followed by second round ratings

• The process you will be following will be more of a standards validation process

11

Details for Standards Validation using the Bookmark Procedure

12

What is the bookmark procedure?

• A standard setting procedure that uses a book of items (ordered from easiest to hardest)

• Panelists place bookmarks in that book of items

13

What is the bookmark procedure?

14

What is the bookmark procedure?

• For purposes of NECAP standard setting, you will be provided with starting cut points and will either validate those cuts, or recommend modifications to them

15

Starting Cut Points

• Starting cuts were determined using teacher judgment data:– At the time of testing, teachers were asked to categorize

each student into an achievement level category, based on classroom performance (not test performance)

– Using these categorizations as well as the students’ test scores, Measured Progress staff calculated starting cut points

16

Starting Cut Points

• Analyses of teacher judgment data indicate:– Good participation rates– Teachers appear to have been conscientious in

completing the task• In other words, available evidence supports

the validity of the starting cuts; therefore…

17

Starting Cut Points

• You are free to recommend changing the starting cuts, but we expect that changes will be relatively minor

• You should have justification for any changes you recommend– rationale for changes should be based on the

Achievement Level Descriptions and the test items

18

Starting Cut Points

• Note that, in some cases, the lowest starting cut appears very early in the Ordered Item Booklet

• What this means is that the test was difficult for these students:– for several of the math tests, borderline Partially

Proficient students (as identified by the teacher judgment data) have less than a two-thirds chance of answering even the easiest items correctly

19

Starting Cut Points

• What you need to know is that the ordered item cut point for a given cut does not equal the raw score a student must obtain to be categorized into the higher achievement level

• For example, the Substantially Below Proficient/ Partially Proficient starting cut for grade 7 math is between ordered items 3 and 4; however, a student must obtain at least 20 points on the test in order to be classified into the Partially Proficient level

20

How to Place a Bookmark

• A few concepts you will need to know:– The starting cut points– The achievement level descriptions– ‘Borderline’ students– What knowledge, skills, and abilities (KSAs)

are needed to answer each question

21

How to Place a Bookmark

• Start at the beginning of the ordered item book• Evaluate whether at least 2 out of 3 students demonstrating

skills at the ‘borderline’ of Partially Proficient would correctly answer item 1

• Moving through the book, make this evaluation of each item• The bookmark should go where you no longer think 2 out of

3 Partially Proficient ‘borderline’ students would correctly answer the question.

• You may decide that the starting cuts are in the right place, or you might recommend moving them.

22

How to Place a BookmarkItem Number

Would at least 2 out of 3 students who demonstrate skills at the Partially Proficient 'borderline' correctly answer this question?

1 Yes

2 Yes

3 Yes

4 Yes

5 Yes

6 Yes

7 Yes

8 Yes

9 No

10 No

11 No

12 No

13 No

14 No

15 No

… No

23

How to Place a Bookmark

• In the example, the bookmark would go between items 8 and 9

• However, it won’t be that easy; there will be gray areas

• You will have the opportunity to discuss your bookmark placements and change them if desired

• Place one bookmark for each cut score

24

How to Place a Bookmark

• To place your bookmarks you will need to be familiar with the achievement level descriptions and the assessment items

25

How to Place a Bookmark

• Don’t worry, we have procedures, materials and staff to assist you in this process.

26

Any questions about the Bookmark Procedure?

27

What Next?

• After this session, you will break into grade-level groups, where you will:– take the assessment to familiarize yourself with

the test items– discuss the Achievement Level Descriptions

and develop definitions of “borderline” Partially Proficient, Proficient, and Proficient with Distinction students

28

What Next?

• You will:– complete the Item Map, which is a document

that will help you with the bookmark placement process

– discuss the starting cut points in table groups and do the first round of bookmark placement

– discuss the first round bookmark placements as a whole room then do the second round of bookmark placement

29

What Next?

• After the second round of bookmark placement, you will have an opportunity to provide feedback about the Achievement Level Descriptions

• As the final step, we will ask you to complete an evaluation of the standard setting process

30

Good Luck!