smarter balanced cut scores

8/10/2019 Smarter Balanced Cut Scores

1/28

Presentation to the Delaware State Board of Education

Brian M. TouchetteOffice of Assessment

December 18, 2014


2/28

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common CoreState Standards. They are designed to let teachers and parents know whetherstudents are on track to be college- and career-ready by the time theygraduate.

Because the new standards set higher expectations for students--and the new

tests are designed to assess student performance against these higherexpectations--our definition of grade level performance is higher than it used tobe.

As a result, it means that fewer students will meet grade level standards ,especially for the first few years. Results should improve as students haveadditional years of instruction aligned to the new standards and become betterequipped to meet the challenges they present.

This does not mean that our students are doing worse than they did last year.Rather, the scores represent a new baseline that provides a more accurateindicator for educators, students, and parents as they work to meet the rigorousdemands of college and career readiness.


3/28

Achievement Level Timeline (2014)

States Approved Achievement Level SettingPlan April 30

Online PanelOctober 6-18

In-Person Panel (Dallas)October 13-19 Cross-Grade Review Committee (Dallas)October 20 Technical Advisory Committee ReviewOctober 30 States vote to approve Achievement Level

RecommendationsNovember 14 State Adoption of Achievement LevelsFall/Winter


4/28







5/28

Online Panel

Crowd sourcing thousands of participantsRecommendations broken down by subgroup (teachers,higher education, etc.)

The process involved:

Online orientation

Review of test questions

Recommended a cut score for the threshold of PerformanceLevel 3

142 Delawareans participated (exceeded SBACs target)


6/28

Three Cut Scores Defines FourPerformance Levels

5


7/28

Reviewing Items in the Ordered Item Booklet(OIB) for Bookmarking

6

1. What do you know about astudent who respondssuccessfully to this item; that is,what skills must a student have in

order to know the correct answer?

2. What makes this itemmore difficult thanpreceding items?

This activity helps

you acquire theknowledge to makecontent-based cutscorerecommendations


8/28

Selection of Placement of Bookmark

A threshold Level 3 student would have about a 50% chanceof success on the item just before the bookmark

A threshold Level 3 student would have less than a 50%likelihood of success on the items from the bookmark on

7


9/28







10/28

In-Person Panel

K-12 Educators, Higher Education, parents, and communityadvocates brought together

Recommended cut scores for all Performance Levels

500 participants in grade/subject panels

Nominated by states

25 Delawareans participated

Used Bookmark procedure

Three Rounds in the process


11/28

Reviewing Items in the Ordered Item Booklet(OIB) for Bookmarking

10

1. What do you know about astudent who respondssuccessfully to this item; that is,what skills must a student have in

order to know the correct answer?

2. What makes this itemmore difficult thanpreceding items?

This activity helps

you acquire theknowledge to makecontent-based cutscorerecommendations


12/28

Three Rounds to Determine Three CutScores

11


13/28

In-Person Panel Agenda

Day 1Morning: Study Common Core State Standards and

Achievement Level Descriptors Afternoon: Study the Ordered Item Booklet

Day 2Morning: Bookmark training and Round 1

Make bookmark recommendations individually

Afternoon: Round 2Discuss differences in Round 1 bookmark placementsat each tableMake Round 2 bookmarks individually

12


14/28

In-Person Panel Agenda

Day 3Morning: Round 3

Discuss Round 2 bookmark placements for the entireroom (all panelists at all tables)

View supporting data based on Round 2 bookmarks(e.g., impact data, online panel results)Place Round 3 bookmarks individuallyReview final recommendations

Evaluate the process

13


15/28







16/28

Cross-Grade Review Committee

Also referred to as Vertical Articulation60 panelists selected from In-Person Panel

One Delawarean participated

Reviewed Online Panel and In-Person Panel results

Recommend changes to smooth the data:

Motion

Second

Discussion

Vote: 2/3 majority required

Aligned cut scores across grades

Eliminated scaled score reversals


17/28

Vertical Articulation

16

This is anexample to

illustrate Vertical Articulation. It isNOT actual data

from AchievementLevel Setting.


18/28







19/28

Setting Achievement Levels(Continued)

States Chiefs ReviewReviewed final recommendations from cross-grade ReviewCommittee

Approval by State Chiefs

State Board Review to Seek Approval


20/28


21/28

English Language Arts/Literacy:Threshold Scale Scores

State Agreement on November 14, 2014

2300

2350

2400

2450

2500

2550

2600

2650

2700

27502800

3 4 5 6 7 8 9 10 11

T h r e s h o

l d S c a

l e S c o r e

Grade

Level 4

Level 3

Level 2


22/28

ELA/Literacy: Estimated Percentage ofStudents at each Achievement Level

35% 37% 33% 30% 34% 28% 28%

27% 22%23% 29%

28%31% 31%

20% 23% 29%30% 30% 32% 30%

18% 18% 15%11% 8% 9% 11%

0%

10%

20%

30%

40%

50%60%

70%

80%

90%

100%

3 4 5 6 7 8 11

E s t

. %

i n E a c h L e v e

l

Grade

Level 4

Level 3

Level 2

Level 1

Estimates based on Spring 2014 Smarter Balanced field test conductedin 21 states and USVI.


23/28

Proposed Scale Score Cut Points

ELA/Literacy Scale Score Thresholds (scale 2000 3000)Grade Level 1-to-2 Level 2-to-3 Level 3-to-4

3 2367 2432 2490

4 2416 2473 2533

5 2442 2502 2582

6 2457 2531 2618

7 2479 2552 2649

8 2487 2567 266811 2493 2583 2682

For Board Action in January 2015

22


24/28

Mathematics: Threshold Scores

State Agreement on November 14, 2014

23002350

2400

2450

2500

2550

2600

2650

2700

27502800

3 4 5 6 7 8 9 10 11

T h r e s h o

l d S c a

l e S c o r e

Grade

Level 4

Level 3

Level 2


25/28

Mathematics: Estimated Percentage ofStudents at each Achievement Level

32% 27%35% 35% 36% 38% 40%

29% 36%32% 32% 31% 30% 27%

27% 24% 18% 19% 20% 19% 22%

12% 13% 15% 14% 13% 13% 11%

0%10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

3 4 5 6 7 8 11

E s t

. %

a t E a c h L e v e

l

Grade

Level 4

Level 3

Level 2

Level 1

Estimates based on Spring 2014 Smarter Balanced field test conductedin 21 states and USVI.


26/28

Proposed Scale Score Cut Points

Math Scale Score Thresholds (scale 2000 3000)Grade Level 1-to-2 Level 2-to-3 Level 3-to-4

3 2381 2436 2501

4 2411 2485 2549

5 2455 2528 2579

6 2473 2552 2610

7 2484 2567 2635

8 2504 2586 265311 2543 2628 2718

For Board Action in January 2015

25


27/28

Preparing for New Test Scores

Smarter Balanced assessments measure the full range of the Common CoreState Standards. They are designed to let teachers and parents know whetherstudents are on track to be college- and career-ready by the time theygraduate.

Because the new standards set higher expectations for students--and the newtests are designed to assess student performance against these higher

expectations--our definition of grade level performance is higher than it used tobe.

As a result, its means that fewer students will meet grade level standards ,especially for the first few years. Results should improve as students haveadditional years of instruction aligned to the new standards and become betterequipped to meet the challenges they present.

This does not mean that our students are doing worse than they did last year.Rather, the scores represent a new baseline that provides a more accurateindicator for educators, students, and parents as they work to meet the rigorousdemands of college and career readiness.


28/28

Learn More About Achievement LevelSetting

Video explanation of processhttps://www.youtube.com/watch?v=bW_yGf4BB1E

Interpretation and Use of Scoreshttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdf

Easy to follow flow charthttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-

Achievement-Level-Setting-Overview-.pdf

Q&A Documenthttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdf

www.DelExcels.org

27
https://www.youtube.com/watch?v=bW_yGf4BB1Ehttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.delexcels.org/http://www.delexcels.org/http://www.delexcels.org/http://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Smarter-Balanced-Achievement-Levels-QA.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/09/Smarter-Balanced-Achievement-Level-Setting-Overview-.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttp://www.smarterbalanced.org/wordpress/wp-content/uploads/2014/11/Interpretation-and-Use-of-Scores.pdfhttps://www.youtube.com/watch?v=bW_yGf4BB1E

smarter balanced cut scores

Documents