Overview of Standard Setting

Leslie WilsonAssistant State Superintendent

Accountability and Assessment

August 26, 2008

Purpose of Presentation

To present the standard setting process

To provide background information to the Board in preparation for setting standards for– Modified High School Assessments– Alternative Maryland School

Assessments

Setting Performance Standards

ContentStandards

Voluntary State

Curriculum

The Tests

Expertise: Student

Knowledge/SubjectMatter

Setting Setting PerformanPerforman

ce ce StandardsStandards

Cut Scores that match students to

their appropriate

Performance Categories

The Threshold Student

A borderline or minimally qualified student– In terms of performance

The least qualified of the Proficient or Advanced students

Mod-HSA Test Performance Standards

Basic

Recommended Standards

threshold students

Proficient Advanced

Standard Setting Work Flow

Content Groups

Psychometric Council

Review & Articulation Committee

State Superintendent

State Board of Education

Set Standards

Step 1: Content Groups

Participants all have appropriate content and grade-level expertise

Include representatives from local Central Offices as well as school level staff

Participate in separate content and grade level discussions

Responsibilities of Content Group

Review Assessment

Discuss what barely proficient and advanced students know and can do

Propose cut scores and discuss

Make recommendations to Review and Articulation group

Content Group Process: Round 1

Break into small groups to encourage independent decision-making

Read, discuss ordered-item booklets

Individuals vote (privately) for a cut score

Small group discussion of votes

The Process: Round 2

Individuals vote (privately) for a cut score

Small group discussion of votes

Examine, discuss impact data

The Process: Round 3

Small groups converge

Individuals vote (privately) for a cut score

Discussion

Individuals vote (privately) for a cut score

The median cut score is the committee’s recommendation.

Bookmark Standard-Setting Procedure

Test items students must know and be able to do to be considered Proficient

Test items students must

know and be able to do to be considered Advanced

A

P

Ordered Item

Booklet

1

23

45

67

8

9

10

1112

13

14

15

Standard-Setting Process

Individual Analysis

Items most

students get

correct

Items fewer

students get

correct

Content every student should know

Review test; vote

Round 1

Discuss data and distribution

Median

Standard-Setting Process

Round 2

View impact data and discuss

VoteMedian

Standard-Setting Process

Small groups join; vote

Round 3

Median cut score is committee’s recommendation

Median

Step 2: Psychometric Council

Nationally-recognized experts

Trained in measurement and psychometrics

Independent

Familiar with Maryland tests

Similar to Technical Advisory Panels in other states

Psychometric Council Responsibilities

Reviews work of all groups

Ensures quality controls were followed

Ensures standard-setting process technically sound

Does not make judgments on whether standard is “good” or “bad”

Forwards comments and recommendations to Review & Articulation Committee

Step 3: Review and Articulation

Superintendents

Assistant Superintendents

Local Accountability Coordinators

Advocates for students with special needs (special education and ELL)

Parents

Local Boards of Education

Teachers Union representatives

Business Community members

MSDE

Review & Articulation Committee

Responsibilities include:

– Reviewing work of all groups and Psychometric Council

– Ensuring rigor is equivalent among grades

– Reviewing articulation among grades

– Forwarding recommendations to the State Superintendent

Step 4: State Superintendent of Schools

Reviews work of all groups, Psychometric Council, and Review & Articulation Committee

Makes recommendation to the State Board of Education

Step 5: State Board Approval

Reviews proposed standards

Approves standards for implementation

Next Steps

1. Review and approve proposed standards to be used to determine proficient and advanced performance on the modified HSAs. The scores will be used to determine AYP for Maryland high schools next year.

2. Review and approve proposed standards for the Alt-MSA. Science will be reported this fall for the first time.

Overview of Standard Setting

Questions and Discussion