10 Things to Ask Your Local Psychometrician

about the Keystone Exams

Dr. Stefan BiancanielloOnHand Schools, Inc.Quality Classroom ConsortiumOctober 3, 2012

First Some Background First Some Background InformationInformation

OnHand Schools Works OnHand Schools Works with Common with Common AssessmentsAssessments

• Started in Florida—Hard to Measure Subjects

• Next came the Keystone Exams and inquiries from Penn Hills and Elizabeth Forward

• Others we are/expect to be working with this year :– Albert Gallatin

– Carlynton

– Highlands

– Clairton

– Sto-Rox

– Keystone Consortium

• 2012-13 - Algebra I, Literature, Biology

• 2015-16 – English Composition

• 2016-17 – Civics & Government, Geometry

• 2017-18 – U.S. History

• 2018-19 – Algebra II

• 2019-20 – Chemistry

• 2020-21 – World History

The Timeline of the The Timeline of the Keystone ExamsKeystone Exams

• …to develop Common Benchmark Assessments – similar in content/rigor to the Keystones in Algebra I, Biology, & Literature

• ….to develop a pool of items to be used for valid and reliable final exams

• ….to develop a pool of items for valid and reliable baseline and mid-course exams

• The results will be used ……

The purpose of the The purpose of the Albert Gallatin Albert Gallatin

assessments is…assessments is…

Now For the 10 Now For the 10 Questions for Your Questions for Your Psychometrician!Psychometrician!

#1#1What the Heck is a Psychometrician?What the Heck is a Psychometrician?

A psychometrician is someone who practices the science of educational and psychological measurement, i.e., testing. Psychometricians measure the validity, reliability, and fairness of an exam program, and are an integral part in the process of creating valid and reliable tests.

Psychometricians are often skilled in statistics, and psychology.

Psychometrician

#2 #2 What is the Difference between What is the Difference between Multiple Choice and Constructed Multiple Choice and Constructed

Response Items?Response Items?

Why are we writing items?

A. to populate a state item bank

B. to build common assessments

C. to work with other teachers

D. to gain experience writing

The Stem

Distractor

Distractor

CORRECT ANSWER

Distractor

Constructed Response ItemsConstructed Response Items•Constructed response items require students to provide a written response. These questions typically ask students to describe, explain, critique, or evaluate the scenario provided in the stimulus

•Constructed response items have multiple correct responses. However, there may be specific content that is required in the response to receive full credit.•Constructed response items can assess one or more benchmarks and range from low to high complexity.

•Constructed response items are scored using a rubric

Module 1 Module 2

Assessment Anchors Covered

Operations & Linear Equations & Inequalities

Linear Functions & Data Organizations

# of Eligible Content 18 15

# of Multiple Choice 18 18

# of Constructed Response 3 3

Multiple Choice Points 18 18

Constructed Response Points 12 12

Algebra Keystone – Algebra Keystone – from other PDE from other PDE

documents documents

60%

40%

Module 1 Module 2

Assessment Anchors Covered

Cells and Cell Processes

Continuity and Unity of Life

# of Eligible Content 16 22

# of Multiple Choice 24 24

# of Constructed Response 3 3

Multiple Choice Points 24 24

Constructed Response Points 9 9

Biology Keystone – Biology Keystone – from other PDE from other PDE

documents documents

73%

27%

Module 1 Module 2

Assessment Anchors Covered Fiction Literature Nonfiction Literature

# of Eligible Content 25 31

# of Multiple Choice 17 17

# of Constructed Response 3 3

Multiple Choice Points 17 17

Constructed Response Points 9 9

Literature Keystone – Literature Keystone – from other PDE from other PDE

documents documents

65%

35%

#3#3How Do You Train Teachers for This?How Do You Train Teachers for This?

We Teach them A New We Teach them A New Language! Try This Language! Try This

ExerciseExercise• Each of you received a stapled

piece of paper when you registered this morning.

• Please open the paper now. It contains a Multiple Choice Question written in “Franzipanic” —take just two minutes and figure out the correct answer!

The Moral of this TestThe Moral of this Test• With poorly written items, you can

get them correct without knowing the intended content.

• Reliability issue?

• Validity issue?

#4#4How Hard are the Keystone Exams?How Hard are the Keystone Exams?

Field Test Results—Field Test Results—Half or More Failed!Half or More Failed!

• Of the 47,000 students who took the Biology exam, 64.1% were NOT proficient: Basic (32.7%) Below Basic (31.4%)

• Of the 95,000 students tested in Algebra I, 61.0% were NOT proficient: Basic (41.2%) Below Basic (19.8%)

• Of the 42,600 tested in Literature, 49.7% were NOT proficient: Basic (31.4%) Below Basic (18.3%)

A recently released A recently released problemproblem

Tim’s scores for the first 5 times he played a video game are listed below:

4526 4599 4672 4745 4818

Tim’s scores follow a pattern. Which expression can be used to determine his score after he played the video game n times?

a.73n+4453

b.73 (n+4453)

c.4453n +73

d.4526n

Keystone Exam Keystone Exam DifficultyDifficulty• The Keystone Exam Test Questions

will almost all be at the Level 2 or 3 from Webb’s Depth of Knowledge

• Which leads us to the next question…

#5#5What Is Webb’s Depth of KnowledgeWhat Is Webb’s Depth of Knowledge

________________________________ ________________________________

Webb’s Depth of Webb’s Depth of KnowledgeKnowledge

• Norman Webb developed a process and criteria for systematically analyzing the alignment between standards and assessments.

• Webb’s Depth of Knowledge Model can be used to analyze the cognitive complexity required for the student to master a standard or complete an assessment task.

Webb’s Depth of Webb’s Depth of KnowledgeKnowledge

• Cognitive complexity refers to the cognitive demand associated with a test item.

• The Depth of Knowledge level of the item is determined by the complexity of the mental processing that the student must use to answer the item.

Level 1 - Recall - Recall of a fact, information, or procedure

Level 2 - Basic Application - of Skill/Concept - Use of information, conceptual knowledge, procedures, two or more steps, etc.

Level 3 - Strategic Thinking - Requires reasoning, developing a plan or sequence of steps; has some complexity; more than one possible answer; generally takes less than 10 minutes to do.

Level 4 - Extended Thinking - Requires an investigation; time to think and process multiple conditions of the problem or task; and more than 10 minutes to do non-routine manipulations.

Webb’s four levels of Webb’s four levels of complexitycomplexity

Cognitive Complexity/ Cognitive Complexity/ VerbsVerbsLow Cognitive Complexity—Level 1• Remember • Recall• Memorize

• Recognize• Translate• Rephrase

• Describe• Explain• Repeat

Moderate Cognitive Complexity—Level 2• Apply• Execute• Solve

• Connect• Classify• Break

Down

• Distinguish• Compare• Contrast

High Cognitive Complexity—Levels 3 & 4• Integrate• Extend• Combine

• Design• Create• Judge

• Perform• Value• Assess

Same Verb—Three Same Verb—Three Different DOK LevelsDifferent DOK Levels

DOK 1- Describe three characteristics of metamorphic rocks. (Requires simple recall)

DOK 2- Describe the difference between metamorphic and igneous rocks. (Requires cognitive processing to determine the differences in the two rock types)

DOK 3- Describe a model that you might use to represent the relationships that exist within the rock cycle. (Requires deep understanding of rock cycle and a determination of how best to represent it)

Describe

#6 #6 Does Depth of Knowledge Does Depth of Knowledge

Mean Difficulty?Mean Difficulty?______________________________________________________________

DOK is NOT about DOK is NOT about difficultydifficulty

DifficultyDifficulty is a reference to how many students answer an item correctly•How many of you know the definition of exaggerate?

DOK Low = Recall

If all or most of you know the definition , this item is an easy one.•How many of you know the definition of illeist?

DOK Low = Recall

If most of you do not know the definition, this item is a difficult one.

Complexity vs. Complexity vs. DifficultyDifficulty• The Depth of Knowledge levels are

based on the complexity of the mental processes the student must use to find the correct answer, not the difficulty of the item!

#7 How Does Webb’s DOK Relate to How Does Webb’s DOK Relate to

Bloom’s TaxonomyBloom’s Taxonomy__________________________________________________________________

COGNITIVE LEVEL COMPARISON MATRIX: BLOOM AND WEBB

Action WordsBLOOM WEBB

Rememberdefine, identify, name, select, state, order (involves a one-step process)

1.0define, identify, name, select, state, order (involves a one-step process)

Understand

convert, estimate, explain, express, factor, generalize, give example, identify, indicate, locate, picture graphically (involves a 2-step process)

2.0

apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use (involves a two-step process)

Apply

apply, choose, compute, employ, interpret, graph, modify, operate, plot, practice, solve, use, (involves a three-or-more step process)

Analyze

compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test

3.0

compare, contrast, correlate, differentiate, discriminate, examine, infer, maximize, minimize, prioritize, subdivide, test

Evaluate

arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data 4.0

arrange, collect, construct, design, develop, formulate, organize, set up, prepare, plan, propose, create, experiment and record data

Createappraise, assess, defend, estimate, evaluate, judge, predict, rate, validate, verify

#8#8Do You Have Item Writing Rules?Do You Have Item Writing Rules?

Item Writing RulesItem Writing Rules• Yes, we offer training in constructing

quality Multiple Choice and Constructed Response Items

• A one-page summary explains the rules that we used last week while training in Florida

• Shula Nedley and I will be available to discuss this in more detail at one of the breakout sessions today

#9#9Can’t I Just Buy Assessment Questions?Can’t I Just Buy Assessment Questions?

Yes You Can Buy or Yes You Can Buy or Find Assessment Find Assessment Questions But… Questions But…

• Concern over Alignment

• Concern over Depth of Knowledge

• Penn Hills Will Discuss Why They Are Creating Their Own

#10#10Is it Worth All This Effort?Is it Worth All This Effort?

Yes, It’s Worth the Yes, It’s Worth the EffortEffort• The Bar is being raised

• The Common Core changes the game

• The Keystone Exams change the game

• The world our students will enter is FAR MORE COMPLEX

• We need to raise the Level of our game

A Tool To Ease the A Tool To Ease the PainPain

• Assessment BuilderAssessment Builder

A demonstration of this tool will take A demonstration of this tool will take place during the breakout sessions place during the breakout sessions later today.later today.

You are trapped in a room with 2 doors, one of which leads to freedom and the other to certain death. You do not know which is which.

Each door is guarded by a man, one of whom always lies, while the other always tells the truth. You do not know which is which.

You can ask but one question of either man to gain your freedom. Which question should you ask?

A.How do I get out of this room and stay alive?

B.Which one of the two door is the correct one?

C.Who is the man who never tells the truth & always lies?

D.Which door would the other man say leads to death?

Test Your Depth of Test Your Depth of KnowledgeKnowledge