CREATING QUALITY ASSESSMENTS

Guidelines

James Greene

Harlan Independent School District

July, 2012

HISD EXPECTATIONS FOR 2012-13

Continue work on realigning curriculum and updating curriculum documents

Continue and refine use of assessment for learning strategies with emphasis on student self-assessment and ownership of learning

Create quality assessments using research to inform your design

Use CIITS to create and store assessments and to track and analyze assessment data

Design ConcernsDesign ConcernsClassroom AssessmentClassroom Assessment

Purpose Audience Content Evidence of learning Process Meaning

Key No. 1 STANDARDS-BASEDWhat do we want students to know and be able to do at the end?

Key No. 3 VALIDITYDoes the assessment measure what it’s supposed to measure?

Key No. 2 FAIRDoes the assessment allow students to show what they know?

Key No. 4 MULTIPLE SOURCESIs there sufficient evidence to judge whether a student has mastered the standard?

KEYS

TO

QU

ALIT

Y

AS

SES

SM

EN

T

Objectivity (no one gets unfair advantage) Absence of bias (cultural, gender, socio-

economic) Opportunity for each student to demonstrate

what he/she knows

FAIRNESS MEANS…

ISSUES RELATED TO VALIDITY

Do the items form a representative sample?

Do the items represent a fresh experience or are they recycled?

Do the items rely on skills that students may lack which are not part of the assessment?

DO ITEMS FORM A REPRESENTATIVE SAMPLE?

Items are not aligned with the SFLTs Items represent teacher convenience

rather than systematic design Items focus on exceptions rather than

key topics Items are not appropriately distributed

among topics Items do not reflect the amount of time

devoted to them during instruction

“The essential characteristic of well-designed assessments is that the processes used to collect and interpret data are consistent with the purpose of the assessment. That match of purpose and process is achieved through thoughtful planning…”

—National Science Education Standards

TEST BLUEPRINTS

•Select targets to be assessed•Select methods to be used•Create items•Reasonable number of items per target (based on type of item)•Group items by targets•Assure appropriate balance among items and among targets •Consider if redo/retake will be permitted and what form it will take

“Any item may function at the rote learning level if the concept involved has been taught in the same way that it is used in the testing situation. In view of this, we must do our utmost to increase the likelihood that an item requires reasoning and understanding.”

—Harry D. Berg

Do the items represent a fresh experience?

SKILLS ISSUESHeavy reliance on a skill in which student is

not proficient (e.g., reading level, drawing)Sentence structure and vocabularyExtraneous knowledge Format (complexity; unfamiliar to student)Student’s ‘testwiseness’Student test anxiety

PRECISE UNAMBIGUOUS LANGUAGE

“It is not sufficient to write a question that can be understood. To paraphrase the advice of Gen. Douglas MacArthur to the Corps of Cadets: Write questions that cannot be misunderstood, not merely questions that can be understood.”

—John P. Sevenair

" The content of assessments should match challenging subject matter standards and be connected to contexts of application. This means assessing learning based on observations, oral questioning, significant tasks, projects, demonstrations, collections of student work, and students' self-evaluations, and it means that teachers must engage in systematic analysis of the available evidence."

—Lorrie A. Shepard

Knowledge

Reasoning

Product

Skills

Constructed response

Performance tasks

Observation

Selectedresponse

Personal Communication

ASSESSMENT METHODSASSESSMENT METHODS

True FalseMultiple True False (MTF)Multiple Choice & Alternate ChoiceMatchingKeyed Lists

SELECTED SELECTED RESPONSE RESPONSE

1.1. Grass is green.Grass is green.

2.2. California is the largest state.California is the largest state.

3.3. The U.S. national debt is too high. The U.S. national debt is too high.

TRUE/FALSE

Benefits• Ease of use and

coverage of material

• Assessing student misconceptions

• Useful in formative assessment

Pitfalls• Trivia, trickery, and

truisms• Overemphasis on

recall • 50% chance of

guessing correctly

TRUE/FALSE

WRITING GOOD T/F ITEMS

Focus statements on ONE fact or concept. Create statements of relatively equal length. Word statements positively using precise,

unambiguous language. Use NEW language (don’t lift statements from

textbook or from class). Don’t use absolutes (e.g., always, never, all) or

conditional verbs (“can, may”). Avoid vague qualifiers like “some,” “usually,”

“typically,” “several,” “many,” “large,” “small”). If it is necessary to test an opinion, attribute it to

a source.

WRITING GOOD T/F ITEMS

Lessen the power of guessing by: having slightly more “false” items than “true”

(when students don’t know, they tend to opt for “true”)

mixing up the sequence of responses so that they do not form patterns

“A major distinction between the [true-false] item and items in multiple-choice or two-choice formats, is that the [true-false] item contains no criterion for answering the item. The criterion is outside of the item, rooted in the characteristics and experiences of each individual examinee. Each examinee must ask the question, true or false with respect to what? It follows that each … true-false item must be unequivocally true or unequivocally false.”

https://www.msu.edu/dept/soweb/writitem.html#m

citem

Use T/F to test CONCEPTUAL UNDERSTANDING

1. Inflation leads to a lower cost of living.

2. Physical change results in formation of a new substance.

3. Profit is the money a store takes in when goods are sold.

LESSEN POWER OF GUESSING

1. Higher prices are one of the results of deflation.

2. The rhythmical succession of single tones producing a distinct musical phrase or idea is referred to as harmony.

•Students can correct entire statement (focuses on the complete proposition) OR

•They can correct an underlined word or phrase (allows teacher to focus questions on particular concepts or facts)

•Correction diminishes the power of guessing

JUSTIFIED TRUE FALSE

Consider the equation a=12+b (when b≠0)

T or F

Why I (we) think so

a is always greater than b

a and b could be equal

etc.

from Mathematics Formative Assessment by Page Keeley and Cheryl RoseTobey, ©2011 Corwin Press

MULTIPLE TRUE FALSE (MTF)MULTIPLE TRUE FALSE (MTF) Consists of a context or situation and several

statements which the student is to evaluate in terms of that information.

Is an alternative way to deal with “All of the above” multiple choice items

EXAMPLE OF MTF

Which of these are characteristics of a square?

Check (√) all that are:Sides are equal in lengthSides join to form acute anglesHas four sidesIs a type of polygonIs a three-dimensional figure

ANOTHER MTF EXAMPLE

Which of these are effects of inflation? Check (√) all that are: Shortage of capital to invest in new businesses Drop in the purchasing power of money Decline in cost of goods and services People on fixed incomes can buy less than before Interest rates favor borrowers rather than lenders

A THIRD MTF EXAMPLE

Place a checkmark (√) by each example of chemical change:

1. Butter melting on warm bread2. Tarnish on a silver charm3. Black edges on overdone toast4. Creases in pants from being ironed5. Rust on a saw blade

MATCHINGMATCHING

Identify each item on the left by matching it with an item from the right.

1. Ship that brought Pilgrims to America

2. Indian who helped the Pilgrims

3. Name of the Pilgrim settlement

4. Important food crop of the Pilgrims

5. Key Pilgrim leader6. Date Pilgrims arrived in

America

A. William BradfordB. SquantoC. Henry HudsonD. MayflowerE. JamestownF. 1620G. PlymouthH. RiceI. Corn

CREATING QUALITY MATCHING ITEMS

Target items should be of the same class Directions indicate the relationship

between target items and choices Have at least one extra choice or allow choices to be used more than

once Sequence choices to make matching

easier (e.g., ABC order) Reasonable number of items

1. “It’s better to have it and not need it, than to need it and not have it.”

2. “He is smart as an owl.”3. “George has been put out to

pasture.”4. “Henry was a lion in battle.”5. “This vase will make a good

centralpiece.”6. “If I don’t make an A, I will just die.”

What type of figure speech is represented by each quote? Match the correct term from the right to the quote on the left. There is one extra term.

A. chiasmusB. euphemismC. hyperboleD. malapropismE. metaphorF. simileG. synedoche

BEYOND RECOGNITION/RECALL

KEYED LIST

Sentence rolesA. SubjectB. Direct objectC. Indirect objectD. Predicate

nominative

1. Is that your car?2. She told him to come in.3. Joe is a great student.4. Which book do you need?5. To whom did you give the

package?6. Why didn’t Mr. Howe

return my call?

Use the key to identify the role the underlined word plays in the sentence.

Key

ANOTHER KEYED LIST

Functions of moneyA. Unit of accountB. Store of valueC. Medium of

exchange

Isabella deposits $25.00 in her savings account at the bank

The supermarket sign that says “Strawberries $2.99 per pint”

Mrs. Wallace writes a check to her beautician for her perm.

The school board receives a bid from Acme Supply to provide ten cases of copy paper for $375.

Dr. Greene finds a hundred dollar bill on the street.

Classify each example using the key:Key

ALTERNATE CHOICE ALTERNATE CHOICE Two choices Useful in formative assessment Can highlight particular misconceptions or

errors students are likely to make Can be expanded to conventional MC by using

“both of these” and “neither of these” as options

1. If you were making shirts, which of these would be a capital resource?

A. the person who designed the shirtsB. the sewing machine used in sewing

them

2. People who make things for others to use are called:A. consumersB. producers

MULTIPLE CHOICEMULTIPLE CHOICE

1. The______ lasted from 1861 to 1865.

A. Revolution

B. Civil War

C. Spanish-American War

2. It marked the turning point of the war.

A. Ft. Sumter B. Antietam C. Gettysburg

3. The Civil War

A. Was the bloodiest war in American history

B. Was fought mainly in the South

C. Could have been avoided

EFFECTIVE STEMS

Focus the question through the stem, not the choices. (Writing the stem as a question helps do this.)

Avoid non-functional language. Use simple, direct language as much as

possible. If stem contains a negative, highlight it in

some manner: e.g., Which of these is NOT true of paintings by Monet?

Focus the question through the stem:

What does a seismologist study?

A) Earthquakes and earth’s internal structure

B) Human social organization and institutions

C) Physical features of earth and human response to them

(preferred form)

Seismology

A) Study of human social organization and institutions

B) Study of earthquakes and internal structure of the earth

C) Study of physical features of earth and human response to them

Make all choices plausible & attractiveUse common misconceptions and errors as

basis for distractorsKeep distractors consistent in content, form,

and lengthUse similar degree of specificity Use “new” language rather than phrases

from the text or class3-4 good choices are enough

EFFECTIVE CHOICES

AVOID COMPLEX MULTIPLE CHOICE ITEMS:

Composers of the Baroque period include:1. J.S. Bach2. Frederic Chopin3. Georg Philipp Telemanna) 1 & 2 b) 1 & 3 c) 2 & 3 d) 1, 2, 3

•Harder for students to process

•Don’t discriminate better than simple choice questions

AVOID “ALL OF THE ABOVE” AS OPTION

• If student finds one exception, then he knows it is NOT “all of the above”

• If he finds at least two choices that are correct, then he knows it IS “all of the above”

ALTERNATIVES1.Stem…… A) ------- B) ------ C) both of these D) neither of these2. Use MTF

“NONE OF THE ABOVE” AS CHOICE

• Useful if the correct response is an absolute• If it is the answer, you can’t be certain that the

student actually knows the information• Better than a weak distractor• Should be the correct answer about half the time

Which of these adds value to Choco-Bars as a product?A) the elegant blue and gold wrappers in which they are soldB) the short travel time they have from factory to storeC) the free gift that comes with a purchase of a five pound boxD) none of the above

SUGGESTIONS FOR MANAGING OPTIONS

• Logical sequencing (e.g., ABC, magnitude)• Random patterns in arrangement of options• Relatively equal use of each option on every

test• Use vertical formatting for longer options• Have students use capital letters rather than

small case if they have to write the letters

1. Which of these artists is known primarily for his work as a sculptor?A) Leonardo da VinciB) Auguste RodinC) George BellowsD) Rembrandt van Rijn

 2. Which of these is an example of

Baroque sculpture?A) Donatello’s DavidB) Rodin’s The Age of BronzeC) Bernini’s Apollo and DaphneD) Miró’s Woman and Bird

1. “Pharaohs were to Egypt as Emperors were to Rome” is an example of an - - -:

A) simile B) chiasmus C) analogy D) metaphor

2. The poetic meter with a foot consisting of an unaccented and an accented syllable is known as:

A) lyricB) epicC) iambicD) trochaic

3. Who composed Rhapsody in Blue?A) Judy GarlandB) Frank SinatraC) Cole PorterD) George Gershwin

GIVEAWAYS Grammatical clues (e.g., a, an) Words from the stem repeated in the

correct choice Correct choice is much longer or

more complicated in wording Absolutes: always, never Choices form pairs Interlocking items (first item answers

second)

• Options not selected by anyone should be replaced as they are not working

• An option chosen by almost everyone over the other options including the right answer may signal an instructional problem

• Items that everyone gets right are not functioning as discriminators

• Items missed by virtually everyone may signal an assessment problem or an instructional problem

Item Analysis

MULTIPLE CHOICE AND HIGHER LEVEL THINKING

Identify the verb you would like to use

Change it to a noun and pair it with a convergent verb

You can also use “which” with the noun

Explain: explanation Select the best

explanation for …. Describe: description

Identify the most accurate description of…

Solve: solution Choose the most

viable solution for….

“Bury the Verb” Mike Dickinson

• Common factor• Example or

illustration• Problem/solution• Conclusion from

data (e.g. trends)• Explanation of a

sequence• Image-based

items

FRAMING QUESTIONS

• Identification/recognition

• Application• Comparison/

contrast• Premise/

consequence; cause/effect

• Case or scenario• Interpretation

CLOSE DECISION MULTIPLE CHOICE (MYRON DUECK)

Student may choose a second answer and provide an explanation “Lifeline”

Reveals student’s thinkingReveals flaws in your questions

CONSTRUCTED RESPONSE TYPES

Short response• Fill in the blank• Graphic organizers and visuals• Relationships (e.g., form pairs, which

doesn’t belong, word splashes)• Exit Cards• Prompts

• Extended response (formerly called “open response”)

• Essay

GRAPHIC ORGANIZERS AND VISUALS

Simple webs Concept maps Venn diagrams Graphs and data

displays Labeling features

in a picture or diagram

Promote “chunking”

Show relationships

between ideas

Promote categorization

USING GRAPHS

Number apples eaten

1 2 3 4 5 6

Kate

Ashley

Ethan

Steve

Ethan ate apples. Show this number on the graph.

1.Who ate more apples: the girls or the boys?2.How many more apples did Ashley eat than Steve?3.Write a number sentence than shows how many apples Kate and Ashley ate.

FILL IN THE BLANK

a. ________________ is the current President of the United States.

b. A good source of protein is __________________.c. The steps in solving a quadratic equation are

1) ________________ 2) __________________, 3) _________________, 4) __________________, and 5) _________________.

Item should have only one right answer Blank should be placed at or near the end Multiple blanks should be avoided

Writing Quality Fill in the Blank Items

1. The name of the current Governor of Kentucky is ________________________.

2. Based on the data in the graph, the month with the highest number of absences was __________________.

SHORT RESPONSE PROMPTS

Focus question on one point. Use clear, unambiguous language. Assess knowledge and reasoning

Compare the role of producers and consumers in the economy.

Sam shaded part of this circle. (Diagram shown)What fraction of the circle is shaded? Explain your answer.

K-PREP sample

1. (Based on reading passage)Tommy learned that watching birds build their nests is a good way to understand them. How does watching the birds build their nests help Tommy understand birds better?

Sample K-PREP short response promptsSample K-PREP short response prompts

2. A bird called the arctic tern flies about 18,600 miles each year. What is this distance rounded to the nearest ten thousand? Show your work or explain how you found your answer.

3. Draw a closed figure with 5 line segments. Your figure must contain 2 right angles. Label the right angles with the correct angle marks.

DIAGNOSTIC QUESTIONS

“The crucial feature of…diagnostic questions is based on a fundamental asymmetry of teaching; in general, is better to assume that students do not know something when they do than it is to assume they do know something when they don’t. What makes a question useful as a diagnostic question, therefore, is that it must be very unlikely that the student gets the correct answer for the wrong reason.”

—Dylan Wiliam

DIAGNOSTIC QUESTIONS (DYLAN WILIAM, PAGE KEELEY)

Ask “Why is…?” rather than “What is…?”“Why is 17 prime and 15 not?” rather than “What

is a prime number?” or “Is 15 a prime number?”

“Why is a square a rectangle?” rather than “Is a square a rectangle?”

“What makes the veto an example of checks and balances?” rather than “What are “checks and balances?”

Take a straight recall question and transform it by phrasing it to require explanation or justification.

DIAGNOSTIC QUESTIONS (DYLAN WILIAM)

Ask for reactions to statements:“All squares are rectangles.”“Light travels from the eye to the object.”Use MTF/MC with multiple correct responses:“In which of these diagrams is one-fourth of the

area shaded?”

“Just as there can be no set formulas for producing a good story or a good painting, so there can be no set of rules that will guarantee production of good test items Principles can be established and suggestions offered, but it is the item writer’s judgment in the application…that determines whether good items or mediocre ones will be produced.”

—R.L.Ebel

CAVEAT