Improving your classroom assessments

Dr. Patty LeBlanc

Southeastern University

Agenda

IntroductionsObjectives of the SeminarPurpose of Educational AssessmentsEducator and Educational StandardsBloomTest Blueprints Item Writing Commandments

Item Critique Activity Item AnalysisExample from grad studentsAuthentic AssessmentRubricsSummaryAction PlanQuestions?

Objective of the Seminar

To help teachers design valid assessments of instruction that inform instruction

What is the purpose of testing?

Participant Ideas

To encourage students and improve teaching

To assess the extent to which students have met the intended learning outcomes, and to look for and evaluate unintended outcomes

To recognize achievements and diagnose learning difficulties so that students learn to build on their strengths and overcome or cope with their weaknesses

To refine instruction and learning experiences in order to improve both individual and class learning

To help students set meaningful and realistic learning goals and assume responsibility for their own learning.

To communicate meaningful information to students, parents, and school authorities about student learning

To give realistic and helpful feedback about achievement, capabilities, behavior, attitudes, and dispositions

To put teachers, students, and parents/guardians in touch with one another about progress over time

To provide guidance for educational and vocational choices

To report learning achievement to school authorities such as school boards and governments

Florida Educator Accomplished Practices in Assessment

The effective educator consistently:Analyzes and applies data from multiple assessments and measures to diagnose students’ learning needs, informs instruction based on those needs, and drives the learning process;Designs and aligns formative and summative assessments that match learning objectives and lead to mastery;Uses a variety of assessment tools to monitor student progress, achievement and learning gains;Modifies assessments and testing conditions to accommodate learning styles and varying levels of knowledge;Shares the importance and outcomes of student assessment data with the student and the student’s parent/caregiver(s); andApplies technology to organize and integrate assessment information.

Educational Standards

Florida Sunshine State StandardsCommon Core StandardsOther

“Backward” Design

The standards dictate the lesson objectives…

The objectives dictate the instructionThe assessment must align with the

objectives, standards, and instruction in order to be valid.

Blessed Assurance…

A fantastic way to ensure that your test items align with your objectives: A Test Blueprint!

Content Validity Think matrix of instruction and assessment:

Objective Item 1

Item 2

Item 3

Item 4

Item 5

1 X X X

2 X

3

4 X X

Bloom’s Taxonomy of the Cognitive Domain

Another blueprint…

Item # Standard/Objective Knowledge/Comp Above Knowledge/Comp

Comment

    

     

    

     

    

     

    

     

    

     

    

     

    

     

    

     

    

     

    

     

    

     

Sung to the tune of the Farmer in the Dell

The teacher follows the standards…The teacher creates the objectives…The teacher writes the test…The teacher teaches the lesson…The students learn a lot…The parents beam with pride…

The Commandments of Test Item WritingItem Writing Item Writing

CommandmeCommandmentsnts

Comunicación y Gerencia

By W.James PophamBy W.James Popham

Click to add TextClick to add Text

General Item Writing Items

Thou shalt not

provide opaque directions to students regarding how to respond to your assessment instruments.

Thou shalt not

employ ambiguous or wordy statements in your assessment items.

The term hypothesis, as used in research, as defined as:a. A conception or proposition formed by speculation or deduction or by abstraction and

generalization from facts, explaining or relating an observed set of facts, given

probability by experimental evidence or by factual or conceptual analysis but not

conclusively established or accepted.

b. A statement of an order or relation of phenomena that so far as is known is invariable

under the given conditions, formulated on the basis of conclusive evidence or tests

and universally accepted, that has been tested and proven to conform to facts.

*c. A proposition tentatively assumed in order to draw out its logical or empirical

consequences and so test its accord with facts that are known or may be determined,

of such a nature as to be either proved or disproved by comparison with observed or know facts.

Better…

The term hypothesis, as used in research, is defined as:

a. An assertion explaining an observed set of facts that has not been conclusively

established.

b. A universally accepted assertion explaining an observed set of facts.

*c. A tentative assertion that is either proved or disproved by comparison with an observed set of facts.

Thou shalt not

Provide unintentional clues as to the correct response.

What meter is used to measure resistance?a. Voltmeterb. Ohmmeterc. Ammeterd. Oscilloscope

Thou shalt not

employ complex or give-away syntax in your assessment items.

In item response theory, the one-parameter model assumes that each item …….a. discriminates equally well.b. students perform equally well.c. students score the same across items.d. guessing affects all items the same.

Thou shalt not

use vocabulary that is more advanced than required.

As the level of fertility approaches its nadir, what is the most likely ramification for the citizenry of a developing nation?

a decrease in the labor force participation rate of women a dispersing effect on population concentration a downward trend in the youth dependency ratio a broader base in the population pyramid an increased infant mortality rate

Use plausible distractors

Which of the following artists is known for painting the ceiling of the Sistine Chapel?

a. Warhol

b. Flintstone

*c. Michelangelo

d. Santa Claus

Thou shalt

Thou Shalt

Avoid using "always" and "never" in the stem or the distractors.

Thou shalt

Rarely us negative statements…

Which of the following is NOT a symptom of osteoporosis? decreased bone density frequent bone fractures raised body temperature lower back pain

Binary Choice Items [True/False]

One of several types of Selected Response Items

Thou shalt not

Phrase items so that a superficial analysis by the student suggests a wrong answer.

Which of the following artists is known for painting the ceiling of the Sistine Chapel?

a. Warhol

b. Flintstone

*c. Michelangelo

d. Santa Claus

Thou shalt

NEVER use double negatives.

True or False: There is no advantage to not using specific determiners in true-false items.

Thou shalt

Have an approximately equal number of items representing the two categories being tested.

Keep item length similar for both categories being tested.

Thou Shalt

Stick with absolute truths or falsehoods.

True or False: Nearsightedness is hereditary in origin.

Thou Shalt Not

Use extreme qualifiers.

All sessions of Congress are called by the President. (F)

The Supreme Court frequently rules on the constitutionality of law. (T)

An objective test is generally easier to score than an essay test. (T)

Multiple Choice Items

Thou shalt

Write the stem so that it consists of a self-contained question or problem.

California:a. Contains the tallest mountain in the United States

b. Has an eagle on its state flag.

c. Is the second largest state in terms of area.

*d. Was the location of the Gold Rush of 1849.

Better…

What is the main reason so many people moved to California in 1849?

a. California land was fertile, plentiful, and inexpensive.

*b. Gold was discovered in central California

c. The east was preparing for a civil war.

d. They wanted to establish religious settlements.

Thou shalt

Include only one concept in each statement and only one correct answer.

The United States should adopt a foreign policy based on:a. A strong army and control of the North American continent.

b. Achieving the best interest of all nations.

c. Isolation from international affairs.

*d. Naval supremacy and undisputed control of the world’s sea lanes.

Thou shalt

NEVER use ‘all of the above’ alternatives, only occasionally use ‘none of the above’ alternatives [to increase item difficulty].

Alternative Weakness

“All of the above” Distractor can be eliminated by noting that one of the other alternatives is incorrect

“None of the above” 

Answer measures the ability to recognize incorrect answers rather than correct answers [remember educational psychology 101?].Distractor does not appear plausible to some students.  

Well-designed MC items CAN measure HOTS

A common goal of the Salt March in India, the Boxer Rebellion in China, and the Zulu resistance in southern Africa was to:

A. overthrow totalitarian leaders

B. force upper classes to carry out land reform

programs

C. remove foreign powers

D. establish Communist parties to lead the countries

Numbers of Items is Important!

Number of 4-Alternative MC Items

2 5 10 15 20 25

Chance of Scoring 70% or Higher by Blind Guessing

• 1 out of 16• 1 out of 64• 1 out of 285• 1 out of 8,670• 1 out of 33,885• 1 out of 942,651

Pop Quiz

Can you identify the problem with this item?

Due to budget cutbacks, the university library now subscribes to fewer than _?_ periodicals. 25,000 20,000 15,000 10,000

Matching Items

Most effective when you need to measure the learner’s ability to identify the relationship or association between similar items. They work best when the course content has many parallel concepts, for example:Terms and DefinitionsObjects or Pictures and LabelsSymbols and Proper NamesCauses and EffectsScenarios and ResponsesPrinciples and Scenarios to which they apply

Thou shalt

Use homogeneous lists.

Hitler A. Year in which WWII began

Mulroney B. A Canadian Prime Minister

Tank C. A German dictator during WWII

1939 D. An armored vehicle used originally to break the trench

war stalement in WWI

Thou shalt

Randomly assign correct answers to alternative positions.

Logical order of responses 

Example

 

Numerical a. 1939b. 1940c. 1941d. 1942 

Alphabetical a. Changing a from .01 to .05.b. Decreasing the degrees of freedom.c. Increasing the spread of the exam scores.d. Reducing the size of the treatment effect.

Sequential a. Heating ice from -100°C to 0°C.b. Melting ice at 0°C.c. Heating water from 0°C to 100°C.d. Evaporating water at 100°C.e. Heating steam from 100°C to 200°C.

Thou shalt not

let the length of alternatives supply unintended clues.

Which of the following is the best indication of high morale in a supervisor’s unit?

a. The employees are rarely required to work overtime.

*b. The employees are willing to give first priority to attaining group objectives,

subordinating any personal desires they may have.

c. The supervisor enjoys staying late to plan the next day.

d. The unit gives expensive birthday presents to each other.

Thou shalt

Use relatively brief lists, placing the shorter words or phrases at the right.

Thou shalt

Employ more responses than premises.

Thou Shalt

Provide good directions!

Directions: Match the following.

____1. City dwellers A. Wild animals

____2. Hunter-gatherers B. Farm

____3. Pastoral nomads C. Apartments

D. Graze animals

Better…

Directions: On the line to the left of each definition, write the letter of the term in the right hand column that is defined. Use each term only once. There are more responses than items. Each correct response is worth 2 points.

_1. Live in areas of high population density. A. Pastoral nomads

__2. Move from place to place in search of wild animals. B. Ranchers

__3. Move from one place to another with grazing animals. C. Hunter-gatherers

__4. Till land for cash crops. D. City dwellers

E. Farmers

Thou shalt

Describe the basis for matching and the number of times responses may be used in the instructions, as well as the point values for each item.

Place all premises and responses for an item on a single page.

Tip: Be sure to decide ahead of time whether you will give partial credit.

Matching HOTS item

Directions: Number (1-8) the following events in the history of ancient Egypt in the order in which they occurred, using 1 for the earliest event. _____Egypt divided; ruled by Libyan kings, Nubian pharaohs,

Assyrians, and Persians _____Seizure of power by Hyksos kings _____Upper and Lower Egypt are united by Menes _____Alexander the Great conquers Egypt _____Reunification of Egypt under pharaoh Mentuhotop II _____Rise of feudal lords leads to anarchy _____Thutmose III expands empire to the Euphrates _____Many kings with short reigns; social and political chaos

Short Answer/Essay Items

Aka Constructed Response Items

Short Answer/Fill in the Blanks

Thou Shalt

Ask a direct question.

Undesirable:Alloys are ordinarily produced by…

Desirable:How are alloys ordinarily produced?

Thou Shalt

Put a blank at the end of a sentence rather than the beginning.

Undesirable:

____________ is the measure of central tendency that is most affected by extremely high or low scores.

Desirable: The measure of central tendency that is most affected by

extremely high or low scores is the ____________.

Thou Shalt

Omit only significant words.

Undesirable:

Every atom has a central ____________ called a nucleus.

Desirable:

Every atom has a central core called a(n) ____________.

Thou Shalt Not

Omit so many words that the meaning in unclear.

Undesirable:

The ____________ were to Egypt as the ____________ were to

Persia and as ____________ were to the early tribes of Israel.

Desirable:

The Pharaohs were to Egypt as the ____________ were to Persia and as ____________ were to the early tribes of Israel.

Thou Shalt Not

Use essay items to measure trivial ideas

Thou Shalt

Give very clear directions. Know the criteria for grading and use a rubric. Make expectations clear—have a good idea of what

the teacher expects in a good response Not provide options… Indicate point values and suggested time limit per item Take the test yourself and see how you do… Ask a colleague to take the test and critique it. Consider using several short essays vs 1 long one. Analyze student responses to improve the item in

future.

How do I decide???

Another Pop Quiz

Objective or Essay?

Rubrics for Essay Items

How can a classroom teacher improve his/her tests?

Before the test: Table of specifications/matrix Pilot testing, especially essay items and rubrics Peer review

During the test: Observation Timing Ask the students to rate the test/items

After the test: Item Analysis Go over the test with students Item and Test Revision Re-examine instruction and re-teach if necessary

That’s a lot of trouble!

Yes, but it is the right thing to do.It’s intellectually honest.It makes grades easier to defend.

Test Improvement Project

By Christina Salas and Amanda Williams

Authentic Assessment

Aka performance based assessment

Great example: Edutopia

Authentic Assessment vs. Traditional Assessment

Traditional --------------------------------------------- Authentic

Selecting a Response ------------------------------------ Performing a Task

Contrived --------------------------------------------------------------- Real-life

Recall/Recognition ------------------------------- Construction/Application

Teacher-structured ------------------------------------- Student-structured

Indirect Evidence -------------------------------------------- Direct Evidence

How to Measure Authentic Assessments

RubricsCooperativelyPanel of Experts

Steps in Rubric Development

1. Determine the learning outcomes.

2. Determine the criteria

3. Determine measurable criteria [4,3,2,1]

4. Develop a matrix

5. Evaluate overall point value relative to the task.

Grading cooperative work…

Participation in Class Discussions

Sit down with a seating chart and mark contributions to the discussion with a rating systemFor example, 5 being outstanding and 1

being poor, 0 being absent or silent

Grading cooperative work

Group gradeRecognize that there will always be

loafers, workers, and compulsive worker beesSee article on grading cooperative projects

Assessment should…

Encourage and improve student learning Be a blessing for students that embraces, grace, justice, and

encouragement Be an integral part of learning, closely related to overall learning

goals Foster excellence in all a students’ gifts on the basis of the

student’s potential as a bearer of God’s image Emphasize assessment FOR learning as well as assessment OF

learning Develop plans that enable teachers and students to interact

frequently about students’ progress as well as challenges, with opportunities to refine or revise

Use a variety of meaningful tasks, products, and performances, with descriptive feedback to students

Use accurate descriptors for all grades based on the achievement of learning outcomes

Communicate meaningful and clear information to students, parents, and authorities.

Other suggestions?

Questions?

Resources

Van Brummelen, H. 2009. Walking with God in the Classroom, 3rd edition. Purposeful Design Publicatons.

Brookhart, S. 2008. Feedback that Fits. Educational Leadership 65, no.4:54-59.

Stiggins, R. 1997. Student-centered classroom assessment. 2nd edition. Merrill.

More Resources

How to prepare effective essay questions http://www.uwgb.edu/oira/teachLearn/betterTests/betteressays.pdf

Steven J. Burton, et al, How to Prepare Better Multiple-Choice Test Items: Guidelines for University Faculty, (http://testing.byu.edu/info/handbooks/betteritems.pdf).

Still More Resources

A Short Guide to Writing Effective Test Questions

Rubric Maker Quiz Star

Thank You!!!

To download this presentation:

http://leblancspace.wikispaces.com/home

pbleblanc@seu.edu