Best Practices for Building Tests and Quizzes

League for Innovation

November 2004

7 question types

• multiple choice• true/false• essay/short answer• matching• ordering• fill in the blank• multiple answers

Part II-Instructional Design Considerations for Creating

Tests

Selected Response Questions

• Selected response questions require students to select from a predetermined list of potential answers.– multiple choice– true false– Matching– fill-in-the blank questions

Created Response Questions

• Ex. Essays measure the student’s ability to communicate effectively, not just their understanding of content.

• Easier to write but harder and (more subjective) to grade.

Match Question Type to Level of Assessment Desired

• Multiple choice and matching questions– offer the most flexibility in terms of content

covered– thinking skills that can be assessed

• True/false are usually limited to fact recall.

Match Instructional Objectives

• Use lesson plans or teacher notes to accurately reflect content that was covered in class

• Choose the most important objectives to assess and use these as the outline for your test

Cover Important Material

• Facts, definitions, comprehension, analysis, applications

• Trivial items result in trivial studying and learning.

Items Should be Independent

• Do not “give away” answers via information in other questions

• Independence maximizes breadth of coverage

Write Simply and Clearly

• Measure knowledge of material and concepts, not vocabulary

• Ambiguous questions create error, frustration, and compound biases related to language and disability

Clearly Specify What Type of Response is Sought

• How long or short an answer is sought? • Should they show their work? • Do you want description? comparisons?

application? evaluation? • Whose opinion do you want (book,

lecture, their own)?

Good Tests Take Time to Write

• Give yourself enough time to evaluate items after a day or two

• Revise, edit, and ask others to read before administering the test

Good Tests Have High Degrees of Reliability and Validity

• Reliability refers to the extent to which measurement is consistent

• Validity refers to the extent to which a test measures what it should.

Poor Example

Construct Questions Carefully

• The most important test-making process is selecting the wording of each question and answer.

• Question Stem and Distractors

Construction Tips

• Construct evaluation items with a single correct answer

• Use plausible distracters• Arrange options in a logical sequence• Include a minimum of 3 and a maximum of

5 options• Alternate the order of the correct

responses randomly among all options

Construction Tips

• Refrain from using the choices “all of the above” or “none of the above”

• Identify an initial difficulty level

Construction Tips

• Increasing item difficulty can be achieved by paying attention to the distracters (options)

• The more “homogeneous” the options and distracters, the higher the degree of difficulty

Higher Degree of Difficulty

The first James Bond movie was released in what year?

a) 1960

b) 1962

c) 1964

d) 1966

Mid-Level Difficulty

The first James Bond movie was released in what year?

a) 1960

b) 1962

c) 1964

d) 1970

Low-Level Difficulty

The first James Bond movie was released in what year?

a) 1958

b) 1962

c) 1970

d) 1975

Points of Concern for Different Formats

• Writing True-False Items:

• Choose words with precise, definite meanings

• Avoid tricks and trivia

• Avoid easy clues

• Follow guidelines for multiple choice

Writing Multiple Choice Questions

• Present the problem, including qualifying statement

• There should be only one correct answer

• Distracters should be plausible but clearly incorrect

• Avoid negative wording (especially double negatives)

Writing Multiple Choice Questions

• When item is controversial, indicate whose opinion is sought

• Avoid irrelevant cues to correct answer (length, grammar)

• Items should test one central idea or concept

Writing Multiple Choice Questions

• Watch out for patterns in alternatives (overuse of "C" as correct answer)

• Choose appropriate level of difficulty• Assure even coverage of material and

types of knowledge

Writing Matching Items

• Keep each matching set short (no more than 5 stimuli/responses)

• Use longer items as stimuli, shorter as responses

• Arrange responses in alphabetical or logical order

Writing Short Answer and Completion Items

• Clearly indicate type of answer you want

• Do not use more than two blanks per completion item

• Make a key before scoring

• Periodically re-score early tests to detect shifting criteria

Writing Essay Questions

• Limit questions to vitally important material

• Clearly define task, scope, and directions for a "good" answer

• Allow time for thought

• Use multiple medium-length essays rather than one long one

Writing Essay Questions

• Use questions that have a limited number of good answers

• Allow choice between alternatives (e.g., "answer 3 of the 4 questions")

Part III- Evaluating your Assessment or Assessing

Your Evaluation

Evaluating the Test

• Once scored, spend some time reviewing the test and observing patterns that may be present.

• Were there any questions that every student got wrong?

• If so, can you deduce whether it was due to poor test item construction or to instruction?

Evaluating the Test

• If poorly written, you may want to consider canceling it out and recalibrating the test.

• If students didn’t understand the concept, perhaps the content needs to be retaught.

A Checklist for Teachers

• Are your directions CLEAR and CONCISE?

• Are your objectives clear?• Is there a logical connection between

questions and answers?• Are you being specific in regard to the

area you are covering?• Are your answers simple and factual?

Higher Order Thinking

• What later events best affirm the ideas set forth in the Declaration of Independence?A) Emancipation Proclamation;

19th AmendmentB) Eminent Domain, Manifest

DestinyC) Civil War

References

• Orlich,Donald C, Harder,Robert J.,Callahan,Richard C, and Gibson,Harry W.(1998), Teaching Strategies: A guide to better instruction, 5th Ed, Houghton Mifflin, New York

• Royse, David (2001) Teaching Tips for College and University Instructors: A Practical Guide, Allyn and Bacon, Boston

References

• Dominowski, Roger L. (2002),Teaching Undergraduates, Lawrence Erlbaum Associates, New Jersey

• Paul Eggen, Don Kauchak (2001),Educational Psychology, Windows on Classrooms, 5th Ed, Pub. Merrill- Prentice Hall, Upper Saddle River, New Jersey.

• Bloom, B.S. (Ed.) (1956) Taxonomy of educational objectives: The classification of educational goals: Handbook I, cognitive domain. New York ; Toronto: Longmans, Green

• Chase, C.I. (1999).  Contemporary Assessment for Educators.  New York: Addison Wesley Longman, Inc. 

 

References

• Dominowski, R.L. (2002).  Teaching Undergraduates.  Mahwah, NJ:  Lawrence Erlbaum Associates. 

• Friedenberg, L. 1995. Psychological testing: Design, analysis, and use.  Needham Heights: Allyn & Bacon.  

 

References

References

• Kay Burke (1999), The Mindful School: How to Assess authentic Learning Pub.SkyLight Professional Development , Arlington Heights, Illinois

• Heinich, R., Molenda, M., Russell, J. D., and Smaldino, S. E. (1996). Instructional media and technologies for learning (5th ed.). (pp. 40-41). Englewood Cliffs: Prentice-Hall.

References

• Jacobs, L. C. & Chase, C. I.  (1992). Developing and using tests effectively:  A guide for faculty.  New York:  Jossey-Bass.

• Kopeikin, Hal S. (n. d.) UC Los Angeles lecturer, unpublished notes on writing effective questions.

• http://web.wi.mit.edu/sive/pub/Lab%20Resources%20Documents/Thoughts%20on%20Exam%20Quests.html

References• http://www.rpi.edu/~verwyc/chap2tm.htm• http://www.glencoe.com/sec/teachingtoday/

educationupclose.phtml/print/40• http://taesig.8m.com/createv.html• http://www.utc.edu/Administration/

WalkerTeachingResourceCenter/FacultyDevelopment/Assessment/test-questions.html

• http://www.psych.ucsb.edu/~taap/writingtests.html (defunct)