aviad haramati, phd peggy a. weissinger, edd georgetown university school of medicine

1

Center for New Designs in Learning & Scholarship and Office of Faculty and Academic Affairs

Writing Multiple Choice Questions

Aviad Haramati, PhDPeggy A. Weissinger, EdD

Georgetown University School of Medicine

Georgetown University

2/12/2013 CNDLS & OFAAGeorgetown University

2

Agenda

Welcome & introductionsWhy do we assess?Common flaws in MCQGuidelines for writing quality MCQLet’s do it!Resources

3

Why do we assess?

4

Objectives

Activities Assessment

Why do we assess?

Adapted from Felder & Brent ( 1991)

5

Objectives


What should mystudents know? Whatshould they be able to do?

Why do we assess?


6

Objectives


What type of activitieswill promote and develop this knowledge?What type of activities can help students achieve the learning objectives?

Why do we assess?


7

Objectives


How will I know that they have achieved the

objectives?

Why do we assess?


8

Objectives


Congruence!


9

What am I testing?

Before creating an exam, ask yourself:What do the students have to know (or do)?What am I testing?

TLISI - Georgetown University 10

What am I testing?

Core Principles: Decide exactly what you want to test. Eliminate as many ambiguities as possible. Create distractors based firmly on what you want to

test. After it is over, use it as a learning opportunity!

– Give feedback for your distracters which explains why they’re wrong.

– Give feedback for the correct answer explaining why it’s right.

Adapted from http://hotpot.uvia.ca/howto/mcquestion.htm


Technical Flaws

Issues Related to TestwisenessGrammatical cues

Reference: Case & Swanson (NBME) Constructing Written Test Questions


Grammatical Cues

A 60-year-old man is brought to the emergency department by the police, who found him lying unconscious on the sidewalk. After ascertaining that the airway is open, the first step in management should be intravenous administration of

a. examination of cerebrospinal fluidb. glucose with vitamin B1 (thiamine)c. CT scan of the headd. phenytoine. diazepam

Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 19


Technical Flaws

Issues Related to TestwisenessGrammatical cuesLogical cues



Logical Cues

Crime isa. equally distributed among the social classesb. overrepresented among the poorc. overrepresented among the middle class and

richd. primarily an indication of psychosexual

maladjustmente. Reaching a plateau of tolerability for the nation


a, b, c are collectively exhaustive d, e may be partially correct but not same dimension as others


Technical Flaws

Issues Related to TestwisenessGrammatical cuesLogical cuesAbsolute termsLong correct answer



Long Correct Answer

Secondary gain isa. synonymous with malingeringb. a frequent problem in obsessive-compulsive

disorderc. a complication of a variety of illnesses and tends to

prolong many of themd. never seen in organic brain damage.



Technical Flaws

Issues Related to TestwisenessGrammatical cuesLogical cuesAbsolute termsLong correct answerWord repeats


18

Word Repeats

A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated. He speaks of experiencing the world as unreal. This symptom is called

a. derealizationb. depersonalizationc. derailmentd. focal memory deficite. signal anxiety


Technical Flaws

Issues Related to TestwisenessGrammatical cuesLogical cuesAbsolute termsLong correct answerWord repeatsConvergence strategy


20

Convergence Strategy

Local anesthetics are most effective in thea. anionic form, acting from inside the nerve

membraneb. cationic form, acting from inside the nerve

membranec. cationic form, acting from outside the nerve

membraned. uncharged form, acting from inside the nerve

membranee. uncharged form, acting from outside the nerve

membrane


Technical Flaws

Issues Related to Irrelevant Difficulty Options long, complicated, or double



Long, Complicated Options Peer review committees in HMOs may move to take action against a

physician’s credentials to care for participants of the HMO. There is an associated requirement to assure that the physician receives due process in the course of these activities. Due process must include which of the following?

a. Proper notice, a tribunal empowered to make the decision, a chance to confront witnesses against him/her, and change to present evidence in defense.

b. Notice, an impartial forum, council, chance to hear and confront evidence against him/her.

c. Reasonable and timely notice, impartial panel empowered to make a decision, a chance to hear evidence against himself/herself and to confront witnesses, and the ability to present evidence in defense.

Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 23)


Technical Flaws

Issues Related to Irrelevant Difficulty Options long, complicated, or double Numeric data not stated consistently Terms in options are vague Language in options not parallel Options in non-logical order Stems are tricky or unnecessarily complicated



Tricky Stems Arrange the parents of the following children with Down’s syndrome in

order of highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are:

I: 46, XX, -14, +T (14q21q) patII: 46, XX, -14, +T (14q21q) de novoIII: 46, XX, -14, +T (14q21q) matIV: 46, XX, -21, +T (14q21q) patV: 47, XX, -21, +T (21q21q) (parents not karyotyped)

a. III, IV, I, V, IIb. IV, III, V, I, IIc. III, I IV, V, IId. IV, III, I, V, IIe. III, IV, I, II, V

Reference: Case & Swanson (NBME) Constructing Written Test Questions, p. 25.


Technical Flaws

Issues Related to Irrelevant Difficulty Options long, complicated, or double Numeric data not stated consistently Terms in options are vague Language in options not parallel Options in non-logical order Stems are tricky or unnecessarily complicated Answer is “hinged” to a related item


5/21/08 TLISI - Georgetown University 26

Item Shape

Long Stem

a.b.c.d.e.

Appropriately Shaped Item

Poorly Shaped Item

Short Stem

a.b.c.d.e.


When Writing Options...

…make sure that they are:Homogenous in content Incorrect or inferior to the correct answerPlausible and attractive to the uninformedSimilar to the correct answer in constructionGrammatically consistent and logically

compatible with the stem


28

Your opinion, please!Osteopenia is caused by which of the following?A. BoronB. PhosphorusC. MagnesiumD. Strontium

29

Edited Item A 55 year-old woman presents with osteopenia. In reviewing her dietary profile, you suspect one of the following may be deficient. Which is it? A. BoronB. PhosphorusC. MagnesiumD. Strontium

30

Your opinion, please!A 60 year-old male runner complains of being confused, irritable and very lethargic. He has hypertension for which he is taking 10 mg of Lisinopril, an angiotensin converting enzyme inhibitor. He states that he drinks ~3L of water and eats “lots of fruits and vegetables” daily. Which of the following is the likely factor producing his symptoms?A. Potassium depletionB. Sodium depletionC. Lisinopril resistanceD. Decreased renal blood flow

31

Edited ItemA 60 year-old male runner complains of being confused, irritable and very lethargic. He has hypertension for which he is taking 10 mg of Lisinopril, an angiotensin converting enzyme inhibitor. He states that he drinks ~3L of water and eats “lots of fruits and vegetables” daily. Which of the following is the likely factor producing his symptoms?A. HypokalemiaB. HyponatremiaC. HypocalcemiaD. Hypomagnesemia


General GuidelinesBulk of question in the stem.Try “cover-the-options” test.Avoid superfluous information.Avoid “tricky” and overly complex items.Write grammatically consistent options.Avoid absolutes! (e.g. always, never, all)

And most important: Focus on important concepts. Don’t waste time testing trivial facts.



Let’s Do It!

Write one multiple choice question.


34

SAMPLE ITEM

A 25 year-old medical student who is using natural family planning to chart her fertility, just started her Surgery rotation. Since the start of the quarter, she has been waking up at 4:00 a.m. every day and getting less sleep than usual, as well as feeling generally stressed. During her next menstrual cycle, which of the following phases is most likely to be longer than normal for her? (MD)

A. Follicular PhaseB. Menstrual PhaseC. Ovulatory PhaseD. Secretory Phase

35

SAMPLE ITEM

When practicing the Billings Ovulation Method of NFP, infertile days of the cycle include… (MD)

A. Dry days after mensesB. Every day from the 4th day after the Peak Day

until the end of the cycleC. Both A and BD. All of the above

36

SAMPLE ITEM

Functional Residual Capacity or FRC (AN)

A. Is best measured by following Peak Flows

B. Is the equilibrium volume when the tendency of the lung to expand is balanced by the tendency of the chest wall to retract after full inspiration

C. Is usually greater than the residual volume

D. Is maximized when the patient is in deep sleep

E. Cannot be maintained by active inspiration

37

Pulling it together!

Summary:Decide what you want to test.Well-written objectives make the job easier.Use guidelines for writing quality MCQ.Local resources here to help!

38

Contact information

Peg [email protected] 202-687-4556

Aviad [email protected]

CNDLS & OFAAGeorgetown University

aviad haramati, phd peggy a. weissinger, edd georgetown university school of medicine

Documents

felder brent

case swanson nbme

learning objectives

technical flawsissues

learning scholarship

learning opportunity

office of faculty

vitamin b1 thiaminect