university of strasbourg ea.sauleau@unistra -...
TRANSCRIPT
Introduction Redaction Scoring Conclusion
Exam with multiple choice questionsNovember 10, 2011
Erik A. Sauleau
Department of Biostatistics - Faculty of MedicineUniversity of Strasbourg
Introduction Redaction Scoring Conclusion
Outline
1 Introduction
2 How to write a multiple choice question?
3 Scoring
4 Conclusion
Introduction Redaction Scoring Conclusion
Outline
1 Introduction
2 How to write a multiple choice question?
3 Scoring
4 Conclusion
Introduction Redaction Scoring Conclusion
Formal definition
Multiple choice question: MCQ
Educational evaluation tool
Def: Student answers to the question by selecting one or moreitems among several
In general among 5 items
Each item is classified in ”correct” or ”not correct”
For what kind of evaluation?
KnowledgeComprehension, application and analysis
Not for what?
Synthesis, evaluation, reasoningCreativityWriting capacities, speaking capacities
Introduction Redaction Scoring Conclusion
Constitution
Here is the question
A A correct answer
B Another correct answer
C Oh, this is a diversion item
D Here another one bait
E A possibility for ”another answer”
Warn students whether they can choose several items or not
Introduction Redaction Scoring Conclusion
Two main types
1 Simple MCQ ⇒ one correct answer
What is my first name?
A Boris
B Erik-Andre
C Sauleau
D Olga
E Other answer
1 Simple MCQ ⇒ one correct answer2 Multiple MCQ ⇒ several correct answers
What are my main skills?
1 Biostatistics
2 Cooking
3 Education
4 Stealing
5 Flying
Pattern for answer: A: 1+3, B: 1+4+5, C: 2 (definitively not), D:3+5, E: Other answer
3 Several other
Introduction Redaction Scoring Conclusion
Two main types
1 Simple MCQ ⇒ one correct answer
2 Multiple MCQ ⇒ several correct answers
What are my main skills?
1 Biostatistics
2 Cooking
3 Education
4 Stealing
5 Flying
Pattern for answer: A: 1+3, B: 1+4+5, C: 2 (definitively not), D:3+5, E: Other answer
3 Several other
Introduction Redaction Scoring Conclusion
Example of answer grid (with repentance)
Introduction Redaction Scoring Conclusion
Pros and cons
Pros
Automatic correction by scanning or data processing ⇒objectivity
Quick, simple and accurate knowledge evaluation
Cover a wide range of a given field
Pros
Cons
Danger: get into corners
Test for memorisation and not understanding
Long preparation time ⇒ lecturer database
Student database ⇒ need for quick renewal
Correct answer by random
Increase the number of diversion itemsDecrease the mark in case of incorrect answerAsk to the student to add his own uncertainty assessment andchange the mark
Introduction Redaction Scoring Conclusion
Pros and cons
Pros
Cons
Danger: get into corners
Test for memorisation and not understanding
Long preparation time ⇒ lecturer database
Student database ⇒ need for quick renewal
Correct answer by random
Increase the number of diversion itemsDecrease the mark in case of incorrect answerAsk to the student to add his own uncertainty assessment andchange the mark
Introduction Redaction Scoring Conclusion
Outline
1 Introduction
2 How to write a multiple choice question?
3 Scoring
4 Conclusion
Introduction Redaction Scoring Conclusion
General issues
Meet educational aimsSyntax
Syntactically correct sentencesAvoid vague terms and ambiguitiesPositive and interrogative forms: avoid negatives and especiallydouble negativesNo value judgments
Independent questions (no cumulative time in prison)Time for students to answer
Simple question → 1 minuteMore complex question → up to 3 minutes
Expert able to answer to the exam (with a good mark)Detail general and specific instructions
Among these propositions which belong to clinical signs of flu?
A Stuffy nose
B Cough
C Fever
D Edema
E Other answer
Introduction Redaction Scoring Conclusion
General issues
Among these propositions which one are not a clinical sign of flu?
A Stuffy nose
B No cough
C Fever less than about 39◦C
D Contact with person with flu
E Other answer
Among these propositions which belong to clinical signs of flu?
A Stuffy nose
B Cough
C Fever
D Edema
E Other answer
Introduction Redaction Scoring Conclusion
General issues
Among these propositions which belong to clinical signs of flu?
A Stuffy nose
B Cough
C Fever
D Edema
E Other answer
Introduction Redaction Scoring Conclusion
The question
Just one question on a given part and not just the question”What is the correct answer?”Use in the question all common words of the answersExpert could be able to answer to the question without thesolutionsAnchoring
Several (10% to 30%) well-known questions (by the lecturer)for calibrating the examPoor global results but correct results to these questions →exam too difficultPoor global results with poor results to these questions →worse students cohort
What is the correct second age milk dose for a one month oldbaby?
A 60g
B 120g
C 180g
D 240g
E Other answer
Introduction Redaction Scoring Conclusion
The question
What is the correct dose for a one month old baby?
A Six baby bottles with 120g of second age milk
B Four baby bottles with 120g of second age milk
C Six baby bottles with 60g of second age milk
D Six baby bottles with 120g of first age milk
E Other answer
What is the correct second age milk dose for a one month oldbaby?
A 60g
B 120g
C 180g
D 240g
E Other answer
Introduction Redaction Scoring Conclusion
The question
What is the correct second age milk dose for a one month oldbaby?
A 60g
B 120g
C 180g
D 240g
E Other answer
Introduction Redaction Scoring Conclusion
The answers
Same syntactic construction and same length for all itemsSame likelihood and same technical complexityDiversion items
Not synonymes, no overlappingRealisticSame field than correct answer (paraclinical exam, treatment,. . . )They are undoubtedly diversion items
The answer ”other answer” for lecturer protection
What is the correct diet in each of the six daily baby bottles for aone month old baby?
A 60g of second age milk
B 120g of second age milk
C 120g of first age milk
D Hamburger
E Other answer
Introduction Redaction Scoring Conclusion
The answers
What is the correct diet in each of the six daily baby bottles for aone month old baby?
A 60g of second age milk
B 120g of second age milk
C 120g of first age milk
D Hamburger
E Other answer
Introduction Redaction Scoring Conclusion
The correct answer(s)
They are undoubtedly correct answers
Random place among diversion items
Follow same syntactic rules than diversion items
Introduction Redaction Scoring Conclusion
Using adverbs
Never say never (or always)
In several adverbs implicit frequency: usually > sometimes >seldom
⇒ Add a warning: ”In this exam ”seldom” means ”less than 10%”or Avoid adverbs and use more neutral terms: ”is associated
with”, ”can be explained by”, . . .
Introduction Redaction Scoring Conclusion
Outline
1 Introduction
2 How to write a multiple choice question?
3 Scoring
4 Conclusion
Introduction Redaction Scoring Conclusion
Simple MCQ
Correct No answer Incorrect
Binary +1 0 0
Penalisation for incorrectness +1 0 − 1P
Valorisation for omission +1 + 1P 0
Both +1 + 1P − 1
P ′
Choice of P and P ′: 1 for strong effect but also 2,3 . . .
Introduction Redaction Scoring Conclusion
Multiple MCQ: c correct among k items
More complex
Two questions1 Penalisation for incorrectness and weight2 Valorisation for omission and weight
Introduction Redaction Scoring Conclusion
Outline
1 Introduction
2 How to write a multiple choice question?
3 Scoring
4 Conclusion
Introduction Redaction Scoring Conclusion
For a good MCQ exam
Be rigourous
Be robust
Be helped
. . . but stay cool and gentle
Introduction Redaction Scoring Conclusion
For a good MCQ exam
Be rigourous
Be robust
Be helped
. . . but stay cool and gentle
Introduction Redaction Scoring Conclusion
”Oui” et ”non” sont les mots les plus courts et les plus faciles aprononcer, et ceux qui demandent le plus d’examen.Charles-Maurice de Talleyrand (1754-1838)
Thank you for your attention