opening up multiple choice: assessing with confidence self-assessment: strategies and software to...
TRANSCRIPT
Opening up multiple choice: Assessing with confidence
Self-assessment: strategies and software to stimulate learningHEA workshop, The Open University, 11th June 2012
MCQ Open question• Pros
– objective marking– reliable marking– easy to implement
• Cons– distractors trivial– distractors engender
misconceptions– working backwards
• Pros– numeric easy to mark– tests deeper learning– can find misconceptions
• Cons– Free text very difficult to
mark reliably
Open question
Multiple-choice question
How (not) to do MCQ…• Give the name of a supervolcano:
• Yellowstone
• Redstone
• Bluestone
• Which is a plate boundary:• Conservative
• Destructive
• Liberal
• Labour
• What policy did the Chinese introduce to limit population:• One child policy
• Two child policy
• The child catcher
Reported from BBC Bite Size
Confidence-Based Marking (CBM)Certainty-Based
CBM scoring
Confidence
Score
Correct Wrong
Low 1 0
Medium 2 -2
High 3 -6
Tentative & correct
Confidently correct
Cocksure – and wrong!
Gardner-Medwin & Curtin (2007) Certainty-Based Marking (CBM) for Reflective Learning and Proper Knowledge Assessment[http://www.ucl.ac.uk/lapt/REAP_cbm.pdf]
CBM motivations• Rewards care and effort• Greater engagement• Encourages reflective learning• Encourages self-assessment
Open CBM
Open CBM advantages• Mechanically, question remains simple MCQ
– answer matching is trivial– easy to implement– possible to reuse existing question banks
• But transformed closed to open question– student must formulate answer to decide confidence– must decide answer without +ve or –ve clues.– cannot work backward – will not be led into misconceptions.
must answer the question as set.
MCQ Open CBM• Pros
– objective marking– reliable marking– easy to implement
• Cons– distractors trivial– distractors engender
misconceptions– working backwards
• Pros– open question– reflection– ...as MCQ
• Cons– not always applicable– intimidating?– personality dependent?
Possible benefits• Students
– better engagement– improved learning– more reflective learners
• Staff– easier question setting
eSTEeM project plan• Implement open CBM question type
– in OpenMark• Trial in course assessment
– controlled experimental design– pilot (formative) on T216 CISCO networking
• Log measures– Assignment scores– Time spent on task
• Structured interviews to probe attitudinal aspects• User-lab trials
T216 Cisco networking• Questions from existing bank• Offered as revision quizzes• One set for each of four blocks• 25 questions drawn randomly from bank of ~100 • Randomly offered as either open CBM or MCQ format• Issues:
– Some questions bad!– OpenMark report doesn’t show confidence– Too much randomness
UptakeQuiz Finished Unfinished
Demo 162 178
CCNA 1 213 262
CCNA 2 86 130
CCNA 3 69 51
CCNA 4 64 67
Demo questions – scoresQuestion Taken by Mean
score
What is 2 + 2? 238 2.51
What is derivative of x³? 223 -0.47
Who painted the 'Mona Lisa'?
212 2.14
Who is the 'Mona Lisa'? 208 0.21
Uncertainty principle -- whose?
207 0.03
Uncertainty principle -- formula?
218 0.01
Easy
Difficult
Tricky!
Demo questions – confidenceQuestion High Medium Low
What is 2 + 2? 210 5 23
What is derivative of x³? 96 46 81
Who painted the 'Mona Lisa'?
165 21 26
Who is the 'Mona Lisa'? 56 41 111
Uncertainty principle -- whose?
62 31 114
Uncertainty principle -- formula?
39 21 158
MCQ Open CBM• Pros
– objective marking– reliable marking– easy to implement
• Cons– distractors trivial– distractors engender
misconceptions– working backwards
• Pros– open question– reflection– ...as MCQ
• Cons– not always applicable– intimidating?– personality dependent?
