“There are known knowns. These are things we know that we know. There are known unknowns. That is to say, there are things that we know we don't know. But there are also unknown unknowns. There are things we don't know we don't know.”

Donald Rumsfeld, February 2002

The use of regular multiple choice quizzes to enhance the

development of students self-assessment and self-feedback skills

Pecha Kucha PresentationAlan Brown

CHEP Annual ConferenceJanuary 2013

4 types of student?

A thirteenth-century Persian poet, Ibn Yamin said there are four types of men:1. One who knows and knows that

he knows... His horse of wisdom will reach the skies.

2. One who knows, but doesn't know that he knows... He is fast asleep, so you should wake him up!

Source: Wikipedia, 2013

4 types of student?

3. One who doesn't know, but knows that he doesn't know... His limping mule will eventually get him home.

4. One who doesn't know and doesn't know that he doesn't know... He will be eternally lost in his hopeless oblivion!

• Our first job is to get students away from 4

Source: Wikipedia, 2013

Fluid Mechanics in Year 1

• Brand new subject• Not on ‘A’ level physics or maths syllabus• Several new concepts– Pressure– Viscosity– Density

Fluid Mechanics in Year 1

• Several ‘Old’ concepts– Forces– Velocity and Acceleration– Momentum

• But these don’t look the same as in physics

Assessments historically based on applied mathematical problems

The problem

• Enable students to find out what they know and do not know

• This is the role of Feedback• Several new concepts requires lots of rapid

feedback

• I taught this module to 88 students

The solution

• Multiple choice quiz questions each week on Blackboard

• Based on the previous week’s material

• The adaptive release system is used to enforce the quiz questions

• Any attempt at the questions allows access to the next set of resources

• Answers and feedback are given instantly

• The questions are designed to test conceptual knowledge, not learned definitions

• Class test uses similar types of multiple choice questions

Result 1: Class test marks

• Scores in the class test were similar to the previous year (applied maths problems)

• Quiz designed to test the understanding of the new and old concepts– misunderstandings much easier to spot and correct

than using applied mathematics type questions

Result 1: Class test marks

Feedback to correct conceptual misunderstandings given 1 week after the test

Result 2: Predicting performance

• Are students more aware of their own knowledge?

• Students were asked to predict their grade:– Before they sat the test– After they sat the test– After they had seen all the answers

Result 2: Predicting performance

• With MCQs the students were much better at predicting their marks

With MCQs Without MCQs

Result 3: Engagement

• Out of 88 students–4 attempted none–7 attempted 1 or 2–15 attempted 3 or 4–62 attempted 5 or

more

Test attemptsMean overall

mark0 241 122 253 424 525 486 537 54

Result 3: Engagement

• Student feedback on the MCQ system :–87 % of respondents found the MCQs

helpful or very helpful, –5 % said they were unhelpful

• Many students answered multiple times

What have I learned

• New concepts are difficult to learn• Students need to start from a position of self-

knowledge• Applied mathematics is not an effective way of

assessing conceptual knowledge• Most students will engage with feedback

opportunities

What I will do next

• Check MCQ answers to try and determine misunderstandings earlier– Don’t wait until the class test

• Refine teaching based on misunderstandings from last year

• Enhance the automated feedback in the MCQs• Use MCQs to monitor engagement

Conclusions

• Rapid feedback should enhance students self-knowledge

• MCQs used to give rapid automated feedback to 88 students on a weekly basis

• Adaptive release gives an incentive without marking and assessment

Conclusions

• Students are better able to predict their marks• Students were overwhelmingly positive about

the quiz questions• Engagement easy to assess

• Hopefully fewer “Unknown Unknowns”