are scenario based items associated with more omitted answers in progress testing?
DESCRIPTION
Study presented at AMEE 2014 about the negative impact of the penalty for wrong answers in the engagement of students in scenario-based items of the International Progress Test.TRANSCRIPT
![Page 1: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/1.jpg)
Are scenario-based items associated with
more omitted answers in progress testing?
AMEE Conference - Milan, September 2nd, 2014.
C.F. Collares, Maastricht UniversityA.M.M. Muijtjens, Maastricht UniversityM.M. Verheggen, Maastricht University
D. Cecilio-Fernandes, University Medical Centre GroningenR.A. Tio, University Medical Centre GroningenC.P.M. van der Vleuten, Maastricht University
![Page 2: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/2.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Background
• Formula scoring (number-right score minus
penalties for wrong answers) has been useful in PT:
- prevention of undeserved scores due to guessing
- generalizability (especially for early years)
- promotion of metacognitive skills
• International Progress Test (IPT) Committee:
increase scenario-based items early exposure of
students to more authentic and relevant items
![Page 3: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/3.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Background
• Apparent increase of omitted answers (“I don’t know
option”, “question mark”)
• Student dissatisfaction, demotivation and
disengagement
• Educational utility and future of progress testing
paradoxically in jeopardy
![Page 4: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/4.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Method• Instrument: one edition of the International Progress Test
• Participants:198 students.
• Dependent variable: % of omitted answers of each item
• Independent variables:
- clinical scenario
- stem word count
- item number
- number of alternatives
- dummy variables related to the content domains (subscores).
• Bootstrapped multiple linear regression (SPSS)
![Page 5: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/5.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Results
![Page 6: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/6.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Regression coefficients (all years)
R = 0,577R2 = 0,333
Durbin-Watson = 2,124
ANOVAF = 4,078 (20) p < 0,001
![Page 7: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/7.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Results by year
• The regression coefficients of scenario-based items
on question mark prevalence were the highest and
p values were the lowest on Year 1.
• The observed increase of omitted answers in
scenario-based items gradually decreased until it
disappeared in the fourth academic year.
![Page 8: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/8.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Conclusion
• Scenario-based items were associated with
increased omitted answers, independently from
content, number of alternatives, item length and
item position, particularly in early years
• The current progress testing framework might be
associated to a delay in the engagement of
students into solving scenario-based items
![Page 9: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/9.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
What now?
Is progress testing as we know it
“wearing off”?
Is it time for a new progress testing
framework?
![Page 10: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/10.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
How can we keep the positive
effects of progress testing on
metacognition while enhancing
item relevance and authenticity
and ensuring accurate scores?
![Page 11: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/11.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
PROPOSAL FOR A NEW
PROGRESS TEST FRAMEWORK
![Page 12: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/12.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
It is common for mentors to coach students to
estimate their degree of “certainty” or
“confidence” in their answers before writing
them in the progress test answer sheet.
>75% confidence = go for it and answer!
50-75% = think again, leave it for later
<50% = “I don’t know” (?)
![Page 13: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/13.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
The key to progress testing future...
...might be in the past.
![Page 14: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/14.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
(Leclercq, 1975)
Separate “confidence marking”
(Leclercq, 1982)
![Page 15: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/15.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Enhanced progress testing framework
• Metacognitive knowledge gets its own score
• “Confidence” levels are expressed in a Likert scale
• Each category of the scale represents a % range
• Penalties for wrong answers go to the
metacognition scores only
![Page 16: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/16.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Expected adjustments
• Less items per test
• Multiaxial blueprint with flexible item tagging
(clusters, competences, categories, disciplines) to
allow more items per subscore
• Separate norms for the metacognition scores
• Scores shall be given in standardized item response
theory “theta scores” (mean = 500; SD = 100)
![Page 17: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/17.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Expected outcomes
• Better item relevance and professional authenticity
• Strengthened metacognitive regulation
• Increased reliability (Ferrando et al., 2013)
• Less construct-irrelevant score variance ( validity)
• Early student engagement in scenario-based items
• Authentic items promoting learning and transfer
• Better educational utility
![Page 18: Are scenario based items associated with more omitted answers in progress testing?](https://reader034.vdocuments.mx/reader034/viewer/2022051818/5499beacb4795929178b45d7/html5/thumbnails/18.jpg)
FHML – Dept. of Educational Development and Research – School of Health Professions Education
Thank you!