programmatic assessment for learning an example of medical education design
TRANSCRIPT
programmatic assessment for learning
an example of medical education design
assignment
Build an assessment programme for a workplace-based learning curriculum.- GP training- practice, assignments plus day-release
education- 1 year, two semesters, 4 terms- supervisor, medical educator, on-line
platform
overview
• a bit of history• programmatic assessment • the assessment programme
old model of medical competence
knowledge
TEST
TEST
skills
TEST
TEST
problem solving
TEST
TEST
attitudes
TEST
TEST
the quest for the best test
history
- oral versus written
- open versus closed items
- computer versus paper-and-pencil
- knowledge versus insight
- norm referenced versus criterion referenced
....and many more
typical approach to assessment
test
pass
fail
competent
incompetent
competent incompetent
pass
fail
but….
assessment is not non-obtrusive, so succeptible to many biases and unwanted effects
what does that mean for quality of a test?
major problems
competence is much more complex than expected
no extrinsic standards for criterion validity exist
U = R V E C A
R = reliability
V = validity
E = educational impact
C = costs
A = acceptance
quality elements of assessment
Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.
reliability
test test’
Penny
Leonard
Amy
Howard
85%
73%
59%
51%
Penny
Leonard
Amy
Howard
85%
73%
59%
51%
cut-off
validity: Kane’s view
observation
observed score
universe score
target domain
construct
practice observationfor clinical reasoningexpertise, observation
standards,scales, forms
reliability, reproducibilitysaturation,expertise
relation withmultiple choice key-feature, EMQ
think aloud,CRT, SCT
educational impact
curriculum
teachers students
assessment
content
format
scheduling
regulatory structure
educational impact
Cilliers, F. J., Schuwirth, L. W. T., Herman, N., Adendorff, H., & van der Vleuten, C. P. M. (2012). A model of the pre-assessment learning effects of summative assessment in medical education. Advances in Health Sciences Education, 17(1), 39-53.
U = Rw Vw Ew Cw Aw
R = reliability
V = validity
E = educational impact
C = costs
A = acceptance
quality elements of assessment
Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.
W = weight
But….
testing requires some strange assumptions
underlying concepts
traits are stable and generic characteristics
1
underlying concepts: stable trait
2 3 4
A
B
C
1 1 1 1
.5 .5 .5 .5
0 0 0 0
4
2
0
T 1’ 2’ 3’ 4’
A
B
C
1 1 1 1
.5 .5 .5 .5
0 0 0 0
4
2
0
PT
1
underlying concepts: stable trait
2 3 4
A
B
C
0 .5 .5 0
1 .5 0 1
1 1 .5 1
1
2.5
3.5
T 1’ 2’ 3’ 4’
A
B
C
1 0 .1 1
1 .5 0 0
0 .5 0 0
2
1.5
.5
PT
traits are stable and generic characteristics
individual items in themselves are meaningless
underlying concepts
underlying concepts: meaningless items
Ms. Smit is 72 years old. She has angina pectoris. Several times her blood pressure is taken andfound to be 170/100 mmHg.Which antihypertensive drug is most indicated for her??a captopril.b chloorthalidon.c metoprolol.
Mr. Johnson, 35 years old, consults his GP withcomplaints of chest pain. Without further informationabout Mr. Johnson the most likely origin of his chest pain is:a the chest wall;b the lungs;c the myocardium;d the esophagus.
underlying concepts: meaningless items
underlying concepts: meaningless items
resuscitation ‘station’ in a skills test
underlying concepts: meaningless items
communication ‘station’ in a skills test
underlying concepts
traits are stable and generic characteristics
individual items in themselves are meaningless
sum scores determine what the test measures
statistics are based on elimination of information
underlying concepts: reductionism
1
2
3
4
5
6
a
c
b
a
a
e
answer
b
c
a
a
b
e
key
0
1
0
1
0
1
50%
0%
100%
failed
underlying concepts
traits are stable and generic characteristics
individual items in themselves are meaningless
sum scores determine what the test measures
statistics are based on elimination of information
one single best instrument for each trait
old model of medical competence
knowledge
TEST
TEST
skils
TEST
TEST
problem solving
TEST
TEST
attitudes
TEST
TEST
competencies
competencies are simple or more complex tasks a successful candidate must be able to handle, and during which s/he uses at the right time the correct and relevant knowledge, skills, attitudes and meta-cognitions tomanage the situation successfully.
competency domains or roles
National Dutch blue print:
1 medical expert2 scientist3 worker in the health care system4 person
overview
• a bit of history• programmatic assessment • the assessment programme
from building blocks…
…to buildings
from methods to programmes
► multiple instruments, various formats
► strengths and weaknesses combined
► assessment moments ≠ decision moments
every assessment moment is a decision moment
test
decision
test
decision
test
decision
+ += competent
every assessment moment is NOT a decision moment
AA
A
AA
A
AA
A
AA
A
AA
A
AA
A
low stakes medium stakes high stakes
from methods to programmes
► multiple instruments, various formats
► strengths-weaknesses combined
► assessment moment ≠ decision moment
► multiple quality approaches
quality: reliability
- consistency
- saturation
- expertise
- organisation
reliability is sampling
testingtime
inhours
1
2
4
8
MCQ1
0.62
0.76
0.93
0.93
shortessay2
0.68
0.73
0.84
0.82
papercases1
0.36
0.53
0.69
0.82
orals3
0.50
0.69
0.82
0.90
obser-vation
assessment4
0.43
0.60
0.76
0.86
practiceVideotest5
0.68
0.81
0.87
0.90
1Norcini et al., 19852Stalenhoef-Halling et al., 19903Swanson, 1987
4Newble & Swanson, 19875Ram et al., 1999
generalisability: saturation
orange
greenblueredyellow
purpleblack
nothing new
nothing new
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
OverviewSept
Dec
Mar
May
June
Introduction to students
Mentors are trained
First portfolio submissionFormative review
Examiner training(benchmark portfolios)
2nd portfolio submission
Summative review
Steps in the year
Mentor/student Recommendation F/P/D
Exam committee decision
YesMentoragrees with
Student
Examineragrees with
Mentor
Examiner 1
Ex 1agrees with
Ex 2
FinalJudgment
Examiner 2
Examiner 1
Full committee (n=20)
Portfolio Committee Assessment Procedure
YesYes
No
No
No
from methods to programmes
► multiple instruments, various formats
► strengths-weaknesses combined
► assessment moment ≠ decision moment
► multiple quality approaches
► many instruments : many competency domains
A B C D
1 role 1 instrument
instruments
med expert
scientist
worker in HCS
person
domains
multi-modal assessment
med expert
scientist
worker in HCS
person
domains
instruments
from methods to programmes
► multiple instruments, various formats
► strengths-weaknesses combined
► assessment moment ≠ decision moment
► multiple quality approaches
► many instruments : many competency domains
► integrative holistic not reductionist
overview
• a bit of history• programmatic assessment • the assessment programme
assignment
Build an assessment programme for a workplace-based learning curriculum.- GP training- practice, assignments plus day-release
education- 1 year, two semesters, 4 terms- supervisor, medical educator, on-line
platform
design
• goals and stated purpose• programme in action• supporting the programme• documenting of the programme• improvement approaches to the
programme• accounting for the programme
Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.
If ‘incompetence’ were an illness, how would we diagnose and treat it?
design
• multiple instruments• meaningful collation• learning focused• self-regulation• assessment moment ≠ decision moment• longitudinal• feasible and efficient
purpose
safe independent practitioner
• medical expert• worker in het healthcare system• person• scholar
what is safe?
what is safe?
mastery + skill + competence +….
self regulation
self regulation
• self driven– analyses– external information seeking– goal orientation– prioritisation– realisation/attainment– time management
1 Bandura A. social cognitive theory: an agentic perspective. Annual Review Psychology 2001;52:1-26.2 Dochy F, M.Segers, Sluijsmans D. The Use of Self-, Peer and Co-assessment in Higher Education;a review. Studies in Higher Education 1999;24(3):331-50.3 Eva KW, Cunnington JPW, Reiter HI, Keane D, G N. How can I know what I don't know? Poor self assessment in a well-defined domain. Advances in Health Sciences Education 2004;9:211-24.
The opposite of good is...
…well intended
perfect assessmentprogram
time competing demands
understanding patient care
motivationcosts
expertise beliefs
laws
expectationscontext
culture
relevant research findings
• meaningfulness
1. Posner MI. What is it to be an expert? In: Chi MTH, Glaser R, Farr MJ, editors. The nature of expertise. Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc, 1988:xxix - xxxvi.2. Schmidt HG, Boshuizen HP. On acquiring expertise in medicine. Special Issue: European educational psychology. Educational Psychology Review 1993;5(3):205-21.
learning in context• a newspaper is better than a glossy magazine• the seashore is better than the street• first it is better to run than to walk• you will have to try several several times• some skills are required but it is easy to learn• even small children can enjoy it• once successful the risk of complications is minimal• birds seldom get too close• rain soaks in very fast• a rock can serve as an anchor• once it breaks loose there is not second chance
learning in context: flying a kite
• a newspaper is better than a glossy magazine• the seashore is better than the street• first it is better to run than to walk• you will have to try several several times• some skills are required but it is easy to learn• even small children can enjoy it• once successful the risk of complications is minimal• birds seldom get too close• rain soaks in very fast• a rock can serve as an anchor• once it breaks loose there is not second chance
relevant research findings
• meaningfulness• transfer and domain specificity
1. Eva K. On the generality of specificity. Medical Education 2003;37:587-8.2. Eva KW, Neville AJ, G.R. N. Exploring the etiology of content specificity: Factors influencing analogic transfer and problem solving. Academic Medicine 1998;73(10):s1-5.
analogous transfer
relevant research findings
• meaningfulness• transfer and domain specificity• deliberate practice
Ericsson KA. An expert-performance perspective of research on medical expertise: the study of clinical performance. Medical Education 2007;41:1124-30.
deliberate practice
deliberate practice
feedback
• concrete• constructive• focused on improvement• ‘connected’• leading to learning goals/learning plans
Shute V. Focus on formative feedback. Review of educational research 2008;78(n):153-89.
loop
learning goals
learningactivities
practiceanalysi
s
feedback
relevant research findings
• meaningfulness• transfer and domain specificity• deliberate practice• self-regulated learning
self-regulated learning
forethought, planning & activation
monitoring
control
reaction & reflection
cf. Schunk DH (2005). Self-regulated learning: The educational legacy of Paul R. Pintrich. Educational Psychologist, 40, 85-94
cognition motivation
behaviour context
phases areas
cognition motivation
behaviour context
cognition motivation
behaviour context
cognition motivation
behaviour context
relevant research findings
• meaningfulness• transfer and domain specificity• deliberate practice• self-regulated learning• reasoning and decision making
1. Boreham NC. The dangerous practice of thinking. Medical Education 1994;28:172-79.2. Klein G. Naturalistic Decision Making. Human Factors 2008;50(3):456-60.3. Plous S. The psychology of judgment and decision making. New Jersey: McGraw-Hill inc., 1993.4. Schmidt HG, Machiels-Bongaerts M, Hermans H, ten Cate TJ, Venekamp R, Boshuizen HPA. The Development of Diagnostic Competence: Comparison of a Problem-based, and Integrated, and a Conventional Medical Curriculum. Academic Medicine 1996;71(6):658-64.
relevant research findings
• reliability• validity• quality frameworks• organisational reliability
1. Williams M, Klamen D, McGaghie W. Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine 2003;15(4):270-92.2. Kane MT. Validation. In: Brennan RL, editor. Educational Measurement. Westport: ACE/Praeger, 2006:17 - 64.3. Govaerts MJB. Climbing the pyramid; Towards understanding performance assessment. Maastricht University, 2011.4. Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93
CCA
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
direct obs. direct obs. direct obs. direct obs.direct obs.
mcq test
mini-releases
MSF
portfolio
mini-releases mini-releases mini-releases mini-releases
term 1
mid-term end-term
CCA
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
direct obs. direct obs. direct obs.
audit audit
mcq test
MSF
mcq test
mini-releases mini-releases mini-releases mini-releases mini-releases
term 2
mid-term
portfolio
end-term
critical case analysis
• 5 write-ups of real patient consultations• relevance• analysis• learning activities• produce exam questions (EMI, KFP, MCQ)• increasingly original literature• any discussion minuted by registrar
directly observed consultations
• 9 real patient consultations• relevance• analysis• learning goals (practical + theoretical)• learning activity• demonstration of success in next observed
consultation• discussion minuted by registrar
clinical audit
• analysis of the practice environment• determination of specific question• collection of data• draw conclusions• describe plan for change• + 3 months: look back and annotate• any discussion minuted by registrar
multiple-choice tests• 3 tests of 60 items each• blueprinted• sit and submit your answers• review items, answer key• comment on an criticise questions for correctness• present in min-release• ‘lodge’ appeal against questions• score calculation and feedback to registrars
mini-releases
• flexible agenda• building informal networks• discuss MCQ test items• compile
– appeal against questions– list of ‘informal’ network
multi-source feedback• 2 times per year• nurses, practice manager, receptionist, other
practice staff and registrar• discussed with supervisor (end-term
assessment) and with ME (minuted by registrar)• simple form: dealing with tasks, other and
yourself• simple ordinal scale• ample room for qualitative comments
mid and end-term assessment
• integrative• reviewing all the information• learning goals and/or remediation plans• advisory to performance review
committee• minuted by registrar
portfolio
• complete dossier including minutes• individual extra information (only if
relevant)• audit trail • basis for future CV or position
applications
example of a ‘line’
CCA
learning
MCQs
test
appeal
group
groupappeal
feedback
analysis
feedback
meaning
test enhanced learning
informal/social networks
transformationresearch: narrativesfor feedback
design
• goals and stated purpose• programme in action• supporting the programme• documentation of the programme• improvement approaches to of the
programme• accounting for the programme
Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.
rules and regulation
• self responsibility comes with accountability (minutes, plagiarism, fraud)
• focus on learning and remediation• information provision to the registrar• documentation• transparency• second opinion/appeals/switch of ME or
supervisor• organisation reliability/credibility
staff development
• efficiency:– short analyses– concrete learning goals– focus on learning– training of staff (expertise efficiency)– admin support by admin staff– division of advisory and decision roles
further requirements
• goals and stated purpose• programme in action• supporting the programme: regulations• documentation of the programme: ELO,
fact sheets, • improvement approaches to of the
programme: systematic evaluation• accounting for the programme: research
Thank you