programmatic assessment for learning an example of medical education design

programmatic assessment for learning

an example of medical education design

assignment

Build an assessment programme for a workplace-based learning curriculum.- GP training- practice, assignments plus day-release

education- 1 year, two semesters, 4 terms- supervisor, medical educator, on-line

platform

overview

• a bit of history• programmatic assessment • the assessment programme

old model of medical competence

knowledge

TEST

TEST

skills

TEST

TEST

problem solving

TEST

TEST

attitudes

TEST

TEST

the quest for the best test

history

- oral versus written

- open versus closed items

- computer versus paper-and-pencil

- knowledge versus insight

- norm referenced versus criterion referenced

....and many more

typical approach to assessment

test

pass

fail

competent

incompetent

competent incompetent

pass

fail

but….

assessment is not non-obtrusive, so succeptible to many biases and unwanted effects

what does that mean for quality of a test?

major problems

competence is much more complex than expected

no extrinsic standards for criterion validity exist

U = R V E C A

R = reliability

V = validity

E = educational impact

C = costs

A = acceptance

quality elements of assessment

Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.

reliability

test test’

Penny

Leonard

Amy

Howard

85%

73%

59%

51%

Penny

Leonard

Amy

Howard

85%

73%

59%

51%

cut-off

validity: Kane’s view

observation

observed score

universe score

target domain

construct

practice observationfor clinical reasoningexpertise, observation

standards,scales, forms

reliability, reproducibilitysaturation,expertise

relation withmultiple choice key-feature, EMQ

think aloud,CRT, SCT

educational impact

curriculum

teachers students

assessment

content

format

scheduling

regulatory structure

educational impact

Cilliers, F. J., Schuwirth, L. W. T., Herman, N., Adendorff, H., & van der Vleuten, C. P. M. (2012). A model of the pre-assessment learning effects of summative assessment in medical education. Advances in Health Sciences Education, 17(1), 39-53.

U = Rw Vw Ew Cw Aw

R = reliability

V = validity

E = educational impact

C = costs

A = acceptance

quality elements of assessment

Van der Vleuten CPM. The assessment of Professional Competence: Developments, Research and Practical Implications. Advances in Health Science Education 1996;1(1):41-67.

W = weight

But….

testing requires some strange assumptions

underlying concepts

traits are stable and generic characteristics

1

underlying concepts: stable trait

2 3 4

A

B

C

1 1 1 1

.5 .5 .5 .5

0 0 0 0

4

2

0

T 1’ 2’ 3’ 4’

A

B

C

1 1 1 1

.5 .5 .5 .5

0 0 0 0

4

2

0

PT

1

underlying concepts: stable trait

2 3 4

A

B

C

0 .5 .5 0

1 .5 0 1

1 1 .5 1

1

2.5

3.5

T 1’ 2’ 3’ 4’

A

B

C

1 0 .1 1

1 .5 0 0

0 .5 0 0

2

1.5

.5

PT

traits are stable and generic characteristics

individual items in themselves are meaningless

underlying concepts

underlying concepts: meaningless items

Ms. Smit is 72 years old. She has angina pectoris. Several times her blood pressure is taken andfound to be 170/100 mmHg.Which antihypertensive drug is most indicated for her??a captopril.b chloorthalidon.c metoprolol.

Mr. Johnson, 35 years old, consults his GP withcomplaints of chest pain. Without further informationabout Mr. Johnson the most likely origin of his chest pain is:a the chest wall;b the lungs;c the myocardium;d the esophagus.

underlying concepts: meaningless items

underlying concepts: meaningless items

resuscitation ‘station’ in a skills test

underlying concepts: meaningless items

communication ‘station’ in a skills test

underlying concepts

traits are stable and generic characteristics

individual items in themselves are meaningless

sum scores determine what the test measures

statistics are based on elimination of information

underlying concepts: reductionism

1

2

3

4

5

6

a

c

b

a

a

e

answer

b

c

a

a

b

e

key

0

1

0

1

0

1

50%

0%

100%

failed

underlying concepts

traits are stable and generic characteristics

individual items in themselves are meaningless

sum scores determine what the test measures

statistics are based on elimination of information

one single best instrument for each trait

old model of medical competence

knowledge

TEST

TEST

skils

TEST

TEST

problem solving

TEST

TEST

attitudes

TEST

TEST

competencies

competencies are simple or more complex tasks a successful candidate must be able to handle, and during which s/he uses at the right time the correct and relevant knowledge, skills, attitudes and meta-cognitions tomanage the situation successfully.

competency domains or roles

National Dutch blue print:

1 medical expert2 scientist3 worker in the health care system4 person

overview

• a bit of history• programmatic assessment • the assessment programme

from building blocks…

…to buildings

from methods to programmes

► multiple instruments, various formats

► strengths and weaknesses combined

► assessment moments ≠ decision moments

every assessment moment is a decision moment

test

decision

test

decision

test

decision

+ += competent

every assessment moment is NOT a decision moment

AA

A

AA

A

AA

A

AA

A

AA

A

AA

A

low stakes medium stakes high stakes

from methods to programmes

► multiple instruments, various formats

► strengths-weaknesses combined

► assessment moment ≠ decision moment

► multiple quality approaches

quality: reliability

- consistency

- saturation

- expertise

- organisation

reliability is sampling

testingtime

inhours

1

2

4

8

MCQ1

0.62

0.76

0.93

0.93

shortessay2

0.68

0.73

0.84

0.82

papercases1

0.36

0.53

0.69

0.82

orals3

0.50

0.69

0.82

0.90

obser-vation

assessment4

0.43

0.60

0.76

0.86

practiceVideotest5

0.68

0.81

0.87

0.90

1Norcini et al., 19852Stalenhoef-Halling et al., 19903Swanson, 1987

4Newble & Swanson, 19875Ram et al., 1999

generalisability: saturation

orange

greenblueredyellow

purpleblack

nothing new

nothing new

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

OverviewSept

Dec

Mar

May

June

Introduction to students

Mentors are trained

First portfolio submissionFormative review

Examiner training(benchmark portfolios)

2nd portfolio submission

Summative review

Steps in the year

Mentor/student Recommendation F/P/D

Exam committee decision

YesMentoragrees with

Student

Examineragrees with

Mentor

Examiner 1

Ex 1agrees with

Ex 2

FinalJudgment

Examiner 2

Examiner 1

Full committee (n=20)

Portfolio Committee Assessment Procedure

YesYes

No

No

No

from methods to programmes

► multiple instruments, various formats

► strengths-weaknesses combined

► assessment moment ≠ decision moment

► multiple quality approaches

► many instruments : many competency domains

A B C D

1 role 1 instrument

instruments

med expert

scientist

worker in HCS

person

domains

multi-modal assessment

med expert

scientist

worker in HCS

person

domains

instruments

from methods to programmes

► multiple instruments, various formats

► strengths-weaknesses combined

► assessment moment ≠ decision moment

► multiple quality approaches

► many instruments : many competency domains

► integrative holistic not reductionist

overview

• a bit of history• programmatic assessment • the assessment programme

assignment

Build an assessment programme for a workplace-based learning curriculum.- GP training- practice, assignments plus day-release

education- 1 year, two semesters, 4 terms- supervisor, medical educator, on-line

platform

design

• goals and stated purpose• programme in action• supporting the programme• documenting of the programme• improvement approaches to the

programme• accounting for the programme

Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.

If ‘incompetence’ were an illness, how would we diagnose and treat it?

design

• multiple instruments• meaningful collation• learning focused• self-regulation• assessment moment ≠ decision moment• longitudinal• feasible and efficient

purpose

safe independent practitioner

• medical expert• worker in het healthcare system• person• scholar

what is safe?

what is safe?

mastery + skill + competence +….

self regulation

self regulation

• self driven– analyses– external information seeking– goal orientation– prioritisation– realisation/attainment– time management

1 Bandura A. social cognitive theory: an agentic perspective. Annual Review Psychology 2001;52:1-26.2 Dochy F, M.Segers, Sluijsmans D. The Use of Self-, Peer and Co-assessment in Higher Education;a review. Studies in Higher Education 1999;24(3):331-50.3 Eva KW, Cunnington JPW, Reiter HI, Keane D, G N. How can I know what I don't know? Poor self assessment in a well-defined domain. Advances in Health Sciences Education 2004;9:211-24.

The opposite of good is...

…well intended

perfect assessmentprogram

time competing demands

understanding patient care

motivationcosts

expertise beliefs

laws

expectationscontext

culture

relevant research findings

• meaningfulness

1. Posner MI. What is it to be an expert? In: Chi MTH, Glaser R, Farr MJ, editors. The nature of expertise. Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc, 1988:xxix - xxxvi.2. Schmidt HG, Boshuizen HP. On acquiring expertise in medicine. Special Issue: European educational psychology. Educational Psychology Review 1993;5(3):205-21.

learning in context• a newspaper is better than a glossy magazine• the seashore is better than the street• first it is better to run than to walk• you will have to try several several times• some skills are required but it is easy to learn• even small children can enjoy it• once successful the risk of complications is minimal• birds seldom get too close• rain soaks in very fast• a rock can serve as an anchor• once it breaks loose there is not second chance

learning in context: flying a kite

• a newspaper is better than a glossy magazine• the seashore is better than the street• first it is better to run than to walk• you will have to try several several times• some skills are required but it is easy to learn• even small children can enjoy it• once successful the risk of complications is minimal• birds seldom get too close• rain soaks in very fast• a rock can serve as an anchor• once it breaks loose there is not second chance

relevant research findings

• meaningfulness• transfer and domain specificity

1. Eva K. On the generality of specificity. Medical Education 2003;37:587-8.2. Eva KW, Neville AJ, G.R. N. Exploring the etiology of content specificity: Factors influencing analogic transfer and problem solving. Academic Medicine 1998;73(10):s1-5.

analogous transfer

relevant research findings

• meaningfulness• transfer and domain specificity• deliberate practice

Ericsson KA. An expert-performance perspective of research on medical expertise: the study of clinical performance. Medical Education 2007;41:1124-30.

deliberate practice

deliberate practice

feedback

• concrete• constructive• focused on improvement• ‘connected’• leading to learning goals/learning plans

Shute V. Focus on formative feedback. Review of educational research 2008;78(n):153-89.

loop

learning goals

learningactivities

practiceanalysi

s

feedback

relevant research findings

• meaningfulness• transfer and domain specificity• deliberate practice• self-regulated learning

self-regulated learning

forethought, planning & activation

monitoring

control

reaction & reflection

cf. Schunk DH (2005). Self-regulated learning: The educational legacy of Paul R. Pintrich. Educational Psychologist, 40, 85-94

cognition motivation

behaviour context

phases areas

cognition motivation

behaviour context

cognition motivation

behaviour context

cognition motivation

behaviour context

relevant research findings

• meaningfulness• transfer and domain specificity• deliberate practice• self-regulated learning• reasoning and decision making

1. Boreham NC. The dangerous practice of thinking. Medical Education 1994;28:172-79.2. Klein G. Naturalistic Decision Making. Human Factors 2008;50(3):456-60.3. Plous S. The psychology of judgment and decision making. New Jersey: McGraw-Hill inc., 1993.4. Schmidt HG, Machiels-Bongaerts M, Hermans H, ten Cate TJ, Venekamp R, Boshuizen HPA. The Development of Diagnostic Competence: Comparison of a Problem-based, and Integrated, and a Conventional Medical Curriculum. Academic Medicine 1996;71(6):658-64.

relevant research findings

• reliability• validity• quality frameworks• organisational reliability

1. Williams M, Klamen D, McGaghie W. Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine 2003;15(4):270-92.2. Kane MT. Validation. In: Brennan RL, editor. Educational Measurement. Westport: ACE/Praeger, 2006:17 - 64.3. Govaerts MJB. Climbing the pyramid; Towards understanding performance assessment. Maastricht University, 2011.4. Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93

CCA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

direct obs. direct obs. direct obs. direct obs.direct obs.

mcq test

mini-releases

MSF

portfolio

mini-releases mini-releases mini-releases mini-releases

term 1

mid-term end-term

CCA

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26

direct obs. direct obs. direct obs.

audit audit

mcq test

MSF

mcq test

mini-releases mini-releases mini-releases mini-releases mini-releases

term 2

mid-term

portfolio

end-term

critical case analysis

• 5 write-ups of real patient consultations• relevance• analysis• learning activities• produce exam questions (EMI, KFP, MCQ)• increasingly original literature• any discussion minuted by registrar

directly observed consultations

• 9 real patient consultations• relevance• analysis• learning goals (practical + theoretical)• learning activity• demonstration of success in next observed

consultation• discussion minuted by registrar

clinical audit

• analysis of the practice environment• determination of specific question• collection of data• draw conclusions• describe plan for change• + 3 months: look back and annotate• any discussion minuted by registrar

multiple-choice tests• 3 tests of 60 items each• blueprinted• sit and submit your answers• review items, answer key• comment on an criticise questions for correctness• present in min-release• ‘lodge’ appeal against questions• score calculation and feedback to registrars

mini-releases

• flexible agenda• building informal networks• discuss MCQ test items• compile

– appeal against questions– list of ‘informal’ network

multi-source feedback• 2 times per year• nurses, practice manager, receptionist, other

practice staff and registrar• discussed with supervisor (end-term

assessment) and with ME (minuted by registrar)• simple form: dealing with tasks, other and

yourself• simple ordinal scale• ample room for qualitative comments

mid and end-term assessment

• integrative• reviewing all the information• learning goals and/or remediation plans• advisory to performance review

committee• minuted by registrar

portfolio

• complete dossier including minutes• individual extra information (only if

relevant)• audit trail • basis for future CV or position

applications

example of a ‘line’

CCA

learning

MCQs

test

appeal

group

groupappeal

feedback

analysis

feedback

meaning

test enhanced learning

informal/social networks

transformationresearch: narrativesfor feedback

design

• goals and stated purpose• programme in action• supporting the programme• documentation of the programme• improvement approaches to of the

programme• accounting for the programme

Dijkstra J, Van der Vleuten C, Schuwirth L. A new framework for designing programmes of assessment. Advances in health sciences education 2010;15. :379–93.

rules and regulation

• self responsibility comes with accountability (minutes, plagiarism, fraud)

• focus on learning and remediation• information provision to the registrar• documentation• transparency• second opinion/appeals/switch of ME or

supervisor• organisation reliability/credibility

staff development

• efficiency:– short analyses– concrete learning goals– focus on learning– training of staff (expertise efficiency)– admin support by admin staff– division of advisory and decision roles

further requirements

• goals and stated purpose• programme in action• supporting the programme: regulations• documentation of the programme: ELO,

fact sheets, • improvement approaches to of the

programme: systematic evaluation• accounting for the programme: research

Thank you