Implementing Work-Based Implementing Work-Based AssessmentsAssessments

Professor T.MasudProfessor T.Masud

Nottingham University Hospitals Nottingham University Hospitals NHS TrustNHS Trust

ST1 ST2

Core Training

GIM (Acute) L1 Curriculum

F1 F2

FP Curriculum

ST3 ST4 ST5 ST6 ST7

Speciality Curriculum

GIM (Acute) L2 Curriculum

Generic C. L1 Generic Curriculum L2

CCT in Geriatric Medicine JRCPTB Cert. in GIM L2

Selection Allocation

MRCP Parts 1, 2 (written), PACES KBA (specialist exam)

WORK BASED ASSESSMENTS

Overview- Curricula and Assessments for Training

*Tooke report recommends separating F1 and F2 & joining F2 to ST1+2

Generic Curriculum - Generic Curriculum - builds on Foundation Curriculumbuilds on Foundation Curriculum(competencies categorised into Knowledge, Skills, Attitudes & Behaviour)(competencies categorised into Knowledge, Skills, Attitudes & Behaviour)

Level 1 (Mandatory CT Competencies)Level 1 (Mandatory CT Competencies)

– 1.1 History taking, examination, record keeping1.1 History taking, examination, record keeping

– 1.2 Time management and Decision Making skills1.2 Time management and Decision Making skills

– 1.3 Good Quality Care and Patient Safety1.3 Good Quality Care and Patient Safety

– 1.4 Infection Control1.4 Infection Control

– 1.6 Valid Consent1.6 Valid Consent

– 3.1 Communication wih patients within a consultation3.1 Communication wih patients within a consultation

– Focus Area 4 Working with ColleaguesFocus Area 4 Working with Colleagues

– Focus Area 6 Professional Behaviour Focus Area 6 Professional Behaviour

Generic Curriculum - Generic Curriculum - builds on Foundation Curriculumbuilds on Foundation Curriculum(competencies categorised into Knowledge, Skills, Attitudes & Behaviour)(competencies categorised into Knowledge, Skills, Attitudes & Behaviour)

Level 2 Competency AreasLevel 2 Competency Areas

– 1.5 Health Promotion and Public Health1.5 Health Promotion and Public Health

– 1.6i Medical Ethics and Public Health1.6i Medical Ethics and Public Health

– 1.6iii Legal framework for Practice1.6iii Legal framework for Practice

– 1.7 Ethical research1.7 Ethical research

– 1.8 Managing Long Term Conditions and Promoting Patient Self-Care1.8 Managing Long Term Conditions and Promoting Patient Self-Care

– Focus Area 2 Governance + Maintaining Good Clinical PracticeFocus Area 2 Governance + Maintaining Good Clinical Practice

– 3.2 Breaking Bad News3.2 Breaking Bad News

– 3.3 Complaints and Medical Error3.3 Complaints and Medical Error

– Focus Area 5 Teaching and TrainingFocus Area 5 Teaching and Training

– Focus Area 7 Management and NHS StructureFocus Area 7 Management and NHS Structure

Curriculum for GIM (Acute Medicine)– Levels1+2 Curriculum for GIM (Acute Medicine)– Levels1+2 (Categorised into levels 1 & 2 by Knowledge, Skills , Attitudes+Behaviour)(Categorised into levels 1 & 2 by Knowledge, Skills , Attitudes+Behaviour)

Emergency PresentationsEmergency Presentations– Cardio-respiratory arrestCardio-respiratory arrest

– Shocked patientShocked patient

– Unconscious patientUnconscious patient

– AnaphylaxisAnaphylaxis

Abdominal Pain Acute back pain Blackout/Collapse Breathlessness Chest pain Confusion- acute Cough Diarrhoea Falls Fever-

Fits/seizures Haematemesis/Melaena Headache Jaundice Limb pain/swelling Palpitations Poisoning Rash Vomiting /nausea Weakness/Paralysis

Top 20 Common Medical Presentations

Curriculum for GIM (Acute Medicine)– Levels1+2 Curriculum for GIM (Acute Medicine)– Levels1+2 (Categorised into levels 1 & 2 by Knowledge, Skills , Attitudes+Behaviour)(Categorised into levels 1 & 2 by Knowledge, Skills , Attitudes+Behaviour)

Abdo mass/hepatosplenomegaly Abdo swelling/constipation Abnormal sensation Aggressive/disturbed behaviour Alcohol/substance dependence Anxiety / panic behaviour Bruising Chance findings Dialysis Dyspepsia Dysuria Genital discharge & ulceration Haematuria Haemoptysis Head Injury Hoarseness and stridor Hypothermia Immobility Involuntary movements Joint swelling Lymphadenopathy

Other important presentations SYSTEM SPECIFIC COMPETENCIES

Loin pain Medical complics of acute illness/surgery Medical problems in pregnancy Memory loss Micturition (difficult) Neck pain Non-organic physical symptoms Polydipsia Polyuria Pruritis Rectal bleeding Skin and mouth ulcers Speech disturbances Suicidal ideation Swallowing difficulties Syncope and presyncope Unsteadiness /balance problems Visual disturbance Weight loss

Allergy Cancer & palliative care Cardiovascular medicine Clinical pharmacology Dermatology Diabetes and endocrinology Gastroenterology / hepatology Haematology Immunology Infectious diseases Medicine in the Elderly Musculoskeletal system Neurology Psychiatry Public Health / H.Promotion Renal medicine Respiratory medicine

Investigation Competencies Procedural Competencies

Geriatric Medicine ST Curriculum (Jan 2007) 1Geriatric Medicine ST Curriculum (Jan 2007) 1

Primary learning objectives (achieve competencies in:) Primary learning objectives (achieve competencies in:) [various [various settings]settings]

1.1. Perform Comprehensive Geriatric AssessmentPerform Comprehensive Geriatric Assessment

2.2. Diagnose and manage acute illness in old ageDiagnose and manage acute illness in old age

3.3. Diagnose and manage those with chronic disease and disabilityDiagnose and manage those with chronic disease and disability

4.4. Provide rehabilitation with the multidisciplinary team to an older patientProvide rehabilitation with the multidisciplinary team to an older patient

5.5. Plan the transfer of care of frail older patients from hospitalPlan the transfer of care of frail older patients from hospital

6.6. Assess a patient’s suitability for and provide appropriate care to those Assess a patient’s suitability for and provide appropriate care to those in long term (continuing) care in the NHS or communityin long term (continuing) care in the NHS or community

7.7. Apply knowledge and skills of a competent geriatrician in an Apply knowledge and skills of a competent geriatrician in an intermediate care and /or community settingintermediate care and /or community setting

Geriatric Medicine ST Curriculum (Jan 2007) 2Geriatric Medicine ST Curriculum (Jan 2007) 2

8. Assess and manage older patients presenting with the common geriatric 8. Assess and manage older patients presenting with the common geriatric problems (syndromes):problems (syndromes):

a. Falls- with and wihout fracturea. Falls- with and wihout fracture b. Deliriumb. Delirium c. Incontinencec. Incontinence d. Poor mobilityd. Poor mobility

9. Demonstrate competence in following subspecialities:9. Demonstrate competence in following subspecialities: a. Palliative carea. Palliative care b. Orthogeriatricsb. Orthogeriatrics c. Old Age Psychiatryc. Old Age Psychiatry d. Specialist Stroke Cared. Specialist Stroke Care

10. Be competent in 10. Be competent in a. Research methodology (basic)a. Research methodology (basic) b. Ethical principles of researchb. Ethical principles of research c. Critical appraisal of medical literaturec. Critical appraisal of medical literature d. Preferably to have personal experience of researchd. Preferably to have personal experience of research [basic science or clinical (health service)][basic science or clinical (health service)]

Assessment Plan in Geriatric Medicine - OverviewAssessment Plan in Geriatric Medicine - Overview

MRCP PACESMRCP PACES

KBA (SE)KBA (SE)

Work Based AssessmentsWork Based Assessments– Mini-CEX, DOPS, ACAT, CbD, MSF, PSMini-CEX, DOPS, ACAT, CbD, MSF, PS

Assessment of Generic Areas Assessment of Generic Areas – Research (portfolio, supervisor reports, publications)Research (portfolio, supervisor reports, publications)– Audit (portfolio, audit reports, supervisors reports)Audit (portfolio, audit reports, supervisors reports)– Clinical Governance (portfolio, supervisor reports)Clinical Governance (portfolio, supervisor reports)– Teaching (portfolio, assessed teaching) Teaching (portfolio, assessed teaching)

Regular Appraisal by Educational Supervisor Regular Appraisal by Educational Supervisor (including Educational Supervisor Reports)(including Educational Supervisor Reports)

Training RecordTraining Record– educational supervisor + consulatant trainer reportseducational supervisor + consulatant trainer reports– completed + signed work based assessmentscompleted + signed work based assessments– reflective learningreflective learning– course certificates, audit reports, publications etccourse certificates, audit reports, publications etc

Work Based Assessment MethodsWork Based Assessment Methods

Mini-Clinical Evaluation Exercise (mini-CEX)Mini-Clinical Evaluation Exercise (mini-CEX)

Direct Observation of Procedural Skills (DOPS)Direct Observation of Procedural Skills (DOPS)

Multi-Source Feedback (MSF)Multi-Source Feedback (MSF)

Case-Based Discussions (CbD)Case-Based Discussions (CbD)

Patient Survey (PS)Patient Survey (PS)

Acute Care Assessment Tool (ACAT)Acute Care Assessment Tool (ACAT)

Mini-CEX (ST3-ST7) [n=25]Mini-CEX (ST3-ST7) [n=25]

4 Acute (1 ward round)

2 Rehab (1 ward round)

1 Pre-op Orthogeriatric

1 Post-op Orthogeriatric

2 Chronic disease (clinic eg DM, OA)

1 MDT chair (discharge)

1 Continence

1 Falls

1 Movement disorder

1 Delirium / Depression

1 Old Age Psychiatry (HV / Ward referral)

1 Osteoporosis / metabolic bone disease

1 Comprehensive Geriatric Assessment

1 Intermediate care / Home visit

1 Continuing care

1 Day Hospital

3 Stroke (acute WR, Rehab WR, TIA clinic

1 Palliative care (break bad news)

CbD (ST3-ST7) [n=26]CbD (ST3-ST7) [n=26]

4 Acute (diagnosis, Mx, prescribing)

2 Rehab

2 Continuing care (≥ 1 non-NHS)

1 Evidence Based Medicine

1 Ethics / Law

1 Health Promotion

1 Complaint

1 Intermediate Care

1 Transfer of Care problem

1 Delirium

1 Old Age Psychiatry

1 Depression / dementia

1 Falls

1 Continence

1 Orthogeriatric – acute

1 Orthogeriatric - rehab

1 Acute Stroke

1 Rehab Stroke

1 Neurovascular investigation (TIA)

1 Palliative care

1 Tissue Viability / Hypothermia

Mid-Trent Experience of MSF (360 degree appraisals for SPRS)

Overview of Earlier PilotOverview of Earlier Pilot

• Early pilot June-August 2004

• 18 SPRs in 6 NHS Trusts

• Then 13 SPRs (1 abroad, 1 maternity leave)

• 11 SPRs performed 360 degree appraisals

• Minimal guidance

• Experience discussed at RITAs Sept 2004 (subjective)

• Questionnaire sent to SPRs in Jan 2005 (objective)

Who decided which people the form should Who decided which people the form should be sent to?be sent to?

0

1

2

3

4

5

6

SPR SEC ED SUP OTHER

No.

Ward Manager

Did you find the exercise useful?Did you find the exercise useful?

Yes- extremely usefulYes- extremely useful 11

Yes- quite usefulYes- quite useful 44

Not sureNot sure 33

No- not that usefulNo- not that useful 22

No- waste of timeNo- waste of time 11

Comments: Those who found process Comments: Those who found process usefuluseful

• Gave useful feedback on how I am performing on the ward

• I think it is a great way of assessing clinicians

• Getting positive feedback improved my confidence.

• The process informed the appraisal meeting with my

educational supervisor

Comments: Those who found process Comments: Those who found process notnot usefuluseful

• Other people should distribute the forms so the SPR does not know who has been asked to complete them (x2)

• Meetings with the educational supervisor more important than the 360 degree appraisal

Further analysisFurther analysis

• Of the 5 SPRs who found the process useful, for 3 of them “others” decided who to send the forms to.

• Of the 6 SPRS who who did not find the process useful or were not sure, all 6 SPRs chose the people to send the forms to

Conclusion of the Early pilotConclusion of the Early pilot

Main hypothesis generated:

The process is more useful if educational supervisor / consultant trainer decides to whom the questionnaires are sent to.

Randomisation- by TrustRandomisation- by Trust

Randomised to

SPRs decides (n=8)

ES/Con decides

(n=7)

3 Trusts 3 Trusts

Questionnaire sent to SPRs and ESs after the RITAs in Oct 2005

Mean no of Qs sent 17.8 (range 10-20)Mean no of Qs sent 17.8 (range 10-20)

Mean returns 15.1 (range 10-20)Mean returns 15.1 (range 10-20)

Mean response rate 85% (range 55%-Mean response rate 85% (range 55%-100%)100%)

No difference between the 2 groupsNo difference between the 2 groups

Did you find the process useful?Did you find the process useful?

TotalTotal SPR SPR DecidesDecides

ES/Con ES/Con DecidesDecides

Yes- extremelyYes- extremely 22 00 22

Yes- quiteYes- quite 99 44 55

Not sureNot sure 22 22 00

No- not that No- not that useful useful

22 22 00

No- waste of No- waste of timetime

00 00 00

“50% useful” “100% useful”

[Chi sq 4.77, df 1, p=0.029; Fishers p=0.051]

Mini-CEXMini-CEX

12 SPRs had performed at least 1 Mini-CEX12 SPRs had performed at least 1 Mini-CEX

9 found them useful, 3 not9 found them useful, 3 not

Egs - Ward round (x4); MDM (x2); Tilt (x1) Egs - Ward round (x4); MDM (x2); Tilt (x1) - Rest not stated - Rest not stated

DOPS + Patient Satisfaction QsDOPS + Patient Satisfaction Qs

4 SPRs had performed at least 1 DOPS4 SPRs had performed at least 1 DOPS

3 found them useful (Temp pacing, OGD, not stated)3 found them useful (Temp pacing, OGD, not stated)

1 not useful (LP)1 not useful (LP)

1 SPR performed a patient satisfaction Q- useful1 SPR performed a patient satisfaction Q- useful

Feedback from Educational supervisors Feedback from Educational supervisors (5 returns from 7) (5 returns from 7)

2 found the 360 degree process extremely 2 found the 360 degree process extremely usefuluseful

2 found it quite useful2 found it quite useful

1 not sure1 not sure

Feedback from Educational supervisorsFeedback from Educational supervisors

Was the process time-consuming?

No- 4

Yes–slightly 1

Yes – very 0

Assessment Plan For Speciality Training in Assessment Plan For Speciality Training in Geriatric Medicine Geriatric Medicine

MRCPMRCP KBEKBE(SE)(SE)

Mini-Mini-CEXCEX

CbDCbD MSFMSF ACATACAT PSPS ES+CTES+CTReportsReports

Audit / Audit / ResearchResearch

Teach Teach AssesAsses

CertCert

ST3ST3 ** 6 6 AcuteAcute

6 6 AcuteAcute

11 ** ALSALS

ST4ST4 66 66 ** 11 ** ** Acute Acute Audit Audit

ALS ALS Res. Res. Meth.Meth.

ST5ST5 ** 66 66 11 ** Audit Audit (eg rehab)(eg rehab)

** ALS ALS TeachTeach

ST6ST6 66 66 ** ** * * PYA PYA

Audit (eg IC) Audit (eg IC) Res. present.Res. present.

ALSALS

ST7ST7 11 22 11 ** PublicationPublication ALS ALS Manag.Manag.

TOTTOT 25 25 2626 22 44 22

Summative Formative

EducationalAppraisal

Annual Review of

Competence Progression

Outcome

Annual Planning

1. Learning agreement:aims & intended learning outcomesbased on specialty curriculum2. Advice on portfolio3. Regular feedback (2 way)4. Personal Development Plan5. Trainer’s structured report

6. Workplace based (NHS) appraisal

Educational supervisor and/or TPD meet with trainee toreview competence outcome with trainee plan next part of training

Based on a paper from PMETB’s Workplace Assessment Group (2005)

Workplace based (NHS)

appraisal

ARCPA. Evidence1 Assessment of performance e.g.workplace based assessments and observational methods E.g. mini-CEX, DOPS, video, CBDexaminationsstructured report

2. Assessment of experience, e.g.portfolio/log bookauditresearchcritical incidents

B. Annual Competence Review Appropriately constituted panel considers evidence Outcome of review

Role of the ARCP

ChallengesChallenges

Overlap between SpRs and StRsOverlap between SpRs and StRs

StRs – proactiveStRs – proactive

Educational supervisors – proactiveEducational supervisors – proactive

Consultant Trainers – proactiveConsultant Trainers – proactive

New documentation – structuredNew documentation – structured

Time in Job plansTime in Job plans

Useful process or tick-box exercise?Useful process or tick-box exercise?

Some Suggestions

• StRs and CTs to have forms for CbDs, Mini-CEX

handy

• 2 CbDs and 2 Mini-CEX per 4 month attachment

• 3 CbDs and 3 Mini-CEX per 6 month attachment

• Importance of Meeting with Education Supervisor

about 1 month before ARCP

• Regular half day / full day teaching sessions

mapped to curricula

PartnershipPartnership

StR

Educational Supervisor TPDs

Consultant Trainer

Specialist Training Committee

Deanery

BGS ETC SAC JRCPTB PMETB

NHS Trust

Regional Advisor

Assessments! Assessments!More Work Based Assessments