Objective Structured Clinical Examination (OSCE)

Arnuparp Lekhakula M.D.,M.S.Faculty of Medicine

Prince of Songkla UniversityHat Yai, Songkhla

What is OSCE?

• Multiple stations, each testing different competencies or tasks

• Time limit for each station• Students rotate through all the station and

are tested on the same material• All are judges by the same preset

standards – usually using checklists or rating scales

Why OSCE?• Before OSCE (1975)• Valid?

Know how NOT show how• Reliable?

Different patients Different examiners• OSCE - more valid, more reliable• Show how• Same task / patient / examiner / structured

marking sheet

Advantages of OSCEs• Candidate’s knowledge and skills over

whole range of topics can be tested• Use of different test methods• Test not only knowledge and skills, but

attitudes as well• Efficient use of limited resources• More objective• Specific profiling of each component of

performance

Climbing the pyramid......

Knows

Shows how

Knows how

Does

Knows Factual tests: MCQ, essay type, oral…..

Knows how (Clinical) Context based tests:MCQ, essay type, oral…..

Shows how Performance assessment in vitro:OSCE, SP-based test…..

DoesPerformance assessment in vivo: Masked SPs, Video, Audits…..

Common Instruments Direct MCQ MEQ CRQ OSCE Observation

Knowledge +++ ++ +++ + +Problem-solving + ++ ++ + ++Practicalskill +++ ++Interpersonalskill ++ +++Attitudes + + + ++

Common Instruments Direct

MCQ MEQ CRQ OSCE observation

Validity Content +++ ++ +++ ++ +++ Face + +++ ++ +++ +++ Construct + +++ + + +++Reliability +++ ++ ++ +++ ++Objectivity +++ ++ ++ +++ ++Feasibility +++ + + + ++

Limitations of OSCEs• Labor-intensive, lengthy preparation• Costly• Require trainees to perform isolated

aspects of the clinical encounter• Rely on task specific checklists and

scoring• What can be simulated constrain the

type of patient problems used• Need more observation skills of staff

Limitations of OSCEs• Standardization of simulated patients

and examiners• Low inter-station correlation• Test security• Repetitive - boring

Factors Leading to Lower Reliability

• Too few stations or too little testing time

• Checklists or items that don’t discriminate

• Unreliable patients or inconsistent portrayals by standardized patients

• Examiners who score idiosyncratically• Administrative problems

What Assessed by OSCE• Various clinical skills – history taking,

physical examination, technical procedure (on model), communication, interpersonal skills

• Knowledge and understanding• Data interpretation• Problem solving• Attitudes

Types of OSCE stations

• Process - various skills, attitudes• Product – data interpretation, diagnosis,

problem solving, report writing, order sheet writing, drug description

• Mixed – process and product

Components of OSCE station

• Stem with clearly stated task• Checklist

- Appropriate for level of training - Task-based - Observable

• Training information

StemJohn Smith, aged 37, arrived in the emergency room Complaining of acute abdominal pain that began 12 hours previouslyInstruction to student In the next five minutes, conduct a relevant physical examination

Checklist

Examiner to fill in box to each item that student successfullycompletes

MarksDrapes patient appropriately 2Inspects abdomen 1Auscultation abdomen 1Percussion abdomen 1Asks patient for the location of pain 1Start to palpate abdomen from the

area which is not pain 2Lightly palpates each quadrant 2Deeply palpates each quadrant 2Checks for peritoneal irritation 2

Stem

Training Information

History of pain The pain started 12 hours ago

SymptomsThe pain is in the right lower quadrant, at “at least 9”,and is constant. His abdomen is tense even when palpatelightly. With deeper palpation, there is guarding in the RLQ,and McBurney point is acutely tender.Obturator (raising right knee against resistance) and psoassigns (extension of right leg at hip-kicking backward) arepositive.

Stem

Checklist

Steps in Developing OSCEs• Assembling working groups• Defining competencies/content assessed• Developing blueprint• Writing an OSCE station – process or

product - Writing a scenario - Develop instructions to the candidate:

precise, concise, unambiguous - Write instructions to the simulator/SPs,

a script for role-playing - Draw up a list of resources: space,

people, equipment, materials

Steps in Developing OSCEs• Developing checklist• Draw a scoring scheme• Revised by committee• Setting standards/passing score• Pilot testing• Reviewing and rewriting

Table of SpecificationArea tested

A1 A2 A3 A3 A5 A6 A7 A8 …..

Hx 3

PE 3

Procedures X X X X 4

Data interpret.

X X X X 4

Problem- solving

2

Com skill 1

Pt Ed 1

Attitudes 2

Total 2 3 3 2 3 3 2 2 20

Organizing an OSCE

• What is to be assessed – produce a table of specification

• Duration of stations – 4-15 minutes, mostly 5 minutes

• Number of stations – 15-20 stations• Format of stations – isolated, linked• Use of examiners – fully briefed prior to

examination• Preparation of checklists• Organization of examination

OSCE Station1 2 3 4

5

6789

10

1

2

2

3 410 min

1 2 3 4

Basic

Double-time

Linked

Organizing an OSCE

• Assigning priority• Resource requirement – examiners,

patients (real, simulated), equipment, paperwork

• Plan of examination and direction• Change signal• Records

How to Improve Checklist

• Stem - Clear - Observable - Not to long

• Overall - Not too long

• Rating scale - More clarification of each scale - Done, complete/Done, incomplete/

Not done - Rater training

Observation

• Direct• Indirect

- One-way mirror

- Monitor

- Video

Instructions to Simulated Patient• Situation• Information about illness• Role playing

- Facial/emotional/non-verbal expression - Response to students :

normal/abnormal - Questions asked to students

- Answers prepared to students’ questions

• Dressing/Costume• Make-up

Summary of OSCE stations

StationCompe-tency

Area tested

Linkedstation Process Product SP

Check-list

Exami-ner

Equip-ments

Timeline for OSCE• 8 wk Select date and appoint

overall coordinator and site coordinator

• 6 wk Decide on the station tasks, book site and refreshments, make local arrangements

• 3 wk Review all station details• 2 wk Have all station paperwork

printed, signs made, equipment prepared, remind examiner

Timeline for OSCE• 24 hr Walk through examination site

with coordinator• 2 hr All coordinator on-site• 1 hr Final briefing of examiners

and SPs• 30 min All examiners and SPs at

stations Students’ orientation/briefing

Standard Setting - Angoff Method• Performance levels at “minimally

acceptable” or “borderline” is used to derive cut-off score

• Undertaken by a panel (usually a minimal of 6)

• Judges independently estimate what proportion (%) of minimally competent examinees would answer the item correctly

Example - AngoffItems Full

markJudge Angoff

Score

Explain to patient 1 80% 0.8

Position of patient 1 80% 0.8

Inspection of skin 2 80% 1.6

Temperature change 1 60% 0.6

Pulsation : Femoral 1 80% 0.8

Popliteal 1 50% 0.5

Posterior tibial 1 50% 0.5

Dorsalis pedis 1 50% 0.5

Capillary refill 1 50% 0.5

Total 10 6.6

Example - AngoffItems Not

doneDone

poorlyDone statis.

Explain to patient 0 0.5 1.0

Position of patient 0 0.5 1.0

Inspection of skin 0 1.0 2.0

Temperature change 0 0.5 1.0

Pulsation : Femoral 0 0.5 1.0

Popliteal 0 0.5 1.0

Posterior tibial 0 0.5 1.0

Dorsalis pedis 0 0.5 1.0

Capillary refill 0 0.5 1.0

Maximum total = 10.0 Angoff score = 6.0

Standard Setting - Borderline Method

Stu 1 2 3 4 5 6 7 8 9

Score 80 90 60 70 75 55 80 65 85

Global rating

S SS BU BS S U S BS S

• Marking form : checklist + global rating• All categorized ‘borderline’ students

• Mean score of borderline group Passing score = (60+70+65)/3 = 65

Conclusions : OSCE• What? Stations + tasks + checklist• Why? More valid, more reliable, more objective• How?

How to organize?How to analyze?

OSCE – Task to Do Ahead (1)

• Create blue print• Set timeline • Get authors for case-writing workshop• Review and finalize case• Arrange workshop on setting standards• Recruit standardized patients/examiners• Train SPs• Print scoring sheet, mark signs

OSCE – Task to Do Ahead (2)

• List all supplies for set-up of OSCE stations• Remind everyone of date• Make sure students have all the

information• Plan of the examination day

- Diagram of station layout- Directions for examiners, SPs and staff- Registration table for examiners, SPs and students- Timing and signals (stopwatch, wristles or

bell)- Procedures for ending examination