from learning objectives to outcomes assessment during simulation paul e. phrampus director winter...

65
From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER) University of Pittsburgh

Upload: charles-ritchie

Post on 27-Mar-2015

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

From Learning Objectives to Outcomes

Assessment During SimulationPaul E. Phrampus

DirectorWinter Institute for Simulation, Education and

Research (WISER)University of Pittsburgh

Page 2: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

• Why• What• When• Where• How

Assessment

Page 3: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

The Future of Simulation………

Page 4: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

What is a simulation for?

1. Education2. Assessment3. Fun4. 1 and 2

Page 5: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

TESTING TEACHING

The Simulation Advantage

Psychomotor SkillsPsychomotor Skills

Decision MakingDecision Making

Skills and DecisionsSkills and Decisions

Individuals Individuals Teams Teams

Page 6: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

What is “Simulation”….. Really…..

Page 7: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

The Job of Healthcare Educators

Psychomotor Skills

Communications Skills

Professionalism Skills

Decision MakingBase Knowledge

Teamwork Skills

Page 8: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Why is this so hard?

Page 9: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Learning Objective

• You have to have a clear reason of why you are running any simulation.

Page 10: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Lets Watch

Page 11: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

• What is most important aspect to debrief?

Page 12: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Where does a simulation fit?

• It depends on what your doing!

Page 13: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

EACH Sim is part of a bigger plan!

Page 14: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

What is this?

Page 15: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Does it belong here?

Page 16: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Course (Global) objectives v sim objectives

Page 17: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Global Objectives

• Course Objectives– 1. Demonstrate a proper patient assessment– 2. Demonstrate following the emergency protocol– 3. Demonstrate Airway Management Skills

Page 18: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Course Schedule / Flow

• Sim 1• Sim 2• Lecture• Skills Lab• Sim 3• Sim 4

Course Objectives

Course Objectives

Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol

Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol

Page 19: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

How do you determine “success” of a simulation?

Students

liked...

Learn

ing objec...

Students

passe...

Not S

ure

25% 25%25%25%1. Students liked it2. Learning objectives

achieved3. Students passed the

scenario4. Not Sure

Page 20: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

How do you determine “success” of a simulation?

Instructor Number 1

Instructor Number 2

Instructor Number 3

Page 21: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

How do you determine “success” of a simulation?

Instructor Number 1 Instructor Number 2 Instructor Number 3

Measure Here

Page 22: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

You can only keep track of so many things at one time

Page 23: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

How many of you feel like this ?

Page 24: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Tools for Success

• Make it easy for your faculty to stay on track

• “In my opinion…..”

Page 25: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Methodology– Identify methods for measuring core aspects in

simulation education / assessment– Discuss development and deployment of

measurement tools

Page 26: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Student Learning

OutcomesCurriculum

Teaching & Learning Assessment Evaluation

Page 27: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment and Evaluation

• Assessment and Evaluation often used interchangeably

• Assessment – learner outcomes• Evaluation – course or program outcomes

Page 28: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Performance Assessment

Page 29: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Rate the Performance

• Pass or Fail

• Score Performance

• Procedural Check

Page 30: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment

• Why• What• When• Where• How

Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.

Page 31: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

ACTION

PERFORMANCE

COMPETENCE

KNOWLEDGE

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

Page 32: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Performance Assessment

• Basic to performance – Do they know it and know how?

• Competence – Can they do it?

• Performance – Do they do it?

Page 33: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

DOES

SHOWS HOW

KNOWS HOW

KNOWS

Knows When

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

Page 34: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

DOES

SHOWS HOW

KNOWS HOW

KNOWS

Knows When

Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.

Assessment of clinical skills/competence/performance

Knowledge Tests

MCQs, Essay, Oral exam

Clinical Based Tests

Pt Mgt Qs, Essay, Oral

Performance Tests

OSCE, TaskDemo, SIM/SP

Practice Review

Audits, Video, UC-SP, 360O

Page 35: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment• Formative Assessment

– Lower stakes assessment– One of several assessments over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level

• Summative Assessment– Higher stakes assessment– Generally final course or program assessment– Primary purpose is performance evaluation– Often results in a Go-No Go outcome

Page 36: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment

• Assessors– Instructors– Observers– SIM/Patients– Peers– Participants

• Instruments– AV annotation– Pencil/Paper– Scantron form– TabletPC or PDA

• Assessment data sources– Live observation– Video recording– SIM device logs

Page 37: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

STUDENT

EVALUATOR

ASSESSMENTPATIENT

Variables in Assessment

Page 38: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Validity

• Validity – – Degree to which an assessment tool measures what it is

intended to measure– Construct validity

– Accuracy of tool to measure intended construct

• Concurrent validity• Content validity• Predicative validity

Page 39: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Reliability• Test Reliability –

– consistency of a test to yield the same measure over time (does not assure validity)

– Test-Retest Reliabilityconsistency among different administrations

– Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest

– Inter-Rater Reliability two or more observers rate the same subjects with strong positive correlation of their ratings

Page 40: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

STUDENT

EVALUATOR

ASSESSMENTPATIENT

Variables in Assessment

Page 41: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Improvement

• Improving Reliability and Validity– Base assessment on outcome/objectives >

event triggers > observable behavior > behavioral rating > assess against competencies

– Define <low – medium – high> performance– Use of Rubric or rating metric– Use (video) training examples of performance– Employ a quality assurance/improvement system

Page 42: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Based on Objectives and Events

Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.

Page 43: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Event – Behavior – Scale - Report

Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.

Page 44: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Metrics

• Procedural or Check List assessment• Global Rating assessment

Page 45: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Metrics• Procedural or Check List assessment

BCLS Y N

Open Airway

Check Breathing

BCLS Y N

Open Airway(< 5 sec of LOC)

Check Breathing(< 5 sec of Airway)

BCLS Y N

Open Airway

Check Breathing

A

Rating Score +1 -1 0 *Assist

Page 46: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Metrics• Global Rating assessment

Code Blue P F

CPR and

ACLS

Code Blue

CPR<1(low) - 5(Hi)> points

ACLS<1(low)- 5(Hi)> points

Code Blue H M

CPR

ACLS

L

Rating Score +1 -1 0

Pts.

Page 47: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Assessment Metrics• Procedural or Check List assessment• Global Rating assessment

< EXAMPLES>

• Pros and Cons to each• Multiple modes & points vs. Single assessment• QA/QI Metric – collective vs single assessment

Page 48: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.

Page 49: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.

Page 50: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.

Page 51: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Interrater Reliability

Page 52: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Evaluator9

Evaluator8

Evaluator7

Evaluator6

Evaluator5

Evaluator4

Evaluator3

Evaluator2

Evaluator 1

5 ScenarioVideos

Novices

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

Andrus, Phrampus, Wang

Page 53: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

0

0.2

0.4

0.6

0.8

1

1.2

9 Raters

6 Raters

3 rookies

Andrus, Phrampus, Wang

Page 54: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations

• CONCLUSIONS: – Simulation instructors exhibited good overall

inter-rater agreement but showed poor agreement for some key airway tasks.

– Variability in instructor ratings may influence real-time evaluation of simulation performance.

– Likely could be improved with careful instructor orientation / training

Andrus, Phrampus, Wang

Page 55: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Evaluation

• Individual Trainee Evaluation– Per episode performance– Course of the Day

Page 56: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Evaluation

• Group Trainee Evaluation– Benchmark

Page 57: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Evaluation

• Instructor Evaluation– Per episode performance– Measure Against Benchmarks

Page 58: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Testing Points

T I M E

Page 59: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Behind the Scenes

Page 60: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Behind the Scenes

Page 61: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Behind the Scenes

Page 62: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Behind the Scenes

Page 63: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Behind the Scenes

Page 64: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Summary

• Know why you are doing the sim• Each sim is part of a bigger picture• Design tools for consistency• Make it easy for your faculty!

Page 65: From Learning Objectives to Outcomes Assessment During Simulation Paul E. Phrampus Director Winter Institute for Simulation, Education and Research (WISER)

Questions and discussion