from learning objectives to outcomes assessment during simulation paul e. phrampus director winter...
TRANSCRIPT
From Learning Objectives to Outcomes
Assessment During SimulationPaul E. Phrampus
DirectorWinter Institute for Simulation, Education and
Research (WISER)University of Pittsburgh
• Why• What• When• Where• How
Assessment
The Future of Simulation………
What is a simulation for?
1. Education2. Assessment3. Fun4. 1 and 2
TESTING TEACHING
The Simulation Advantage
Psychomotor SkillsPsychomotor Skills
Decision MakingDecision Making
Skills and DecisionsSkills and Decisions
Individuals Individuals Teams Teams
What is “Simulation”….. Really…..
The Job of Healthcare Educators
Psychomotor Skills
Communications Skills
Professionalism Skills
Decision MakingBase Knowledge
Teamwork Skills
Why is this so hard?
Learning Objective
• You have to have a clear reason of why you are running any simulation.
Lets Watch
• What is most important aspect to debrief?
Where does a simulation fit?
• It depends on what your doing!
EACH Sim is part of a bigger plan!
What is this?
Does it belong here?
Course (Global) objectives v sim objectives
Global Objectives
• Course Objectives– 1. Demonstrate a proper patient assessment– 2. Demonstrate following the emergency protocol– 3. Demonstrate Airway Management Skills
Course Schedule / Flow
• Sim 1• Sim 2• Lecture• Skills Lab• Sim 3• Sim 4
Course Objectives
Course Objectives
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
Session Objective1.Ventilate w BVM2.Recognize Narcotic Overdose3.Activate Protocol
How do you determine “success” of a simulation?
Students
liked...
Learn
ing objec...
Students
passe...
Not S
ure
25% 25%25%25%1. Students liked it2. Learning objectives
achieved3. Students passed the
scenario4. Not Sure
How do you determine “success” of a simulation?
Instructor Number 1
Instructor Number 2
Instructor Number 3
How do you determine “success” of a simulation?
Instructor Number 1 Instructor Number 2 Instructor Number 3
Measure Here
You can only keep track of so many things at one time
How many of you feel like this ?
Tools for Success
• Make it easy for your faculty to stay on track
• “In my opinion…..”
Assessment Methodology– Identify methods for measuring core aspects in
simulation education / assessment– Discuss development and deployment of
measurement tools
Student Learning
OutcomesCurriculum
Teaching & Learning Assessment Evaluation
Assessment and Evaluation
• Assessment and Evaluation often used interchangeably
• Assessment – learner outcomes• Evaluation – course or program outcomes
Performance Assessment
Rate the Performance
• Pass or Fail
• Score Performance
• Procedural Check
Assessment
• Why• What• When• Where• How
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim Healthcare 3:2008;33–41.
ACTION
PERFORMANCE
COMPETENCE
KNOWLEDGE
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
Performance Assessment
• Basic to performance – Do they know it and know how?
• Competence – Can they do it?
• Performance – Do they do it?
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
DOES
SHOWS HOW
KNOWS HOW
KNOWS
Knows When
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical skills/competence/performance
Knowledge Tests
MCQs, Essay, Oral exam
Clinical Based Tests
Pt Mgt Qs, Essay, Oral
Performance Tests
OSCE, TaskDemo, SIM/SP
Practice Review
Audits, Video, UC-SP, 360O
Assessment• Formative Assessment
– Lower stakes assessment– One of several assessments over time of course or program– May be evaluative, diagnostic, or prescriptive– Often results in remediation or progression to next level
• Summative Assessment– Higher stakes assessment– Generally final course or program assessment– Primary purpose is performance evaluation– Often results in a Go-No Go outcome
Assessment
• Assessors– Instructors– Observers– SIM/Patients– Peers– Participants
• Instruments– AV annotation– Pencil/Paper– Scantron form– TabletPC or PDA
• Assessment data sources– Live observation– Video recording– SIM device logs
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
Assessment Validity
• Validity – – Degree to which an assessment tool measures what it is
intended to measure– Construct validity
– Accuracy of tool to measure intended construct
• Concurrent validity• Content validity• Predicative validity
Assessment Reliability• Test Reliability –
– consistency of a test to yield the same measure over time (does not assure validity)
– Test-Retest Reliabilityconsistency among different administrations
– Parallel Forms Reliabilityto assure that memory effects do not occur use a different pre- and posttest
– Inter-Rater Reliability two or more observers rate the same subjects with strong positive correlation of their ratings
STUDENT
EVALUATOR
ASSESSMENTPATIENT
Variables in Assessment
Assessment Improvement
• Improving Reliability and Validity– Base assessment on outcome/objectives >
event triggers > observable behavior > behavioral rating > assess against competencies
– Define <low – medium – high> performance– Use of Rubric or rating metric– Use (video) training examples of performance– Employ a quality assurance/improvement system
Assessment Based on Objectives and Events
Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.
Event – Behavior – Scale - Report
Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare. Sim Healthcare 3:2008;33-41.
Assessment Metrics
• Procedural or Check List assessment• Global Rating assessment
Assessment Metrics• Procedural or Check List assessment
BCLS Y N
Open Airway
Check Breathing
BCLS Y N
Open Airway(< 5 sec of LOC)
Check Breathing(< 5 sec of Airway)
BCLS Y N
Open Airway
Check Breathing
A
Rating Score +1 -1 0 *Assist
Assessment Metrics• Global Rating assessment
Code Blue P F
CPR and
ACLS
Code Blue
CPR<1(low) - 5(Hi)> points
ACLS<1(low)- 5(Hi)> points
Code Blue H M
CPR
ACLS
L
Rating Score +1 -1 0
Pts.
Assessment Metrics• Procedural or Check List assessment• Global Rating assessment
< EXAMPLES>
• Pros and Cons to each• Multiple modes & points vs. Single assessment• QA/QI Metric – collective vs single assessment
Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics 121(3):2008;597-603.
Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings. Anesthesiology, 89(1)July 1998:8-18.
Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.
Interrater Reliability
Evaluator9
Evaluator8
Evaluator7
Evaluator6
Evaluator5
Evaluator4
Evaluator3
Evaluator2
Evaluator 1
5 ScenarioVideos
Novices
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
Andrus, Phrampus, Wang
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
0
0.2
0.4
0.6
0.8
1
1.2
9 Raters
6 Raters
3 rookies
Andrus, Phrampus, Wang
Interrater Reliability of Data Collected with SimMan During Difficult Airway Simulations
• CONCLUSIONS: – Simulation instructors exhibited good overall
inter-rater agreement but showed poor agreement for some key airway tasks.
– Variability in instructor ratings may influence real-time evaluation of simulation performance.
– Likely could be improved with careful instructor orientation / training
Andrus, Phrampus, Wang
Evaluation
• Individual Trainee Evaluation– Per episode performance– Course of the Day
Evaluation
• Group Trainee Evaluation– Benchmark
Evaluation
• Instructor Evaluation– Per episode performance– Measure Against Benchmarks
Testing Points
T I M E
Behind the Scenes
Behind the Scenes
Behind the Scenes
Behind the Scenes
Behind the Scenes
Summary
• Know why you are doing the sim• Each sim is part of a bigger picture• Design tools for consistency• Make it easy for your faculty!
Questions and discussion