Evaluating Clinical Evaluating Clinical SimulationsSimulations

Pamela R. Jeffries, DNS, RN, FAANJohns Hopkins University School of Nursing

Mississippi – February 25, 2011

Objectives of the Session:Objectives of the Session:

The participant will be able to:The participant will be able to:

Discuss the evaluation process using Discuss the evaluation process using simulationssimulations

Describe strategies using simulations as Describe strategies using simulations as an evaluation tool in courses or programsan evaluation tool in courses or programs

Describe different components that need Describe different components that need to be assessed when using simulationsto be assessed when using simulations

WHAT IS EVALUATION?WHAT IS EVALUATION?

Feedback

Coaching

Assigning Grades

Judgmental: objective or subjective

Form of quality improvement

Assessment

ASSESSMENTASSESSMENT

What it is:“Systematic collection, review and use of information about educational programs undertaken for the purpose of improving student learning and development.” (Palumba and Banta, 1999)

Focus: development and improvement rather than judgment and grading

WHY EVALUATE?WHY EVALUATE?

Determine learning outcomes and program goals achieved

Give feedback while learning Improve effectiveness of teaching

and learning Attain performance standards Assure patient safety

WHEN TO EVALUATE?WHEN TO EVALUATE?

1. Frequently when: Learning is complex There is a high risk of failure When the consequences of error would be serious

2. When outcomes are critical: Ensure that learners are prepared for clinical practices When findings can be used to alter the direction of the

project

3. End of the module, activity, or project to be certain of learning outcomes

EVALUATION PROCESSEVALUATION PROCESS

Is…A systematic effort involving identifying what, who, when and why, and then gathering, analyzing and interpreting the data

Concludes when…

the findings of the evaluation are reported and used

EVALUATION PROCESSEVALUATION PROCESS

Judgment: Relative worth or value of something

Focus: Specifically on the student learner in e-learning

Throughout educational process:Interwoven throughout the learning process usually in the form of feedback

EVALUATION INSTRUMENTEVALUATION INSTRUMENT

1.1. Norm-referencedNorm-referenced: :

Focus: how learners rank in comparison to each other

Outcome: The interpretation of the evaluation is to determine who has the most knowledge, best skill performance, etc. and who has the least

Evaluation InstrumentsEvaluation Instruments

2.2. Criterion-referenced Criterion-referenced

Focus: the learner’s ability to attain the objectives or competencies that have been specified

Purpose: to determine who has achieved the outcomes and who has not

Outcome: The outcome standard must be specified by the educator, and then the learner is evaluated to determine if the standard is met

PRINCIPLES OF EVALUATING PRINCIPLES OF EVALUATING ADULT LEARNERSADULT LEARNERS

Involve in planning Capable of self-evaluation Motivated to achieve Deal with real world problems and

need real world evaluation Like variety in teaching, learning,

and evaluation

PRINCIPLES OF EVALUATING PRINCIPLES OF EVALUATING ADULT LEARNERSADULT LEARNERS

Respond to feedback (evaluation) Need frequent and informative

feedback Are self-directed Learn from each other and can

evaluate each other Respond to choices…provide options

for evaluation

GUIDELINES FOR SELECTING GUIDELINES FOR SELECTING EVALUATION STRATEGIESEVALUATION STRATEGIES

Appropriate for the domain of learning The learner should have the opportunity to

practice in the way he/she will be evaluated

Used to provide multiple ways of assessing learning

Ease of development/use on the part of the educator

Evaluation instrument should be valid and reliable

Steps to Take When EvaluatingSteps to Take When Evaluating

1.1. Identify the purpose of the Identify the purpose of the evaluationevaluation

2.2. Determine a time frameDetermine a time frame3.3. Identify when to evaluateIdentify when to evaluate4.4. Develop the evaluation planDevelop the evaluation plan5.5. Select the instrument(s)Select the instrument(s)6.6. Collect the dataCollect the data7.7. Interpret the dataInterpret the data

Evaluations and SimulationsEvaluations and Simulations

Four areas of Evaluation:Four areas of Evaluation:

Evaluating the simulation itselfEvaluating the simulation itself Evaluating the Implementation phaseEvaluating the Implementation phase Evaluating student learning Evaluating student learning

outcomesoutcomes Using simulation as an evaluation Using simulation as an evaluation

tooltool

Evaluating the simulation (design)Evaluating the simulation (design)

A good simulation A good simulation design and design and development is development is needed to obtain needed to obtain the outcomes you the outcomes you needneed

The SDS provides a The SDS provides a measure of the measure of the importance of each importance of each design featuredesign feature

Data ResultsData Results

Simulation Design Scale Simulation Design Scale (SDS) analyzed using factor (SDS) analyzed using factor analyses with varimax analyses with varimax rotation on items for each rotation on items for each scalescale

Cronbach alphas calculated Cronbach alphas calculated on each subscale and the on each subscale and the overall scale to assessoverall scale to assess instrument reliabilityinstrument reliability

Simulation Design ScaleSimulation Design Scale

Information/ObjectivesInformation/Objectives 5 items5 items 0.87 alpha0.87 alpha

Student SupportStudent Support 4 items4 items 0.90 alpha0.90 alpha

Problem-SolvingProblem-Solving 5 items5 items 0.82 alpha0.82 alpha

Feedback/DebriefingFeedback/Debriefing 4 items4 items 0.89 alpha0.89 alpha

Fidelity (realism)Fidelity (realism) 2 items2 items 0.76 alpha0.76 alpha

Overall ScaleOverall Scale 20 items20 items 0.95 alpha0.95 alpha

Incorporating the Educational Incorporating the Educational Practices into the SimulationPractices into the Simulation Assemble the simulation with the Assemble the simulation with the

educational practices in mind when educational practices in mind when implementing the simulationimplementing the simulation

Active learningActive learning CollaborationCollaboration Diverse ways of learningDiverse ways of learning High expectationsHigh expectations

Data ResultsData Results

Educational Practices within Educational Practices within a Simulation Scale (EPSS) a Simulation Scale (EPSS) analyzed using factor analyzed using factor analyses with varimax analyses with varimax rotation on items for each rotation on items for each scalescale

Cronbach alphas calculated Cronbach alphas calculated on each subscale and the on each subscale and the overall scales to assess overall scales to assess instrument reliabilityinstrument reliability

Educational Practices ScaleEducational Practices Scale

Using factor analyses with varimax rotation on itemsUsing factor analyses with varimax rotation on items

for each scale, the analyses revealed 4 factorsfor each scale, the analyses revealed 4 factors

Active LearningActive Learning 10 items10 items 0.91 alpha0.91 alpha

CollaborationCollaboration 2 items2 items 0.92 alpha0.92 alpha

Diverse Ways of LearningDiverse Ways of Learning 2 items2 items 0.87 alpha0.87 alpha

High ExpectationsHigh Expectations 2 items2 items 0.83 alpha0.83 alpha

Overall ScaleOverall Scale 16 items16 items 0.92 alpha0.92 alpha

Instrument TestingInstrument Testing

Testing of the two instruments, Testing of the two instruments, Simulation Design Scale and Simulation Design Scale and Educational Practices Scale Educational Practices Scale continued in this Phase.continued in this Phase.

Reliabilities for both scales were Reliabilities for both scales were found to be good.found to be good.

It was important we found measures It was important we found measures to assess the quality of the designed to assess the quality of the designed simulations that were being used.simulations that were being used.

Evaluating Student Evaluating Student Learning OutcomesLearning Outcomes

Simulation technologies used to Simulation technologies used to measure both process and outcomes measure both process and outcomes range from case studies to range from case studies to standardized patients (OSCE), to task standardized patients (OSCE), to task trainers and high fidelity mannequinstrainers and high fidelity mannequins

Example of a Clinical Competency Measure to Measure Example of a Clinical Competency Measure to Measure a Curriculuar Threada Curriculuar Thread

(J. Cleary – Ridgewater, MN)(J. Cleary – Ridgewater, MN)

Critical competency

Performance Standard

Novice (1)

Accomplished (2)

Exemplary (3)

Selects appropriate channels of communication

Applies professional communication theories and techniques

Participates in an interdisciplinary team

Uses interdisciplinary resources to assist clients

Attempts to establish rapport

Generally establishes rapport

Written communication is generally accurate

Tentative to communicate to an interdisciplinary team

Consistently establishes rapport and uses therapeutic communication

Evaluating outcomesEvaluating outcomes

Formative evaluation measuresFormative evaluation measures: : simulation is used by the simulation is used by the learner/faculty to mark progress learner/faculty to mark progress toward a goaltoward a goal

Summative evaluation measuresSummative evaluation measures: : include determining course include determining course competencies, licensing and competencies, licensing and certification examinations, and certification examinations, and employment decisionsemployment decisions

Exemplars of student outcome Exemplars of student outcome measures used todaymeasures used today

KnowledgeKnowledge Skill performanceSkill performance Learner satisfactionLearner satisfaction Critical ThinkingCritical Thinking Self-confidenceSelf-confidence Skill proficiencySkill proficiency Teamwork/collaborationTeamwork/collaboration Problem-solving SkillsProblem-solving Skills

Using simulations as the Using simulations as the Evaluation ToolEvaluation Tool

When skill sets, When skill sets, clinical reasoning, clinical reasoning, and selected and selected clinical clinical competencies need competencies need to be measured, to be measured, simulations can be simulations can be used as the used as the mechanism to do mechanism to do this.this.

Simulations to EvaluateSimulations to Evaluate

Set-up a simulation Set-up a simulation as an evaluation as an evaluation activityactivity

Issues to address:Issues to address: Make sure student Make sure student

is aware it is an is aware it is an evaluationevaluation

Describe the Describe the evaluation metricsevaluation metrics

Is it objective?Is it objective?

Ways to Use Simulations as an Ways to Use Simulations as an Evaluation MetricEvaluation Metric

As an Objective As an Objective Structured Clinical Structured Clinical Exam (OSCE)Exam (OSCE)

Hired simulated Hired simulated patients are patients are portrayed by actors portrayed by actors or actressesor actresses

Students are Students are immersed into immersed into specific scenariosspecific scenarios

Ways to Use Simulations as an Ways to Use Simulations as an Evaluation MetricEvaluation Metric

Set-up a simulation Set-up a simulation to measure such to measure such concepts as concepts as teamwork, patient teamwork, patient safety safety competencies, competencies, ACLS protocol, ACLS protocol, communication communication skills, any selected skills, any selected interventionintervention

Ways to Use Simulations as an Ways to Use Simulations as an Evaluation MetricEvaluation Metric

Simulations used Simulations used for evaluation can for evaluation can come in the form come in the form of computer-based of computer-based learning, scores, learning, scores, and metrics to and metrics to demonstrate demonstrate knowledge, skill knowledge, skill competency, and competency, and proficiencyproficiency

Examples of products that can be Examples of products that can be used for evaluation purposes:used for evaluation purposes:

MicroSim DVD- scoring mechanismMicroSim DVD- scoring mechanism ACLS computer-based software ACLS computer-based software

scenario packages – scoring scenario packages – scoring Cath simulators – scoring devices – Cath simulators – scoring devices –

standardized scales, benchmarksstandardized scales, benchmarks CPR models/mannekins – CPR models/mannekins –

programming and scoringprogramming and scoring

SummarySummary

Simulations require evaluation of Simulations require evaluation of many variables, including the many variables, including the simulation design, the simulation design, the implementation process, and implementation process, and learning outcomeslearning outcomes

In addition, simulations can serve as In addition, simulations can serve as the mechanism to evaluate studentsthe mechanism to evaluate students

Nursing ImplicationsNursing Implications

When developing the course and When developing the course and curriculum planning in nursing, decide curriculum planning in nursing, decide what the purpose of the simulation what the purpose of the simulation encounters serve and evaluate to make encounters serve and evaluate to make sure the outcomes or purpose is being sure the outcomes or purpose is being achieved.achieved.

More evidence/documentation is needed More evidence/documentation is needed that simulations are serving the need for that simulations are serving the need for improved clinical performance, critical improved clinical performance, critical thinking, and improved diagnostic thinking, and improved diagnostic reasoning reasoning

ConclusionConclusion

““How to tell students what to look for How to tell students what to look for without telling them what to see is without telling them what to see is the dilemma of teaching.”the dilemma of teaching.”

Lascelles AbercrombieLascelles Abercrombie

Any Questions?Any Questions?

pjeffri2@son.jhmi.edu