methods of evaluating simulation experiences 1 of... · impact on student learning. ... methods of...
TRANSCRIPT
The Henderson Repository is a free resource of the HonorSociety of Nursing, Sigma Theta Tau International. It isdedicated to the dissemination of nursing research, research-related, and evidence-based nursing materials. Take credit for allyour work, not just books and journal articles. To learn more,visit www.nursingrepository.org
Item type Article
Title Methods of Evaluating Simulation Experiences in NursingPrograms: An Integrative Review
Authors Paik Page, Michelle
Citation Paik Page, M. (2014, March 17). Methods of evaluatingsimulation experiences in nursing programs: An integrativereview. Virginia Henderson Global Nursing e-Repository.Retrieved fromhttp://www.nursinglibrary.org/vhl/handle/10755/346805
Downloaded 9-Jul-2018 08:50:01
Item License http://creativecommons.org/licenses/by/4.0/
Link to item http://hdl.handle.net/10755/346805
METHODS OF EVALUATING SIMULATION EXPERIENCES 1
Methods of Evaluating Simulation Experiences in Nursing Programs: An Integrative Review
Michelle Paik Page MS RN
METHODS OF EVALUATING SIMULATION EXPERIENCES 2
Abstract
Evaluation of simulation activities in nursing programs is a continuing process and different
methods have been applied in order to capture the effectiveness. Most of these methods are
subjective in nature and incorporate rubrics and qualitative feedback data of both students and
faculty. As simulation becomes a more utilized teaching method in nursing programs, evaluation
tools will be necessary to measure outcomes and success of the simulation implementation. The
varying methods currently used are not standardized and inter-rater reliability comes into
question when using these tools for evaluation of students in the simulation setting. Tools must
be refined and work must be done to standardize these in order for educators to quantify the
impact on student learning. The aim of this review was to synthesize the varying techniques and
rubrics identified in the literature used to measure simulation experiences and discuss its impact
on future development of a more standardized tool. Twelve articles were reviewed that identified
rubrics to measure and evaluate outcomes and perceptions of simulation experiences. Analysis of
the results revealed that there is no universal standardized tool used to measure simulation in
nursing programs. Evaluation rubrics do not have inter-rater reliability across educator users and
programs. Although the literature suggests a strong positive correlation between the ability of
simulation to produce effective learning, the measurement of the significant impact of that
learning cannot be quantified. Although a quantitative tool for measurement of simulation does
not exist it seems that the current rubrics being used in the simulation arena are becoming more
comprehensive and present similar determinants that measure these experiences more
effectively. Comparative and longitudinal studies will be needed with the current and developing
rubrics across varying curricula and simulation programs to produce a reliable and valid tool to
METHODS OF EVALUATING SIMULATION EXPERIENCES 3
measure the effect of simulation. Simulation standardization may first have to be achieved prior
to a standardized tool to measure it.
Keywords: simulation, nursing, evaluation
METHODS OF EVALUATING SIMULATION EXPERIENCES 4
Methods of Evaluating Simulation Experiences in Nursing Programs: An Integrative Review
“Human patient simulation (HPS) is becoming ubiquitous in nursing education as a
supplement or alternative to traditional clinical education. In order to develop evidence about the
efficacy of HPS as a teaching strategy, nurse educators must adequately evaluate student learning
from HPS activities. One major barrier to the accurate evaluation of learning outcomes is a lack
of evaluation instruments that allow nurse educators to make valid and reliable evaluations of
student performance in HPS activities (Adamson & Kardong-Edgren, 2012).” Because of the
current faculty shortage and limited clinical placement facilities more simulation activities will
need to be implemented in nursing programs to provide students with practical skill acquisition
but in order to replace the traditional clinical experience with simulated experience educators
must be able to evaluate that simulated experiences provide similar outcomes (Schlairet &
Pollock, 2010). Although simulation is widely used throughout the nursing curricula it has not
replaced clinical experiences because educators are hesitant, without evidence, sufficient
evaluative methods and adequate technology training, to justify simulation as a scientifically
effective teaching strategy (Schlairet & Pollock, 2010). The literature is robust in supporting
positive faculty and student perspectives related to implementation of simulation and its impact
on outcomes among varying nursing programs but inversely so in the ability to describe adequate
research to support strong simulation evaluation rubric measurements to ensure reliability and
validity in these tools being used (Schlairet, 2011). Simulation instruments have been designed
by organizations and individuals representing various nursing programs. These tools have been
studied in the simulation environments. The seven tools discovered in the literature developed by
organizations, universities and individual researchers include the Lasater Clinical Judgement
Rubric (LCJR), the Clark Simulation Evaluation Rubric (CSER), the Seattle University
METHODS OF EVALUATING SIMULATION EXPERIENCES 5
Evaluation Tool (SUET), the Creighton-Simulation Evaluation Instrument (C-SEI), the
Sweeney-Clark Simulation Performance Rubric (SCSPR), the QSEN-based Simulation and
Evaluation Rubric (QSER) developed by faculy at Colorado Mountain Community College, and
the Objective Structured Clinical Examination (OSCE). This literature review will analyze these
tools used to study simulation outcomes and hopefully provide discussion of further research on
standardization of simulation evaluation.
Methodology
The databases of CINAHL, OmniFile, Academic Search Premier, and Health Source:
Nursing/Academic Edition were searched for literature within a ten year time frame including
search terms of “simulation”, “nursing”, and “evaluation” in a combination query. Criteria for
the search were further limited to peer-reviewed, English language, and research articles that
yielded 531 results. An abstract review was conducted to search for relevant sources. An
additional ancestor and hand search was conducted to seek specific articles related to evaluation
tools used for simulation. From this search an additional four articles were identified. Articles
that did not identify specific evaluation tools used and studied along with simulation activities
were excluded. After further reviewing the articles, studies were excluded if the results discussed
evaluation criteria on student reflection or perception of a simulation experience. Faculty
reflection and perception interview articles were also excluded from those chosen for the
literature review. 28 articles were examined. Eleven articles were chosen for the literature review
(see Appendix A for Prisma diagram). The articles were then reviewed for elemental themes and
a chart showing a comparison of included themes for each evaluation tool was prepared (see
Appendix B for theme evaluation tool comparison chart). The results were discussed using the
METHODS OF EVALUATING SIMULATION EXPERIENCES 6
themes identified in the review. The evaluation tools that showed the highest assessment for a
theme were highlighted within that theme.
Results
The design of the simulation evaluation tools were very different among those studied but
common themes for evaluation of students emerged from those tools. Suggestions for those
commonalities will be discussed later. The emerging themes used in simulation evaluation tools
included patient-centered care and assessment, evidence-based nursing interventions and clinical
judgment, communication and teamwork, and safety. The synthesized data uncovered that nurse
educators are attempting to identify similar attributes of quality among the nursing student
population in all the tools used for evaluation but are scoring those attributes with different
scales or using various methods to hopefully discover and pass competent new providers. These
studies were not done among the same level across nursing programs and it is well know that
simulation arenas and equipment can be significantly different from program to program. The
results below are grouped into the themes extracted from the literature review. The tools that
represented the strongest points and assessment for the theme listed are discussed within that
theme below.
Patient-centered Care and Assessment
Every evaluation tool reviewed incorporated the element of patient-centered care and
assessment. It is well established that practicing nurses must have a high level of assessment
skills in order to correctly intervene with appropriate interventions for the best patient outcomes.
Luetke & Bembenek (2012) of Colorado Mountain College identified competency areas that
encompassed all themes of the literature review. Their study developed a rubric that defined
expected skill achievements with a write in for faculty to list an example of those skills as
METHODS OF EVALUATING SIMULATION EXPERIENCES 7
observed within the simulation activity (Luetke & Bembenek, 2012). The rubric score for
patient-centered care and assessment weighed the heaviest in the grading process. Prior to
developing their tool, they identified current rubric’s used in the field of simulation and
integrated Quality and Safety Education for Nurse (QSEN) competencies into the evaluation
language (Luetke & Bembenek, 2012). Using a convenience sample of 19 first year students, a
high-fidelity simulation was implemented and evaluated using the Sweeney-Clark Simulation
Performance Rubric (SCSPR) and the newly faculty developed QSEN-based Simulation and
Evaluation Rubric (QSER). The SCSPR, which measured simulation experience using Benner’s
theory of novice to expert, was identified by this faculty cohort as the comparitive model in
which they wanted to base their study on (Luetke & Bembenek, 2012). The findings according to
Luetke & Bembenek, (2012) indicated that:
“Students within the study group have a strong need for an evaluation rubric to be
used as part of simulations to provide for structured feedback and to be used as a
guide for expected learning outcomes (Luetke & Bembenek, 2012).”
“The results also demonstrate that debriefing and verbal feedback are essential
components to students learning experiences in a post simulation discussion
session (Luetke & Bembenek, 2012).”
“Students stated that self evaluation using the Sweeney Clark rubric was
challenging and adversely impacted their perceived nursing self confidence
(Luetke & Bembenek, 2012).”
“In the Sweeney Clark rubric all faculty using the rubric need to have a consensus
of what behaviors define each level in order to appropriately evaluate students
(Luetke & Bembenek, 2012).”
METHODS OF EVALUATING SIMULATION EXPERIENCES 8
“The benefit of the QSEN rubric is that it provides additional areas of teaching
outside of standard simulations including the ability to add evidence of a quality
improvement project and cultural assessment (Luetke & Bembenek, 2012).”
“The QSEN rubric allows for faculty consensus and consistency in expected
simulation learning outcomes (Luetke & Bembenek, 2012).”
Evidence-based Nursing Interventions and Clinical Judgment
The Lasater Clinical Judgment Rubric (LCJR) as described in Ashcraft, et al. (2013)
evaluated senior nursing students in a high-fidelity simulator lab equipped with hospital
equipment and computerized charting via a laptop. The scenarios were used to evaluate clinical
judgment integrating safety and effective use of assessment for each scenario. This two year
study included the Bachelor of Science in Nursing (BSN) students in their final semester prior to
graduation (Ashcraft, et al., 2013). The descriptive study’s design included role assignment, pre-
simulation study of pathophysiology, post-performance reflection and debriefing activities
(Ashcraft, et al., 2013). 86 senior nursing students participated in the first year of the study and
102 students in the second year (Ashcraft, et al., 2013). Students were consented and evaluated
formatively in the first half of the semester and summatively in the final half of the semester.
Randomization of simulation groups and diagnosis for patients took place initially (Ashcraft, et
al., 2013). Objectives for each group was consistent. Each group received 20 minutes from the
introduction of the patient via a report and the Lasater Clinical Judgement Rubric was used to
score student performance (Ashcraft, et al., 2013). The rubric was assigned values for each
section that ranged from 0-3 in the first year and then modified to a point scale of 0-11 for the
second year group to avoid negative scoring (Ashcraft, et al., 2013). Student clinical scores based
on the rubric improved for both groups between the formative and summative evaluations in
METHODS OF EVALUATING SIMULATION EXPERIENCES 9
clinical dimension and judgment areas (Ashcraft, et al., 2013). Inter-rater reliability was reported
at a high-level after a Cronbach’s alpha testing used to rate evaluator error (Ashcraft, et al.,
2013). One of the limitations of the study reported dependency of instructor interpretation of
rubric descriptors (Ashcraft, et al., 2013). Dimensions measured by this rubric include effective
noticing, effective interpreting, effective responding, and effective reflecting (Dillard, et al.,
2009).
Communication and Teamwork
Studies using the Clark Simulation Evaluation Rubric (CSER), the Seattle University
Evaluation Tool (SUET), and the Creighton-Simulation Evaluation Instrument (C-SEI) speak to
all themes of this review but are good examples of the communication and teamwork evaluation
strength. In a study done by Adamson & Kardong-Edgren (2012) video archived simulations
were rated using the SUET, the C-SEI, and the LCJR. A review board at Washington State
University approved the use of the videos via an exemption certificate regarding the use of
human subjects (Adamson & Kardong-Edgren, 2012). This study focused on the inter-rater
reliability of the evaluation tools (Adamson & Kardong-Edgren, 2012). Nurse faculty
participants were solicited to be evaluators and met criteria as follows:
“Currently teaching in an accredited, pre-licensure, baccalaureate nursing
program in the United States (Adamson & Kardong-Edgren, 2012).”
“A minimum of one year’s experience using HPS in pre-licensure, baccalaureate
nursing education (Adamson & Kardong-Edgren, 2012).”
“Clinical teaching or practice experience in an acute care setting as an RN during
the past 10 years (Adamson & Kardong-Edgren, 2012).”
METHODS OF EVALUATING SIMULATION EXPERIENCES 10
The faculty sample participated in orientation meetings to become familiar with the
evaluative tools and study particulars (Adamson & Kardong-Edgren, 2012). The participants
were sent six video-taped National League for Nursing (NLN) simulation videos and were
instructed to watch one video at a time and use the evaluative tools to rate each one (Adamson &
Kardong-Edgren, 2012). Over a period of weeks this process was repeated in order to validate
the reliability of the tools provided (Adamson & Kardong-Edgren, 2012). “Descriptive statistics,
including means and standard deviations of scores assigned to each of the scenarios using each
of the instruments, were calculated using all of the data (Adamson & Kardong-Edgren, 2012).”
Inter-rater reliability, inter-instrument reliability, test-retest reliability, and internal consistency
were measured using various statistical methods. Inter-rater reliability measured 0.858-0.952
(Adamson & Kardong-Edgren, 2012). Test-retest reliability measured 0.849-0.910 with the
LCJR showing the highest reliability in this study.
In the study published by Gantt (2010) two cohorts of students were evaluated using the
Clark Simulation Evaluation Rubric (CSER) to study elements of the rubric and its reliability of
use. Prior to the studies faculty had to meet and discuss the rubric and what determined a passing
score within each measure and therefore the researchers did not focus on inter-rater reliability as
the aim for the study (Gantt, 2010). Gant (2010) discussed the need for standard patient
simulation in order for grading of performance domains to be more objective as it was noted that
rating was quite subjective and domains overlapped which led to the question of checklist use
versus rubric use for simulation. Performance measures included the themes identified in the
review and defined objectives for each domain in a range of not being able to see the whole
picture to anticipation of a changing picture at the highest level of the evaluation in each section
(Gantt, 2010). Consistent grading practices could only be achieved on 10 of the student
METHODS OF EVALUATING SIMULATION EXPERIENCES 11
participants only after review and discussion of the faculty took place regarding the same
scenario (Gantt, 2010). The study discusses more work needed for evaluative tool to be used
consistently among raters (Gantt, 2010).
Safety
According to Alinier, Hunt, Gordon, & Harwood (2006) the OSCE is a reliable and valid
method to evaluate simulation activities because the design of the evaluation content can be
marked by someone without skill or knowledge in the area and still reliably evaluate
performance of participants within the skill simulation stations. “The OSCE included four
theoretical stations with questions on safety and nursing practice (Alinier, Hunt, Gordon, &
Harwood, 2006).” The highlight of safety in this study where 4 out of 15 testing stations were
focused on safety denotes an importance of this element as part of the evaluation. 99 students
participated as part of a control and experimental group where an OSCE was given pre and post-
simulation for both groups (Alinier, Hunt, Gordon, & Harwood, 2006). The experimental group
was exposed to the simulation activities and the control group followed a traditional clinical
pathway. “A pretest/post-test design was employed with volunteer undergraduate students from
second year Dimploma of Higher education in Nursing prgramme in United Kingdom using a
15-station Objective Structured Clinical Examination (Alinier, Hunt, Gordon, & Harwood,
2006).” This was a two year study using high-fidelity HPS. The results of the comparison in the
pre and post simulation tests revealed a significant statistical increase in improvement in students
performance on the OSCE after exposure to simulation when compared to those in the traditional
clinical exposure (Alinier, Hunt, Gordon, & Harwood, 2006). Although this evaluation tool
covered other elemental themes, its specific focus on patient safety was noted.
METHODS OF EVALUATING SIMULATION EXPERIENCES 12
Discussion
This review reveals several examples of evaluative tools that are used in the simulation
arena and its varying strengths including the themes of patient-centered care and assessment,
evidence-based nursing and clinical judgment, communication and teamwork, and safety. The
varied strengths of each tool discussed include: reliability to assess simulation teaching methods,
inter-rater ability to capture non-standard scenarios, and the ultimate use of standard scenarios
across nursing programs in order to measure outcomes more scientifically. It is not clear whether
one tool can capture all elements needed for practice or even if standardization of patient
simulation scenarios would improve the reliability of these tools, but it is clear that some rubrics
are more concise and are stronger at capturing certain elements of practice. The literature review
does suggest that evaluation is 1) an important part of the simulation process 2) was able to
strengthen simulation programs and 3) facilitate discussion among faculty regarding common
denominators of objectives that educators are seeking from student performance in simulation.
The relationship between which evaluation tool remains the best for use among those out there
remains unanswered and more studies are needed to improve or develop a more comprehensive
or standardized tool for evaluation of simulation.
Conclusion and Recommendations
There are tools that are being used to measure simulation and evaluate its effectiveness as
discussed in this review. The tools being used are few but appear to be well developed using the
values of assessment, teamwork, safety, evidence-based nursing, communication, patient-
centered care, and clinical judgment.
METHODS OF EVALUATING SIMULATION EXPERIENCES 13
Implications for Research
One of the problems noted in the literature is that these tools have not been used for a
larger random-control trial study in order to test for validity and inter-rater reliability. Simulation
is not standardized across programs and tools developed cannot capture completely the
effectiveness of simulation when it is measuring differing variations of implementation of
simulation. If simulation scenarios were standardized in formal nursing programs it would be a
platform for more standardized studies and hopefully a more standardized measurement tool for
evaluation could emerge. This would be a large endeavor but would help programs develop
simulation strategies more readily and more easily to adopt.
Implications for Education
Funding for simulation seems robust at this time and as educators we need to grab this
opportunity to attempt the standardization of simulation in nursing education. By using the tools
that have been developed it may be possible to integrate the strong elements from each and
create an evaluation rubric that is objective and comprehensive for use in simulation.
METHODS OF EVALUATING SIMULATION EXPERIENCES 14
References
Adamson, K., & Kardong-Edgren, S. (2012). A method and resources for assessing the reliability
of simulation evaluation instruments. Nursing Education Perspectives, 33(5), 334-339.
Alinier, G., Hunt, B., Gordon, R., & Harwood, C. (2006). Effectiveness of intermediate-fidelity
simulation training technology in undergraduate nursing education. Issues and
Innovations in Nursing Education, 359-369.
Ashcraft, A., Opton, L., Bridges, R., Caballero, S., Veesart, A., & Weaver, C. (2013). Simulation
evaluation using a modified Lasater clinical judgment rubric. Nursing Education
Perspectives, 34(2), 122-126.
Dillard, N., Sideras, S., Ryan, M., Carlton, K., Lasater, K., & Siktberg, L. (2009). A
collaborative project to apply and evaluate the clinical judgment model through
simulation. Nursing Education Research, 30(2), 99-104.
Gantt, L. (2010). Using the Clark simulation evaluation rubric with associate degree and
baccalaureate nursing students. Nursing Education Perspectives, 31(2), 101-105.
Howard, M., Englert, N., Kameg, K., & Perozzi, K. (2011). Integration of simulation across the
undergraduate curriculum: Student and faculty perspectives. Clinical Simulation in
Nursing, 7(1), e1-e10. doi:10.1016/j.ecns.2009.10.004
Kardong-Edgren, S., Starkweather, A., & Ward, L. (2008). The integration of simulation into a
clinical foundations of nursing course: Student and faculty perspectives. International
Journal of Nursing Education Scholarship, 5(1), 1-16.
uetke, R., Bembenek, B. ( ). Simulation evaluation A comparison of two simulation
evaluation rubrics PD document . Retrieved from SEN web site
www.qsen.org/docs/ conference/ SEN uetke.pdf
METHODS OF EVALUATING SIMULATION EXPERIENCES 15
McCaughey, C., & Traynor, M. (2010). The role of simulation in nurse education. Nurse
Education Today, 30, 827-832. doi:10.1016/j.nedt.2010.03.005
Mills, J., West, C., Langtree, T., Usher, K., Henry, R., Chamberlain-Salaun, J., & Mason, M.
(2014). 'Putting it together': Unfolding case studies and high-fidelity simulation in the
first-year of an undergraduate nursing curriculum. Nurse Education in Practice, 14, 12-
17. doi:10.1016/j.nepr.2013.06.003
Schlairet, M. (2011). Simulation in an undergraduate nursing curriculum: Implementation and
impact evaluation. Journal of Nursing Education, 50(10), 561-568.
Schlairet, M., & Pollock, J. (2010). Equivalence testing of traditional and simulated clinical
experiences: Undergraduate nursing students' knowledge acquisition. Journal of Nursing
Education, 49(1), 43-47. doi:10.3928/01484834-20090918-08
Sinclair, B., & Ferguson, K. (2009). Integrating simulated teaching/learning strategies in
undergraduate nursing education. International Journal of Nursing Education
Scholarship, 6(1), 1-11.
METHODS OF EVALUATING SIMULATION EXPERIENCES 16
Appendix A
Prisma Diagram
PRISMA 2009 Flow Diagram
Records identified through database searching
(n = 531)
Scre
enin
g In
clu
ded
El
igib
ility
Id
enti
fica
tio
n
Additional records identified through other sources
(n = 4)
Records after duplicates removed (n = 535)
Records screened (n = 28)
Records excluded (n = 17)
Full-text articles assessed for eligibility
(n = 28) Full-text articles excluded, with reasons
(n = 17) Not relevant to criteria for evaluation tools/methods
in simulation
Studies included in qualitative synthesis
(n = 11) (Moher, Liberati, Tetzlaff,
& Altman, 2009)
METHODS OF EVALUATING SIMULATION EXPERIENCES 17
Appendix B
Simulation Evaluation Theme Elements
Simulation Evaluation Elements
Patient-centered Care and Assessment
Evidence-based Nursing Interventions and Clinical
Judgment
Communication and Teamwork
Safety
Clark Simulation Evaluation Rubric (CSER)
X X X
Lasater Clinical Judgment Rubric (LCJR)
X X X
Sweeney-Clark Simulation Performance Rubric (SCSPR)
X X X X
Colorado Mountain College 2012
QSEN Based Simulation Evaluation Rubric (QBSER)
X X X X
Objective Structured Clinical Examination (OSCE)
X X X X
The Seattle University Evaluation Tool (SUET)
X X X
The Creighton-Simulation Evaluation Instrument (C-SEI)
X X X
METHODS OF EVALUATING SIMULATION EXPERIENCES 18
Appendix C
Study Purpose Theory Sample Methods Design Findings Limitations Comments
Schlairet, M.
(2011). Simulation in an undergraduate
nursing curriculum:
Implementation and impact evaluation.
Journal of Nursing
Education, 50(10),
561-568.
To explore the influence
of simulation across an undergraduate BSN
curriculum
Nursing Education
Simulation Framework NESF developed by
Jeffries 2005
161 Junior and senior
level BSN students enrolled at a regional
university in the
southeastern US representing traditional
and accelerated nursing
tracks. Convienence
sampling, qualitative and
quantitative data
Broad evaluation
model, mixed methods approach. Surveys,
program eval data,
faculty reports, externally validated
data collection
instruments.
Used NLN Laerdal Multisite
simulation group instruments. Demographic
form for student
characteristics. Faculty data derived from two
investigator designed survey
instruments
150/161 of student
survey packets returned 94%
response rate.
Student findings:
Students found
simulation provided
active learning in a
safe environment.
Convienence
sample Group size small
Conceptual
learning not measured
Faculty level
instruments
designed by in
house faculty
and not validated by external
source
Medium strength
d/t convenience sample at one
university
Support
continued
integration of sim
through
curriculum
Gantt, L. (2010).
Using the Clark simulation
evaluation rubric
with associate degree and
baccalaureate nursing students.
Nursing Education
Perspectives, 31(2), 101-105.
To study the rubric to
find if it captures more contextual and critical
thinking components in
simulation scenarios in nursing program
Benner and Bloom 69 adn students and 109
bsn students
Grant from CON goal to
expand clinical capacity to supplement or replace
traditional clinical rotations. 2nd goal assist
comm college faculty to
develop experience with simulators and scenarios
Medium strength
d/t one cohort convenience
sample
Clark Simulation
Evaluation Rubric: Concept
and skills
acquisition tool, needs to be
further developed
so that interrater reliability is
achieved
McCaughey, C., &
Traynor, M. (2010). The role of
simulation in nurse
education. Nurse Education Today,
30, 827-832.
doi:10.1016/j.nedt.2010.03.005
To evaluate the role of
medium to high fidelity simulation in the
students prep for
clinical practice and to evaluate how it helped
students make the
transition to staff nurse
NMC proficiency
standards expected of third year nursing
students provided
framework and to extrapolate themes
An entire cohort of adult
branch nursing students (153) from higher
education institution
chosen as a group because they had recently
been exposed to
simulated learning and were approaching
transition from student to
staff nurse within study time frame
During simulated
learning students were observed through a
one way mirror as
sessions were videotaped to facilitate
a debrief process post
scenario.
Thirty two item
questionnaire with 27 likert scales and five
demographic questions
to construct profiles of population.
Data from questionnaire
coded for a number value and inputted into SPSS.
Descriptive stats used to
illustrate frequencies and central tendencies mean
mode median.
Ordinal level of data, no
inferential stats.
Qualitative data summarized
using recurrent themes.
93 completed
questionaires for a 60% response rate.
All responders had
used the simulator for one 4 hour period in
past 8 months.
Perceived effect sim has on student prep
for role as RN:
72% sim will help them prep for
transition in role.
92.5% sim promoted confidence to deal
with similar situation as new staff RN.
METHODS OF EVALUATING SIMULATION EXPERIENCES 19
Adamson, K., &
Kardong-Edgren, S. (2012). A method
and resources for
assessing the reliability of
simulation
evaluation instruments.
Nursing Education
Perspectives, 33(5), 334-339
Describes a successfully
piloted method for facilitating rapid
psychometric
assessments of three simulation evaluation
instruments
Kolb theory of
experiential teaching and need for ebp eval
and practice in nursing
education
47 faculty teaching in an
accredited prelicensure bsn program in US with
one year experience using
HPS and clinical or practice experience in
acute care as Rn during
past 10 years
IRB approval of
Washington state University. Exemption
from human subject
testing as videos were used. Video archived
simulation sent to
participants
Evaluators used same tool
for each video simulation using 3 instruments: LCJR,
SUET and CSEI. 22 items of
assessment. Instruments selected on their conceptual
frameworks. Training given
to participants for interrater reliability. NLN standardized
scenarios used for sim
Interrater reliability of
3 tests .85-.95 Test Retest reliability
consistent
Internal consistency cronbach’s alpha .96
to .97
Intraclass
correlation, convienence
sampling,
recruits had higher than
average interest
in simulation,
Med to high
strength d/t broad range of faculty
used and number
of instruments tested.
Move toward a more
standardized
approach to simulation?
Alinier, G., Hunt,
B., Gordon, R., &
Harwood, C. (2006).
Effectiveness of
intermediate-fidelity simulation
training technology
in undergraduate nursing education.
Issues and Innovations in
Nursing Education,
359-369
Critically appraise value
of use of simulation in
nursing education by comparing performance
in a practical exam of
two groups of students
n.a Three consecutive
cohorts of students in
second year diploma program. 133 volunteers
One group exposed to
scenario based sim
training and one was not. Experimental
group was
hypothesized to do better. OSCE tool used
Pretest posttest experiement
to compare both groups.
Data collected 2 years. Students followed normal
curriculum. Students in the
experiment group took part in scenario based clinical
intensives for two
afternoons. Random assignment.
Experimental group
did better. Positive
impact of sim training
No correlation
between
confidence levels.
Medium strength
d/t convenience
sampling.
Stressors for
some participants included being in
a tech
environment. Facilitators
should have less interaction with
participants in
scenarios
Ashcraft, A., Opton,
L., Bridges, R.,
Caballero, S.,
Veesart, A., & Weaver, C. (2013).
Simulation
evaluation using a modified Lasater
clinical judgment
rubric. Nursing Education
Perspectives, 34(2),
122-126
Describe process of
evaluating senior
nursing students in sim
lab using a modified LCJR
n.a Study conducted over
two years with students
enrolled in final bsn
semester using HFS. 86 in phase one and 102 in
phase 2
LCJR provides
numeric grading and
clarity for expectations
of clinical competency. IRB
approval obtained.
Descriptive study. Students
assigned to roles in scenarios
and concept map along with
pathophys prior to simulation. Sim in lab setting
with HFS. Phase I was done
in first year of study and phase II in second year.
Random assignment. Four
standard diagnosis used. Two faculty managed
scenarios-each faculty
evaluated only one student. 20 minute scenarios and
debriefing occurred
Clinical judgment
profiles increased
between phase one
and 2 with improvement in
summative scores vs
formative scoring
Students
informing other
students about
scenarios. Faculty training
on equipment.
Instructor interp of rubric
Medium strength
d/t convenience
sample. Good
length of time for study and two
phases compared.
Dillard, N., Sideras,
S., Ryan, M.,
Carlton, K., Lasater,
K., & Siktberg, L. (2009). A
collaborative
project to apply and evaluate the clinical
judgment model
To evaluate
effectiveness of faculty
development workshop
focused on evaluating students clinical
thinking during sim.
To eval student learning
after sim case.
Cervero model for
faculty evaluation
Tanners clinical judgment model
68 students enrolled in
junior adult health course.
Sim occurred for all. 25
of those students participated in the clinical
portion cardiovascular
practicum.
Mixed method.
All students exposed
to sim. 25 of those did
clinical practicum. Each student provided
written reflections of
care provided to heart failure patients. All
faculty completed a
Faculty evaluations of study
done and Student evaluations
of study done each with their
own set of subscales for evaluation.
Positive impact on
sim noted. Results
based on lessons
learned for each group with an understanding
that more work needs
to be done to incorporate into
curriculum.
Small study with
very small
cohort. All
faculty but no number of
faculty given. 2
schools collaborated on
project and
Weak strength d/t
sample and
design of
evaluation tool.
No standardized
method for sim has been
developed and no
METHODS OF EVALUATING SIMULATION EXPERIENCES 20
through simulation.
Nursing Education Research, 30(2), 99-
104
To explore the perceptions of students
and faculty regarding
impact of sim session on actual clinical practice
workshop evaluation. qualitative data
not compiled systematically
standard method
for evaluating sim has been
developed
Howard, M.,
Englert, N., Kameg,
K., & Perozzi, K. (2011). Integration
of simulation across
the undergraduate curriculum: Student
and faculty
perspectives. Clinical Simulation
in Nursing, 7(1), e1-
e10. doi:10.1016/j.ecns.2
009.10.004
To implement and
integrate the use of HFS
as a teaching activge learning strategy
throughout undergrad
curriculum and evaluate this instructional
technology.
Kolbs experiential
Jeffries nursing ed sim
framework
151 students Accelerated
and traditional BSN
program 6 faculty
Mixed methods survey
and focus group.
Research design.
Sim helped students to better
understand concepts
Valuable learning experience
Help to stimulate critical thinking
Knowledge gained can be transferred to clinical setting
Faculty and students
both agreed that use
of sim beneficial to achievement of
learning objectives
Small cohort
studied. Students
had been exposed to sim
throughout
program
Weak strength d/t
sample and
convenience size. Also based on
opinions of sim.
Supports use of
sim in undergrad
nursing curricula
Adequate
resources including funding
noted for use of
sim in programs
Luetke, R., &
Bembenek, B.
(2012). Simulation evaluation: A
comparison of two
simulation
evaluation rubrics
[PDF document].
Retrieved from QSEN web site:
www.qsen.org/docs
/2012_conference/ SEN uetke.
To evaluate 2 sim eval
tools within a planned
sim for sound clinical judgement.
Benners novice to
expert
QSEN as guide for
objectives
19 students in first year
of nursing program
Planned simulation with
group. Sweeney Clarke
eval tool based on
Benner’s theory novice to
expert and QSEN rubric
developed by faculty used to score each
students. QSEN rubric
based on QSEN safety objectives and standards.
Comparative study
using two rubrics
during a simulation at college in Colorado,
one developed by
faculty. Minimum of
77% on rubric to pass.
If no pass student need
to repeat sim and below 85 write out self
study action plan to
learn skills needed for sim
One class (n=19) of
first year students completed the same
two high fidelity
simulation scenarios and were evaluated
as a group on their
performance in each
scenario. The
students were
evaluated with the Sweeney Clark rubric
in the first scenario, the QSEN Rubric in
the second scenario
Students felt positive about
sim and positive about
knowing the objectives and evaluation elements prior to
testing. Faculty felt that
QSEN competencies were a
stronger basis for producing
the NOF.
Students found that
rubric avail prior to
sim let them know ahead of time what
will be expected.
Faculty found it a
good framework to
evaluate the work and
eliminates subjective components. Happens
concurrently with sim
Not published.
One cohort
Faculty interrater reliability. Small
group. Unsure if
they tested more
than one group.
Eval tool not
widely used, developed by
faculty at that
college
Weak strength,
d/t convenience
sample and unable to assess
number faculty
involved in small
study
Sim grading rubrics allow
faculty to assess
students on critical thinking
and contextual
components of pt care scenarios
Mills, J., West, C., Eval first year Adaptation of traditional 47 bsn students Unfolding case studies Case study design conducted Qual and quant First year Supported model
METHODS OF EVALUATING SIMULATION EXPERIENCES 21
Langtree, T., Usher,
K., Henry, R., Chamberlain-
Salaun, J., &
Mason, M. (2014). 'Putting it together':
Unfolding case
studies and high-fidelity simulation
in the first-year of
an undergraduate nursing curriculum.
Nurse Education in
Practice, 14, 12-17. doi:10.1016/j.nepr.2
013.06.003
undergrad students level
of satisfaction with new model of teaching
clinical skills using case
students in HFS
case study
method/theory
consenting for sim
3 faculty Two standard patients
5 of above students
consented for interview process for qual info
with standard pts in
sim hospital setting. Live video taping and
debriefing. Each
scenario introduced 5 minutes prior to
scenario. 15 minutes to
complete scenario. Quantitative data using
combo of three
instruments form NLN and used in other
research groups in
previous sim studies, permission received
from NLN. SSSL,
EPQ and SDS.
over 4, 6hr sim session
Descriptive stats analyse quant data for survey
findings. Transcripts open
coded and grouped which produced two themes
emerged (being watched and
putting it together)
evidence of student
satisfaction to support a HFS model of
teaching clinical
skills. High value of learning experience
students level of
experience still building
confidence and
competence.
No authenticity
of real hospital environment
noted in study
text.
Comparing of
more than one cohort over a
period of
consecutive years would
have given
results to analyze
of teaching.
Positive scores in
all areas of
evaluation.
Improvement of
design elements
Funded by
university grant
METHODS OF EVALUATING SIMULATION EXPERIENCES 22