competency evaluation, simulation and the future of ...c.ymcdn.com/sites/ · competency evaluation,...
TRANSCRIPT
Competency Evaluation, Simulation and the Future of Certification and
Recertification
John M. O’Donnell CRNA, MSN, DrPHDirector, University of Pittsburgh Nurse Anesthesia Program
Associate Director, WISER
Objectives
• Review the current processes for competency evaluation, certification and recertification
• Discuss emerging trends in healthcare education toward proof of competency using new educational methods
In the health care setting, clinical
competence is….
The mastery of relevant knowledge, attitude and
skill at a satisfactory level …….at a certain
point in the educational process
modified from www.qualityresearchinternational.com
In the case of clinical training, which is primarily
based on an apprenticeship model, teachers
define what the student is expected to do and
then test their ability to do it.
modified from www.qualityresearchinternational.com
….most clinical actions are concerned with
problems for which there are no clear answers
and no single solution…..in some cases the
solution will be something he or she has never
arrived at before.
modified from www.qualityresearchinternational.com
Complexity in Clinical Problem Solving
Clinical Problem
Correct or IncorrectSolution
Knowledge AExperience
Knowledge BBook Knowledge
Knowledge CYour Course
Decisions
Adapted. With permission from Dr. Paul Phrampus MD
Feedback
Competence itself is only of value as a
prerequisite for performance in a real clinical
setting and does not always correlate highly
with performance in practice.
modified from www.qualityresearchinternational.com
Bottom Line: We are looking for the threshhold level of competence established
through some sort of standard setting
The assessment is thus:Is he or she good enough?
Does Competence Increase with Years of Practice?
• Conclusions: Physicians who have been in practice longer may be at risk for providing lower‐quality care. Therefore, this subgroup of physicians may need quality improvement interventions. Niteesh, K, Review: The Relationship between Clinical Experience and Quality of Health Care Ann Intern Med. 2005;142:260‐273.
Experience does not always equal better quality
“I’ll know when it’s time”Hines Ward, 2011
When talking about competence‐
Does the person know when they have lost it?
How many times have you encountered a student/ or other provider who is not
consciously aware of their performance issues?
1. Never2. Once or twice3. Several times4. Frequently5. Daily18%
47%
30%
3%
2%
N = 170 responses, 2010 Assembly of School Faculty
Self‐Assessment and Competence“the preponderance of evidence suggests that physicians have a limited ability to accurately self‐assess. The processes currently used to undertake professional development and evaluate competence may need to focus on more external assessment.” JAMA 2006;296: 1094‐1102
Skills needed to be competent = skills needed to self assess
Levels of Competence?
• Unconsciously incompetent• Consciously incompetent• Consciously competent• Unconsciously competent
Dreyfus, SE, Dreyfus HL. Mind over Machine. 1986. New York NY. Blackwell PublishersKruger J, Dunning D. Unskilled and Unaware of it: How Difficulties in Recognizing One’s Own IncompetenceLead to Inflated Self-Assessment. Journal of Personality and Social Psychology. 1999. 77, 6: 1121-1134
??? 10,000 hrs
So What Does Increase Competence?
• Deliberate practice (DP)• Ericsson, 2008,
– Concept of ‘deliberate practice’ as the route for development of new skills (up to the expert level)
Source: Academic Emergency Medicine 2008; 15:988–994
Developing Metrics
RNs MDs PharmDsRRTs
Anesthesia Providers
What kind of competency evaluation is used in anesthesia for entry to practice?
How about for ongoing practice?
What are our current ways to teach and measure ongoing competencies?
Lectures and testsAnnual ‘competencies’
Annual performance reviewsRequest cases: OR street level
Do we need new methods?
Lecture (5%)*
Reading (10%)
Audiovisual (20%)
Demonstration (30%)
Discussion/Interaction (50%)
Practice / Role Playing / Simulation (75%)
Application and Utilization of New Learning Situations (90%)
*Average Retention Shown as % of Material Retained Source: National Training Laboratories Institute (1970)
Pyramid of Learning RetentionPyramid of Learning Retention
With permission: Dr. Jeffrey Groom
Lecture (5%)
Reading (10%)
Audiovisual (20%)
Demonstration (30%)
Discussion/Interaction (50%)
Practice / Role Playing / Simulation (75%)
Application and Utilization of New Learning (90%)
Pyramid of Teaching PracticesPyramid of Teaching Practices
*Average Retention Shown as % of Material Retained Source: National Training Laboratories Institute (1970)
With permission: Dr. Jeffrey Groom
Industry and Military Recognition of Value in Simulation• Aviation, military, NASA, nuclear etc
• Simulation used and in many cases now required• Adopted as a matter of necessity and expediency• Now culturally embedded e.g. FAA requirement for pilot
simulation• Until the 1990’s‐ Obscure area in healthcare education
– Technology advances and miniaturization– 1999 IOM Report and subsequent national safety movement
Link Trainer- 1929 Boeing 747 Simulator Mrs. Chase- 1905SimMan™ - 1999
Captain Chesley Sullenberger
DidacticClinical
Simulation is One Bridge to Teach Skills and Measure Some Areas of Competence
Present clinical problems authentically…..
My current experience level in simulation education is:1. Novice2. Some experience3. Moderate experience4. Significant experience5. Expert
2010 AOSF
Current Pitt NAP Sim Contact Hrs (110+ per student)
Year 1‐Participant (70 hrs)
Year 2‐Participant and Volunteer
(30 hrs)
Year 3‐Participant, Volunteer,Student Instructor
(10‐40 hrs)
We use it to develop and measure a variety of competencies
AssessmentIndividual Psychomotor Skills
Monitoring and Intervention Skills
Clinical Problem Solving
Learning roles and goals within a teamClinical Reasoning
Ex: Inducing Anesthesia
• Value:• Confidence• Basic skills• Decreased anxiety
• Strategy is Scaffolding• Didactic• Standardized patients• Part task trainers• Full task trainers• Practice sessions• Full context simulation
Lt. Jason Trudell, CRNA, MSNLt. John Jones, CRNA, MSNLt. Brian Keller, CRNA, MSNCpt. Bill Teskey, CRNA, MSNCpt. Kurt Kline, CRNA, MSNLt. Scott Pearson, CRNA, MSN
Critical event simulation has helped our grads prepare for real life..
July 2008
• …we all agree that such learning activities as anesthesia crisis leadership training (ACLT), other WISER simulations, and the variety of clinical experiences that Pitt had to offer are extremely beneficial downrange. We provide anesthesia in some of the most austere and unforgiving environments, sand, lack of transport for supplies due to bad weather conditions, and overall enemy activity necessitating the need for adaptation and quick thinking when faced with taking care of critical patients.
Henrichs (2009): CRNAs vs. MDs
• Head to head comparison of providers• Population:
– 35 MDs, 24 CRNAs
• Intervention– Scenario management
• Outcome‐– Comparison of completion of critical steps
• MDs did a little better overall
Henrichs, B. M., M. S. Avidan, et al. (2009). "Performance of certified registered nurse anesthetists and anesthesiologists in a simulation-based skills assessment.[see comment]." Anesthesia & Analgesia 108(1): 255-62.
How Serious is the ASA?
• MOCA‐maintenance of certification in anesthesiology• Once q 10 years‐must take simulation as part of accreditation‐ ABA is offering incentives (enhanced CE credit)
• ASA endorsed (accredited) centers‐ 18 in US• 3000‐5000 participants per year starting this year
Ziv et. al. (Israeli National Simulation Center‐MSR) Standardized Testing,
Selected Specialties (2004‐present)
Schaeffer, Quinlan et. al. DAW Course University of Pittsburgh (2006‐present)
Is high stakes testing on the
way?
Where does AANA Stand?
• No widespread agreement or white paper• No infrastructure• Simulation Users Group of National Faculty
• New NBCRNA recertification initiative counts simulation but there is no mandatory requirement
NBCRNA vs. AANA Missions
Slightly Different Missions
NBCRNA MissionTo promote patient health and safety throughcredentialing programs that support
lifelong learning‐ Credentialing
AANA MissionAANA advances patient safety, practice excellence, and its members’ profession‐ Practice/Profession
Why Recertification Change ‐ NOW?
IOM reports?
Public expectations?APRN equivalence?
Financial motives?
MD equivalence?
Raising the bar?
PEW/ Credentialing bodies
AA equivalence?
Outdated system?
NCSBN expectations?
Drivers for Change
• Citizens Advisory Council and AARP• Politics• Financial pressures? AANA/NBCRNA• Regulators‐ accreditors of certifying bodies• Institute of Medicine Reports
– Crossing the Quality Chasm: A new Health System for the 21st Century
– To Err is Human– The Future of Nursing– Redesigning Continuing Education in Healthcare Professions
Implementing Continuing Competency Requirements for Health Care Practitioners Swankin, et al.. Citizen Advocacy Center #2006‐16 July 2006
Drivers for Change‐ IOM
“The absence of a comprehensive and well‐integrated system of continuing education (CE) in the health professions is an important contributing factor to knowledge and performance deficiencies at the individual and system levels”
Institute of Medicine
Redesigning Continuing Education in the Health Professions Committee on Planning a Continuing Health Professional Education Institute; Institute of Medicine. ISBN: 978‐0‐309‐14078‐2, (2010)
Does Certification Improve Care?
• .. Holding all other variables constant, treatment by a certified physician was associated with a 15% reduction in mortality among patients with an AMI
• .. Less patient mortality was associated with treatment by physicians who were cardiologists, cared for large number of AMI patients, were closer to graduation from medical school, and were certified – Acad. Med. 2000;75:1193‐1198
Less mortality from AMI when certified
Does Certification Improve Care?
• Conclusions: Few published studies (5%) … [of the studies published ]….more than half support an association between board certification status and positive clinical outcomes
• Specialty Board Certification and Clinical Outcomes: The Missing Link,
Sharp et al. Acad. Med. 2002;77:534‐542
Does Certification Improve Care?
• Grosch: “Sharp et al. ignored the evidence …and proposed enthusiastic but unjustified conclusions in support of specialty board certification
• Conclusion: No evidence supports the clinical benefit of specialty board certification…..”
Grosch, E. N. (2006). "Does specialty board certification influence clinical outcomes?.[Erratum appears in J Eval Clin Pract. 2006 Dec;12(6):704]." Journal of Evaluation in Clinical Practice 12(5): 473‐481
Bottom Line
• “The public perceives that board certification …..is the “gold standard” and if a clinician possesses this credential, he or she has the knowledge and skills required to be competent.
• Brennan, T. A., R. I. Horwitz, et al. (2004). "The role of physician specialty board certification status in the quality movement." JAMA 292(9): 1038‐1043.