competency-based medical education (cbme): milestones...
TRANSCRIPT
8/6/2014
1
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Competency-based Medical Education (CBME):
Milestones and Assessment Systems
© 2014 Accreditation Council for Graduate Medical Education
Objectives
Purposes of Milestones
Role of Milestones in NAS
System perspective: Assessment and Milestones
Clinical Competency Committees
© 2014 Accreditation Council for Graduate Medical Education
When I say “Milestone”…
• What first comes to mind?
• Share your initial reaction with a neighbor• Why this reaction?
8/6/2014
2
© 2014 Accreditation Council for Graduate Medical Education
Purposes and Implications
ACGME• Accreditation – continuous monitoring
of programs; lengthening of site visit cycles
• Public Accountability – report at a national level on competency outcomes
• Community of practice for evaluation and research, with focus on continuous improvement
Training Programs• Framework for CCC• Guide curriculum development• More explicit expectations of trainees• Support better assessment• Enhanced opportunities for early
identification of under-performers
Certification Boards• Potential use – inform eligibility
decisions for certification
Residents and Fellows• Increased transparency of
performance requirements• Encourage informed self-assessment
and self-directed learning• Better feedback
Milestones
4
Milestones are a Formative Assessment Framework
© 2014 Accreditation Council for Graduate Medical Education
The Milestones and NAS in a Nutshell
• A Continuous Accreditation Model based on assessment of annual data – this list is not all encompassing and is subject to change• Annual program data (resident/faculty information, major
program changes, citation responses, program characteristics, scholarly activity, curriculum)
• Aggregate board pass rate
• Resident clinical experience
• Resident survey and faculty survey (latter is new)
• Semi-annual resident Milestone evaluations
• 10 year Self-Study and Self-Study Visit
• Clinical Learning Environment Review (CLER) Visits
© 2014 Accreditation Council for Graduate Medical Education
* From TeamSTEPPS/AHRQ
COMPETENCE
Shared Mental Model Challenge
MILESTONES
8/6/2014
3
© 2014 Accreditation Council for Graduate Medical Education
Milestones
• By definition a milestone is a significant point in development.
• Milestones should enable residents, fellows and the training program to better determine an individual’s trajectory of competency acquisition.
© 2014 Accreditation Council for Graduate Medical Education
PC1. History (Appropriate for age and impairment)
Level 1 Level 2 Level 3 Level 4 Level 5Acquires a general medical history
Acquires a basic physiatric history including medical, functional, and psychosocial elements
Acquires a comprehensive physiatric history integrating medical, functional, and psychosocial elements
Seeks and obtains data from secondary sources when needed
Efficiently acquires and presents a relevant history in a prioritized and hypothesis driven fashion across a wide spectrum of ages and impairments
Elicits subtleties and information that may not be readily volunteered by the patient
Gathers and synthesizes information in a highly efficient manner
Rapidly focuses on presenting problem, and elicits key information in a prioritized fashion
Models the gathering of subtle and difficult information from the patient
CompetencyDevelopmental
Progression or Set of Milestones Sub-competency
Specific Milestone
© 2014 Accreditation Council for Graduate Medical Education
Entrustable Professional Activities
• EPAs represent the routine professional-life activities of physicians based on their specialty and subspecialty
• The concept of “entrustable” means:• ‘‘a practitioner has demonstrated the necessary
knowledge, skills and attitudes to be trusted to perform this activity [unsupervised].’’1
1Ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and
clinical practice? Acad Med. 2007; 82(6):542–547.
8/6/2014
4
© 2014 Accreditation Council for Graduate Medical Education
Competencies, Milestones and EPAs
COMPETENCY
Entrustable Professional Activity
MILESTONES
Characteristic Competencies Milestones EPAs
Granularity Low Moderate to High Low to Moderate
Synthetic/Integrated Moderate Low to Moderate High
Practicality (application)
Low Moderate High
Conceptual High Low Low to Moderate
COMPETENCY
MILESTONES
COMPETENCY
MILESTONES MILESTONES
COMPETENCY
© 2014 Accreditation Council for Graduate Medical Education
Milestones and EPAs as Roadmap
Observations:
1) Journey not a straight line
2) More than one path (but not infinite)
3) “If you don’t know where you are going, any road will get you there”
© 2014 Accreditation Council for Graduate Medical Education
Dreyfus & Dreyfus Development Model
Dreyfus SE and Dreyfus HL. 1980Carraccio CL et al. Acad Med 2008;83:761-7
Time, Practice, Experience
Novice
Advanced Beginner
Competent
Proficient
Expert/Master
8/6/2014
5
© 2014 Accreditation Council for Graduate Medical Education
Dreyfus & Dreyfus Development Model
Dreyfus SE and Dreyfus HL. 1980Carraccio CL et al. Acad Med 2008;83:761-7
Time, Practice, Experience
Novice
Advanced Beginner
Competent
Proficient
Expert/Master
MILESTONESCurriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Curriculum
Assessment
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Assessment System for Effective CBME
© 2014 Accreditation Council for Graduate Medical Education
Professional Self-Regulatory Assessment System
Assessments withinProgram:
• Direct observations• Audit and
performance data• Multi-source FB
• Simulation• ITExam
Judgment and Synthesis:Committee
Residents
Faculty, PDs and others
Milestones and EPAs as Guiding Framework and Blueprint
Accreditation
Unit of Analysis:Program
Certification and Credentialing
Unit of Analysis:Individual
Institution and Program
This is a Human Process
8/6/2014
6
© 2014 Accreditation Council for Graduate Medical Education
Linked Aims of Improvement
Better Patient (and population)
Outcomes
Better Professional Development
Better System Performance
Everyone
Batalden PB and Davidoff F. Qual Saf Health Care. 2007;16:2–3.
© 2014 Accreditation Council for Graduate Medical Education
Evaluating Residency Programs Using Patient Outcomes JAMA 2009;302(12):1277-1283. Asch, DA, et.al.
0
2
4
6
8
10
12
14
Q5 Q4 Q3 Q2 Q1 Q1-Q5
Residency Program of Origin, Ranked (Quintile) by Program Complication Rate
Rate of Major Obstetric Complications by Graduates (%)
∆
Difference remainsafter correction for
USMLE performance
Excess Risk ∆ 32%Q1 vs Q5
Care of the Vulnerable Elderly Study
Performance on Geriatric Process of Care
ResidentClinics
Mean %
PracticingPhysiciansMean %
UnivariateF
Structurecoefficients
Documentation of:
Gait evaluation 28.4% 74.2% 77.53** .90
Balance evaluation 21.6% 66.4% 65.51** .82
Medical surrogate 28.0% 54.4% 24.00** .65
End-of-life preferences 29.5% 49.3% 12.85** .55
Vision testing done 40.0% 64.7% 19.09** .55
Hearing assessment 23.3% 40.3% 8.06* .41
Screens for:
Falls risk 18.6% 60.8% 49.60** .67
Cognitive impairment 18.3% 52.0% 29.02** .60
Depression 33.7% 62.6% 24.09** .57
Lynn LA, et al. Acad Med. 2009.
8/6/2014
7
© 2014 Accreditation Council for Graduate Medical Education
Linked Aims of Improvement
Better Patient (and population)
Outcomes
Better Professional Development
Better System Performance
Everyone
Batalden PB and Davidoff F. Qual Saf Health Care. 2007;16:2–3.
© 2014 Accreditation Council for Graduate Medical Education
Norcini: How do we train faculty?
Faculty development
• Methods of assessment will need to be based largely on observation• Faculty are the measurement
instrument and they need training
• Milestones make training easier but they are not a substitute for it • 2-4 hour training exercise with periodic follow-up
important (deliberate practice)
From J. Norcini; AMEE 2013; FAIMER
© 2014 Accreditation Council for Graduate Medical Education
Linked Aims of Improvement
Better Patient (and population)
Outcomes
Better Professional Development
Better System Performance
Everyone
Batalden PB and Davidoff F. Qual Saf Health Care. 2007;16:2–3.
8/6/2014
8
© 2014 Accreditation Council for Graduate Medical Education
Effective Assessment Process
Most important component of curriculum is the clinical care residents provide and experience Clarity on right outcomes linked to curriculum
Integration of the educational and clinical systems
Right combination and synthesis of assessment methods
Critical importance of shared understanding & mental models of competence Competencies, milestones, entrustable professional
activities (EPAs)
© 2014 Accreditation Council for Graduate Medical Education
Assessment and Safe Patient Care
Importance of appropriate supervision
Entrustment
Trainee performance* X
Appropriate level of supervision**
Must = Safe, effective patient-centered care
* a function of level of competence in context
**a function of attending competence in context
Kogan JR, Conforti LN, Iobst WF, Holmboe ES. Reconceptualizing variable rater assessments as both an educational and clinical care problem. Acad Med. 2014 May;89(5):721-7.
© 2014 Accreditation Council for Graduate Medical Education
Discussion: Your Efforts in the Triangle
Better Patient (and population)
Outcomes
Better Professional Development
Better System Performance
Everyone
Batalden PB and Davidoff F. Qual Saf Health Care. 2007;16:2–3.
8/6/2014
9
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Competency Committees
© 2014 Accreditation Council for Graduate Medical Education
The Assessment “System”
Assessments withinProgram:
• Direct observations• Audit and
performance data• Multi-source FB
• Simulation• ITExam
Qual/Quant “Data”
Synthesis:Committee
Residents
Faculty, PDs and others
Milestones and EPAs as Guiding Framework and Blueprint
Accreditation
Unit of Analysis:Program
Certification and Credentialing
Unit of Analysis:Individual
JUDGEMENT
D
FB
FB
DD FB
PUBLIC
© 2014 Accreditation Council for Graduate Medical Education
8/6/2014
10
© 2014 Accreditation Council for Graduate Medical Education
Group Decision Making
Key Issues What is the environment in which the committee
performs its work? What is the local culture?
Groups within groups
What are the effects of hierarchy on group decision making? David Berg: Medicine one of the most hierarchical of all
professions
Single variable of effectiveness: extent to which people are willing to say “positive” and “negative” comments and observations in a group
© 2014 Accreditation Council for Graduate Medical Education
• The wisdom of many is often better than the wisdom of the few
• To maximize the probability of good judgments:• Sample• “Independence”• Diversity
are important…
The Wisdom of Crowds
© 2014 Accreditation Council for Graduate Medical Education
Basic Committee Principles
• Evidence-based versus verdict-based “jury”• Start and review all evidence before a decision
• Do not start with a conclusion/decision
• Confirmation bias
• Be careful not to emphasize consensus over dissent• Minority opinions, even if “wrong”, still helpful
• Be sure all voices are “heard” and watch carefully for negative effects of hierarchy
8/6/2014
11
© 2014 Accreditation Council for Graduate Medical Education
Committee Benefits
• Develop group goals and shared mental models
• “Real-time” faculty development
• Key for dealing with difficult residents and fellows
• Share and calibrate strengths and weaknesses of multiple faculty assessments (“observations”)
• Key “receptor site” for frameworks/milestones• Synthesis and integration of multiple assessments
© 2014 Accreditation Council for Graduate Medical Education
“Wisdom of the Crowd”
Hemmer (2001) Group conversations more likely to uncover
deficiencies in professionalism among students.
Schwind (2004) 18% of resident deficiencies requiring active
remediation became apparent only via group discussion. Average discussion 5 minutes/resident (range
1 – 30 minutes)
© 2014 Accreditation Council for Graduate Medical Education
Narratives and Judgments
Pangaro (1999) Matching students to a “synthetic” descriptive
framework (RIME) reliable and valid across multiple clerkships
Key component: good process with facilitation
Regehr (2012) Faculty created narrative “profiles” (16 in all)
found to produce consistent rankings of excellent, competent and problematic performance.
8/6/2014
12
© 2014 Accreditation Council for Graduate Medical Education
Milestone Journey: Revised Conceptual Model of Rapid Cycle Change
Tomolo A M et al. Qual Saf Health Care 2009;18:217-224
Accreditation Council for Graduate Medical Education
© 2014 Accreditation Council for Graduate Medical Education
Thank You and Questions