Milestones: Are You Ready for July 1st ?

Kirk Lalwani, MD, FRCA, MCR

Associate Professor, Chair,

Resident Evaluation and Clinical Competence Committee,

Anesthesiology and Perioperative Medicine.

On behalf of the AMIGOS(Anesthesiology Milestones Implementation

Group Operatives)

Disclosure

No conflict of interest

Some of my slides are borrowed from the faculty educational resources on the ACGME website

Implementing MilestonesObjectives:

Outline the Next Accreditation System (NAS)

Define a Milestone

Review our Milestones strategy

Discuss assessment of Anesthesiology Milestones

Abbreviations

NAS- Next Accreditation System

CLER- Clinical Learning Environment

ACGME- Accreditation Council for Graduate Medical Education

RRC- Residency Review Committee

Accreditation Council for Graduate Medical Education

© 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013

The Next Accreditation System

Why ‘Next Accreditation System’ (NAS)?

“Self-regulation is a fundamental professional responsibility, and the system for educating physicians answers to the public for the graduates it produces.”

“The Next GME Accreditation System – Rationale and Benefits”Nasca T.J., Philibert I., Brigham T., Flynn T.C. N Engl J Med 2012; 366:1051-1056

Goals of NAS

Strengthen resident development in Professionalism Communication Skills Systems Based Practice Practice Based Learning

Enhance public accountability More explicit definition of a good physician (Milestones) Patient safety (Clinical Learning Environment Review Program)

Reduce burden required for accreditation

‘Next Accreditation System’ …in a Nutshell

Accreditation on the basis of educational outcomes

NAS : Advance from an episodic ` biopsy ’ model to annual data collection or ` continuous oversight ’.

RRCs will measure compliance through the evaluation of annual program data elements.

The Next Accreditation System

ContinuousObservations

Identify Opportunities for Improvement

Program MakesImprovement(s)

AssessProgramImprovement(s)

PromoteInnovation

© 2013 Accreditation Council for Graduate Medical Education Information Current as of December 2, 2013

The Next Accreditation System

We will now have annual data collection

Trends in annual data Milestones Resident and Faculty Surveys Resident and Faculty Scholarly activities Case Log Data ITE / ABA Examination pass rates

Clinical Learning Environment Review Visits

• Oversight of • Transitions of Care• Duty Hours Policy• Fatigue Management and Mitigation• Education and Monitoring of Professionalism• Involvement in Institutional Quality Improvement and

Safety initiatives• Supervision policies

Additional information available at: http://www.acgme-nas.org/CLER

Milestones

Click icon to add picture

What are Milestones?

Specific behaviors, attributes, or outcomes in the six general competency domains to be demonstrated by residents during residency. (ACGME and Specialty Boards)

Skill and knowledge-based development that commonly occurs by a specific time.

Why ????

ACGME requires it Program Accreditation

Who, When, and How???? All residents in all ACGME-accredited programs

Subspecialty fellows from July 2015

New assessment methodology as well as existing tools

6-monthly reporting of Milestones to ACGME in January and July

Reporting starts July 2014

Expected Benefits of Milestone Assessments

For Residents

Clarify expectations

Feedback should identify specific areas to work on

Earlier identification of under-performers

Aspirational goals for residents who exceed expectations

Expected Benefits of Milestone Assessment

For the Program Curriculum development Accreditation requirements Earlier identification of under-performers

For the Public Better definition of what a physician can do at the

completion of training Program is accountable to a common standard Possible use for board certification

17

MinimalistCCC assigns milestone levels using existing tools

ComprehensiveProgram changes assessment system, invests time in faculty development and education to improve rater expertise and CCC validity

PragmaticProgram implements new milestone aligned tools, which help residents understand expectations

Milestones Development Strategy

Process: June 2013Assembled AMIGOS Group from RECCC, faculty, and residents to represent all clinical areas

Survey to rank importance of each Milestone, following which 17 / 25 chosen

Small groups assigned 2-3 Milestones each to develop instruments or methodology

Large group review and feedback

Education office worked with Evalue to incorporate tools into system

General Strategy“Good now is better than perfect later….”

Priorities

Ease of use Should not be burdensome for residents, faculty, RECCC or education

office staff No PAPER ! App for ‘on-the-go’ assessments - procedures, H&P

Implementation feasibility Ready by July 2014 Integrate with existing system (Evalue)

Ensuring adequate data to assess Milestones comprehensively A Milestone for every competency 17/25 Milestones chosen

Work in progress, as refinements will be made and new tools added over time

Which?

Which Milestones did the AMIGOS select?

Medical Knowledge (MK): 1

MK1- Knowledge of biomedical, clinical, epidemiological, and social sciences as outlined in the American Board of Anesthesiology Content Outline

ITEAKTABA Basic ExaminationACLS certificationDirect Clinical Observation

Patient Care (PC): 8 of 10 PC1-Preanesthetic Evaluation, Assessment, and Preparation PC2- Anesthetic choice and conduct PC3- Periprocedural pain management PC4- Management of perianesthetic complications PC5- Crisis management PC6- Triage and management of critically ill patient in a non-

operative setting PC7- Acute, chronic, and cancer related pain consultation and

management PC8- Technical skills: Airway management PC9- Technical skills: Monitoring and Equipment PC10-Technical skills: Regional anesthesia

Example of PC-8 Assessment

PC-8 (Technical skills: Airway management)Direct Clinical Observation (DCO)

Daily evaluation form Airway management checklist

Objective Structured Clinical Examination (OSCE) / Objective Structured Assessment Test (OSAT)

Simulation / Standardized Patient (SP)

Professionalism (Prof): 3 of 5

Prof1- Responsibility to patients, families, and society

Prof2- Honesty, integrity, and ethical behavior Prof 3- Commitment to institution,

department and colleagues Prof4- Receiving and giving feedback Prof 5- Responsibility to maintain personal,

emotional, physical and mental health

Example of Prof-1 Assessment

Prof 1 (Responsibility to Patient, Family, and Society)360 degree evaluations

Patients Healthcare team members Peer evaluations Administrative staff

Direct Clinical Observation (DCO) Daily evaluation form Written comments

Interpersonal and Communication Skills (ICS): 2 of 3

ICS1- Communication with patients and families

ICS2- Communication with other professionals

ICS3- Team and Leadership skills

Practice Based Learning (PBL): 2 of 4

PBL1 Incorporates quality improvement and patient safety initiatives into personal practice

PBL2- Analysis of practice to identify areas in need of improvement

PBL3- Self-directed learning PBL4- Education of patient, family, students,

residents and other health professionals

Systems Based Practice (SBP) 1 of 2

SBP1- Systems-based approaches to patient care

SBP2- Coordination of patient care within the healthcare system

How will we Assess Milestone Levels? Milestones are a summary of how a resident is

progressing based on information from other sources

Milestones are not assessment tools to be used for gathering detailed information

They do not replace end-of-rotation forms, simulation, 360’s, Daily evaluations etc.

More pieces of data allow for more precision

There will be a minimum number of evaluations required for procedural assessments

Clinical Competency Committee (RECCC)

Clinical Competency Committee

End-of-Rotation

Evaluations

Peer Evaluations

SelfEvaluations

Case Logs

Student Evaluations

Patient/Family

Evaluations

ProfessionalismIndicators like

pharmacy, billing, evals

etc

Nursing and Ancillary

Personnel Evaluations

Assessment of Milestones

Clinical Workplace Evaluations

Mock Orals

OSCE

ITESimLab

UnsolicitedComments

Anesthesiology Milestones

Level 4 is the Graduation Target Not every resident will achieve Level 4 in every Milestone

Residents required to substantially meet most Milestones at Level 4

Residents are not expected to achieve Level 5 during residency

Residents may achieve a level of competency in specific Milestones sooner than expected

Milestone Template

PC1. Preanesthetic Evaluation, Preparation

Level 1 Level 2 Level 3 Level 4 Level 5

Performs comprehensive H&P

Identifies medical issues that may affect anesthesia care

Identifies elements and process of informed consent

Identifies medical issues relevant to anesthesia care

Optimizes preparation of non-complex patients

Obtains informed consent but recognizes when assistance is needed

 Identifies medical issues relevant to subspecialty anesthesia care

Optimizes preparation of complex or subspecialty patients

Obtains informed consent of patients with complex problems or requiring subspecialty anesthesia care  

Performs assessment of complex or critically ill patients with conditional independence

 

Obtains informed consent in complicated clinical situations with conditional independence 

 Independently performs assessment for all patients

 Independently serves as a consultant to other members of the healthcare team

Consistently ensures that informed consent is obtained by using all available resources

General Competency Developmental Progression or

Set of Milestones Subcompetency

Milestone

Instruments to Measure Milestones Daily Evaluation Form for OR (Evalue) Daily Evaluation Form for ICU (Evalue) Daily Evaluation Form for Pain (Evalue) Technical Skills forms (Evalue) (‘App’?)

Arterial line CVP Line Nerve Block / Epidural / Caudal Airway Management

Peer evaluations Assessment of Patient Interaction Regional assessments Simulation assessments

‘QUASAR’ Project:Quality in Anesthesiology, Systems Assessment and Research

Resident ‘QUASAR’ Project: Quality in Anesthesiology, Systems Assessment and Research Quality Improvement Systems Based Practice Practice-Based Learning

CA1- Background research, identify area of study

CA2- Identify quality measure to be studied and change to be implemented, study effects, refine

CA3- Write summary, submit abstract to national conference, or write a manuscript for publication

Summary

Next logical extension of competency-based evaluation

Mostly familiar, not burdensome Levels 1 and 2 do NOT signify a ‘Fail’ grade Evaluations will be more specific and focused Direct clinical observation to evaluate

procedures and patient interactions

We Need Your Data !

APOM NEEDS YOU

Acknowledgements AMIGOS

Nicole Conrad Oumou Diallo Judy Freeman Julio Gonzalez Karen Hand Izumi Harukuni Amy Miller Juve Diana Kim Ed Kahl Dean Lao David Larsen Kim Mauer Michele Noles Annie Riley Peter Schulman Katie Seligman Chris Swide Linda Wylie David Wilson Glenn Woodworth