residency review committee for internal medicine … · residency review committee for internal...

Accreditation Council for Graduate Medical Education

© 2013 Accreditation Council for Graduate Medical Education (ACGME)

Residency Review Committee for Internal Medicine (RRC-IM) Update

APDIM Fall MeetingOctober, 2013

Jerry Vasilias, PhD, Executive DirectorJames A. Arrighi, MD, Chair

Program Director, Cardiology, Brown University


APDIM Apr 2013 Oct 2013

“Doubt is an uneasy state from which we struggle to free ourselves and pass into the state of belief.”

Charles Sanders Pierce


Since we last met (Apr 2013)…• NAS is here• Experience with ADS• First milestones reporting due May-Jun 2014• ABIM & AAIM working feverishly in sub milestones• ~ 50 CLER visits done• Changes to RC membership and eligibility rules• ACGME and AOA broke their engagement• RRC

• Old business (new apps, short cycle programs)• NAS prep: data, citations, site visits


The Building Blocks or Components ofThe “Next” Accreditation System (NAS)

10 year Self-Study

10 year Self-Study Visit

prn Site Visits (Program or Institution)

Continuous RRC and IRC Oversight and Accreditation

Clinical Learning Environment Review CLER Visits



NAS ADS


Annual Data Review Elements

The following are the “primary” annual data elements:

1) Program Attrition2) Program Changes3) Scholarly Activity4) Board Pass Rate5) Clinical Experience Data6) Resident Survey7) Faculty Survey8) Milestones

ADS


Annual Data Review Element #3:Scholarly Activity: Faculty (Core)

Pub Med Ids (assigned by PubMed) for articles published between 7/1/2011 and 6/30/2012. List up to 4.

Number of abstracts, posters, and presentations given at international, national, or regional meetings between 7/1/2011 and 6/30/2012

Number of other presentations given (grand rounds, invited professorships), materials developed (such as computer-based modules), or work presented in non-peer review publications between 7/1/2011 and 6/30/2012

Number of chapters or textbooks published between 7/1/2011 and 6/30/2012

Number of grants for which faculty member had a leadership role (PI, Co-PI, or site director) between 7/1/2011 and 6/30/2012

Had an active leadership role (such as serving on committees or governing boards) in national medical organizations or served as reviewer or editorial board member for a peer-reviewed journal between 7/1/2011 and 6/30/2012

Between 7/1/2011 and 6/30/2012, held responsibility for seminars, conference series, or course coordination (such as arrangement of presentations and speakers, organization of materials, assessment of participants' performance) for any didactic training within the sponsoring institution or program. This includes training modules for medical students, residents, fellows and other health professionals. This does not include single presentations such as individual lectures or conferences.

Faculty Member

PMID 1

PMID 2

PMID 3

PMID 4

Conference Presentations Other Presentations Chapters /

TextbooksGrant

LeadershipLeadership or Peer-

Review Role Teaching Formal Courses

John Smith 12433 32411 3 1 1 3 Y N

RC-IM Expectation/Threshold: Within the last academic year, at least 50% of the program’s “core” faculty need to have done at least one type of scholarly activity from the list of possible activities in the table above.


Annual Data Review Element #3:Scholarly Activity: Residents

Pub Med Ids (assigned by PubMed) for articles published between

7/1/2011 and 6/30/2012. List up to 3.

Number of abstracts, posters, and presentations given at international, national, or regional meetings between 7/1/2011 and 6/30/2012

Number of chapters or textbooks published between 7/1/2011 and 6/30/2012

Participated in funded or non-funded basic science or clinical outcomes research project between 7/1/2011 and 6/30/2012

Lecture, or presentation (such as grand rounds or case presentations) of at least 30 minute duration within the sponsoring institution or program between 7/1/2011 and 6/30/2012

Resident PMID 1 PMID 2 PMID 3

Conference Presentations

Chapters / Textbooks Participated in research Teaching / Presentations

June Smith 12433 1 0 N Y

RC-IM Expectation/Threshold: At least 50% of the program’s recent graduates need to have done at least one type of scholarly activity from the list of possible activities in the table above.

Although the form itself indicates that data entry is for only a single year, IM was granted an exception to allow entry of scholarship data once upon completion of training, reflecting scholarship performed for the entirety of training.


Annual Data Review Element #3:Scholarly Activity: Residents

Although the form itself indicates that data entry is for only a single year, IM was granted an exception to allow entry of scholarship data once upon completion of training, reflecting scholarship performed for the entirety of training.

• Communicated through AAIM/APDIM• Special instructions (IM only) were

outlined on ADS


Milestones


Example of Reporting Milestone (Core IM)

Milestone

Developmental Progression or Set of Milestones

Sub-Competency

Competency Committees + Milestones

Clinical Competency Committee

End of Rotation Evaluations

Peer Evaluations

SelfEvaluations

Case Logs

Student Evaluations

Patient / Family Evaluations

Operative Performance Rating Scales

Nursing and Ancillary Personnel Evaluations

Assessment of Milestones

Clinic Work Place Evaluations

Mock Orals

OSCE

ITESimLab

UnsolicitedComments


What specific elements of the system are ACGME?

Curricular Milestones

EPA’s

Specific curriculum

EvaluationSystem

ReportingMilestones

ACGME(and ABIM)

PartlyACGME

NotACGME


Reporting Milestones• De-identified, aggregate (program) data will

gradually be used as one element of accreditation decisions

• Individual reports by trainee will be provided to PD• In time, goal is to make final reports available to

fellowship PD’s upon matriculation of the graduated resident into his/her fellowship program

• Semiannual reporting remains a foundation of NAS


Reporting Milestones: Timeline• First reporting period (AY 2013-14):

• May 1 – June 15, 2014• In AY 2014-15, IM programs will be reporting

twice annually• First window: November 1 – December 31• Second Window: May 1 – June 15

• For those interested…• Test-Run = November 1-December 31, 2013• Data entered will not be used for accreditation

matters or trending reports and will be purged


Reporting Milestones: Med-Peds

• Semiannual evaluation process as usual• ANNUAL reporting of the reporting

milestones• Report BOTH medicine and pediatrics

milestones


Milestones: Use by RC-IM

• Cannot be fully used for several years• Will be one important aspect of

accreditation status• Initially, ascertain that programs are

reporting• Next, check for completeness of data, etc• Ultimately, indentify patterns indicating

need for program improvement


Sidebar on Subspecialty Milestones


Subspecialty Reporting Milestone Development

Reporting Milestones

ABIM

AAIM

SubspecSocieties

Competencies (6)Competencies (6)

Sub-Competencies (n = ??)Sub-Competencies (n = ??)

Reporting Milestones (5 per sub-competency)Reporting Milestones (5 per sub-competency)


Subspecialty Reporting Milestones

• Draft completed by working group• All specialty societies represented• Aim for finalizing by Dec 2013• Current draft:

• Scholarship subcompetency added• Other subcompetencies are

extensions/adaptations of IM milestones• Individual specialties will have opportunity to

edit


CLER Visits


CLER Program• Clinical Learning Environment Review• Institutions will be visited every 18-24

months• Formative evaluation, not judgement• Data will not be used for accreditation,

but…….• Programs must ensure that residents and

fellows are aware of and participating inpatient safety/quality improvement efforts of the institution


Change to RC-IM Membership

• Public member to be added• Non-MD• Specific profile defined by RC


Changes to Eligibility Rules• Approved at last ACGME BOD meeting• Effective date 7/1/2016


Changes to Eligibility RulesResidency Eligibility

• Any re-requisite training (for entry or transfer) must be done in programs accredited by:• ACGME • Royal College of Physicians & Surgeons• College of Family Physicians of Canada

• If a physician has completed an IM residency not accredited (by above), they may enter at PGY1 level and be advanced (early) to PGY2 level based on milestones assessments at PD discretion.

• No other exceptions for residency


Changes to Eligibility RulesFellowship Eligibility

• Any re-requisite training (for entry or transfer) must be done in programs accredited by:• ACGME • Royal College of Physicians & Surgeons• College of Family Physicians of Canada

• Exceptions may be allowed by RC’s• RC-IM has not voted yet

• RC-IM’s board take/pass rate PR is present (“outcome”)


Changes to Eligibility RulesFellowship Eligibility Exceptions

• Qualifications:• Selection committee assessment• Review/approval/oversight by GME office• Completion of USMLE 1, 2, and if applicable 3• ECFMG verification if applicable• Milestones assessment at 6 weeks after entry

• Remediation needed?


Changes to Eligibility RulesFellowship Eligibility Exceptions

• What constitutes “exceptionally qualified”?:• Completed a non-ACGME core residency• Demonstrated clinical excellence• Additional potential evidence includes:

• Additional clinical or research training• Scholarship• Leadership• ACGME-international program graduates


Decisions, Decisions


When is my program reviewed?

• Each program is reviewed annually• NAS is a continuous process• Annual data supplemented by:

• Reports of self study visits• Progress reports (when requested)• Reports of site visits (as necessary)

• Cycle lengths not used• Feedback given to program annually


New Applications

New Application

New Application

InitialAccreditation


AccreditationWithheld

AccreditationWithheld

Reapplicationanytime or Appeal

Reapplicationanytime or Appeal


Initial Accreditation

Initial accreditation SV in 2 yrs

Initial accreditation SV in 2 yrs

Continued accreditation (CA)

Continued accreditation (CA)

Initial accreditation w/ warningSV in 1 yr

Initial accreditation w/ warningSV in 1 yr

CACA

Withdrawal of accreditation (WD)

Withdrawal of accreditation (WD)

WDWD


Continued Accreditation

Accredited Program

Accredited Program

Continued accreditation

(CA)

Continued accreditation

(CA)

CA w/ warningCA w/ warning

Other (e.g. egregious)Other (e.g. egregious)

Site visitSite visit

Probationary accreditationProbationary accreditation

Withdrawal of accreditationWithdrawal of accreditation

CA w/ warningCA w/ warning

CACA

Note: After SV, any status may be granted (including withdrawal)


Citations

• Citations are given only by RC committee (not by staff)

• Require response in ADS as long as they are active

• Require an RC member to review annually• Removed once issue is solved

• Annual data• Progress report• Site visit


Areas for Improvement• May be given or removed by staff (RC

rules) or RC-IM• Do not require ADS response• Staff will know there is an “area of

improvement” on annual data review• Work in progress


• No site visits (as we know them)

but…

• Focused site visits for an “issue”

• Full site visit (no PIF)

• Self-study visits every ten years

NAS: What’s Different?


Focused Site Visits• Assesses selected aspects of a

program and may be used:• to address potential problems identified

during review of annually submitted data;• to diagnose factors underlying

deterioration in a program’s performance

• to evaluate a complaint against a program


Focused Site Visits• Minimal notification given (~ 1 month)

• Minimal document preparation expected

• Team of site visitors

• Specific program area(s) looked at as

instructed by the RRC


Full Site Visits• Application for a new core program

• At the end of the initial accreditation period

• RRC identifies broad issues/concerns

• Other serious conditions or situations

identified by the RRC

• Notification given ~ 60 days

• Minimal document preparation

• Team of site visitors


• “Cycle Lengths” will not be given – that’s OAS, not NAS• Citations may be given or removed• Areas for Improvement may be given

• Areas for Improvement are different from citations• Will not be reviewed annually by RC• Are not necessarily linked to a PR• Programs do not need to provide response in ADS• RC will monitor whether addressed using annual data

• Status Options:Continued AccreditationAccreditation with WarningProbationary Accreditation *Withdrawal of Accreditation *


What happens after data are reviewed?

* Status conferred only after a site visit.


What is a self-study visit?

NAS and Self-Study Visits



What is a Self-Study Visit?

• Not fully developed

• Scheduled every ten years

• Conducted by a team of visitors

• Minimal document preparation

• Interview residents, faculty, leadership



What is a Self-Study Visit?• Examine annual program evaluations

• Response to citations• Faculty development

• Focus: Continuous improvement in program• Learn future goals of program• Will verify compliance with core

requirements



Ten Year Self-Study Visit

Yr 0 Yr 1 Yr 2 Yr 3 Yr 4 Yr 5 Yr 6 Yr 7 Yr 8 Yr 9 Yr 10

Self-StudyVISIT

Self-Study Process

AE AE AE AE AE AE AEAE AE AE


Accreditation & Innovation

• Program Requirements (PRs) classified:• Outcome• Core• Detail

• Programs in good standing:• May freely innovate in detail standards



The program director must appoint the Clinical Competency Committee. (Core)

• At a minimum the Clinical Competency Committee must be composed of three members of the program faculty. (Core)

• There must be a written description of the responsibilities of the Clinical Competency Committee. (Core)



The Clinical Competency Committee should:

review all resident evaluations semi-annually; (Core)

prepare and assure the reporting of Milestones evaluations of each resident semi-annually to ACGME; and, (Core)

advise the program director regarding resident progress, including promotion, remediation, and dismissal. (Detail)


Clinical Competency CommitteeComposition

• PR’s do not specify composition; each program may decide best structure

• PR’s do not limit PD’s role• PR’s do not define specialty, degree,

role for members of CCC• “Best practices” may be defined by

community• New FAQ’s are posted


Where did the NAS annual data elements come from?


Annual Data Review Elements

The following are the “primary” annual data elements:

1) Program Attrition2) Program Changes3) Scholarly Activity4) Board Pass Rate5) Clinical Experience Data6) Fellow Survey7) Faculty Survey8) Milestones


Where did the NAS annual data elements come from?

Analysis to determine what combination of data elements may predict a “problem” program.

History of prioraccreditation decisions

Data analysis & modeling

Adequate sensitivityMinimize false negative and positives

Importance of trends

Understand that this is a…

New data elements will likely be introduced in future.


Data Modeling• In 2009, data modeling project began to

identify factors that predicted high and low program performance

• A number of statistical methods were used• Model was replicated using more recent

data• Indicators were assessed to determine

“relative risk” to predict low performance


Selection of Indicators

• Obtainable• Meaningful• Correlates w/ prior decisions• Passed statistical “muster”• Used in combination


Categorizing Programs Using Indicators


Accredited Programs

N = 9,022

Good (95%)Borderline (4%)Problem (<1%)


1 “Real Issue” defined as most recent accreditation decision (up to 5 years earlier)

Indicators (detected issue)Screening Detected Concerning Parameters

NO YES

None 1 2 ≥3

Continued Accreditation 2%

Accreditation w/Warning 2%

PreviousReal Issue1

NO

YES

What happened when predictions from NAS data were compared to actual accreditation decisions

(IM core programs)?


Indicators (detected issue)Screening Detected Concerning Parameters

NO YES

None 1 2 ≥3



New Problem since last review: 8 of 9 programs have most recent Certification Board Pass Rate below 80%

PreviousReal Issue1

NO

YES

Using Indicators to Screen ProgramsIM Core Programs


Using Indicators to Screen ProgramsIM Core Programs

Indicators (detected issue)Screening Detected Concerning ParametersNo Yes

None 1 2 ≥3



Improving8 of 9 have since been reviewed by RRC-IM. Issued Continued Accreditation, Cycle Length >2 years.–no longer on warning

PreviousReal Issue1

NO

YES


Work of the RRC in NAS• Utilize data and judgment to:

• Concentrate efforts on problem programs

• Determine whether accreditation standards are violated and provide useful feedback for programmatic improvement

• Whether these violations (citations) rise to a level requiring alteration in accreditation status

• Motivate programs to rapidly improve, rather than playing the “accelerating accreditation action game”

• Over time, understand and refine the nuances of the process

• Conduct complete review of the program q10 years, using a “PIF-less”, team based, department wide evaluation of programs


Thank you!


Conceptual Model of NAS

ProgramRequirements

Core ProcessDetailed Process

Outcomes


New Programs

Core ProcessDetailed ProcessOutcomes

Withhold AccreditationWithdrawal of AccreditationWithhold AccreditationWithdrawal of Accreditation

2-4% 15% 75%

6-8%

Accreditation w/ Warning

Accredited Programs with Major Concerns

Probationary Accreditation


ContinuedAccreditation

Accredited Programs without Major Concerns



NAS = Innovation withoutpermission.

It should allow you to “innovate”


Categorization of Program Requirements (Example of IM)

Majority of Common PRs -- “core” Majority of IM PRs -- “detail”

Common Program RequirementsTotal # %

Core 89 45%Detail 66 34%Outcome 42 21%

IM Program RequirementsTotal # %

Core 56 34%Detail 83 51%Outcome 24 15%


RRC-IM

• 3 nominating organizations - ABIM, ACP, AMA• Currently 20 voting members • 6 year terms -- except resident (2 years)• Generalists and subspecialists

Cardiology, CCEP, Critical Care Medicine, Endocrinology, Gastroenterology, General Internal Medicine, Geriatric Medicine, Hematology/Oncology, Infectious Disease, Medicine-Pediatrics, Nephrology, Pulmonary/Critical Care Medicine, Rheumatology, Sleep Medicine, Transplant Hepatology

• Ex-officio members from each nominating organization (non-voting)


Who is the RRC-IM?James A. Arrighi, MD – Chair

Beverly M.K. Biller, MD

Robert Benz, MD

Christian Cable, MD

Andres Carrion, MD

Gates Colbert, MD

E. Benjamin Clyburn, MD – Vice-Chair

John Fisher, MD

Andrew S. Gersoff, MD

Lynne Kirk, MDBetty Lo, MD Brian Mandell, MDFurman McDonald, MD Elaine A. Muchmore, MD Susan Murin, MD Victor J. Navarro, MD Andrea Reid, MD Ilene Rosen, MD Stephen M. Salerno, MD Jennifer C. Thompson, MD


Information on NAS: http://www.acgme-nas.org/

residency review committee for internal medicine … · residency review committee for internal...

Documents