Bridge over Troubled WaterLinking UME and GME

• Monica L. Lypson, MD Assistant Dean, Graduate Medical Education

University of Michigan

Jeff Fabri, MD

Rita M. Patel, MD

Education Across the Continuum Standardization of Assessment

Student Portfolios to Baseline Assessment

• Monica L. Lypson, MD Assistant Dean, Graduate Medical Education

University of Michigan

• Associate Chief of Staff, VA Ann Arbor Health Care System

Institutional OSCE – Post-Graduate Orientation Assessment (POA)

Developed by the Graduate Medical Education Committee (GMEC) at the University of Michigan

It is our initial step in training our residents

It was established to determine residents’ baseline proficiency in particular aspects of the ACGME’s six general competencies

Lypson ML. et.al. Academic Medicine. 79(6):564-70, 2004 Jun.

POA CONTENT

Knowledge and skills needed during the first six to eighteen weeks of residency/internship

Emphasizes clinical situations that are often encountered without formal supervision

9 Assessment & Educational Stations

The POA as Formative Assessment

Results of the POA determine the basis for individualized “learning agendas”

Remediation is provided after the completion of each station Residents received educational materials that provide the

“answers” to the information assessed during the POA

Standardized Patient feedback is provided to the program director within 24 hours if the resident performs exceptionally well or poorly during the POA

Implementation

Administered over 4 days of Paid hospital orientation

Provided to approximately 150 PGY-1 residents in over 15 specialties

Cost Approximately $250 per resident for the assessment - $500 if you add in salary

There are some discipline specific scenarios – e.g. Pediatric cases and examples

Scores are provided to the resident and program director within 7-10 days of the POA

Subspecialty Programs

• Dentistry• Emergency Medicine• Family Medicine• General Surgery

– Preliminary Residents

• Internal Medicine– Preliminary Residents

• Internal Medicine – Pediatrics

• Neurosurgery• Obstetrics & Gynecology

• Otolaryngology• Orthopedics• Pathology• Pediatrics• Plastic Surgery• Psychiatry• Urology• PGY-2s (Residents that did not complete their Internships at UMHS)

– Physical Medicine & Rehabilitation– Dermatology– Neurology

• Critical Values / Multiple Choice – Computer Based assessment – Review and diagnose 17 short

patient scenarios– Multiple Choice Format

similar to United States medical Licensing Exam (USMLE) Step 3

– Competencies:• Patient Care• Medical Knowledge

• Socio-Cultural Communication – Standardized patient assessment – Assesses the understanding of

disease and treatment recommendations in the context of the patient’s health beliefs and socio-economic setting

– Competencies:• Interpersonal and

Communication Skills• Professionalism • Patient Care• Practice-Based Learning and

Improvement

Station Content

Station Content

• Evidence-Based Medicine (EBM) – Computer-based

Assessment– Generate a clinical

question– Residents reviewed

abstracts & identify the appropriate treatment

– Competencies:• Practice-based

Learning and Improvement

• Medical Knowledge

• Images (X-Rays)– Computer-based

Assessment– Review & diagnose 18

images– Many of the common

films reviewed in the middle of the night while “on call”

– Competencies:• Patient Care• Medical Knowledge

Station Content

• Informed Consent– Standardized patient

Assessment

– Obtain informed consent from a patient for a procedure

– Hospital & JCAHO standards and policies

– Competencies:

• Interpersonal and Communication Skills

• Professionalism

• Patient Care

• Patient Safety– Pen & Paper Assessment

– Order Writing Station

• Legibility

• Signature, Date, Time

– Respiratory Distress

• Treatment of the acute Asthmatic

– Competencies:

• Systems-Based Practice

• Professionalism

• Patient Care

• Medical Knowledge

Station Content

• Aseptic Technique – Checklist Evaluation by Expert Nursing Staff and Standardized

Patients– Create & maintain a sterile field while performing a “mock” I&D– Universal Protocol & “Time-out” procedures covered– Assessment/Remediation materials include a review of JCAHO

requirements – Improved nursing and house officer interactions– Competencies:

• Patient Care• Medical Knowledge• Systems-Based Practice

Station Content

• System Compliance / Fire Safety Station– Surgical Fire Safety (20/20)

– Housestaff involvement with a Patient fire

– Questions covering use of safety equipment & review of JCAHO requirements

– Video & Computer Based Assessment

– Competencies:• Patient Care

• Systems-Based Practice

• Pain Assessment– Pain Assessment Tools

– Educates on the appropriate medications to use for pain

– Explains hospital and JCAHO Pain assessment mandates

– PowerPoint & Computer Based Assessment

– Competencies:

• Patient Care

• Medical Knowledge

Resident Satisfaction with the POA-Survey Response Rate 93%2002-2004

Question YES

Have you learned any new clinical skills during this assessment?

70.6%

Do you think this was a useful way to spend part of orientation?

83.4%

Do you feel better prepared for some aspects of internship after this assessment?

80.6%

Do you expect to refer back to the teaching materials handed to you today?

84.7%

Would you recommend that we continue the POA next year?

84.7%

The POA was based on the following…

The University of Michigan

Comprehensive Clinical Assessment

(CCA)

Comprehensive Clinical Assessment 1991-Present

• A high-stakes examination for University of Michigan Medical Students

• Fourth Year students must pass the CCA in order to Graduate

• 10-15 stations over ~4-5 hours• ~150 M4 students per year• Summative Evaluation• Standards have been set for Pass/Fail• Remediation for failures

Rochester AB. et.al.. Academic Radiology. 5(3):169-72, 1998

Summary of Station Comparison:CCA & POA

POA- GME

• Formative

• Informed Consent & Policy

• Pain Assessment

• Socio-Cultural Communication

• Evidence Based Medicine

• Images

CCA-UGME

• Summative

• History & Physical Diagnosis

• Geriatrics

• Socio-Cultural Communication

• Evidence Based Medicine

• Images

Institutional Implications for GME & UGME

• Training the Faculty in the Teaching Skills of competency assessment– Medical School Objectives Project (MSOP)– Accreditation Council of Graduate Medical Education

• Reimbursement of faculty time and effort• Payment for both assessments – Medical

School vs. Hospital– Is the medical school or the hospital responsible for ‘educational

cost’?

• Trainee awareness of “competency gap” between the expectations of medical School and residency

POA Demographics 2002-2004

Gender

Male 54%

Race

White / Non-Hispanic Underrepresented Minorities

Other

70.1%

6.5%

23.4%

Medical Schools

Public

International

62%

1.5%

United States Medical Licensing Examination (USMLE)

Step 1

Step 2231

233

The University of Michigan Health System Plan for Competency Based Resident Education

• Joint Hire faculty member with the Department of Medical Education

• Centralized OSCE – Post-Graduate Orientation Assessment (POA)

This is based on the system already in place for Undergraduate Medical Education at UMHS.

The Department of Medical EducationChairman: Larry Gruppen, PhD

• One of ~5 departments of medical education at LCME accredited medical schools

• The department had focused on CME and UGME in the past with assessment and research expertise

• The office of Graduate Medical Education and the Department of Medical Education

– Joint Hire – Job Description

• 50% appointment GME office, 50% appointment Dept. of Med. Ed.• Aid Residency programs with curriculum development, core competency

assessment• Use this work as a vehicle of scholarly pursuit

– The department has a long history of UGME – now will develop its expertise in GME