using medhub and process streamlining to make ......using medhub and process streamlining to make...

Using MedHub and Process Streamlining to Make Organizing & Documenting GME / Accreditation Data Easy

Nancy Piro, PhD – Program Manager/Ed Specialist Stanford GMEBrett Toensing, MS – MedHub/Web Coordinator

Confidential – For Discussion Purposes Only

AGENDA – Leveraging MedHub

Process Streamlining Tips & Tricks

Setting Priorities

− Citation Analysis

Program Evaluations

− Standardizing Delivery and Documentation

Summative Evaluations

− Incoming and Outgoing Trainees

− Standardized Template and Electronic Filing

− Program Needs Analysis

Organizing data for:

− CCCs

− APEs / Self-Studies

Confidential – For Discussion Purposes Only

Thinking Outside the Box – Looking to Industry and Corporations

What does industry have that we can use?

− Conveyor belt to move patients in and out of OR’s? …..No……

But they do use a “Lean Philosophy”

− Respect for People

− Total Elimination of Waste

And they use “Lean Tools” such as

− Kaizen

− 3 M’s

− 5S’s

Confidential – For Discussion Purposes Only

Lean – Toyota Production System (TPS)

TPS system managed to get by with “half of everything”!

− Physical space

− Personnel

− Capital Investment

− Inventory

Resulted in far fewer than half the defects and safety incidents

Make problems visible – not hide them or “not talking about mistakes/problems”

Fix problems permanently – get to the “root cause” and eliminate it.

Focus on the value of people and respect for people.

Confidential – For Discussion Purposes Only

Sample of “Lean Tools” … Kaizen

What is ‘Kaizen’?

Gradual, unending continuous improvement of processes

Processes must be improved to get improved results

By improving and standardizing activities and processes, Kaizen aims to eliminate waste

Focus is on small, incremental change (not necessarily huge leaps, innovation)

Quality Control = Quality of People

− “A company that is able to build quality into its people is halfway to building quality outcomes”

Respect for People

− “Only people produce improvements”

Confidential – For Discussion Purposes Only

Lean Tools: - 3M’s

All about Waste- Identifying and Removing it

− MURI = Waste of overburdening people or equipment/resources

− MURA = Waste of unevenness, variability in processes

− MUDA = Waste of using resources without creating added value

Confidential – For Discussion Purposes Only

Five Why’s Problem Solving Method

Keep asking ‘Why?’ until you discover the root cause of the problem

− No magic in 5 –

might be 3, or 7, or 10

Why do we? (…conduct orientation in person, fill out multiple forms, have residents take on line training for non MDs? ….)

Combine with Lean Tools: – Fishbone Charts using the 4Ms – 4 Ps

Manpower/Personnel

Materials

Method(s)

Machines / Equipment

People

Process

Policies

Principles

Confidential – For Discussion Purposes Only

Ishikawa Diagram of ER Prolonged Wait Times

Materials Methods/Process

Manpower/People Machine \ Equipment

WHY? WHY?

WHY? WHY?

xxx …wait time in the ER

Confidential – For Discussion Purposes Only

5Ss - Mnemonics Retained

Sorting

Simplifying

Sweeping

Standardizing

Self Discipline

= Seiri

= Seiton

= Seiso

= Seiketsu

= Shitsuke

QUICK EXERCISE

Confidential – For Discussion Purposes Only

5Ss in Action

10

Confidential – For Discussion Purposes Only

AFTERBEFORE

ER Rooms

Confidential – For Discussion Purposes Only

Putting Philosophy into Practice

Flowcharted the processes

− Asked: “Why are we doing this?”

‘Fish boned’ the problem

Looked for the 3Ms

Applied the 5Ss

Confidential – For Discussion Purposes Only

CURRENT CULTURE

96 programs (96 fiefdoms all doing their own thing….but having common requirements)

Lack of standardization/inconsistencies (MURA)

Overburden (MURI)

Waste (MUDA) of both materials and time

− Lots of files

− Time to file personnel folders or

ACGME letters

Time-consuming institutional oversight of programs (MUDA, MURI)

Confidential – For Discussion Purposes Only

Standardizing Program Evaluations in MedHub

Factors driving decision to standardize our Program Evaluations:

− Largest number of program citations from ACGME/RRCs

− Lack of consistent data on program evaluation for APEs (MURA)− Huge amount of paper generated to produce suboptimal evaluations (MUDA)− Burden of work on the coordinators (MURI)

What did we do?

Developed standardized comprehensive core competency-based Program Evaluations – by Faculty and Trainees

Presented the draft templates to the Program Directors who edited and approved them.

Confidential – For Discussion Purposes Only

Standardized Program Evaluations – Process Implementation

Then implemented - Annually the GME Department:

− Delivers (via MedHub, of course!)

>190 Program evaluations to > 1500 trainees and faculty

Tracks Evals Delivered for each program and Average Results for each Eval

Aggregates each program’s data after the evals have been completed

‘Prints’ (electronically to pdfs) their aggregate Program evaluation reports

Posts the Program Eval Reports (pdfs) on each Program’s APE site in MedHub

Confidential – For Discussion Purposes Only

Program Evaluation Standardization Example

Confidential – For Discussion Purposes Only

Standardization of Program Evaluations

Benefits

− Fewer citations

− Can be input to our Institutional Report Card

− Easy oversight by DIO/GMEC

− Early warning system

Confidential – For Discussion Purposes Only

Summative Evaluations – What did we do?

Developed Standard Evaluation Template for Summative Evaluations

Comprehensive over program training years

Core-competency based

User friendly word document

Held Program Director and Program Coordinator Teaching Sessions / Workshops

Taught the use of the template and how to pair it with aggregated evaluation data MedHub

Programs could select the evaluations to use and aggregate–

− We recommended:

Evaluations Competencies by Resident

Resident/Faculty Ranking Report (aggregate)

Aggregate Comments Report

Evaluation Competencies Report

Confidential – For Discussion Purposes Only

Coordinate with Aggregated Evaluation Data

Confidential – For Discussion Purposes Only

Summative Evaluation Template

Confidential – For Discussion Purposes Only

Macro-enabled Word Doc

Confidential – For Discussion Purposes Only

Summative Evaluations - Additional Considerations

Special Cases

− At the end of a preliminary year (internship)

Need to list Rotations

Review report with trainee

Place a copy of the summative evaluation in the Trainee’s permanent file and upload to MedHub

We also obtain Summative Evaluations from our trainees coming in from another GME program for all the programs and upload them to MedHub also

Confidential – For Discussion Purposes Only

Switching Gears From Summatives to CCCs

CCCs

Summatives

Confidential – For Discussion Purposes Only

CCCs

CCC MedHub Online Functionality –

− Assign Files / Aggregated Evaluations to Program Faculty for Pre-Work

− Upload all required data for Preliminary Review and the CCCX meeting

Confidential – For Discussion Purposes Only

Pulling the Data Together

Clinical Competency Committee

End-of-Rotation

Evaluations

Safety Incident Reports

Case Logs

Patient/ Family

Evaluations

Clinical Skills

Assessment

Nursing and Staff / Techs

EvaluationsProgress on Milestones

SimLab

In service In-service training exams

Quality Improvement

Activities

Confidential – For Discussion Purposes Only

Data used in CCC meetings for trainee assessment

26

65.38%

93.85%

76.92%

44.62%

63.08%

36.92%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Aggregate DirectMilestone

Evaluations

Aggregate RotationEvaluations

Staff or patient(360) Evaluations

Technical SkillsAssessments/Mini-

CEX's

Scholarly Activities

Case Volumes

Confidential – For Discussion Purposes Only

EvaluationsMilestone End of Rotation (Total)Medical KnowledgePatient CareInterpersonal & Communication SkillsProfessionalismSystems-Based PracticePractice-Based Learning & Improvement

Creating a Resident Performance ProfileStep 1 – Defining what to track

Confidential – For Discussion Purposes Only

In-service Assessments (MK; PC)Routine procedure technical skills assessment: Level 2-3Complex procedure technical skills assessment: Level 3-4Medical Knowledge Assessments

Creating a Resident Performance ProfileStep 1 – Defining what to track

Confidential – For Discussion Purposes Only

Case Logs / Clinical Experience (PC)VAGINAL DELIVERYCAESAREAN SECTIONPEDIATRICSPEDIATRICS UNDER 3CARDIACENDOVASCULAR

Creating a Resident Performance Profile Step 1 –Defining what to track

Confidential – For Discussion Purposes Only

Quality Improvement/Patient SafetyQI ProjectQI Committee ParticipationQI Course Work (e.g., IHI)SAFE Report / Adverse Event ReviewPatient Handover Evaluations

Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only

Practice-Based LearningPatient outcomes / Case Study PresentationScholarly Activity: Research studyScholarly Activity: PublicationsPresentation at Internal and National Meeting

Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only

CommunicationPatient Feedback Staff EvaluationMedical Student FeedbackPatient Handover Evaluations

Creating a Resident Performance Profile Step 1 – Defining what to track

Confidential – For Discussion Purposes Only

Creating a Resident Performance ProfileStep 2 - Link data sources to milestones

Evaluations MilestonesMilestone End of Rotation (Total) 24Medical Knowledge MK A1Patient Care PC A1-10Communication ICS A1-2Professionalism PROF 1-5Systems-Based Practice SBP A1-2Practice-Based Learning PBLI A1-4Quality Improvement/Patient SafetyQI Project PBLI1-2QI Committee Participation PBLI1-2QI Course Work (e.g., IHI) PBLI1-2SAFE Report / Adverse Event Review PROF1/2; SBP1Patient Handover Evaluations SBP1; ICS1

Confidential – For Discussion Purposes Only

Milestone Data Translation to Numerical Data

Confidential – For Discussion Purposes Only

Creating a Resident Performance ProfileStep 3: CCC defines performance ranges

At or Above Expectation: 2.8 and higher

Below Expectation:1.7 – 2.7

Remediation:Below 1.7

Example:

For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges:

Confidential – For Discussion Purposes Only

Creating a Resident Performance ProfileStep 4 – Set conditional formatting

Confidential – For Discussion Purposes Only

Example: Aggregate milestone evaluation data cells

Highlight cells to apply the conditional formatting

Creating a Resident Performance ProfileStep 4 – Set conditional formatting

Confidential – For Discussion Purposes Only

Click on “Conditional Formatting”“Highlight Cells Rules”

Creating a Resident Performance ProfileStep 4 – Set conditional formatting

Confidential – For Discussion Purposes Only

Select, “Greater Than” “Less Than” or “Between” to Set Value RangesChoose the corresponding fill color (e.g., red, yellow, green)

Creating a Resident Performance ProfileStep 4 – Set conditional formatting

Confidential – For Discussion Purposes Only

Creating a Resident Performance ProfileCompiling and centralizing data

Confidential – For Discussion Purposes Only

Resident Performance Profile:Step 5: Enter in data

Confidential – For Discussion Purposes Only

Creating a Resident Performance ProfileVisual Trends and Detailed Data

Confidential – For Discussion Purposes Only

Creating a Resident Performance Profile-Visual Trends and Detailed Data: Another Example

Confidential – For Discussion Purposes Only

There was no way we could have had an effective CCC meeting without completing pre-work.

44

90.58%

8.07%

1.35%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

TRUE

FALSE

Notapplicable

Confidential – For Discussion Purposes Only

MedHub CCC Functionality – Resources and Files

Confidential – For Discussion Purposes Only

Leveraging More MedHub Functionality

Conference attendance statistics

− Core competencies linked to specified conferences

− Attaching conference materials for later reference

Procedures and levels; linked procedure evaluations

Confidential – For Discussion Purposes Only

• Resident portfolio toolsQI participation and outcomesScholarly Activity logs

Leveraging MedHub Functionality

Confidential – For Discussion Purposes Only

Leveraging MedHub Functionality

Aggregate reporting and graphic summaries

Peer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores

Confidential – For Discussion Purposes Only

PGY 1 Ranked at Graduation Level

1

Confidential – For Discussion Purposes Only

When Remediation is Evident

Confidential – For Discussion Purposes Only

Switching Gears From CCCs to APEs

CCCs

APEs

Confidential – For Discussion Purposes Only

Program Evaluation Committee (PEC) Must Monitor and Track (V.C.2):

1. RESIDENT PERFORMANCE

2. FACULTY DEVELOPMENT

3. GRADUATE PERFORMANCE

4. PROGRAM QUALITY

5. PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN

Confidential – For Discussion Purposes Only

RESIDENT PERFORMANCE

The most recent aggregated written evaluations of the residents submitted by faculty and other evaluators

In-training/In-service exam scores

Procedure logs (if applicable)

Scholarly activity (publications, presentations, grant awards, etc.)

Learning portfolios: documented quality improvement activities

Confidential – For Discussion Purposes Only

FACULTY DEVELOPMENT

ABMS certification status for all faculty

Updated faculty CVs

Documentation (faculty survey; attendance logs) of faculty participation in:− CME-type activities directed toward acquisition of clinical

knowledge and skills and also activities directed toward developing teaching abilities, professionalism, and abilities for incorporating the core competencies into practice

− Teaching (conferences, grand rounds, journal clubs, lecture-based CME events, workshops, directed QI projects, practice-improvement self study).

Faculty actively involved in mentor relationships with residents/fellows.

Confidential – For Discussion Purposes Only

GRADUATE PERFORMANCE

Aggregated board exam pass rates

Aggregated alumni survey results (typically, such surveys target physicians one year and five years after graduation –survey questions may inquire about such items as current professional activities of graduates and perceptions on how well prepared they are as a result of the program)

Other outcome measures

− Practice location (underserved areas)

− Academic Affiliations

− Scholarly Activity

Confidential – For Discussion Purposes Only

Alumni Surveys …

Confidential – For Discussion Purposes Only

PROGRAM QUALITY

The most recent aggregated written evaluations of the program submitted by faculty

The most recent aggregated written evaluations of the program (and/or specific rotations) submitted by residents

The most recent aggregated written evaluations of the faculty submitted by residents

Faculty’s recent scholarly activity (publications, presentations, grant awards, etc.)

Confidential – For Discussion Purposes Only

PROGRAM QUALITY - Continued

The most recent ACGME survey results

The most recent GME House Staff survey results

The most recent GMEC Internal Review Report

Any recent communications from the ACGME or RRC

Program Report Card/Scorecard

− Trend Analyses

Confidential – For Discussion Purposes Only

PROGRAM QUALITY - Continued

Curriculum

Overall and rotation-specific goals and objectives (Are they appropriate? Do they align with the core competencies?)

Didactic curriculum (Is there at least one regular conference targeted to the residents’ level?)

Opportunities for scholarly activity

Compliance with any new standards established by the ACGME, RRC, ABMS, etc.− Assessment Methods (Are evaluation tools appropriate? Do they align with

the core competencies?)

− Resources: Personnel (PD, PC, faculty), Affiliated Training Sites, Patient/Procedure Volume,

Learning Environment (space, call rooms, books, computers, etc.)

Confidential – For Discussion Purposes Only

PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN

Review progress / (attempts to resolve problems) with respect to last year’s Annual Review delineating identified areas of weakness.

Confidential – For Discussion Purposes Only

Now… SWOT Analyses are being required

STRENGTH

OPPORTUNITY

OPPORTUNITY

THREATS

Confidential – For Discussion Purposes Only

APE “SWOT” Analysis Tool - Fishbone / IshikawaPROGRAM

Confidential – For Discussion Purposes Only

Program Aims – ACGME PerspectiveThe AIM setting is part of the annual program evaluation

Relevant considerationsWho are our residents/fellows?What do we prepare them for? Academic practice Leadership and other rolesWho are the patients/populations we care for?

AIMS are a way to differentiate programs Self-study will ultimately evaluate program effectiveness in

meeting these aimsMoves beyond improvement solely based on compliance with

minimum standards Assessment of relevant initiatives and their outcomes

Confidential – For Discussion Purposes Only

Strengths and Weaknesses – Internal Factors

Strengths

Program factors that are likely to have a positive effect on (or be an enabler to) achieving your program’s aims are strengths.

Important to acknowledge and celebrate

What should definitely be continued (important question in an environment of limited resources)

Weaknesses

Program factors that are likely to have a negative effect on (or be a barrier to) achieving your program’s objectives are weaknesses.

Citations, areas for improvement and other information from ACGME

The Annual Program Evaluation and other program/institutional data sources

Confidential – For Discussion Purposes Only

Factors and contexts external to your programs (institutional, local, regional and national) that affect the program

Opportunities - Factors that favor the program, that the program may take advantage of / leverage

External Factors that are likely to have a positive effect on achieving or exceeding your program’s objectives not previously considered are called opportunities.

What are capabilities for further evolving the program; how can the program capitalize on them?

Has there been recent change in the program’s context that that creates an opportunity?

Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

Opportunities – External Factor

Confidential – For Discussion Purposes Only

Threats – External Factors

Threats - Factors that pose risks.

External Factors and conditions that are likely to have a negative effect on achieving the program’s objectives, or making the objective redundant or un-achievable are called threats.

While the program cannot fully control them, beneficial to have plans to mitigate their effect

What external factors may place the program at risk?

What are changes in residents’ specialty choice, regulation, financing, or other factors that may affect the future success of the program?

Are there challenges or unfavorable trends in immediate context that may affect the program? e.g., faculty burdened with heavy clinical load that prevents effective teaching and mentorship

Confidential – For Discussion Purposes Only

Fishbone – Ishikawa diagram - SWOT Analysis Completed Example

Confidential – For Discussion Purposes Only

APE – Program SWOT Action Plan in Workbook for Self Study Retention

Confidential – For Discussion Purposes Only

Sample Stanford APE Page

Confidential – For Discussion Purposes Only

Tools Can Be Downloaded @ www.gme.stanford.edu

GME Community Templates

Confidential – For Discussion Purposes Only

Questions