using medhub and process streamlining to make ......using medhub and process streamlining to make...

Using MedHub and Process Streamlining to Make Organizing & Documenting GME / Accreditation Data Easy

Nancy Piro, PhD – Program Manager/Ed Specialist Stanford GMEBrett Toensing, MS – MedHub/Web Coordinator

Confidential – For Discussion Purposes Only

AGENDA – Leveraging MedHub

Process Streamlining Tips & Tricks

Setting Priorities

− Citation Analysis

Program Evaluations

− Standardizing Delivery and Documentation

Summative Evaluations

− Incoming and Outgoing Trainees

− Standardized Template and Electronic Filing

− Program Needs Analysis

Organizing data for:

− CCCs

− APEs / Self-Studies


Thinking Outside the Box – Looking to Industry and Corporations

What does industry have that we can use?

− Conveyor belt to move patients in and out of OR’s? …..No……

But they do use a “Lean Philosophy”

− Respect for People

− Total Elimination of Waste

And they use “Lean Tools” such as

− Kaizen

− 3 M’s

− 5S’s


Lean – Toyota Production System (TPS)

TPS system managed to get by with “half of everything”!

− Physical space

− Personnel

− Capital Investment

− Inventory

Resulted in far fewer than half the defects and safety incidents

Make problems visible – not hide them or “not talking about mistakes/problems”

Fix problems permanently – get to the “root cause” and eliminate it.

Focus on the value of people and respect for people.


Sample of “Lean Tools” … Kaizen

What is ‘Kaizen’?

Gradual, unending continuous improvement of processes

Processes must be improved to get improved results

By improving and standardizing activities and processes, Kaizen aims to eliminate waste

Focus is on small, incremental change (not necessarily huge leaps, innovation)

Quality Control = Quality of People

− “A company that is able to build quality into its people is halfway to building quality outcomes”

Respect for People

− “Only people produce improvements”


Lean Tools: - 3M’s

All about Waste- Identifying and Removing it

− MURI = Waste of overburdening people or equipment/resources

− MURA = Waste of unevenness, variability in processes

− MUDA = Waste of using resources without creating added value


Five Why’s Problem Solving Method

Keep asking ‘Why?’ until you discover the root cause of the problem

− No magic in 5 –

might be 3, or 7, or 10

Why do we? (…conduct orientation in person, fill out multiple forms, have residents take on line training for non MDs? ….)

Combine with Lean Tools: – Fishbone Charts using the 4Ms – 4 Ps

Manpower/Personnel

Materials

Method(s)

Machines / Equipment

People

Process

Policies

Principles


Ishikawa Diagram of ER Prolonged Wait Times

Materials Methods/Process

Manpower/People Machine \ Equipment

WHY? WHY?

WHY? WHY?

xxx …wait time in the ER


5Ss - Mnemonics Retained

Sorting

Simplifying

Sweeping

Standardizing

Self Discipline

= Seiri

= Seiton

= Seiso

= Seiketsu

= Shitsuke

QUICK EXERCISE


5Ss in Action

10


AFTERBEFORE

ER Rooms


Putting Philosophy into Practice

Flowcharted the processes

− Asked: “Why are we doing this?”

‘Fish boned’ the problem

Looked for the 3Ms

Applied the 5Ss


CURRENT CULTURE

96 programs (96 fiefdoms all doing their own thing….but having common requirements)

Lack of standardization/inconsistencies (MURA)

Overburden (MURI)

Waste (MUDA) of both materials and time

− Lots of files

− Time to file personnel folders or

ACGME letters

Time-consuming institutional oversight of programs (MUDA, MURI)


Standardizing Program Evaluations in MedHub

Factors driving decision to standardize our Program Evaluations:

− Largest number of program citations from ACGME/RRCs

− Lack of consistent data on program evaluation for APEs (MURA)− Huge amount of paper generated to produce suboptimal evaluations (MUDA)− Burden of work on the coordinators (MURI)

What did we do?

Developed standardized comprehensive core competency-based Program Evaluations – by Faculty and Trainees

Presented the draft templates to the Program Directors who edited and approved them.


Standardized Program Evaluations – Process Implementation

Then implemented - Annually the GME Department:

− Delivers (via MedHub, of course!)

>190 Program evaluations to > 1500 trainees and faculty

Tracks Evals Delivered for each program and Average Results for each Eval

Aggregates each program’s data after the evals have been completed

‘Prints’ (electronically to pdfs) their aggregate Program evaluation reports

Posts the Program Eval Reports (pdfs) on each Program’s APE site in MedHub


Program Evaluation Standardization Example


Standardization of Program Evaluations

Benefits

− Fewer citations

− Can be input to our Institutional Report Card

− Easy oversight by DIO/GMEC

− Early warning system


Summative Evaluations – What did we do?

Developed Standard Evaluation Template for Summative Evaluations

Comprehensive over program training years

Core-competency based

User friendly word document

Held Program Director and Program Coordinator Teaching Sessions / Workshops

Taught the use of the template and how to pair it with aggregated evaluation data MedHub

Programs could select the evaluations to use and aggregate–

− We recommended:

Evaluations Competencies by Resident

Resident/Faculty Ranking Report (aggregate)

Aggregate Comments Report

Evaluation Competencies Report


Coordinate with Aggregated Evaluation Data


Summative Evaluation Template


Macro-enabled Word Doc


Summative Evaluations - Additional Considerations

Special Cases

− At the end of a preliminary year (internship)

Need to list Rotations

Review report with trainee

Place a copy of the summative evaluation in the Trainee’s permanent file and upload to MedHub

We also obtain Summative Evaluations from our trainees coming in from another GME program for all the programs and upload them to MedHub also


Switching Gears From Summatives to CCCs

CCCs

Summatives


CCCs

CCC MedHub Online Functionality –

− Assign Files / Aggregated Evaluations to Program Faculty for Pre-Work

− Upload all required data for Preliminary Review and the CCCX meeting


Pulling the Data Together

Clinical Competency Committee

End-of-Rotation

Evaluations

Safety Incident Reports

Case Logs

Patient/ Family

Evaluations

Clinical Skills

Assessment

Nursing and Staff / Techs

EvaluationsProgress on Milestones

SimLab

In service In-service training exams

Quality Improvement

Activities


Data used in CCC meetings for trainee assessment

26

65.38%

93.85%

76.92%

44.62%

63.08%

36.92%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

Aggregate DirectMilestone

Evaluations

Aggregate RotationEvaluations

Staff or patient(360) Evaluations

Technical SkillsAssessments/Mini-

CEX's

Scholarly Activities

Case Volumes


EvaluationsMilestone End of Rotation (Total)Medical KnowledgePatient CareInterpersonal & Communication SkillsProfessionalismSystems-Based PracticePractice-Based Learning & Improvement

Creating a Resident Performance ProfileStep 1 – Defining what to track


In-service Assessments (MK; PC)Routine procedure technical skills assessment: Level 2-3Complex procedure technical skills assessment: Level 3-4Medical Knowledge Assessments

Creating a Resident Performance ProfileStep 1 – Defining what to track


Case Logs / Clinical Experience (PC)VAGINAL DELIVERYCAESAREAN SECTIONPEDIATRICSPEDIATRICS UNDER 3CARDIACENDOVASCULAR

Creating a Resident Performance Profile Step 1 –Defining what to track


Quality Improvement/Patient SafetyQI ProjectQI Committee ParticipationQI Course Work (e.g., IHI)SAFE Report / Adverse Event ReviewPatient Handover Evaluations

Creating a Resident Performance Profile Step 1 – Defining what to track


Practice-Based LearningPatient outcomes / Case Study PresentationScholarly Activity: Research studyScholarly Activity: PublicationsPresentation at Internal and National Meeting



CommunicationPatient Feedback Staff EvaluationMedical Student FeedbackPatient Handover Evaluations



Creating a Resident Performance ProfileStep 2 - Link data sources to milestones

Evaluations MilestonesMilestone End of Rotation (Total) 24Medical Knowledge MK A1Patient Care PC A1-10Communication ICS A1-2Professionalism PROF 1-5Systems-Based Practice SBP A1-2Practice-Based Learning PBLI A1-4Quality Improvement/Patient SafetyQI Project PBLI1-2QI Committee Participation PBLI1-2QI Course Work (e.g., IHI) PBLI1-2SAFE Report / Adverse Event Review PROF1/2; SBP1Patient Handover Evaluations SBP1; ICS1


Milestone Data Translation to Numerical Data


Creating a Resident Performance ProfileStep 3: CCC defines performance ranges

At or Above Expectation: 2.8 and higher

Below Expectation:1.7 – 2.7

Remediation:Below 1.7

Example:

For all aggregate milestone evaluation scores for a PGY 3, the CCC defines these ranges:


Creating a Resident Performance ProfileStep 4 – Set conditional formatting


Example: Aggregate milestone evaluation data cells

Highlight cells to apply the conditional formatting



Click on “Conditional Formatting”“Highlight Cells Rules”



Select, “Greater Than” “Less Than” or “Between” to Set Value RangesChoose the corresponding fill color (e.g., red, yellow, green)



Creating a Resident Performance ProfileCompiling and centralizing data


Resident Performance Profile:Step 5: Enter in data


Creating a Resident Performance ProfileVisual Trends and Detailed Data


Creating a Resident Performance Profile-Visual Trends and Detailed Data: Another Example


There was no way we could have had an effective CCC meeting without completing pre-work.

44

90.58%

8.07%

1.35%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

TRUE

FALSE

Notapplicable


MedHub CCC Functionality – Resources and Files


Leveraging More MedHub Functionality

Conference attendance statistics

− Core competencies linked to specified conferences

− Attaching conference materials for later reference

Procedures and levels; linked procedure evaluations


• Resident portfolio toolsQI participation and outcomesScholarly Activity logs

Leveraging MedHub Functionality


Leveraging MedHub Functionality

Aggregate reporting and graphic summaries

Peer or departmental average, individual average, minimum and maximum scores, standard deviation or listing of all scores


PGY 1 Ranked at Graduation Level

1


When Remediation is Evident


Switching Gears From CCCs to APEs

CCCs

APEs


Program Evaluation Committee (PEC) Must Monitor and Track (V.C.2):

1. RESIDENT PERFORMANCE

2. FACULTY DEVELOPMENT

3. GRADUATE PERFORMANCE

4. PROGRAM QUALITY

5. PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN


RESIDENT PERFORMANCE

The most recent aggregated written evaluations of the residents submitted by faculty and other evaluators

In-training/In-service exam scores

Procedure logs (if applicable)

Scholarly activity (publications, presentations, grant awards, etc.)

Learning portfolios: documented quality improvement activities


FACULTY DEVELOPMENT

ABMS certification status for all faculty

Updated faculty CVs

Documentation (faculty survey; attendance logs) of faculty participation in:− CME-type activities directed toward acquisition of clinical

knowledge and skills and also activities directed toward developing teaching abilities, professionalism, and abilities for incorporating the core competencies into practice

− Teaching (conferences, grand rounds, journal clubs, lecture-based CME events, workshops, directed QI projects, practice-improvement self study).

Faculty actively involved in mentor relationships with residents/fellows.


GRADUATE PERFORMANCE

Aggregated board exam pass rates

Aggregated alumni survey results (typically, such surveys target physicians one year and five years after graduation –survey questions may inquire about such items as current professional activities of graduates and perceptions on how well prepared they are as a result of the program)

Other outcome measures

− Practice location (underserved areas)

− Academic Affiliations

− Scholarly Activity


Alumni Surveys …


PROGRAM QUALITY

The most recent aggregated written evaluations of the program submitted by faculty

The most recent aggregated written evaluations of the program (and/or specific rotations) submitted by residents

The most recent aggregated written evaluations of the faculty submitted by residents

Faculty’s recent scholarly activity (publications, presentations, grant awards, etc.)


PROGRAM QUALITY - Continued

The most recent ACGME survey results

The most recent GME House Staff survey results

The most recent GMEC Internal Review Report

Any recent communications from the ACGME or RRC

Program Report Card/Scorecard

− Trend Analyses


PROGRAM QUALITY - Continued

Curriculum

Overall and rotation-specific goals and objectives (Are they appropriate? Do they align with the core competencies?)

Didactic curriculum (Is there at least one regular conference targeted to the residents’ level?)

Opportunities for scholarly activity

Compliance with any new standards established by the ACGME, RRC, ABMS, etc.− Assessment Methods (Are evaluation tools appropriate? Do they align with

the core competencies?)

− Resources: Personnel (PD, PC, faculty), Affiliated Training Sites, Patient/Procedure Volume,

Learning Environment (space, call rooms, books, computers, etc.)


PROGRESS ON THE PREVIOUS YEAR’S ACTION PLAN

Review progress / (attempts to resolve problems) with respect to last year’s Annual Review delineating identified areas of weakness.


Now… SWOT Analyses are being required

STRENGTH

OPPORTUNITY

OPPORTUNITY

THREATS


APE “SWOT” Analysis Tool - Fishbone / IshikawaPROGRAM


Program Aims – ACGME PerspectiveThe AIM setting is part of the annual program evaluation

Relevant considerationsWho are our residents/fellows?What do we prepare them for? Academic practice Leadership and other rolesWho are the patients/populations we care for?

AIMS are a way to differentiate programs Self-study will ultimately evaluate program effectiveness in

meeting these aimsMoves beyond improvement solely based on compliance with

minimum standards Assessment of relevant initiatives and their outcomes


Strengths and Weaknesses – Internal Factors

Strengths

Program factors that are likely to have a positive effect on (or be an enabler to) achieving your program’s aims are strengths.

Important to acknowledge and celebrate

What should definitely be continued (important question in an environment of limited resources)

Weaknesses

Program factors that are likely to have a negative effect on (or be a barrier to) achieving your program’s objectives are weaknesses.

Citations, areas for improvement and other information from ACGME

The Annual Program Evaluation and other program/institutional data sources


Factors and contexts external to your programs (institutional, local, regional and national) that affect the program

Opportunities - Factors that favor the program, that the program may take advantage of / leverage

External Factors that are likely to have a positive effect on achieving or exceeding your program’s objectives not previously considered are called opportunities.

What are capabilities for further evolving the program; how can the program capitalize on them?

Has there been recent change in the program’s context that that creates an opportunity?

Are these opportunities ongoing, or is there a narrow window for them? How critical is the timing?

Opportunities – External Factor


Threats – External Factors

Threats - Factors that pose risks.

External Factors and conditions that are likely to have a negative effect on achieving the program’s objectives, or making the objective redundant or un-achievable are called threats.

While the program cannot fully control them, beneficial to have plans to mitigate their effect

What external factors may place the program at risk?

What are changes in residents’ specialty choice, regulation, financing, or other factors that may affect the future success of the program?

Are there challenges or unfavorable trends in immediate context that may affect the program? e.g., faculty burdened with heavy clinical load that prevents effective teaching and mentorship


Fishbone – Ishikawa diagram - SWOT Analysis Completed Example


APE – Program SWOT Action Plan in Workbook for Self Study Retention


Sample Stanford APE Page


Tools Can Be Downloaded @ www.gme.stanford.edu

GME Community Templates


Questions

using medhub and process streamlining to make ......using medhub and process streamlining to make...

Documents