Procedural Assessment: Procedural Assessment: Where do we stand? Where do we stand?

March 2013March 2013

Walter J. Coyle MD, FACG, FASGE

ObjectivesObjectives

The apprentice model– The way we were

Competency based education– Where we are now

Outcomes based learning– Procedural focus– Milestone development

ApprenticeshipApprenticeshipSuccessful for thousands of yearsKey properties:

– Good mentor– Motivated student– Adequate exposure AND hands on time

Problems: – Consistency– Objective measures of success– Low ceiling for promotion

The Mystery of Mastery. Psychology Today 1986;20:32

ApprenticeshipApprenticeship

Works well for very sub-specialized areas and few centers of excellence

Still model for advanced endoscopy– AEF match– Variation in level of exposure and mastery– Who monitors the mentors?– How do the graduates do?

Medicine resents outside monitoring– Better if we did it ourselves

Competency based trainingCompetency based training

ACGME initiative from the 1990s– Applied to all aspects of training– Knowledge, professionalism, procedures

Ineffective for procedural training– GI procedures still in apprentice model– Little consensus on assessment and outcomes– Little data to define milestones

How should a 2nd year fellow scope?

Lurie. Med Educ 2012;46:1365

Competency based trainingCompetency based training

Diverse training methods and assessment techniques

Small programs vs large; research fellowsSilo mentality: no consistent standardExplains why we have this problem now

Outcomes based learningOutcomes based learning

More process orientedFocus on the process not the problemACGME wants us to move hereStarting point: 1st year fellowEnding point: Staff GIMilestone development: easier for

knowledge core vs procedures

N Engl J Med 2012;3686:1051-56

Milestone developmentMilestone development

Final milestone: Colonoscopy– >95% cecal intubation rate– >25% ADR– Low complication– Patient satisfaction

Stepwise milestones: None with great data or evidence– 1st year vs 2nd vs 3rd

Gastrointest Endosc 2010;71:319-24

Procedural Education: initial Procedural Education: initial focus on processfocus on process

Intense didactic– FYF course, DVDs, local resources

Intense hands on training with scope– ? Simulators– Training box/tool– Standardized patient– Example of pilot training?

Procedural Education: Procedural Education: subsequent focus on processsubsequent focus on process

Ongoing, continuous assessment:– Mentor feedback; patient feedback– Objective outcomes based assessment tool– Universal tool ?

Development of outcomes based, data driven milestones that apply throughout fellowship– How???

Procedural Education: A Procedural Education: A Proposal Proposal

Universal assessment tool agreed uponWeb-based submission of assessments

– Collection and development of milestone– Feedback to fellow and program– Fellow compared to peers nationally

Progression through milestones will be fellow driven, not fixed year driven

Data on FellowData on Fellow

Sedlack, GIE 2010;72:1125-33

Procedural Education: A Procedural Education: A Proposal Proposal

Requirements of system– Ease of use: minutes, APP for phone, link on

desktop– Secure– Can provide data back to program and fellow in

real time– Dynamic and progressive

Procedural Education: A Procedural Education: A Proposal Proposal

Cost: GI programs, GI societies, ACGMEWeb site location and maintenance

– ACGME– CORI database like initiative– GIQuik– Endoscopic report generating systems

Provation initiative with Mayo Clinic

Time frame

ASGE Proposed FormsASGE Proposed Forms

Integrated AssessmentIntegrated Assessment

Are we ready and committed??Resource CommitmentStaff CommitmentBarriers breaking silos

Only definite: change is here

SummarySummary

Prior models and procedural mentoring are probably inadequate

Classic competency based assessment is flawed for procedures

Outcomes and milestones are a next stepGI directed development of milestones and

tools is critical

SummarySummary

Need to think of the process NOT the problem