web-enabled evaluation techniques: applying a competency-based curriculum

June, 2002 AFPRD Workshop

Web-Enabled Evaluation Techniques: Web-Enabled Evaluation Techniques: Applying a Competency-Based Applying a Competency-Based

CurriculumCurriculum

Charles O. Frazier, M.D.Charles O. Frazier, M.D.

Jon J. Kaminer, M.D.Jon J. Kaminer, M.D.


ObjectivesObjectives

1.1. Why effective evaluation is criticalWhy effective evaluation is critical

2.2. How the ACGME guidelines can fit in How the ACGME guidelines can fit in an effective evaluation schemean effective evaluation scheme

3.3. How a web-enabled evaluation tool How a web-enabled evaluation tool developed at Riverside can be used to developed at Riverside can be used to apply a competency-based curriculumapply a competency-based curriculum


Why is an evaluation system Why is an evaluation system needed?needed?

– It’s a requirementIt’s a requirement– Further defines expectationsFurther defines expectations– Promotes goal directed behaviorPromotes goal directed behavior– Provides comfortable atmosphere for Provides comfortable atmosphere for

constructive criticism and positive constructive criticism and positive reinforcementreinforcement


What is effective evaluation?What is effective evaluation?– Specific/focusedSpecific/focused– Timely Timely – ConsistentConsistent– Target behavior/performance reasonableTarget behavior/performance reasonable– Conducted in collegial mannerConducted in collegial manner– Ends with defined goal/expectation of future Ends with defined goal/expectation of future

behavior/performancebehavior/performance


What are the main evaluation modalities?What are the main evaluation modalities?

• feedbackfeedback

• summative evaluationsummative evaluation


FeedbackFeedback

• Immediate commentImmediate comment


Summative EvaluationSummative Evaluation

• Longitudinal measure of performanceLongitudinal measure of performance


Modality ComparisonModality Comparison


How does the ACGME outcomes How does the ACGME outcomes project fit in to this evaluation project fit in to this evaluation framework?framework?


CompetenciesCompetencies


List of ACGME competenciesList of ACGME competencies

• Patient CarePatient Care• Medical KnowledgeMedical Knowledge• Practice-Based Learning and Improvement Practice-Based Learning and Improvement • Interpersonal and Communication SkillsInterpersonal and Communication Skills• ProfessionalismProfessionalism• Systems-Based PracticeSystems-Based Practice


www.ACGME.org/Outcomewww.ACGME.org/Outcome


ToolboxToolbox• Record reviewRecord review

• Chart Stimulated recallChart Stimulated recall

• ChecklistChecklist

• Global ratingGlobal rating

• SPSP

• OSCEOSCE

• Simulations and ModelsSimulations and Models


ToolboxToolbox

• 360360ºº Global Rating Global Rating

• PortfoliosPortfolios

• Exam MCQExam MCQ

• Exam oralExam oral

• Procedure case or logsProcedure case or logs

• Patient surveyPatient survey


BenefitsBenefits– defines the ACGME’s expectations of usdefines the ACGME’s expectations of us– Defines our expectations for residents to Defines our expectations for residents to

graduategraduate– allows for consistencyallows for consistency– changes focus of evaluation to resident changes focus of evaluation to resident

performance rather than residency structureperformance rather than residency structure


LimitationsLimitations– Incredible amount of work and paper workIncredible amount of work and paper work– How will we use all this information to How will we use all this information to

provide meaningful feedback to the resident provide meaningful feedback to the resident – Will this take away from resident teaching Will this take away from resident teaching

timetime


Riverside Family Practice Riverside Family Practice ResidencyResidency


Our former evaluation frameworkOur former evaluation framework

FeedbackFeedback

PreceptingPrecepting


Our former evaluation frameworkOur former evaluation framework

Summative EvaluationSummative Evaluation– Trimester Faculty meetingsTrimester Faculty meetings– Trimester directors meetingsTrimester directors meetings

• Rotational evalsRotational evals

• Full faculty impressionFull faculty impression


Pros Pros – Plenty of face-time and feedback during Plenty of face-time and feedback during

preceptingprecepting– Gestalt of faculty about resident shared and Gestalt of faculty about resident shared and

discusseddiscussed– Not much paperwork for facultyNot much paperwork for faculty


Cons Cons – No specificsNo specifics– Summative evaluation disconnected from Summative evaluation disconnected from

feedbackfeedback– InefficientInefficient– No good individualized action planNo good individualized action plan– Overloaded residency directorOverloaded residency director


Structural Changes MadeStructural Changes Made

– Advisor systemAdvisor system– Quarterly advisor meetings with Quarterly advisor meetings with

Educational PrescriptionEducational Prescription– Semiannual director meetingsSemiannual director meetings

Advising - March 2001


Competency-based SystemCompetency-based System• Paper evaluation forms developed in Paper evaluation forms developed in

light of competencieslight of competencies• Expectations defined for residents and Expectations defined for residents and

facultyfaculty• Our residency “toolbox” was expanded Our residency “toolbox” was expanded

and redefined based on ACGME and redefined based on ACGME guidelinesguidelines

Tool MatrixTool MatrixComp-Eval™ Competency Evaluation Tool Matrix

Competency (from www.acgme.org)

PATIENT CARE Residents must be able to provide patient care that is compassionate, appropriate, and effective for the treatment of health problems and the promotion of health. Residents are expected to:

communicate effectively and demonstrate caring and respectful behaviors when interacting with patients and their families

gather essential and accurate information about their patients make informed decisions about diagnostic and therapeutic interventions based on patient information and preferences, up-to-date scientific evidence, and clinical judgment

develop and carry out patient management plans counsel and educate patients and their families use information technology to support patient care decisions and patient education

perform competently all medical and invasive procedures considered essential for the area of practice

provide health care services aimed at preventing health problems or maintaining health

work with health care professionals, including those from other disciplines, to provide patient-focused care

MEDICAL KNOWLEDGE Residents must demonstrate knowledge about established and evolving biomedical, clinical, and cognate (e.g. epidemiological and social-behavioral) sciences and the application of this knowledge to patient care. Residents are expected to:

demonstrate an investigatory and analytic thinking approach to clinical situations know and apply the basic and clinically supportive sciences which are appropriate to their discipline

PRACTICE-BASED LEARNING AND IMPROVEMENT Residents must be able to investigate and evaluate their patient care practices, appraise and assimilate scientific evidence, and improve their patient care practices. Residents are expected to:

analyze practice experience and perform practice-based improvement activities using a systematic methodology

locate, appraise, and assimilate evidence from scientific studies related to their patients’ health problems

obtain and use information about their own population of patients and the larger population from which their patients are drawn

apply knowledge of study designs and statistical methods to the appraisal of clinical studies and other information on diagnostic and therapeutic effectiveness

PagePage425425

RotationsRotations


DOPDOP

PatientsPatients

NursesNurses

PeersPeers

Medical Medical StudentsStudents

Care Care ManagersManagers

Research Research ProjectProject

Procedure Procedure ReportReportRecordRecord

CompletionCompletionReportReport

RotationsRotations


DOPDOP

PatientsPatients

NursesNurses

PeersPeers

Medical Medical StudentsStudents

Care Care ManagersManagers

Research Research ProjectProject

Procedure Procedure ReportReportRecordRecord

CompletionCompletionReportReport

OnlineOnlineComp-Eval Comp-Eval

ServerServer

FacultyFacultyResidentResident


August/September - Nurse Evaluation / Faculty Advisor Evaluation

November/December - Nurse Evaluation / Faculty Advisor Evaluation

December - Semi-Annual Program Director Evaluation

February/March - Nurse Evaluation / Faculty Advisor Evaluation

May/June - Nurse Evaluation / Faculty Advisor Evaluation

June - Semi-Annual Program Director Evaluation


Web-Enabled SystemWeb-Enabled System

• A stand-alone evaluation systemA stand-alone evaluation system

• Universally accessibleUniversally accessible

• Highly customizableHighly customizable

• Data automatically collected in databaseData automatically collected in database– stored, studied, and sharedstored, studied, and shared


Technical description - IntranetTechnical description - Intranet

• Compaq Proliant ServerCompaq Proliant Server• Microsoft Windows NT 4.0Microsoft Windows NT 4.0• Microsoft Internet Information Services Microsoft Internet Information Services

5.05.0• Microsoft FrontPage 2000Microsoft FrontPage 2000

– Active Server PagesActive Server Pages

• Microsoft Internet ExplorerMicrosoft Internet Explorer


Active Server PagesActive Server Pages

• Script (programming) is executed on the Script (programming) is executed on the serverserver, and resulting HTML is sent back , and resulting HTML is sent back to and displayed on browserto and displayed on browser– Connects to databaseConnects to database– Underlying script secureUnderlying script secure


WebWebBrowserBrowser

DatabaseDatabase


Technical Description - IntranetTechnical Description - Intranet

• SecuritySecurity– NT Server AuthenticationNT Server Authentication

• CostCost– Server $3,000Server $3,000– Software $1500Software $1500– Labor (estimate) 450 hoursLabor (estimate) 450 hours


Technical Description - InternetTechnical Description - Internet

• VCU IIS Public Information ServerVCU IIS Public Information Server

• Security with ASP database passwordsSecurity with ASP database passwords

• No direct cost to residencyNo direct cost to residency

• Domain name registrationDomain name registration– www.pubinfo.vcu.edu/webevalwww.pubinfo.vcu.edu/webeval


DemonstrationDemonstration


The futureThe future– Ability to study competency-based educationAbility to study competency-based education– Evaluate Program CurriculumEvaluate Program Curriculum– Ability to describe expected “Natural Ability to describe expected “Natural

History” or “Growth Curve” for residentHistory” or “Growth Curve” for resident– Generate documentation to support putting Generate documentation to support putting

a resident on academic probationa resident on academic probation


Questions?Questions?

http://www.http://www.pubinfopubinfo..vcuvcu..eduedu//webevalwebeval

charlescharles..frazierfrazier@@rivhsrivhs.com.com

jonjon..kaminerkaminer@@rivhsrivhs.com.com

web-enabled evaluation techniques: applying a competency-based curriculum

Documents