1 ambulatory pediatric association educational guidelines for pediatric residency tutorial 5 written...

1

Ambulatory Pediatric Ambulatory Pediatric AssociationAssociation

Educational Guidelines forEducational Guidelines forPediatric ResidencyPediatric Residency

Tutorial 5Tutorial 5

Written byWritten byConstance D. Baldwin, Ph.D., Carol Carraccio, M.D., Constance D. Baldwin, Ph.D., Carol Carraccio, M.D.,

and the and the Educational GuidelinesEducational Guidelines Team Team

Source: Kittredge, D., Baldwin, C. D., Bar-on, M. E., Beach, P. S., Trimm, R. F. (Eds.). (2004). APA Educational Guidelines for Pediatric Residency. Ambulatory Pediatric Association Website. Available online: www.ambpeds.org/egweb. ©2004 Ambulatory Pediatrics Association. All Rights Reserved.

2

Part IPart I

Principles of Principles of

Competency-based Competency-based EvaluationEvaluation

Ambulatory Pediatric Association Educational Guidelines for Pediatric Residency

3

Basic Principles: Basic Principles: Definition of Definition of “Competencies”“Competencies”

ACGME has defined six broad competency domains, each composed of several competency elements.

Competency elements are more complex than educational objectives: they encompass knowledge, skill and attitude components.

The competency elements are best evaluated through behavioral and measurable educational objectives.


4

Specification of Specification of CompetenciesCompetencies

• While competency domains and elements are general, evaluation must be specific: In most teaching settings,

competencies can be referenced to specific educational objectives that relate to that setting.

Example: Communication and Interpersonal Skills might be evaluated in the nursery by observing how well a resident educates a mother on newborn care or breastfeeding.


5

Criteria for Sound Criteria for Sound LearnerLearnerEvaluation MethodsEvaluation Methods

• Validity: Do ratings accurately depict what the learner knows, without bias or distortion?

• Reliability: Do ratings of the learner return the same result in multiple settings with multiple evaluators?

• Generalizability: Does the sampling of a learner’s performance provide a true and complete picture of his/her strengths and weaknesses?


6

Criteria for Sound Criteria for Sound Evaluation Methods, Evaluation Methods, con’dcon’d

• Feasibility: Can the evaluation method be implemented relatively easily and efficiently (i.e., will faculty and learners comply in using it)?

• Utility for learners: Is the method sufficiently complete and specific to help learners improve their performance?

How can these criteria be applied to evaluations in the

“real world”?


7

Use Authentic, Relevant Use Authentic, Relevant AssessmentsAssessments

• Competency-based evaluation requires authentic assessment. One must evaluate performance in

tasks that are relevant to the practicing physician, not just measure knowledge

Example: Evaluation of the resident’s ability to evaluate “best evidence” and apply it to a particular patient


8

Don’t Use Global Don’t Use Global Evaluations AloneEvaluations Alone

• A proxy of assessment is not sufficient. Global evaluations alone do not

measure actual skills. They are subject to “halo” effect (e.g.,

an articulate resident with good interpersonal skills may be assumed to be skilled in physical examination).

Direct observation is needed to measure performance accurately.


9

Clarify Clarify Evaluation CriteriaEvaluation Criteria

• Transparency in evaluation is critical for both evaluator and learner. Learners need to understand the

criteria by which their performance will be judged.

Evaluators need to understand how to use the evaluation tool and the criteria by which they must judge a learner’s performance.


10

Include Criterion-Include Criterion-Referenced ToolsReferenced Tools

• Consider using criterion-referenced methods in addition to norm-referenced methods. Norm-referenced tools compare

learners to each other. Criterion-referenced tools measure

learners against a predetermined threshold.


11

Use Multiple Evaluation Use Multiple Evaluation MethodsMethods

• Multiple methods make an assessment more valid. Example: To evaluate a resident’s

ability to care for a patient, observe the resident performing a history & physical in addition to reviewing the resident’s written documentation on that patient.


12

Emphasize Formative Emphasize Formative FeedbackFeedback

• Focus on formative feedback rather than summative evaluation. Learners deserve guidance in

meeting the threshold you have set for achieving competence.

If learners are falling short, timely feedback and a remediation plan are critical.


13

Emphasize Formative Emphasize Formative FeedbackFeedback

• Good feedback and counseling should be: Frequent and timely (during experience) Based on careful observation Expressed in specific behavioral terms Constructive—non judgmental Accompanied by a plan for improvement

and a schedule for follow-up


14

Encourage Residents to Encourage Residents to Self-evaluateSelf-evaluate

Self-directed learners becomelife-long learners!• Use table of goals/objectives for

resident self-evaluation on first and last day of an experience.

• Assign self-assessment tasks to reinforce importance of self-direction.

• Encourage long-range planning by resident, based on learning needs at end of rotation.

15

Part IIPart II

Practical Tips for Creating Practical Tips for Creating

a Competency-based a Competency-based

Evaluation SystemEvaluation System


16

Planning an Evaluation Planning an Evaluation System: Basic PrinciplesSystem: Basic Principles

• Evaluation methods must be based on specific, measurable or observable objectives.

• Evaluation tools should precisely specify level of expected performance.

• Summative evaluations can be brief and still valid only if they are based on iterative, detailed and specific formative evaluations.

• Faculty and residents need orientation for the system to work!


17

1. Develop tools and processes to evaluate residents in each competency domain.

Note RRC suggestions for methods in each domain.

2. Provide faculty development and resident orientation.

Residents must know what target they are aiming for and whether they hit it.

Faculty must become skilled at conducting valid and reliable evaluations.

3. Establish a regular feedback system.

Planning an Evaluation Planning an Evaluation System: A PrimerSystem: A Primer


18

Planning an Evaluation Planning an Evaluation System: A PrimerSystem: A Primer

4. Assess resident performance on each rotation and document progress over time.

• Monitor each rotation’s progress in competency integration.

• Periodically evaluate teaching and program performance, so you can tune the system.


19

Practical Applications: Practical Applications: Choose Targets and Choose Targets and ToolsTools

• To create a competency-based evaluation system, first:Select the competencies and

related educational objectives from the APA Guidelines that you want to evaluate.

Decide which evaluation tools are most effective for assessing these learning objectives.


20

Choosing Tools: Choosing Tools: ACGME ToolboxACGME Toolbox

• Written/oral examinations• Global ratings • Procedure/case logs• Record review• Checklist evaluation• Direct observation of clinical skill• OSCE/standardized patients• Simulations and models• Portfolios• 360-degree evaluations (peers, staff)• Chart stimulated recall oral exam• Patient surveys


21

Choosing Choosing Evaluation ToolsEvaluation Tools

• Most commonly used instruments (each measures a different aspect of competence) Global rating by supervisor Multiple choice exams Standardized patients

• RRC would prefer more use of: Direct observationsMulti-source (“360”) evaluations


22

How to Improve Global How to Improve Global Ratings?Ratings?

• Backup ratings with direct observation. • Annotate form with detailed coding

instructions that are context-specific. • Diversify raters (e.g., peers, staff,

patients). • Provide systems to support faculty

training, form completion and collection, analysis, and feedback systems.


23

Direct Observation Direct Observation is Essentialis Essential

• OSCE’s & standardized patients expensive Valuable teaching and feedback tools Need multiple cases for good reliability in

“high stakes” exams• Clinical evaluation exercises more practical

CEX = Observed H&P, presentation and structured feedback

Mini CEX = 20 min focused on 1-2 elements • Bottom line: You need to observe and give

feedback!


24

Choose Evaluators and Choose Evaluators and SettingsSettings

• Plan your evaluation strategy: Decide where residents most

practically can be evaluated within your program.

Be sure all competency domains and elements are covered (not every competency needs to be measured in every rotation).

Decide who are the best evaluators.


25

Educate Faculty and Educate Faculty and ResidentsResidents

• Provide faculty development for your evaluators so that they understand the tool they are using and how to apply it. Emphasize the need to give formative feedback.

• Educate your learners about the process and criteria upon which they will be evaluated.


26

Evaluate the Evaluation Evaluate the Evaluation ProcessProcess

• Don’t stop with evaluating the learner. • You need to evaluate the program’s

success in:Teaching and evaluating the learnersDelivering an effective and efficient

curriculumAssessing and refining the program


27

Evaluation: Summary of Evaluation: Summary of LiteratureLiterature

• Evaluation of clinical competence is a daunting, complex task.

• Evaluation of life-long learning skills, professionalism, and systems-based practice is an even newer, bigger challenge.

• No single evaluation tool can adequately assess everything.

• A multifaceted approach is required. • The “right combination” depends on program

goals, needs and resources.

28

Part IIIPart III

Building Competency-based Building Competency-based

Evaluation FormsEvaluation Forms


29

Building Evaluation Building Evaluation Forms: Basic PrinciplesForms: Basic Principles

• Type of form (e.g., global, multi-source) should reflect setting, evaluators, program requirements.

• Items on form should reflect the core content of the learning experience and be specified clearly.

• Evaluation coding system should be well defined, so it is consistently applicable across evaluators.

• Specify performance expectations for each item.


30

Building Evaluation Building Evaluation Forms:Forms:A PrimerA Primer

A typical evaluation form iscomposed of 3 parts:1. Header (top row)2. Content rows3. Footer

Evaluation FormEvaluation Form Template Template

Form TitleResident:

Date: Evaluator:

Date:

Evaluation Anchors1= 3= 5= N/0 = No opportunity to evaluate

Content Row 1

Content Row 2, etc.

Open-ended Comments

Column Labels

Head

er

Foot

er

Con

ten

t R

ow

s


32

Designing FormDesigning FormHeadersHeaders

• Evaluation codes should be:As specific as possible in describing

expected performance Well matched to the behavior being

evaluatedEasy to reproduce consistently across

evaluators• Include an escape option to

minimize guesswork (e.g., NE=not evaluable).


33

Designing Form Designing Form Headers, cont’dHeaders, cont’d

Column labels can be used to establish:Evaluation time points (e.g., mid-

rotation, end of rotation; PGY level)Performance benchmarks (behaviors

expected at evaluation time points or for final certification of competence)

Action plans for learner improvementComments by learner or evaluator


34

Selecting Row Content Selecting Row Content (Evaluation Items) (Evaluation Items)

Use Goals/objectives or Competencies/ elements:

• Make items match the actual activities observed in a resident.

• Make items as specific as space allows. • Include enough items to allow a fair

sampling of a resident’s performance. Length of a form, however, cannot

exceedan evaluator’s willingness to complete it!


35

Flexible Uses ofFlexible Uses of the Footer the Footer

Open-ended comments can be used:

• By evaluator, to clarify and specify ratings

• By evaluator, to help the learner improve performance

• By learner, to comment on the evaluation or action plan


36

What Matters MostWhat Matters Most in the End in the End

• An evaluation method is only as good as the evaluator who implements it.

• Work with faculty and residents to teach them about evaluation and promote cooperation.

1 ambulatory pediatric association educational guidelines for pediatric residency tutorial 5 written...

Documents