Department of Graduate Medical Education (GME)1

Faculty Development Session on Evaluations

Graduate Medical EducationAnn Dohn, MA,

Designated Institutional Official (DIO)

Nancy Piro, PhDProgram Manger/Education Specialist

Kim Walker, PhDProgram Manger/Education Specialist

Department of Graduate Medical Education (GME)2

ACGME Requirements: Seven Domains

I. InstitutionII. Program Personnel & ResourcesIII. Resident AppointmentsIV. Educational ProgramV. Evaluation

I. ResidentII. FacultyIII. Program

VI. Resident Duty HoursVII. Experimentation & Innovation

Department of Graduate Medical Education (GME)

Faculty Development SessionEVALUATIONS

3

How do we eliminate unintended bias from our evaluation process?

Our focus today:

Department of Graduate Medical Education (GME)4

What is cognitive bias…

• Cognitive bias is distortion in the way we perceive reality / information.

• Response bias is a type of cognitive bias which can affect the results of an evaluation if evaluators answer questions in the way they think they are designed to be answered, or with a positive or negative bias toward the fellow being evaluated

Department of Graduate Medical Education (GME)5

Where does response bias occur?

• Response bias most occurs most often in the wording of the question. – Response bias is present when a question

contains a leading phrase or words. – Response bias can also occur in rating scales.

• Response bias can be in the raters themselves

– Central Tendency– Halo Effect– Similarity Effect

Department of Graduate Medical Education (GME)6

Examples of Question Bias

• Example 1: "I can always talk to my Program Director about residency related problems."

• Problem: Terms such as "always" and "never" will bias the response in the opposite direction.

• Result: Data will be skewed.

Department of Graduate Medical Education (GME)7

Examples of Question Bias

• Example 2: “Career planning resources are available to me and my program director supports my professional aspirations."

• Problem: Double-barreled ---resources and aspirations… Respondents may agree with one and not the other. Evaluator cannot make assumptions about which part of the question respondents were rating.

• Result: Data is useless.

Department of Graduate Medical Education (GME)8

Examples of Question Bias

• Example 3: "Communication in my program is good."

• Problem: Question is too broad. If score is less than 100% positive, researcher/evaluator still does not know what aspect of communication needs improvement.

• Result: Data is of little or no usefulness.

Department of Graduate Medical Education (GME)9

Rating Scale Bias

Competence and knowledge in general medicine.

Poor Fair Good Very Good Excellent

The data will be artificially skewed in the positive direction with this scale because there are far more (4:1) positive than negative rating options.

Department of Graduate Medical Education (GME)10

Rater/Evaluator Bias

Response bias can be in the evaluators themselves• Central Tendency• Similarity Effect• Halo Effect

Department of Graduate Medical Education (GME)11

Beware the Halo Effect

• The halo effect refers to a cognitive bias whereby the perception of a particular behavior or trait is influenced by the perception of the former traits in a sequence of interpretations.

• Thorndike (1920) was the first to support the halo effect with empirical research. – People seem not to think of other individuals in

mixed terms; instead we seem to see each person as roughly good or roughly bad across all categories of measurement.

Department of Graduate Medical Education (GME)12

The Halo Effect and Expectations

• The halo effect is involved in Kelley's implicit personality theory– the first traits we recognize in other

people influence our interpretation and perception of later ones because of our expectations.

Department of Graduate Medical Education (GME)13

Halo Effect Extends to Products / Marketing

The iPod has had positive effects on perceptions of Apple’s other products…

Department of Graduate Medical Education (GME)14

Could this impact our evaluations here?

Empirical evidence from our HouseStaff…. A question from the most recent GME HouseStaff survey:

“The general feeling in my program is that your ability will

be labeled based on your initial performance.”

54.7%

24.4%

15.6%

71.8%

17.9%

10.3%

Overall SHC-LPCH Overall Peds Fellows

Department of Graduate Medical Education (GME)15

Reverse Halo Effect

• A corollary to the halo effect is the reverse halo effect (devil effect) –individuals, brands or other things

judged to have a single undesirable trait are subsequently judged to have many poor traits, allowing a single weak point or negative trait to influence others' perception of the person, brand or other thing in general.

Department of Graduate Medical Education (GME)16

Blind Spots

• In the 1970s, the social psychologist Richard Nisbett demonstrated that we may have no awareness of when the halo effect influences us (Nisbett, R.E. and Wilson, T.D., 1977)

The problem with Blind Spots is that we are blind to them…

Department of Graduate Medical Education (GME)17

Know the ACGME Core Competenciesand How You are Evaluating Them

Department of Graduate Medical Education (GME)18

Questions?