ExtensionProgram EvaluationDale A. MooreVMTRCSchool of Veterinary MedicineUC Davis

What’s the purpose of evaluation? Improve the program Improve teaching Make a difference Scholarly pursuit Get the higher-ups off your back Meet promotion requirements

Goals of This Presentation Discuss “traditional” program evaluation

Help you focus your evaluations Learn the different levels of evaluation Help you construct evaluation questions Provide you a tool for data analysis

Discuss educational program evaluation using Learning Stage Theory

Pre-Test Take out the Pre-Test Take 5 minutes to

answer the questions Turn them in to Pat

Who will use the results? The public Your extension division Colleagues You and program planning staff … All your stakeholders

Who will use the results?

Your Audience

Do I have the resources to conduct this evaluation project? Time Money People Expertise Or – are the data already available?

Levels of Evaluation Perceptions of the program or course Competence with new skills, knowledge,

abilities or new attitudes Individual performance – a change in

behavior within their environment Industry outcomes

What are your outcomes?

Mechanisms for Evaluation Direct observation Audience response systems Surveys Telephone or face-to-face interviews Focus groups Expert or peer-review (formative evaluations) Testimonials Industry data

Audience Response System

Surveys Who is the audience? How are you going to deliver it? Are you going to randomly choose possible

responders? What percent response do you need to make it

meaningful? Pre- and Post-program delivery to address

immediate changes in knowledge/attitudes?

Pre-Test --–-- Post-Test

Pre-Test --–-- Post-Test Advantages: Get attention at beginning of the program Provide some level of anxiety so that they

“look” for the answers during the presentation Gives instructors some measure of learning

needs / areas upon which to focus Identify short-term changes in knowledge /

attitudes

Pre-Test – Post-Test Disadvantages: Takes time away from the program Need to be able to quickly “grade” and analyze

your data Need to be able to make adjustments to the

program if needed Only addresses short-term learning and not

behavior change Adults may not enjoy being graded

What kind of questions? Reaction Learning Behavior Results

?

What kind of questions/data if the outcome is behavior change?

Questions about… Satisfaction with different aspects of the

program Speaker effectiveness New knowledge or skills A change in attitude A “commitment to change” New program ideas

Question Types Open- ended questions Closed

Set of responses Likert scale Numeric responses Categorical responses

What are the advantages and disadvantages of each kind of question?

Data Analysis How do you go about analyzing responses to

different kinds of questions? Comparison of means – Student’s T-test Comparison of proportions Categorical data analysis – Chi-Square Non-parametric like the Sign test Multivariate analyses

* EpiCalc2000 – free program from Web; URL in handout

Documenting the Impacts of Continuing Education

Dale Moore and Hank Slotnick

Do you sometimes feel that…

despite…• Results of your needs assessments, and • Your best intentions to create an appealing

program

Those who need to attend the program, don’t? The minds of some who come aren’t open?

Objectives Provide the theoretical framework for

documenting change Demonstrate applications of the theory Demonstrate a new method of

representing change data

Background on the Problem Key meta-analytic findings of CME programs:

Findings consistent with adult learning: Simple provision of information is often

insufficient. Active learning appears to be more effective. Multiple exposures predispose toward success.

Most traditional CME does not changedoes not change physician behavior

Why not?

Behavioral Change Theories Prochaska & Mezirow & Slotnick

Suggest that change occurs in stages In each stage the learner has different needs To become interested in the idea of change

• the learner’s mind must be opened• the learner must feel a need to attend to

something, something has happened to unsettle them, or they have an immediate problem to solve

INTERESTED, BUT…. (contemplation)

CHOICES:DECISIONS (preparation)

ACTION

FINE TUNING & MAINTAINING CHANGE (maintenance)

UNAWARE OR UNIMPRESSED (precontemplation)

Adapted from: Prochaska et al, 1992

STAGES OF CHANGE

Mezirow: Theory of Perspective Transformation

1) Disorienting dilemma

2) Self-examination with feelings dissonance or discomfort

3) Critical re-assessment of assumptions

4) Recognition of discontent and that others have changed

5) Exploration of options for new ways of seeing/behaving

6) Planning a course of action, gathering resources

7) Acquisition of knowledge and skills

8) Provisional trying of new perspective/behavior

9) Building of competence and self-confidence in new activity

10) Integration of new perspective/behavior into one’s life

Traditional CE Efforts

Potential for mismatch between the intervention and learner readiness Over-emphasis on encouraging action

Ask them to take the leap because we say so! Presumes learners minds are open

Documenting the Outcomes A taxonomy of outcomes

Awareness of problem Knowledge/skill changes and Practice changes Improved herd outcomes

Aware

ChangeImproveSlotnick 2001

Taxonomy of Outcomes Taxonomy based on stages in

learning episodes Evaluation: Learner decides

whether to learn the solution to the problem at hand

Learning: Learner gains required skills and knowledge

Gaining experience: Learner familiarized with the skills, knowledge & problems.

Aware

ChangeImprove

Evaluation Learn

Gain Experience

Documenting Change

Hypothesis: It should be possible to document changes in prevalence of learners at different stages in addressing an educational program.

How to Document Outcomes Present clinical/practical vignettes. Ask respondents to indicate their ‘learning status’ for each

vignette/problem. Vignette example: Mary was a 58-year-old woman who had

been in good health generally. She did not exercise regularly and was about 20 pounds over her ideal weight, but her diet was relatively low in fat, she did not smoke, and she had no history of diabetes. She had been taking estrogen and progesterone hormone replacement therapy for 6 years. Her recent blood lipid panel results were as follows: (blah blah blah). She was taking hydrochlorothiazide to treat hypertension, and her mother had experienced an MI at 60 years of age. Mary was seen at the emergency department for a prolonged episode of left shoulder and upper back pain which had begun after dinner that evening.

Responses Evaluation: I’m comfortable that I can handle the

problem in the vignette; all I might be interested in is information so I can decide when next to update.”

Learning: I need to update myself in the skills and knowledge needed to address this problem.

Gaining experience: I’ve recently updated skills and knowledge. All I might be interested in is hearing about others’ experiences with similar problems.

How to Document Outcomes The AMEE (Slotnick) study

Goal: Introduce the idea of estimating prevalence for people at different stages of learning in a lecture setting (90 minutes)

Measurement: Practice vignettes were examined pre- and post, some based on issues addressed in the lecture and some not.

Analysis: Expected to see stage-to-stage movement for the first set, but not the second.

The Triangular Graph

Each side representsa response – 3 possibleresponses.

Evaluation

Learning GainingExperience

0

10

20

30

40

50

Eval Learning Gain Exp

Response

% R

espo

nse

50%

50% 50%

Eval Dominated

Learning Domin. Gain Exp.

Evaluation

Learning GainingExperience

0

10

20

30

40

50

Eval Learning Gain Exp

Response

% R

espo

nse

50%

50% 50%

Evaluation

Learning GainingExperience

0

10

20

30

40

50

Eval Learning Gain Exp

Response

% R

espo

nse

50%

50% 50%

How to Document Outcomes The AMEE study.

3 Vignettes where no changes were expected.

Evaluation

Learning GainingExperience

“The task force chair needs a list of physicians who might be called on in an emergency. She asks you to locate such a list.”

How to Document Outcomes The AMEE study.

4 Vignettes where changes were expected.

Evaluation

Learning GainingExperience

“You and two others are to put together a pencil-and-paper survey questionnaire to determine what doctors know about preventing, diagnosing, and managing anthrax and small pox.”

Dairy Veterinarian Study

Dairy Informatics Vignettes

“A 500-cow dairy client is going to expand his operation to add 500 more cows. To help support his application for a loan, the bank has asked him for information on the productivity of the dairy herd, including milk production and herd replacements. The bank would like information for the last 3 years of the dairy’s operation.”

Developed 8 practice vignettes: An example --

Moore’s Study

Evaluation

Learning Gaining Experience

Moore’s Study

Evaluation

Learn Gain Experience

Conclusions Evaluate group changes dependent on

stage of learning Documents the changes Provides for parsimonious graphical

display of change Currently working on statistical methods

Post-Test Take out the Post-

Test Take 5 minutes to

complete Turn in to Pat

What did we learn? There are different levels of program evaluation Clearly-defined program objectives lead to

easier evaluations There are a number of mechanisms of program

evaluation Learning stage theory helps us evaluate WHERE

the learner is with regard to making a change

Thank you for

your attention

References Dr. Jack Mezirow: Transformative Dimensions of Adult

Learning; Fostering Critical Reflection in Adulthood: A Guide to Transformative and Emancipatory Learning.

Prochaska, J. O. and C. C. DiClemente (1982). "Transtheoretical therapy: Toward a more integrative model of change." Psychotherapy: Theory, Research and Practice 19(3): 276-288.

Slotnick HB. How Doctors Learn: Physicians' Self-directed Learning Episodes. Acad Med 1999;74:1106-17.

Moore DA. Evaluating the stages of veterinary practitioner learning for continuing education needs assessment and program evaluation. J Vet Med Education 2003;30(1):50-56.