Measuring Program Outcomes and Impacts

Ayanava MajumdarExtension Entomologist, ACES

Gulf Coast Research and Extension Center8300 State Hwy 104, Fairhope, AL 36532

Email: azm0024@auburn.eduCell phone: 251-331-8416

Fax: 251-990-8912

Objectives• Why measure outcomes and impacts?• Discuss timing of evaluations (Logic model)• Discuss 12 major evaluation techniques• How you can make evaluations useful?• Provide critical sources of information

This presentation contains information unique to the presenter and should be taken as suggestions. You may modify content of this presentation to suit your situation.

Few critical sources of information

• Developing a Logic Model (Taylor-Powell & Renner, 2000, UWEX)

• Logic Model Development Guide (W.K. Kellogg Foundation, 1998)

• Kirkpatrick’s Four Levels of Evaluation (1994), Encyclopedia of Education

Conventional measurements (old-school evaluations)

• Participant reaction: usefulness of program

• Teaching & facilitation: suggestions for improvement

• Outcomes: what did you learn today?

• Future programming: what do want to learn more about?

Taylor-Powell & Renner, 2000

…focus on outputs and immediate effects (learning)

…no information about action and conditions

…in effect, this was the push strategy (linear TOT)

Welcome to the Accountability Era!

What gets measured gets done!

If you don’t measure results, you can’t differentiate success from failure.

If you demonstrate results, you can win public support.

Osborne & Gaebler, 1992

Accountability & evolution of concepts

Outputs. The activities, products, and participation generated through the investment ofresources. Goods and services delivered.

Outcomes. Results or changes from the program such as changes in knowledge, awareness,skills, attitudes, opinions, aspirations, behavior, practice, decision-making,policies, social action, condition, or status.

Impact. The social, economic, civic and/or environmental consequences of the program. Impacts tend to be longer-term and so may be equated with goals.

Taylor-Powell & Renner, 2000

Revisiting the LOGIC MODEL

Remember…• It is not a theory, it is not a reality• It is only a MODEL…a framework for visualizing

relationshipsTaylor-Powell & Renner, 2000

Measuring success is complicated!

• Measuring outcomes, by itself, will need resources!• Assumptions and external factors create variations in

outcomes.

Types of evaluationFour basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation

When to conduct? • First thing we should be doing!• Establishes priorities

What questions to ask?• Characteristics of audience• Needs of audience (prioritize)• Where do they find information?• Best learning method• Find barriers to knowledge adoption

Don’t forget your camera, writing instruments, survey print & give time to respond

Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation When to conduct?

• During program implementation• E.g., quality survey, satisfaction survey, future needs survey

What questions to ask?• Were you satisfied with delivery methods?• Was there too much information?• Are you reaching to targeted audience?

Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation

When to conduct? • Measure learning• During on-site programs: workshops, field days, etc.

What questions to ask?• Short-term change: key words in questions “awareness”, “knowledge”, “opinion”, “motivation”• Document who is not benefiting (analyze sample and understand biases)

Part 1

Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation

When to conduct? • Measure behavioral changes• During one-to-one visits, farm visits, telephonic, mail, email…repeat surveys!

What questions to ask?• Medium-term changes: key words in questions “behavior”, “practices”, “decision”, “action”• Are you meeting goals? Unintended outcomes?

Part 2

Types of evaluation contd.Four basic types:1. Needs assessment2. Process evaluation3. Outcome evaluation4. Impact evaluation

When to conduct? • You should have partially achieved this if you did previous steps right.

What questions to ask?• Long-term changes: change in “condition” • Separate real impact from “background noise”• Try to document final consequences: new products, innovations, services, community changes, motivation to act in the absence of program

Determine some INDICATORS

Relatively EASY

Relatively DIFFICULT

Indicators: can be qualitative or quantitative

Evaluation techniques• Survey: collect standardized

information, may be mailed, done on-site, structured interviews (N, P)

• Case study: in-depth examination of particular groups or individuals (O, I)

• Interviews: face-to-face interaction, conversational, one-on-one or small groups (P, I)

• Observation: collecting information via seeing or listening, structured or unstructured (I)

Tayl

or-P

owel

l, 20

02

N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.

Evaluation techniques (contd.)• Group assessment: use of

nominal techniques like focus groups, brainstorming, community forum (N)

• Expert/peer review: examination by a review committee, Delphi method (“indicator”, I)

• Portfolio reviews: collection and presentation of materials and samples of work that indicate breadth of program (“indicator”)

• Testimonials: individual statements by people indicating personal reactions, household drop-off (O, I) Ta

ylor

-Pow

ell,

2002

N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.

Evaluation techniques (contd.)

• Tests: assess knowledge, skills, performance, e.g., pre-test & post-test (P, O)

• Photos, videos: group or one-on-one interviews ( I)

• Success or problem stories: narrative account by participants about adoption of new practices(“indicator”, N, I)

• Unobtrusive methods: gathering information without making participants aware, e.g., indirect measures, content analysis (N, P, O)

Tayl

or-P

owel

l, 20

02

N = Needs asses., P = Process eval., O = Outcome eval., I = Impact eval.

Four evaluation criteria• Utility:

– Goal: how useful is your program evaluation to you & your audience?

– Know the following information: • State purpose clearly• Consider your audience• Communicate findings & relevance of findings

Boyd, 2002

• Feasibility: – Goal: how practical is your assessment

technique?– Know the following information:

• Keep evaluation practical, nondisruptive• Calculate cost: benefit• Use appropriate evaluation technique/s

Boyd, 2002

Byod’s evaluation criteria contd.

• Appropriateness: – Goal: how appropriate is your program evaluation for

those involved?– Know the following information:

• Respect people and their rights• Use appropriate choice statements• Disclose findings properly

Boyd, 2002

Byod’s evaluation criteria contd.

Byod’s evaluation criteria contd.• Accuracy:

– Goal: how accurate is your program evaluation to you & your audience?

– Know the following information: • Design repeatable surveys• Use appropriate analyses • Draw justifiable conclusions

Boyd, 2002

Final tips on program evaluations• Consult specialist in planning phase• Think backwards in LOGIC model (impact >>

output >> input) & allocate resources• Think about “indicators” of success• If you conduct surveys, allocate time to respond

(don’t rush)• Publicize your programs, use multiple channels

>> create a “pull” system >> more success

Thank you for patient listening!

QUESTIONS FOR AYANAVA?