evaluation: from objectives to outcomes janet myers, phd mph aids education and training centers...
DESCRIPTION
What is Program Evaluation? Evaluation is the systematic collection of information about a program in order to enable stakeholders: to better understand the program, to improve program effectiveness, and/or to make decisions about future programming. Program evaluation is the use of social research procedures to systematically investigate the effectiveness of … programs.TRANSCRIPT
![Page 1: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/1.jpg)
Evaluation:from Objectives to Outcomes
Janet Myers, PhD MPHAIDS Education and Training Centers National Evaluation Centerwww.ucsf.edu/aetcnec
![Page 2: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/2.jpg)
Overview
Evaluation overview Why evaluate? Case studies
![Page 3: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/3.jpg)
What is Program Evaluation?
Evaluation is the systematic collection of information about a program in order to enable stakeholders: to better understand the program, to improve program effectiveness, and/or to make decisions about future programming.
Program evaluation is the use of social research procedures to systematically investigate the effectiveness of … programs.
![Page 4: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/4.jpg)
Why Evaluate?
Ensure program effectiveness and appropriateness
Demonstrate accountability Contribute to HIV/AIDS knowledge
base Improve program operations and
service delivery
![Page 5: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/5.jpg)
Components of Program Evaluation
There are 3 general components to comprehensive program evaluation: Process evaluation: How was the program
implemented? Outcome evaluation: Did the program meet
its objectives? Impact evaluation: Was the ultimate goal of
the program achieved?
![Page 6: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/6.jpg)
Every program has…
Evaluation 101
Every program evaluation should have…
Impact Indicators
Outcome Indicators
Process Indicators
GoalsGoals
ObjectivesObjectives
ActivitiesActivities
![Page 7: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/7.jpg)
Every program has…
Evaluation 101
Every program evaluation should have…
Impact Indicators
Outcome Indicators
Process Indicators
GoalsGoals
ObjectivesObjectives
ActivitiesActivities
![Page 8: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/8.jpg)
What is Process Evaluation? Process evaluation:
Addresses how, and how well, the program is functioning
It can help to… Create a better learning environment Improve presentation skills Show accountability to funder Reflect the target populations Track service units
![Page 9: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/9.jpg)
Process Evaluation con’t
Key questions in process evaluation: Who is served? What activities or services are
provided? Where, when, and how long is the
program?
![Page 10: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/10.jpg)
Process Evaluation con’t
Identify how a product or outcome is produced
Identify strengths & weaknesses of a program
Create detailed description of the program
![Page 11: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/11.jpg)
Every program has…
Evaluation 101
Every program evaluation should have…
Impact Indicators
Outcome Indicators
Process Indicators
GoalsGoals
ObjectivesObjectives
ActivitiesActivities
![Page 12: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/12.jpg)
Outcome Evaluation
Outcome evaluation: Measures the extent to which a
program produces its intended improvements
Examines effectiveness, goal attainment and unintended outcomes
In simple terms, “What’s different as a result of your efforts?”
![Page 13: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/13.jpg)
Outcome Evaluation con’t Key questions in outcome evaluation:
To what degree did the desired change(s) occur?
Outcomes can be immediate, intermediate or longer-term
Outcomes can be measured at the patient, provider, organization, or system level.
![Page 14: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/14.jpg)
Every program has…
Evaluation 101
Every program evaluation should have…
Impact Indicators
Outcome Indicators
Process Indicators
GoalsGoals
ObjectivesObjectives
ActivitiesActivities
![Page 15: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/15.jpg)
Impact Evaluation
Impact is sometimes used to mean “outcome.”
Impact is perhaps better defined as a longer-term outcome. For clinical training programs, impacts may be improved patient outcomes.
In global M&E, incidence or prevalence of disease
![Page 16: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/16.jpg)
A note about impact…
Most program evaluations focus on measuring the process and outcomes of a program.
Measuring impact requires significant resources that most programs don’t have.
It’s also difficult to link the more immediate effects of a program to broad, often community level, impacts.
![Page 17: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/17.jpg)
1. Identify program goals and objectives2. Define the scope of the evaluation3. Define evaluation questions & indicators4. Define methods5. Design instruments and tools6. Carry out the evaluation7. Analyze data and write a report8. Disseminate and use data
Essential Steps to Evaluation (FHI, Impact, USAID manual)
![Page 18: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/18.jpg)
Program Goals and Objectives Well developed goals and objectives are
critical to evaluation. Objectives are specific steps that contribute
to a goal. Often several objectives per goal. Good objectives are SMART:
SS – specific MM – measurable AA – attainable RR – realistic TT – time-bound
![Page 19: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/19.jpg)
Good objectives include (McKenzie & Smeltzer 2001)
1. What will change: Outcome that will be achieved
2. When will it change: Conditions under which the outcomes will be observed
3. How much change: Criterion for deciding whether the outcomes has been achieved
4. Who will change: Target population
![Page 20: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/20.jpg)
Scope of the Evaluation Determine your resources
StaffingTimeMaterials$$$
![Page 21: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/21.jpg)
Questions & Indicators Figure out your questions:
What will this be used for? Guided by objectives…select
process and outcome indicators Relevant Measurable Improvable
![Page 22: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/22.jpg)
Methods, Instruments, Tools
Some questions to ask: Primary v. secondary data? Qualitative v. quantitative?
Instrument/Tool DevelopmentDon’t reinvent the wheel!
![Page 23: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/23.jpg)
Research Design
Qualitative methods: interviews, focus groups, observation, document analysis
Quantitative methods: surveys, medical record abstraction, pre-test, post-test
This is another course…
![Page 24: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/24.jpg)
Analysis
Evaluation is not clinical trials research.
Analysis can be straightforward. Easy stats are often more
useful, depending on audience.
![Page 25: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/25.jpg)
Dissemination
Essential in Evaluation Planning for it is important Framing is important Think about broad audience
(consumers, stakeholders, policymakers)
See: www.caps.ucsf.edu/dissemination/
![Page 26: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/26.jpg)
Resources galore…
http://www.ucsf.edu/aetcnec/
![Page 27: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/27.jpg)
Some Case Studies…
![Page 28: Evaluation: from Objectives to Outcomes Janet Myers, PhD MPH AIDS Education and Training Centers National Evaluation Center](https://reader036.vdocuments.mx/reader036/viewer/2022062523/5a4d1aeb7f8b9ab05997b092/html5/thumbnails/28.jpg)
AETC Program