![Page 1: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/1.jpg)
Evelyn GonzalezProgram Evaluation
![Page 2: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/2.jpg)
AR Cancer Coalition Summit XIVMarch 12, 2013
MAKING A DIFFERENCEEvaluating Programmatic Efforts
![Page 3: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/3.jpg)
Overview of evaluationDefining SMART objectives for your
goalsKnow how to use different methods of
evaluation Be more willing to evaluate your
efforts
OBJECTIVES
![Page 4: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/4.jpg)
…the systemati c collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program eff ectiveness, and/or inform decisions about future programming.
(Patt on, Uti lizati on Focused Evaluati on, 1997)
WHAT IS PROGRAM EVALUATION?
![Page 5: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/5.jpg)
Did the program/intervention
work?
Was it worth it?
What worked; what didn’t?
Who did we reached?
Did we get our monies worth?
WHY EVALUATE
![Page 6: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/6.jpg)
WHEN SHOULD WE BEGIN EVALUATION?
![Page 7: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/7.jpg)
An evaluation plan is the “blueprint”
What will be evaluated What information will be
collected When will it be collected
What will be done with the results
EVALUATION PLAN
![Page 8: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/8.jpg)
CDC FRAMEWORK FOR EVALUATION
![Page 9: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/9.jpg)
• Collect
• Analyze Data
START
• Implementation of the program
• Gather data as you go
• Monitor
Planning
• Establish Goals & Objectives
• Establish baseline •Identify an Evidence- Base Program (EBP)
Evaluation
• As you implement
• End of program
Community/Audience
Stakeholders
Planning Phase
Implementation
Phase
Evaluation
Phase
Involve Stakeholders
Share results with Community & Stakeholders
![Page 10: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/10.jpg)
The “grand reason” for engaging in your public health effort
Span 3 or more years
State the desired end result of the program.
GOALS: DEFINITION
![Page 11: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/11.jpg)
More specific than goals.
They state how the goals will be achieved in a certain timeframe.
Well written objectives are SMART:
Specific
Measurable
Achievable
Realistic and Relevant
Time-framed
OBJECTIVES: DEFINITION
![Page 12: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/12.jpg)
Specific
Who are you reaching (priority audience)?
What intervention will you use?
Where, setting
S.M.A.R.T.
![Page 13: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/13.jpg)
Measurable
Dosing, how many times will you do the intervention
What is the expected outcomeIncrease of X% following the intervention
Decrease of smoking by X%
S.M.A.R.T.
![Page 14: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/14.jpg)
AttainableIs your intervention feasible?
Realistic and RelevantDoes the objective match the goal?
Is it evidence-based program (EBP)?
S.M.A.R.T.
![Page 15: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/15.jpg)
Time-framedBy when do you anticipate the change?End of the session
3,6,9 months
5 years
S.M.A.R.T.
![Page 16: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/16.jpg)
You are working on an intervention that will increase awareness about breast cancer risk
Objective 1: Participants will be aware of the major risk factors for developing breast cancer.
How can this be re-written to be SMART?
SMART OBJECTIVE EXERCISE
![Page 17: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/17.jpg)
Original:Participants will be aware of the major risk factors for developing breast cancer.
SMART Objective:Upon post test following the intervention, participants will be able to identify 3 major risk factors for developing breast cancer.
SMART OBJECTIVE EXERCISE
![Page 18: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/18.jpg)
Original:This program will increase screening for colorectal cancer in Arkansas.
SMART:Colorectal screening will be increased by 5%, over the prior year for age appropriate males in Arkansas.
RE-WRITTEN:
![Page 19: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/19.jpg)
Objective 1: Public Education for Breast Cancer Screening –
Increase knowledge and improve attitudes of all women withregards to the importance of breast cancer screening
Strategy 1 – Promote campaigns to educate the public about the importance of mammography.
Action 1 – Increase awareness among all women 40 and older of the importance of regular breast cancer screening
GOAL: PROMOTE AND INCREASE THE APPROPRIATE UTILIZATION
OF HIGH-QUALITY BREAST CANCER SCREENING
![Page 20: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/20.jpg)
Planning—Develop the questions, consult with the program stakeholders or resources, make a timeline
Data Collection—Pilot testing. How will the questions be asked? Who will ask them?
Data Analysis—Who will analyze the data and how?
Reporting—Who will report and how? Who will receive the data and when? How will it affect the program
Application—How could your results be applied in other places?
THE EVALUATION PROCEDURE
![Page 21: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/21.jpg)
Look at the evaluation methods used in the original EBP.
When discussing evaluation, think about these questions:What is important to know?What do you need to know versus what is nice to know?
What will be measured and how?How will this information be used?
PLANNING FOR EVALUATION
![Page 22: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/22.jpg)
![Page 23: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/23.jpg)
Indicators or measures are the observable and measurable data that are used to track a program’s progress in achieving its goals.
Monitoring (program or outcome monitoring, for example) refers to on-going measurement activity
SOME DEFINITIONS…
![Page 24: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/24.jpg)
Process evaluation can find problems early on in the program. It includes an assessment of the staff, budget review, and how well the program is doing overall.
For this kind of evaluation, it may be useful to keep a log sheet to record each of your activities.
From Windsor et al., 1994
PROCESS EVALUATION
![Page 25: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/25.jpg)
Impact evaluation can tell if the program has a short-term effect on the behavior, knowledge, and attitudes of your population.
It also measures the extent to which you have met your objectives.
From Green and Kreuter, 1991
IMPACT EVALUATION
![Page 26: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/26.jpg)
Outcome evaluation looks to see if the long-term program goals were met.
These goals could be changes in rates of illness or death, as well as in the health status of your population.
From McKenzie & Smeltzer, 1997
OUTCOME EVALUATION
![Page 27: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/27.jpg)
Identify Program Goals
For each goal: Identify Process Objectives
Identify Outcome Objectives
For each objective: Identify Indicators
Identify Data Source
Plan Data Collection
Plan Data Analysis
APPLICATION TO YOUR PROGRAM:
![Page 28: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/28.jpg)
DATA COLLECTION METHODS
SurveysInterviewsFocus GroupsObservationDocument Review
![Page 29: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/29.jpg)
You may develop a way to compare the baseline data from the needs assessment with the final outcome of your program.
Pre/Post survey in an education session.
This will let you see if you have achieved your objectives.
PRE- AND POST-EVALUATION
![Page 30: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/30.jpg)
Primary sources Quantitative:
Surveys/questionnaires Qualitative: Focus groups, public
meetings, direct observation Qualitative: In-depth interviews
with community leaders, interviews with other program planners.
INFORMATION COLLECTION
![Page 31: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/31.jpg)
Will depend on which EBP/Intervention selected
Answer these questions: What specific behaviors do I want my audience to acquire or enhance?
What information or skills do they need to learn to act in a new way?
What resources do I need to carry out the program?
What methods would best help me meet my objectives?
STRATEGIES
![Page 32: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/32.jpg)
USING MIXED DATA SOURCES/METHODS
Involves using more than one data source and/or data collection method.
![Page 33: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/33.jpg)
Your objectives should be measurable so that they can be evaluated.
The evaluation should be in line with your objectives.
Try not to make up new things to evaluate.
PROGRAM OBJECTIVES AND EVALUATION
![Page 34: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/34.jpg)
![Page 35: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/35.jpg)
You may want to do a pilot test in order to evaluate the effect of your program.
A pilot test is a practice run using a small group who are similar to your target audience.
PILOT TESTING
![Page 36: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/36.jpg)
Evidence-based programs have already done some type of evaluation.
Look to see how the program was evaluated before. Try to use the same methods.
You do not have to evaluate everything!
REPLICATING THE EVALUATION
![Page 37: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/37.jpg)
MONITORING PROGRESS
![Page 38: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/38.jpg)
NOW THAT YOU’VE COLLECTED THE DATA, WHAT DO YOU DO WITH IT?
Analyzing data Who When How
Interpretation of results and sharing findings
![Page 39: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/39.jpg)
Must be able to answer this!
Do not just look for the good outcomes
Learn from what didn’t work
Share both the positive and negative outcomes
SO WHAT?
![Page 40: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/40.jpg)
DEVELOPING RECOMMENDATIONS
Your evaluation’s recommendations should be:
Linked with the original goals/SMART objectives.
Based on answers to your evaluation questions.
Should have stakeholder input
Tailored to the end users of the evaluation results to increase ownership and motivation to act.
![Page 41: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/41.jpg)
SHARING RECOMMENDATIONS
CommunityExecutive SummaryFinal ReportNewsletter article(s)Website articleTown hall
meeting(s)Radio interviewsLocal newspapers
Institution & YourselfExecutive SummaryFinal ReportJournal articlesProfessional
conferencesPoster sessionsMeetings with
colleagues
![Page 42: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/42.jpg)
TIPS & CONSIDERATIONS
Consult with partners with evaluation experience
Budget 10-15% for evaluation Staffing Build a database Analysis
Consider pilot testing your programPilot test your evaluation method &
tool(s)
![Page 43: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/43.jpg)
Trust yourself. You know more than you think you
do!
Benjamin Spock
![Page 44: Evelyn Gonzalez Program Evaluation. AR Cancer Coalition Summit XIV March 12, 2013 MAKING A DIFFERENCE Evaluating Programmatic Efforts](https://reader036.vdocuments.mx/reader036/viewer/2022062517/56649e885503460f94b8c613/html5/thumbnails/44.jpg)