program evaluation webinar series part 2:
DESCRIPTION
Program Evaluation Webinar Series Part 2:. “Getting Started and Engaging Your Stakeholders”. Presented by: Leslie Fierro and Carlyn Orians. Getting Started and Engaging Your Stakeholders. Leslie A. Fierro , MPH TKCIS Contractor NCEH/CDC [email protected]. Carlyn Orians , MA, - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/1.jpg)
Program Evaluation Webinar Series Part 2:“Getting Started and
Engaging Your Stakeholders”Presented by: Leslie Fierro and Carlyn Orians
![Page 2: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/2.jpg)
Getting Started and Engaging Your Stakeholders
Leslie A. Fierro, MPHTKCIS Contractor NCEH/[email protected]
Carlyn Orians, MA, Battelle Centers for Public Health Research and [email protected] Presented November 20, 2008
![Page 3: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/3.jpg)
Agenda
Definition of evaluation.
Difference between evaluation & research.
CDC’s Evaluation Framework.
The who, why, when & how of stakeholders.
Tangible examples of engaging stakeholders.
![Page 4: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/4.jpg)
What is the Purpose of Evaluation?
The purpose of evaluation can be:
social betterment
program planning and decision making
empowerment of individuals
![Page 5: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/5.jpg)
Evaluation: Definition 1
Evaluation is the systematic investigation of the merit, worth, or significance of an object.
-- Michael Scriven
![Page 6: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/6.jpg)
Evaluation: Definition 2
Evaluation is the systematic assessment of the operation and/or the outcomes of a program or policy, compared to a set of explicit or implicit standards as a means of contributing to the improvement of the program or policy.
-- Carol Weiss
![Page 7: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/7.jpg)
Evaluation: Definition 3
Evaluation is the systematic collection of information about the activities, characteristics, and outcomes of programs to make judgments about the program, improve program effectiveness, and/or inform decisions about future programming.
-- Michael Patton
![Page 8: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/8.jpg)
The Common Element
Note that all 3 of these statements share a common element. ..
…they all define evaluation as a systematic and formalized endeavor.
![Page 9: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/9.jpg)
Summative Evaluations
Summative evaluations seek to judge a program by asking, “Should this project be…”
cancelled? continued? expanded?
![Page 10: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/10.jpg)
Formative Evaluations
Formative evaluations seek to use the evaluation findings. They ask the question:
“Is the program being conducted as planned?”“Is the program doing well?”
Formative evaluations seek to improve programs or policies.
![Page 11: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/11.jpg)
The CDC Definition
Evaluation is the systematic collection of information about the activities, characteristics and outcomes of the program to make judgments about the program, improve program effectiveness and/or inform decisions about future program development.
![Page 12: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/12.jpg)
Research vs. Evaluation
Specific to Research
Specific toEvaluation
Research and evaluation share methodologies but ask different questions.
Commonalities
![Page 13: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/13.jpg)
The Research Context
Purpose To add to an existing body of theoretical knowledge.
Typical Questions
Does the theory hold in this situation?How can we improve the theory to make
better predictions in every situation?
![Page 14: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/14.jpg)
Research Findings
test and improve theories.develop generalizable theories.theories apply across different
settings, people, and times.
Research is conducted to:
![Page 15: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/15.jpg)
The Evaluation Context
PurposeJudge the merit or worth of a program.
Provide information for decision making.
Typical Questions
Is the program doing what was intended?
What steps might be taken to improve it?
![Page 16: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/16.jpg)
Evaluation Findings
Evaluation results:
are not usually generalizable. focus on a specific situation. evaluation of a single program in a particular context.
![Page 17: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/17.jpg)
Research vs. Evaluation
Evaluation asks: “Is this program working?”
Research asks:“Will this program work across multiple settings?”
![Page 18: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/18.jpg)
The Role of Researchers
Researchers: Play a single role as content experts.
![Page 19: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/19.jpg)
The Role of Researchers
Researchers: Play a single role as content experts.Identify gaps in current knowledge.
![Page 20: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/20.jpg)
The Role of Researchers
Researchers: Play a single role as content experts.Identify gaps in current knowledge.Derive their own questions.
![Page 21: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/21.jpg)
The Role of Researchers
Researchers: Play a single role as content experts.Identify gaps in current knowledge.Derive their own questions.Perform their own research.
![Page 22: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/22.jpg)
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.
![Page 23: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/23.jpg)
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.Involve stakeholders.
![Page 24: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/24.jpg)
The Role of Evaluators
Evaluators:Play multiple roles—facilitator, educator, scientific expert, etc.Involve stakeholders.Collaborate to identify and prioritize questions.
![Page 25: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/25.jpg)
“Research seeks to prove, evaluation seeks to improve.”
M.Q. Patton
In a Nutshell…
![Page 26: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/26.jpg)
The Findings Must be Useful
To improve a program, the findings must be useful! How?
The stakeholders must be involved. The questions must be relevant to the program.The findings must be credible to key stakeholders.
![Page 27: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/27.jpg)
Getting Started and Engaging Your Stakeholders
The Who, When, Why, and How of Stakeholder Involvement
in CDC’s Evaluation Framework
![Page 28: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/28.jpg)
“Who” are the Stakeholders?
Stakeholders are:people and/or organizations that are interested in the program, are interested in the results of the evaluation and/or have a stake in what will be done with the results of the evaluation.
![Page 29: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/29.jpg)
Examples of Stakeholders
Persons involved in program operations• Example: asthma program staff and partners
Persons served or affected by the program• Example: participants in asthma interventions,
asthma patients, families, caregivers, etc.Intended users of evaluation findings• Example: policy makers, managers,
administrators, advocates, funders, and others
![Page 30: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/30.jpg)
Each Program is Different
Develop a list of stakeholders at the start of any evaluation activity.
![Page 31: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/31.jpg)
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we need to…”
![Page 32: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/32.jpg)
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we need to…” enhance credibility?
![Page 33: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/33.jpg)
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we need to…” enhance credibility? implement program changes?
![Page 34: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/34.jpg)
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we need to…” enhance credibility? implement program changes? advocate for changes?
![Page 35: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/35.jpg)
Which Stakeholders Matter Most?
Review your list of stakeholders and think strategically about these questions:
“Who do we need to…” enhance credibility? implement program changes? advocate for changes? fund, authorize, or expand the program?
![Page 36: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/36.jpg)
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.
Involving stakeholders may be a requirement of your program.
![Page 37: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/37.jpg)
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework
Stakeholders will add credibility.
If you want the results to be acted upon, they must be credible.
![Page 38: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/38.jpg)
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.Stakeholders will add credibility.
Stakeholders may have resources to help.
Stakeholders may be able to contribute data, skills, analytical skills, etc.
![Page 39: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/39.jpg)
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.Stakeholders will add credibility. Stakeholders may have resources to help.
Stakeholders may be critical to implementing or advocating for action based on the results.
![Page 40: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/40.jpg)
“Why” Engage Stakeholders?
Engaging stakeholders is an important part of the CDC Evaluation Framework.Stakeholders will add credibility. Stakeholders may have resources to help.Stakeholders may be critical to implementing or advocating for action based on the results.You will build trust and understanding among program constituents. Involving stakeholders helps to reduce fear of the evaluation process.
![Page 41: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/41.jpg)
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.
![Page 42: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/42.jpg)
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.Learn about evaluation.
![Page 43: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/43.jpg)
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.Learn about evaluation.Influence the design and methods.
![Page 44: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/44.jpg)
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.Learn about evaluation.Influence the design and methods.Protect their constituents.
![Page 45: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/45.jpg)
“Why” Would Stakeholders Want to be Involved?
Get answers to their questions.Learn about evaluation.Influence the design and methods.Protect their constituents.Motivated to help program succeed.
![Page 46: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/46.jpg)
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEPS
StandardsUtility
FeasibilityProprietyAccuracy
![Page 47: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/47.jpg)
“When” and “How” Can You Engage Stakeholders?
You can use the CDC Evaluation Framework to engage stakeholders in: identifying and prioritizing evaluation
questions, selecting credible sources, and developing a plan for action based on
evaluation results.
![Page 48: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/48.jpg)
“When” and “How” Can You Engage Stakeholders?
You can engage stakeholders in every step of the evaluation process.
But… you don’t have to engage stakeholders in all these ways in every evaluation.
![Page 49: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/49.jpg)
“How” Depends on…
Evaluator preference.
![Page 50: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/50.jpg)
“How” Depends on…
Evaluator preference.Stakeholder preference.
![Page 51: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/51.jpg)
“How” Depends on…
Evaluator preference.Stakeholder preference.Resources.
![Page 52: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/52.jpg)
“How” Depends on…
Evaluator preference.Stakeholder preference.Resources.Degree of trust or threats to credibility.
![Page 53: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/53.jpg)
“How” Depends on…
Evaluator preference.Stakeholder preference.Resources.Degree of trust or threats to credibility.
If there is a high degree of mistrust, engage stakeholders early in the evaluation process. This helps ensure that the results are viewed as credible and are acted upon.
![Page 54: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/54.jpg)
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEPS
StandardsUtility
FeasibilityProprietyAccuracy
![Page 55: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/55.jpg)
Who should be involved? Develop list of
potential stakeholders.
Decide which stakeholders are the most important to include.
CDC’s Framework Step 1
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Engage stakeholders
![Page 56: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/56.jpg)
CDC’s Framework Step 2Do stakeholders share a vision of what the program does and its intended outcomes?
Diverse views? Similar views? Engage them early in the
process.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Describe the
program
![Page 57: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/57.jpg)
CDC’s Framework Step 3What are the most pressing and important evaluation questions for stakeholders?
What questions do they need answered to be able to take action?
What methods are available and preferred?
What will be considered “credible evidence”?
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Focus the evaluation
design
![Page 58: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/58.jpg)
CDC’s Framework Step 4Gather evidence stakeholders will find credible.Ensure use
and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Gather credible evidence
![Page 59: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/59.jpg)
CDC’s Framework Step 5How do diverse stakeholders interpret the findings?
May engage stakeholders in the analysis.
Perhaps solicit their interpretation before results are finalized.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Justify conclusions
![Page 60: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/60.jpg)
CDC’s Framework Step 6Which stakeholders can play a role in disseminating results or acting on findings?
This is a critical step for stakeholder involvement.
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEP 1
StandardsUtility
FeasibilityProprietyAccuracy
Ensure use and
share lessons learned
![Page 61: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/61.jpg)
Getting Started and Engaging Your Stakeholders
Tangible Examples ofStakeholder Involvementin the Evaluation Process
![Page 62: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/62.jpg)
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
![Page 63: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/63.jpg)
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:• Provide education and training for apartment
owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.
![Page 64: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/64.jpg)
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.
• Work with city officials to enhance existing housing code.
![Page 65: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/65.jpg)
Example #1 – Asthma and Home Environment in Low-Income Apts
Intervention to improve indoor environment for children with asthma via following activities:
Provide education and training for apartment owners, building inspectors, maintenance vendors, and tenants regarding asthma triggers and housing codes.
Work with city officials to enhance existing housing code.
• Promote smoke-free housing.
![Page 66: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/66.jpg)
CDC’s Evaluation Framework
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEPS
StandardsUtility
FeasibilityProprietyAccuracy
![Page 67: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/67.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
![Page 68: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/68.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families
![Page 69: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/69.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families those who receive training (e.g., owners,
inspectors, maintenance personnel, tenants)
![Page 70: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/70.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families those who receive training (e.g., owners,
inspectors, maintenance personnel, tenants) trainers
![Page 71: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/71.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families those who receive training (e.g., owners,
inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code
![Page 72: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/72.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families those who receive training (e.g., owners,
inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code
smoke-free advocates
![Page 73: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/73.jpg)
Step 1: Engage the Stakeholders
Low-Income Housing Scenario
Who are the stakeholders?
affected families those who receive training (e.g., owners,
inspectors, maintenance personnel, tenants) trainers city officials responsible for the housing code
smoke-free advocates
Stakeholders may include advocates as well as those resistant to change . It’s important to include a mix of views so you don’t get surprised at the end.
![Page 74: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/74.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
![Page 75: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/75.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is the current indoor environment?
![Page 76: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/76.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is the current indoor environment? What changes are both desired and seen as
realistic?
![Page 77: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/77.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is the current indoor environment? What changes are both desired and seen as
realistic? What outcomes do they think are most
important?
![Page 78: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/78.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is the current indoor environment? What changes are both desired and seen as
realistic? What outcomes do they think are most
important? Which activities contribute to which outcomes?
![Page 79: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/79.jpg)
Step 2: Describe the Program
Low-Income Housing Scenario
Obtain stakeholders’ views on:
What is the current indoor environment? What changes are both desired and seen as
realistic? Which outcomes do they think are most
important? Which activities contribute to which
outcomes? (You could use a logic model here… Stay tuned for the next Webinar in this series…)
![Page 80: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/80.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
![Page 81: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/81.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code?
![Page 82: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/82.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers?
![Page 83: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/83.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design?
![Page 84: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/84.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code? Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best?
![Page 85: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/85.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best? Should there be a control group?
![Page 86: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/86.jpg)
Step 3: Focus the Evaluation DesignLow-Income Housing Scenario
What questions do the stakeholders think are the most important to ask?
Should the evaluation focus on changes to the housing code?
Focus on trigger reduction? Which triggers? What is the best design? Is a pre-post design best? Should there be a control group? Are observational methods appropriate?
![Page 87: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/87.jpg)
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play?
![Page 88: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/88.jpg)
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play? Are there stakeholders that can facilitate access to
residences and/or translate if necessary?
![Page 89: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/89.jpg)
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play? Are there stakeholders that can facilitate access to
residences and/or translate if necessary? Can the trainers help collect data before, during, or after
each training? Other opportunities for data collection?
![Page 90: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/90.jpg)
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play? Are there stakeholders that can facilitate access to
residences and/or translate if necessary? Can the trainers help collect data before, during, or after
each training? Other opportunities for data collection? What information will the stakeholders need to act? For
example, will policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage?
![Page 91: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/91.jpg)
Step 4: Gather Credible Evidence
Low-Income Housing Scenario
What role can stakeholders play? Are there stakeholders that can facilitate access to
residences and/or translate if necessary? Can the trainers help collect data before, during, or after
each training? Other opportunities for data collection? What information will the stakeholders need to act? For
example, will policy makers be more influenced by information on environmental conditions, asthma symptoms, or medical care usage?
Data in what form? Pictures? Stories? Numerical data?
![Page 92: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/92.jpg)
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look like to them?
![Page 93: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/93.jpg)
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look like to them?
Do stakeholders have alternative interpretations of the initial findings?
![Page 94: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/94.jpg)
Step 5: Justify Conclusions
Low-Income Housing Scenario
How do stakeholders view success? What does “success” look like to them?
Do stakeholders have alternative interpretations of the initial findings?
Do they perceive barriers you might have missed?
![Page 95: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/95.jpg)
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
![Page 96: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/96.jpg)
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?
![Page 97: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/97.jpg)
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?
Is information needed in a different form to prompt action? A report? An executive brief? An oral presentation?
![Page 98: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/98.jpg)
Step 6: Ensure Use and Share Lessons Learned
Low-Income Housing Scenario
Can stakeholders help disseminate evaluation findings?
Is there an owner’s newsletter? A tenant meeting? A neighborhood association?
Is information needed in a different form to prompt action? A report? An executive brief? An oral presentation?
Is there interest beyond the community involved in the intervention?
![Page 99: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/99.jpg)
What if you ignored stakeholders?
If you ignore the stakeholders, potential pitfalls include…
![Page 100: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/100.jpg)
What if you ignored stakeholders?
Failure to ask important questions.
• For example, it may be important to know if tenants have access to quit-smoking services.
If you ignore the stakeholders, potential pitfalls include…
![Page 101: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/101.jpg)
What if you ignored stakeholders?
Difficulty gaining access to or trust of program constituents.
• For example, will you gain access to data or be received into residents’ homes?
If you ignore the stakeholders, potential pitfalls include…
![Page 102: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/102.jpg)
What if you ignored stakeholders?
Failure to uncover unanticipated consequences.
• For example, if improvements are made, will rents increase?
If you ignore the stakeholders, potential pitfalls include…
![Page 103: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/103.jpg)
What if you ignored stakeholders?
Failure to communicate results in ways that are understood and/or accessible.
• For example, are there literacy or language issues? Cultural preferences?
If you ignore the stakeholders, potential pitfalls include…
![Page 104: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/104.jpg)
What if you ignored stakeholders?
If you fail to involve the stakeholders…
…your evaluation may not lead to action!
![Page 105: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/105.jpg)
105
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma.
![Page 106: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/106.jpg)
106
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma.
Involves standardizing protocols across care systems, including:
Medicaid HMOsHome nursing agenciesHealth departments
![Page 107: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/107.jpg)
107
Example #2 – Care Coordination Across Health Systems
Intervention to provide and integrate care coordination and case management for high-risk children with asthma.
Involves standardizing protocols across care systems, including:
Medicaid HMOsHome nursing agenciesHealth departments
The goal: to prove success and thus convince insurers to continue reimbursing case management services.
![Page 108: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/108.jpg)
Engage the Stakeholders
Housing Scenario Healthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
Who are the stakeholders in this scenario?
![Page 109: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/109.jpg)
Engage the Stakeholders
Housing Scenario Healthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers
![Page 110: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/110.jpg)
Engage the Stakeholders
Housing Scenario Healthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers case managers
![Page 111: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/111.jpg)
Engage the Stakeholders
Housing Scenario Healthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers case managers families
![Page 112: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/112.jpg)
Engage the Stakeholders
Housing Scenario Healthcare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
insurers case managers families health system
administrators
![Page 113: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/113.jpg)
Focus the Evaluation Design
Housing Scenario Healthcare Scenario
What changes are both desired and seen as realistic?
What outcomes do they think are most important?
Which activities contribute to which outcomes?
What outcome questions?
![Page 114: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/114.jpg)
Focus the Evaluation Design
Housing Scenario Healthcare Scenario
What changes are both desired and seen as realistic?
What outcomes do they think are most important?
Which activities contribute to which outcomes?
Is this intervention sustainable?
![Page 115: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/115.jpg)
Focus the Evaluation Design
Housing Scenario Healthcare Scenario
What changes are both desired and seen as realistic?
What outcomes do they think are most important?
Which activities contribute to which outcomes?
Is this intervention sustainable?
What is the cost of doing this kind of case management?
![Page 116: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/116.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
What type of data do they need to make a decision?
![Page 117: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/117.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Is quantitative data more highly valued than qualitative data?
![Page 118: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/118.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Is quantitative data more highly valued than qualitative data?
Do the data need to show cost savings?
![Page 119: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/119.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Be sensitive to the stakeholders’ interests and time constraints.
![Page 120: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/120.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Be sensitive to the stakeholders’ interests and time constraints.
Be aware of and consider the political environment.
![Page 121: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/121.jpg)
Example #3 – Daycare EducationIntervention to train childcare providers to identify triggers and to manage children with asthma.
![Page 122: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/122.jpg)
Example #3 – Daycare EducationIntervention to train childcare providers to identify triggers and to manage children with asthma.
Intervention reaches out to:Large daycare centersLicensed home daycare programsParticipation is encouraged through continuing education credits
![Page 123: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/123.jpg)
Example #3 – Daycare EducationIntervention to train childcare providers to identify triggers and to manage children with asthma.
Intervention reaches out to:Large daycare centersLicensed home daycare programsParticipation is encouraged through continuing education credits
Long-term goal: to expand beyond pilot providers to larger community.
![Page 124: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/124.jpg)
Engage the Stakeholders
Housing Scenario Daycare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
Who would be the stakeholders in this scenario?
![Page 125: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/125.jpg)
Engage the Stakeholders
Housing Scenario Daycare Scenario
affected families
those who receive training
trainers
city officials
smoke-free advocates
trainers daycare providers licensing boards families
![Page 126: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/126.jpg)
Focus the Evaluation Design
Housing Scenario Daycare Scenario
What changes are both desired and seen as realistic?
What outcomes do they think are most important?
Which activities contribute to which outcomes?
What outcome questions?
![Page 127: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/127.jpg)
Focus the Evaluation Design
Housing Scenario Daycare Scenario
What changes are both desired and seen as realistic?
What outcomes do they think are most important?
Which activities contribute to which outcomes?
behavior change environmental
outcomes evidence to support
decision to expand beyond pilot
![Page 128: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/128.jpg)
Focus the Evaluation Design
Housing Scenario Healthcare Scenario
Focus on changes to the housing code?
Trigger reduction? Should there be a
control group? Observational
methods?
What design mix?
![Page 129: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/129.jpg)
Focus the Evaluation Design
Housing Scenario Healthcare Scenario
Focus on changes to the housing code?
Trigger reduction? Should there be a
control group? Observational
methods?
Consider pre-post design.
Consider control group design.
![Page 130: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/130.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
What data collection methods?
![Page 131: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/131.jpg)
Gather Credible Evidence
Housing Scenario Healthcare Scenario
Data in what form? Pictures? Stories? Numerical data?
Stakeholders that can facilitate access ?
Trainers help collect data ?
Success stories may be important .
![Page 132: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/132.jpg)
Getting Started and Engaging Your Stakeholders
Tangible Examples ofStakeholder Involvement in
Applying the Evaluation Standards
![Page 133: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/133.jpg)
Evaluation Standards Apply to Every Step
Ensure use and share lessons learned
Gather credible evidence
Engage stakeholders
Describe the
program
Focus the evaluation
designJustify
conclusions
STEPS
StandardsUtility
FeasibilityProprietyAccuracy
![Page 134: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/134.jpg)
The Evaluation Standards
Standards
• Utility• Feasibility• Propriety• Accuracy
![Page 135: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/135.jpg)
The Evaluation Standards
There are actually 30 evaluation standards grouped into four categories.
A complete list of the standards is published in CDC’s Framework for Program Evaluation in Public Health. (See link under “Learning Aids”.)
![Page 136: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/136.jpg)
Why Use Standards?Standards provide a way to:
Make difficult decisions when designing and implementing an evaluation.Judge the quality of an evaluation.Determine where an evaluation can be better balanced.
![Page 137: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/137.jpg)
The “Utility” StandardWho needs the evaluation information and
what information do they need?
Identify affected stakeholders. Collect data that addresses important
questions. Ensure the data is responsive to the needs of
the stakeholders.
![Page 138: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/138.jpg)
The “Feasibility” StandardHow much money, time, and effort can we put into this?
Keep the burden of data collection as low as possible.
Consider the political interests of the stakeholders.
Ensure that the evaluation is cost effective.
![Page 139: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/139.jpg)
The “Propriety” StandardWhat steps need to be taken for
the evaluation to be ethical?
Ensure the ethical treatment of evaluation participants.
Ensure the ethical treatment of those who commissioned the evaluation.
Ensure the ethical treatment of those affected by the findings of the evaluation.
![Page 140: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/140.jpg)
The “Accuracy” StandardWhat design will lead to
accurate information?
Clearly and accurately document the program. Gather data in a way that produces reliable
information. Analyze and report the data in a systematic and
unbiased way. Assure that the source of the information is
defensible.
![Page 141: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/141.jpg)
CDC’s Asthma Control Program
CDC’s Asthma Control Program funds 35 states and territories to implement statewide asthma control programs using a public health perspective.
Their approach involves:Engaging, enhancing, and maintaining relationships with partners.Developing, improving, and conducting asthma surveillance activities.Designing and implementing interventions with partners.
![Page 142: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/142.jpg)
Example #4 - Asthma Program Monitoring System
This example involves collecting information for the purpose of evaluation across multiple sites.
![Page 143: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/143.jpg)
Example #4 - Asthma Program Monitoring System
Purpose:
Develop a strategic, systematic approach to collecting information about our program activities, progress, and accomplishments across all funded states.
![Page 144: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/144.jpg)
Employing the Framework ModelStep 1: Involve the stakeholders CDC Air Pollution and Respiratory
Health Branch. Representatives of funded State
Asthma Programs.Step 2: Describe the Program Develop logic models.
Step 3: Focus the evaluation design Identify and prioritize evaluation
questions.Step 4: Gather credible evidence Create data collection instrument.
![Page 145: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/145.jpg)
Developing the Data Collection Instrument
In this example, developing the data collection instrument was a highly collaborative endeavor aided by the program evaluation standards.
![Page 146: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/146.jpg)
The Iterative Process• CDC internal workgroup identifies core information needs and drafts survey
instrument.
• State workgroup members review instrument, pilot test, provide pilot data and comments.
• Create mock report to illustrate type of information obtained from instrument.
• CDC internal workgroup discusses comments and mock report.
• Modify instrument as necessary.
![Page 147: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/147.jpg)
Applying the StandardsHow were the evaluation standards employed in this example?
How was “engaging the stakeholders” important in applying each of the standards?
![Page 148: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/148.jpg)
The “Utility” StandardWho are the intended users of the information?
Why do they need the information?
CDC Air Pollution and Respiratory Health Branch
State partners
Examine progress. Identify promising
patterns. Share successful practices Identify areas for
improvement. Compare and contrast
program designs and performance.
![Page 149: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/149.jpg)
The Utility Standard in Action• Determined what information the CDC needed to do their job and how they will use the
information.
• Engaged states in discussions about how to make the information useful for them.
• Developed mock reports to facilitate conversations with stakeholders about how the data could be used.
![Page 150: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/150.jpg)
The “Feasibility” StandardHow much money, time, and effort can we put into this?
Remained conscious of how much time this data collection will require of the states.
Identified areas where processes could be streamlined.
Recognized that there will be less time for other programmatic activities.
![Page 151: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/151.jpg)
The Feasibility Standard in Action• Carefully selected “need-to-know” questions.• Asked states:
• What information do you already have?• How feasible is it for you to obtain?• How long does it take to locate information?• How long does it take to fill out the data collection instrument?
• Always kept the burden of data collection as low as possible.
![Page 152: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/152.jpg)
The Feasibility Standard in ActionThe content of the data collection instrument was a careful balance between feasibility and utility.
High priority data requiring higher burden was balanced by eliminating lower priority questions.
![Page 153: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/153.jpg)
The “Propriety” StandardWhat steps need to be taken for
the evaluation to be ethical?
Program monitoring systems can be used to generate performance measures.
The data collected could have significant implications.
Understand that the information collected can affect the stakeholders’ programs.
![Page 154: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/154.jpg)
The Propriety Standard in Action
• Used a highly collaborative process.
• Fostered open, honest dialogue about expectations and concerns.
• Remained highly responsive to stakeholder concerns.
• Modified instrument to address stakeholder concerns.
![Page 155: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/155.jpg)
The “Accuracy” StandardWhat design will lead to
accurate information?
Be certain that the information collected is valid and reliable.
If the information is inconsistent or irrelevant to the concept, reconsider whether to collect it.
![Page 156: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/156.jpg)
Balancing the Standards
When balancing feasibility vs. accuracyconsider the effect on propriety.
![Page 157: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/157.jpg)
The Accuracy Standard in Action
Involved states in:
• Pilot tests and reviews.
• Discussions about “guessing”.
• Analyzing the quality of the information provided.
• Revising questions or collecting information that would help to improve accuracy.
• Decisions to discard some questions.
![Page 158: Program Evaluation Webinar Series Part 2:](https://reader034.vdocuments.mx/reader034/viewer/2022052603/56815e3c550346895dcca555/html5/thumbnails/158.jpg)
What We Have Reviewed Today• The goal of evaluation and role of stakeholders.
• Who to engage as stakeholders.
• Why it is important to engage stakeholders.
• When and how to engage stakeholders.
• Stakeholder involvement across all steps of CDC framework.
• Stakeholder involvement in fulfilling the evaluation standards.
• Examples: Single and multiple sites