intro to logic models for community organization program ... · logic modeling/evaluation planning...
TRANSCRIPT
Intro to Logic Models for Community Organization Program Evaluation
February 5, 2018
Smith Child Health Research, Outreach & Advocacy (SCHROA)
Evaluation Core
Goal: improve evaluation reach & quality to inform programs and initiatives in pediatric health
Offers:
• pre-proposal consultation on evaluation design & budget
• post-funding evaluation planning, consulting, staffing and implementation
• TA and training opportunities
Alliance for Research in Chicagoland Communities (ARCC)
Mission: Catalyze & support meaningful community & academic engagement across research spectrum to improve health & health equity
Based in Center for Community Health. Serving
and Institute of Public Health & Medicine
Websites: www.ARCConline.net http://www.feinberg.northwestern.edu/sites/cch/
Resources:
Partnership brokering/development Education (workshops, team training, online)
Consultation, proposal review/support Funding assistance
Introductions
• Name
•Organization
• Experience with logic models
Today’s goals
• Introduce logic models, their components and uses
• Share examples of logic model formats
• Gain practice in applying logic models in program evaluation example
• Get started on creating a logic model for your program
Program Evaluation –Rapid Recap
Program evaluation
•Systematic collection of information about a program to generate credible information used to make decisions about the program
System: A set of detailedmethods, procedures and routines created to carry out a specific activity, perform a duty, or solve a problem
Program evaluation
•Tip: try to think about all the things your program/organization does in the beginning to include in your evaluation planning.
•Easier now than adding things in later on when you might not be able to have baseline measures.
What goes into program evaluation?
Stakeholder involvement
Planning
ImplementationData
analysis
Dissemination & application of findings
What goes into program evaluation?
Stakeholder involvement
Planning
ImplementationData
analysis
Dissemination & application of findings
Questions?
Evaluation planning: creating an overall vision for the evaluation
• Key program and evaluation planning
tool: Logic Model– Articulate/organize logic behind program
– Identify gaps in thinking
– Provide a visual guide
– Involve stakeholders
– Map relationships among elements
Opening of the second movement of Beethoven's String Quartet in A-Minor Op. 132
https://en.wikipedia.org/wiki/String_Quartet_No._15_(Beethoven)
Logic modeling/evaluation planning
• A picture of how your organization does its work
• A way to make explicit assumptions/theories underlying a program
• A systematic presentation of relationshipsamong resources you have to operate a program, the planned activities and the changes/results you would like to achieve
• A planning tool
• A management tool
• An evaluation tool
Logic modeling/evaluation planning
•Project specific
•Ranges from simple to complex
•Built collaboratively with stakeholders
Logic modeling/evaluation planning
• Logic models can be used at any program stage.–Design: specify & apply program theory
–Planning: identify resources needed, timing
–Implementation monitoring: fidelity check – are things being done according to the model?
–Evaluation: what outputs, outcomes, and impacts were achieved?
Questions?
Basic logic model
Inputs Activities Outputs Outcomes Impact
Planned work Intended/expected results
Logic Model Development Guide. Kellogg Foundation
Narrative
Logic model elements: Impact
• The greater goal you are striving for
• Be sure you can Identify/clarify the program’s ultimate goal or mission – that drives everything else.
–Decrease in child BMI
–Reduced unemployment
–Reduced homelessness
–Improve quality of life
https://www.bing.com/images/search?view=detailV2&ccid=w0wKt3Qh&id=387267BC797F65B1CA2847FF21F30E06EED38EB1&thid=OIP.w0wKt3QhLecbM0JKxIKpKAHaLJ&mediaurl=http%3a%2f%2fwww.wpclipart.com%2feducation%2fawards%2fribbons%2fribbon_achievement.png&exph=343&expw=228&q=Achievement+Ribbon&simid=608047872278398121&selectedIndex=0&ajaxhist=0
Logic model elements: Outcomes
•Results
–Range from immediate to very long term
–Must be narrowed down to fit with your organization’s mission and capacity
Logic model elements: OutcomesLong-term
• Build on changes in short term outcomes attitudes, behavior, knowledge, and skills.
– Consumption of 2 additional servings of fruits and vegetables
– 2 point decrease in body mass index
Medium - Short term
• Attitudes, behavior, knowledge, and skills that result from program activities.
– 80% of participants report deciding to eat healthier
– 25% Increase in knowledge on a quiz about healthy eating after 4 sessions
Logic model elements: Outputs
•Direct product of/completed program activities
•Often measured in numbers.
•Think of as the “dose” received by participants.
–# people attending class
–# items distributed
–# sessions held…
Logic model elements: Activities
•Program activities (the things your program does with resources)
•The driving force behind your intervention/program
–classes
–events
–counseling sessions
–referrals....
Logic model elements: Inputs
•Resources put into program
–staff time
–materials
–volunteers…
Program Development Logic Model
Activities Outputs Outcomes
• Intro to Program Evaluation Workshop
• Networking
• # of workshops
• # of attendees
• # of networkers attended
• Participant satisfaction
• ↑ participant Evaluation
Knowledge & confidence
• Participants apply learning in their programs
What you do How much of it you do What happens
Evaluation Logic ModelActivities Outputs Outcomes
• Intro to Program Evaluation Workshop
• Networking
• # of workshops
• # of attendees
• # of networkers attended
• 100% of participant report satisfaction with training
• 75% of participants report increased evaluation knowledge
• 50% of participants report feeling prepared to implement learning
• 50% of participants apply learning in their programs over the next year
What you do How much of it you do
What happens
Closer look at components
Activities Outputs Outcomes Impact
Bike helmet distribution
1000 helmetsdistributed
50% increase in access to a helmet by youth and parents in target neighborhoods
25% reduction in youth head trauma injuries related to wheeled recreation activities.
Bike helmet ambassadors visit schools to talk with students and advocate for helmet use
Bike ambassadorsvisit 50 schools
Youth attitude towards helmets increases by 50%
80% of youth use helmet every ride
Closer look at components
Activities Outputs Outcomes Impact
Train staff on risk factors and signs of child abuse in children brought into the ED.
100% of ED staff trained on signs of and riskfactors associated with child abuse
85% of staff report confidence in being able to identify risk factors and follow appropriate procedures.
75% reduction in readmittance of children with signs and risks for child abuse.
Reduced child injury and mortality related to child abuse among hospital patients.
Closer look at componentsActivities Outputs Outcomes Impact
• Identify contractor
• Createdesigns
• Build gateway entrances
• Install signage
• 4 trails improved through the addition of gateway entrances and wayfinding signage
• Users of the trails report using the trails and average of 4 additional days per month from baseline.
• 75% of users have positive feelings about new signage
• Trail counters record increase of users by 20% from baseline to May 2018.
• Trail counters record increase of users by 20% from May 2018 to October 2018.
• Average yearly use of FPCC trails of 100,000people
• Create and field media campaign
• 20% of users report starting to use, or increasing usage of the trails because of media campaign
Questions?
How to build a logic model • Collaborate! Bring in stakeholders – organizational
leadership, staff, potential program participants
• Start from the rightmost column, moving to the left
32
Start
on the
right
How to build a logic model • As you complete the columns, continually ask
– “What do we need in order to make this happen?”
– “What else needs to happen first?”
• The answers to those questions go into the column to the left of the one you are completing.
33
What do we need to
make this happen?
Answer
goes here
ExampleA parents’ group at a church has noticed that many of the church congregation have been having health issues related to chronic diseases like heart disease, hypertension, obesity, and diabetes. They are concerned and want to try to address this issue as best they can.
ExampleInputs Activities Outputs Outcomes Impact
The health of our
congregation will
improve.
Example• Improving health…many options
– reducing chronic diseases
– increasing vaccination
– improving residents’ diets
– Increasing physical activity
• Need to narrow by making sure that you are focusing on:
– things that most closely relate to your organization’s goals and mission
– thinking about what things you are really going to be capable of influencing (scope, scale, resources).
• Can we buy everyone a gym membership?
• Can we start a walking group?
ExampleInputs Activities Outputs Outcomes Impact
The health of our
congregation will
improve through
improved
nutrition habits.
Example• Many families are eating most meals at restaurants
• Restaurant food can be high in calories, fat, sugar and salt.
• What if families cooked more at home?
Note:It can be very helpful to use available data about your
community to shape ultimate impact goals and mid- and long-term outcomes.
For example, your organization may have information from a needs assessment or a community planning process that could be used to inform your thinking.
Example• Next we ask
– what needs to happen for families to cook healthier meals?
– what do the families need?
• Again, there are many answers to these questions
– families need access to healthy foods
– they need to feel they have the time to cook for their family
– they need to want to cook for their family
– they need skills to cook healthy foods, and so on
ExampleInputs Activities Outputs Outcomes Impact
100% of families in
our congregation
report having
greater access to
fresh produce close
to home
Families in our
congregation report
purchasing %25
more fresh produce
100% of families in
our congregation
report cooking at
least 1 healthy
meal each week
75% of families in
our congregation
report eating more
food cooked at
home with less
sale, fat, and fewer
calories
The health of our
congregation will
improve through
improved nutrition
habits.
Example• Next we ask
– What do we need to do in order to have people in our community have greater access to, and purchase, fresh fruits and vegetables?
– What else needs to happen first?
– How can we help make this happen?
• In answering these questions we think about the actual activities we will do and the outputs that will occur.
• You can complete the activities and outputs columns at the same time.
ExampleInputs Activities Outputs Outcomes Impact
Meet with corner
store owners and
ask them to offer
fresh produce
Work with store
owners to make
changes
Share information
about community
benefit
Share information
about tax credits
Provide incentives
to store that
participate
2 corner stores
engaged
Participating corner
stores stock at least
2 fresh fruits and 2
fresh vegetables
100% of families in
our congregation
report having
greater access to
fresh produce close
to home
Families in our
congregation report
purchasing %25
more fresh produce
100% of families in
our congregation
report cooking at
least 1 healthy
meal each week
75% of families in
our congregation
report eating more
food cooked at
home with less
sale, fat, and fewer
calories
The health of our
congregation will
improve through
improved nutrition
habits.
Example• Next we ask
– What resources do we need to carry out these activities?
ExampleInputs Activities Outputs Outcomes Impact
4 volunteers
Technical assistance
from someone who
has done this work
before
Incentives for
corner store owners
Materials – surveys,
handouts, posters
etc.
Meet with corner
store owners and
ask them to offer
fresh produce
Work with store
owners to make
changes
Share information
about community
benefit
Share information
about tax credits
Provide incentives
to store that
participate
2 corner stores
engaged
Participating corner
stores stock at least
2 fresh fruits and 2
fresh vegetables
100% of families in
our congregation
report having
greater access to
fresh produce close
to home
Families in our
congregation report
purchasing %25
more fresh produce
100% of families in
our congregation
report cooking at
least 1 healthy
meal each week
75% of families in
our congregation
report eating more
food cooked at
home with less
sale, fat, and fewer
calories
The health of our
congregation will
improve through
improved nutrition
habits.
Logic model review• Go back and review completed logic model. Ask:
– Has information been completely filled? No missing steps or piece?
– Will all of the listed inputs allow the program to carry out the activities listed?
– Will program activities lead to the listed outputs?
– Will activities and outputs lead to outcomes?
– Will the program outcomes eventually lead to the desired impact?
• If the answer is “no” to any of these questions, go back and edit your logic model as needed.
46
Logic model elements: Narrative
•Description of your logic model
–Ensures everyone will
interpret the same and
come away with the
same understanding.
–Summary ~2 pages
https://i.pinimg.com/236x/e3/9e/38/e39e38f4f5256b0b0afd73bcbcc3f7e8.jpg
Things to watch out for
•Check assumptions–Resources available to deliver outputs
–Realistic change from outputs
•Thinking holistically about what is “the program”
•Not taking a step back and looking at everything in the larger context–Connects with the impact you desire?
Questions?
Resources
Recommended reading• McLaughlin, J.A., and Jordan G.B. (1999). Logic models: a tool for telling
your program’s performance story. Evaluation and Program Planning, 22, 65-17.
• U.S. Department of Health and Human Services. Centers for Disease Control and Prevention. Office of the Director, Office of Strategy and Innovation (2005). Introduction to program evaluation for public health programs: A self-study guide. “Describe the program.” Atlanta, GA: Centers for Disease Control and Prevention, 19-36. Available from http://www.cdc.gov/eval/evalguide.pdf
• W.K. Kellogg Foundation. (2001). Using Logic Models to bring Together Planning, Evaluation, & Action. W.K. Kellogg In: Logic Model Development Guide. Battle Creek, MI. Available from http://www.wkkf.org/knowledge-center/resources/2010/Logic-Model-Development-Guide.aspx
Resources
• CLOCC Evaluation Guidebook– Introduction to Program
Evaluation
– The Evaluation Planning Process
– Developing Evaluation Questions
– Evaluation Design Basics
– Data Collection
– Data Analysis Basics for Program Evaluation
– Disseminating Program Evaluation Findings
Contact us! http://www.clocc.net/
Monthly Resources & Opportunities Listings
Online Resource Directory: www.ARCCresources.net
www.ARCConline.net
54
Alliance for Research in Chicagoland
Communities Resources
Evaluation, Data Integration, and Technical Assistance (EDIT) Program
Fostering a learning community to improve the health and wellbeing of sexual and gender minority populations in
Chicago and beyond.
Director: Dr. Gregory Phillips IIAssociate Director: Dr. George J [email protected]://isgmh.northwestern.edu/about/edit/
Northwestern University Evaluation and Research Alliance
NU-ERA is a group of evaluators building a community to efficiently facilitate information and resources sharing across Northwestern and to increase visibility in Chicago.
Vision
Kickoff event was held on 6/28/17 while hosting American Evaluation Association President Kathy Newcomer
Formation
Email [email protected] to join the listserv and learn about future events!Learn More
Current Collaborators
Maryann Mason, PhD312-227-7026
Sarah Welch, MPH312-227-7045
Jen Brown, MPH312-503-2942
Your Turn!
Discussion