1 using logic models to enhance evaluation westat center to improve project performance (cipp)...
TRANSCRIPT
1
Using Logic Models to Enhance Evaluation
WESTATCenter to Improve Project Performance (CIPP)
Office of Special Education Programs
Amy A. Germuth, Ph.D.
Compass Consulting Group
Durham NC
2
Today’s Schedule
12:30-3:30 Workshop
12:30-2:00 Workshop
2:00-2:15 Break
2:15-3:30 Workshop/Wrap-up
Please feel free to ask questions at any time or take a break if you need it.
3
What we Will Cover What a logic model is
Why we develop logic models The components of a logic model Vocabulary associated with logic models
What is an output versus an outcome Why satisfaction is (almost always) an output versus an outcome
How to develop a logic model Simple logic model Complex logic model
How logic models can guide formative and summative evaluation efforts Formative versus summative evaluation Connecting evaluation to the logic model
4
Learning Goals
1. Understand what a logic model is and why you might want to develop one for your program.
2. Understand the components of a logic model.
3. How to develop a logic model for your program.
4. How to use your logic model to guide evaluation.
5
What is a Logic Model?
A depiction of a program showing what the program will do and what it is to accomplish.
A series of “if-then” relationships that suggest (domino effect): If the program is implemented as intended (and
designed) desired outcomes will be realized A systematic and visual way to depict the
relationship that exists among the resources, activities, and desired results of a project or program. Thus, a completed logic model depicts how the project will work and is sometimes referred to as the program theory.
6
Why are Logic Models used?
To describe group work, team work, community-based collaboratives and other complex organizational processes.
To guide clarity on sequencing. To guide planning, implementation, evaluation and
communication. To help depict the theory of change associated
with a program.
7
Constructing Logic Models Examining program descriptions, missions, visions, plans,
or extracting them from a narrative (useful if program in place)
Forward mapping—starting with activities and asking “so what” to generate expected outputs and outcomes (useful if program in place)
Reverse mapping—starting with outcomes and asking “how best to” identify the activity that is most likely to generate them (useful if developing a program)
Importance of incorporating Evidence Based Practices
8
Simple Logic Model
Inputs → Outputs → OutcomesExample:
Inputs
→ Outputs
→ Outcomes
Books
→ New vocabulary words
→ Increased reading comprehension
9
More Complex Logic Models
10
11
Another Logic Model
12
Our Logic Model Framework - 1
Objectives/GoalsInputs→→
Activities→→Outputs→→
Outcomes→→Impacts
AssumptionsExternal Factors / Context
13
The Visual…
Need
Goal(s) Inputs Activities Outputs Outcomes Impacts
Evidence-based practices are used to:1)establish goals as effective and efficient ways to meet needs; 2)establish activities as effective and efficient ways to operationalize goals;3)establish the direct outcomes likely to result from activities; and/or4)establish that direct outcomes eventually will lead to impacts in some proportion of the target population.
14
Our Logic Model Framework-2
Inputs = what the program needs to invest in order to provide activities
Activities + Outputs = what the program does Outcomes + Impacts = who or what will change and how
Needs reflected in Legislation = The legislation that provides resources for a program is grounded in identified or observed needs
Assumptions + External Factors / Context = Factors that influence getting to outcomes
15
Logic Model Definitions
1. NEEDS – Inspire legislation.2. GOALS / OBJECTIVES– The primary changes that
are necessary to serve the need.3. INPUTS - Resources, contributions, investments
(financial and other) that go into the program.4. ACTIVITIES - What you do as part of the program.5. OUTPUTS - Activities, services, events and products
that reach people who participate or who are targeted. These are often discrete.
6. OUTCOMES - Results or measures of change for individuals, groups, communities, organizations, or systems. Can be short-term and long-term.
7. IMPACTS – Overall result of multiple outcomes acting together.
16
Activity 1- Output or Outcome?
Directions:
Worksheet 1 contains multiple outputs and outcomes selected from a diverse range of policy, personnel, and project / program evaluations.
1. On your own, identify each as an Output or Outcome.
2. Next compare your results with those at your table, and be prepared to present your findings to the broader group.
17
Developing a Logic Model
Let’s start by developing a simple logic model (Inputs →Outputs →Outcomes), assuming the situation we are addressing is:
Developing and operating a science lab.
1. What are possible inputs?
2. What are possible outputs?
3. What are possible outcomes?
18
Inputs: Developing and Operating a Science Lab
Funds to buy lab equipment Lab procedures curriculum Guide to lab procedures and safety rules Safety equipment
19
Outputs: Developing and Operating a Science Lab
Number of labs completed Number of different procedures completed Number of times each procedure done Number of safety rules learned Number of safety measures taken
20
Outcomes: Developing and Operating a Science Lab
Improved lab skills Increased understanding of safety needs Increased number of safety precautions
taken
21
Activity 2 – Another Simple Logic Model
Situation: Family vacation. Let’s distinguish between inputs and activities on this one.
Inputs – what we invest
Activities – what we do
Outputs – who we reach and how
Outcomes – what results
What would a potential logic model look like?
22
Adding More Detail: Short- and Long-term Outcomes
The USED Math-Science Partnership Program was developed to increase students’ achievement in math and science. The assumption behind this program is that by providing teachers professional development in math and science content and pedagogy, teachers would be better prepared to help their students attain greater achievements in these areas.
23
MSP Program Logic Model
Inputs: $$$, trainers, college professors, materials, technology
Activities: Professional development in content and pedagogy, shared planning time, departmental meetings, book discussions
Outputs: Attendance at PD, feedback on PD, # persons trained, average number of hours of training received
Outcomes: ??????
Impact: Increased student achievement
24
MSP Program Logic Model: Short- and Long-term Outcomes
What are some short-term outcomes associated with teachers?
What about with students?
What are some long-term outcomes associated with teachers?
What about with students?
25
Additional Considerations Assumptions - the beliefs we have about the program, the
people involved, the context in which the program is operating , and the way we think the program will work.
Objectives –our goals / objectives for our program – these are based on our assumptions and guide our inputs and activities.
External Factors / Context - the context or environment in which the program exists. This includes external factors that interact with and influence the program action (e.g., need, interest, economic factors, etc.). They may guide your assumptions and choices of inputs and activities, but also may be things that affect outcomes and impacts.
26
Activity 3:Putting it all Together
In groups, please review the scenario/ program assigned to you and identify the following:
AssumptionsExternal Factors / ContextObjectives / GoalsInputsActivitiesOutputsOutcomes (short-term and long-term)Impacts
The more detail / specific, the more useful your logic model will be, particularly for guiding your evaluation!
27
Using Logic Models to Guide Evaluation
Logic Models are extremely useful in guiding both formative and summative evaluation efforts.
- Formative evaluation is undertaken to assess whether the program is proceeding as planned, the fidelity of implementation to program design, and the degree to which changes need to be made.
- Summative evaluation is conducted to asses whether planned outcomes have been achieved and what impacts (intended and unattended) have occurred.
28
Logic Models and Formative Evaluation
Logic Models identify the key inputs, activities, and outputs associated with the program. This information in turn identifies data sources and key data that can be used to answer such questions as:
1. Are key inputs being utilized as expected? As designed?2. How are activities being received by participants?3. Are key outputs occurring at the desired level?4. What do participants and stakeholders like about the current
program?5. What changes do they suggest should be made?
What other questions might we ask?What other data sources can we consider?
What data should we collect?
29
Logic Models and Summative Evaluation
Logic Models also identify key outcomes or measurable changes and impacts associated with the program. This information in turn identifies data sources and key data that can be used to answer such questions as:
1. What outcomes (expected and unexpected) have occurred?2. What expected outcomes have not occurred?3. Where is change the greatest?4. What impacts (expected and unexpected) have occurred?5. What expected impacts have not occurred?6. To what degree have outcomes occurred?7. What is the unique contribution of the program to the observed
change?8. What is the cost / benefit of these outcomes?
What other questions might we ask?What other data sources can we consider?
What data should we collect?
30
Updating our Logic Model
These questions, data sources, and data to be collected can easily be added to our logic model.
31
Activity 4 – Adding Evaluation Components
Using the same scenario/program from Activity 3, work in your group to identify potential formative and summative evaluation questions, and data / data sources needed to answer them.
Be prepared to share these with the broader group.
32
References (Web) Kellogg Foundation Logic Model Development Guide
http://www.wkkf.org
CDC Guide on Logic Model developmenthttp://www.ojp.usdoj.gov/BJA/evaluation/guide/documents/cdc-logic-model-development.pdf
University of Wisconsin Extension Program Logic Model Development Guidehttp://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html
InnoNetTool for Planning and Evaluationhttp://www.innonet.org/index.php?section_id=64&content_id=185
33
References (Texts)
Logic Modeling Methods in Program Evaluation: Research Methods for the Social Sciences - J Frechtling
Theory Driven Evaluations – H. Chen
34
The End
Questions?
