Intervention Logic
A Presentation to
the Pathfinder Project
Karen Baehler
Victoria University of Wellington463 5711
Victoria University of Wellington 2
The problem
Citizens want to know if government is making a difference. Are we getting results in return for our taxes?
Ministers want better advice: “The most common problems which can be discerned in recent experience are:
• overstatement of what will be achieved;• under-explanation of how policy actions will achieve the
claimed outcomes.”
From Improving Policy Advice (1993) by G. R. Hawke, p. 27
Victoria University of Wellington 3
The solution
• Identify goals (outcomes)• Chart a course to those goals• Measure current progress• Stop things that don’t work• Alter things that sort of work• Keep improving things that do work• Discover/invent new avenues to success
Victoria University of Wellington 4
Outline
• What is intervention logic?
• What are its prerequisites?
• What are its uses?
• What are its blind spots?
• How do we minimise the blind spots?
• How do we know if an IL is working?
Victoria University of Wellington 5
What is intervention logic?
• A testable theory of causation– Linked “if-then” statements– Action/reaction pairs
• A chain of conditions to be achieved• Ultimate/end outcome
= policy goal
• Intermediate outcomes and immediate impacts– Lead to the end outcome– But are not ends themselves
• A basis for confirming performance
Victoria University of Wellington 6
Start with a backbone
• The vertical dimension of IL• Outcomes logic, not processes or activities
– Outcome grammar– Connectedness
• The necessary but not sufficient rule
• Advisor’s mindset– Optimistic– Skeptical
Victoria University of Wellington 7
The “If you build it, they will come” backbone
• Political benefits of simplicity
• Analytical pitfalls
• What’s wrong with this backbone?
• Can the matrix fill in the gaps?
Reducetraffic
congestion
Buildbypass
Peopledriveon it
Output
Immediateimpact
End outcome
Victoria University of Wellington 8
The more complex backbone
• Add intermediate outcomes
• More assumptions about indirect causation
Victoria University of Wellington 9Output: School vouchers
Ultimate outcome: Increased educational achievement/smarter kids
Parents aware of & understand program
Parents possess “appropriate” information about schools
Parents choose “best” schools for their children
“Good” schoolsgain pupils
“Bad” schoolslose pupils
“Good” schoolsget even better
Some “bad”schools fold
Some “bad”schools lifttheir game
New “good”schools come
on line
Victoria University of Wellington 10
The black box at the top of the backbone
Social policy: Note the large leaps in logic that often occur at the top
Client responds wellto services
Client’s behaviour changes(How? Why?)
Ultimate outcomerealised
Victoria University of Wellington 11
Plot twists
• When is an intermediate outcome also an end outcome?
• Can one agency’s / department’s intermediate outcome be another agency’s end outcome?
Victoria University of Wellington 12
The backbone as a manage-ment tool: Links below outputs
Inp u t 1 Inp u t 2 Inp u t 3
A c tiv ity 1 A c tiv ity 2 A c tiv ity 3
O utp u t 1 O utp u t 2 O utp u t 3
Im m ed ia te im p a c ts
Source: R Waite
Victoria University of Wellington 13
The backbone as a risk ID tool: Collateral outcomes
A c tu a l risks (p = ? )
U lt im a te o u tco m e(u n in te nd e d)
In te rm ed ia te ou tco m e(u n in te nd e d)
Im m e d ia te im p a ct(u n in te nd e d)
O u tp u t
Im m e d ia te im p a ct(in te nd e d)
In te rm ed ia te ou tco m e(in te nd e d)
U lt im a te o u tco m e(in te nd e d)
Victoria University of Wellington 14
What are IL’s prerequisites?
• Agreed outcomes for the top row– Sources
• Statement of intent• Agency/departmental mission
– The importance of first principles review– The role of problem definition
• Outcomes (goals) are the flipside of problems• The “problem logic” model and the black box
• Intervention option(s) for the bottom row• Common sense
Victoria University of Wellington 15
Group Exercise 1
• Work in pairs
• Choose a familiar policy/output from your work or from the news
• Produce a backbone linking the output to intermediate and ultimate outcomes
• Identify strong and weak links
Move to a matrix (Funnell 1997)
1
Outcomes
Hierarchy
2 Success Criteria
3
Factors
Within
Control
4
Factors
Outside
Control
5
Activities & Resources
6
Perfor-
mance
Ultimate outcome
Inter-mediate
outcomes
Immediate
outcomes
Output
Victoria University of Wellington 17
What are its uses?
Conventional uses• Testing existing
policy hypotheses• Testing performance• Improving impacts
through design & management of risk
• Making better use of existing data
Unconventional uses• Comparing policy
options• Identifying generic
intervention templates for a department
• Discovering/inventing new interventions
Victoria University of Wellington 18
Testing existing policy hypotheses
• If X, then Y• Y = f (X)
– Does the raw logic hold? (ex ante)– Does the available evidence support the logic? (ex
ante and ex post)– What additional evidence is needed to test the
logic?
• IL breaks an impact evaluation into chunks.
Victoria University of Wellington 19
Testing/confirming performance
Column 6 in the IL matrix allows us to
• disaggregate performance into chunks
• distinguish chunks that are working well from those working less well– based on achievements compared agains
success criteria/targets
Victoria University of Wellington 20
Improving impacts
• Identify conceptual and operational gaps in existing policy
• Target issues for review (weak links)
• Monitor– Internal and external risks– Counter-intuitive causes and effects
• Revise design
Victoria University of Wellington 21
Making better use of data
• Evidence need not relate to ultimate/end outcomes to be useful
• Findings to date (from NZ or international) may shed light on immediate and intermediate links in the chain
• Role of research in the “problem logic”• Examples
– School choice research and its place in the IL
Victoria University of Wellington 22
Comparing policy options (via the conventional matrix)
Criteria Option A Option B Option C
Life years saved
SA1 SB1 SC1
New incidents prevented
SA2 SB2 SC2
Equity SA3 SB3 SC3
Multiple outcomes = unlinked “criteria”
Victoria University of Wellington 23
Using IL to compare options
• Compare #’s of links– More links = more
chances to stuff it up/more resources required?
– More links = less uncertainty, more robust theory?
– Fewer links = political plus?
• Compare #’s and magnitude of weak links
• Compare #’s and magnitude of possible unintended outcomes
Step 1: Prepare a backbone for each option
Victoria University of Wellington 24
Using IL to compare options
Step 2: Prepare an IL matrix for each option
Victoria University of Wellington 25
Using IL to compare options
See next slide• Compare risks across
A, B, C• Compare resources
needed A, B, C• Compare performance
contract possibilities• Compare evaluability
IL sets up more accurate cost-effectiveness analysis– Remove
unnecessary steps before costing
– Identify possible sources of extra costs
Step 3: Compare across IL matrices
Victoria University of Wellington 26
A cross-cutting matrix
Option A Option B …
Outcomes (1)* Weak logic Strong logic
Success criteria (2) Measurable Unmeasurable
Internal control (3) High Low
External risks (4) Low High
Costs (5) $ per X $ per X
Institutional capacity (5)
High Low
Management (2-6) ? ?
*Numbers in parentheses refer to columns in the IL matrix
Victoria University of Wellington 27
Identifying IL templates
• The case management model – Generic steps (slide 28)– Early intervention example
• The information campaign model– Generic steps (slide 29)
• The deterrence model– Mandatory sentencing laws example (slide 30)
• The pollution permits model– GHGs example (slide 31)
Victoria University of Wellington 28Output: Target group enters programme
Individual’s needs & prospects assessed accurately
Realistic objectives set for (and with) the individual
Individualised programme put in place to meet objectives
Short-term objectives for individual progressively achieved
Life circumstances/chances of individual are improved; long-term objectives are achieved
Reduced long-term costs and/or increased long-term benefitsto the community
Victoria University of Wellington 29Output: Educational literature produced
Literature passes pretest for readability, etc.
Appropriate audience receives literature
Audience reads literature
Readers learn the facts
Readers change their opinions
Readers change their behaviour
Readers influence others to change opinions/behaviour
Behaviour change leads to improved outcomes
Victoria University of Wellington 30Output: Mandatory prison sentences for crime X
Judges understand and apply them
More Xoffenders
jailed longerPast & potential offenders
aware of sentencing
Past & potential offenders includenew X sentencing risk in their
personal decision making
X offenders workharder to avoidapprehension Potential offenders
avoid crime X Fewer Xoffenders on
the street
Rates of crime X
Victoria University of Wellington 31Output: Tradable emissions permits created for GHGs
Permits auctioned to bidders (or other allocation made)
Plants calculate costs & benefits of investing in cleaner technologyv buying add’l permits v paying
fines for excessive GHGs
Regulatorssanction
Some plants invest in
clean R & D andtechnology
Innovations in cleantechnology diffuse
Govt investsauction revenuein clean R&D
Some plants buy add’lpermits
Some plantsemit GHGs
above permit
“Clean”firmsprofit
GHGs& costs
Victoria University of Wellington 32
Discovering/inventing new interventions
• The brainstorming approach– Pick generic policy instruments– Apply to the problem at hand, using quick,
back-of-the-envelope backbones
• The engineering approach– Start with the “problem logic”– Find the entry points in the model– Fashion interventions for the entry points
Victoria University of Wellington 33
Group exercise 2
• Same pairs• Choose an end/ultimate outcome and make it
the top “vertebra” of a backbone• Work down to identify intermediate outcomes
that might lead to that end outcome (based on your knowledge of how that outcome is “naturally” produced)
• What interventions suggest themselves as you move down?
Victoria University of Wellington 34
What are IL’s blind spots?
• Might the chain of outcomes look different for different population groups of interest?
• Might risk factors differ across groups?• Might different groups need different
activities and resources to reach each intermediate outcome?
Equity
Victoria University of Wellington 35
What are IL’s blind spots?
• Hidden portions of the backbone– Inputs and activities (below)– Collateral outcomes (beside)– Program/theory assumptions (beside)
• Getting trapped in a paradigm– Tikanga v cognitive-behavioural paradigms for
explaining crime
• Focusing on the lower levels of the hierarchy, where managers have more control
Victoria University of Wellington 36
How do we minimise the blind spots?
• Research that contributes to robust problem logics– The poverty example– The drug harms example
• Evaluation that contributes to robust intervention logics– The welfare to work example
• Outcomes that reflect actual results in the community/real consequences
Victoria University of Wellington 37
How do we know when an IL is working (or not)?
• Does it help us distinguish between apparently more and less promising interventions?
• If it just rationalises everything, not robust• Does it systematically favour some types of interventions
over others? Why? Is this warranted (cross check)?
• Does it help us make better use of existing evidence?
• Does it help us generate a research agenda?
Victoria University of Wellington 38
Ex ante criteria for a good IL• Proper “grammar” in the backbone
– Outcomes, not processes or activities
• Each intermediate outcome represents a necessary but not sufficient cause of the next outcome
• Success criteria are measurable and lend themselves to targets
• Activities and resources cover all of the key factors within the programme’s control
• Activities and resources supply what is needed to get from one outcome to the next
Victoria University of Wellington 39
Ex post criteria for a good IL
• Are outcomes being produced more cost effectively than prior to use of IL?
• Are intended and unintended outcomes predicted more accurately?
• Are there fewer unintended outcomes?• Are there fewer unexpected
outcomes/surprises?• Is the department accumulating better
information about its own performance?
Victoria University of Wellington 40
How can IL evolve?
• Through multiple applications, learning by doing
• Through cross-breeding with other soft & hard systems approaches
• Through peer review
Challenges to be met– Accounting for new
sets of surprises– Facilitating cross-
departmental thinking on partnerships for particular outcomes
– Facilitating equity analysis
– Other