exploring innovations & solutions x real-time policy...
TRANSCRIPT
exploring innovations & solutions x real-time policy-making
Arianna Legovini
Development Impact Evaluation (DIME)
World Bank
1 Legovini
What does dime do?
• Evaluate programs at scale to answer policy-
relevant questions
• Build capacity to strengthen country
institutions for evidence-based policymaking
• Build theory in strategic development areas
based on high-quality evidence
Improve Bank’s operations and country policies
2
How does DIME work?
Assess
demand
Identify priority research questions
Define best identification & data generation
strategy
Support implementation
Collect high-quality data
Analyze
results
Disseminate results to direct
client
Disseminate to global
communities of practice
Inform policy design
Guide policy implementation,
make mid-course
corrections
Provide policy feedback to
inform adoption and scale-up
Syst
em
atic
use
of evid
ence
Workshops & clinics with
operational staff and government
clients
Joint research/gov teams, field coordination, research team
visits
Workshops, seminars, papers, briefs, community
of practice
Capacity
build
ing
3
Discuss
1. Elements of success in policy
design
2. Using impact evaluation (IE) to drive
decisions and results
5 Legovini
What is the Impact of Public
Policy & Investments?
IT DEPENDS
Do people know what services are available?
Do they value them? Use them?
What is their quality?
…
6
Focus on the problem
• what policy do we need ?
• the returns are not fixed
• how to make policy work ?
1. get the delivery right
2. get people to do it
7 Legovini
How to
• Behavioral biases
• limited attention, self-control, cognitive capacity, understanding, asymmetric valuation of gains and losses
• Delivery mechanisms
• targeting rules, centralized/decentralized modality of public service delivery, collective vs. individual or peer-to-peer, private sector vs. public sector delivery, paid vs. voluntary, use of media and technology
• Accountability mechanisms
• top-down accountability (audits, inspections, supervision, performance assessment and feedback, laws and regulations)
• demand-side accountability (information, report cards, user participation and monitoring).
• Incentives
• supply or demand conditional monetary and non-monetary incentives
• Constraints to capital formation and productivity
• credit, financial, cash-flow, risk-management,
• financial literacy and life skills,
• information, inputs,
• institutional, legal, tax burden, corruption, property, public goods,
• managerial, skills, technology, and transaction costs.
Legovini 9
Testing WHAT and HOW
to
We evaluate the effectiveness of packages of interventions (“what”)
and experiment with mechanisms (“why/how”) to make them work
better:
Legovini 10
Reminders increase HIV
adherence by 35%
Legovini 12
How effective are the billions of dollars going to HIV
treatment when 40% of patients adhere to treatment?
With (costless) SMS reminders 53% of patients adhere to
treatment and increase effectiveness by 35%!
Pre-commitment
increases investment
Legovini
60
73.2
0
10
20
30
40
50
60
70
80
Control Precommitment savings
Precommitment and use of agricultural inputs
13
Automatization
increases TAX compliance
without
Automatization
Less compliance
With
Automatization
More compliance
14 Legovini
Default
doubles take up
49%
86%
40%
45%
50%
55%
60%
65%
70%
75%
80%
85%
90%
Before After
Control
Treatment
Legovini 15 [Madrian and Shea, 2001]
Public recognition
increases performance
6.96
7.807 8.177
14.502
4
6
8
10
12
14
16
Sales
Legovini 16 [Ashraf et al., 2011]
What is the POINT?
USE IE EVIDENCE TO INFORM POLICY DESIGN
USE IMPACT EVALUATION TO TEST IF IT WORKS IN SPECIFIC SECTOR/ COUNTRY CONTEXT
17 Legovini
Presentation
1. Elements of success in policy design
2. Using impact evaluation (IE) to
drive decisions and results
18 Legovini
Improve results by learning
19
Prescription-based policy
Evidence-based policy
We know We
learn
Legovini
FOCUS ON the Causal Links
Inputs
• Financial and human resources
Activities
• Actions to convert inputs into outputs
Outputs
• Tangible products
Intermediate outcomes
• Effective access
• Use
• Compliance
• Service quality
Final Outcomes
• Growth
• Lives saved
• Productivity
20
WHAT (SUPPLY SIDE) HOW TO (DEMAND RESPONSE) & RESULTS
Legovini
monitoring & Reporting:
trends and correlations
• DO WE NEED MONITORING? YES!
• Monitoring tracks indicators over time
• But only among participants
• It is descriptive before-after analysis
• It tells us whether things are moving in the
right direction
• It does not tell us why things happen 22 Legovini
Invest in the cause to
get more results
• It is easy to confuse correlation with causation
• By investing in the real causes we get more results
Small business growth
SME training
“Selection”
Entrepreneurial SME owners get more credit and
training
24 Legovini
Impact Evaluation:
causality
• Tracks mean outcomes over time in
• the treatment group relative to
• the control group
• Compares
• what DID happen with
• what WOULD HAVE happened (counterfactual)
• Identifies causal output-outcome link
• separately from the effect of other time-varying
factors
25 Legovini
Counterfactual criteria
Treated & control groups
• Have identical initial average characteristics
(observed and unobserved)
• The only difference is the treatment
• Therefore the only reason for the difference
in outcomes is due to the treatment
26 Legovini
Importance to plan ahead: prospective evaluation
Example SME credit
• What determines loan take up?
• Test different offers (with more or less choice)
• What determines loan repayment?
• Test different risk assessment processes
• What determines SME growth?
• Test whether business services improve credit outcomes
Legovini 27
How to define treatment & control
and measure impact?
• Randomly assign the intervention or different
elements of it to different SMEs to obtain
comparable treatment and control groups
• Use clear measures of outcome (take-up,
repayment, growth)
• Collect data on treatment and control groups
• Compare mean outcomes over time
28 Legovini
identify the elements
that cause more impact
Legovini 29
SME credit/training
programs
Advertise the terms of one
loan
Risk assessment type 1
Business advisory services
Risk assessment type 2
Advertise the terms of different
loan options
Max loan take-up
Max loan repayments
Max SME growth
Max IMPACT
Why iE is important to do?
• Know what improves results
• Take better decisions
• Demonstrate to others
• Invest responsibly/make better
fiscal choices
30 Legovini
CONCLUSIONs
1. Use IE evidence now to improve policy
design
2. Use IE to drive decisions and results
31 Legovini
DIME resources
fb
http://www.facebook.com/IEknow
web
http://www.worldbank.org/dime
blog
http://blogs.worldbank.org/impactevaluations
data
http://microdata.worldbank.org/index.php/catalog/impact_evaluation 32