Monitoring and Impact Evaluation Second DM2013 Convening Workshop

Overview: Why ,What, and How

Delhi, India April 23 2014

Joost de Laat, World Bank

DM Example 1: Embrace

1. What is the program?

Deployment of embrace infant warmers in government hospitals

2. Where is it conducted?

Jharkhand, India

3. When is it conducted?

Since 2009

4. Who is the implementing organization?

Embrace

5. Who is targeted?

Low birth weight and premature babies

DM Example 2: Micro-insurance academy (MIA)

1. What is the program?

Micro-health insurance scheme

2. Where is it conducted?

178 villages, Kalahandi district, India

3. When is it conducted?

2009- onward

4. Who is the implementing organization?

MIA & Mahashakti Foundation

5. Who is targeted?

Tribal villagers

Monitoring and (Impact) Evaluation Methods

Needs Assessment

Process Evaluation

Impact Evaluation

Analysis of Beneficiary Needs

Analysis of Program

Implementation

Measures How Much the Program

Impacts Beneficiaries

1. Services being delivered?

2. Clients satisfied? 3. Services reaching

target population?

4. Can we improve cost-effectiveness?

1. Health of mothers and children improved?

2. Learning outcomes improved?

3. More people in jobs?

4. Less poverty?

1. Why are we doing the project? 2. Who is the target population? 3. Why do we think this approach will work in this context? 4. What else could we do?

Introduction to Impact Evaluation

• Why

• What

• How

We need more than programs

we need programs that work

Scale Promising Programs

With good evidence, we can spend money where it matters most

Scale Promising Programs

Sustain Successful Programs

Improve Existing Programs

Supply of Malaria Medicine

52%

78%

94%

CONTROL COMMODITY PLANNER

DIRECT PURCHASE

Improve Existing Programs

Video Link

Impact Evaluation

• Why

• What

• How

• Insert Animation…

Using impact evaluations to truly determine the impact of the program

Using impact evaluations to truly determine the impact of the program

Does promoting hand-washing work?

From Meena's Three Wishes, “Meena tackles the issues of hygiene and sanitation.” UNICEF

Outcome: Children’s

health

Time

Intervention

Impact?

Impact of the program

before after

Treatment villages

What would have happened without the program?

Time

Intervention

Counterfactual

Actual Impact

= Treatment

- Control

Impact of the program Control needs to be a good counterfactual

Before After

Treatment villages

Control villages

Outcome: Children’s

health

Non-random program selection leads to comparisons of apples and oranges

Non-random program selection leads to comparisons of apples and oranges

Randomization creates the right comparison group to measure impact

Randomization creates the right comparison group to measure impact

Impact Evaluations

• Why? Provide sound evidence that can help us: sustain, scale, and improve promising new programs.

• What? Need a good comparison group to measure impact. Randomization can achieve this.

• How…?

Impact Evaluation

• Why

• What

• How

Randomize when not enough resources to reach everyone

Northern Uganda business grant

• 560 youth groups applied.

• 280 received funding through a lottery. 280 comparison.

Pre-School Program in Mozambique • 76 communities identified to receive the program • 30 randomly assigned to the treatment. • 46 acted as the control. • Government so impressed that it expanded it to 600

communities

Randomize when there are not enough resources to reach every village

…or randomize who receives the program first

Sanitation program in Indonesia

• 80 villages received the program in phase one.

• Remaining 80 villages in phase two

Learn more with multiple treatments

Scholarship program in Cambodia. • In 52 schools, well-performing pupils received a merit

scholarship • In 51 schools, poor pupils received a poverty-targeted

scholarship • 105 schools were control

Learn more with multiple treatments

Teacher bonus program in India • 100 schools – individual bonus pay for

teachers’ pupil performance • 100 schools – group bonus pay for overall

school performance. • 100 schools – cash grant for additional teacher • 100 control

Learn more with multiple treatments

Early Child Nutrition and Development program in Colombia

• Villages randomized into 4 groups:

• 24 villages received stimulation program

• 24 received micro-nutrients

• 24 received both.

• 24 in control

• On average 15 pupils

Now is your chance to integrate an impact evaluation in your own program