1

What is Impact Evaluation … and How Do We Use It?

Deon FilmerDevelopment Research Group, The World Bank

Evidence-Based Decision-Making in Education WorkshopAfrica Program for Education Impact Evaluation (APEIE)

Accra, GhanaMay 10-14 2010

2

Some examples

• Should my government distribute free textbooks to students to promote learning?

• Should we distribute scholarships to poor children to promote attendance?

• Should teachers be rewarded for learning improvements of their students?

• Should management decisions be devolved to the school level?

Impact evaluation is a way to start answering those questions using rigorous evidence

3

What is Impact Evaluation?Example

• You would like to distribute textbooks to students as a way of improving learning outcomes

• Your intuition tells you that textbooks should matter.• But what is that intuition based on?

– Own experience– “Common sense”– Observing children in schools– Comparisons between children in schools with textbooks and in those without

• Impact evaluation, in this situation, would aim at providing – rigorous evidence, – based on actual experience, – of what the actual impact of providing textbooks is.

4

What is Impact Evaluation?

• How would impact evaluation achieve this aim?– By establishing the causal impact of textbooks on learning outcomes– This is the ultimate goal of impact evaluation.

• This workshop will be about– What makes for a good estimate– How to estimate that impact– How to interpret the estimate

5

Why do we use impact evaluation?

• Understand if policies work– Might an intervention work (“proof of concept”)?– Can an intervention be done on a large scale?– What are alternative interventions to achieve a

particular goal, and how do they compare?

6

Why do we use impact evaluation?

• Understand the net benefits of the program, and cost-effectiveness of alternatives– Requires good cost and benefit data

• Understand the distribution of gains and losses

Budget constraints force selectivityBad policies and programs are wasteful and can be

hurtful

7

Why do we use impact evaluation?

• Demonstrate to politicians, population, donors that a program is effective

This can be key to sustainabilityInforms beliefs and expectations

8

Putting impact evaluation in context

Monitor

ing

Evaluation

Impact evaluation

Regular collection and reporting of information to track whether actual results are being achieved as planned

Analytical efforts to answer specific questions about performance of a program/activities.

Analytical efforts to relate cause and effect.Key part is establishing “what would have happened in the absence of the intervention”

9

Monitoring, Evaluation, and Impact Evaluation

• Periodically collect data on the indicators and compare actual results with targets

• To identify bottle-necks and red flags (time-lags, fund flows)

• Point to what should be further investigated

Monitor

ing

Regular collection and reporting of information to track whether actual results are being achieved as planned

10

Monitoring, Evaluation, and Impact Evaluation

• Analyzes why intended results were or were not achieved

• Explores targeting effectiveness

• Explores unintended results• Provides lessons learned

and recommendations for improvement

Evaluation

Analytical efforts to answer specific questions about performance of a program/activities.

11

Monitoring, Evaluation, and Impact Evaluation

• What is effect of program on outcomes?

• How much better off are beneficiaries because of the intervention?

• How would outcomes change under alternative program designs?

• Does the program impact people differently (e.g. females, poor, minorities)

• Is the program cost-effective?

Impact evaluation

Analytical efforts to relate cause and effect.Key part is establishing “what would have happened in the absence of the intervention”

12

The central problem in Impact Evaluation Analysis:

The counterfactual

• In order to establish the impact of the program, we need to know what would have happened in the absence of the program

• Not in general, but specifically for the people who actually received the program

13

The central problem in Impact Evaluation Analysis: The counterfactual

• What is the effect of a scholarship on school enrollment?

• We want to observe the units of treatment in two states

• What’s wrong with this picture?

Elizabethon 1 July 2010

with scholarship

Elizabethon 1 July 2010

withoutscholarship

14

The central problem in Impact Evaluation Analysis: The counterfactual

• This is impossible!– we never observe the same individual with and

without program at same point in time• The counterfactual is never actually observed• It needs to be estimated• Impact Evaluation Analysis is all about

alternative approaches to estimating the counterfactual

15

Why is the counterfactual important?

• The next session will discuss the counterfactual in more detail

• Here, just one illustration

16

Illustration of the importance of the counterfactual

• Question: What is the best estimate of the impact of the program on enrollment?

Time

Enrollment

AfterBefore

A

B

2008 2010

Program

Impact?

17

Illustration of the importance of the counterfactual

• Question: What is the best estimate of the impact of the program on enrollment?

Enrollment

AfterBefore

A

B

C?

2008 2010

D?

Time

Program

Impact?

18

What is impact evaluation?

• Impact is …the difference between outcomes with the

program and without it• Impact evaluation involves …

estimating the counterfactual so that changes in outcomes can be attributed to the program

19

What is involved in implementing an impact evaluation?

• Determine why an IE is called for• Understand program and its results chain• Determine what to measure• Determine methodology• Carry out – Data collection (baseline, follow up)– Program implementation– Analysis and reporting

• Adjust policy

20

Determine why the evaluation called for

• Specific intervention – Cash transfer to specific students– Specific teacher training program with a particular curriculum– School grant program with a particular structure

• Alternative interventions/complementary interventions– Teacher training versus school grants as a way to improve outcomes– Information and school grants as a way to boost school accountability

and performance• Entire program/cluster of activities

• Reform program

What is the audience (policymakers, technocrats, public at large)?

21

Understand the program using the results chain

• Teachers• Text books• Grants

(What the program does)

• Teachers’ training• School councils

established• Conditional Cash

transfers• Salary incentives

(Goods & Services)

• Increased enrollment

• Lower teachers’ absenteeism

Outcomes

• Higher primary education completion rates

• Higher student learning achievements

Long-term Results

• Lower unemployment• Poverty reduction• Better income

distribution

Inputs Activities Outputs

This results chain provides guidance on what to measure

22

Determine what to measure

• Based on the results chain• Choose indicators for the evaluation– Carefully defined indicators– Can be measured in a precise way– Are expected to be affected by the program…– … within the timeframe of the evaluation

• What are the important sub-populations– E.g. age, gender, urban/rural, SES…

23

Determine indicators using the results chain

• Teachers• Text books• Grants

(What the program does)

• Teachers’ training• School councils

established• Conditional Cash

transfers• Salary incentives

(Goods & Services)

• Increased enrollment

• Lower teachers’ absenteeism

Outcomes

• Higher primary education completion rates

• Higher student learning achievements

Long-term Results

• Lower unemployment• Poverty reduction• Better income

distribution

Inputs Activities Outputs

Indicators related to the implementation of a program

24

Determine indicators using the results chain

• Teachers• Text books• Grants

(What the program does)

• Teachers’ training• School councils

established• Conditional Cash

transfers• Salary incentives

(Goods & Services)

• Increased enrollment

• Lower teachers’ absenteeism

Outcomes

• Higher primary education completion rates

• Higher student learning achievements

Long-term Results

• Lower unemployment• Poverty reduction• Better income

distribution

Inputs Activities Outputs

Indicators related to the results of a program

25

Determine the methodology

• We’ll be talking a lot more about this– Experimental methods– Quasi-experimental methods

• Some principles:– Prefer method that complements program best– Prefer method that does not alter program design

or implementation substantively – Prefer method that does not deny anyone

benefits

26

The IE “cycle”

Baseline data collection

Program implementation

Follow-up data collection

Data analysis and reporting

Program adjust-ements

Design phase

27

The goal of impact evaluation

• To improve policies• For example, to find out how to turn this

teacher…

28

The goal of impact evaluation

• To improve policies• …into this teacher

29

Thank you!