$

Manage Successful Impact Evaluations

FIELD COORDINATOR WORKSHOP

6-10 JUNE 2016WASHINGTON, DC

How Much Does It Cost To Get That Impact?

Measuring Cost Effectiveness8 June 2016

David K. Evans, Senior Economist, Office of the Chief Economist for Africa

The ChallengeIncreasing Evidence of Impact

Lego

vini

et a

l. 20

12

We have all this amazing evidence of interventions that

will work in your country!

Yes, but how much is this going to cost me?

Not just you, man! How much is it going to cost me?

Government MinisterEntrepreneur

IE Expert #1

1. Summarize a complex program in terms of a return on investment

2. Permit comparison across contexts & times

Combining cost & impact data has two goals

Dhal

iwal

et a

l. 20

11

Intuitive appeal of cost-effectiveness estimates

JPAL

201

4

Cost Effectiveness Can Reverse Ranking of Effective

Interventions

What are the most effective education programs?

Source: J-PAL, “Increasing test score performance”

Cost Effectiveness Can Reverse Ranking of Effective

Interventions

What are the most cost-effective education programs?

No overlap between those two groups.

Source: J-PAL, “Increasing test score performance”

Cost Effectiveness Can Reverse Ranking of Effective

Interventions

If we want to responsibly use impact evaluation to inform policy,cost analysis is crucial.

Source: J-PAL, “Increasing test score performance”

Definitions and Debates

Cost benefit analysis

Cost effectiveness analysis

Cost-utility analysis

Cost Benefit Analysis

𝑁𝑒𝑡 𝑝𝑟𝑒𝑠𝑒𝑛𝑡𝑣𝑎𝑙𝑢𝑒=∑𝑡=0

𝑛 𝐵𝑡

(1+𝑟 )𝑡−∑

𝑡=0

𝑛 𝐶𝑡

(1+𝑟 )𝑡

1. Monetize the benefits of the

program

2. Sum those up over time (for n

years) 3. Discount future gains

4. Do the same thing for costs

6. Voilà! A summary measure of the lifetime

net value of the intervention!

5. Subtract costs from benefits

McE

wan

201

2

Cost Effectiveness Analysis

McE

wan

201

2

1. Calculate the cost

2. Divide it by incremental

effect

𝐶𝐸=𝑐𝑜𝑠𝑡 𝑝𝑒𝑟 𝑎𝑑𝑑𝑖𝑡𝑖𝑜𝑛𝑎𝑙𝑢𝑛𝑖𝑡 𝑜𝑓 𝑏𝑒𝑛𝑒𝑓𝑖𝑡

What the h#@! is an additional unit of benefit?

10% increase in business profits 1 standard deviation increase in crop yield

10% increase in farmer knowledge Additional woman receiving prenatal care

Additional student in school 1 s.d. increase in student learning

CBA Absolute desirability of

intervention Allows comparison across

totally different interventions (health versus agriculture)

Requires very strong assumptions (lifetime benefits of intervention…)

Pros and Cons

CEA Much more transparent: No

need to monetize range of benefits

Simpler to explain Useful for comparing

interventions with common outcome (agricultural extension versus targeted savings)

Implicit assumption: Common post-intervention trajectory

McE

wan

201

2

Question: The Minister of Finance is deciding whether to scale up an Agricultural Extension program or a Business Training

program. What kind of analysis will be most useful?

1. Cost-Benefit Analysis

2. Cost-Effectiveness Analysis

Answer: (1)

Question: The Minister of Education wants to increase student test scores. What kind of analysis will be most useful?

1. Cost-Benefit Analysis

2. Cost-Effectiveness Analysis

Answer: (2)

Whose costs?Take a social perspective

Seed distribution program

Ministry of Finance

DeliveryCommunity leadersHouseholds

Ministry of Agriculture NGO Partner

McE

wan

201

2

Question: The government is giving grants to entrepreneurs. Where do these enter the costing equation?

1. As cost to the government.

2. As benefit to the entrepreneur.

3. Both.

Answer: It depends. In CBA, the answer is (3), so it cancels out.

In CEA, we don’t include other benefits (e.g., increased revenues, better health), so why this? So (1).

McEwan 2012 and Dhaliwal et al 2011

Ingredients method• Personnel• Facilities• Equipment• Materials• Other program inputs• Client inputs

Recipe for Cost AnalysisMetaphor for

successful cost analysis

Looks delicious!

McE

wan

201

2

Where do we get the info?Budget• What it was supposed to cost• Not actual cost• Often misses client costs

Interviews & direct observations• Did it go how it was supposed to go? (No)• How many seeds were actually distributed? How much time did

it actually require?

Data collected in impact evaluation• Remuneration• Time use

McE

wan

201

2

Need early: Guides rest of cost analysis

Very difficult to gather ex post

Must be designed ex

ante

Where do we get the info?Budget• What it was supposed to cost• Not actual cost• Often misses client costs

Interviews & direct observations• Did it go how it was supposed to go? (No)• How many seeds were actually distributed? How much time did

it actually require?

Data collected in impact evaluation• Remuneration• Time use

McE

wan

201

2

Need early: Guides rest of cost analysis

Very difficult to gather ex post

Must be designed ex

ante

Field coordinators are uniquely

placed to gather these data.

Value each ingredientAdjust for *– Inflation ---– Time-value *

– Currency ---

+ Add it all togetherCost of the program

What do we do with the info?

McE

wan

201

2

Question: My agricultural extension program has community volunteers who spend 10 hours a week and

receive a bag of rice. How do I cost that?1. Market value of a bag of rice2. Market value of 10 hours of

time for people with similar background

3. Both4. I don’t know; I was checking

my email.

Answer: (3). Hopefully not (4). Come on team, heads in the game. Really, it depends.

If “government only” perspective, could leave out, but volunteers in pilot may not be available at scale.McEwan 2012 and Dhaliwal et al 2011

JPAL

201

4

What Cost Effectiveness Looks Like in Practice

• Cost analysis is extra work• Most studies don’t do it– Of 77 RCTs, 56% reported

zero data on incremental costs (McEwan 2014)

• Not doing it significantly limits policy relevance

Well, that was easy. Not!

That was exhausting!

Now that you know how to do it

A few caveats to keep in mind

Caveats

Cost-Effectiveness Sensitivity to Errors in

Estimates

Additional s.d. of student learning per $100

Evan

s & P

opov

a 20

14

Gray: Measured cost

effectiveness

Black: 90% confidence interval on

impact estimateLesson: Be humble about cross program ordering.

Cost-Effectiveness Sensitivity to Place

Evan

s & P

opov

a 20

14

Actual effectiveness-cost ratio for

India programGray: Other programs

Additional s.d. of student learning per $100

Red: Use other country costs of

community teachers

Lesson: Adapting cost-effectiveness across countries takes work.

Pilot vs Scaled up

Evan

s & P

opov

a 20

14

Cost Sensitivity to Exchange Rates

Dhal

iwal

et a

l. 20

11

Standard exchange rate Purchasing power parity

Assumes relative prices in country Adjusts for price levels across countries but messes up relative prices

Ignoring:1. Categories of ingredients: facilities, equipment,

client inputs2. Sensitivity (to place, to scale, etc.)

Common Pitfalls

Adap

ted

from

De

La B

riere

3. Exchange rates4. Reality (costing ideal vs.

actual program).

Not providing enough details to verify 1 to 4

1. No policymaker adopts a program without knowing the cost (we hope!)

2. Costing and cost effectiveness take us from scholarship to practical policy implementation

3. It is just as important to be prospective with costs as with impact evaluation

4. Field coordinators are particularly well placed to gather cost data

Remember!

• Papers!– Cost-Effectiveness Analysis of Education & Health Interventions in Dev

eloping Countries, by McEwan (2012)

– Comparative Cost-Effectiveness Analysis to Inform Policy in Developing Countries, by Dhaliwal et al (2011)

– Cost-Effectiveness Measurement in Development: Accounting for Local Costs & Noisy Impacts, by Evans & Popova (2014)

• A book! – Cost-Effectiveness Analysis: Methods & Applications, by McEwan &

Levin (2000)

Additional Resources - 1

• IDB Evaluation Hub: Cost Benefit and Cost-Effectiveness– Background materials– Terms of reference– Examples– Templates

• Poverty Action Lab– Examples for student participation and student learning– What does this look like in Excel?

Additional Resources - 2

Credits• All clip art from openclipart.org• “Just Do It” from Adweek.com• NGO partners from OneAcreFund.org• Some other photos from

World Bank Flickr stream

Thank you!