The why and the how of randomized experiments

Abhijit Vinayak BanerjeeSantiago, December 2009

Why evaluate?

Theory is a very limited guide to policy

• It is very good at alerting us to various possible mechanisms

• It just does not tell us which of those mechanisms actually work and which don’t

• Because theory is always necessarily incomplete. – It is based on concepts, which come out of our

need to limit the complexity of reality– It is based on behavioral assumptions and

assumptions about the environment that are untested and often untestable

An example of a challenging concept• Education

• Let us accept the evidence that education promotes growth

• How to promote education?

• Concepts often used in the growth literature are things like “fraction of population completing primary school”– These are not policy levers

• Policy levers are things like:– Teacher-student ratio, textbooks, school uniforms,

school meals, etc. • Theory tells us nothing about which of works best

• Can we assume that the policy makers know?

An example of unexpected behavior• The demand for health goods

– Like immunization, insecticide treated bed-nets, chlorine for purifying drinking water

• Standard economic theory– These goods are valuable to some people (people who

live in malarial zones, say) less to others

– Pricing at marginal social cost correctly discriminates between those who want them more than the social costs and those who don’t

– However there are externalities: infectious diseases, intra-family issues

– So some subsidy may be justified.

– But no argument for giving them away

– Sunk cost fallacy argues for charging a positive price.

The pricing of health goods• Several studies are exploring

the impact of charging or subsidizing people for health behavior

• First example: Lentils for vaccine. – Immunization is really low in

Rajasthan (less than 5%)

– What is the price elasticity of vaccine demand?

Abdul Latif Jameel Poverty Action Lab

Testing Demand and Supply• One first possibility is that the supply channel is the

problem:– Conducted monthly immunization camps in 60 villages:

regular camps held rain or shine

• One second possibility is that there is a problem of demand: – People not interested in immunization, scared, etc. – Can demand be affected?– Extra incentive: in 30 of the camps, provided a kg of

lentils for each immunization

• 60 camps’ villages remained the control group. Immunization rates were followed in treatment villages, control villages, and one neighboring village of each of the treatment villages

Abdul Latif Jameel Poverty Action Lab

High price elasticity• Surprisingly high price elasticity, in light of the

expected benefits. Why might that be so: – Information?

– Discount rate?

– Something else?

• Smokeless stoves• Chlorine for water: same effect• Bed-Nets:

– Dupas and Cohen : randomize the price at which bednets are offered in pre-natal clinic

– Dupas: similar experiment among households.

• Deworming drugs: take up collapses with (small) positive price

Should People Pay for Health Goods? • Conventional economic theory:

– If the health good has a high externality and is expensive, you want to subsidize its use (possibly even subsidize people to do it, like in the case of the immunization program), in particular when the price elasticity is high

• “New” conventional wisdom (social marketing)– Health goods you give away may not end up in

the hands of those who need it the most– The fact of paying may encourage people to use

the good (sunk cost fallacy)

Abdul Latif Jameel Poverty Action Lab

Evidence from Bed nets

• Do people who get a free bed net use it less than those who have to pay?

• Do people who get a free bed net need it less than those who have to pay?

Abdul Latif Jameel Poverty Action Lab

Effective Coverage: Share of Prenatal Clients Sleeping Under ITN, by Price

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0 10 20 40 50ITN Price (in Ksh)

Nu

mb

er

Sle

ep

ing

un

de

r IT

N

Abdul Latif Jameel Poverty Action Lab

Selection Effects of Price on Health

• Do higher insecticide treated net prices induce selection of more vulnerable women (i.e. sickest)?

• Result: No. People who buy nets at current cost-sharing price are healthier than the average prenatal client in the region – Ability to pay seems to be

the binding factor

Abdul Latif Jameel Poverty Action Lab

Lessons• Find no evidence that cost-sharing reduces wastage on

those who will not use the product• Also find no evidence that cost-sharing induces selection

of those who need net more• Cost-sharing does considerably dampen demand

– As the price goes from zero to the prevailing cost-share price, uptake drops by 75%

• Another example where the cost benefit analysis is not what you thought!– Free distribution seems at least as cost-effective as

cost-sharing (because of externalities)• The debate generated by the paper: does this generalize?

– Proponents of cost sharing proposed various theories to explain why the results may not generalize.

– But they had to formulate new arguments, which can then be tested: this moved the debate beyond where it was… “cost sharing is good”.

Abdul Latif Jameel Poverty Action Lab

Why experiment?

Policy has to be based on learning from experience• History is a set of experiments…

• Why do we need Experiments?

Why experiment: First answer• Selection bias in observational data

• Can sometimes be solved using natural experiments or other non-experimental methods

• But that limits the set of programs that can evaluated.

• And can be less than transparent

Why Experiment: Second answer• You typically know exactly who was treated or

something close

• And you control data collection

• Therefore you can target your data collection.

• As against in an observational study where the quasi-randomization emerges opportunistically. – You study a policy and typically decide ex post who was treated

and who was not. Unless you were directly involved in the policy making—in which case why not try to randomize?

• And especially with Instrumental Variables methods, you don’t always know who you are comparing

• And even when you do, the actual event may be many years past

• Therefore you cannot rely on data collected for you

• Hence detail/creativity in data collection is lower

Example: Decoy visits in the Evaluation of Police Reform in Rajasthan • Can the police be taught to investigate better, treat

people better and not to try to fob off complainants?

• In collaboration with Rajasthan Police Department

• How do you measure the willingess to register crimes

• Decoy visits: Surveyors act as real complainants. – Revealed themselves if the case was about to be registered

– Only 54% of complaints were registered

• Neither training nor more vacations for the police nor community observers in the police station had any effect on registration

• However the (random) number of previous revealed decoy visits matters: each visit increased the probability of FIR registration by 4%

Why experiment: Third answer• Experiments can be designed to test very

specific hypotheses

• Whereas observational studies, because they rely on policy variation that exists in the world, can rarely be mapped to something quite so narrow.

• Therefore we can test individual elements of a policy rather than the policy as a whole

Technology adoption: fertilizer in Kenya (Duflo-Kremer-Robinson)

• Current hike in food price underscore the needs to improve the productivity of agriculture in developing countries.

• The Green revolution has changed the face of Asian Agriculture since the 1970s. Need a green revolution for Africa

• Part of the solution is to develop new technologies, and part is to make sure the ones we have get used.

• Big puzzle: why do so few farmers adopt fertilizer in Western Kenya– The technology is well known– It has high returns (which we confirmed in experiments)– It can be done in small quantity

Possible reasons• Development economists have worked on

this topic for decades. Hypotheses:– Low actual returns– Lack of information/knowledge– Financing difficulties

• Duflo-Kremer-Robinson set up a series of field experiments over several years in Western Kenya to investigate these issues. – Controlled trial on (randomly selected) farmers’

plot: fertilizer seem profitable

Information• Several experiments to understand the role and

the diffusion of information:– Experiments on farmers plot were done with randomly

selected farmers: we follow their adoption and that of the control farmers

– Other ways to disseminate information: • Starter kits (Malawi)• Demonstration farms• Neighbors

– Invited– Non invited (natural diffusion)

• Bottom line: – information plays a role (10% increase in adoption

among pilot farmer)– However it does not diffuse naturally (no effect of

friends network)– Financial barriers remain a concern for farmers

Present-bias• Farmers seem to make plans to use fertilizer at harvest

time, but not carry them out• By the time of planting they have no money left. • Possibly it is because they postpone the purchase until

they need the fertilizer, but then they consume too much in the intervening period (may be because of family demands).

• If this is the case, a program that encourages them to purchase fertilizer right after harvest, rather than later, should be effective.

• With an NGO (ICS), they offered free delivery of fertilizer right after harvest (the “SAFI” program)

Results• SAFI is taken up by 40% of farmer and

led to a 14 percentage point increase in fertilizer use (a 60% increase).

• As large as the impact of a 50% subsidy• Offering free delivery at the time they

need it is not sufficient (it led to a 7 percentage point increase in use).

• When farmer are asked to chose the timing, they chose the early delivery.

What can’t experiments do?• Tell you what exchange rate to set

• But it is not clear that regressions of any kind are the right methodology for answering such questions

• May better to be build simulation models based on reliable parameter estimates.

• From micro regressions.

• Experiments may play an important role here

The question of external validity• How do we know that the result from an experiment

will generalize to other places

• Problem for all forms of empirical research.

• However it might seem that it is less of a problem for cross-country comparisons– Because the result is an average covering a wide range of

environments

• Not exactly right– Different results when you include different controls

– Could be because the set of countries being compared are different

• One advantage of experiments is you are much more likely to know the exact population that is being compared.

However• There is no dismissing this issue.

• Replications help build some confidence in the results– Theory helps give us identify the kind of

replications that make sense.

• The problem remains that as of now organizations that participate in randomized trials tend to be “strong” organizations

• We need more randomized evaluations in weak organizations

Most importantly• There is an infinity of policy questions out

there. – Textbooks, uniforms, meals, teachers,

blackboards, toilets, scholarships, computers..

• Theory is the only reliable guide to what are the right questions to answer

• Unfortunately in many cases we do not have a useable theory: that tells us, for example, textbooks are more like computers or more like teachers.

On balance however• We have learnt things from experiments we

could have not learnt otherwise• The benefits from the experimental approach go

beyond the specific learnings (with their limitations).– Have helped popularize a culture of emphasizing

experimentation and learning from experiments

– And have imposed some discipline on what can and cannot be claimed as known

• Does not mean that we learn only from Randomized Experiments

• But experiments set a standard of evidence and a culture of being willing to embrace failures.

• THANK YOU!