randomized controlled trials, development...

Randomized Controlled Trials, Development Economics and Policy Making in Developing Countries

Esther Duflo Department of Economics, MITCo-Director J-PAL [Joint work with Abhijit Banerjee and Michael Kremer]

Randomized controlled trials have greatly expanded in the last two decades

Randomized controlled Trials were progressively accepted as a tool for policy evaluation in the US through many battles from the 1970s to the 1990s.

In development, the rapid growth starts after the mid 1990s Kremer et al, studies on Kenya (1994) PROGRESA experiment (1997)

Since 2000, the growth have been very rapid.

J-PAL | THE ROLE OF RANDOMIZED EVALUATIONS IN INFORMING POLICY 2

Cameron et al (2016): RCT in development


0

50

100

150

200

250

300

1975 1980 1985 1990 1995 2000 2005 2010 2015Publication Year

Figure 1: Number of Published RCTs

BREAD Affiliates doing RCT


0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

1980 or earlier 1981-1990 1991-2000 2001-2005 2006-todayPhD Year

Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs

* Total Number of Fellows and Affiliates is 166.

Top Journals


3ie_time

Sector2000200120022003200420052006200720082009201020112012% of IER pre-2013

Agriculture and rural development126681251522291548489.70%

Economic policy01000001046220.70%

Education911121028223425356667908223.10%

Energy00001102021610.60%

Environment and disaster management00110138781118173.40%

Finance0121364915141728175.50%

Health, Nutrition, and Population 2133315450787410110017219621623364.90%

Information and communications technology0000119336814182.80%

Private sector development012335585122226215.10%

Public sector management02032435614613133.30%

Social protection1268217141820294751525715.10%

Transportation00001111002210.40%

Urban development00101201244130.80%

Water and sanitation012547558121312134.20%

Total studies3045466782114111143173274296370377.

YearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCTYearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCT

198111100%1981-1990111385%

198211100%1991-200011715874%

1983000%2001-200523636664%

1984000%2006-today1405238659%

1985010%

198622100%

198711100%

19882367%

198922100%

199022100%

199133100%

19921250%

19935683%

199481080%

199591182%

1996192095%

199791753%

1998192383%

1999213560%

2000233174%

2001304764%

2002274560%

2003516974%

2004548762%

20057411863%

20067812761%

20079315460%

200811719261%

200916629057%

201019732161%

201125339564%

201227243063%YearNumber of Published RCTsNumber of Published Impact EvalsFraction of Published Impact Evaluations that are RCT

201313423457%1981-2000ERROR:#REF!ERROR:#REF!ERROR:#REF!

20147615748%2001-2005ERROR:#REF!ERROR:#REF!ERROR:#REF!

2015198622%2006-todayERROR:#REF!ERROR:#REF!ERROR:#REF!

Figure 1: Fraction of Published Impact Evaluations that are RCTs

Fraction of Published Impact Evaluations that are RCT1981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201220132014201511000110.666666666666666631110.50.833333333333333370.80.818181818181818230.950.529411764705882360.826086956521739140.60.741935483870967750.638297872340425560.60.739130434782608650.620689655172413810.62711864406779660.614173228346456710.603896103896103930.6093750.572413793103448310.613707165109034230.640506329113924020.632558139534883710.572649572649572610.484076433121019110.22093023255813954

Publication Year

Figure 1: Fraction of Published Impact Evaluations that are RCT

Fraction of Published Impact Evaluations that are RCT1981-19901991-20002001-20052006-today0.846153846153846150.7405063291139240.644808743169398960.58885163453478628

Publication Date

Number of Published RCTs1981-19901991-20002001-20052006-today111172361405


Number of Published RCTs1981198219831984198519861987198819891990199119921993199419951996199719981999200020012002200320042005200620072008200920102011201211000212223158919919212330275154747893117166197253272

Publication Year



Publication Year



Publication Date




Publication Year



Publication Year



Publication Date




Publication Year



Publication Year



Publication Date




Publication Year

3iePaper

SectorDifference-in-DifferenceInstrumental VariablesRCTRegression Discontinuity DesignPSM or OMMNotesDifferences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method

Agriculture and rural development32.00%21.00%20.50%0.00%43.80%16.70%8.30%66.40%2.40%16.10%

Economic policy43.80%31.30%0.00%6.30%37.50%

Education20.90%10.90%58.30%6.30%17.70%

Energy35.70%35.70%0.00%0.00%57.10%

Environment and disaster management27.60%22.40%30.30%2.60%34.20%

Finance29.80%17.70%41.10%0.00%24.20%

Health, Nutrition and Population9.50%4.40%83.20%0.70%6.50%

Information and communications technology25.40%3.20%66.70%3.20%20.60%

Private sector development36.50%19.10%26.10%7.00%33.90%

Public sector management31.10%23.00%44.60%5.40%16.20%

Social protection31.40%12.30%40.80%6.20%34.30%

Transportation55.60%11.10%22.20%11.10%22.20%

Urban development36.80%15.80%15.80%5.30%36.80%

Water and sanitation13.80%5.30%72.30%0.00%17.00%s

Total16.70%8.30%66.40%Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 20162.40%16.10%Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 2016

Source: Drew B. Cameron , Anjini Mishra , Annette N. Brown "The growth of impact evaluation for international development: how much have we learned?" Journal of Development Effectiveness Vol. 8, Iss. 1, 2016

Figure 2: Evaluations by Type

Differences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method0.167000000000000018.3000000000000004E-20.664000000000000032.4E-20.161Differences in DifferencesInstrumental VariablesRCTRegression DiscontinuityPropensity Score Matching, or Other Matching Method0

Aidgrade

YearPercent RCTs out of TotalPercent RCTs in EconPercent RCTs in Other FieldsTotal Number of EvaluationsTotal # of Econ EvaluationsTotal # of Other EvaluationsTotal # of RCTsTotal # of RCTs (Econ)Total # of RCTs (other)YearPercent RCTs in EconPercent RCTS in Other FieldsNon-RCTs

19821000100101101198201000

19891000100101101198901000

19901000100202202199001000

19911000100101101199101000

199250050202101199205050

19931000100101101199301000

19951000100101101199501000

19961000100404404199601000

19971000100202202199701000

19981000100707707199801000

1999100100100321321199966.666666666733.33333333330

20001000100303303200001000

200188.8888950100927817200111.111111111177.777777777811.1111111111

200266.666665075624413200216.66666666675033.3333333333

20039066.6666610010379272003207010

200458.3333344.444441001293743200433.33333333332541.6666666667

20056033.333331005323122005204040

200658.3333328.571431001275725200616.666666666741.666666666741.6666666667

200762.525100844514200712.55037.5

200856.2536.363641001611594520082531.2543.75

200947.3684237.510019163963200931.578947368415.789473684252.6315789474

20104018.1818210015114624201013.333333333326.666666666760

20115038.88889100221841174201131.818181818218.181818181850

201272.7272766.666661001192862201254.545454545518.181818181827.2727272727

201386.6666684.615391001513113111201373.33333333336.666666666720

2014100100440440201410000

Figure 3A. Aidgrade.org Evaluations

Total Number of Evaluations1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201411212114273396101251281619152211154Total # of Econ Evaluations198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320140000000000202239374111611189134Total # of Other Evaluations1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201411212114271374732545344210

Figure 3B: Aidgrade.org Evaluations By Type

Percent RCTs out of Total198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010010010088.88889000000000466.6666599999999939058.3333299999999976058.33332999999999762.556.2547.36842405072.72727000000000486.666659999999993100Percent RCTs in Econ1982198919901991199219931995199619971998199920002001200220032004200520062007200820092010201120122013201400000000001000505066.66665999999999344.4444433.33332999999999728.5714299999999992536.36363999999999737.518.18181999999999838.88889000000000466.66665999999999384.615390000000005100Percent RCTs in Other Fields198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010010010010075100100100100100100100100100100100

Figure 3. Aidgrade.org Evaluations by Type

Percent RCTs in Econ19821989199019911992199319951996199719981999200020012002200320042005200620072008200920102011201220132014000000000066.666666666666657011.11111111111111116.6666666666666642033.3333333333333292016.66666666666666412.52531.57894736842105113.33333333333333431.81818181818181754.5454545454545473.333333333333329100Percent RCTS in Other Fields198219891990199119921993199519961997199819992000200120022003200420052006200720082009201020112012201320141001001001005010010010010010033.33333333333332910077.7777777777777865070254041.6666666666666715031.2515.78947368421052626.66666666666666818.18181818181818318.1818181818181836.6666666666666670Non-RCTs19821989199019911992199319951996199719981999200020012002200320042005200620072008200920102011201220132014000050000000011.111111111111133.3333333333333431041.6666666666666714041.66666666666666437.543.7552.631578947368425605027.27272727272728200













BREAD Affiliates

Total Number of Affiliates & Fellows# with some RCTs# mostly RCTs

1658662Year of PhD# with some RCTs# mostly RCTs# of Affiliates in yearPercent of Affiliates doing some RCT

1980 or earlier321225%

1981-1990962733%

Number of BREAD Affiliates who do RCTs1991-200015133444%

Year of PhD# with some RCTs# mostly RCTs# of Affiliates in yearPercent of Affiliates doing some RCT2001-200521154151%

19620010%2006-today38265767%

19660030%

19720010%

1973111100%

1974101100%

19750020%

19760010%

19770010%

1979111100%

198110333%

198221367%

19830020%

19840010%

198511333%

198610250%

19870020%

198822633%

198922450%

19900010%

19910010%

199222633%

19930010%

1994111100%

19950030%

199610333%

19970010%

199855771%

199932650%

200033560%

200142757%

200222450%

2003757100%

200444850%

2005421040%

2006861080%

200765875%

200832475%

200954956%

201032560%

2011521050%

201264875%

201321367%

Total8662

*Total Number of Fellows and Affiliates is 165

Total Number of Some RCTTotal Number of Mainly RCT

8662

Figure 4. Fraction of BREAD Affiliates & Fellows with 1 or more RCTs

Percent of Affiliates doing some RCT1980 or earlier1981-19901991-20002001-20052006-today0.250.333333333333333310.441176470588235280.512195121951219520.66666666666666663

PhD Year

BREAD Conf Graph

Row LabelsSum of Total # of PapersSum of Total # of RCTs

2003144

2004122

2005121

2006132

2007143

2008205

2009154

2010138

2011145

20122211

2013209

2014156

2015146

201671

Grand Total20567

YearTotal # of PapersTotal # of RCTs% of Papers using a RCT

200314429%

200412217%

20051218%

200613215%

200714321%

200820525%

200915427%

201013862%

201114536%

2012221150%

201320945%

201415640%

201514643%

20167229%

Figure 5. Percent of BREAD Concerence Papers using a RCT

% of Papers using a RCT

200320042005200620072008200920102011201220132014201520160.28571428571428570.166666666666666668.3333333333333329E-20.153846153846153850.214285714285714270.250.266666666666666660.615384615384615420.357142857142857150.50.450.40.428571428571428550.2857142857142857

Year

Top Journals

Info Needed:How many development papers, how many RCTs

MIT Lib?199020002015

AERYesYesYesYes

QJE1 year lagYesYesYes

JPEYesYes

Restud

Econometrica

Number of Development Papers

Year

Journal Title201520001990

AER

QJE

JPE

Restud

Econometrica

Number of RCTs

Year

Journal Title201520001990

AER

QJE

JPE

Restud

Econometrica

Table 6: Papers in Top Journals

JournalYearTotal # of Papers# of Development Papers# of which are RCTs

AER2015101154

20004860

19905720

QJE20154011

20004350

19905230

JPE20153643

20005170

19906590

Restud20154872

20003630

19904010

Econometrica20154650

20003700

19906420

* These include papers in the following journals: AER, QJE, JPE, Restud, Econometrica

Table 2: Papers in Top 5 Journals

YearTotal # of Papers# of Development Papers# of which are RCTs

20152713210

2000215210

1990278170

* These include papers in the following journals: AER, QJE, JPE, Restud, Econometrica

https://www.aeaweb.org/journals/aer/issueshttp://qje.oxfordjournals.org/content/by/yearhttp://www.journals.uchicago.edu/toc/jpe/currenthttp://www.restud.com/accepted-papers/http://www.econometricsociety.org/publications/econometrica/browse

NEUDC Conference

Info Needed:How many RCTs by year

Host UniversityYearTotal # of PapersTotal # of RCTsPercent RCTRounded

Brown20152204018.181818181818.2%

BU20142013617.910447761217.9%

Harvard 20132024924.257425742624.3%

Dartmouth20121692715.976331360916.0%

Table 3: NEUDC Conference Papers

YearTotal # of RCTsPercent RCTs

20154018.2%

20143617.9%

20134924.3%

20122716.0%

Many sectors, many countries


Why have RCT had so much impact?

Focus on identification of causal effects (across the board)

Assessing External Validity

Observing Unobservables

Data collection

Iterative Experimentation

Unpack impacts


Focus on Identification across the board! The key advantage of RCT was perceived to be a clear

identification advantage

With RCT, since those who received a treatment are randomly selected in a relevant sample, any difference between treatment and control must be due to the treatment

Most criticisms of experiment also focus on limits to identification (imperfect randomization, attrition, etc. ) or things that are not identified even by randomized trials (distribution of treatment effects, effects elsewhere).


Focus on Identification across the board! Before the explosion of RCT in development, a literature

on RCT in labor and public finance has thought of other ways to identify causal effects

In development economics, there was a joint development of the two literatures (natural experiment and RCT), which has made both literatures stronger, and perhaps less different than we initially thought they would be: Natural experiments think of RCT as a natural benchmark (not just

an hypothetical gold standard). Development of methods to go beyond simple comparison of

treatment and control in experiments, and richer designs


Encouragement design

Difference in take up caused by encouragement

People who take up program

Focus on Identification across the board! Before the explosion of RCT in development, a literature on

RCT in labor and public finance has thought of other ways to identify causal effects

In development economics, there was a joint development of the two literatures (natural experiment and RCT), which has made both literatures stronger, and perhaps less different than we initially thought they would be: Natural experiments think of RCT as a natural benchmark (not just an

hypothetical gold standard). Extremely well identified non randomized studies.

Development of methods to go beyond simple comparison of treatment and control in experiments, and richer designs

Ultimately, the advantage of RCT in terms of identification is a matter of degree, rather than a fundamental difference.






Data collection


Unpack impacts


External Validity

Will results obtained somewhere generalize elsewhere?

A frequent criticism of RCT is that they dont guarantee external validity

Which is quite right, but it is not like they are less externally valid

And because they are internally valid, and because you can control where they will take place: compared across contexts. they can be purposefully run in different contexts Prediction can be made of what the effects of related programs

could be.


Bayesian Hierarchical Modelling of all the MF results : Profits Meager (2015)

Bayesian Hierarchical Modeling-- Meta analysis (consumption)

15

Example 2: Targeting the Ultra Poor Program: Coordinated evaluation in several countries

Beneficiary

Productive asset transfer

Health

Consumption support Technical skills training

Home visits

Savings

Banerjee et al, 2015

16

http://www.sciencemag.org/content/348/6236/1260799.full

Country by country results: Assets

Banerjee et al, 2015

-0.1

0.2

0.5

0.8Endline 1 Endline 2

Ass

et c

hang

e (s

tand

ard

dev

iatio

ns)

17

http://www.sciencemag.org/content/348/6236/1260799.full

Country by country results: Consumption

-5%

0%

5%

10%

15%

20%

Endline 1 Endline 2

% C

hang

e in

per

ca

pita

con

sum

ptio

n

18

Structured Speculation

Ultimately, if the results are similar it is nice, but if they are different the ex-post analysis is speculative.

Banerjee, Chassang, Snowberg (2016) propose to be explicit about such speculation, and that researchers should predict what the effect may be for other interventions, or in other contexts.

This can then motivate running such experiments, and guesses can be falsified.

Example: Dupas (2014)Effect of short run subsidies on long run adoption depend on the timing of costs and benefits, and how quickly uncertainty about them is resolved: this allows her to classify the goods.






Data collection


Unpack impacts


Observing unobservables

Some things simply cannot be observed in the wild, with naturally occurring variation

Negative income tax experiment was designed as an experiment to separate income and substitution effects

Many experiments in development are designed likewise to capture such effects: Karlan Zinman Observing Unobservables Cohen Dupas and Ashraf Dupas Shapiro: selection and treatment

effect of prices. Bertrand et al. Corruption in driving licences in Delhi.






Data collection


Unpack impacts


Innovative data collection

Innovative data collection does not require an experiment.

But experiments have two features which have motivated creativity in measurement We know precisely what we are trying to measure: payoff to the

person who is designing the questionnaire We know that there will likely be enough power to measure such

effects

As a result, lots of innovation in measurement: Borrowing from other fields : psychology, political science,

agriculture, web scraping, wearable techology, Inventing new methods: e.g. Olken 2007






Data collection


Unpack impacts


Iterative experimentations

Some great natural experiment leave us with some unanswered questions: Why are elite school not working for the marginal child? Why are (some) charter school working so well?

One other advantage of experiments is that one is never stuck with one particular surprising answer: you can continue to experiment in the same setting till you have some clarity.

Example: Duflo, Kremer, Robison multi-year work on fertilizer. People dont use fertilizer, even though it is profitable One set of experiment on financing One set on learning and social learning.






Data collection


Unpack impacts


Unpack impacts

This is a related point, but more narrowly focused on policy design.

There are many many possible ways to design a particular programs

Usually, one version is tried out

But if it works what was essential? Effort to unpack Conditional Cash Transfer Example of doing everything at once: Raskin program, Indonesia

Many people do not receive the rice they are eligible for, or over pay Would transparency help?


They distribute 4 version of a cards to eligible villagers in 378 villages, randomly chosen out of 572


Other sources of variation and results

They also varied: Public (common) knowledge of the program Fraction of people who get the phyisical card

Results: Making the card distribution public knowledge makes it more

effective The physical card matter: information (in the form of list) alone is not

sufficient (Perception of) accountability does not seem to make much of a

difference.

The government decided to scale the version of the card with most info and the list to 65 million beneficiaries!


What has been the policy impacts of RCTs? Is the Raskin Case unique or unusual?

A study designed by researchers with several treatments and an underlying economic model, destined to be published in a to academic journal

but that still had large policy impact

Some have argued that the research impact of RCT has potentially come at the expense of real-world impact:

Researchers and policy makers interests may diverge Research slow down the process of iteration

Evidence Out of 700 projects on going or completed on the J-PAL site, there are

only 9 story of scale up or policy impact. However, this is not a census of J-PAL study (or of RCT). Story selected for

high impacts : the sum of people reached is about 200 million.

30

Over 200 million people reached through scale-ups of programs evaluated by J-PAL researchers

ProgramPeople

Reached (mn)

School-based Deworming 95

Raskin: Subsidized Rice (Indonesia) 66

Teaching at the Right Level (India) 34

Generasi: Conditional Community Block Grants (Indonesia)

6

Chlorine Dispensers for Safe Water (East Africa) 0.5

Free Insecticidal Bednets Policy influence

Police Skills Training Policy influence

TOTAL 202 mn31

Getting a sense of overall influence is difficult Returns to R&D highly skewed.

- Most scholarly articles are never cited- Most start-ups fail- Venture capitalists get most of their revenue from a small number of

investments

Still huge payoffs to R&D

Ideas take a long time to percolate through the system, and many RCT are fairly recent

Many RCT find that things DO NOT work as well as hoped (microcredit, smokeless stoves)


RCTs and real world impact: The case study of DIV

To solve the census problem, we focus on one case study: USAID DIV.

USAIDs Development Innovation Ventures (DIV) offers an opportunity to compare outcomes in selected sample of award winners:

DIV has open approach: no top-down restriction on sector, strategy Grantees include social entrepreneurs, NGOs and development

researchers

Staged financing (Pilot $150,000; Testing $1,500,000; Transition to Scale $15,000,000)

Openness on scaling strategy (Commercial, public-sector or hybrid funding

Emphasis on cost effectiveness; Attention to management team, external commitments, but no

rigid litmus test

33

Methods

Coverage: 43 DIV awards made from 2010-2012; total value $17.3m

Here just examine reach, the estimated number of people exposed to the original and adapted versions of the innovation, after the DIV funding.

Do not compare measures of the size of impact per beneficiary

Do not estimate the likelihood that reach will be sustained or increased in the future

Does not assume the credit to further expansion all goes to DIV

One (of several) components of social return calculation

Specifically, we focus on number of awards reaching more than 100k or more than 1 M people.

34

DIV-Supported several Innovations Reaching > 100K People5 INNOVATIONS REACHED MORE THAN 1 MILLION PEOPLE Voter Report Cards (2

awards)

Election Monitoring Technology

Consumer Action and Matatu Safety

Digital Attendance Monitoring

Dispensers for Safe Water

6 INNOVATIONS REACHED MORE THAN 100K AND LESS THAN ONE MILLION PEOPLE Scaling CommCare for

Community Health Workers (2 awards)

d.light Innovative Financing for Solar Systems

Sustainable Distribution for Improved Cookstoves

Recruiting and Compensating Community Health Workers

VisionSpring BoPtical Care

Renewable Powered Microgrids for Rural Lighting

35

High reach of innovations with RCTs/involvement of development economics researchers 24 awards had RCT/researcher involvement, of which:

42% (10 awards) reached more than 100,000 people 25% (6 awards) reached more than one million people

19 awards did not have RCT, of which: 16% (3 awards) reached more than 100,000 people No awards have yet reached more than one million

people

Overall DIV numbers favorable relative to many impact investors Arbitrage opportunity from openness to multiple type s of

innovation? Discipline of evidence useful?


While early stage awards have low probability of attaining reach, they have high expected reach per dollar spent

Award StageNumber

of Awards

Award Value

Fraction Reaching More than

100,000 people

Fraction Reaching More than 1,000,000 people

People Reached

Expenditure per Person Reached

Stage 1 (< $100,000) 24 $2,353,136 17% (4/24) 8% (2/24) 6,723,733 $0.35

Stage 2 (

Pathways to reach

DIV awards for innovations with RCTs reached > 100,000 people through a variety of partnerships: Country governments (e.g. Zambia CHW recruiting, India

biometric monitoring) Donors (e.g. cookstoves in Ethiopia and Sudan) NGOs/Social Enterprises (e.g. Dispensers for Safe Water) Private sector firms (e.g. newspapers, banks, insurance

companies, Qualcomm, Safaricom) Three of five innovations reaching more than one million people

had earlier RCTs demonstrating impact, potential for cost effectiveness: researcher/project selection?

38

Why might projects involving RCT be more likely to have future reach? Convincing force of evidence [most projects that do not

involve RCT try to scale through retails sale, which is harder]

Nothing to do with the RCT per se: Close involvement of researchers help ideas grounded in basic

science percolate research (like in biotech). In particular: Influence of behavioral economics/information:

focus on low cost interventions , which are more likely to scale

Selection of good projects [willing to do an RCT]


Conclusion

The projects evaluated by RCT that then have reached many people tend to be low-cost, well defined, simple.

So what has made RCT useful as a research tool (ability to iterate, zero-down to component, test a theory) is exactly what has turned out to make them policy relevant: details matter tremendously, and RCT tend to get the details right.

An alternative pathway: BRAC, PROGRESA. Complexedinterventions replicated in many contexts.

And a third one: innovation within existing governments and institutions.

Tamil Nadu innovation Fund, Nudge Unit Gujarat Pollution Control Board


Conclusion

For RCT to move from the large research impacts to large policy impact, we need a range of complementary institutions: Meta-analysis Review article Review panels Registry of Experiments has started and is successful (706 studies as of

June 8), Appropriate support and experiment to support the learning

needed to move from successful pilot to policy at scale Iterations to design scalable (robust) versions and measure their

effects Equilibrium effects Political economy /industrial organization of implementation

J-PAL | THE IMPACT OF RCT ON RESEARCH AND POLICY 41

Randomized Controlled Trials, Development Economics and Policy Making in Developing CountriesRandomized controlled trials have greatly expanded in the last two decades Cameron et al (2016): RCT in developmentBREAD Affiliates doing RCT Top Journals Many sectors, many countriesWhy have RCT had so much impact? Focus on Identification across the board!Focus on Identification across the board!Encouragement designFocus on Identification across the board!Why have RCT had so much impact? External ValidityBayesian Hierarchical Modelling of all the MF results : Profits Meager (2015)Bayesian Hierarchical Modeling-- Meta analysis (consumption)Example 2: Targeting the Ultra Poor Program: Coordinated evaluation in several countries Country by country results: Assets Country by country results: ConsumptionStructured Speculation Why have RCT had so much impact? Observing unobservables Why have RCT had so much impact? Innovative data collection Why have RCT had so much impact? Iterative experimentationsWhy have RCT had so much impact? Unpack impactsThey distribute 4 version of a cards to eligible villagers in 378 villages, randomly chosen out of 572 Other sources of variation and resultsWhat has been the policy impacts of RCTs? Over 200 million people reached through scale-ups of programs evaluated by J-PAL researchersGetting a sense of overall influence is difficultRCTs and real world impact: The case study of DIVMethodsDIV-Supported several Innovations Reaching > 100K PeopleHigh reach of innovations with RCTs/involvement of development economics researchers While early stage awards have low probability of attaining reach, they have high expected reach per dollar spentPathways to reach Why might projects involving RCT be more likely to have future reach?ConclusionConclusion

randomized controlled trials, development...

Documents