Towards laun hing Randomized Controlled Trials inEuropePresentation prepared for the workshop�E onomi Experiments for EU Agri ultural Poli y Evaluation:Methodologi al hallenges�Angers, 6-7 June 2017Ulri h B. MorawetzUniversity of Natural Resour es and Life S ien es Vienna (BOKU)Slide 1 / 16

Introdu tionIntrodu tionFundamental di� ulties in ex-post evaluation (of CAP) 1:• la k of appropriate ontrol group (unbiased expe ted value)• heterogenous e�e ts (varian e)• Nonlinear e�e ts• temporal and spatial lags & general equilibrium e�e ts• CAP programs that omprise multiple interventionsMy fo us on �rst point, with some dis ussion of se ond.1based on Ferraro (2009) Slide 2 / 16

Unbiased estimated treatment e�e tExpe ted value of treatment e�e tAverage e�e ts of CAP program on the treated:E (ATT ) = E (y1|D = 1)− E (y0|D = 1)︸ ︷︷ ︸Average treatment e�e t on treated =E (y |D = 1)− E (y |D = 0)

︸ ︷︷ ︸Observed di�eren e in average −E (y0|D = 1)− E (y0|D = 0)︸ ︷︷ ︸Sele tion biaswhere y = y0 + (y1 − y0)D.

• y measure out ome (e.g. fertilizer utilization)• D = 1 if farm is treated: omplian e with agri-environment programmandatory Slide 3 / 16

Unbiased estimated treatment e�e tSolutions to sele tion biasObservational data:• Di�eren e in Di�eren e ombined with Propensity S oreMat hing i.e. Chabé-Ferret&Suberview (2013), Kir hweger et al.(2015)• Regression Dis ontinuity Design . i.e Obje tive 1 Funds: eligible ifin ome<0.75 of EU average (Be ker et al, 2013)• Instrument variables from the USA (Roberts and Bu holtz, 2005)Experimental approa hes:• Randomized ontrolled trials (RCT): On-farm-s ale e ologi almodels (Firbank et al., 2003). Raineau&Giraud-Héraud(2017) Slide 4 / 16

Unbiased estimated treatment e�e tExample �Refrain from using silage�• E.g.: Austrian agri-environment program �Refrain from usingsilage�

• Hey produ ed instead of silage• Grass is ut later, more biodiversity• Farms eligible 2:

• if > 0.5 livesto k/ha: ompensation 150 Euro/ha• 10.000 parti ipants, 140.000 ha, 18 Mio Euro in 2009

2simpli�ed eligibility rule Slide 5 / 16

Unbiased estimated treatment e�e tObservational DataEvaluation with observational data:Out ome = hayhay+silage• Di�eren e in Di�eren e with PSM: if pre-treatment observationsand omparable non-parti ipants are available .• Regression Dis ontinuity Design: given the eligbility threshold at0.5 livesto k units/ha, we an apply.• Instrument Variable Regression: no instrument available.

Slide 6 / 16

Unbiased estimated treatment e�e tExperimental approa h• RCT with random farms ex luded from parti ipation: a eptan e inCAP still untested.To in rease a eptan e of RCTs, �Close to Random RCTs� (Du�o et al.,2007; Shadish et al. 2002)• Pilot proje t, phase-in: Randomize in whi h areas program isintrodu ed �rst.• Over-subs ription: If appli ants > budget allows: randomize who ofappli ants an parti ipate.• En ouragement design:

• promote program among randomly sele ted farms• Use promotion intensity as instrument variable to estimate ATE

• �Free-Lun h Randomization� for Agri-Environment measures(Morawetz, 2014) Slide 7 / 16

Unbiased estimated treatment e�e t�Free lun h� randomizationAfter deadline for appli ation for agri-environment program:• Eligible farms whi h applied for silage program• Eligible farms whi h did not apply for silage programs• non-eligible farms

Farms:

applying non−applying non−eligibleFigure: Randomly hosen oordinates of farm lo ations. Slide 8 / 16

Unbiased estimated treatment e�e t�Free lun h� randomizationRandomly sele ted farms get a �free lun h� :• get agri-environment payments (independent whether they applied)• do not have to omply with the rules

Farms:

Randomly selectedFigure: Randomly hosen oordinates of farm lo ations. Slide 9 / 16

Unbiased estimated treatment e�e t�Free lun h� randomizationAt end of period, al ulate the average treatment e�e t on the treatedE (y1|D = 1)︸ ︷︷ ︸applying, non �free lun h� − E (y0|D = 1)

︸ ︷︷ ︸applying, �free lun h�• D = 1: farms willing to parti ipate in agri-environment program• y1: out ome of farms that have to omply to the rules• y0: out ome of farms whi h do not need to omply to the rulesWhy in lude non-applying farms in randomization?• Otherwise biased, be ause expe ted payment would depend onappli ation. Slide 10/ 16

Unbiased estimated treatment e�e tDemonstration Free Lun h RandomizationWho is willing to do a ontra t with me?

• Your part: you a ept the next review request from a journal• May part: I pay you a ho olate ball now

Slide 11/ 16

Unbiased estimated treatment e�e tDemonstration Free Lun h RandomizationWho is born in De ember?• All De ember born are freed from having to a ept the next reviewrequest• All De ember born an keep the a ho olate ball• If born in De ember, you get a ho olate ball, independent from a ontra tIn one year I will evaluate if ho olate balls had an e�e t:• % �next review requests a epted� of those with ontra t bornJanuary to November• minus• % �next review requests a epted� of those with ontra t born inDe ember Slide 12/ 16

Unbiased estimated treatment e�e tDis ussion �free lun h� randomizationAdvantages:• A eptan e hopefully higher as nobody is ex luded• Estimate dire tly ATT• Only minor hange in CAP program ne essary• Applied among FADN parti ipants to redu e survey ostsDisadvantages:• E�e t of being a �free lun h� farm:- applying farms: �re ipro al obligation� (Corrigan and Rousu, 2006)- in ome e�e t• Negative environmental e�e ts in the short run Slide 13/ 16

Varian e estimated treatment e�e tObservational data or experiments?Trade-o� bias and varian e in RCTs (Deaton and Cartwright, 2016).Measure pre ision of ATT with Mean Squared Error (MSE) ):MSE = E (ATT − ATT )2 = Var(ATT ) + bias(ATT )2• RCT with small number of observations (e.g. 200):

• ATT unbiased, but possibly large varian e• Observational study with many observations (e.g. 10.000):

• ATT possibly biased but smaller varian e. Slide 14/ 16

Con lusionsCon lusions• Suitability of method depends on spe i� program

• Pre-treatment and omparable non-treated observations available?DiD-PSM• Arbitrary threshold available? RDD• Instrument available? IV• Pilot, phase-in, over-subs ription? Close to random RCT.• Ex-post evaluation of Agri-Environment Program? Free lun hrandomization.

• Suitability depends on joint e�ort of program designers and evaluators:• olle t pre-treatment observations• in lude arbitrary eligibility rules• run phase-in• apply �free lun h randomization� Slide 15/ 16

Con lusionsReferen e• Angrist, J.D., and J. Pis hke. 2010. �The Credibility Revolution in Empiri al E onomi s: How BetterResear h Design is Taking the Con out of E onometri s.� Journal of E onomi Perspe tives 24(2): 3�30.• Be ker SO, Egger PH, von Ehrli h M (2013) Absorptive Capa ity and the Growth and Investment E�e ts ofRegional Transfers: A Regression Dis ontinuity Design with Heterogeneous Treatment E�e ts. Ameri anE onomi Journal: E onomi Poli y 5:29-77.• Chabé-Ferret S, Subervie J (2012) E onometri methods for estimating the additional e�e ts ofagri-environmental s hemes on farmers' pra ti es. Evaluation of Agri-Environmental Poli ies: Sele tedMethodologi al Issues and Case Studies 185-198.• Corrigan, J.R., and M.C. Rousu. 2006. �The E�e t of Initial Endowments in Experimental Au tions.�Ameri an Journal of Agri ultural E onomi s 88(2): 448-457.• Deaton A, Cartwright N (2016) Understanding and Misunderstanding Randomized Controlled Trials. NBERWorking Paper• Du�o, E., Glennerster, R., and M. Kremer. 2007. �Using Randomization in Development E onomi sResear h: A Toolkit.� In Handbook of Development E onomi s, eds. T.P. S hultz and J.A. Strauss. p.3895�3962.• Ferraro, P.J. 2009. �Counterfa tual thinking and impa t evaluation in environmental poli y.� New Dire tionsfor Evaluation 2009(122): 75�84.• Firbank LG, Heard MS, Woiwod IP, et al (2003) An introdu tion to the Farm-S ale Evaluations ofgeneti ally modi�ed herbi ide-tolerant rops. Journal of Applied E ology 40:2-16.• Kir hweger S, Kantelhardt J, Leis h F (2015) Impa ts on e onomi farm performan e fromgovernment-supported investments in Austria. Agri ultural E onomi s - Cze h (Zem¥d¥lská ekonomika)61:343-355.• Morawetz UB (2014) A on ept for a randomized evaluation of agri-environment measures. In: S hmid E,Vogel S (eds) The Common Agri ultural Poli y in the 21st Century. fa ultas.wuv, Vienna, Austria, pp113-130• Raineau, Y and Giraud-Héraud, E.(2017) Nudge �eld experiment for pesti ide use redu tion.• Roberts MJ, Bu holtz S (2005) Slippage in the Conservation Reserve Program or Spurious Correlation? AComment. Am J Agr E on 87:244-250.• Shadish, W.R., T.D. Cook, and D.T. Campbell. 2002. Experimental and quasi-experimental designs forgeneralized ausal inferen e. Boston: Houghton Mi�in. Slide 16/ 16