fls 6415 - causal inference for the political economy of
TRANSCRIPT
Political Economy Causal Inference
FLS 6415 - Causal Inference for the PoliticalEconomy of Development
Week 3 - The Effects of Democracy & Field Experiments
Jonathan Phillips
August 2017
1 / 27
Political Economy Causal Inference
The Effects of Democracy
É What are the key features of democracy as an institution?
É What institutional feature do these studies test?
2 / 27
Political Economy Causal Inference
The Effects of Democracy
É What are the key features of democracy as an institution?
É What institutional feature do these studies test?
2 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why might direct democracy affect economic outcomes?
É Median voter theoremÉ Item-unbundlingÉ Elite knowledge
É Political outcomes?É ParticipationÉ Elite captureÉ Procedural legitimacy
3 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why might direct democracy affect economic outcomes?É Median voter theoremÉ Item-unbundlingÉ Elite knowledge
É Political outcomes?É ParticipationÉ Elite captureÉ Procedural legitimacy
3 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why might democracy NOT affect economic outcomes?
É Elite respond by using de facto power to re-assert control(Olken?)
É Weak accountability: Elections don’t sanction politiciansÉ Strong accountability: Local decision-making already
responsiveÉ Selection: democracy only happens in countries where the
poor already have powerÉ Reverse Causality: Economic outcomes drive democratic
transition
4 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why might democracy NOT affect economic outcomes?É Elite respond by using de facto power to re-assert control
(Olken?)É Weak accountability: Elections don’t sanction politiciansÉ Strong accountability: Local decision-making already
responsiveÉ Selection: democracy only happens in countries where the
poor already have powerÉ Reverse Causality: Economic outcomes drive democratic
transition
4 / 27
Political Economy Causal Inference
The Effects of Democracy
É What does Olken test?
É PopulationÉ SampleÉ TreatmentÉ ControlÉ Potential OutcomesÉ Treatment Assignment MechanismÉ Outcome Measures
5 / 27
Political Economy Causal Inference
The Effects of Democracy
É What does Olken test?É PopulationÉ SampleÉ TreatmentÉ ControlÉ Potential OutcomesÉ Treatment Assignment MechanismÉ Outcome Measures
5 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why does he use a field experiment?
É Threats to causal inference in observable data:É Omitted variables: Political power (or anything else) shapes
both political rules and economic outcomesÉ Selection: Countries which can benefit from direct democracy
implement it (Switzerland?)É Reverse causation: Economic growth leads to direct
democracy
6 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why does he use a field experiment?É Threats to causal inference in observable data:
É Omitted variables: Political power (or anything else) shapesboth political rules and economic outcomes
É Selection: Countries which can benefit from direct democracyimplement it (Switzerland?)
É Reverse causation: Economic growth leads to directdemocracy
6 / 27
Political Economy Causal Inference
The Effects of Democracy
É What results does Olken find?
É Project TypeÉ Project LocationÉ Satisfaction
7 / 27
Political Economy Causal Inference
The Effects of Democracy
É What results does Olken find?É Project TypeÉ Project LocationÉ Satisfaction
7 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?
É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
processÉ What explains the positive effects?
É We don’t knowÉ Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
process
É What explains the positive effects?É We don’t knowÉ Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
processÉ What explains the positive effects?
É We don’t knowÉ Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
processÉ What explains the positive effects?
É We don’t know
É Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
processÉ What explains the positive effects?
É We don’t knowÉ Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É What explains the null effects?É De facto elite power and lobbyingÉ Elites doing a good jobÉ Less de facto power among women to ’rig’ election resultÉ Broader institutions, eg strategically anticipating funding
processÉ What explains the positive effects?
É We don’t knowÉ Procedural legitimacyÉ Knowledge?É Pre-implementationÉ Women alter a pre-set elite agenda?
É What’s the theory for a change in satisfaction without achange in projects?
8 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why isn’t this all about hamlet population size?
É Collective action?É Small villages form coalitions?
9 / 27
Political Economy Causal Inference
The Effects of Democracy
É Why isn’t this all about hamlet population size?É Collective action?É Small villages form coalitions?
9 / 27
Political Economy Causal Inference
The Effects of Democracy
É How does Beath et al’s experiment differ?
É ?É How do their results differ?
É Project type/location effectsÉ Similar findings for women’s projects
10 / 27
Political Economy Causal Inference
The Effects of Democracy
É How does Beath et al’s experiment differ?É ?
É How do their results differ?
É Project type/location effectsÉ Similar findings for women’s projects
10 / 27
Political Economy Causal Inference
The Effects of Democracy
É How does Beath et al’s experiment differ?É ?
É How do their results differ?É Project type/location effectsÉ Similar findings for women’s projects
10 / 27
Political Economy Causal Inference
The Effects of Democracy
É How do we explain the differing results of Olken and Beathet al?
É Chance!É Sample sizeÉ Treatment duration expectationsÉ Local accountability for councilsÉ Status of womenÉ Ability to strategically manipulate agenda
11 / 27
Political Economy Causal Inference
The Effects of Democracy
É How do we explain the differing results of Olken and Beathet al?É Chance!É Sample sizeÉ Treatment duration expectationsÉ Local accountability for councilsÉ Status of womenÉ Ability to strategically manipulate agenda
11 / 27
Political Economy Causal Inference
The Effects of Democracy
É So what do we learn about direct democracy?
É Are these ’real’ results; would you use them to informpolicy?
É Could they just reflect chance?
É How far do the results travel?
12 / 27
Political Economy Causal Inference
The Effects of Democracy
É So what do we learn about direct democracy?
É Are these ’real’ results; would you use them to informpolicy?
É Could they just reflect chance?
É How far do the results travel?
12 / 27
Political Economy Causal Inference
The Effects of Democracy
É So what do we learn about direct democracy?
É Are these ’real’ results; would you use them to informpolicy?
É Could they just reflect chance?
É How far do the results travel?
12 / 27
Political Economy Causal Inference
The Effects of Democracy
É So what do we learn about direct democracy?
É Are these ’real’ results; would you use them to informpolicy?
É Could they just reflect chance?
É How far do the results travel?
12 / 27
Political Economy Causal Inference
The Effects of Democracy
É What are the differences in Harding and Stasavage?É TreatmentÉ ControlÉ Treatment Assignment MechanismÉ Analysis MethodologyÉ PopulationÉ SampleÉ Outcome measures
13 / 27
Political Economy Causal Inference
The Effects of Democracy
É What is their evidence?É Attendance Fees + DemocracyÉ Remove fees election dateÉ Support for candidate who removed fees support for
removing fees
14 / 27
Political Economy Causal Inference
The Effects of Democracy
É How much do you trust the results of Harding andStasavage?
É Is Olken or Harding and Stasavage more useful forunderstanding the effects of democracy?
15 / 27
Political Economy Causal Inference
The Effects of Democracy
É How much do you trust the results of Harding andStasavage?
É Is Olken or Harding and Stasavage more useful forunderstanding the effects of democracy?
15 / 27
Political Economy Causal Inference
Field Experiments
É Field experiments provide confidence because treatmentassignment is controlled by the researcher
É But still take place in real-world environments, so theyidentify (hopefully) meaningful treatment effects
16 / 27
Political Economy Causal Inference
Field Experiments
É Why does randomization help us achieve causal inference?
É A treatment assignment mechanism that balances potentialoutcomesÉ Every unit has exactly the same probability of treatmentÉ No confoudingÉ No self-selectionÉ No reverse causation
17 / 27
Political Economy Causal Inference
Field Experiments
É Why does randomization help us achieve causal inference?É A treatment assignment mechanism that balances potential
outcomesÉ Every unit has exactly the same probability of treatmentÉ No confoudingÉ No self-selectionÉ No reverse causation
17 / 27
Political Economy Causal Inference
Field Experiments
É Why does randomization help us achieve causal inference?É We want to estimate:
E(Y1 − Y0) (1)
É Our data provides:
E(Y1|D = 1) , E(Y0|D = 0) (2)
É With randomization, Y1, Y0 ⊥ D:
E(Y1|D = 1) = E(Y1) (3)
E(Y0|D = 0) = E(Y0) (4)
E(Y1|D = 1) − E(Y0|D = 0) = E(Y1) − E(Y0) (5)
= E(Y1 − Y0) (6)
18 / 27
Political Economy Causal Inference
Field Experiments
É Why does randomization help us achieve causal inference?É We want to estimate:
E(Y1 − Y0) (1)
É Our data provides:
E(Y1|D = 1) , E(Y0|D = 0) (2)
É With randomization, Y1, Y0 ⊥ D:
E(Y1|D = 1) = E(Y1) (3)
E(Y0|D = 0) = E(Y0) (4)
E(Y1|D = 1) − E(Y0|D = 0) = E(Y1) − E(Y0) (5)
= E(Y1 − Y0) (6)
18 / 27
Political Economy Causal Inference
Field Experiments
É Why does randomization help us achieve causal inference?É We want to estimate:
E(Y1 − Y0) (1)
É Our data provides:
E(Y1|D = 1) , E(Y0|D = 0) (2)
É With randomization, Y1, Y0 ⊥ D:
E(Y1|D = 1) = E(Y1) (3)
E(Y0|D = 0) = E(Y0) (4)
E(Y1|D = 1) − E(Y0|D = 0) = E(Y1) − E(Y0) (5)
= E(Y1 − Y0) (6)
18 / 27
Political Economy Causal Inference
Field Experiments
É But these are just expectations (averages)
É On average, potential outcomes will be balancedÉ More likely in larger samplesÉ We cannot verify potential outcomesÉ But we can assess balance in observable covariatesÉ What if some covariates are imbalanced?
19 / 27
Political Economy Causal Inference
Field Experiments
É But these are just expectations (averages)É On average, potential outcomes will be balancedÉ More likely in larger samplesÉ We cannot verify potential outcomesÉ But we can assess balance in observable covariatesÉ What if some covariates are imbalanced?
19 / 27
Political Economy Causal Inference
Field Experiments
É Analysing field experimentsÉ Comparison of means: t-test to test significanceÉ Regression achieves the same thing
É Y ∼ α + βD + εÉ Y = Y0 + (Y1 − Y0)D + εÉ Just the conditional expectation function: E(Y |D = d)
É Include covariates if:É There is residual imbalanceÉ To increase precision of standard errors
20 / 27
Political Economy Causal Inference
Field Experiments
É Clustered treatment/samplingÉ Fixed effects: Adjust coefficients to remove variation in an
unhelpful dimensionÉ Eg. if we randomized within schools but not between them
É Clustered standard errors: Adjust standard errors forcorrelated errors, usually from clustered sampling ortreatmentÉ Eg. If we sampled one village in every municipalityÉ Or if treatment at the village level
21 / 27
Political Economy Causal Inference
Field Experiments
É AssumptionsÉ Compliance with randomization - Treatment was truly
random and acceptedÉ SUTVA - Treatment of one unit doesn’t affect potential
outcomes of other unitsÉ Excludability - Effects of treatment assignment operateonly through treatmentÉ Depends if these effects are part of the causal chain
22 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: Answerable Questions
É Small sample sizes still prevent inferenceÉ EthicsÉ Logistics/FinanceÉ Some treatments can’t be manipulated (history)É Lack of control over treatment content and context - is it
informative?É Long-term effects/adaptation?
23 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: Answerable QuestionsÉ Small sample sizes still prevent inferenceÉ EthicsÉ Logistics/FinanceÉ Some treatments can’t be manipulated (history)É Lack of control over treatment content and context - is it
informative?É Long-term effects/adaptation?
23 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: Internal Validity
É No guarantee of actual balance (and Inefficient if we alreadyknow confounders)
É Hawthorne effect: participants adapt behaviour inexperiments
É Biased measurement if not double-blind (non-excludability)É Average Treatment Effect can be skewed by OutliersÉ Always complications of non-compliance, SUTVA, attritionÉ Publication/Selection biasÉ Unbiased but imprecise; variation still high if lots of other
variables also affect YÉ Treatment assignment mechanism itself affects outcomes
24 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: Internal ValidityÉ No guarantee of actual balance (and Inefficient if we already
know confounders)É Hawthorne effect: participants adapt behaviour in
experimentsÉ Biased measurement if not double-blind (non-excludability)É Average Treatment Effect can be skewed by OutliersÉ Always complications of non-compliance, SUTVA, attritionÉ Publication/Selection biasÉ Unbiased but imprecise; variation still high if lots of other
variables also affect YÉ Treatment assignment mechanism itself affects outcomes
24 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: External Validity
É What theory are we testing? Can’t accumulate knowledgewithout theory. Causal mechanisms still a black box.
É What is our real population? Do we really have arepresentative sample?
É Limited portability of findings - context, population changemarginal treatment effects.É Eg. CCTs improve child health only where clinics are availableÉ Requires structural theory and background knowledgeÉ Average effects may not apply to any individual
É Naive application of policy implicationsÉ How much do the results depend on researcher oversight?
25 / 27
Political Economy Causal Inference
Field Experiments
É Limitations of Field Experiments: External ValidityÉ What theory are we testing? Can’t accumulate knowledge
without theory. Causal mechanisms still a black box.É What is our real population? Do we really have a
representative sample?É Limited portability of findings - context, population change
marginal treatment effects.É Eg. CCTs improve child health only where clinics are availableÉ Requires structural theory and background knowledgeÉ Average effects may not apply to any individual
É Naive application of policy implicationsÉ How much do the results depend on researcher oversight?
25 / 27
Political Economy Causal Inference
Field Experiments
É All these complications mean we need lots of assumptionsand background knowledge
É Just as with other methodologies
26 / 27
Political Economy Causal Inference
Field Experiments
É What is the purpose of field experiments?
É To inform policy choices?É To identify causal mechanisms?
É Are field experiments the gold standard?
27 / 27
Political Economy Causal Inference
Field Experiments
É What is the purpose of field experiments?É To inform policy choices?É To identify causal mechanisms?
É Are field experiments the gold standard?
27 / 27
Political Economy Causal Inference
Field Experiments
É What is the purpose of field experiments?É To inform policy choices?É To identify causal mechanisms?
É Are field experiments the gold standard?
27 / 27