deep iv: a flexible approach for counterfactual...
TRANSCRIPT
Deep IV: A Flexible Approach for Counterfactual Prediction
Jason Hartford 1 Greg Lewis 2 Kevin Leyton-Brown 1 Matt Taddy 2
AbstractCounterfactual prediction requires understandingcausal relationships between so-called treatmentand outcome variables. This paper provides arecipe for augmenting deep learning methods toaccurately characterize such relationships in thepresence of instrument variables (IVs)—sourcesof treatment randomization that are conditionallyindependent from the outcomes. Our IV specifi-cation resolves into two prediction tasks that canbe solved with deep neural nets: a first-stage net-work for treatment prediction and a second-stagenetwork whose loss function involves integrationover the conditional treatment distribution. ThisDeep IV framework1 allows us to take advantageof off-the-shelf supervised learning techniquesto estimate causal effects by adapting the lossfunction. Experiments show that it outperformsexisting machine learning approaches.
1. IntroductionSupervised machine learning (ML) provides many effec-tive methods for tasks in which a model is learned basedon samples from some data generating process (DGP) andthen makes predictions about new samples from the samedistribution. However, decision makers would often like topredict the effects of interventions into the DGP through pol-icy changes. Such changes impact the relationship betweeninputs and outcomes, making straightforward prediction ap-proaches inappropriate. In order to accurately answer suchcounterfactual questions it is necessary to model the struc-tural (or causal) relationship between policy (or “treatment”)and outcome variables.
For example, consider an airline that wants to use historical
1University of British Columbia, Canada 2Microsoft Research,New England, USA. Correspondence to: Jason Hartford <[email protected]>, Matt Taddy <[email protected]>.
Proceedings of the 34th International Conference on MachineLearning, Sydney, Australia, PMLR 70, 2017. Copyright 2017 bythe author(s).
1Implementation with all simulation experiments is availableat https://github.com/jhartford/DeepIV
data to optimize the prices it charges its customers: in thiscase, price is the treatment variable and the customer’s deci-sion about whether to buy a ticket is the outcome. There aretwo ways that a naive analysis could lead to incorrect coun-terfactual predictions. First, imagine that price varies in thetraining data because the airline gradually increases pricesas a plane fills. Around holidays, more people want to flyand hence planes become fuller leading to higher prices. So,in our training set we observe examples with high prices andhigh sales. A direct ML approach might incorrectly predictthat if the airline were to increase prices at other times in theyear they would also observe increased sales, whereas thetrue relationship between price and sales is surely negative.Typically we can observe holidays, and include them in themodel, so that we can correct for their effects. This casewhere an observable feature (holidays) is correlated withboth the outcome and treatment variable is called “selectionon observables”.
But say there is also sometimes high demand because ofconferences, which the airline is not able to observe. Thispresents a more challenging problem: even if price dis-tortions due to holidays were taken into account, a naiveanalysis could conclude that higher prices drive higher de-mand, whereas in fact high price is only correlated withhigh demand via the latent conference variable. This is themore challenging case of “selection on unobservables”.
The gold standard for establishing causal relationships isto conduct “AB” experiments, with subjects randomly as-signed to different values of the treatment variable. In theairline example, this would mean assigning passengers ran-dom prices that do not depend on the number of seats sold.Enough data collected in this way would make it straight-forward to identify the true relationship between price anddemand even if the latent variable were never observed.However, such a strategy would be exceedingly expensive(the airline would sometimes turn away interested passen-gers for nearly-empty flights and offer steep discounts fornearly-full flights). Depending on the timescale over whichsuch randomization were conducted, it could also hurt theairline’s long-term interests (because it could be perceivedas unfair) and could fail if passengers were able to hide theiridentities (e.g., by logging in from a different computer ifthey didn’t like the price).
Deep IV: A Flexible Approach for Counterfactual Prediction
The alternative is to work with observational data, but doingso requires explicit assumptions about the causal structure ofthe DGP (Bottou et al., 2013). Most recent approaches to us-ing machine learning methods such as trees (Wager & Athey,2015; Athey & Imbens, 2016) and deep networks (Johans-son et al., 2016; Shalit et al., 2016) for causal inference inobservational data leverage an “unconfoundedness” assump-tion that the treatment is conditionally independent of anylatent variables given the observed features. This amountsto assuming away selection on unobservables, which mayor may not be reasonable depending on the setting.
We can do without this assumption and get around bothtypes of selection if we can identify one or more instru-mental variables (IVs) that only affect treatment assignmentand not the outcome variable. In our airline example, thecost of fuel could be such an instrument: its variation isindependent of demand for airline tickets and it affects salesonly via ticket prices. Changes in the cost of fuel thus createmovement in ticket prices that is independent of our latentvariable, and this movement is as good as randomization forthe purposes of causal inference. See Figure 1 for an graph-ical illustration of this example and of the general class ofcausal graphs that we consider.
The IV framework has a long history, especially in eco-nomics (e.g., Wright, 1928; Reiersøl., 1945). It providesmethods for learning the regression function that relates thetreatment and response variables under the “interventionaldistribution” for DGPs that conform to the graphical modelshown in Figure 1 (Pearl, 2009). Most IV applications makeuse of a two-stage least squares procedure (2SLS; e.g., An-grist & Pischke, 2008) that applies a model of linear andhomogeneous treatment effects (e.g., all airline customersmust have the same price sensitivity). Nonparametric IVmethods from the econometrics literature relax these as-sumptions (e.g., Newey & Powell, 2003; Darolles et al.,2011). However, these methods typically work by modelingthe outcome as an unknown linear combination of a pre-specified set of basis functions of the treatment and othercovariates (e.g. Hermite polynomials, wavelets, or splines)and then modeling the conditional expectation of each ofthese basis functions in terms of the instruments (i.e., thenumber of parameters is quadratic in the number of basisfunctions). This requires a strong prior understanding ofthe DGP by the researcher; also, the complexity of bothspecification and estimation explodes when there are morethan a handful of inputs.
Advances in deep learning have demonstrated the power oflearning latent representations of complex features spaces(for recent surveys, see eg. LeCun et al., 2015; Schmid-huber, 2015). This paper’s goal is to use these powerfullearning algorithms for IV analysis. We do this by breakingIV analysis into two supervised stages that can each be tar-
geted with deep networks and that, when solved, allow usto make counterfactual claims and perform causal inference.Specifically, we first model the conditional distribution ofthe treatment variable given the instruments and covariatesand then target a loss function involving integration over theconditional treatment distribution from the first stage. Bothstages use deep neural nets trained via stochastic gradientdescent (Robbins & Monro, 1951; Bottou, 2010). We alsopresent an out-of-sample causal validation procedure forselecting hyper-parameters of the models on a validation set.We refer to this setup as the Deep IV framework.
Section 2 describes our general IV specification and its de-composition into two learning tasks. Section 3 outlinesneural network estimation for these tasks with particularattention paid to the SGD routine used in model trainingand our causal validation procedure. Section 4 presents ex-perimental results that illustrate the benefits of our methods.
2. Counterfactual predictionWe aim to predict the value of some outcome variable y (e.g.,sales in our airline example) under an intervention in a policyor treatment variable p (e.g., price). There exists a set ofobservable covariate features x (e.g., holidays) that we knowaffect both p and y. There also exist unobservable latentvariables e that may affect x, p and y (e.g., conferences).Counterfactual prediction aims to recover E[y|do(p), x] inthe context of the graphical model given by Figure 1, wherethe do(·) operator indicates that we have intervened to setthe value of the policy variable p (as per Pearl, 2009). Weassume the y is structurally determined by p, x and e as
y = g (p, x) + e. (1)
That is, g (·) is some unknown and potentially non-linearcontinuous function of both x and p, and we assume that thelatent variables (or “error”) e enter additively with uncondi-tional mean Ee = 0. We allow for errors that are potentiallycorrelated with the inputs: E[e|x, p] , 0 and, in particular,E[pe|x] , 0.
Define the counterfactual prediction function
h (p, x) ≡ g (p, x) +E[e|x], (2)
which is the conditional expectation of y given the observ-ables p and x, holding the distribution of e constant as p ischanged. Note that we condition only on x and not p in theterm E[e|x]; this term is typically nonzero, but it will remainconstant under arbitrary changes to our policy variable p.2
Thus h (p, x) is the structural equation that we estimate. It is
2It may be easier to think about a setting where e ⊥⊥ x, so thatthe latent error is simply defined as being due to factors orthogonalto the observable controls. In that case, h(p, x) = g (p, x). All ofour results apply in either setup.
Deep IV: A Flexible Approach for Counterfactual Prediction
holidays
conference
sales
fuel costs price
x e
yz p
Figure 1. (Left) Our air-travel demand example, with arrows representing causal effects. Price is the policy variable, sales is the outcome,and holidays are observable covariates. There is a big ‘conference’, unobserved to the policy-maker, that drives demand and (due tothe airline’s pricing algorithms) price. The instrument is the cost of fuel, which influences sales only via price. (Right) The generalstructure of the DGP under our IV specification; x represents observable features, p is our treatment variable of interest, z represents theinstruments, and latent effects e influence the outcome y additively.
useful because to evaluate policy options (e.g. changing theticket price from p0 to p1) we can look at the difference inoutcomes h (p1, x)−h (p0, x) = g (p1, x)− g (p0, x).
In standard supervised learning settings, the predictionmodel is trained to fitE[y|p, x]. This will typically be biasedagainst our structural objective because
E[y|p, x] = g (p, x) +E[e|p, x] , h (p, x) (3)
since our treatment is not independent of the latent errorsby assumption and hence E[e|p, x] , E[e|x]. This object isinappropriate for policy analysis as it will lead to biasedcounterfactuals:
E[y|p1, x]−E[y|p0, x]
= g (p1, x)− g (p0, x) +
(E[e|p1, x]−E[e|p0, x]
). (4)
In our airline example, high prices during conferences implythat E[e|p1, x]−E[e|p0, x] will be positive, resulting in theincorrect prediction that higher prices are associated withhigher sales if this bias is sufficiently large.
Fortunately, the presence of instruments allows us to esti-mate an unbiased h(p, x) that captures the structural rela-tionship between p and y. These are sets of variables z thatsatisfy the following three conditions.
Relevance F(p|x,z), the distribution of p given x and z, isnot constant in z.
Exclusion z does not enter Eq. (1)—i.e., z ⊥⊥ y | (x, p,e).
Unconfounded Instrument z is conditionally independentof the error—i.e., z ⊥⊥ e | x.3
3Under the additive error assumption made in Eq. (1), uncon-foundedness of the instrument is not necessary: we could replacethis assumption with the weaker mean independence assumptionE[e|x,z] = 0 without changing anything that follows. We use thestronger assumption to facilitate extensions, e.g. to estimatingcounterfactual quantiles. Our assumption is similar to the ‘un-confoundedness’ assumption in the Neyman–Rubin potential out-comes framework (Rosenbaum & Rubin, 1983) (i.e. p⊥⊥ e | x). Butour assumption is weaker—in particular, we allow for p 6⊥⊥ e|x—and so the matching and propensity-score re-weighting approachesoften used in that literature will not work here.
Taking the expectation of both sides of Equation (1) condi-tional on [x,z] and applying these assumptions establishesthe relationship (cf. Newey & Powell, 2003):
E[y|x,z] = E[ g (p, x)|x,z] +E[e|x]
=
∫h (p, x)dF(p|x,z), (5)
where, again, dF(p|x,z) is the conditional treatment distri-bution. The relationship in Equation (5) defines an inverseproblem for h in terms of two directly observable functions:E[y|x,z] and F(p|x,z). IV analysis typically splits this intotwo stages: first estimating F(p|xt,zt) ≈ F(p|xt,zt), and thenestimating h after replacing F with F.
Most existing approaches to IV analysis assume linear mod-els for the treatment density function F and the counterfac-tual prediction function h to solve Equation (5) in closedform. For example, the two-stage least-squares (2SLS) pro-cedure (e.g., Angrist et al., 1996) posits y = γp + xβy + eand p = τz+ xβp +v, with the assumptions that E[e|x,z] = 0,E[v|x,z] = 0, andE[ev] , 0 (which impliesE[ep] , 0). Thisprocedure is straightforward: fit a linear model for p givenx and z and use the predicted values p in a second linearmodel of y. This is a statistically efficient way to estimatethe effect of the policy variable (i.e. γ) as long as two strongassumptions hold: linearity (i.e., both first- and second-stageregressions are correctly specified) and homegeneity (i.e.,the policy affects all individuals in the same way).4
Flexible nonparametric extensions of 2SLS either replacethe linear regressions with a linear projection onto a seriesof known basis functions (Newey & Powell, 2003; Blundellet al., 2007; Chen & Pouzo, 2012) or use kernel-based meth-ods as in Hall & Horowitz (2005) and Darolles et al. (2011).This system of series estimators is an effective strategy forintroducing flexibility and heterogeneity with low dimen-sional inputs, but the approach faces the same limitations askernel methods in general: their performance depends onthe choice of kernel function; and they often become com-
4The estimated γ remains interpretable as a ‘local averagetreatment effect’ (LATE) under less stringent assumptions (seeAngrist et al., 1996, for an overview).
Deep IV: A Flexible Approach for Counterfactual Prediction
putationally intractable in high-dimensional feature spaces[x,z] or with large numbers of training examples.
3. Estimating and validating DeepIVWe now describe how to use deep networks to perform flex-ible, scalable IV analysis in a framework we call DeepIV.We make two contributions that are each necessary compo-nents of the approach. First, we propose a loss function andoptimization procedure that allows us to optimize deep net-works for counterfactual prediction. Second, we describe ageneral procedure for out-of-sample validation of two-stageinstrument variable methods. This allows us to performcausally valid hyper-parameter optimization, which in gen-eral is necessary for achieving good predictive performanceusing deep networks.
Our approach is conceptually simple given the counterfac-tual prediction framework described in Section 2. Ratherthan constraining ourselves to analytic solutions to the in-tegral in Equation (5), we instead directly optimize ourestimate of the structural equation, h . Specifically, to mini-mize `2 loss given n data points and given a function spaceH (which may not include the true h ), we solve
minh∈H
n∑t=1
(yt −
∫h (p, xt)dF(p|xt,zt)
)2
. (6)
Since the treatment distribution is unknown, we estimateF(p|x,z) in a separate first stage.
So the DeepIV procedure has two stages: a first stage den-sity estimation procedure to estimate F(p|x,z) and a secondthat optimizes the loss function described in Equation (6).In both stages hyper-parameters can be chosen to minimizethe respective loss functions on a held out validation set, andimprovements in performance against this metric will cor-relate with improvements on the true structural loss whichcannot be evaluated directly. We briefly discuss these twostages before describing our methods for optimizing the lossgiven in Equation (6) and our causal validation procedure.
First stage: Treatment network In the first stage welearn F(p|x,z) using an appropriately chosen distributionparameterized by a deep neural network (DNN), say F =
Fφ(p|x,z) where φ is the set of network parameters. Sincewe will be integrating over Fφ in the second stage, we mustfully specify this distribution.
In the case of discrete p, we model Fφ(p|x,z) as a categoricalCat (p | π(x,z;φ)) with p(p = pk) = πk(x,z;φ) for each treat-ment category pk and where πk(x,z;φ) is given by a DNNwith softmax output. For continuous treatment, we model Fas a mixture of Gaussian distributions where componentweights πk(x,z;θ) and parameters
[µk(x,z;φ),σk(x,z;φ)
]form the final layer of a neural network parametrized by
φ. This model is known as a mixture density network, asdetailed in §5.6 of Bishop (2006). With enough mixturecomponents it can approximate arbitrary smooth densities.To obtain mixed continuous–discrete distributions we re-place some mixture components with point masses. In eachcase, fitting Fφ is a standard supervised learning problem.
Second stage: Outcome network In the second stage,our counterfactual prediction function h is approximatedby a DNN with real-valued output, say hθ. We optimizenetwork parameters θ to minimize the integral loss functionin Equation (6) over training data D of size T = |D| from thejoint DGPD,
L(D;θ) = |D|−1∑
t
(yt −
∫hθ(p, xt)dFφ(p|xt,zt)
)2
. (7)
Note that this loss involves the estimated treatment distribu-tion function, Fφ, from our first stage.5
3.1. Optimization for DeepIV networks
We use stochastic gradient descent (SGD; see algorithmsin, e.g., Duchi et al., 2011; Kingma & Ba, 2014) to trainthe network weights. For Fφ, standard off-the-shelf meth-ods apply, but second stage optimization (for hθ) needs toaccount for the integral in Equation (7). SGD convergenceonly requires that each sampled gradient ∇θLt is unbiasedfor the population gradient, ∇θL(D;θ). Lower variance for∇θLt will tend to yield faster convergence (Zinkevich, 2003)while the computational efficiency of SGD on large datasetsrequires limiting the number of operations going into eachgradient calculation (Bousquet & Bottou, 2008).
We can approximate the integral with respect to a probabil-ity measure with the average of draws from the associatedprobability distribution:
∫h(p)dF(p) ≈ B−1 ∑
b h(pb) for
{pb}B1
iid∼ F. Hence we can get an unbiased estimate of Equa-
tion (7) by replacing the integral with a sum over samplesfrom our fitted treatment distribution function, Fφ:
L(D;θ)≈ |D|−1∑
t
yt −1B
∑p∼Fφ(p|xt ,zt)
hθ( p, xt)
2
:= L(D;θ).
(8)This equation can be used to estimate ∇θLwith an importantcaveat: if we want to maintain unbiased gradient estimates,independent samples must be used for each instance of theintegral in the gradient calculation. To see this, note that the
5 We can replace Eq. (7) with other functions, e.g., a softmaxfor categorical outcomes, but use `2 loss for most of our exposition.
Deep IV: A Flexible Approach for Counterfactual Prediction
gradient of Equation (8) has expectation
ED∇θLt = −2ED(EFφ(p|xt ,zt)
[yt −hθ(pk, xt)
]·EFφ(p|xt ,zt)
[h ′θ(pk, xt)
])(9)
, −2EDEFφ(p|xt ,zt)[(
yt −hθ(pk, xt))h ′θ(pk, xt)
],
where the inequality holds so long ascovFφ(p|xt ,zt)
[(yt −hθ(pk, xt)
)h ′θ(pk, xt)
], 0. We thus
need a gradient estimate based on unbiased MC estimatesfor each EFφ(p|xt ,zt) term in Equation (9). We obtain such an
estimate by taking two samples { pb}B1 , { pb}
B1
iid∼ Fφ(p|xt,zt)
and calculating the gradient as
∇BθLt ≡ −2
yt −B−1∑
b
hθ( pb, xt)
B−1∑
b
h ′θ( pb, xt). (10)
Independence of the two samples ensures that E∇BθLt =
ED∇θLt = ∇θL(D;θ), as desired. The variance of our es-timate depends on B, the number of samples that we draw.Each of these samples is relatively expensive to computebecause they require a forward pass through the networkhθ(p, xt). If this varies significantly with p we might need alarge number of samples to get a low-variance estimate ofthe gradient, which is computationally intensive.
An alternative is to optimize an upper bound on Equation (8).By using Jensen’s inequality and the fact that the squarederror function is convex we get that
L(D;θ) ≤ |D|−1∑
t
∑p∼Fφ(p|xt ,zt)
(yt −hθ( p, xt)
)2. (11)
Taking the RHS of Equation (11) as the objective and cal-culating the gradient leads to a version of Equation (10) inwhich a single draw can be used instead of two independentdraws. This objective is easy to implement in practice asit just involves drawing samples during training. This iswell-supported in deep network implementations because itis analogous to the data augmentation procedures that arecommonly used to encourage invariance in deep networks.The analogy is more than just aesthetic—by optimizing thisloss, we are essentially encouraging the network to be invari-ant to variations in the treatment that cannot be explained byour features and instrument. This encourages the networkto ignore the effects of unobserved confounding variables.
However, we do not have theoretical guarantees that optimiz-ing this upper bound onL(D;θ) leads to good counterfactualperformance. While it may converge more quickly becauseit exhibits lower variance, it will have worse asymptotic per-formance as it only approximates the desired loss function.We evaluated this tradeoff experimentally by comparingoptimizing the upper bound to the more computationally
expensive unbiased procedure. In our simulations, we foundthat upper bound loss tended to give better performanceunder practical computational limitations.
Discrete treatment and outcome spaces The discussionthus far has focused on continuous treatment and outcomespaces because they are more challenging mathematically.When the treatment space is discrete and low dimensional,Fφ is modeled as a categorical response so the gradient ofEquation (7) can be expressed exactly as
∇θLt = −2
yt −∑
k
πk(xt,zt;φ)hθ(pk, xt)
·∑
k
πk(xt,zt;φ)h ′θ(pk, xt). (12)
Thus when the outcome space is discrete, we also have lessto worry about with respect to bias in gradient updates. Fordiscrete outcomes, we use the softmax loss and can usesingle-sample MC gradient estimates without introducingbias or large amounts of variance because the gradient withrespect to the softmax loss does not involve a product ofrandom variables.
3.2. Causal validation
There is a widespread belief that “[s]tandard methods forhyperparameter selection, such as cross-validation, are notapplicable when there are no samples of the counterfactualoutcome” (Shalit et al., 2016). This would be a signifi-cant problem in our setting because out-of-sample (OOS)validation procedures are crucial for tuning deep networkhyper-parameters and optimization rules. Fortunately, bothsteps in our Deep IV procedure can be validated by simplyevaluating the respective losses on held out data. Considera held-out dataset Dh-o. Our first stage, fitting Fφ, is astandard density estimation problem and can be tuned tominimize the OOS deviance criterion
minφ
∑dl∈Dh-o
− log fφ(pl|xl,zl), (13)
where fφ is either the probability mass function or densityfunction associated with Fφ, as appropriate. Second stagevalidation proceeds conditional upon a fitted Fφ, and weseek to minimize the held-out loss criterion
minθ
∑dl∈Dh-o
(yl−
∫hθ(p, xl)dFφ(p|xl,zl)
)2
. (14)
The integral here can either be exact or MC approximate viasampling from Fφ.
Each stage is evaluated in turn, with second stage validationusing the best-possible network as selected in the first stage.
Deep IV: A Flexible Approach for Counterfactual Prediction
This procedure guards against the ‘weak instruments’ bias(Bound et al., 1995) that can occur when the instrumentsare only weakly correlated with the policy variable. Tosee why, consider the worst case where the instruments areindependent of the policy variable, i.e., F(p|z) = F(p). With-out validation, a sufficiently powerful model will perfectlyoverfit by approximating the conditional distribution F(p|zi)with a point mass at pi. As a result, the causal loss functionwill approximate the standard prediction loss, leading tospurious conclusions. By contrast, the expected first-stageloss minimizer on the validation set is the model that bestapproximates the unconditional distribution F(p), and giventhat, the second stage minimizer best approximates g(p) = ywhich predicts no relationship when there is no evidence infavor of it. That said, it should be noted that these criteriaprovide relative performance guidance: improving on eachcriterion will improve performance on counterfactual pre-diction problems, but without giving any information abouthow far hθ(p, x) is from true h (p, x).
Our causal validation procedure is sequential: stage two’svalidation depends on the model chosen in the first stage.This greedy procedure is computationally efficient andcausally valid, but potentially suboptimal because the first-stage loss is not of independent interest. Peysakhovich &Eckles (2017) concurrently developed a casual validationframework that considers both stages jointly in domains withcategorical instruments and no observable features. Theseassumptions are too restrictive for the problems we consider,but it would be interesting to investigate whether there existsa joint optimization approach that offers practical benefitsunder more permissive assumptions.
4. ExperimentsWe evaluated our approach on both simulated and realdata. We used simulations to assess DeepIV’s ability torecover an underlying counterfactual function both in a low-dimensional domain with informative features and in a high-dimensional domain with features consisting of pixels of ahandwritten image. We compared our approach to 2SLS andto a standard feed-forward network, evaluated the effective-ness of hyper-parameter optimization, and contrasted ourbiased and unbiased loss functions with various numbersof samples underlying the SGD step. We also considereda real-world dataset where ground truth was not available,showing that we could replicate the findings of a previousstudy in a dramatically more automatic fashion.
4.1. Simulations
Our simulation models a richer version of the airline ex-ample described in Section 1. We assume that there are 7customer types s ∈ {1, ...,7} that each exhibit different lev-els of price sensitivity. We model the holiday effect on
sales by letting the customer’s price sensitivity vary continu-ously throughout the year according to a complex non-linearfunction, ψt = 2
((t−5)4/600 + exp
[−4(t−5)2
]+ t/10−2
).
The time of year t is an observed variable, generated ast ∼ unif(0,10). Prices are a function of ψt and of the fuelprice z, with the motivation that they are chosen strategicallyby the airline in order to move with average price sensitivity.In our example, the high demand that results from con-ferences breaks the conditional independence between ourtreatment variable p and the latent effects e, thereby vio-lating the “unconfoundedness” assumption. We model thisabstractly by generating our latent errors e with a parameterρ that allows us to smoothly vary the correlation between pand e. Sales y are then generated as
y = 100 + (10 + p)sψt −2p + e, p = 25 + (z + 3)ψt + v
z, v ∼ N(0,1) and e ∼ N(ρv,1−ρ2).
Our target counterfactual function is h(t, s, p) = (10 +
p)sψt − 2p. To evaluate the model, we consider the coun-terfactual question, “What would sales have been if priceshad been changed to p′?” Thus the price in our test set is setdeterministically over a fixed grid of price values that spansthe range of training set prices. The observed features [t, s]are sampled as in the original data generating process andwe compare estimated h against the ground truth h .
Low dimensional domain We evaluated structural meansquare error (MSE) while varying both the number of train-ing examples and ρ, the correlation between e and p. Inaddition to Deep IV, we considered a regular feed-forwardnetwork (FFNet) with the same architecture as our outcomenetwork, a non-parametric IV polynomial kernel regres-sion (NonPar, Darolles et al., 2011) using Hayfield et al.(2008)’s R implementation, and standard two-stage leastsquares (2SLS). Full details of model architectures and hy-perparameter choices for all the models are given in theAppendix.
The results are summarized in Figure 2. The performance ofNonPar, of 2SLS, and of our Deep IV model was mostly un-affected by changes in ρ, reflecting the fact that these modelsare designed to be resilient to unobserved confounders. For1000 data points, NonPar’s mean performance was betterthan 2SLS but failed to match DeepIV. Because of Non-Par’s excessive computational requirements we were notable to fit it to the larger datasets. 2SLS is constrained byits homogeneity and linearity assumptions, and so did notimprove with increasing amounts of data. Adding regu-larized polynomial basis functions to 2SLS (2SLS(poly))gives some empirical improvements in performance over2SLS on larger datasets but the procedure is not causallyvalid because it violates 2SLS’s linearity assumptions. Bothforms of 2SLS performed far better than FFNet which did a
Deep IV: A Flexible Approach for Counterfactual Prediction
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
ρ = 0.9
●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
ρ = 0.75
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
ρ = 0.5
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
ρ = 0.25
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
ρ = 0.1
●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
●
●
1 5 10 20
0.02
0.10
0.50
2.00
10.0
0
Training Sample in 1000s
Out
−of
−S
ampl
e M
SE
(lo
g sc
ale)
FFNet2SLS2SLS(poly)NonParDeepIV
Figure 2. Out-of-sample predictive performance for different levels of endogeneity (ρ). Note that the test samples were generated withindependent errors conditional upon a fixed grid of price values, breaking the endogeneity that existed in the training sample; this is whythe feed-forward network did so poorly. Each model was fitted on 40 random samples from the DGP for each sample size and ρ-level.
●
●●
●
●
●
●
●
●
●
●
●
●●
●
●●●
●●
●
●●
●
●
●
●
●
●
●
●
●
●
●●
●●
●●
●
●
●
●
●
●
●●
●●
●
●
●
●
●
●
●
●
●
●
●
●
●
●
●●
2.6 2.8 3.0 3.2 3.4
0.2
0.4
0.6
0.8
1.0
1.2
Cou
nter
fact
ual l
oss
(MS
E)
Validation loss (MSE)
●
●
●
●
●
●
●
1e−045e−040.0010.0050.010.050.1
●
1 5 10 20
0.05
0.20
0.50
2.00
5.00
20.0
0
●●
●
●●●●
●●
●
●●
●
Training Samples in 1000s
Out
−of
−S
ampl
e C
ount
erfa
ctua
l MS
E
Controlled ExperimentDeepIV2SLSNaive deep net
Figure 3. For the high-dimensional feature space problem we useda four-layer convolutional network to build an embedding of theimage features which was concatenated with the observed timefeatures and the instrument (first stage) and the treatment samples(second stage) and fed the resulting vector through another hiddenlayer before the output layer. (Left) Grid search over L2 anddropout parameters for the embedding used in the convolutionnetwork. (Right) Performance on an image experiment.
good job of estimating h (t, s, p)+E[e|p] but a terrible job ofrecovering the true counterfactual. As ρ dropped, decreas-ing E[e|p], FFNet’s performance improved but even withlow levels of correlation between p and e it remained farworse than simple 2SLS. This occurred because we eval-uated the models with respect to a fixed grid of treatmentvalues which induced a covariate shift at test time. In con-trast, Deep IV was the best performing model throughoutand its performance improved as the amount of data grew.
High dimensional feature space In real applications, wedo not typically get to observe variables like customer typethat cleanly delineate our training examples into explicit
classes, but may instead observe a large number of featuresthat correlate with such types. To simulate this, we replacedthe customer type label s ∈ {0,1, ...,7} with the pixels of thecorresponding handwritten digit from the MNIST dataset(LeCun & Cortes, 2010). The task remained the same, butthe model was no longer explicitly told that there were 7customer types and instead had to infer the relationshipbetween the image data and the outcome.
In this far more challenging domain, performance is sensi-tive to the choice of hyper-parameters, necessitating opti-mization on a validation set. Figure 3 shows an evaluation ofthe appropriateness of our loss function for hyper-parametertuning, comparing our validation-set loss after grid searchover Dropout and L2-regularization parameters to test setloss. We found a clear linear relationship between the losses;the best performing validation set model was among the bestfive performing models under the true causal loss.
We can get an upper bound on the performance of a partic-ular model architecture by comparing its performance tothe same architecture trained on data from a simulated ran-domized experiment on the same data-generating process.We simulated this by generating the outcome y with inde-pendent noise e and by generating p uniformly at randomover its support. Thus the controlled model had to solvea standard supervised learning problem where errors weregenerated independently and there was no test-time covari-ate shift. As before, the naive deep network also sharedthe same architecture in addition to taking the instrumentas input. This experiment showed that DeepIV was able tomake up most of the loss in counterfactual prediction perfor-mance that the naive network suffered by not accounting forthe causal prediction problem. However, there was still agap in performance: with 20 000 data points the controlledexperiment achieved an average mean squared error of 0.20while DeepIV managed 0.32.
Deep IV: A Flexible Approach for Counterfactual Prediction
Loss Function # Samples Mean Stdev
Upper bound 1 0.32 0.085Unbiased 2 0.48 0.107Unbiased 4 0.50 0.158Unbiased 8 0.44 0.100Unbiased 16 0.39 0.098
Table 1. Comparing loss functions on the high-dimensional imagetask. Bold indicates best performing models. Although the upperbound loss offered better mean performance, the two results arenot statistically significantly different.
Performance of the unbiased loss relative to the upperbound loss We tested how our two approaches to op-timization affected performance on the image task with20 000 training examples. The results are summarized inTable 1. The upper bound loss loss gave significantly betterperformance than all but the unbiased loss version basedon 16 samples, without requiring multiple passes throughthe network to evaluate the gradient. Thus, the bias intro-duced by the upper bound did not meaningfully degradecounterfactual performance in our experiments.
4.2. Application: Search-advertisement position effects
Our experiments so far have considered synthetic data. Wenow evaluate the utility of our approach on real data forwhich we do not have access to ground truth. This meansthat we cannot evaluate models in terms of their predictions;instead, we show that we can replicate the results of a pre-viously published study in a dramatically more automatedfashion. Specifically, we examine how advertiser’s positionon the Bing search page (their “slot”) affects the probabilityof a user click, allowing for different treatment effects fordifferent advertiser-query pairs. For example, we aim to de-tect differences in the importance of ad position when Cokebids on the word “Coke” (an “on-brand query”) versus whenPepsi bids on “Coke” (an “off-brand query”) versus whenCoke bids on “www.coke.com” (an “on-nav query”, occur-ring when a user types a url in the search box by mistake)versus when Pepsi bids on “www.coke.com” (an “off-navquery”). This question was studied extensively by Gold-man & Rao (2014) using a nonparametric IV estimationapproach that involved a detailed construction of optimalinstruments, as well as a separate hand-coded classificationof advertiser–query pairs into the four categories above.
Our goal is to replicate these results in an automated fashion.Advertiser position is correlated with latent user intent (e.g.when a user searches for “Coke”, it is likely both that theywill click and that “Coke” will be the top advertiser), so weneed instruments to infer causation. The instruments pro-posed by Goldman & Rao (2014) are a series of indicatorsfor experiments run by Bing in which advertiser–query pairs
200 400 600 800 1000
2030
4050
6070
on_brand off_brand
200 400 600 800 1000
4050
6070
8090 on_nav off_nav
website US rank
rela
tive
clic
k ra
te
Figure 4. The inter-quartile range in advertiser-query specific esti-mates of the relative drop in click rate from moving from position1 to position 2 (i.e. y-axis denotes cr1−cr2
cr1%), as the popularity of
the advertiser varies along the x-axis (as measured by visit rank onAlexa.com), for on-brand versus off-brand queries (left panel) andon-nav versus off-nav queries (right panel) for a single combinationof advertiser and search-query text.
were randomly assigned to different algorithms that in turnscored advertisers differently in search auctions, resultingin random variation in position. Our estimation algorithmtakes the experiment ID directly as an instrument.
As features, we gave the deep nets the query url and userquery as text. The url was parsed into tokens on dashesand dots and these tokens were then parsed on punctuationand whitespace. Given the outcome variable (an indica-tor for user click), the features, and the instruments, weapplied our methodology directly to the data without anyof the additional feature engineering and construction ofoptimal instruments performed by Goldman & Rao (2014).This approach was tractable despite the fact that the datasetcontained over 20 million observations. Figure 4 showsthe results. We were able to replicate the original study’sbroad findings, namely that (i) that for “on-brand” queries,position was worth more to small websites; and (ii) that thevalue of position for on-nav queries was much smaller thanfor off-nav queries. A naive non-causal regression found anunrealistic average treatment effect (ATE; across sampledadvertisers and queries) drop in click rate of 70% from 1stto 2nd position; in contrast, the causal estimate of the ATEwas a more modest 12% drop.
5. DiscussionWe have presented DeepIV, an approach that leverages in-strument variables to train deep networks that directly min-imize the counterfactual prediction error and validate theresulting models on held-out data. DeepIV significantlyreduced counterfactual error measured in simulation ex-periments and was able to replicate previous IV experi-ments without extensive feature engineering. In future work,we plan to discuss interference techniques for the DeepIVframework and explore how this approach generalizes toother causal graphs given appropriate assumptions.
Deep IV: A Flexible Approach for Counterfactual Prediction
AcknowledgementsWe would like to thank Susan Athey, Xiaohong Chen andDemian Pouzo for their helpful discussions and the anony-mous reviewers for their useful comments on the paper. Wewould also like to thank Holger Hoos for the use of the Adacluster, without which the experiments would not have beenpossible.
ReferencesAngrist, J. D., Imbens, G.W., and Rubin, D. B. Identification
of causal effects using instrumental variables. Journal ofthe American Statistical Association, 91(434):444–455,1996.
Angrist, Joshua D and Pischke, Jorn-Steffen. Mostly harm-less econometrics: An empiricist’s companion. Princetonuniversity press, 2008.
Athey, Susan and Imbens, Guido. Recursive partitioning forheterogeneous causal effects. Proceedings of the NationalAcademy of Sciences, 113:7353–7360, 2016.
Bishop, Christopher M. Pattern Recognition and MachineLearning. Springer, 2006.
Blundell, Richard, Chen, Xiaohong, and Kristensen, Den-nis. Semi-nonparametric iv estimation of shape-invariantengel curves. Econometrica, 75:1630–1669, 2007.
Bottou, L. Large-scale machine learning with stochasticgradient descent. In Proceedings of COMPSTAT’2010,pp. 177–186. Springer, 2010.
Bottou, Leon, Peters, Jonas, Quinonero-Candela, Joaquin,Charles, Denis X, Chickering, D Max, Portugaly, Elon,Ray, Dipankar, Simard, Patrice, and Snelson, Ed. Coun-terfactual reasoning and learning systems: The exampleof computational advertising. The Journal of MachineLearning Research, 14(1):3207–3260, 2013.
Bound, John, Jaeger, David A, and Baker, Regina M. Prob-lems with instrumental variables estimation when thecorrelation between the instruments and the endogenousexplanatory variable is weak. Journal of the AmericanStatistical Association, 90:443–450, 1995.
Bousquet, Olivier and Bottou, Leon. The tradeoffs of largescale learning. In Advances in neural information pro-cessing systems (NIPS), pp. 161–168, 2008.
Chen, X. and Pouzo, D. Estimation of nonparametric condi-tional moment models with possibly nonsmooth general-ized residuals. Econometrica, 80:277–321, 2012.
Darolles, Serge, Fan, Yanqin, Florens, Jean-Pierre, andRenault, Eric. Nonparametric instrumental regression.Econometrica, 79:1541–1565, 2011.
Duchi, John, Hazan, Elad, and Singer, Yoram. Adaptivesubgradient methods for online learning and stochasticoptimization. Journal of Machine Learning Research, 12:2121–2159, 2011.
Goldman, Mathew and Rao, Justin M. Experiments asinstruments: heterogeneous position effects in sponsoredsearch auctions. 2014.
Hall, Peter and Horowitz, Joel L. Nonparametric methodsfor inference in the presence of instrumental variables.Ann. Statist., 33(6):2904–2929, 12 2005.
Hayfield, Tristen, Racine, Jeffrey S, et al. Nonparametriceconometrics: The np package. Journal of statisticalsoftware, 27(5):1–32, 2008.
Johansson, F. D., Shalit, U., and Sontag, D. Learning repre-sentations for counterfactual inference. In Proceedings ofthe 33nd International Conference on Machine Learning,(ICML), pp. 3020–3029, 2016.
Kingma, Diederik and Ba, Jimmy. ADAM: A method forstochastic optimization. International Conference onLearning Representations (ICLR), 2014.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. Na-ture, 2015.
LeCun, Yann and Cortes, Corinna. MNIST handwrittendigit database. 2010. URL http://yann.lecun.com/exdb/mnist/.
Newey, W. K. and Powell, J. L. Instrumental variable esti-mation of nonparametric models. Econometrica, 71(5):1565–1578, 2003.
Pearl, J. Causality. Cambridge University Press, 2009.
Peysakhovich, A. and Eckles, D. Learning causal effectsfrom many randomized experiments using regularizedinstrumental variables. ArXiv e-prints, January 2017.
Reiersøl., O. Confluence analysis by means of instrumentalsets of variables. Arkiv for Matematik, Astronomi ochFysik, 32a(4):1–119, 1945.
Robbins, H. and Monro, S. A stochastic approximationmethod. The Annals of Mathematical Statistics, pp. 400–407, 1951.
Rosenbaum, P. R. and Rubin, D. B. Assessing sensitivity toan unobserved binary covariate in an observational studywith binary outcome. Journal of the Royal StatisticalSociety. Series B (Methodological), 45(2):212–218, 1983.
Schmidhuber, J. Deep learning in neural networks: Anoverview. Neural Networks, 2015.
Deep IV: A Flexible Approach for Counterfactual Prediction
Shalit, U., Johansson, F., and Sontag, D. Estimating indi-vidual treatment effect: generalization bounds and algo-rithms. ArXiv e-prints, June 2016.
Wager, Stefan and Athey, Susan. Inference of het-erogeneous treatment effects using random forests.arXiv:1510.04342, 2015.
Wright, P. G. The Tariff on Animal and Vegetable Oils.Macmillan, 1928.
Zinkevich, Martin. Online convex programming and gener-alized infinitesimal gradient ascent. In Proceedings of theTwentieth International Conference on Machine Learning(ICML), 2003.