epse 581c: causal inference for applied researchers · in observational or quasi-experimental...

EPSE 581C: Causal Inference for Applied Researchers

Ed Kroc

University of British Columbia

[email protected]

June 10, 2019

Ed Kroc (UBC) Causal Inference June 10, 2019 1 / 39

Last time

Matching to balance confounders over treatment groups

Propensity score matching


Today

Propensity score matching

implementation

limitations

case studies


HW1 feedback

Interpreting regression output with marginal curvature

Regression models the average response, not the individual response


The stable unit treatment value assumption (SUTVA)

One key assumption of the NRCM is called the stable unit treatmentvalue assumption (SUTVA):

The counterfactual (potential outcome) of one sample unit should beunaffected by the particular assignment of treatments to the othersample units.

More simply, whatever treatment one sample unit receives should notaffect the outcome of whatever treatment another sample unit receives.

Such an assumption should hold in a tightly controlled experiment,e.g. where sample units are not allowed to interact with each other.

But the SUTVA assumption may fail if sample units are not isolated.


The assignment to treatment mechanism

We have already argued that randomization of treatment allows foran unconfounded estimate of the ACE (assuming patients stay ontheir assigned treatment - we will later see a technique that relaxesthis assumption: instrumental variables).

We have also seen that a deterministic assignment of treatment canallow for an unconfounded estimate of the ACE locally (RD design).

In practice, true randomization can be very difficult to achieve:blinding of patients and researchers/doctors helps.


Matching

In observational or quasi-experimental research designs, we usually do nothave the ability to assign treatment at all; we simply observe outcomesbased on assignment to treatment mechanisms that we cannotcontrol/manipulate (e.g. self-selection to treatment, or comorbidities(covariates) increasing likelihood of assignment to treatment).

The most widely used way to attempt to correct for this confoundingof the assignment to treatment mechanism is by some kind ofmatching of sample units.

Generally speaking, the idea is to match up sample units from onetreatment group to another based on how similar they are over allmeasured covariates.

Matched sample units of different treatments can then mimiccounterfactuals, assuming no omitted confounders.


Matching: problems

Three fundamental problems of matching:

(1) Can never measure all possible confounders, so always imperfectcorrections.

Issue (1) can never be addressed by matching.

(2) Matching on many covariates simultaneously requires a lot of data(curse of dimensionality).

Propensity score matching addresses problem (2), but relies on aregression framework that is susceptible to all the usual issues withmodel misspecification.

(3) Matching on multiple covariates requires that we observe enoughsample units in each possible subcategory/strata so that we can finda “match”.

Fix that’s not a fix for (3): restrict inferences to subgroups whereenough data exist.


Matching to minimize confounding of causal effects

Consider the following example:

We want to assess the causal effect of a new tutoring service beingoffered free to all students in MATH 105, Winter Session II, on finalcourse grade.

Graduate students in MATH are trained via a series of pedagogicalworkshops. Students can then book one thirty-minute appointmentvia an online system with one of these tutors anytime from 9 AM to 4PM, the week before an exam.

Problem: self-enrolment in treatment (tutor) group - meansassignment to treatment mechanism will be confounded.

Problem: SUTVA violations - one student may meet with a tutor thenpeer-tutor other friends in the course; one student may not be able toset a tutor session because of too many other students already usingthe tutors.Ed Kroc (UBC) Causal Inference June 10, 2019 9 / 39


Ignore the SUTVA violations for the sake of this example (notreasonable in practice!)

How could we deal with the self-enrolment in treatment issue?

ANSWER: try matching students from the treatment (tutor) groupwith students who do not come in for this tutoring.

The idea here is to try to create subgroups of students that areexchangeable over treatment assignment, given their similar values ofrelevant covariates.



Student Sex Faculty AgeGPA from

previous termTutor?

Finalgrade

1 F SCIE 19 3.5 1 902 F SCIE 19 3.6 1 883 F SCIE 21 3.7 0 854 M SCIE 18 3.0 0 825 F SCIE 20 3.5 0 826 M SCIE 19 3.7 0 837 M SCIE 19 3.5 1 908 F ARTS 18 3.8 0 959 M SCIE 21 3.5 1 9410 M SCIE 20 3.6 0 9411 F ARTS 19 2.8 0 75...

......

......

......

113 M SCIE 20 3.1 0 85



Colours indicate student “matches” from T “ 1 with students fromT “ 0 over all covariates.

Assuming no omitted confounders, we could now treat thesestudents as exchangeable over treatment assignment: i.e. we observe:

Y1p1q, Y2p1q, Y3p0q, Y5p0q,

but can consider the assignment to treatment mechanism as randomover the subgroup:

Sex “ F , Faculty “ SCIE , Age « 20, GPA « 3.6.

Thus, can estimate the ACE pT | F , SCIE , 20, 3.6q via

pEpY | T “ 1, F ,SCIE , 20, 3.6q ´ pEpY | T “ 0, F ,SCIE , 20, 3.6q

“ 89´ 83.5 “ 5.5



If we could create other matched subgroups, then we could estimatethe

ACE pT | Sex , Faculty , Age, GPAq

for other values of the covariates, assuming no omittedconfounders.

Let’s do this for the subgroup:

Sex “ M, Faculty “ SCIE , Age « 20, GPA « 3.6.




previous termTutor?

Finalgrade

1 F SCIE 19 3.5 1 902 F SCIE 19 3.6 1 883 F SCIE 21 3.7 0 854 M SCIE 18 3.0 0 825 F SCIE 20 3.5 0 826 M SCIE 19 3.7 0 837 M SCIE 19 3.5 1 908 F ARTS 18 3.8 0 959 M SCIE 21 3.5 1 9410 M SCIE 20 3.6 0 9411 F ARTS 19 2.8 0 75...

......

......

......

113 M SCIE 20 3.1 0 85



Assuming no omitted confounders, we could now treat these(green) students as exchangeable over treatment assignment: i.e. weobserve:

Y6p0q, Y7p1q, Y9p1q, Y10p0q,

but can consider the assignment to treatment mechanism as randomover the subgroup:

Sex “ M, Faculty “ SCIE , Age « 20, GPA « 3.6.

Thus, can estimate the ACE pT | M, SCIE , 20, 3.6q via

pEpY | T “ 1, M,SCIE , 20, 3.6q ´ pEpY | T “ 0, M, SCIE , 20, 3.6q

“ 92´ 88.5 “ 3.5



Notice: we now have an estimate of the ACE pT q for Faculty = SCIE,Age « 20, GPA « 3.6, and for both sexes.

Now we can use properties of averages (double expectation formula)to estimate the ACE pT q unconditional on Sex (still conditional onFaculty, Age, GPA).

Double expectation formula:

EpY q “ ErEpY | X qs

“ÿ

x

EpY | X “ xq ¨ PrpX “ xq

Saw the algebra last time. Can then remove the dependency of ourestimate on covariates via this procedure.

Lots of software exists that will do this kind of matching for youautomatically (e.g. R ‘MatchIt’).



Several problems with practical implementation of this:

(1) What if we can’t find a match?

(2) What if we have too many covariates to match on, thereby creatingsome sample units that can’t be matched?

(3) For non-categorical variables, how do we decide what a ‘match’ is?E.g. we equated ages of 19 to 21, and GPAs of 3.5 to 3.7. Is thisreasonable? Optimal?

LOTS of work has been done to try to answer these questions.

To address (3), many different matching algorithms exist to find“best” matches. (No one algorithm is always best.)

To address (2), very common to use propensity scores.


Propensity scores

What are propensity scores? (Definitive source: Rosenbaum & Rubin,1984, JASA)

Estimating the ACE pT q requires (at least) two mathematicalproperties to hold:

(1) The assignment to treatment mechanism is unconfounded;i.e. PrpT “ 1q and PrpT “ 0q do not depend on any other variables,measured or unmeasured.

(2) Finding estimates of EpY | T “ 1q and EpY | T “ 0q to thensubstitute for the counterfactual averages EpYi p1qq and EpYi p0qq(justified if assignment to treatment mechanism is unconfounded).

Recall:ACE pT q “ EpYi p1qq ´ EpYi p0qq


Propensity scores

The propensity score is defined as the conditional probability that asample unit with observed covariates X will be assigned to treatment,T “ 1:

spX q :“ PrpT “ 1 | X q

Claim: if sample units all have about the same propensity score spX q,then the distribution of X is about the same for both treatmentgroups.

That is, sample units with similar propensity scores are(approximately) exchangeable over treatment, assuming nounobserved confounders.

Thus, propensity scores are a way of removing confounding effects ofthe observed covariates on the assignment to treatment mechanism.


Propensity scores

The propensity score is defined as the conditional probability that asample unit with observed covariates X will be assigned to treatment,T “ 1:

spX q :“ PrpT “ 1 | X q

Claim: if sample units all have about the same propensity score spX q,then the distribution of X is about the same for both treatmentgroups.

Notice: this claim is equivalent to the statement that X and T areconditionally independent given s “ spX q:

PrpX ,T | sq “ PrpX | sqPrpT | sq

Saw proof of this last time.


Propensity scores

Propensity scores solve the problem of having too many covariates tomatch on: instead of matching on all covariates simultaneously,simply estimate the propensity score and then match sample unitsbased on this single value.

Two immediate questions arise:

(1) How do we estimate a propensity score from sample data?

(2) What is the best way to match units over their propensity scores (acontinuous quantity)?


Estimating propensity scores

(1) How do we estimate a propensity score from sample data?

ANSWER: propose a regression model relating assignment totreatment to the observed covariates.

Assignment to treatment is a binary variable, and the propensity scoreis the probability of assignment to treatment (real number between 0and 1), given the observed covariates.

How do we specify a regression model for a binary response variable?

ANSWER (most commonly): logistic regression (primer last time)



Goal: estimate spX q “ PrpT “ 1 | X q.

Solution: logistic regression:

log

ˆ

PrpT “ 1 | X q1´ PrpT “ 1 | X q

˙

“ β0 ` βXX ` ε

Note: it is entirely possible that we may want to include interactionsand other marginal curvature, etc. terms in this regression model: thegeneric vector notation (bold-faced coefficients, covariates) should beconstrued as allowing for these possibilities.



Using the logistic model, we can then model the propensity score:

PrpT “ 1 | X q1´ PrpT “ 1 | X q

“ exppβ0 ` βXX ` εq

PrpT “ 1 | X q “ p1´ PrpT “ 1 | X qq exppβ0 ` βXX ` εq

exppβ0 ` βXX ` εq “ PrpT “ 1 | X qp1` exppβ0 ` βXX ` εqq

spX q “ PrpT “ 1 | X q “exppβ0 ` βXX ` εq

1` exppβ0 ` βXX ` εq

So once we have estimates for the parameters (regression coefficients)of the logistic model, we can then get an estimate for the propensityscores.



Return to our tutor intervention example:

Model assignment to treatment T conditional on observed covariates,for example:

logitpT q “ β0 ` β1Sex ` β2Faculty ` β3Age ` β4GPA` ε

Fit in R with ‘glm’ command:

Now convert to propensity scores using formula we previously derivedand add these scores to our data frame:




previous termTutor?

Finalgrade

Propensityscore est.

1 F SCIE 19 3.5 1 90 0.2472 F SCIE 19 2.3 1 88 0.3383 F SCIE 21 3.7 0 85 0.2254 M SCIE 18 3.0 0 82 0.4985 F SCIE 20 2.5 0 82 0.3166 M SCIE 19 3.7 0 83 0.4287 M SCIE 19 3.5 1 90 0.4468 F ARTS 18 3.8 0 95 0.1189 M SCIE 21 3.5 1 94 0.43410 M SCIE 20 3.6 0 94 0.43111 F ARTS 19 2.8 0 75 0.159...

......

......

......

...113 M SCIE 20 3.1 0 85 0.476


Matching with propensity scores

So now that we have estimates for the propensity scores, how do wematch sample units on them (recall: these are continuous quantities)?

Many techniques have been proposed, but one of the most commonand validated techniques is to match on the quintiles or deciles of thepropensity score distribution; i.e. match on the strata of thep.s. distribution.



Quintiles: assume sample units are exchangeable over treatment(conditional on observed covariates and assuming no omittedconfounders) within the quintiles of the propensity score distribution:

[0,20th percentile), [20th,40th percentile), . . . , [80th,100th percentile]

If you have enough data, can match over finer quantiles of thepropensity score distribution, e.g. the deciles:

[0,10th percentile), [10th,20th percentile), . . . , [90th,100th percentile]

Once again, many algorithms for choosing matches within each ofthese strata.

For our example, quintiles of p.s. distribution are:

0, 0.13, 0.22, 0.36, 0.44, 1




previous termTutor?

Finalgrade

Propensityscore est.

1 F SCIE 19 3.5 1 90 0.2472 F SCIE 19 2.3 1 88 0.3383 F SCIE 21 3.7 0 85 0.2254 M SCIE 18 3.0 0 82 0.4985 F SCIE 20 2.5 0 82 0.3166 M SCIE 19 3.7 0 83 0.4287 M SCIE 19 3.5 1 90 0.4468 F ARTS 18 3.8 0 95 0.1189 M SCIE 21 3.5 1 94 0.43410 M SCIE 20 3.6 0 94 0.43111 F ARTS 19 2.8 0 75 0.159...

......

......

......

...113 M SCIE 20 3.1 0 85 0.476



Now match sample units with T “ 1 to those with T “ 0 within eachstrata.

Can do this manually, or most commonly use software to creatematches.

Some potential issues:

No guarantee that you will have the same number of treated units asuntreated units within each strata.

No guarantee that your strata will be equal-sized.

Can get different matches if you choose different strata; thus, willproduce different estimates of ACE pT q.

We will explore these issues soon.


Case study 1

Rubin (1997)


Limitations of propensity scores

How to specify the best propensity score model?

How much balance is enough balance?

What if there aren’t enough matches to be made in a stratum?


Specifying the propensity score model

Lots of conflicting research out there on this.

Different goals of model specification than before:

Previously (RD designs), wanted to properly specify the model for thedata-generating process: Y “ f pX q ` ε.

But now, we are trying to specify a model for theassignment-to-treatment process: spX q “ PrpT “ 1 | X q.

Main goal: to acheive balance of covariates between treated groups,thus ensuring exchangeability of treatment over measured covariates.

Best way to do this: fit a PS-model, then check the balance ofcovariates empirically.


Checking empirical propensity score balance

Most common ways to check covariate balance:

Do standardized mean differences of the covariate between T “ 0 andT “ 1 approximately equal 0 over each propensity score subclass?

Does ratio of variances of covariate between T “ 0 and T “ 1approximately equal 1 over each propensity score subclass?

More generally, do the distributions of the covariate between T “ 0and T “ 1 approximately equal?

Also more generally, are the propensity scores balanced between theT “ 0 and T “ 1 groups in each subclass?

See case studies for examples.

How much balance is enough? No good answer.


Outline for working with propensity scores

Fit a PS-model for the assignment-to-treatment mechanism.

Check the empirical balance of propensity scores over subclasses(e.g. quintiles, deciles).

If balance is poor, specify revised PS-model and check new balance.

Once balance is adequate, then estimate ACE within each subclass:

pEpY | spX qq

Finally, can combine strata estimates to get an overall estimate of theACE, if desired: usually combine estimates via inverse-sample-sizeweighting.


Case study 2

Connors et al. (1996)


Limitations of propensity scores

How to specify the best propensity score model?

How much balance is enough balance?



What if we can’t find a match?


Equivalently, what if the distribution of the propensity scores are toounbalanced between treatment groups?

There is NO solution to this problem, and it is quite common.

Partial fixes:

Restrict your inferences to subsets where this problem does not arise.

Collect more data to produce enough matches in each stratum.


Next time

Instrumental variables


epse 581c: causal inference for applied researchers · in observational or quasi-experimental...

Documents