douglas n. harris associate professor of economics university endowed chair in public education...
TRANSCRIPT
Douglas N. HarrisAssociate Professor of Economics
University Endowed Chair in Public Education
Tulane UniversityPresentation at Hualien
Dec.17, 2012
Using Randomized Trials and Behavioral Economics to Address Problems in U.S.
College Market
Two Recent Advances in Economics
• Behavioral economics
• Randomized trials
• The two advances are connected:• Most behavioral economics theories are rooted
in psychology research and randomized trials are more common there• 20 years ago, this field was called “experimental
economics” reflecting the methodological focus
Experiments vs. Quasi-Experiments
• Quasi-experiments have high external validity• We tend to prefer more complex techniques and randomized
trials involve less statistical complexity• The real advantage is that estimates may generalize to other
“real-world” settings• However, quasi-experiments can be biased (low internal
validity) and often have low statistical power
• Randomized trials have low bias (high internal validity) • However, the situations are often unrealistic and therefore
have low external validity• Some would say that internal validity is more important
because we cannot generalize something to other situations if it is not accurate about the particular setting where it occurred
Financial Incentives for College a Case in Point
• U.S. spends $155 billion annually on financial aid programs for low- and middle-income people
• Until about five years ago, we only had quasi-experiments of aid
• These suggest aid effects of about 3-4 percentage point on entry per $1,000 in aid• There is debate about whether these are small or
large, however, note that if we compare financial aid to other programs, this does not appear cost-effective
Stylized Facts of U.S. College Market
Two Initial Stylized Facts
• The return to human capital investment (education) is much higher than the return on other forms of capital investment (i.e., people seem to under-invest in education)
• Underinvestment is especially common among students from low-income families • They invest less even when they have similar
academic ability as those from higher income families (and there are no credit constraints)
College attendance among 18-24 year old males (Carneiro & Heckman, 2002)
B.A. degree completion by age 24 (conditional on entry)
Source: Postsecondary Education Opportunity
College attendance by income and test scores (Carneiro & Heckman, 2002)
Theories that might explain these
patterns
Old Neoclassical Theories
• Risky returns: people are risk-averse and, obviously, not everyone obtains the average return (Heckman, 2006)• However, people would have to be extremely risk-averse for this
to be the only explanation (Palacios-Huerta, 2006)
• Option value• When investments are irreversible and timing is flexible, risk-
neutral agents may delay if they can obtain more information in the future (Jacobs, 2007)
• Psychic costs• Education is a consumption good and some do not like it
• Misinformation: under-estimate return or over-state costs• Unlikely; people over-state both the costs and the benefits of
higher education by similar amounts (Kane & Avery)
Apparently New Explanations from within Neoclassical Model
• Variation on risky returns: Uncertainty leads students to invest less in learning during primary and secondary school (perhaps because of misinformation or risk)• Uncertainty for all but especially low-income families
• Initial under-investment worsens the consumption disutility; i.e., people who do poorly in school are more likely to dislike college because it is too hard (they may not anticipate this)• Related to Heckman’s idea that investments have to
be made early for cognitive reasons
Behavioral Economic Theories
• Loss aversion• General meaning: The marginal disutility of a loss is larger
(in absolute value) than the utility of an equal-sized gain• In human capital: People may fear taking the chance that
they will lose what they have (or lose what they think they can obtain with no education)
• Endogenous preferences (peer effects)• General meaning: People’s preferences are influenced by
their friends and family• In human capital: Students’ preferences about education
are influenced by classmates and parents (hard to separate parent influence from availability)
More Behavioral Economic Theories
• Availability• General meaning: People over-state risks of negative
events for which they have vivid examples in mind• Human capital investment: Low-income students are more
likely to have parents who failed in college; they have a negative example available
• Ambiguity aversion• General meaning: People prefer lotteries where the
probabilities are easy to assign (holding constant actual risk)
• Human capital investment: Probabilities are hard to assign because there are many forms of uncertainty (cost, psychic cost, returns)
Theory Summary
• Above discussion suggests many possible explanations for underinvestment
• How can we test them?
Empirical Evidence from Two
Randomized Trials
College Student Experiment: Wisconsin Scholars Longitudinal
Study
• Low-income four-year college students randomized to receive $17,500 for college if they stay in school full-time
• Students located in many public colleges throughout the state of Wisconsin
• We have data from the first three years
Impact on Aid Package: Year 1 (2008-2009)
CONTROL TREATMENT IMPACT ($3,500)
Total Aid $11,426 $1,665***
Pell (%) 99.8 - 0.1
State Grant (%) 99.0 - 3.7 ***
SEOG (%) 63.6 -9.8***
ACG (%) 80.4 0.3(3.0)
Institutional Aid (%) 54.7 1.6
Work Study (%) 18.3 - 5.0 *
Sub. Staff (%) 77.9 - 11.1***
Unsub Staff (%) 38.9 - 3.8
Total loans ($) 3428.3 -909.5
Average Impacts of the Grant on Enrollment: 2008-2011
Control Mean
Treatment Effect
Total # terms enrolled (f/s, %)
5.192 0.05 (.09)
Ever transferred (%) 23.7 -0.4 (2.7)
Ever attended 2-year college (%)
14.4 -0.4 (2.2)
Completed associates degree (%)
3.1 0.4 (1.1)
Average Impacts of the Grant on Credits: 2008-2010
Control Mean
Treatment Effect
Credit Accumulation
Average credits completed 46.9 0.9 (1.0)
Earned 1-29 credits (%) 18.2 0.4 (2.4)
Earned 30-47 credits (%) 16.0 2.6 (2.4)
Earned 48-59 credits (%) 42.3 - 8.2 (3.1) ***
Earned 60+ credits (%) 22.2 6.3 (2.8) **
Progress toward 4-year Degree
60+ credits, 2-2.5 GPA 2.2 0.0 (0.9)
60+ credits, 2.5-3.0 GPA 6.8 -0.5 (1.6)
60+ credits, 3.0-3.5 GPA 8.5 4.7 (2.0) **
60+ credits, 3.5+ GPA 4.3 2.4 (1.4) *
Results from College Student Experiment
• No effect on college persistence
• Small positive effects on GPA and number of credits
• Pattern of effects is not predicted by the neoclassical model
• Results look slightly better in more recent cohorts
• Perhaps most interesting is evidence of loss aversion• Some students lost the grant (arguably exogenously)
and these students had worse outcomes than the control group
High School Student Experiment: Degree Project
• 2,500 low-income high school students in Milwaukee, Wisconsin were randomly assigned to receive $12,000 for college
• Main requirements: 2.5 GPA and 90% attendance rate
• Program has only been operating for one year and students are now in 10th grade
• As an aside, the methodology is also interesting: a paired cluster RCT
Distinctive Experiment
• Cluster RCT
• Paired randomization
• Baseline Equivalence
• Standard error calculations
• Statistical power associated with correct std errors
• Generalizability
Paired Randomization
• When we get the standard errors right, we often have insufficient power in a cluster randomized trial
• Paired randomization can be a good solution, especially in this case when pairing on a lagged value of the dependent variable
• But this affects everything about how the analysis is carried out (more to come)
Statistical Power
• The main concern with cluster RCTs is that, with correct standard errors, power is insufficient to identify plausible effects
• Pairing combined with paired randomization largely solves this problem
• Power in this design is the same as an individual-level RCT with the same N, but no covariates• Only slightly worse than individual-level RCT
with the same covariate R2
Generalizability with Power
• This design has the best of both worlds: strong statistical power with improved generalizability• Nothing is really generalizable but cluster RCTs are
almost certainly an improvement
• We really can have our cake and eat it too• Key is to have lagged dependent variable and use
these for both the pairing and for covariate adjustment
• Significant implications for the design of program evaluations
Analysis Issues
• Treatment stays with student even if switch schools
• Cluster RCT means analysis“at level of cluster” (j)
• Ambiguous terminology; can still use student-level data (i) to (slightly) increase power (Bloom)
• Pairing means cannot simply use diff-in-means tests
• For statistical power, we wouldn’t want to anyway
yij=α+βTj+κp+γ1Pj+γ1Xij+εij
Outcome
Treatment Pairing var.
Stud/sch char.Pair effects
Standard Errors
• Ignoring clustering in cluster RCT yields std errors 1/5 to 1/2 actual size
• The “unit of analysis is the unit of randomization” is usually interpreted to mean that analysis is at the cluster level of aggregation and that this yields correct standard errors
• In reality, estimating at cluster level yields std errors that are too small, though still need to account for clustering
Correct Standard Errors
• Stata “cluster” command helps, but still yields std errors that are too small
• Bootstrapping is generally a good solution (Cameron, Gelbach, & Miller, 2008)• Re-sample with replacement to obtain
thousands of “new” samples and re-estimate
• But there are no prior examples of bootstrapping with paired cluster RCT• In short, bootstrap the pairs, not the clusters
Results of High School Student Experiment
• So far, no effect on GPA or attendance
• But large positive effects on college expectations
• Summary: effects on perceptions but not behavior
• We are just now formulating ways of testing the behavioral economic assumptions
Back to the Theory
Tests of Behavioral Economics Theories Using Behavioral Responses
• Loss aversion• We already have some direct evidence of this
as noted above
• Peer effects• We have exogenous variation in the number of
classmates students have who received the scholarship
• If there are peer effects, we should see that students with recipient classmates are affected more
More Tests with Surveys
• Survey questions below asked in college experiment and we are asking similar ones in high school exper.
• Consumption value/psychic cost• “How much fun is college?”• “How interesting are your classes?”• “How difficult is the material in your classes?”• “How much do you like the freedom of being in
college?”• “How much do you enjoy the people you go to
college with?”
More Tests (cont.)
• Combined risk-aversion during high school and psychic costs• Ask consumption value questions similar to
above• Also, compare the effects of the two
experiments; if there is very little difference in effects, then this theory is rejected
Tests of Behavioral Economic Theories Using Surveys
• Availability• We ask whether students’ parents had a “bad
experience” in college• We also ask the level of education, so we can
distinguish other ways in which parents education might matter
• Ambiguity aversion• More difficult to test• No question planned at this time
Conclusion
• To be announced
• However, these experiments provide a unique opportunity to test multiple theories
• The experiments will also have broader implications for financial aid and incentive programs and for our general understanding of human behavior