Douglas N. HarrisAssociate Professor of Economics

University Endowed Chair in Public Education

Tulane UniversityPresentation at Hualien

Dec.17, 2012

Using Randomized Trials and Behavioral Economics to Address Problems in U.S.

College Market

Two Recent Advances in Economics

• Behavioral economics• Randomized trials• The two advances are connected:• Most behavioral economics theories are rooted

in psychology research and randomized trials are more common there• 20 years ago, this field was called “experimental

economics” reflecting the methodological focus

Experiments vs. Quasi-Experiments

• Quasi-experiments have high external validity• We tend to prefer more complex techniques and randomized

trials involve less statistical complexity• The real advantage is that estimates may generalize to other

“real-world” settings• However, quasi-experiments can be biased (low internal

validity) and often have low statistical power

• Randomized trials have low bias (high internal validity) • However, the situations are often unrealistic and therefore

have low external validity• Some would say that internal validity is more important

because we cannot generalize something to other situations if it is not accurate about the particular setting where it occurred

Financial Incentives for College a Case in Point

• U.S. spends $155 billion annually on financial aid programs for low- and middle-income people

• Until about five years ago, we only had quasi-experiments of aid

• These suggest aid effects of about 3-4 percentage point on entry per $1,000 in aid• There is debate about whether these are small or

large, however, note that if we compare financial aid to other programs, this does not appear cost-effective

Stylized Facts of U.S. College Market

Two Initial Stylized Facts

• The return to human capital investment (education) is much higher than the return on other forms of capital investment (i.e., people seem to under-invest in education)

• Underinvestment is especially common among students from low-income families • They invest less even when they have similar

academic ability as those from higher income families (and there are no credit constraints)

College attendance among 18-24 year old males (Carneiro & Heckman, 2002)

B.A. degree completion by age 24 (conditional on entry)

Source: Postsecondary Education Opportunity

College attendance by income and test scores (Carneiro & Heckman, 2002)

Theories that might explain these

patterns

Old Neoclassical Theories• Risky returns: people are risk-averse and, obviously, not

everyone obtains the average return (Heckman, 2006)• However, people would have to be extremely risk-averse for this to

be the only explanation (Palacios-Huerta, 2006)

• Option value• When investments are irreversible and timing is flexible, risk-

neutral agents may delay if they can obtain more information in the future (Jacobs, 2007)

• Psychic costs• Education is a consumption good and some do not like it

• Misinformation: under-estimate return or over-state costs• Unlikely; people over-state both the costs and the benefits of higher

education by similar amounts (Kane & Avery)

Apparently New Explanations from within Neoclassical Model• Variation on risky returns: Uncertainty leads

students to invest less in learning during primary and secondary school (perhaps because of misinformation or risk)• Uncertainty for all but especially low-income families

• Initial under-investment worsens the consumption disutility; i.e., people who do poorly in school are more likely to dislike college because it is too hard (they may not anticipate this)• Related to Heckman’s idea that investments have to

be made early for cognitive reasons

Behavioral Economic Theories

• Loss aversion• General meaning: The marginal disutility of a loss is larger

(in absolute value) than the utility of an equal-sized gain• In human capital: People may fear taking the chance that

they will lose what they have (or lose what they think they can obtain with no education)

• Endogenous preferences (peer effects)• General meaning: People’s preferences are influenced by

their friends and family• In human capital: Students’ preferences about education

are influenced by classmates and parents (hard to separate parent influence from availability)

More Behavioral Economic Theories

• Availability• General meaning: People over-state risks of negative events

for which they have vivid examples in mind• Human capital investment: Low-income students are more

likely to have parents who failed in college; they have a negative example available

• Ambiguity aversion• General meaning: People prefer lotteries where the

probabilities are easy to assign (holding constant actual risk)• Human capital investment: Probabilities are hard to assign

because there are many forms of uncertainty (cost, psychic cost, returns)

Theory Summary

• Above discussion suggests many possible explanations for underinvestment

• How can we test them?

Empirical Evidence from Two

Randomized Trials

College Student Experiment: Wisconsin Scholars Longitudinal

Study• Low-income four-year college students

randomized to receive $17,500 for college if they stay in school full-time

• Students located in many public colleges throughout the state of Wisconsin

• We have data from the first three years

Impact on Aid Package: Year 1 (2008-2009)

CONTROL TREATMENT IMPACT ($3,500)

Total Aid $11,426 $1,665***Pell (%) 99.8 -

0.1 State Grant (%) 99.0 - 3.7 ***SEOG (%) 63.6 -9.8***ACG (%) 80.4 0.3(3.0) Institutional Aid (%) 54.7

1.6 Work Study (%) 18.3 - 5.0 *Sub. Staff (%) 77.9 -

11.1***Unsub Staff (%) 38.9 -

3.8 Total loans ($) 3428.3 -

909.5

Average Impacts of the Grant on Enrollment: 2008-2011

Control Mean

Treatment Effect

Total # terms enrolled (f/s, %)

5.192 0.05 (.09)

Ever transferred (%) 23.7 -0.4 (2.7)

Ever attended 2-year college (%)

14.4 -0.4 (2.2)

Completed associates degree (%)

3.1 0.4 (1.1)

Average Impacts of the Grant on Credits: 2008-2010

Control Mean

Treatment Effect

Credit Accumulation Average credits completed 46.9 0.9 (1.0) Earned 1-29 credits (%) 18.2 0.4 (2.4) Earned 30-47 credits (%) 16.0 2.6 (2.4) Earned 48-59 credits (%) 42.3 - 8.2 (3.1) *** Earned 60+ credits (%) 22.2 6.3 (2.8) **Progress toward 4-year Degree 60+ credits, 2-2.5 GPA 2.2 0.0 (0.9) 60+ credits, 2.5-3.0 GPA 6.8 -0.5 (1.6) 60+ credits, 3.0-3.5 GPA 8.5 4.7 (2.0) ** 60+ credits, 3.5+ GPA 4.3 2.4 (1.4) *

Results from College Student Experiment

• No effect on college persistence• Small positive effects on GPA and number of credits• Pattern of effects is not predicted by the neoclassical

model• Results look slightly better in more recent cohorts• Perhaps most interesting is evidence of loss aversion• Some students lost the grant (arguably exogenously)

and these students had worse outcomes than the control group

High School Student Experiment: Degree Project• 2,500 low-income high school students in

Milwaukee, Wisconsin were randomly assigned to receive $12,000 for college

• Main requirements: 2.5 GPA and 90% attendance rate

• Program has only been operating for one year and students are now in 10th grade

• As an aside, the methodology is also interesting: a paired cluster RCT

Distinctive Experiment

• Cluster RCT• Paired

randomization

• Baseline Equivalence

• Standard error calculations

• Statistical power associated with correct std errors

• Generalizability

Paired Randomization

• When we get the standard errors right, we often have insufficient power in a cluster randomized trial

• Paired randomization can be a good solution, especially in this case when pairing on a lagged value of the dependent variable

• But this affects everything about how the analysis is carried out (more to come)

Statistical Power

• The main concern with cluster RCTs is that, with correct standard errors, power is insufficient to identify plausible effects

• Pairing combined with paired randomization largely solves this problem

• Power in this design is the same as an individual-level RCT with the same N, but no covariates• Only slightly worse than individual-level RCT

with the same covariate R2

Generalizability with Power

• This design has the best of both worlds: strong statistical power with improved generalizability• Nothing is really generalizable but cluster RCTs are

almost certainly an improvement

• We really can have our cake and eat it too• Key is to have lagged dependent variable and use

these for both the pairing and for covariate adjustment

• Significant implications for the design of program evaluations

Analysis Issues• Treatment stays with student even if switch schools

• Cluster RCT means analysis“at level of cluster” (j)• Ambiguous terminology; can still use student-level data (i) to

(slightly) increase power (Bloom)

• Pairing means cannot simply use diff-in-means tests• For statistical power, we wouldn’t want to anyway

yij=α+βTj+κp+γ1Pj+γ1Xij+εij

Outcome

Treatment Pairing var.Stud/sch char.Pair effects

Standard Errors• Ignoring clustering in cluster RCT yields std

errors 1/5 to 1/2 actual size• The “unit of analysis is the unit of

randomization” is usually interpreted to mean that analysis is at the cluster level of aggregation and that this yields correct standard errors

• In reality, estimating at cluster level yields std errors that are too small, though still need to account for clustering

Correct Standard Errors

• Stata “cluster” command helps, but still yields std errors that are too small

• Bootstrapping is generally a good solution (Cameron, Gelbach, & Miller, 2008)• Re-sample with replacement to obtain

thousands of “new” samples and re-estimate• But there are no prior examples of

bootstrapping with paired cluster RCT• In short, bootstrap the pairs, not the clusters

Results of High School Student Experiment

• So far, no effect on GPA or attendance• But large positive effects on college

expectations• Summary: effects on perceptions but not

behavior• We are just now formulating ways of testing

the behavioral economic assumptions

Back to the Theory

Tests of Behavioral Economics Theories Using Behavioral Responses

• Loss aversion• We already have some direct evidence of this

as noted above

• Peer effects• We have exogenous variation in the number of

classmates students have who received the scholarship

• If there are peer effects, we should see that students with recipient classmates are affected more

More Tests with Surveys

• Survey questions below asked in college experiment and we are asking similar ones in high school exper.

• Consumption value/psychic cost• “How much fun is college?”• “How interesting are your classes?”• “How difficult is the material in your classes?”• “How much do you like the freedom of being in

college?”• “How much do you enjoy the people you go to college

with?”

More Tests (cont.)

• Combined risk-aversion during high school and psychic costs• Ask consumption value questions similar to

above• Also, compare the effects of the two

experiments; if there is very little difference in effects, then this theory is rejected

Tests of Behavioral Economic Theories Using Surveys

• Availability• We ask whether students’ parents had a “bad

experience” in college• We also ask the level of education, so we can

distinguish other ways in which parents education might matter

• Ambiguity aversion• More difficult to test• No question planned at this time

Conclusion

• To be announced• However, these experiments provide a

unique opportunity to test multiple theories• The experiments will also have broader

implications for financial aid and incentive programs and for our general understanding of human behavior