Do Students Know Best? Choice, Classroom Time and Academic Performance

Ted Joyce Baruch College and Graduate Center, CUNY and NBER

Sean Crockett

Baruch College and Graduate Center, CUNY

David A. Jaeger Graduate Center, CUNY, University of Cologne, CESifo, and NBER

Onur Altindag

Graduate Center, CUNY

Stephen D. O’Connell Graduate Center, CUNY

Dahlia Remler

Baruch College and Graduate Center, CUNY and NBER

Effect of Classroom Time

• Does time in the classroom matter for student outcomes? – Given rich online materials available today

• Randomized experiments solve biased treatment selection problem – Irreplaceable – Notation: Randomized Clinical Trial (RCT)

Randomized Experiments: Limitations in a World of Choice

• Unwillingness to participate in RCT less generalizable

• Students get to choose format – Being able to choose motivation – Students choose based on what works for them

(heterogeneous treatment effects)

• Time, expense limit RCTs… – How close can observational studies come?

Research Questions • What is the effect of time in the classroom on

student outcomes? • Do students sort into different learning formats?

– What factors drive sorting?

• Does that sorting affect the estimates of classroom time on performance? – Parallel experimental and observational study – How close can observational estimates come? – Do students know best? [can’t fully answer…]

Outline of Talk – Part I

• Review key studies on online effects

• Introduce the DRPT—outline our “quasi-DRPT”

• Review Potential Outcomes Framework

• Evidence to support quasi-DRPT assumptions

Outline of Talk – Part 2

• Econometric model: experimental v. observational effects

• Selection – Survey about likely selection drivers

• Results by class size • Implications

Several Relevant Literatures

• Effect of Attendance on performance (Romer 1993; Dobkin et al. 2010) – 1 GPA (Romer); -.17 SD (Dobkin)

• Large database analyses (Xu & Jaggers 2013;

Bettinger et al. 2014; Hart et al. 2014) -.14 SD (X-J); -10 pts (H); -0.09 , IV=-0.28 SD

Several Relevant Literature (cont)

• RCTs of online learning: (Figlio et al. 2013;

Bowen et al. 2014; Alpert et al. 2015; Joyce et al. 2015)

2-3 pts (F); no diff (B); no diff blended (A)

Participation issues

Several Relevant Literature (cont)

• Doubly Randomized Preference Trials (DRPT) • Medical literature (Bradley 1993)

• Janevic et al. (2003); • Shadish et al. (JASA 2008); Marcus et al

(2012) • Long, Little, Lin (JASA 2008a,2008b)

Doubly Randomized Preference Trial

Consent

Randomization

Randomization

Experiment Treated

Controls

Choice Treated

Controls

DRPT uses • Observational and experimental estimates

generalize to same population

• Can selection bias can be eliminated in observational studies...which variables needed

• How effects (experimental estimates) differ by preference – & from choosing

Cites: Shadish, Clark & Steiner (JASA 2008); Janevic et al (2003); Marcus et al (2012); Long, Little & Lin (2008a, b); Cook, Shadish & Wong (2008)

Our Quasi DRPT

• Run a Randomized Experiment in fall of 2013

• Repeat study in fall of 2014 – Identical except students choose format

• In 2014 we do a detailed baseline survey: Can we predict format choice with a rich set of covariates and “eliminate” Omitted Variables Bias?

Recap of Results from RCT Joyce, et al. Econ of Ed. Review, June 2015

• Randomized intro econ students into – Treatment: format with a 50% reduction in class time – Control: traditional – Providing online content to both groups

• Primary Results: – Class time matters: treatment group scored average

2.5 percentage points lower on exams (about .2 SDs) – Relatively little evidence of treatment heterogeneity – No differences in non-cognitive effort (attendance, time

spent with online materials), nor withdrawal. – Experimental estimates of attendance are about ¼ of

observational estimates from past studies

Micro Principles (ECO 1001) • ECO 1001 required course for all students

applying to the business program – Enrollment ≈ 1000 students each fall – Over 100 are evening students, 25 Honors students

& another 40 students taught by adjunct faculty

• Professor A and Professor B – Both Taught ECO1001 for the past 6 years – Both have similar and strong teaching evaluations

• Registrants offered 5 points on final grade for participating

Professors and Classrooms Monday Tuesday Wednesday Thursday Friday

Morning A/Small: Traditional

A/Small: Traditional

B/Large:

Compressed

Afternoon A/Small: Compressed

B/Large:

Traditional

B/Large: Traditional

Research Design • Course: ECO 1001, Microeconomic Principles • Two class formats

– Traditional meets 2 x per week for 75 minutes each – “Compressed” meets 1 x per week for 75 minutes – Online materials identical for both formats (quizzes, videos) – Professors used same slides in both formats, but went faster

with less time for questions, etc. in the compressed format • Two professors, each teaching both formats

– Professor A: always in small classroom with 114 seats – Professor B: always in large classroom with 274 seats

• Two semesters of ECO 1001, Microeconomic Principles – Fall 2013: 725 Students were randomized into formats – Fall 2014: 766 Students chose format

Doubly Randomized Preference Trial

Consent

Randomization

Randomization

Experiment Traditional

Compressed

Choice Traditional

Compressed

Potential Outcomes Framework Define the causal effect for person “i” δi = (P1i – P0i) f=1 compressed format; f=0 traditional format Since δi is unobsevable we estimate the expected value

E(δi) = E(P1i – P0i)

Potential Outcomes Framework: Randomized Arm

Pi = (1-di)*P0i + di*P1i di = 1 if compressed; di = 0 if traditional

E(Pi|di=1) = (P1i|di=1) E(Pi|di=0) = (Pi0|di=0) E{(Pi|di=1) - (Pi|di=0)} = E(P1i|di=1) - E(P0i|di=0)

δr = E(P1i|di=1) - (P0i|di=1) + E(P0i|di=1) - (P0i|di=0) Treatment of

Treated (TOT) Selection bias

Potential Outcomes Framework: Choice Arm

Pi = (1-di)*P0i + di*P1i di = 1 if compressed and 0 traditional

E(Pi|X, di=1) = (P1i|X, di=1) E(Pi|X, di=0) = (Pi0|X, di=0) E(Pi|X,di=1) - (Pi|X,di=0) = E(P1i|X,di=1) - (P0i|X,di=0) δc = E(P1i|X,di=1) - (P0i|X, di=1) + E(P0i|X,di=1) - (P0i|X,di=0)

Treatment of Treated (TOT)

Selection bias

Diff-in-Diff of Treatment Effects

𝛿𝛿𝑐𝑐 = 𝐸𝐸𝑥𝑥{𝐸𝐸 𝑃𝑃1𝑖𝑖 − 𝑃𝑃0𝑖𝑖 𝑋𝑋 }

𝛿𝛿𝑟𝑟 = 𝐸𝐸[𝑃𝑃1𝑖𝑖 − 𝑃𝑃0𝑖𝑖}

𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 = 𝛿𝛿𝑟𝑟 − 𝛿𝛿𝑐𝑐

Treatment effect from randomized arm

Treatment effect from choice arm

Why Might Choice Affect Outcomes? • Learning style may be better suited to one

format or another – If students choose the format that they expect will

improve their performance, then the effect of format likely will be less in the choice than in the randomized arm

• If students choose format for reasons such as convenience, it could increase difference in the treatment effect of format between the two designs

Quasi Doubly Randomized Preference Trial (as Implemented)

Haphazard Randomization

Experiment (Fall 2013)

Choice (Fall 2014)

Randomization

Traditional

Compressed

Traditional

Compressed

Consent (96%)

Consent (100%)

Haphazard Randomization into Format

• “Haphazard randomization” into format – Can’t postpone…

• Covariate balance betw semesters excellent

• No reason at all to think there are any systematic differences between students in the two semesters – Almost as good as doing DRPT in one semester, but

with more power

Baseline Characteristics 2013 2014 Diff NormD

Baruch GPA 3.03 3.04 -0.01 -0.01

SAT Verbal 540.7 556.5 -15.8*** -0.13

SAT Math 604.0 608.7 -4.7 -0.04

Credits 44.59 44.88 -0.29 -0.01

Part time 0.07 0.08 -0.01 -0.01

White 0.27 0.29 -0.02 -0.03

Asian 0.45 0.46 -0.01 -0.02

Native English 0.53 0.55 -0.03 -0.04

*** p<.01

Baseline Characteristics

Baseline Characteristics Within 2013

Balance Across Formats in Choice Year

• Formats balanced in randomized experiment year (2013), of course

• Balance across formats in choice year (2014) – For variables measured in both years – No treatment selection bias seen in variables

strongly predictive of performance • Math SAT, GPA…

– From this evidence, could still be selection bias in survey variables (2014 only) or unmeasured variables

Baseline by Format: Choice Comp Trad Diff NormD

Baruch GPA 3.05 3.03 0.02 0.03

SAT Verbal 555.2 557.8 -2.6 -0.02

SAT Math 604.9 612.4 -7.5 -0.06

Credits 44.76 44.99 -0.24 -0.01

Part time 0.07 0.09 -0.01 -0.04

White 0.28 0.30 -0.02 -0.03

Asian 0.46 0.47 -0.01 -0.01

Native English 0.59 0.53 0.06 0.08

*** p<.01

Baseline Characteristics within 2014

Selection into Compressed

Outline of Talk – Part 2

• Choice effect: – How does effect of compressed vary choice v.

experiment?

• Preference effect: – How does effect of compressed vary by

preference for compressed v. hybrid?

• Selection: What factors drive format choice? • Results by class size • Implications

Econometric Specification

Difference between randomized and choice estimates, controlling for variables available in both years: Choice Effect

δ = δr − δc

Student Performance Measures • Midterm:

– % of 30 multiple choice questions answered correctly • Final

– % of 40 multiple choice questions answered correctly • Midterm+Final

– % of 70 Midterm and Final questions answered correctly

• Overall Grade: – Curved Midterm (35%) + curved Final (45%) + Aplia

(20%) + 5% points experiment participation bonus. • Also: Withdrawal

Effect of Class Time on Student Performance

Cognitive Outcomes: 2013 vs. 2014¥ (standard deviations)

Joyce et al. (2014) 36 1/23/2014

¥All estimates adjusted for student characteristics

Midterm Final MT+F Overall

2013 (δr) -0.20*** -0.14 -0.19** -0.20***

2014 (δc) -0.12 -0.13* -0.14* -0.14**

(δr-δc) -0.08 -0.01 -0.05 -0.07 N 1,332 1,333 1,333 1333

Cognitive Outcomes: 2013 vs. 2014¥ (standard deviations)

Joyce et al. (2014) 37 1/23/2014

¥All estimates adjusted for student characteristics; includes only those with preferred choices in 2014

Midterm Final MT+F Overall

2013 (δr) -0.19*** -0.13* -0.18*** -0.19***

2014 (δc) -0.13* -0.09 -0.12* -0.12*

(δr-δc) -0.06 -0.04 -0.06 -0.07

N 1,243 1,244 1,243 1243

Expressed Preferences • At each semester start, students asked what format

they preferred

• In Choice year, 14% (of those who expressed preference) did not get preference, roughly equal between formats

• In Experimental year, being randomized to a

format made you decide you liked it: – In compressed arm: 77% preferred compressed – In traditional arm: 43% preferred compressed – Better to ask before assignment, if possible

Preference Differences

Pref dummies are: Actual_Preferred Omitted category: Got Traditional, Preferred Traditional (TT)

Does the compressed effect vary by (expressed) preferred format?

Midterm Final MT+F Overall Got Comp, Want Comp

-0.21** (0.08)

-0.17* (0.09)

-0.22** (0.09)

-0.17*** (0.06)

Got Comp, Want Trad

-0.26* (0.13)

-0.04 (0.14)

-0.16 (0.14)

-0.10 (0.11)

Got Trad, Want Comp

-0.04 (0.08)

-0.09 (0.09)

-0.08 (0.08)

-0.06 (0.06)

Comp effect, among want Comp

-0.17 (0.08)

-0.08 (0.09)

-0.14 (0.08)

-0.12 (0.07)

Comp effect Diff pref comp – pref trad)

-0.43 (0.17)

-0.11 (0.18)

-0.30 (0.17)

-0.22 (0.13)

N 605 605 605 605

Standard Errors in Parentheses; Only Known Preferences Omitted category: Got Trad, Want Trad

2013 (Experiment): Preference Differences

Midterm Final MT+F Overall Got Comp, Want Comp

-0.13* (0.07)

-0.11 (0.08)

-0.13* (0.07)

-0.12** (0.05)

Got Comp, Want Trad

-0.15 (0.12)

-0.35** (0.15)

-0.30** (0.14)

-0.22** (0.10)

Got Trad, Want Comp

-0.18 (0.15)

0.01 (0.15)

-0.08 (0.15)

-0.09 (0.11)

Comp effect, among want Comp

0.05 (0.15)

-0.12 (0.16)

-0.06 (0.16)

-0.03 (0.11)

Comp effect Diff pref comp – pref trad)

-0.10 (0.20)

-0.47 (0.23)

-0.35 (0.22)

-0.26 (0.16)

N 648 648 648 648

2014 (Choice): Preference Differences

Standard Errors in Parentheses; Only Known Preferences Omitted category: Got Trad, Want Trad

Results • Compressed effect about .2 SD • Effect sizes smaller when students choose

– but not statistically significantly

• Differences by preference in compressed effect suggest – Those who prefer traditional do better in

traditional • But little power, stat sig

• In theory, small selection bias on unmeasured variables possible

Original Research Goal on Selection • Cook, Rubin on prospective observational

studies • Every possible approach to learn every

possible variable that might drive format choice (selection) – Conducted informal interviews

• Create and field survey to measure all variables prospectively

• See if can reproduce randomized experiment – But we came close without them!

Survey: Hypothesized Format Drivers • Online vs. in-person orientation • Perceived strength in quant, non-quant courses • Need for structure • Attitudes towards risk • Hours worked, if any • Commute time

• Survey Instrument

– Some constructs had multiple indicators – Most on 5-point Likert response format (“Strongly

Agree,” “Agree,” “Neither Agree or Disagree,” “Disagree”, “Strongly Disagree”)

Traditional Compressed P-value (χ2 )

My learning style well-suited suited to hybrid w/ less face-to-face class time

3.30 2.71 0.00***

Ever took hybrid/fully online 0.32 0.29 0.26

If yes, worked well 2.93 2.58 0.06*

Traditional lectures work well for me 2.30 2.51 0.02**

Prefer quant courses to writing courses

2.57 2.54 0.00***

Prefer electronic devices to read 3.38 3.23 0.16

Interaction w/ prof & other students helpful

1.81 2.12 0.00***

Commute time to school 2.42 2.52 0.30

Paid work 2.51 2.35 0.15

Prior Student Learning Experiences: 2014

Selection • Few variables drive format selection • Variables that drive format selection

– Mostly learning style – Uncorrelated w/ performance predictors (e.g.,

GPA, Math SAT) – Do not affect performance

• No detectible treatment selection bias! • In theory, small choice effect could be

contaminated by residual treatment selection bias due to unmeasured variables…. Unlikely

Effects by Class Size/Professor

• Superb balance across format in class-size & professor

• But design makes it impossible to disentangle effect of professor from effect of class size

• We believe effect is class size, not professor • Professor A and Professor B

– Both Taught ECO1001 for the past 6 years – Both have similar and strong teaching evaluations

Professors and Classrooms Monday Tuesday Wednesday Thursday Friday

Morning A/Small: Traditional

A/Small: Traditional

B/Large:

Compressed

Afternoon A/Small: Compressed

B/Large:

Traditional

B/Large: Traditional

Prior Teaching Evaluations scale is 1 (worst) through 5 (best)

Cognitive Outcomes: 2013 vs. 2014 Professor A/Small Class¥

(standard deviations)

Joyce et al. (2014) 51 1/23/2014

¥All estimates adjusted for student characteristics

Midterm Final MT+F Overall

2013 (δr) -0.12 -0.10 -0.12 -0.12

2014 (δc) -0.11 -0.14 -0.14 -0.15

(δr-δc) -0.01 0.04 0.02 0.03 N 383 383 383 383

Performance within Professor A

Cognitive Outcomes: 2013 vs. 2014 Professor B/Large Class¥

(standard deviations)

Joyce et al. (2014) 53 1/23/2014

¥All estimates adjusted for student characteristics

Midterm Final MT+F Overall

2013 (δr) -0.27** -0.18** -0.26*** -0.28***

2014 (δc) -0.16** -0.12 -0.14 -0.16

(δr-δc) -0.11 -0.06 -0.08 -0.12 N 949 950 950 950

Performance within Professor B

Implications • Lessons for RCTs

– Choice effects may be important (although not so much in this case)

– Possible to get estimates of choice effects using DRPT framework

• Lessons for observational studies – May not suffer from treatment selection bias! – Prospective observational studies doable

• Lessons for both – Prospectively measure likely heterogeneity

dimensions, like format preference!

Implications • Administrators looking to reduce costs with

fewer class meetings: be aware of performance penalty – Trade off with class size

• Treatment effects may be heterogeneous – Students may know what works best and choose

accordingly • Preference effects may matter

– Beware forcing those who don’t want hybrid/online into hybrid/online