marc frenette

***LILE and Aspirations questionReducing Barriers to Postsecondary Education Through Innovative Interventions: Evidence from Canadian Field ExperimentsMcGill University Social Statistics Speaker SeriesMarch 21, 2012Marc Frenette

1

Preliminary: not for quotation

Reducing Barriers to Postsecondary: The Need for Hard Evidence

Improving access to PS important for many reasons:

• Private gains (Card, 1999; many others..)• Societal gains, including productivity enhancement (Coulombe

and Tremblay, 2006), crime reduction (Lochner and Moretti, 2004), longer life expectancy (Lleras-Muney, 2005), improved health (Arendt, 2008), and civic participation (Milligan, Moretti, and Oreopoulos, 2004)

• Equalizing opportunities / increasing intergenerational mobility

2


The Canadian literature has successfully identified several important barriers to PS:

• Income (Frenette, 2008)• Financial literacy (Frenette and Robson, 2011)• Academic performance (Frenette, 2008)• Career information (Frenette, 2009)

3


Although all four barriers are amenable to policy intervention, we have little to no hard evidence on best practices in Canada:

• Students who qualify for grants or participate in academic / career counseling are likely a select group

• Lack of natural experiments in Canada• Field experiments, using random assignment, are ideal, but

rare…

4

The Canadian Millennium Scholarship Foundation Recognized the Need for Hard Evidence (and they had hard cash!!!)

Commissioned the Social Research and Demonstration Corporation (SRDC) to design, implement, and evaluate three interventions to reduce specific barriers to PSE:

• BC AVID– Academic intervention for middle-achieving students

• Learning Accounts (Future to Discover)– Early promise of a large non-repayable grant for low-income

students

• Explore Your Horizons (Future to Discover)– Career / education information

5

AVID

Advancement Via Individual Determination (AVID)

Serves 400,000 students in 4,500 schools worldwide

Goal: raise academic performance, and subsequently, university attendance

Targets students on the margin• Mostly Bs and Cs (middle-achieving)• No extreme behaviour issues• Desire to attend university

6

AVID

What is AVID?

• Academic intervention

• Elective class (replacing a regular elective) offered throughout high school

• Consists of:– Curriculum studies (40%)– Tutorials assisted by local college students (40%)– Motivational activities regarding PS (20%)

7

AVID

Mechanisms through which AVID may help students (Dunn et al., 2008)?

• Study skills (time management, note taking, etc.)

• “Untracking” students (students choosing advanced courses)

• Mentoring effects (continued contact with AVID teacher and tutors)

• Peer effects (continued contact with students sharing similar characteristics)

8

AVID

Can AVID make a difference?

• Cognitive skills are not malleable after age 14 (Heckman, 1995), when AVID begins

– But AVID helps students use their existing cognitive skills more efficiently by helping them become better learners

• Furthermore, non-cognitive skills (motivation, self-discipline) are controlled by the prefrontal cortex, which is malleable until late adolescence (Heckman, 2000; Shonkoff and Phillips, 2000)

9

AVID

Non-experimental evidence on AVID’s effectiveness:

• Mehan et al. (1996):– Improvements in college participation

• Watt et al. (2006)– AVID districts in Texas saw gains in graduation rates, advanced course

enrollments, and international baccalaureate testing• Watt et al. (2007)

– AVID participants had higher aspirations, knowledge about college, and academic preparation compared to peers

• Nagaoka and LaForce (2010)– Propensity score matching study in Chicago– Small improvements in English and Math in grade 9; fewer absences

All suffer from selection bias

• Those most likely to benefit from AVID will sign up

10

BC AVID

First (and only) scientific evaluation of AVID

1,241 students recruited at 14 BC high schools*• Two cohorts (2005 and 2006) followed throughout HS and until

university age• Administrative and survey data used for follow-up

‘AVID eligible’ students recruited through rigorous process:• Middle-achieving (mostly Bs and Cs)• No extreme behaviour issues• Desire to attend university

*Schools and students had to apply to participate and waiting lists were created when classes reached the limit of 30 (just like the real AVID). Informed consent was also required from the parents to collect data.

11

Evaluation approach

Random assignment within high schools

Impact:

• Treatment group outcome – Control group outcome• Controlling for baseline characteristics

12

Focus of this study

Impact of the offer of AVID on highest level of education aspired*:

• Early demand indicator

Future work will report on university / college application / attendance

13

* “No matter what you plan to be doing in a year from now (grade 12), what is the highest level of education you would like to get?”

Large impact on university aspirations for boys and first-generation students

14

Table 2: Impact of offer of AVID on probability of aspiring to university

Outcomes (%)Treatment Control Impact

All 55.31 51.09 4.23(3.10)

Boys 53.31 45.68 7.62 *(4.41)

Girls 57.22 55.19 2.03(4.33)

No parent with postsecondary 52.09 43.32 8.77 **(4.45)

Parent with postsecondary 60.62 61.56 -0.94(4.68)

Note: Standard errors are in parentheses. Significant impacts are denoted by '***' (1%), '**' (5%), and '*' (10%).

Weaker AVID skills to begin with?

No impact on non-university PS, as expected

15

Outcomes (%)Treatment Control Impact

All 39.37 41.86 -2.49(3.17)

Boys 42.86 44.78 -1.92(4.74)

Girls 36.09 40.01 -3.93(4.11)

No parent with postsecondary 42.79 47.57 -4.79(4.56)

Parent with postsecondary 34.14 33.45 0.69(4.61)

Note: Standard errors are in parentheses. Significant impacts are denoted by '***' (1%), '**' (5%), and '*' (10%).

Table 3: Impact of offer of AVID on probability of aspiring to non-university postsecondary

AVID may need more time to work…

16

-

Grade 10 course mark Grade 12 course mark Grade 12 final markOutcomes (%) Outcomes (%) Outcomes (%)

Treatment Control Impact Treatment Control Impact Treatment Control Impact

All 23.00 24.02 -1.02 20.99 18.88 2.11 11.29 9.27 2.03(2.38) (2.10) (1.59)

Boys 15.37 16.43 -1.06 16.98 12.48 4.51 9.05 6.43 2.62(3.29) (3.07) (2.41)

Girls 29.70 30.54 -0.84 24.48 24.36 0.12 13.42 11.40 2.03(3.62) (3.39) (2.73)

19.96 20.77 -0.81 16.34 9.22 7.13 *** 9.31 4.78 4.53 **(3.10) (2.52) (1.88)

26.82 27.84 -1.01 27.71 30.79 -3.09 13.74 15.33 -1.59(3.84) (4.19) (3.29)

Table 5: Impact of offer of AVID on probability of obtaining 80% or more on grade 10 and 12 English course marks and on

Note: Standard errors are in parentheses. Significant impacts are denoted by '***' (1%), '**' (5%), and '*' (10%). The grade 12 English final is a weighted average of the grade 12 English course mark (80%) and the grade 12 English

No parent with postsecondaryParent with postsecondary

Potential biases/evaluation challengesThose we can rule out

Program take-up

• 96.7% accepted AVID offer

Spillover of AVID techniques

• Very minor issue based on earlier comparison with similar non-AVID schools (Cornell notes)

Attrition bias (survey data analysis only)

• 19.6% (treatment) vs 23.8% (control) between grade 9 and 12• No important changes in baseline characteristics

Teacher grading bias

• AVID students are well known• Argument does not apply to differences in sub-group impacts

17

Potential biases/evaluation challengesThose we can not rule out

Program drop-out

• 37.6% withdrew by grade 12 (mostly in that year: wanted to pick other courses to apply skills)

• Estimated impacts = Intention-to-treat effects

Substitution bias (displacing non-experimental treatment)

• Earlier report found reverse: course load was more challenging among treatment group

Timing

• Lagged effect • Students coped with more difficult course load as they were only beginning to learn AVID

skills• AVID now being run before HS in some jurisdictions

18

Potential biases/evaluation challengesThose we can not rule out

Reactions to inequity caused by randomization

• Positive or negative impact on control group• Currently following outcomes of similar students in matched non-AVID schools

Sample size

• Affects statistical significance (especially in sub-group analysis)

19

BC AVID Final Report

Final report (including cost-benefit study) in 2013

20

Future to Discover Overview

Two interventions designed to increased PSE access:

• Learning Accounts

– Early promise of non-repayable grant (up to $8,000) for low-income youth– Offered in NB

• Explore Your Horizons

– Career and PSE planning intervention– Several components (workshops, magazines, parental outreach, etc.)– Offered in NB and MB

21

How Can Learning Accounts Help? Reduces cost of post-secondary, like other grants (clawed-

back from loans)

Potentially addresses several design issues with existing grants:

– Students are informed of the grant– Early promise of aid– No obligation to apply for loans

o May encourage loan averse students to seek grant Palameta and Voyer (2010) Lab experiment: 5-20% students rejected grants when couple with a loan

offer, but accepted same grant without loan offer (even though loan could be forfeited without compromising grant)

o With no loans, Learning Accounts represents additional aid22

How Can Explore Your Horizons Help? Informs students of costs and benefits of PSE

• Financial literacy of students very poor regarding PSE (Frenette and Robson, 2011)

Helps students understand educational requirements for the careers they wish to pursue

• Knowledge of educational requirements an issue for many students (Frenette, 2009)

23

Methodology

Students are randomly assigned to receive the intervention (treatment group) or not (control group)

Three possible interventions:

• LA• EYH• LA + EYH

24

25

Table 1: LA IMPACTS ON PSE ENROLMENT

LA Comparison Impact LA Comparison ImpactGroup Group (s.e) Group Group (s.e)

Enrolled in PSE institution (%)ALL 76.34 66.07 10.27 ** 67.69 61.00 6.68

(4.30) (4.21)LILE 75.20 61.40 13.80 *** 65.70 56.98 8.71 *

(4.92) (4.69)Parents with High school or less (FGF) 67.33 55.28 12.04 * 60.87 55.11 5.76

(6.75) (6.54)Parents with any PSE 88.25 75.68 12.57 ** 74.12 65.82 8.30

(5.25) (6.06)Boys 71.20 58.60 12.60 * 60.98 45.48 15.50 **

(6.85) (7.84)Girls 82.74 70.81 11.93 ** 73.18 72.25 0.93

(5.33) (5.24)Sample size 247 262 240 255Source: FTD 66 month survey, FTD 66 month proxy survey, FTD Administrative dataNotes: Estimates regression adjusted. Sample sizes vary for individual measures because of missing values. Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent. Rounding may cause slight discrepancies in sums and differences.

New Brunswick

Fr-LA-Eligible En-LA-Eligible

No PSE credential

26

Table 2: LA IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION


ENROLLED IN UNIVERSITYALL 29.42 25.75 3.68 25.55 26.28 -0.73

(4.08) (3.96)LILE 25.35 17.74 7.61 * 22.45 23.04 -0.59

(4.36) (4.41)Parents with High school or less (FGF) 22.52 16.52 6.01 24.51 18.11 6.40

(5.16) (5.48)Parents with any PSE 37.99 34.42 3.56 25.75 35.21 -9.46

(6.54) (5.92)Boys 16.81 18.03 -1.22 12.21 13.71 -1.50

(5.36) (4.95)Girls 39.59 33.35 6.24 36.97 34.37 2.60

(5.37) (5.77)ENROLLED IN COLLEGEALL 49.20 36.56 12.64 ** 32.68 25.48 7.20 *

(4.98) (3.96)LILE 51.48 35.42 16.07 *** 31.96 24.50 7.46 *

(5.28) (4.16)Parents with High school or less (FGF) 46.18 28.79 17.39 *** 28.34 26.61 1.73

(6.22) (6.52)Parents with any PSE 54.51 43.27 11.24 35.70 24.74 10.96 *

(7.84) (5.90)Boys 57.18 35.97 21.21 *** 34.25 25.43 8.82

(7.59) (6.71)Girls 44.66 34.96 9.70 29.56 27.34 2.21

(6.44) (5.87)

New Brunswick


Why did Learning Accounts work in Francophone sector?

Possible explanation: supply constraints

Data from New Brunswick government:

• Anglophone programs were oversubscribed• Francophone programs were undersubscribed

27

28

LA IMPACTS ON PSE APPLICATIONS BY TYPE OF INSTITUTION


APPLIED TO UNIVERSITYALL 32.08 25.31 6.77 * 30.22 28.18 2.04

(4.08) (3.98)LILE 28.91 17.51 11.40 ** 26.69 24.72 1.97

(4.44) (4.58)Parents with High school or less (FGF) 23.78 15.72 8.06 28.32 16.45 11.87 **


(6.61) (5.94)Boys 19.68 17.71 1.98 16.66 15.26 1.40

(5.43) (6.08)Girls 42.29 33.02 9.26 * 40.26 38.20 2.06

(5.50) (6.12)APPLIED TO COLLEGEALL 48.89 35.26 13.63 *** 41.64 27.93 13.71 ***

(4.64) (4.21)LILE 51.92 33.21 18.71 *** 42.80 28.55 14.26 ***

(5.16) (4.71)Parents with High school or less (FGF) 48.87 25.82 23.05 *** 40.51 29.10 11.40

(6.25) (7.35)Parents with any PSE 50.04 44.88 5.16 41.88 27.42 14.46 **

(7.54) (6.27)Boys 56.89 34.90 21.99 *** 43.04 24.08 18.96 ***

(7.74) (6.88)Girls 43.25 34.38 8.87 40.67 30.72 9.95

(6.31) (6.52)

New Brunswick


29

Table 3: EYH IMPACTS ON PSE ENROLMENT

EYH Comparison Impact EYH Comparison Impact EYH Comparison ImpactGroup Group (s.e) Group Group (s.e) Group Group (s.e)

Enrolled in PSE institution (%)ALL 72.62 67.82 4.80 81.73 77.54 4.19 * 74.10 70.26 3.85

(3.04) (2.31) (2.63)LILE 61.43 52.88 8.55 74.01 60.63 13.38 *** 63.87 55.52 8.35 *

(6.89) (5.01) (4.66)NOT LILE 76.32 73.78 2.54 83.81 88.13 -4.32 80.42 79.69 0.74

(3.52) (2.63) (3.20)Parents with High school or less (FGF) 62.90 55.64 7.27 72.38 59.42 12.96 *** 58.98 56.15 2.83

(6.27) (5.01) (5.54)Parents with any PSE 76.83 72.46 4.36 84.99 86.95 -1.97 80.87 76.76 4.11

(3.45) (2.47) (2.73)Boys 66.68 57.53 9.15 * 77.64 71.55 6.09 67.69 60.70 6.99 *

(4.86) (3.96) (4.06)Girls 78.66 77.53 1.12 86.68 81.81 4.86 * 80.29 79.20 1.08

(3.82) (2.87) (3.21)Aboriginal 63.72 61.67 2.06 . . . . . .

(11.13) . .Sample size 478 395 484 677 471 646Source: FTD 66 month survey, FTD 66 month proxy surveyFTD Administrative data.Notes: Estimates regression adjusted. Sample sizes vary for individual measures because of missing values. Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent. Rounding may cause slight discrepancies in sums and differences.

ManitobaNew Brunswick

Francophone Anglophone

30

Table 4: EYH IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION

EYH Comparison

Impact EYH Comparison

Impact EYH Comparison

ImpactGroup Group (s.e) Group Group (s.e) Group Group (s.e)

ENROLLED IN UNIVERSITYALL 51.33 47.40 3.93 48.74 43.78 4.97 * 43.13 41.04 2.08

(3.20) (2.71) (2.51)LILE 34.55 33.78 0.77 33.05 19.10 13.95 *** 29.76 22.01 7.74 *

(6.81) (4.28) (4.43)NOT LILE 57.41 53.48 3.93 55.92 58.23 -2.31 49.51 53.05 -3.54

(3.79) (3.46) (3.62)Parents with High school or less (FGF) 34.69 32.81 1.87 29.86 22.87 7.00 17.54 20.92 -3.38

(5.85) (4.61) (4.20)Parents with any PSE 57.94 53.42 4.52 57.33 54.07 3.26 53.81 50.25 3.56

(3.94) (3.47) (3.25)Boys 42.31 35.45 6.86 38.14 34.81 3.33 34.02 29.95 4.07

(4.63) (4.16) (3.93)Girls 60.40 58.87 1.53 58.69 51.42 7.26 * 51.54 51.49 0.04

(4.25) (3.71) (3.82)Aboriginal 35.87 36.41 -0.54 . . . . . .

(12.24) . .ENROLLED IN COLLEGEALL 25.23 20.33 4.89 38.47 36.97 1.50 27.57 24.19 3.38

(3.00) (3.00) (2.63)LILE 25.70 15.17 10.53 * 41.19 34.73 6.46 25.50 24.89 0.61

(5.92) (5.42) (4.47)NOT LILE 24.84 21.72 3.12 36.10 38.90 -2.80 29.84 24.04 5.80 *

(3.55) (3.38) (3.30)Parents with High school or less (FGF) 29.76 15.42 14.35 ** 38.77 30.59 8.18 29.75 26.80 2.94

(5.97) (5.31) (5.78)Parents with any PSE 23.60 21.76 1.84 37.62 40.36 -2.74 26.87 23.15 3.72

(3.32) (3.48) (3.12)Boys 24.63 17.87 6.76 46.65 38.44 8.21 * 28.72 26.95 1.77

(4.14) (4.65) (4.11)Girls 26.35 22.16 4.19 32.48 34.30 -1.82 26.33 21.78 4.55

(4.40) (3.79) (3.75)Aboriginal 29.53 23.89 5.64 . . . . . .

(11.74) . .

ManitobaNew Brunswick

Francophone Anglophone

31

Table 5: EYHLA IMPACTS ON PSE ENROLMENT

EYHLA Comparison Impact EYHLA Comparison ImpactGroup Group (s.e) Group Group (s.e)

Enrolled in PSE institution (%)ALL 72.60 66.23 6.37 69.41 59.41 10.00 ***

(4.42) (3.80)LILE 72.14 60.62 11.52 ** 66.52 55.01 11.50 **

(4.97) (4.51)Parents with High school or less (FGF) 70.86 56.72 14.15 ** 60.89 53.77 7.11

(6.81) (6.54)Parents with any PSE 75.29 76.14 -0.85 75.67 67.01 8.66

(6.06) (5.27)Boys 62.42 59.78 2.64 67.88 41.75 26.13 ***

(7.35) (6.39)Girls 84.52 70.11 14.41 ** 70.41 73.10 -2.69

(5.63) (5.44)Sample size 244 262 237 255Source: FTD 66 month survey, FTD 66 month proxy survey, FTD Administrative dataNotes: Estimates regression adjusted. Sample sizes vary for individual measures because of missing values. Statistical significance levels are indicated as * = 10 per cent; ** = 5 per cent; *** = 1 per cent. Rounding may cause slight discrepancies in sums and differences.

New Brunswick


32

Table 6: EYHLA IMPACTS ON PSE ENROLMENT BY TYPE OF INSTITUTION

EYHLA Comparison Impact EYHLA Comparison ImpactGroup Group (s.e) Group Group (s.e)

ENROLLED IN UNIVERSITYALL 31.89 24.30 7.59 * 32.32 25.54 6.78 *

(4.03) (3.76)LILE 26.32 16.69 9.63 ** 28.97 21.63 7.34 *

(3.98) (4.09)Parents with High school or less (FGF) 24.66 16.79 7.88 * 23.35 16.23 7.12

(4.49) (4.85)Parents with any PSE 38.22 33.68 4.54 41.14 34.58 6.56

(6.30) (5.92)Boys 22.48 17.28 5.20 26.79 13.64 13.15 **

(5.63) (5.43)Girls 40.04 31.54 8.49 39.25 33.55 5.71

(5.67) (5.40)ENROLLED IN COLLEGEALL 42.52 38.17 4.34 24.79 24.99 -0.20

(5.15) (3.78)LILE 43.30 36.58 6.72 26.30 24.62 1.68

(5.61) (4.15)Parents with High school or less (FGF) 42.38 31.37 11.01 24.94 27.83 -2.88


(7.09) (5.94)Boys 40.29 37.64 2.64 30.29 22.06 8.23

(7.76) (6.38)Girls 46.21 37.25 8.96 17.78 28.04 -10.26

(6.83) (6.26)

New Brunswick


Potential biases/evaluation challenges Program take-up/drop-out

• Everyone received offer of LA, but about 30% did not remember a couple of years later• Attendance in EYH workshops was poor, especially in Manitoba

– Non-mandatory, after school

Spillover effects

• Not an issue for LA• EYH workshops held after class

Substitution bias (displacing non-experimental treatment)

• Minor impact in LA (small reduction in non-repayable aid for some groups)• EYH: looking into it

Attrition bias

• Enrolment numbers use administrative data

33

Potential biases/evaluation challenges

Reactions to inequity caused by randomization

• Positive or negative impact on control group• No mechanism for testing this (did not follow non-experimental schools)

Sample size

• Affects statistical significance (especially in sub-group analysis)

34

Future to Discover Final Report

Final report (including cost-benefit study) in September 2012

35

Appendix: Related academic interventions

QOP (Hahn et al., 1994; Rodriguez-Planas, 2010)• Includes financial incentives

IHAD (Kahne and Bailey, 1999)• Includes early promise of ‘last dollar scholarship’

Career Academies (Kemple and Willner, 2008)• Focused on career themes

GEAR UP

Upward Bound (Seftor et al., 2009)

Upward Bound Math-Science (Seftor and Calcagno, 2010)

36

Treatment broadly similar to AVID, but focus on low-income students

marc frenette

Documents

untracking students

avid helps students

lowincome students

students dunn

important barriers

avid teacher

avidcan avid

local college students