real caine rolleston
TRANSCRIPT
Assessment for Effectiveness and Equity:
Lessons from a Longitudinal Study
Caine Rolleston
REAL, CambridgeJune 15, 2016
• Extent of ‘learning crisis’ becoming clearer – e.g. from citizen led assessments UWEZO etc.
• Cross-sectional data & benchmarking key altho still absent in many contexts
• But other key questions (especially effectiveness/equity) require more sophisticated data, including longitudinal or combined household and school data:
• Which schools/school systems are more effective (add more value) & why?
• To what extent a crisis of school quality? (not only learning)• When do gaps arise/develop during the life-course?• What are the causes of/remedies for poor attainment?• Cost-effectiveness, intervention choices
• Some potentially require combined longitudinal HH plus longitudinal school data – E.g. Are schools equally effective for more and less advantaged pupils?
ASSESSMENT IN DEVELOPING COUNTRIES
Young Lives longitudinal survey of children, households & communities every 3 years since 2002
• 12,000 index children in two cohorts (now aged 13 & 19)
• Ethiopia, India, Peru, Vietnam• 20 sentinel sites in each country• Tested in maths at each round with common
items• Primary school surveys implemented since
2010• Secondary school surveys from 2016
Allows comparison of • Learning levels• Learning trajectories• Change over time between cohorts
YOUNG LIVES STUDY
YOUNG LIVES STUDY
BENCHMARKING COMPETENCY LEVELS (ETHIOPIA)Competency level Number of Pupils Percentage of
PupilsCumulative Percentage
0 (Below Level 1) 853 8.66 8.661 (Early Foundational) (~ G1) 2,121 21.54 30.212 (Foundational) (~G2-3) 5,152 52.33 82.533 (Emerging) (~G3-4) 1,473 14.96 97.494 (Grade level) (~G4-5) 247 2.51 100.00Total 9,846 100.00
020
4060
8010
0pe
rcen
t
Rural Urban5 4 5 4
Level 0 Level 1Level 2 Level 3Level 4
LONGITUDINAL STUDY WITHIN A STUDY
.000
5.0
01.0
015
.002
.002
5D
ensi
ty
200 400 600 800Score
Test 1 Test 2
Maths Scores at First and Second Round Tests
• Mean Test 1 = 500, SD= 100. Mean Test 2=530 (gain 0.3 SD)
MEASURING PROGRESS OVER THE SCHOOL YEAR (ETHIOPIA)
INTERVALE-SCALE METRICS
Mathematics Reading Comprehension
October 2012
May 2013
Gain October 2012
May 2013
Gain
Mean
500.0 530.0 30.0 500.0 530.6 30.6
Gender
Boy 502.1 532.5 30.4 498.0 527.5 29.5
Girl 497.9 527.6 29.7 501.9 533.6 31.7
Difference 4.2 4.9 0.7 -3.9 -6.1 -2.2
Location
Urban 517.6 548.5 30.9 521.1 551.1 30.0
Rural 456.4 481.2 24.8 447.3 476.6 29.3
Difference 61.2 67.3 6.1 73.8 74.5 0.7
PREDICTORS OF ATTAINMENT AND PROGRESS OVER TIME (ETHIOPIA)
VARIABLES Maths T1 Maths T2 Reading T1 Reading T2
Girl -10.2031 *** -4.4330 *** 0.1650 4.8107 ***
Has 3+ meals per day 18.2975 *** 6.6820 *** 12.6032 *** 2.1339
PCA pupil durable assets 4.5648 *** 1.7210 *** 4.9427 *** 0.8929 *
% days absence W1-W2 -3.3064 *** -1.8387 *** -3.1255 *** -1.4661 ***
Orphan (single or double) 2.8872 -3.2279 * 2.6308 -3.3137 **
No-one in household literate -9.1339 ** -5.7685 * -14.3049 *** -1.8481
Attended pre-school 2.6854 -0.3207 6.3172 *** 3.3394 **
Ever repeated a grade -39.6920 *** -5.0135 *** -38.7614 *** -4.2894 ***
Ever dropped-out -6.1516 ** -2.1293 -13.4413 *** -3.7113 **
Reads books at home 17.5734 *** 6.2704 *** 17.3460 *** 3.6314 *
Child learns in home language 3.8083 -2.2792 14.5838 *** 5.4488 *
Pastoralist -24.1437 *** -4.3839 -37.9640 *** -8.0455 **
Pupil spends time on paid work -10.4830 *** 0.2447 -12.1998 *** -1.0128
SCHOOL-LEVEL VALUE-ADDED (ETHIOPIA)
-100
-50
050
100
Sch
ool V
alue
-Add
ed
0 20 40 60 80 100School Rank
Urban Rural
DUAL COHORT STUDY WITH A 7 YEAR INTERVAL:IMPROVING TEST SCORES, MIXED PATTERNS OF
EQUITYPeru Vietnam
Test score gains with/without equity improvement
Children aged 12 in 2006 and 2013
DECLINING TEST SCORES WITH WIDENING INEQUALITY
India (AP) EthiopiaChildren aged 12 in 2006 and 2013
020
4060
80M
edia
n M
aths
Sco
re R
3 %
0 20 40 60 80 100CDA-Q Score R2 %
Ethiopia PeruIndia Vietnam
WIDENING GAPS ARE DRIVEN BY DIFFERENCES IN LEARNING PROGRESS OVER TIME BETWEEN SYSTEMS:
AGE 5 TO 8
0.2
.4.6
01
23
4
-2 0 2 4 -2 0 2 4
Ethiopia Vietnam
Most Poor Least Poor
Mat
h sc
ores
(201
3)
Math scores (2009)
Graphs by country
Maths scoresLearning divergence by wealth groups
• Steeper gains by prior score in Ethiopia, broadly similar gaps by wealth• Pupils typically make more gains over time in Vietnam
• Distribution of school quality accentuates disadvantage
• E.g. differences in school quality explain more of the differences in test scores in India and Peru than Vietnam (schools are more heterogeneous)
• Differential effectiveness may mean triple disadvantage (two ways in which schools widen gaps)
• Reasons could include elitism, curriculum, language of instruction, discrimination
• A recent study Glewwe, Krutikova & Rolleston (EDCC, forthcoming) compares Vietnam and Peru – using both longitudinal household and school data
EQUITY ISSUES
Vietnam Peru-0.3
-0.2
-0.1
0
0.1
0.2
0.3
0.4
0.5
-0.18
0.41*
Proportion of 1 SD of maths test score distribut...
Difference in test score gains by increasing school quality by 1SD between richest 40% and poorest 60% of pupils
In Vietnam, schools equally effective in teaching Maths
In Peru, schools significantly less effective at teaching children from disadvantaged backgrounds
WITHIN THE SAME SCHOOL, DISADVANTAGED PUPILS MAKE LESS PROGRESS IN PERU, BUT NOT
IN VIETNAM
FINDING OUT MORE