Ms Marie-Louise Samuels and

Dr Stephen Taylor

28 APRIL 2015

Constitutional Hill

MOVING FROM EVALUATION TO

IMPLEMENTATION

Presentation at the DPME/DSD Roundtable on emerging evidence on impact of programmes on wellbeing

of young children

OVERVIEW

• Introduction to the Grade R programme

• Impact Evaluation of Grade R on learning

outcomes

• Data, methodology and results

• Improvement plan

• Reflections on the whole evaluation

process

• Progress to date

1994 1999 2004 2009 Pre-1994 2014

• Racial discrimination of ECD provisioning

• 75% fee-based

• 80% private and 20% public

• Audit of Early Childhood Development Policies

• Nationwide Audit of ECD services

• Education White Paper 5 on ECE

• Conditional Grant

• Children’s Act

• Regulations

• National Integrated Programme of Action

• Outcome 1

• NELDS

• NDP

• Education White Paper 1 on Transformation in Education

• Interim Policy on Early Childhood Education

• National Pilot Project

• Equitable share (Grade R)

• National Integrated Plan for ECD

• Tshwaragano Ka Bana

• ECD International Conference

• Expanded Public Works Programme

• ECD Policy and Programmes

• National Curriculum Framework for Children from Birth to Four

• Outcome 13

LOOKING BACK, LOOKING AHEAD

ACCESS TO GRADE R

CHALLENGE - QUALITY

THE IMPACT EVALUATION

• Department of Planning, Monitoring and

Evaluation (DPME) in the Presidency works

together with other government departments to

evaluate key programmes and policies

• In 2011, The Grade R programme was selected

as one of the first evaluations on the NEP.

• A team of researchers from the University of

Stellenbosch was contracted to conduct an

impact evaluation.

• This impact evaluation of the Grade R

programme has now been completed, has been

presented to Cabinet, and is publicly available.

THREE KEY FACTS EMERGE

7

1. A person’s life trajectory is

established very early

2. Family wealth matters for young

children

3. It is possible to intervene effectively &

to improve the trajectories of young

children, but the later the remediation,

the less effective it is

DATA SET

• Dataset of 18102 schools

• Obtained by merging SNAP data on learners registered for each grade, test data from Annual National Assessments (ANA) of 2011 and 2012, and EMIS Masterlist

• ANA provides data on performance in maths and home language for Grades 1 to 6; – Converted to normalised score with mean 0 and standard deviation of 1 for

each grade, to make scores comparable (in relative terms) across grades

• EMIS provides school quintile and school fees – School fees in 2007 provide a measure of affluence and resources

• Large datasets allows more precise estimation of effect sizes

8

MEASURING TREATMENT

Treatment measure: % of learners in a given cohort that attended Gr R

Limitations:

• School may provide Grade R to a wider catchment area, i.e. learners move to other schools after Gr R

• Ratio exceeds 100% in some cases; we top censor these measures to be at most 100%

• Some Gr X learners may have attended Gr R at another educational facility

• Where data for Grade R are missing, we assume no treatment

9

METHODOLOGY

% OF LEARNERS WITH GRADE R BY YEAR & SCHOOL QUINTILE

11

School quintile 2005 2006 2007 2008 2009 2010 2011 2012

1 43.9 51.4 63.1 72.0 79.2 83.0 85.4 86.1

2 45.3 53.6 65.1 74.5 82.3 87.0 89.2 90.3

3 50.7 59.8 67.7 73.8 80.3 85.2 87.3 89.2

4 54.9 60.8 66.4 71.3 76.4 79.6 82.3 84.2

5 57.3 60.5 64.0 66.0 71.5 75.7 77.3 78.9

All 48.0 55.1 64.2 71.7 78.5 83.1 85.3 86.6

12

AVERAGE TREATMENT BY SCHOOL QUINTILE AND GRADE

DETERMINING CAUSAL IMPACT

• Other factors may also influence outcomes – Some we can control for (e.g. SES) – 0bservables – Others we cannot – Unobservables

• Example: Factors that could affect both treatment and learning, e.g.: – Better managed schools may more easily introduce Gr

R, and would usually have better learning outcomes – Departments may encourage weaker schools more to

introduce Gr R

14

TREATMENT AND RESULTS ACROSS SCHOOLS AND GRADES

15

Re

lati

ve p

erf

orm

ance

Treatment (attending Gr R)

TREATMENT AND RESULTS ACROSS SCHOOLS AND GRADES

16

Re

lati

ve p

erf

orm

ance

Treatment (attending Gr R)

FIXED EFFECTS: EACH SCHOOL TREATED SEPARATELY TO ELIMINATE UNOBSERVABLES

17

Re

lati

ve p

erf

orm

ance

Treatment (attending Gr R)

School A

School B

School C

School D

School E

FIXED EFFECTS: EACH SCHOOL TREATED SEPARATELY TO ELIMINATE UNOBSERVABLES

18

Re

lati

ve p

erf

orm

ance

Treatment (attending Gr R)

School A

School B

School C

School D

School E

INTERPRETING EFFECT SIZES

• Treatment effects are measured in standard deviations (SD) of test scores

– Coefficients on the treatment variable reflect the effect of a change from zero treatment to full treatment, i.e. from having no children attending Grade R to having all children attending

• International literature assumes that a year’s learning improves test scores by ± 40% to 50% SD

– But NSES data indicate SA learning might only be 33% SD in a year (less than 18% in weakest 3 provinces)

19

EFFECT OF TREATMENT (FIXED EFFECTS MODEL)

• Maths score +2.5% of a standard deviation (SD) for 2012 sample • Home language score +10.2% SD

– Assuming 40% SD to be equivalent to one grade level in school, this is equivalent to

• 6% of a year of learning in maths, or what average learner should learn in 12 days (if a school year is 200 days of instruction)

• 25% for home language, or 50 days of learning

– No clear evidence of fade-out

• Quite small effects: – A review of programmes in USA found average effects on cognitive

outcomes of 42% SD – Oklahoma’s high quality universal preschool programme saw

• an 80% SD gain in pre-reading and reading skills • a 65% SD gain in pre-writing and spelling skills, and • a 38% SD gain in early math reasoning and problem-solving

– In Argentina, one year of pre-primary increased average third grade test marks in maths and Spanish by 23% SD

20

21

OLS regression

Pooled 2011 2012

Dependent variable: Standardised Mathematics test score

Treatment 0.159** 0.199** 0.145**

School fixed effects No No No

Grade fixed effects Yes Yes Yes

Observations 47694 14954 32740

R-squared 0.230 0.267 0.223

Dependent variable: Standardised Home Language test score

Treatment 0.151** 0.153** 0.165**

School fixed effects No No No

Grade fixed effects Yes Yes Yes

Observations 47696 14957 32739

R-squared 0.315 0.306 0.338

Ordinary least squares and school fixed effects regression results

Note: * p<0.05, ** p<0.01; robust standard errors in parentheses.

22

OLS regression School fixed effects

Pooled 2011 2012 Pooled 2011 2012

Dependent variable: Standardised Mathematics test score

Treatment 0.159** 0.199** 0.145** 0.053** 0.074** 0.025*

School fixed effects No No No Yes Yes Yes

Grade fixed effects Yes Yes Yes Yes Yes Yes

Observations 47694 14954 32740 129410 41451 87959

R-squared 0.230 0.267 0.223 0.001 0.001 0.000

Dependent variable: Standardised Home Language test score

Treatment 0.151** 0.153** 0.165** 0.093** 0.060** 0.102**

School fixed effects No No No Yes Yes Yes

Grade fixed effects Yes Yes Yes Yes Yes Yes

Observations 47696 14957 32739 129419 41461 87958

R-squared 0.315 0.306 0.338 0.001 0.001 0.001

Ordinary least squares and school fixed effects regression results

Note: * p<0.05, ** p<0.01; robust standard errors in parentheses.

SA effect sizes in comparison (in % SD)

2.5 20.3

10.1

-0.8

0.0

1.5

10.2 19.4

11.5

-0.2

7.7

1.7

40 42

80

65

38

23

-10 0 10 20 30 40 50 60 70 80

Maths: All

Maths: Q5

Maths: Q4

Maths: Q3

Maths: Q2

Maths: Q1

Home Language: All

Home Language: Q5

Home Language: Q4

Home Language:Q3

Home Language: Q2

Home Language: Q1

One year of learning

US average preschool

Oklahoma pre-writing & spelling

Oklahoma pre-reading

Oklahoma early maths reasoning

Argentina Gr 3 Maths & Spanish

Impact by provincial and quintile groups: Fixed effects estimates (in % SD)

24

CONCLUSIONS ABOUT GRADE R IMPACT

1. Dataset enables estimation of effects with great

accuracy

2. Fixed effects control for unobserved

heterogeneity (endogeneity), thus causal effects

can be estimated

3. Grade R has had a positive impact on learning

Effects may be lasting: little sign of fade-out (decay)

in higher grades

Channels not clear, however (e.g. role of nutrition/

school feeding)

Negligible effects in bottom quintiles

25

CONCLUSIONS ABOUT GRADE R IMPACT

4. Measured effects are small for full sample Maths: overall effect less than 1 month worth of learning

2.5% SD for 2012

Home Language: ± 2 months 10.2% SD

5. Effects stronger for better performing

provinces & for higher quintiles: they share

characteristics in programme delivery that have a

positive impact

6. Thus programme quality needs to be

prioritised

26

RECOMMENDATIONS

1. Grade R underfunded according to DBE’s own

criteria (30% rather than 70% of other learners),

with large inter-provincial differences

Quality requires threshold levels of funding of both

personnel and LTSM – need to ensure this

Provinces must ensure Grade R is not crowded out by

other spending

2. Funding levels obscured by not disaggregating

funding within schools, using different categories

Needs greater attention at school and provincial level

27

RECOMMENDATIONS CONT.

3. A quality year of preschool is critical to effectively manage the transition to Grade 1 Close monitoring of teaching and learning in Gr R justified and

possible; requires dedicated personnel Develop common tools to assess language, emergent literacy, maths

development

Establish quality criteria for schools & ECD centres to self-assess & for M&E

4. More specific developmental research needed to inform educational strategies

5. Grade R curriculum has key role to play in closing the gaps Indicators which reflect elements of quality Grade R need to recognise the

critical importance of mediated language enrichment

Provide on-going structured curriculum support for teachers in implementing CAPS, with practical ideas on ‘how’ to achieve learning outcomes

Increase opportunities for in-service training focused on providing teachers with practical strategies for supporting early learning & opportunities to see & practice best teaching

Develop/evaluate evidence-based learning programmes & resources designed for the local context & appropriate for children from poor backgrounds

KEY ELEMENTS OF THE IMPROVEMENT PLAN

ELEMENT PROGRESS

Interim Grade R policy The recommendations on the amendment of legislation were presented to the Ministerial Task Team for consideration.

The development of a human resource development strategy.

A Training and Curriculum committee consisting of key stakeholders has been established as a sub-committee of the National Interdepartmental Committee. The HRD plan is part of the committee’s responsibilities. 3, 860 teachers have been enrolled in a course towards a Level 6 qualification

KEY ELEMENTS OF THE IMPROVEMENT PLAN

ELEMENT PROGRESS

The development of a programme to support curriculum implementation in all Grade R classes

Established a Foundation Phase Subject Committee. Aligned the Grade R resource pack to the National Curriculum Statement. Delivered Grade R workbooks Training of Grade R practitioners towards the Level 6 qualification.

The development of a monitoring and evaluation system integrated with the other objectives developed at all levels, including Grade R in the Annual National Assessment (ANA) system.

Activity has been included in the 2015/16 plan for the DBE.

CONCLUSION

Website: www.education.gov.za

Call Centre: 0800 202 933 | callcentre@dbe.gov.za

Twitter: @DBE_SA | Facebook: DBE SA

THANK YOU

32