ecec research, evaluation and monitoring what should be

ECEC Research, Evaluation and Monitoring What should be monitored, why, and how?

Building on SSIII

“…information is lacking across all areas of ECEC provision…”

“…we not only need to develop and build extensive data bases that can connect information across ECEC sectors, but we must ensure that these are connected with rigorously designed research studies if we are to use the data to inform programme improvements leading to increasing the effectiveness of our early learning efforts globally…”

Public Policy

Public Policy

Research can point us towards what is optimal…

…but public policy must deal in trade-offs

Where to spend dollars, euros, or political capital is easy

Where to spend your next dollar is more important

…and how you spend your existing dollars is probably more important than that

Public Policy

0

0,5

1

1,5

2

2,5

3

3,5

4

4,5

5

Level 2

Level 3/4

Level 5

ECERS-E Literacy Sub-Scale Scores by Manager Qualification

Derived from EPPE (Quality in Early Childhood Settings, Kathy Sylva)

Public Policy

0

5.000

10.000

15.000

20.000

25.000

2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012

Qualified

Unqualified

Numbers of New Zealand ECE Teachers, Qualified and Unqualified

Public Policy

Poor Fair Good Very good

05

10

15

20

25

2004

2006

2009

Overall quality shifts 2004 to 2006 to 2009

Public Policy

Expenditure, $ per full time equivalent child

$0

$2.000

$4.000

$6.000

$8.000

$10.000

$12.000

2001/02 2002/03 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12

Public Policy

% of services with more than 80% of teachers qualified, by SES decile

40%

42%

44%

46%

48%

50%

52%

54%

56%

1 (highest SES)

2 3 4 5 6 7 8 9 10 (lowest SES)

Policy Questions

What works?

What works best?

How do Governments make it work?

How are the gains distributed?

Does our existing body of knowledge enable us to

answer the key policy questions?

Where we get our information – what it tells us

What We Know at the Moment

Research – a number of significant studies have highlighted, across a wide variety of contexts, the importance of the early years

e.g. Strengths and Needs in the Early Years NZ study found statistically significant improvements in reading scores for children attending ECE (but no correlation with disruptive behaviour)

Evaluation – large scale evaluations have considered and highlighted important factors for delivering gains (access, quality)

e.g. the NZ evaluation of the ECE Strategic Plan found correlations between observed process quality and improved structural quality where more qualified teachers were present (but no consistent effect for better ratios)

What Does ECEC Look Like?

Intervention (e.g. Perry, Abecedarian)

Programme (e.g. Head Start, Sure Start)

System (e.g. schools)





Very small scale, experimental design delivered with high control / fidelity

Broader scale, tends to be targeted, delivered with a range of variability

Near-universal, commonly understood, features of boundary object, politically significant





Research

Evaluation

Monitoring

Monitoring

Monitoring

Frequent

Long-term

High cost

Financial

Compliance

Highly complex systems required

Difficult to focus – e.g. easy to collect what’s easy, not what’s useful

Best Practice

www.ecedata.org

Early Childhood Data Collaborative

Are children, birth to age 5, on track to succeed when they enter school and beyond?

Which children have access to high-quality early care and education programs?

Is the quality of programs improving?

What are the characteristics of effective programs?

How prepared is the early care and education workforce to provide effective education and care for all children?

What policies and investments lead to a skilled and stable early care and education workforce?

Early Childhood Data Collaborative

Unique statewide child identifier

Child-level demographic and program participation information

Child-level data on child development

Ability to link child-level data with K–12 and other key data systems

Unique program site identifier with the ability to link with children and the ECE workforce

Program site data on structure, quality and work environment

Unique ECE workforce identifier with ability to link with program sites and children

Individual ECE workforce demographics, including education, and professional development information

State governance body to manage data collection and use

Transparent privacy protection and security practices and policies

Examples of data collection structures

Types of national data collection by subject area

0 1 2 3 4 5 6 7

Overall

Demographic

Attendance

Service quality

Workforce

Outcomes / achievement

Health

Individual

Aggregate

Case Study – New Zealand

Current state –

two data collections, few linked data, no child-level data

paper-based collection

most data from one-week census

Problems include -

Infrequent, inaccurate data; hard to react or assess impact of policy

We don’t know how many children there are or what they’re up to

Case Study – New Zealand

Early Learning Information System

Improve Knowledge Help to link ECE information

Help to target resources better Achieve 98% participation

- $25m project - Idea to implementation – 6 years

- Hugely complex - Enormous change programme for

5,000 providers

Case Study – Singapore

Target of 98.8% participation of 6yos by 2021

92

93

94

95

96

97

98

99

100

2006 2007 2008 2009 2010 2011

6YO children attending pre-school (%)

Case Study – Singapore

Strong reliance on data use and matching – use of cross-Government administrative data matching

Matched with other strategies:

Outreach (home visiting)

Awareness, information and direct contact

Pre-school placement

International Monitoring


Focus mostly on indicators from national administrative data

Or trans-national studies (PISA, PIRLS, TIMMS)

Increases reliability and comparability

Extends sample size, e.g.


Focus mostly on indicators from national administrative data

Or trans-national studies (PISA, PIRLS, TIMMS)

Increases reliability and comparability

Extends sample size, e.g.

Ratios for 3 year olds

In NZ, I can experiment on a range between 1:7 and 1:15

Across the OECD, I can experiment between 1:6 and 1:25

International Indicators

How good is our ECEC system?


0

10

20

30

40

50

60

70

80

90

100

New Zealand Country A Country B Country C


Excellent

Rubbish


0

10

20

30

40

50

60

70

80

90

100



Excellent

Rubbish

Is this a useful, representative measure of how good the system is? (What works?)


0

10

20

30

40

50

60

70

80

90

100



Excellent

Rubbish

Is this country really half as good as New Zealand? (What works best?)

Improving Indicators

0

10

20

30

40

50

60

70

80

90

100



Excellent

Rubbish

Should this country be trying to get to 100? Or is 80 good enough? Are the greatest gains between 0 and 50, or 90 and 100? (What works best?)


0

10

20

30

40

50

60

70

80

90

100



Excellent

Rubbish

How many children in this country get 40? Do some get 20 and some 60? If so, who, and how much? (How are the gains distributed?)


0

2

4

6

8

10

12

14

16

18

20

22

24

26

Chart C2.3. Ratio of students to teaching staff in early childhood education (2010)Public and private institutions

Countries are ranked in descending order of students to teaching staff ratios in early childhood education.

Source: OECD. Argentina and Indonesia: UNESCO Institute for Statistics (World Education Indicators Programme). Table C2.2. See Annex 3 for notes (www.oecd.org/edu/eag2012).

Student to teaching staff ratio

Improving Indicators - conclusions…

We can probably collect quite a lot of comparative data regularly:

Levels of qualification

Mixture of qualification

Group size and ratio

Space and facilities

But it’ll need additional contextual data to make it really useful

Challenges – Answering Policy Questions

How do Governments make it work?

Outcomes


Marginality

Impacts

Balance of interventions

Funding

Taxation

Regulation

Information

Relationships

Choice architecture


0,0

10,0

20,0

30,0

40,0

50,0

60,0

70,0

80,0

90,0

100,0

$0

$200.000

$400.000

$600.000

$800.000

$1.000.000

$1.200.000

$1.400.000

$1.600.000

$1.800.000

2001/02 2002/03 2003/04 2004/05 2005/06 2006/07 2007/08 2008/09 2009/10 2010/11 2011/12

Government Expenditure % of 4YOs participating

New Zealand – participation and expenditure


• Funding • Regulation • Information • Staff • Qualifications • Resources • Curriculum



• Responsive interactions

• High quality attachments

• Safe and welcoming environments






• Child wellbeing

• Child learning






• Child wellbeing

• Child learning

(Structural Quality) (Process Quality)






• Child wellbeing

• Child learning

To find out what works and what works best, we need information on all of these stages






• Child wellbeing

• Child learning

(Structural Quality) (Process Quality)

Almost all of our current data sit here


350

370

390

410

430

450

470

490

510

530

550

Did not attend One year or less More than one year

Pasifika

Non-Pasifika

PISA 2009 – reading point score difference from attending ECE after controlling for SES


PISA 2009 – reading point score difference from attending ECE after controlling for SES

400

420

440

460

480

500

520

540

560

Did not attend One year or less More than one year

Māori

Non-Māori

ecec research, evaluation and monitoring what should be

Documents