Measuring Early Learning Quality and Outcomes (MELQO)

Process and Lessons Learned

May 27, 2015

Amanda Devercelli, Abbie Raikes, Kate Anderson

Goals of MELQO-What did we start out trying to solve?Calls for globally comparable measures of young children’s

development • (SDGs, impact investors, making measurement more effective, etc)

Expensive and/or difficult to adapt tools

Need for tools that can be used from settings level up to systems level (esp. true in context of World Bank projects)

Linking quality of settings with child development outcomes

Can we build on what exists to achieve this?

What do the MELQO instruments cover?

• Child Development • Parent/Teacher report and direct assessment

• Constructs• social-emotional

• domain specific (early literacy and math)

• executive functioning/approaches to learning

• Quality of Settings• Looking at setting and systems

• Trying to find the middle ground

Steering Committee(UNESCO, UNICEF, World Bank, Brookings)• UNESCO (Maki Hayashikawa, Abbie Raikes)

• UNICEF (Jo Bourne, Pia Britto)

• World Bank (Luis Benveniste, Amanda Devercelli)

• Center for Universal Education at Brookings (Tamar Manyuelyan Atinc, Kate Anderson)

• GPE (Karen Mundy)

• UIS (Silvia Montoya, Albert Motivans)

Technical Advisory Group-Child Development and Learning

• Mariavittoria Ballotta, UNICEF WCARO

• Ivelina Borisova, Save the Children

• Sally Brinkman, University of Adelaide

• Magdalena Janus, McMaster University

• Sylvia Linan-Thompson, University of Texas at Austin

• Linda Platas, UC Berkeley

• Nirmala Rao and Yvonne Belcher, Hong Kong University

• Aimee Verdisco, Inter-American Development Bank

• Michael Willoughby, RTI

• Child Development Assessment (WCARO Prototype)

• IDELA• EDI• EGRA/EGMA• East Asia Pacific Scale• PRIDI

Some of the tools representedMembers

Technical Advisory Group- Quality

• Frances Aboud, McGill University

• Caroline Arnold and Sheila Manji, Aga Khan Development Network

• Kim Boller, Mathematica

• Ivelina Borisova, Save the Children

• Belinda Hardin, University of North Carolina

• Katelyn Hepworth, Bernard Van Leer Foundation

• Michaela Ionescu, ISSA

• Yoshie Kaga, UNESCO

• Hellen Kimathi, Kenya Institute for Curriculum & Development

• Sarah Klaus, Open Society Foundation

• Abigail Lanceta, SEAMEO

• Joan Lombardi, Bernard Van Leer Foundation

• Florencia Lopez-Boo, Inter-American Development Bank

• Kelly Maxwell, Child Trends

• Michelle Neuman, R4D

• Venita Kaul, Ambedkar University

• Kofi Marfo, Aga Khan University

• Stephanie Olmore, NAEYC

• Nirmala Rao, University of Hong Kong

• Nitika Tolani-Brown, Save the Children

• Hiro Yoshikawa, NYU

• The Early Childhood Environment Rating Scale (ECERS-R)The Global Guidelines EDI

• International Step by Step (ISSA) Principles of Quality Pedagogy

• ACEI Global Guidelines Assessment (GGA)

• Save the Children’s Quality Learning Environment (QLE)

Some of the tools represented

Members

Why might the MELQO tools be helpful to a SIEF Team or WB Task Team?• For use in impact evaluation…maybe?

• Tools that are globally comparable, locally adaptable

• Open source (free, with technical guidance available)

• Building into the evaluation/operation measurement which is feasible to scale

• Child development includes both direct assessment and parent/teacher report

• Able to link quality of settings with child development outcomes

• Peer reviewed widely to confirm validity and reliability and disseminate knowledge and expertise

Key Design Features

• Focus on readiness for school

• Designed for flexible use• Within household surveys

• As a diagnostic

• At the population level

• Stimulating improvement• Classroom level, by encouraging focus on developmentally-appropriate activity

• Within M&E system, by providing information on how well children are doing and on the quality of children’s learning environments

• At the policy level, by encouraging dialogue on changes to policies and practices required to support young children’s development

• NOT a screening tool

Process

• Convene experts to draw from existing assessments and knowledge• Technical Advisory Groups

• Child Development and Learning• Quality

• Extensive item bank

• Draft tools

• Field tests• Bangladesh, Kenya, LPDR, Madagascar, Mongolia, Sudan, Tanzania

• In-country institutional work

• Peer review

How have we approached assembling tools and selecting specific items?

Steering Committee

• UNESCO (Maki Hayashikawa, Abbie Raikes)

• UNICEF (Jo Bourne, Pia Britto)

• Center for Universal Education at Brookings (Tamar Manyuelyan Atinc, Kate Anderson)

• World Bank (Luis Benveniste, Amanda Devercelli)

• GPE (Karen Mundy)

• UIS (Silvia Montoya, Albert Motivans)

Technical Advisory Group-Child Development and Learning

• Mariavittoria Ballotta, UNICEF WCARO

• Ivelina Borisova, Save the Children

• Sally Brinkman, University of Adelaide

• Magdalena Janus, McMaster University

• Sylvia Linan-Thompson, University of Texas at Austin

• Linda Platas, UC Berkeley

• Nirmala Rao and Yvonne Belcher, Hong Kong University

• Aimee Verdisco, Inter-American Development Bank

• Michael Willoughby, RTI

• Child Development Assessment (WCARO Prototype)

• IDELA• EDI• EGRA/EGMA• East Asia Pacific Scale• PRIDI

Some of the tools representedMembers

Technical Advisory Group- Quality

• Frances Aboud, McGill University

• Caroline Arnold and Sheila Manji, Aga Khan Development Network

• Kim Boller, Mathematica

• Ivelina Borisova, Save the Children

• Belinda Hardin, University of North Carolina

• Katelyn Hepworth, Bernard Van Leer Foundation

• Michaela Ionescu, ISSA

• Yoshie Kaga, UNESCO

• Hellen Kimathi, Kenya Institute for Curriculum & Development

• Sarah Klaus, Open Society Foundation

• Abigail Lanceta, SEAMEO

• Joan Lombardi, Bernard Van Leer Foundation

• Florencia Lopez-Boo, Inter-American Development Bank

• Kelly Maxwell, Child Trends

• Michelle Neuman, R4D

• Venita Kaul, Ambedkar University

• Kofi Marfo, Aga Khan University

• Stephanie Olmore, NAEYC

• Nirmala Rao, University of Hong Kong

• Nitika Tolani-Brown, Save the Children

• Hiro Yoshikawa, NYU

• The Early Childhood Environment Rating Scale (ECERS-R)The Global Guidelines EDI

• International Step by Step (ISSA) Principles of Quality Pedagogy

• ACEI Global Guidelines Assessment (GGA)

• Save the Children’s Quality Learning Environment (QLE)

Some of the tools represented

Members

Process and Research Questions

•Do items work well? Are they feasible to use and train?

•Do we have preliminary evidence of validity?

Phase 1a

•Can we link child development/quality in a meaningful way in the item design?

•Can we establish concurrent validity for both instruments?

Phase 1b •How do measures facilitate improvement at policy and practice levels?

•How do measures link to longitudinal achievement in early grades?

Phase 2

What have we learned so far?

• On process• Willingness to collaborate • Challenge to collaborate across institutions and the range of

skills/experience/capacity/presence required

• On items• Range of strong cognitive items from which to draw• Less consensus and less evidence on non-cognitive (executive function/approaches to

learning, socio-emotional)

• From countries• Strong demand for the tools• Desire to include more items than might be feasible• Some pushback on executive function/approaches to learning and socio-emotional • More in place to implement and scale quality, as opposed to child development

Relevance for SIEF/WB staff

• Comparable measures across countries/evaluations

• Linking of quality and child development

• Useful from setting to systems level

• Affordable and feasible

• Reliable and valid

Questions for discussion

• How do the goals of MELQO resonate with you?

• Does the goal of a globally comparable measure seem possible?

• How can we work within existing institutional settings of countries to make measurement of young children feasible and sustainable?

• What strategies can we use to ensure this is not used as a screening tool?