february 2018 breadth of learning opportunities · breadth of learning opportunities: a fresh...

February 2018

Breadth of Learning Opportunities

Kate Anderson, Seamus Hegarty, Martin Henry, Helyn Kim, and Esther Care

A fresh approach to evaluating education systems

BREADTH OF LEARNING OPPORTUNITIES: A FRESH APPROACH TO EVALUATING EDUCATION SYSTEMS

i

Kate Anderson is an associate fellow at the Center for Universal Education at BrookingsSeamus Hegarty is a visiting professor at the University of WarwickMartin Henry is research coordinator at Education InternationalHelyn Kim is a postdoctoral fellow at the Center for Universal Education at BrookingsEsther Care is a senior fellow at the Center for Universal Education at Brookings

An international expert group supported the authors to develop this report and accompanying toolkit.

Technical Working Group Members

Rachel Bolstad New Zealand Council for Educational Research

David Cameron People for Education, Ontario

Sergio Cárdenas-Denham CREFAL, Mexico

Inés Cruzalegui Secretariat of Innovation and Quality in Education, Argentina

Anil Kanjee Tshwane University of Technology, South Africa

Joyce Kinyanjui ziziAfrique, Kenya

Carina Omoeva FHI 360, USA

Sharon Ritchie Frank Porter Graham Institute at UNC-Chapel Hill, USA

Claire Scoular University of Melbourne, Australia

AcknowledgmentsThe authors would like to thank Tyler Ditmore of the Center for Universal Education, and Erin Mason, Fabiola Lara and Tom Eats for their contributions to this work.

The Brookings Institution is a nonprofit organization devoted to independent research and policy solutions. Its mission is to conduct high-quality, independent research and, based on that research, to provide innovative, practical recommendations for policymakers and the public. The conclusions and recommendations of any Brookings publication are solely those of its author(s), and do not reflect the views of the Institution, its management, or its other scholars.

Brookings gratefully acknowledges the program support provided to the Center for Universal Education by the LEGO Foundation, Echidna Giving, the Government of Norway, The Leona M. and Harry B. Helmsley Charitable Trust, the John D. and Catherine T. MacArthur Foundation, and the William and Flora Hewlett Foundation.

Brookings recognizes that the value it provides is in its absolute commitment to quality, independence, and impact. Activities supported by its donors reflect this commitment.

Access a digital version of the Breadth of Learning Opportunities (BOLO) report and tool package:www.brookings.edu/research/breadth-of-learning-opportunities

http://www.brookings.edu/research/breadth-of-learning-opportunities

ii

Mini-Pilot Teams

Argentina Andrés DelichNatalí SaranskyMarianela Giovannini

Organización de Estados Iberoamericanos, Argentina

El Salvador Pauline MartinJosé Rubén Merino Iglesias Erika Ivonne MestizoBriseyda Elizabeth Hernández

Central American University, El Salvador

Kenya Joyce KinyanjuiAruna Ramakrishnan

ziziAfrique

Joan Kamau Kenya Institute of Curriculum Development (KICD)

Martha OdundoEvelyn OwokoDorah Kitala

Ministry of Education

Pakistan Saba SaeedAfzal Ahmad

Idara-e-Taleem-o-Aagahi (ITA)

Aurangzeb KhanMuhammad Ayaz

Department of Elementary and Secondary Education, Khyber Pakhtunkhwa

Palestine Mohammed Matar The Palestinian Commission for Mathematics (RAFA’H)

Wahid JubranIbrahim IteweeMohammad SubohAmani ShehadehAmal Abu HejlehIsam Abu Kahlil

United Nations Relief and Works Agency for Palestine Refugees (UNRWA)

Rwanda Eugene Ndabaga College of Education, University of Rwanda

Tunisia Chedia Belaid Mhirsi Independent Researcher

Zambia Anna MurruCarl BeelMaimuna Ginwalla

VVOB Zambia

Mahuba Hazemba Ministry of General Education

Teacher Tool Development Team, led by Education International

Lucy Barimbui Kenya National Union of Teachers

Jedidah Rutere

Hesbon Otieno

Wilson Sossion

Christopher Yalukanda Zambia National Union of Teachers

Gideon Kalale Bulwani

Mildah Hanengeta Nyirongo

Newman Bubala

Kristel Bergmans Education International

Stephane Ponsard

Rocio Nieto


iii

Kenya Alignment Pilot Team, led by ziziAfrique

Joyce Kinyanjui ziziAfrique

Sara Ruto ziziAfrique

Aruna Ramakrishnan ziziAfrique

Amina Sharbaidi Centre for Mathematics, Science and Technology Education

Joan Kamau Kenya Institute of Curriculum Development

Flora Malasi Kenya Institute of Special Education

Janeanne Kiviu Kenya National Examination Council

Ali Hussein Abdi Kenya National Union of Teachers

Idris Aden Kenya National Union of Teachers

Ephantus Njoroge Kenya Primary School Heads Association

Darius Mogaka Ministry of Education

Esther Onyango Ministry of Education

Loice Kimani Ministry of Education

Martha Odundo Ministry of Education

Samuel Gichuhi Ministry of Education

Serah Mwirigi Thogoto Teacher Training College

Mexico Alignment Pilot Team, led by Centro de Cooperación Regional para la Educación de Adultos en América Latina y el Caribe (CREFAL)

Sergio Cárdenas-Denham CREFAL

Francisco Cabrera-Hernández CREFAL

Roberto Franco-Alba CREFAL

David Sánchez-Valdez CREFAL

School Tool Development Team at FHI 360

Carina Omoeva FHI 360

Rachel Hatch FHI 360

Celeste Carano FHI 360

iv

CONTENTS

Introduction 1The importance of a broad set of learning opportunities 2The conceptual model 2What do breadth of learning opportunities across the curriculum look like? 3Where did the breadth of learning opportunities measurement idea originate? 5

The BOLO Tools 6Policy tool 7School tool 8Teacher tool 9Putting the tools together 9Education levels and types of education assessed 11Stakeholders participating in the study 11Learning domains included 11

Adapting the Tools 11Jurisdictional level 12Policy questions asked 12Aligning with additional data sources 12

Using the BOLO tools 131. Does the jurisdictional curriculum offer breadth of learning opportunities? 132. Are students offered breadth of learning opportunities by their schools and teachers? 143. Are the education system components aligned to support breadth of learning opportunities? 144. Are there discrepancies in stakeholder interpretations? 14Tool availability 14

References 15

TABLES

Table 1 Levels of the curriculum and corresponding measurements 4Table 2 BOLO tools and piloting 8

FIGURES

Figure 1 LMTF Seven Domains of Learning 5Figure 2 BOLO theory of change 7Figure 3 Sample heat map of three LMTF domains and three curricular subjects 10


1

The global community has made a commitment to high-quality education for every child and young person. By the terms of the fourth Sustainable De-velopment Goal (SDG), target 4.1, every girl and boy should complete primary and secondary education by 2030. This education should be freely available to all and should lead to “relevant and effective learning outcomes” (United Nations Statistics Division, 2017).

Even if this target is achieved, however, the education delivered may be narrow. There is risk of a focus on a subset of learning that young people need—especial-ly literacy and numeracy—that may fail to address the complete set of skills needed to thrive in a changing world, such as collaboration, problem solving, critical thinking, and creativity.

Historically, school curricula have been organized by discipline, prioritizing the acquisition of knowl-edge in discrete subject areas, particularly language, mathematics, and science. This approach to school learning is entrenched in the education sector: textbooks, school organization, teacher education, and assessment, all reflect this discrete, disciplinary

basis. Despite some exploration of project learning, especially in primary schools, the tradition of separate subject learning, remains current. A recent report has documented how many students continue to learn in traditional school environments, where they sit at desks, listen passively, and memorize fixed bodies of knowledge (Care, Kim, Anderson, & Gustafs-son-Wright, 2017). This report notes also that teachers often receive little professional support to deliver a balanced curriculum.

This tradition is reinforced by assessment and exam-inations, both as part of daily routines and high stakes. When students need to pass examinations in order to progress through the education system, their focus—and that of their teachers—will likely be on the content of those examinations, to the exclusion of broader learning. Therefore, to the extent that examinations fo-cus on traditional subjects, and not on social respon-sibility, collaborative work, or the arts, they reinforce the traditional focus of the curriculum.

What if, in addition to evaluating an education system on the learning outcomes demonstrated by students,

Introduction

2

evaluation also targeted the opportunities students have to learn a broad range of skills? With this dual approach, looking at both opportunities and out-comes, education systems might be able to achieve the development goals without narrow delivery. The Breadth of Learning Opportunities (BOLO) initiative seeks to contribute to this approach by providing tools to document 1) whether opportunities are provided for learning across a diverse group of domains, and 2) how the key dimensions of an education system (curriculum, assessments, teacher supports, monitor-ing, and school resources) align to support delivery of breadth of learning opportunities.

This report describes an initiative undertaken from 2016-2017 by the Center for Universal Education (CUE) at the Brookings Institution and Education Inter-national (EI) to develop tools to measure the breadth of learning opportunities to which children and youth are exposed in an education system. First, we provide the rationale for developing these tools. Next, we de-scribe the tools developed through this initiative. We then explain how the tools can be adapted, and some of the questions the tools might be used to answer. This report is part of a package of documents, which includes the most recent version of the tools, a de-scription of how the tools were developed, and case studies of tool piloting in Kenya and Mexico.

The importance of a broad set of learning opportunities

The disciplinary basis of contemporary schooling may have served well in the past but, as a means of struc-turing students’ learning, it is no longer fit for purpose: it does not match the dynamic knowledge environ-ment already surrounding us, and it is not optimal for equipping young people for their adult lives.

Whether focusing on civil society or the workplace, substantial changes can be anticipated in living and working arrangements over the coming decades.

We know already that teamwork and creativity, for instance, are more highly valued in the workforce than they are currently emphasized in schooling. A flexible skill set is increasingly necessary for success in life and work. The International Commission on Financing Global Education Opportunity (The Education Com-mission) predicts that half of the world’s four billion jobs will be at risk due to automation by 2030 (The Education Commission, 2016). The increasing com-plexity of technology and the rise of globalization have ushered in the Information Age, and content knowl-edge will no longer be enough for students to suc-ceed; they need to be equipped with the breadth of skills required to adapt to societal change and thrive in the 21st century.

More critically, there is much that we do not know about the future: many of the jobs that today’s school-children will be doing have not been created yet, and there is little point in preparing them for jobs that may not exist in the future (The Education Commission, 2016). Therefore, the most effective preparation we can give young people is one that develops their per-sonal capacities and equips them with a broad set of flexible competences. It will be for them to respond to challenges as yet unknown and create new futures. In this new world, the advantage will be with well-round-ed individuals who are able to apply their acquired knowledge and skills to new and different situations.

The conceptual model

The curriculum can be seen from the perspective of the student, teacher, school, or system. In the BOLO work, we focus on the teacher, school, and system. This work sits within the broader education model, which is described below.

The key question for a young person is: what have I learned during my time in school? Learning is tak-en here to encompass knowledge, skills, attitudes, and behavior. This is the curriculum as internalized.


3

Ideally, a comprehensive portfolio covering examina-tion and test results, behavioral indicators, self-as-sessments, project work, and measures of workplace readiness would measure this. This broad approach to measurement is necessary because young people express what they have learned in individual ways, and key learning outcomes are not always pre-deter-mined.

This learning could derive from a range of sources, including school. For some young people the school may be a minor or even negligible source of learning but for the purposes of this project, we consider the subset of learning experiences that schools provide. This is the curriculum as experienced, sometimes described in terms of opportunity to learn. These experiences provide coherence, continuity, and links between different areas of learning throughout a learn-er’s life.

Examining the experienced curriculum sheds light on the situation of many students whose learning patterns differ from the norm. Teaching directed at the average student is likely to constitute different learning experi-ences for highly able students, as well as for students with learning difficulties. Information on the curriculum as experienced should come primarily from students, by questionnaire, focus group, and individual inter-view, though it can be supplemented by observation of both classroom activities and non-classroom inter-actions.

A major determinant of students’ learning experiences is what they are presented with, referring to the curric-ulum as implemented. This includes what students are taught in lessons, and opportunities for independent learning through homework, collaboration with peers, structured access to information technology, partici-pation in the life of the school, and social interactions. This can be measured by examining teacher plans and lesson delivery and gathering information on homework, use of information technology, opportuni-

ties for collaborative working, social interactions, and so on.

Teachers operate within a curriculum, assessment, and organizational framework set by their school, which is the curriculum as planned. This sets out the school’s vision for student learning: how it will be real-ized in terms of learning and participation opportuni-ties; how achievement and progression will be mea-sured; and what forms of academic organization the school will use. Information on this can be gathered by questionnaire, the study of school documentation, and the observation of school practices.

Schools, in turn, operate within frameworks laid down by national authorities, usually a national curriculum body. These set out the national curriculum, or the curriculum as intended, specifying in greater or lesser detail what students should learn at different grades and how their achievement should be assessed and certificated. Information on this can be obtained by questionnaire or by scrutiny of documents which are normally in the public domain.

In many countries, there is an intermediate level between the intended or official curriculum and the planned curriculum—the examined curriculum. This is because of the impact of assessment practices, particularly high-stakes examinations. The official curriculum may well set out a broad view of learning but, commonly, examinations restrict themselves to a restricted set of learning outcomes, which are then prioritized in schools’ curricula.

These different levels of the curriculum are summa-rized in Table 1.

What do breadth of learning opportunities across the curriculum look like?

Concerns over the narrowness of student learning are not new, and there have been numerous attempts to

4

set out what a broad-based curriculum would look like. One of the best known is the United Nations Educational, Scientific and Cultural Organization (UN-ESCO) Report Learning: The Treasure Within (UNE-SCO, 1996). This put forward four pillars, which were deemed the foundations of education:

• Learning to know. Broad general knowledge with in-depth understanding of a few areas, but also learning to learn so as to facilitate lifelong learning

• Learning to do. Occupational skills but also the competences to function in many different situa-tions, both social and work-related

• Learning to live together. Understanding other people, being respectful of divergent perspec-tives and being able to manage conflict

• Learning to be. Developing one’s own personality and growing in autonomy, judgement, and per-sonal responsibility

As discussed by Care and colleagues (2017), there has been growing awareness in recent years of the importance of a broad range of skills, and a shift in focus from academic, vocational, and technical skills toward an aspiration for education that informs both work and life more generally (Pellegrino & Hilton, 2012; Brewer, 2013; Lippman, Ryberg, & Moore, 2015).

There have also been efforts by national governments to expand national curricula and education delivery to include a broader set of learning domains. In a recent

Designation What it means Level of measurement How it is measured

Internalized What students have learned Individual student Analysis of student work, assessment data, attitude scales, behavioral indicators, workplace readiness

Experienced Students’ opportunity to learn Teacher/classroomIndividual student

Student surveys, observation of school practice (e.g. tools that measure student experiences as opposed to what the teacher does)

Implemented What and how teachers teach, use of homework and indepen-dent learning, group work

SchoolTeacher/classroom

School surveys and inspections, analysis of lesson plans, lesson observation

Planned The school’s vision for learning and teaching, assessment prac-tices, academic organization

School Analysis of school syllabi, school and classroom assessments, academic organization

Examined The learning deemed import-ant for testing and examination purposes

Country or jurisdiction

Analysis of assessment guidelines and test papers, and any assign-ments, projects or other activities considered in final grades or marks.

Intended National (or other governmental unit) curriculum as defined by authorities

Country or jurisdiction

Analysis of national, provincial or state curriculum statements

Table 1

Levels of the curriculum and corresponding measurements


5

study of national curriculum and policy statements in 152 countries, most (76 percent) were found to include 21st century skills somewhere in their poli-cy documents, and nearly half included skills such as communication, creativity, critical thinking, and problem solving in their curriculum statements (Care, Anderson, & Kim, 2016; Roth, Kim, & Care, 2017). However, when it comes to the learning outcomes actually assessed in an education system, the focus is primarily on reading, mathematics, and, to a lesser degree, science (Cheng & Omoeva, 2014).

Where did the breadth of learning opportunities measurement idea originate?

The work outlined in this report takes its point of de-parture from the Learning Metrics Task Force (LMTF). This was an initiative convened jointly from 2012-2015 by the Center for Universal Education at the Brook-ings Institution and the UNESCO Institute for Statistics

(UIS). The Task Force located its work specifically within the post-2015 global development agenda. Its purpose was to shift the global conversation on education from a focus on access to access plus learning. Within that framework, it sought to establish a consensus around the knowledge and domains of learning deemed important to equip all children and young people for a productive, fulfilled life. Following a process of review, input from experts and wide-spread consultation (in 118 countries), the Task Force published its proposals on the future of learning in the report What Every Child Should Learn (Learning Metrics Task Force, 2013).

At the heart of this report lie the Seven Domains of Learning, as set out in Figure 1.

Figure 1

LMTF Seven Domains of Learning

6

While measuring learning outcomes across all sev-en domains may not be feasible or desirable, it is possible to focus on students’ breadth of learning opportunities: establish how their learning experienc-es match up with an agreed framework (in this case, the LMTF Seven Domains of Learning); identify gaps and repetition; and take remedial action as neces-sary. Analysis of breadth of learning opportunities is one way to review curriculum and identify if it meets country aspirations. While this project used the LMTF Seven Domains of Learning framework as its definition of breadth, the process can be generalized to other frameworks as relevant for the users of the tools.

The BOLO tools bring together the conceptual mod-el of curriculum described above and a generally agreed-upon framework of broad learning, in this case, the LMTF Seven Domains of Learning. In an ideal world, students’ actual learning—the internalized curriculum—would be the starting point. Gathering the requisite information would be extremely labor-in-tensive, however, and would require considerable re-sources. In addition, student factors would complicate this approach. This project has therefore focused on students’ learning opportunities, as opposed to their

achievement, leading to an emphasis on the curricu-lum as intended, examined, planned, and implement-ed.

For each level of the curriculum, the official policies and, when relevant, the practices of schools and teachers are mapped onto the LMTF Seven Domains of Learning. The three tools capture the perspectives of policymakers, school administrators, and teachers to determine where there are differences among these three groups. The triangulation capacity of the tools is an important feature as it can shed light on how different stakeholder groups may experience or per-ceive the education system differently. The data can be analyzed to determine to what extent the curricu-lum embodies breadth of learning opportunities, and each curricular subject can be examined to determine the barriers and bottlenecks to implementation in the classroom. The resulting “map” of domains across the levels of curriculum can be used as a basis for discourse among education stakeholders within a country.

Three sets of tools examine breadth of learning oppor-tunities at the policy, school, and teacher levels, which

The BOLO Tools:Measuring Breadth of Learning Opportunities across the Curriculum Levels


7

maps onto the intended, planned, and implemented components of the curriculum. Accordingly, the inten-tion at the policy level may be reflected in what is de-livered at the school and classroom levels. The tools are designed to be adaptable to country context and needs. Adaptation is discussed further on page 11. The full details of how the tools were developed and piloted are available in the technical paper Breadth of Learning Opportunities: Technical Report 2018. Table 2 describes the BOLO instruments, respondents, curriculum levels, and countries where the tools were piloted.

An important aspect of the BOLO tools is that they are aligned to capture and triangulate the policy, school, and teacher perspectives. This alignment was achieved by first developing a framework for the poli-cy tools with an international group of researchers and policy makers. That framework was used as a starting point for drafting the teacher tool through workshops and piloting with teachers in Kenya and Zambia. Survey development experts used these versions to refine the teacher tool and draft the school tool. Final-ly, the three tools were piloted together in Kenya and Mexico with policymakers, school administrators, and teachers.

The three tools are described below.

Policy tool

An international group of experts in curriculum, assessment, and education policy developed the policy tool. It provides a framework to review curricu-lum and assessment documents at the jurisdictional level, which is defined as the administrative level responsible for setting policies on curriculum. The tool examines the breadth of learning opportunities for students set out at the official level, in one focus grade for each educational level studied. In some countries, the curriculum documents might be developed and sourced at a national level, and in others at a sub-na-tional level; the policy tool is designed to be adapt-able to either situation. The current tool examines the extent to which the curriculum covers the LMTF Seven Domains, and how a breadth of learning opportuni-ties approach is incorporated throughout monitoring, assessment, teacher training, and resource provi-sion. The policy tool is designed to be completed by an inter-agency team including government and non-government stakeholders. The tool contains five components:

1. User Information: Information about the lead respondent and the names, positions, and affil-iations of the committee or team members who completed the tool.

Figure 2

BOLO theory of change

Data are presented to education stakeholders to initiate dialogue on areas of strong and weak appearance of BOLO, and alignment through the system

BOLO tools collect information on where breadth does and does not appear throughout the curriculum levels

Policies and practices are modified to encourage greater breadth of learning opportunities

A broader set of learning opportunities are experienced by children and youth

https://www.brookings.edu/research/breadth-of-learning-opportunities/

https://www.brookings.edu/research/breadth-of-learning-opportunities/

8

2. System Information: An overview of the ed-ucation system and the curriculum using the International Standard Classification of Education (ISCED) (UNESCO Institute for Statistics, 2011) to determine the levels of education on which the remainder of the tool will focus. This section also includes questions about the length of the school day, week and year, mandated class sizes, and general requirements for teachers.

3. Subjects and Timetable: For each grade level, a list of the mandatory and optional subjects in the curriculum, and the hours per week scheduled according to the curriculum policy.

4. Subject Background – Curriculum Policies for Mandatory Subjects: For each mandatory sub-ject, a series of questions on the policies related to monitoring, examinations and assessments, and teacher training and professional develop-ment.

5. LMTF Domains Mapping: A matrix of the LMTF subdomains mapped to the subjects in the na-tional curriculum, with boxes to check on whether

there is a major or minor focus on the subdomain in each curricular subject. This matrix is complet-ed for the focus grade for each education level examined in the study.

School tool

The school tool was developed by CUE with support from FHI 360, in order to align the policy and teacher tools. It is intended to examine the breadth of learning opportunities for students as evidenced by school-lev-el curriculum policies, available facilities, and govern-ment oversight of school instruction. The school tool is intended for completion by school administrators (principals, head teachers, etc.). The school tool con-tains five components:

1. User and School Information: Information about the school type, location, management, grades, socio-economic status of students, and facilities.

2. Grade Information: For the grade level of focus, information on characteristics of teachers and

Instrument Respondent Curriculum level

Countries piloted to date

Education levels piloted to date

Policy tool Policymaker Intended, exam-ined

Argentina, El Salvador, Kenya, Mexico, Pales-tine, Pakistan, Rwanda, Tunisia, and Zambia

Pre-primary, primary, lower secondary

School tool School administrator Planned Kenya, Mexico Primary, lower secondary

Teacher tool Teacher Implemented Kenya, Mexico, Zambia Primary, lower secondary

Policy, school, and teacher tools (alignment pilot)

Policy makers, school administra-tors, teachers

Intended, exam-ined, planned, implemented

Kenya, Mexico Primary, lower secondary

Table 2

BOLO tools and piloting


9

learners. This section also includes questions about the length of the school day, week and year, class sizes, and general qualifications for teachers.

3. Subjects and Timetable: For each grade level, a list of the mandatory and optional subjects in the curriculum, and the hours per week scheduled according to the school’s policy.

4. Information on Mandatory Subjects: For each mandatory subject, a series of questions on the government and the school’s policies and prac-tices related to monitoring, examinations and as-sessments, and teacher training and professional development.

5. LMTF Domains Mapping: A matrix of the LMTF subdomains mapped to the subjects in the school’s curriculum, with boxes to check on whether there is a major or minor focus on the subdomain in each curricular subject. This matrix is completed for the focus grade.

Teacher tool

The teacher tool was developed by EI with teacher unions in Kenya and Zambia to examine breadth of learning at the classroom and teacher levels (Kenya National Union of Teachers & Education International, 2017; Zambia National Union of Teachers & Education International, 2017). It was refined to align with the school and policy tools by CUE and FHI 360. In addi-tion to gathering information on what is actually taught in schools and time allocations to the different do-mains, the tools survey the pre-service and in-service education received, time spent on various activities across the learning domains, types of assessments used, and resources available. The teacher tool con-tains six components:

1. Background Information: Basic information about the enumerator, country, grade level, and school.

2. Teacher Information: The teacher’s education level, grades taught, gender, specialty, and school facilities to which the teacher has access.

3. Class information: Information about learners and class schedule.

4. Subjects and Timetable: For each grade level, a list of the mandatory and optional subjects the teacher teaches, and the hours per week sched-uled according to the school’s policy.

5. Information on Mandatory Subjects: For each mandatory subject, a series of questions on the government policies and teacher practices related to monitoring, examinations and assessments, teacher education and training, and support of-fered to teachers to deliver the curriculum.

6. LMTF Domains Mapping: A matrix of the LMTF subdomains mapped to the subjects taught by the teacher, with boxes to check on whether there is a major or minor focus on the subdomain in each curricular subject. This matrix is completed for the focus grade.

Putting the tools together

The policy, school, and teacher tools contain similar questions to allow triangulation of data from the three sources. They are meant to be administered together, which can be done in multiple ways. In the alignment pilot in Kenya, for example, the policy tool was com-pleted at a workshop of government officials, teacher union officials, and civil society organizations. Then a group of policymakers and a representative from the Kenya National Union of Teachers (KNUT) ad-ministered the school and teacher tools in a random sample of schools. This has the advantage of increas-ing government buy-in for the results, but as reported by stakeholders in Kenya, can also lead to biased re-

10

sponses in favor of what the school administrators and teachers think the government officials want to hear. In Mexico, one policymaker in each state completed the policy tool, and a team of independent researchers administered the school and teacher tools. This had the advantage of potentially reducing bias, but the re-search team also encountered some difficulty access-ing schools and teachers, and the individual policy-makers were not always aware of the current policies. During the piloting of the teacher tools in Kenya and Zambia, teachers were trained as enumerators, which facilitated access to the schools and teachers, but in some cases, the tools were not completed adequate-ly. There are advantages and disadvantages to each approach, and they need to be considered along with the purpose and intended use of the tools.

Before administering the tools, a small pilot should be undertaken to check for relevance and understanding among participants. The data can be triangulated such that a policy or research team should ask fol-low-up questions of participants to determine whether misalignment between policy, school, and teacher surveys is real or due to participants not interpreting the questions in the same way. Rubrics concerning how a “major” or “minor” emphasis in the curriculum is defined may need to be developed in each jurisdiction before a full administration of the tools. The next sec-tion provides more guidelines on adapting the tools.

The results of the tools can be put together in a data visualization, or “heat map”, which shows the degree to which the LMTF Seven Domains of Learning are emphasized within the curricular subjects, according to policy, school administrators, and teachers. For example, the results from policymakers, school admin-istrators, and teachers on how the LMTF subdomains within literacy and communication are emphasized within the national curriculum subject of English could be examined as shown below in Figure 3, with darker colors signifying a major emphasis and lighter colors signifying little to no emphasis. In this example, the

policy makers (denoted by ‘P’) noted that topics such as oral fluency, oral comprehension, and reading fluency are a major part of the curriculum for English, and school administrators (denoted by ‘S’) and teach-ers (denoted by ‘T’) also said these subdomains were emphasized in the English subject. However, teach-ers, and to a lesser extent, school administrators, said that they emphasize the literacy and communication domain in Social Studies and Science, but the cur-riculum policy does not appear to emphasize these subjects in Social Studies and Science.

Figure 3

Sample heat map of three LMTF domains and three curricular subjectsKenya – ISCED 1 – LMTF Domains

English Social Studies

Science

P S T P S T P S T

Num

erac

y &

mat

hem

atics Number concepts & operations

Geometry & patterns

Mathematics application

Lite

racy

& c

omm

unica

tion

Oral fluency

Oral comprehension

Reading fluency

Reading comprehension

Receptive vocabulary

Expressive vocabulary

Written expression/composition

Cultu

re &

ar

ts

Creative arts

Cultural knowledge

A heat map can also be generated to show the degree to which the country or jurisdiction’s curric-ular subjects are emphasized throughout the levels of the system (intended, examined, planned, and implemented). Case studies from Kenya and Mexico (forthcoming) describe the process of developing heat maps and engaging the study participants in dialogue using these tools.

P: Policy makersS: School administratorsT: Teachers


11

unions along with all government agencies involved in developing and implementing policies for teach-ing, curriculum, and assessment. The results can be organized to map the perspectives of policymakers, administrators and teachers side-by-side to facilitate a dialogue between the various stakeholders in the system. Additional stakeholder levels (such as school district supervisors) could be included in the study, if relevant.

Learning domains included

The LMTF framework provides a common framework for defining breadth of learning. However, some countries have developed their own frameworks that incorporate 21st century skills, global citizenship, so-cial and emotional skills, and other aspects of a broad education. Introducing multiple frameworks could be confusing for school personnel in countries where oth-er frameworks are being introduced, as was demon-strated in the pilot studies in Kenya and Mexico.

In countries where a new curriculum is being intro-duced and the government is interested in learning

The BOLO tools are intended to be adapted to the context in which they are to be used. Variations can be made to the tools in multiple ways.

Education levels and types of education assessed

Pre-primary, primary, and lower secondary are the schooling levels for which questionnaires were developed for this initiative, but upper secondary education, technical and vocational education, infant and toddler programs, and non-formal education programs could also be examined using these tools. Programs for students with special needs or learning disabilities could also be examined, either as part of the traditional education system or separately. To adapt the tools to other education contexts, users could identify existing frameworks at the national and international levels to define breadth.

Stakeholders participating in the study

For maximum stakeholder engagement, a study should bring together teacher organizations or

Adapting the Tools

12

the extent to which the content of the new curriculum is already being covered at a school level, the focus areas of the new curriculum could be used for map-ping, rather than the LMTF domains. In a country that has not yet embarked on a curriculum reform, using a global framework like the LMTF domains might be a useful exercise, and the resulting information on how the LMTF domains are represented in the existing curriculum could be used to inform the development of a new curriculum.

Jurisdictional level

In a centrally-administered education system, the national government makes policies and they are expected to be carried out in each region, so adminis-tering one policy tool and triangulating the results from the school and teacher tools to this questionnaire are possible. The data can also be disaggregated at the sub-national level using the same policy tool data in each region.

In a state-administered education system, a separate policy tool may be completed for each state. Using a common framework of learning domains, such as the LMTF framework or the national curriculum and standards, is recommended as it enables comparison across states.

Policy questions asked

For the policy, school, and teacher tools, the ques-tions can be modified to meet the needs of the re-search and policy interests determined by the country. For example, more questions on teacher professional development can be included if a country is planning to use the results to inform teacher education.

Aligning with additional data sources

The BOLO tools could be aligned with previously published classroom observation tools to examine the

experienced curriculum. While developing and vali-dating a classroom observation tool were outside the scope of this study, an existing tool that measures the classroom environment and student experience could be used and the BOLO policy, school, and teacher tools could be aligned with the domains included therein. Student logs could also be used as potential data collection tools aligned with the BOLO tools.


13

Together, the BOLO tools use multiple sources of evidence to gather information on:

• Curriculum documents• Assessment and examination frameworks• Teacher pre-service and in-service preparation• Learning materials• Classroom experiences

The results from the three sets of tools can be com-bined to generate a picture of the breadth of learning opportunities in the education system at the national and classroom levels. Education ministries and other actors can use this information to inform curriculum reform or review efforts or to examine the implementa-tion of the existing curriculum.

These tools are not accountability mechanisms. They are designed to shed light on relevant issues and pose specific questions to explore and to inform action, particularly by national officials, on how the system is operating and where there is and is not alignment. By reviewing the strengths and challenges in the system, from the national to local levels, this

information can help to fine-tune policies that seek to provide broad learning opportunities.The tools are intended to contribute toward improv-ing learning outcomes in a wide array of domains by giving educators a means to analyze their education system. Ultimately, the tools should support gov-ernment interventions that increase the breadth and depth of learning in their constituencies. Examples of questions that the BOLO tools are intended to answer are described below.

1. Does the jurisdictional curriculum offer breadth of learning opportunities?

This question examines to what extent the LMTF Sev-en Domains of Learning and subdomains are repre-sented in the country’s (or jurisdiction’s) mandatory curricular subjects. This information is collected in the LMTF domains and subdomains matrix of the policy tool. It asks if each LMTF subdomain is covered in the curriculum content for each subject, and, if so, wheth-er it has a major or minor focus. The policy tool also asks the hours per week for which each mandatory

Using the BOLO tools

14

and optional subject is scheduled according to policy. Taken together, the mapping and scheduling data can be used to describe how the LMTF Seven Domains of Learning map onto the subjects in the curriculum to establish whether breadth exists and to what extent.

2. Are students offered breadth of learning opportunities by their schools and teachers?

This question examines to what extent school admin-istrators and teachers report that the LMTF Seven Domains of Learning are featured in their school and classroom curricula. This information is collected in the LMTF domains and subdomains matrix of the school curriculum and teacher curriculum question-naires. Similar to the policy survey, teachers and school administrators are asked if each LMTF sub-domain is covered in the curriculum content for each subject, and, if so, whether it has a major or minor focus. The school and teacher tools also ask the num-ber of hours per week each mandatory and optional subject is taught. This information can be aggregated to examine the extent to which schools and teachers focus on teaching across the LMTF Seven Domains of Learning and the subjects of the curriculum.

3. Are the education system components aligned to support breadth of learning opportunities?

The sections of the tools that include detailed infor-mation on mandatory subjects can be analyzed to answer this question. BOLO data can be organized to view how the curricular subjects are supported at the policy, school, and teacher level on six aspects:

• Curriculum• Monitoring• Examinations• Assessments• Teaching materials• Professional development opportunities

If a subject is included in the curriculum policy, it would be expected that this subject would be mon-itored by the school inspection agencies; appear in national examinations, classroom assessments, and any other type of learning assessment; be part of the teaching materials provided to schools (where this happens); and be included in professional develop-ment opportunities. The data from the BOLO tools can be organized to describe how each curricular subject is supported, or not supported, both in policy and in practice.

4. Are there discrepancies in stakeholder interpretations?

The BOLO tools are aligned so that the respons-es from policy makers, school administrators, and teachers can be compared to determine where there are discrepancies in interpretations of the curriculum across the various groups. Presenting the data from all three stakeholder groups together can initiate a di-alogue among the groups about whether the discrep-ancies are real, and what can be done about them.

Tool availability

The BOLO tools, along with a description of how the tools were developed and case studies from Kenya and Mexico, are available at www.brookings.edu/research/breadth-of-learning-opportunities




15

References

Brewer, L. (2013). Enhancing youth employability: What? Why? and How? Guide to core work skills: International Labour Organization.

Care, E., Anderson, K., & Kim, H. (2016). Visualizing the Breadth of Skills Movement Across Education Systems. Washington, DC: Brookings Institution.

Care, E., Kim, H., Anderson, K., & Gustafsson-Wright, E. (2017). Skills for a Changing World: National Perspectives and the Global Movement. Washington, DC: The Brookings Institution.

Cheng, X., & Omoeva, C. (2014). EPDC National Learning Assessments Mapping Project: Key Findings. https://www.epdc.org/epdc-data-points/epdc-national-learning-assessments-mapping-project-key-findings

Kenya National Union of Teachers, & Education International. (2017). Breadth of Learning: Assessment of the Breadth of Learning Opportunities in Public Secondary Schools in Kenya. Brussels.

Learning Metrics Task Force. (2013). Toward Universal Learning: What Every Child Should Learn. Washington/Montreal: Brookings Institution/UNESCO Institute for Statistics.

Lippman, L. H., Ryberg, R. C., & Moore, K. A. (2015). Workforce Connections Key “Soft Skills” That Foster Youth Workforce Success: Toward a Consensus Across Fields. Washington, D.C.: Child Trends.

Pellegrino, J. W., & Hilton, M. L. (2012). Education for Life and Work: Developing Transferable Knowledge and Skills in the 21st Century. doi: 10.17226/13398

Roth, A., Kim, H., & Care, E. (2017). New data on the breadth of skills movement: Over 150 countries included. Education Plus Development. https://www.brookings.edu/blog/education-plus-development/2017/08/31/new-data-on-the-breadth-of-skills-movement-over-150-countries-included/

The Education Commission. (2016). The Learning Generation: Investing in education for a changing world.

UNESCO. (1996). Learning: The Treasure Within.

UNESCO Institute for Statistics. (2011). International Standard Classification of Education: ISCED 2011. Montreal: UNESCO Institute for Statistics.

United Nations Statistics Division. (2017). Revised list of global Sustainable Development Goal indicators. New York: United Nations.

Zambia National Union of Teachers, & Education International. (2017). Breadth of Learning: Assessing Breadth of Learning in Zambian Primary Schools. Brussels: Education International.

Access a digital version of the Breadth of Learning Opportunities (BOLO) report and tool package: www.brookings.edu/research/breadth-of-learning-opportunities


february 2018 breadth of learning opportunities · breadth of learning opportunities: a fresh...

Documents