endline evaluation · cbo community-based organization cba cost-benefit analysis cea...
TRANSCRIPT
April 18, 2016
ENDLINE EVALUATION
UNICEF KENYA’S IMPLEMENTATION OF INTEGRATED
INTERVENTIONS UNDER THE CHILD FRIENDLY SCHOOLS
FRAMEWORK IN ASAL COUNTIES
2
ACKNOWLEDGEMENTS
This report was prepared by the FHI 360 Education Policy and Data Center, authored by Audrey Moore,
Carina Omoeva, Charles Gale, and Wael Moussa. Peter Mwarogo and Racheal Kamau managed the field
component of the study. The evaluation was conducted by FHI 360, and commissioned by UNICEF with
funding and technical oversight from the UK Department for International Development. The project was
monitored by Daniel Baheta, Charles Olaka Kesa, and Shweta Sandilya of UNICEF Kenya Country
Office, and Sandra Barton and Caroline Wangeci of DfID provided technical review of the design and
analysis.
3
TABLE OF CONTENTS Table of contents ........................................................................................................................................... 3
List of Tables ................................................................................................................................................ 5
List of Figures ............................................................................................................................................... 5
Abbreviations ................................................................................................................................................ 6
Executive Summary ...................................................................................................................................... 7
Introduction ................................................................................................................................................. 10
Background ............................................................................................................................................. 10
UNICEF Interventions in the ASAL Region .......................................................................................... 10
Methodology ............................................................................................................................................... 11
Key data sources ..................................................................................................................................... 11
Constructing the comparison group ........................................................................................................ 12
Descriptive analysis............................................................................................................................ 12
Regression analysis ............................................................................................................................ 13
Description of the study population ........................................................................................................ 15
Characteristics of the sample .................................................................................................................. 16
Results ......................................................................................................................................................... 16
Key Findings .......................................................................................................................................... 16
Overview of changes in key outcomes ................................................................................................... 18
Enrollment trends over the intervention period .................................................................................. 19
Attendance rate trend over the intervention period ............................................................................ 19
Changes in retention ........................................................................................................................... 20
Indicative results of the WASH Interventions ........................................................................................ 21
Status of WASH interventions ........................................................................................................... 21
How effective are the WASH interventions for outcomes of interest? .............................................. 21
WASH regression analysis ................................................................................................................. 23
Challenges in implementation and monitoring of WASH interventions. ........................................... 24
Findings from focus groups ................................................................................................................ 25
Indicative Results of Solar-Power Interventions .................................................................................... 27
Status of Solar interventions .............................................................................................................. 27
How effective are the solar interventions for outcomes of interest? .................................................. 27
Solar regression analysis .................................................................................................................... 29
Challenges in implementation and function of solar power ............................................................... 30
Findings from focus groups ................................................................................................................ 31
Indicative Results of Child-Friendly Schools Interventions ................................................................... 33
How effective are the CFS interventions for outcomes of interest? ................................................... 33
4
Classroom Observation - Background ............................................................................................... 34
Academic and non-academic activities .............................................................................................. 34
CFS regression analysis ..................................................................................................................... 36
Findings from Focus Groups .............................................................................................................. 37
Indicative Results of C4D Interventions ................................................................................................. 37
How effective are the C4D interventions for outcomes of interest? .................................................. 37
C4D regression analysis ..................................................................................................................... 40
Findings from Focus Groups .............................................................................................................. 41
Cost-Benefit Analysis ................................................................................................................................. 43
Assumptions ........................................................................................................................................... 43
Results of the benefit-cost analysis......................................................................................................... 45
Limitations of the Benefit-cost model .................................................................................................... 46
Concluding remarks for cost-benefit analyses ........................................................................................ 47
Discussion ................................................................................................................................................... 47
Understanding Dosage, Duration and Enabling Environment ........................................................... 47
Duration .............................................................................................................................................. 48
Dosage ................................................................................................................................................ 49
Enabling environment ........................................................................................................................ 50
References ................................................................................................................................................... 52
Annexes ...................................................................................................................................................... 54
Annex A: Supplemental Methodology Information ............................................................................... 54
Instruments and Key Data Sources ......................................................................................................... 55
Hypotheses to be analyzed in the observational study: .......................................................................... 56
Variables and analyses ............................................................................................................................ 57
Qualitative analysis ............................................................................................................................ 57
Changes to the data collection and analysis process from baseline to endline ....................................... 57
Annex B: Computation of Net Present Value in Cost-Benefit Analysis ................................................ 59
Annex C: Additional Focus Group Results ............................................................................................ 60
5
LIST OF TABLES Table 1 - Number of schools by type of intervention ................................................................................. 15 Table 2 - Enrollment changes over the intervention period ........................................................................ 19 Table 3 - Highlights of the classroom observation ..................................................................................... 35 Table 4 – Changes in enrollment, C4D status ............................................................................................. 38 Table 5 – Estimated enrollment benefits for WASH, Solar, and CFS ........................................................ 43 Table 6 – Grade promotion, dropout, and repetition rates – Kenya, 2014 .................................................. 44 Table 7 – Increased enrollment and grade attainment, by intervention ...................................................... 45 Table 8 – Costs per school, per student, and total, by intervention ............................................................ 45 Table 9 – Lifetime benefit-to-cost ratios per intervention, by discount rate ............................................... 46 Annex table 1 – Number of schools by reported treatment status .............................................................. 54 Annex table 2 – Benefits-to-cost ratios for WASH, Solar, and CFS .......................................................... 59
LIST OF FIGURES Figure 4 – Map of Kenya ............................................................................................................................ 10 Figure 5 - Propensity Score Distribution, by Treatment and Pre-Designated Comparison Groups ........... 13 Figure 6 - Propensity Score Distribution, by Treatment and Constructed Comparison Groups ................. 14 Figure 7 - Number of schools reached by county ....................................................................................... 16 Figure 8 - Pupil-teacher ratio ...................................................................................................................... 16 Figure 9 – Attendance rate changes over the intervention period ............................................................... 20 Figure 10 – Changes in retention over the intervention period ................................................................... 20 Figure 11 – WASH changes in key outcomes ............................................................................................ 22 Figure 12 – Head teacher assessment of WASH outcomes ........................................................................ 22 Figure 13 – Changes in availability of latrines ........................................................................................... 23 Figure 14 – Margins plot of WASH outcomes ........................................................................................... 24 Figure 15 - WASH challenges in implementation, as reported by Head Teachers ..................................... 25 Figure 16 – Solar: type of power ................................................................................................................ 27 Figure 17 – Solar: UNICEF and Non-UNICEF .......................................................................................... 27 Figure 18 – Solar: differences in evening study.......................................................................................... 28 Figure 19 – Hours of study ........................................................................................................................ 28 Figure 20 – Solar: head teacher assessment ................................................................................................ 29 Figure 21 - Margins plot of Solar outcomes ............................................................................................... 29 Figure 22 - Margins plot of Solar effect on frequency of studying ............................................................ 30 Figure 23 – Solar challenges in implementation ......................................................................................... 31 Figure 24 – CFS: Changes in key outcomes ............................................................................................... 33 Figure 25 – Number of schools in which classroom observations were conducted.................................... 34 Figure 26 – Percentage of class time devoted to academic activities ......................................................... 35 Figure 27 – Percentage of class time spent on nonacademic activities ....................................................... 35 Figure 28 – Margins plot of CFS outcomes ................................................................................................ 36 Figure 29 – Changes in attendance, C4D status .......................................................................................... 38 Figure 30 – Changes in promotion, C4D status .......................................................................................... 38 Figure 31 – Has received C4D training ...................................................................................................... 39 Figure 32 – OOSC communication plan ..................................................................................................... 39 Figure 33 – C4D: Engaged in advocacy or social mobilization .................................................................. 40 Figure 34 – C4D: Head teachers’ assessment ............................................................................................. 40 Figure 35 – Margins plots of C4D outcomes ............................................................................................. 41 Figure 36 – Age-earnings profile by educational attainment – Kenya, 2013 ............................................. 44 Figure 37 – Cost-Benefit simulation results – NPV of cumulative taxes paid over lifecycle ..................... 46
6
ABBREVIATIONS
ASAL Arid and Semi-Arid Lands
C4D Communication for Development
CBO Community-Based Organization
CBA Cost-benefit analysis
CEA Cost-effectiveness analysis
CFS Child Friendly Schools
DFID Department for International Development
DRR Disaster Risk Reduction
GOK Government of Kenya
LCPBS Low Cost Primary Boarding School
MOEST Ministry of Education Science and Technology
QED Quasi-experimental Design
RCT Randomized Control Trial
BoM School Board of Management
WASH Water Sanitation and Hygiene
7
EXECUTIVE SUMMARY Description of interventions. In 2013, UNICEF Kenya partnered with the Keya Ministry of Education
Science and Technology (MOEST) to launch a series of program interventions with the goal of
strengthening the quality of educational experience for children in the Kenyan Arid and Semi-Arid Lands
(ASAL). The United Kingdom’s Department for International Development (DFID) provided funding and
technical oversight for the program, which benefits an estimated 66,000 children (UNICEF, 2014). The
program consists of four discrete school-based interventions:
1) Water, sanitation, and hygiene (WASH) package, which involves installation of WASH
infrastructure and training;
2) Installation of solar panels to increase availability of electric lighting in classrooms, WASH
facilities, and dormitories;
3) Child-friendly schools (CFS) training and capacity development of head teachers, teachers and
BOM;
4) Communication for development (C4D) efforts, aimed at reducing dropout and encouraging school
enrollment among out-of-school children
Evaluation design. FHI 360 was contracted to carry out an external evaluation of the program in 2014. Due
to the nature of selection for interventions, and the prior presence of similar program interventions in the
target areas, a quasi-experimental methodology (matching on the propensity score) was selected as the
primary method for this evaluation. In addition, the study factored in the timing and duration of program
interventions, and was able to account for the differential effects of the interventions resulting from longer
duration. The study draws on data from 2,136 student interviews, 321 head teacher interviews, and 26
student/community focus group discussions, to examine program effects, implementation challenges, and
perceptions of the beneficiaries on program effectiveness. In addition, classroom observations were
conducted at 54 CFS and C4D intervention schools provided a gauge of time use and the application of
child-friendly instructional practices. The endline reached 321 out of 349 schools (92%) across all eight
counties, as some schools were dropped due to security concerns.
Results This evaluation showed mostly positive outcomes for schools supported by the program.
Enrollment gains were the greatest among schools that received WASH interventions. We exploit variation
in the reported exposure to the treatment, with some schools reporting having received an intervention prior
to the UNICEF ASAL program period (starting in June 2014), to identify strong and substantively large
positive enrollment effects of longer duration of interventions. This is true in the case of both WASH and
CFS interventions, that were reported present since prior to June 2014, while the Solar intervention was
implemented only after June 2014. Lastly, we find that WASH, Solar power, and CFS exhibit benefits that
exceed the costs of implementation with CFS being the most cost-effective. Brief summaries of results for
each treatment arm follow.
WASH – Both the descriptive and regression analyses
show that schools that implemented WASH the longest
(since prior to June 2014) exhibited the largest gains in
enrollment relative to all other schools. Specifically, we
estimate that enrollment among schools that
implemented WASH the longest increased by
approximately 20 percent relative to similar schools in
the constructed comparison group that never received
the WASH intervention. Attendance rates are also
highest among WASH schools with the longest
duration.
8
Solar power – Regression results show that enrollments are only modestly higher - by about 8.6 percent
among UNICEF Solar schools than similar schools in the constructed comparison group. These schools
also exhibited greater study time, which was one of the intended outcomes of the Solar intervention. All
else being equal, the estimated likelihood that a given student reports studying at least once a day is 13
percent higher among UNICEF Solar schools than similar schools that never received solar panels. The
effect of duration could not be estimated for the solar intervention, as it was limited to the period of program
performance.
CFS – Regression results suggest that enrollment growth among schools that implemented CFS prior to
June 2014 as well as those that implemented after June 2014 is 8.6 percent higher than the constructed
comparison group. However, the schools that reported having the CFS intervention, but were not supported
through the UNICEF/DFID program, that exhibited the largest gains in enrollment. Although, it is likely
that these differences are driven by differences in the level of implementation. Classroom observations
suggest that teachers in schools receiving the CFS intervention are more likely to exhibit characteristics
consistent with the approach that is promoted by the intervention, e.g. ask probing questions such as how
and why, ask students to explain in class discussions, and ask students to apply understanding to relevant
life experience. However, teachers in schools receiving the CFS intervention are slightly less likely to
integrate gender sensitive material into pedagogy.
C4D – The C4D intervention was largely implemented in conjunction with other interventions. Only six
schools report receiving only C4D from UNICEF, therefore, we are unable to isolate the impacts of C4D
from that of other interventions. Further, none of the schools report receiving C4D prior to June 2014. As
a result, we are unable to account for the duration of treatment effect. Overall, the descriptive analyses show
a positive effect on enrollment and attendance, but not grade promotion.
Combination Schools – Schools that receive two or more UNICEF interventions were more likely to show
positive effects on attendance and enrollment. Schools that received Solar power and C4D exhibited 10.5
percent higher enrollment relative to the constructed comparison group. The same combination of
interventions yielded a 46.5 percentage point increase in attendance rates and a 14.8 percent increase in the
likelihood of students studying more than once a day. Schools implementing C4D in conjunction with CFS
are estimated to have an attendance rate that is 28 percentage points higher than that of the constructed
comparison group. Lastly, schools that implemented both CFS and WASH also exhibited gains in
enrollment. Although, the gains achieved by schools with the CFS and WASH combination were relatively
smaller at 9.4 percent. Further, the estimated gain for the CFS and WASH combination was not statistically
significant.
Cost-Benefit Analysis – We compute the benefits of WASH, installation of solar lighting, and CFS
interventions as additional taxes paid over a lifetime as a result of increased educational attainment from
each intervention.
CFS yields the highest return on investment according to our simulation model. Initial investments
in CFS are returned in full after 4-5 years and the lifetime benefits-to-cost ratios for CFS range
between 4.0 and 12.5 per US Dollar invested. This means that an initial investment of 112,206 USD
yields approximately between USD 432,000 and USD 1,395,000 in increased tax revenues over 36
years.
WASH and Solar may be returned in full 10 to 21 years post-implementation depending on the
discount rate, and we calculate lifetime benefit-to-cost ratios between 1.2 and 4.2 per US Dollar
invested. This translates to an initial investment of USD 505,357 in WASH yields a return between
USD 602,000 and USD 1,944,000, and an initial investment of USD 111,667 in Solar panels
produces a return between USD 136,000 and USD 440,000 over 36 years.
9
Recommendations We offer the following conclusions and recommendations based on the findings.
Dosage and duration of the treatments play a significant role when determining intervention
impacts. In this report, we provide evidence that interventions that were implemented for a longer
duration exhibited larger effects on enrollment when compared to schools that implemented the
same intervention for a shorter duration. This is especially true with the WASH and CFS
interventions. This also supports the hypothesis, that certain programmatic effects require some
time to manifest themselves in education outcomes.
Further research is required to ascertain the level of program implementation fidelity in each
intervention. When examining the impact of the CFS or C4D interventions, we found that overall
results were either relatively small in magnitude and/or provided some evidence that similar
interventions supported by other organizations had larger gains. In both cases, the findings point
to a certain amount of untapped potential gains in enrollment that could be had. As such, we
recommend that context-specific elements of implementations should be investigated to
determine where and how certain practices could be improved from a programming standpoint.
For instance, some of the focus group discussion results pointed toward community members and
other stakeholders not being aware of the C4D intervention. Additional efforts into ensuring
community involvement and awareness could have possibly benefitted the schools more.
The investments in WASH, CFS, and Solar panels yield positive returns on investment over the
long-term and to varying degrees. It is important to be cognizant of the assumptions of the cost-
benefit analysis, especially with the discount rate, which acts as a proxy for the value of future
money. In addition, wage based returns are estimated and do not account for any unforeseen
shocks to the economy at least in the ASAL region. However, even with the underlying
assumptions of the CBA, we are able to inform program implementers on the relative cost-
effectiveness of the different interventions, where CFS is the most cost-effective. On the other
hand, WASH and Solar power both yield returns that recover all of the initial investment, but
after at least 10-11 years. This means that the depreciation rate of physical capital in terms of
when reinvestment in WASH facilities and Solar panels is necessary.
10
INTRODUCTION This report presents the results of an observational study conducted by FHI 360 for UNICEF/ Kenya,
examining the implementation and early outcomes of the different interventions implemented in the Arid
and Semi-Arid Lands (ASAL) of Kenya. The study examines schools affected by four different types of
interventions, including water, hygiene and sanitation (WASH), solar panels, child-friendly schools (CFS),
and community involvement through communication for development (C4D), as well as their
combinations.
Background The ASAL region forms 84 percent of Kenya’s land mass and are characterized by low rainfall (see Figure
4). These regions are sparsely populated making it difficult to provide infrastructure and social services
within reasonable distances. The economic mainstay of
these areas is pastoralism. The regions have low
enrollment rates compared to other areas in the country.
According to the 2014 Education Statistical Yearbook,
the bottom 7 counties in the country in terms of primary
gross and net enrollment rates were in the ASAL region,
with only Isiolo (14th lowest, but still below the national
average) above this (MoEST, 2014). Further, an
analysis conducted to map schooling levels of all
persons aged 6 and above indicates that only 32 percent
of the population in Northern Kenya have ever enrolled
in school compared to the national average of 77
percent (KNBS, 2012). The low levels of primary and
secondary education attainment are attributed to early
or forced marriages; child labor, and other socio-
cultural practices that place a low value on formal
education. To overcome some of these challenges, the
Government of Kenya (GOK) has invested in a series
of initiatives to promote education in the ASALs. These
initiatives are complemented by international
development partners, nongovernmental organizations
(NGOs) and community-based organizations (CBOs),
including: DFID, UNICEF, the World Food Program, and the World Bank.
UNICEF Interventions in the ASAL Region In 2013-2014, UNICEF partnered with the Ministry of Education Science and Technology (MOEST) and
the United Kingdom’s Department for International Development (DFID) to establish a policy and
institutional framework to provide quality and child-friendly education services for nomadic children.
Today, this program benefits an estimated 3.5 million boys and girls aged 4–18 years in arid counties in
Kenya. The program consists of four discrete school-based interventions:
1. Water, sanitation, and hygiene (WASH), including constructing or connecting improved water
sources to schools for hand washing, drinking, bathing, and sanitation; building gender-sensitive
school latrines with hand washing facilities; conducting hygiene promotion training for teachers
and school children; and building capacity of Schools Boards of Management (BoM) to ensure the
financial and operational sustainability of services.
2. Solar lighting for two dormitories (boys and girls), ablution facilities, and two upper primary
classrooms in each school, to maximize afterschool time and improve security.
Figure 1 – Map of Kenya
11
3. Child-Friendly Schools (CFS) capacity development training developed teacher skills in child-
centered pedagogy; addressed issues of gender inclusiveness; developed teaching and learning
materials; and trained primary school head teachers to improve school management. These
activities were undertaken through a CFS monitoring toolkit, action research, and a mobile phone-
based school data monitoring and evaluation system. The program also established children’s
governance groups to enhance student leadership, participation, and empowerment in school
decision making.
4. Communication for Development (C4D) aims to build community capacity to meaningfully engage
and participate in analyzing challenges and barriers and identify priority solutions for education
access and quality issues in schools. The intervention used local channels to communicate and
promote understanding of the legal framework of education governance. Community members
were inducted in CFS principles and school management and disaster risk reduction (DRR), and
engaged parents, teachers, and pupils together in school-based change.
5. Combination schools received a combination of more than one of the above interventions, or
received all four.
METHODOLOGY Given the complexities of the operational environment in the ASAL regions, the number of substantive
changes that took place with respect to the assignment of schools to interventions, as well as the duration
and timing of the interventions, UNICEF and FHI 360, in consultation with DFID, agreed to change the
methodology of this evaluation from a randomized control trial (RCT) to a quasi-experimental design
(QED).
The study focuses on the full spectrum of interventions implemented in the ASAL regions by UNICEF with
funding from DFID, including WASH, installation of solar lighting, CFS, and C4D, as well as the
combination of these interventions, with the goal of understanding the magnitude of interventions and their
expected effects, using a framework that accounts for the duration, “dosage” and enabling environment
surrounding interventions and their implementation. The framework also links interventions to their
intended impacts at the student, school, and community levels (see Annex D for specific outcomes by
intervention). Finally, we unpack the how and why of these linkages in an in-depth way through focus group
discussions. While the initial design presumed single-treatment arm interventions, it was revealed during
the baseline that most schools had already had more than one treatment in place or completed shortly before
the baseline. Therefore, the endline data are designed to capture the net effect of a combination of different
interventions completed at different times while our analysis controls for pre-baseline conditions and
background characteristics of the schools.
Selection bias cannot be fully ruled out from the study since the selection of schools into treatments was
not purely random and many schools had experienced prior interventions similar in nature to the ones being
evaluated which may have affected their participation in the interventions. However, the QED using the
full sample provides useful insights into what worked and what did not in the implementation of the
interventions participating schools. In addition, the study allows for an initial testing of the theories of
change that drove the design of the interventions. The results allow for a more nuanced understanding of
the complementary effects as well as the sequencing of different interventions, and an evidence base about
what changes can be expected given the nature of the operating environment and competing pressures
facing the schools and communities in the ASAL regions.
Key data sources The following instruments were used in the endline study. Following the baseline, it was determined that
additional instruments were needed to collect more in-depth data related to the interventions. The team
12
added a school observation protocol; classroom observation protocol; student survey; and expanded the
existing Head Teacher questionnaire and focus group protocols.
1. Head Teacher questionnaire
2. School observation tool, including a) a checklist of school facilities; and b) enrollment,
attendance and retention data tool capturing data from school registers
3. Focus group guides, including: a) student focus group and b) community/ parent focus
group;
4. Classroom observation tool
5. Rapid student survey on study behaviors
Each of these instruments were administered following an extensive series of training sessions carried out
by FHI 360 staff in consultation with a local data collection firm. Enumerators were recruited from a
selection of individuals that participated in the baseline, as well as recommendations from FHI 360 staff in
Nairobi. A series of training workshops were carried out during the last two weeks of June and into early
July, 2015, in Nairobi. The classroom observations were carried out using the Stallings methodology1
(Stallings, 1980) and the instrument was designed to capture differences in teaching practices and level of
resources across different types of schools. In each data collection team, at least one enumerator received
training on the Stallings methodology by FHI 360 staff, prior to the launch of endline data collection.
Training was carried out during a day-long workshop at the FHI 360 Nairobi country office, and follow-up
support was provided the following week during a more extensive week-long training on all of the data
collection instruments.
The data sources listed above provide information on the key immediate, lower-level outcomes that can be
plausibly expected in affected schools at the time of the endline. The initial data collection took place from
July 6 – 31, 2015. The follow-up and re-verification of data took place from September 21 – September 30,
2015. A teacher strike that was carried out during re-verification did not impact data collection, because
head teachers remained at public schools to administer examinations.
Constructing the comparison group Descriptive analysis The construction of comparison groups in this study is based upon the availability of data, the timing of
intervention durations, and the implementing organizations associated with the different interventions.
UNICEF vs Non-UNICEF interventions – We classify interventions by source of the implementing
partner at each school. During the baseline, we found that WASH, installation of solar lighting,
CFS, or C4D interventions were implemented in a large number of schools by non-UNICEF
partners.2 For example, the Government of Kenya is implementing a large-scale solar panel
intervention, and many schools in the sample received support from either/or, or both programs.
Including them in the comparison group without any additional markers may not be appropriate,
and therefore we separate schools into categories based upon whether they received the intervention
from UNICEF (or implementing partner organization), from a different organization, or not at all.
“Longer duration” vs, “shorter duration” – In accordance with our duration/dosage/ enabling
environment framework, we note the length/duration of the intervention as a way of predicting
impact on outcomes. As we note above, the baseline revealed that some schools had already begun
1 The Stallings Observation System was developed in the 1970s in the United States to evaluate time use in
classrooms, and has since been adapted to other country contexts by the World Bank and other organizations. 2 We identify an intervention as being implemented by UNICEF if the Head Teacher of a given school identified
UNICEF as one of the implementing partners of a given intervention.
13
implementation of their intervention prior to data collection. It was therefore determined that the
endline data collection instruments would capture whether the intervention had been implemented
by a UNICEF partner or a different organization, both prior to June 2014 and after June 2014. This
is particularly relevant for the WASH and CFS interventions, as no schools report a C4D
intervention and only 4 schools report that UNICEF installed solar lighting prior to June 2014.
In this report, the above categories are applied in descriptive analysis, baseline-endline changes, and
regression analysis, and comparisons to a constructed control group are done on the basis of propensity
score matching as outlined below.
Regression analysis To evaluate the impact of an intervention on key outcomes among schools receiving the intervention, we
estimate count and linear regression models, respectively. It is important to note that for the regression
models to isolate the impact of the intervention from potential confounders such as school size, number of
teachers, head teacher characteristics, and geographic location, we must compare schools that received the
intervention to similar schools that did not.3
Figure 2 below plots the propensity score distributions for schools that received the WASH intervention,
and schools that never received any treatments. In a randomized controlled trial, or in a quasi-experimental
design, schools receiving the treatment and schools in the control group would exhibit overlapping
propensity score distributions. However, we can see that the two groups of schools display rather distinct
distributions with little overlap in propensity scores providing evidence that the two groups are dissimilar
and may not serve as a viable point of comparison to evaluate the interventions.
Figure 2 - Propensity Score Distribution, by Treatment and Pre-Designated Comparison Groups
3 We compute propensity scores using logistic regressions of the probability of being assigned the treatment as a function of their
observable characteristics to ascertain the level of similarity between intervention and pre-designated control schools. The
propensity score measures the likelihood that a given school will be assigned the treatment.
14
As a result, we exclude schools from the designated control group that never received any treatments that
are dissimilar to the treatment schools. Therefore, for each treatment, we construct a control group that is
composed of schools that are somewhat similar to the treatment schools that did not receive any treatments
in the period prior to June 2014, and did not receive the corresponding treatment in the period following
June 2014. For example, the constructed control group for the WASH schools will have received no
interventions in the period before June 2014 and will receive no WASH intervention after June 2014. Figure
3 plots the propensity score distributions of each intervention and its corresponding comparison group.
Figure 3 - Propensity Score Distribution, by Treatment and Constructed Comparison Groups
Although not perfect, we observe a larger degree of similarity between the treated schools and the
‘constructed’ comparison groups, as evidenced by the larger degree of overlap in the propensity score
15
distributions of the treatment and comparison groups. As such, we will use these comparison groups in our
regression analyses that evaluate the impact of each intervention on outcomes of interest such as student
enrollment, attendance rates, and grade promotion rates.
We estimate a count (Poisson) regression model to ascertain the impact of each intervention on enrollment
growth between 2014 and 2015, and a linear regression model for measuring growth in attendance and
promotion rates over the same timeframe.4 Specifically, we regress 2015 enrollment in 5th and 6th grade on
WASH, installation of solar lighting, CFS, or C4D intervention status, 2014 enrollment for the same grades,
characteristics of the head teacher, number of teachers at the school, teacher experience distribution,
distance to the nearest source of water, and county fixed effects. Lastly, we assess the effectiveness of the
Solar intervention on an additional outcome, students’ reported frequency of study, using a logistic
regression model.5
Description of the study population The target population of the schools benefiting from UNICEF interventions consist of low cost primary
boarding schools (LCBPS) located in the ASAL region of Kenya. The schools received one of the five
interventions (i.e. WASH, installation of solar lighting, CFS, C4D, or combined), sponsored by UNICEF
Kenya in the counties of Garissa, Isiolo, Mandera6, Marsabit, Samburu, Tana River, Turkana, and Wajir.
These eight regions are chosen to achieve broad coverage of the ASAL region in Kenya, known for its low
enrollment and educational attainment rates. The unit of analysis is at the school level for the quantitative
data collection, and at the intervention cluster level for the qualitative focus groups. Cluster sampling by
intervention was done at the regional level and is not representative at the county level.
The baseline data were collected through a school head teacher questionnaire and complemented by a series
of focus groups with students, parents/community members, and teachers. The endline data collection
process included visits to the same schools to gather data using the instruments focused on the study
objectives listed above. The sample size for the study was 349 schools distributed across the counties.
During endline data collection, the teams were able to reach 321 schools, while 28 schools were unable to
be reached due to security reasons. Table 1 below highlights the number of schools reached by treatment
intervention. As the table shows, most of the schools that were unable to be reached were control schools.
Table 1 - Number of schools by type of intervention Treatment Total number of schools by
UNICEF intervention Number of schools reached in
endline survey
C4D 49 47
CFS 50 47
Solar 48 44
WASH 53 51
Combination 55 51
Control 94 81
Total 349 321
4 The Poisson model is appropriate when the dependent variable represents counts as in the case of total enrollment. 5 We estimate the impact of solar panels on the probability that a given student reports studying at least once a day. 6 Baseline data collection in Mandera County could not be completed due to security concern.
16
Characteristics of the sample As discussed in the methodology section, the original number of endline schools was 349 compared with a
sampling frame of 440 schools in the baseline7. For the endline, FHI 360 was able to reach approximately
28 percent more schools (321 compared to 250) than in the baseline, producing a larger dataset. In Isiolo,
Samburu and Tana River, the teams were able to reach all UNICEF supported schools. Mandera and Garissa
had the highest number of schools not visited due mainly to security issues that prevented the teams from
reaching those schools.
Figure 4 and Figure 5 provide some descriptive background statistics about the schools and counties that
were visited. 104 out of the 321 schools visited were in Turkana, while a small number were visited in Tana
River (11) and Isiolo (14). Error! Reference source not found. Figure 5 shows that pupil-teacher ratios
range across treatment arms from 71 in WASH schools to 98 in Solar schools.
Figure 4 - Number of schools reached by county
Figure 5 - Pupil-teacher ratio
RESULTS
Key Findings This section presents the overall summary findings for key outcomes associated with each intervention.
The data are disaggregated by intervention cluster and gender where appropriate. Since schools across all
treatment categories have received interventions from UNICEF and non-UNICEF sources, we unpack the
treatments by duration and source to get a better assessment of how UNICEF partner supported schools are
performing comparatively across the region.
WASH – We are able to divide schools that received the WASH intervention into those that received the
intervention prior to June 2014 (77 schools), after June 2014, and from non-UNICEF organizations.
Both the descriptive and regression analyses show that schools that implemented WASH the
longest (prior to June 2014) exhibited the largest gains in enrollment relative to all other schools.
Specifically, we estimate that enrollment among schools that implemented WASH the longest
7 Prior to the endline, UNICEF and FHI360 worked with the implementing partners to verify the schools where the interventions
had been implemented. This verification process produced the final list of 349 schools. We know from the baseline process that
many schools that were part of the 440 ceased to exist for different reasons.
104
52
51
35
32
22
14
11
0 50 100
(#)
Turkana
Marsabit
Samburu
Garissa
Wajir
Mandera
Isiolo
Tana River
98 9587 85
7771
02
04
06
08
01
00
(#)
Solar
Con
trol
CFS
Com
bo.
C4D
WASH
17
increased by approximately 20 percent relative to similar schools in the constructed comparison
group that never received the WASH intervention.
Attendance rates are also highest among WASH schools with the longest duration. Relative to
similar schools that never implemented WASH, we estimate an 11 percent increase in attendance
rates, although this estimate is not statistically significant at the conventional levels, due possibly
to the amount of variability in the attendance data given the sample size.
Head Teacher perceptions of WASH were more positive among those that had received WASH for
a longer duration than those that received WASH after June 2014.
Solar Power – There are only 4 schools that implemented the solar intervention prior to June 2014. As a
result, we consolidate all schools that received Solar power interventions from UNICEF and designate them
as UNICEF Solar schools.
A higher proportion of students in UNICEF Solar schools report studying for more hours per day
as compared to non-UNICEF Solar schools and schools that never received Solar.
All else being equal, the estimated likelihood that a given student reports studying at least once a
day is 13 percent higher among UNICEF Solar schools than similar schools that never received the
Solar intervention.8
Regression results show that enrollments are only modestly higher among UNICEF Solar schools
than similar schools in the constructed comparison group, by about 6 percent.
Head Teacher perceptions of the Solar intervention’s efficacy are more positive among UNICEF
Solar schools than non-UNICEF schools.
CFS – We are able to identify 18 schools that received the CFS intervention prior to June 2014.
Regression results suggest that enrollment growth among schools that implemented CFS prior to
June 2014 as well as those that implemented after June 2014 is 8.6 percent higher than the
constructed comparison group. However, non-UNICEF CFS schools exhibited the largest gains in
enrollment.
Changes in attendance and grade promotion rates among all UNICEF CFS schools are not
statistically different from those of the comparison group. However, descriptive analyses show that
changes in attendance rates among UNICEF schools that implemented CFS after June 2014 were
higher than schools that did not implement CFS. Changes in grade promotion rates among boys
were higher for all UNICEF CFS than non-CFS.
Classroom observations suggest that teachers in schools receiving the CFS intervention are more
likely to exhibit characteristics consistent with the approach that is promoted by the intervention,
e.g. ask probing questions such as how and why, ask students to explain in class discussions, and
ask students to apply understanding to relevant life experience. However, teachers in schools
receiving the CFS intervention are only slightly less likely to integrate gender sensitive material
into pedagogy.
C4D – The C4D intervention was largely implemented in conjunction with other interventions.
Only six schools report receiving only C4D from UNICEF, therefore, we are unable to accurately
isolate the impacts of C4D from that of other interventions. Further, there are no schools that report
receiving a C4D intervention prior to June 2014, so we are unable to account for the duration of
treatment effect.
8 Study frequency of students is self-reported by the students in response to the student survey item “How often do
you study in a typical week?”
18
Overall, the descriptive analyses show a positive effect on enrollment and attendance, but not grade
promotion.
Additional descriptive evidence suggests that C4D schools are more likely to be involved in
community mobilization efforts around the importance of schooling and the promotion of education
for out of school children.
Combination Schools – Schools that receive two or more UNICEF interventions are more likely to show
positive effects on attendance and enrollment. We also analyze differences in the outcomes of interest
among specific pairwise intervention combinations.
Schools that received a combination of Solar power and C4D exhibited large gains in enrollment
relative to the constructed comparison group by 10.5 percent. The same combination of
interventions yielded a 46.5 percentage point increase in attendance rates and a 14.8 percent
increase in the likelihood of students studying more than once a day.
Schools implementing C4D in conjunction with CFS are estimated to have an attendance rate that
is 28 percentage points higher than that of the constructed comparison group.
Schools that implemented both CFS and WASH experienced approximately a 9.4 percent increase
in enrollment relative to the constructed comparison group. Although the estimate is positive and
certainly non-negligible, it is not statistically significant at the conventional levels.
Cost-Benefit Analysis – We compute the benefits of WASH, installation of solar lighting, and CFS
interventions as additional taxes paid over a lifetime as a result of increased educational attainment from
each intervention. Additionally, to compute the total benefits accrued to each intervention, we aggregate
across the total number of students who are estimated to have increased their educational attainment but
would not have if not for the intervention. Lastly, we compute the number of years required for each
intervention (investment) to yield a full return, i.e. each Kenyan shilling invested is returned in full.
We estimate that the total investment in WASH schools, where we are able to identify the
enrollment impact, may be returned in full between 11 and 21 years post-implementation,
depending on the discount rate. Lifetime benefits-to-cost ratios for WASH range between 1.19 and
3.85 per US Dollar invested.9 This means that every dollar invested in WASH is returned between
1.19 and 3.85 times 36 years after implementation.
Investments made for Solar power are estimated to be returned in full after 10-18 years, depending
on the discount rate. The lifetime benefits-to-cost ratios for the investment in Solar lighting range
between 1.27 and 4.17 per US Dollar invested.
Child friendly schools yielded the highest return on investment according to our simulation model.
Initial investments in CFS are returned in full after only 4-5 years and the lifetime benefits-to-cost
ratios for CFS range between .04 and .12 per USD invested.
Overview of changes in key outcomes This section presents trends in enrollment, attendance and retention over the intervention period. Schools
are organized according to the originally designated UNICEF intervention categories. Note that this is a
description of a trend, not accounting for the differences in the composition of the schools or duration of
their interventions. This contrasts with how interventions will be designated in the following sections,
9 Benefit-to-cost ratios are impervious to the currency used in the calculation. As such, we may use Kenyan
Shillings or US Dollars interchangeably in benefit-to-cost ratios.
19
where we seek to account for the duration of exposure to treatment and restrict analysis to comparable
schools.
Enrollment trends over the intervention period Table 2Error! Reference source not found. shows average enrollment per school across the initially
designated treatment groups. As the table shows, enrollments generally decreased across all schools in the
sample. The reason for the general trend is not known, although the focus group discussions provide some
indication – namely safety and security. However, some increases are seen, particularly for boys. Growth
in boys enrollment was observed in the Combination and Solar category schools (+14 percent for each
category), while enrollment in WASH schools has stayed relatively even. Enrollment also increased for
girls in Combination schools (+5 percent), but decreased across all other interventions. Also notable is the
change in Solar schools, which saw a large decrease in enrollment for girls (-17 percent) but also a large
increase for boys (+14 percent). These results should be interpreted as general trends, while the regression
results presented later are intended to serve as a means of comparison on enrollment trends between groups.
Table 2 - Enrollment changes over the intervention period10 Treatment Girls Boys % change
2013 2015 2013 2015 Girls Boys
C4D 441 459 345 382 4.1% 10.7% CFS 413 344 301 270 -16.7% -10.3% Combo. 488 422 351 327 -13.5% -6.8% Control 405 413 325 383 2.0% 17.8% Solar 369 343 317 358 -7.0% 12.9% WASH 322 351 290 299 9.0% 3.1%
Attendance rate trend over the intervention period In addition to enrollment, we measured attendance by taking information from class registers for the day of
enumeration, and dividing it by the corresponding enrollment figure for that grade. Figure 6 presents a
mixed picture of attendance changes from baseline to endline. While overall, attendance rates are relatively
high (generally around 90 percent, see Annex D), the positive changes one would like to see as a result of
the interventions are not seen across all categories. There are small but clear drops in attendance rates across
all interventions. Similar to enrollment, there are drops in attendance among control schools. Combination
schools are the only intervention type that see increases, most notably for boys (from 84 percent to 92
percent, see Annex D for overall numbers).
10 Note that intervention designations changed between baseline and endline, which is why enrollment numbers do not match
what is in baseline report.
20
Figure 6 – Attendance rate changes over the intervention period
Changes in retention A similarly mixed picture emerges when we look at grade-to-grade retention over the intervention period.
Error! Reference source not found. Figure 7 shows promotion rates between grades 5 and 6 for the two
year (Figure 7). Interestingly the combination schools see drops in promotion rates, while enrollment and
attendance remain high. There is also a drop in promotion for girls in CFS schools. However, on the whole,
promotion rates are relatively high and retention is not a major problem across the intervention schools (see
Annex D)11.
Figure 7 – Changes in retention over the intervention period
Taken together, the data on our outcomes of interest show that there is a substantial degree of fluctuation
on all key outcomes – enrollment, attendance, and retention (grade promotion). Girls’ attendance levels
were lower at the time of the endline survey, compared to the baseline. Promotion rate changes differed
across genders, and the measured changes in promotion rates are not consistent with our hypotheses (e.g.
WASH schools did not show growth in girls attendance). Overall, on face value there do not seem to be
clear differences in outcomes between intervention schools and schools in the designated control group.
However, as the next section will show, when duration of the intervention is taken into account and the
analysis sample is restricted to comparable schools the differences in outcomes are better identified.
11 It should be noted that there is automatic promotion between grades at the primary level, in Kenya. That said, many schools do
report repeaters, and the numbers presented are for a subset of schools for which the data are available.
-6.2-3.2
7.5
-3.8
-14.1-12.4
-20.0
0.0
20.0
% c
han
ge
C4D CFS Combo. Control Solar WASH
Girls Attendance Rate Change, Grades 5-6
-2.1
-11.7
8.0
-6.3
-14.3
-19.9
-20.0
0.0
20.0
% c
han
ge
C4D CFS Combo. Control Solar WASH
Boys Attendance Rate Change, Grade 5-6
-4.7
-16.6
-5.8
2.23.9
-11.8
-20
.0
0.0
20
.0
% c
ha
ng
e
C4D CFS Combo. Control Solar WASH
Girls Promotion Rate Change, Grades 5-6
-5.6-3.6
-9.0
1.1
-9.0
1.2
-20
.0
0.0
20
.0
% c
ha
ng
e
C4D CFS Combo. Control Solar WASH
Boys Promotion Rate Change, Grade 5-6
21
Indicative results of the WASH Interventions Status of WASH interventions Beginning in 2014, three international NGOs
(World Vision, CARE, and Food for the Hungry)
began implementing WASH interventions in
schools across the counties of Samburu, Turkana,
Tana River and Wajir. The objectives included
building latrines and hand washing facilities in
schools where none existed; connecting improved
water sources to schools for hand washing, drinking,
bathing and sanitation; building gender-separated
school latrines; conducting hygiene promotion
training and building the capacity of school
management committees to ensure the financial and
operational sustainability of services. In schools
where facilities already existed, the organizations
either refurbished existing facilities or
complemented the work that had been completed by
other organizations or the government.
How effective are the WASH interventions for outcomes of interest? The hypothesis regarding WASH interventions is
that improved water and sanitation systems will
increase school attendance, particularly among girls. As a result, the intervention will increase their overall
exposure to instruction leading to higher academic performance.
As noted earlier, our data allow us to distinguish schools that reported having received the WASH
interventions from UNICEF providers since prior to the program implementation period in question, from
those that had been receiving the intervention only since June 2014. Such a distinction is important for
determining the extent to which the duration of treatment may have had an effect on outcomes of interest.
As seen in Figure 8 presented below, enrollment increases were greater in schools that reported receiving
the WASH treatment from UNICEF since prior to the start of the program period in question (i.e. since
prior to June 2014), compared to a) schools that reported receiving the WASH intervention funded by
UNICEF for a shorter duration, and b) schools that did not receive a WASH intervention from UNICEF.
The enrollment increase in the “long duration UNICEF intervention” group was four times as large for
boys, and twice as large for girls as either of the comparison groups. Attendance rate changes should be
interpreted with caution, since overall level of observed attendance is relatively high (80%-95%);
nonetheless, when broken down by grade, small increases in girls’ attendance may be observed in “longer
duration” UNICEF WASH schools (Figure 8).
World Vision: World Vision (WV) implemented
WASH interventions in 56 schools across the four
counties. Their interventions consisted of building or
rehabilitating latrines; building or rehabilitating water
sources; provision of hygiene training to teachers; and
supporting the dissemination of school health policies.
Food for the Hungry: Food for the Hungry
implemented WASH in 25 schools across Marsabit and
Isiolo. FFH also GPS mapped the schools and
implemented similar interventions to WV.
CARE: CARE implemented WASH in 20 schools in
Garissa. Their focus was primarily on establishing a
stable water source (particularly piped water); and
building the capacity of staff to teach about WASH. In
six of these schools, there was no piped water so CARE
provided treatment chemicals to treat the local water
source, ensuring cleaner water for students and staff at
the schools. Similar to WV and FFH, CARE began
interventions in June 2014 with a rapid assessment and
initiated actual implementation in September 2014.
Implementation was completed in March 2015.
22
Figure 8 – WASH changes in key outcomes
The perceptions of head teachers in WASH schools were more unambiguously positive about the effects of
the interventions, particularly in schools that had received WASH interventions for a longer period of time.
As shown in Figure 9Error! Reference source not found., 94% of head teachers in schools that had
received UNICEF WASH interventions since prior to June 2014 agreed that it had positively impacted
attendance, compared to 83% in schools where WASH was implemented after this period. Similarly, head
teachers were more positive about effects on enrollment (84% to 76%) and dropout (75% compared to 66%)
in the former types of schools.
Figure 9 – Head teacher assessment of WASH outcomes
-23.3
76.7
-7.2
100
.0
0.0
-100
.0
# c
han
ge
Other WASHpre & post 6/14
WASHpost 6/14
Boys Mean Enrollment Change
-20.7-3.3
-30.9
100
.0
0.0
-100
.0
# c
han
ge
Other WASHpre & post 6/14
WASHpost 6/14
Girls Mean Enrollment Change
-4.9
8.2
-14.5
-20.0
0.0
20.0
% c
han
ge
Other WASHpre & post 6/14
WASHpost 6/14
Boys Attendance Rate Change, Grades 5-6
-8.0
11.0
-5.5
-20.0
0.0
20.0
% c
han
ge
Other WASHpre & post 6/14
WASHpost 6/14
Girls Attendance Rate Change, Grades 5-6
23
Analyzing the impact of the intervention in more depth, it is clear that there are moderate to large positive
differences between schools that have received the WASH intervention for a longer period and other types
of schools. For example, 56 percent of schools that report receiving the intervention since prior to June
2014 (hereinafter referred to as “longer duration” schools) have a water source on the school compound,
compared to 38-39 percent of shorter duration WASH schools and comparison schools with no WASH
intervention (see Annex D for a more detailed breakdown). Similarly, 29 percent of “longer duration
WASH” schools report a water source that is piped or from a well or borehole, compared to only 20-21
percent of comparison schools. Head teachers also considered handwashing and latrine facilities to be
somewhat safer in such schools, compared to comparison schools. Similarly, we see large differences
between types of schools on whether they have received training in hygiene promotion. 94 percent of longer
duration WASH schools have received such training, compared to 74 percent of WASH post-June 2014
only and only 56 percent of comparison schools (see Annex D).
One emphasis of the UNICEF WASH program is to provide separate latrine facilities for boys and girls,
with the goal of improving school attendance among girls. While most schools regardless of intervention
had separate latrines for girls and boys, a slightly higher (97 percent) percentage of longer duration WASH
schools reported separate latrines, and WASH longer duration schools reported a much greater number of
latrines compared to other types of schools (see Figure 10Error! Reference source not found.). The mean
number of latrines per school is significantly less for girls (5.0 compared to 7.6) and boys (5.2 compared to
8.2) in the baseline, compared to the endline. The pupil-toilet ratio has also dropped significantly from the
baseline. While the ratio decreased from 91 to 72 in non-UNICEF WASH schools, the ratio in UNICEF
WASH schools is roughly half what it was during the baseline.
Figure 10 – Changes in availability of latrines
Finally, a goal of the UNICEF WASH program is to establish clean handwashing facilities close to latrines.
While this remains a challenge across all types of schools, a greater percentage of WASH long schools (34
percent) report clean handwashing facilities close to latrines, compared to other types (32 percent in WASH
short and 24 percent in other types of schools).
WASH regression analysis We disaggregate the WASH treatment group into schools that received the intervention for at least two
periods (pre- and post-June, 2014) and those receiving the intervention only post-June 2014. Further, we
interact the group of schools who received the WASH treatment only in the post-June 2014 period with
their treatment status from the other interventions, i.e. for WASH, we identify the pairwise combinations
of schools that received WASH in conjunction with other interventions.
5.1 5.4 5.0
7.6
4.7
6.5
05
10
(#)
Other WASHpre & post 6/14
WASHpost 6/14
Mean No. of Girls Latrines
Baseline Endline
5.05.8
5.2
8.2
5.0
6.7
05
10
(#)
Other WASHpre & post 6/14
WASHpost 6/14
Mean No. of Boys Latrines
Baseline Endline
24
For ease of interpretation, we compute the marginal effects associated with the Poisson regression results
and are interpreted as the increase in the number of students enrolled from one year to the next for WASH
schools relative to similar comparison schools. The marginal effects and regression coefficients from the
linear regression model are one and the same. For attendance and promotion rates, we measure the
percentage point difference between WASH schools and similar schools in the comparison group.
Figure 11 – Margins plot of WASH outcomes Growth in Enrollment
Growth in Attendance Rates
Growth in Promotion Rates
Notes: Marginal effects associated with enrollment are expressed as changes in enrollment in response to the intervention, in counts.
Marginal effects associated with attendance and promotion rates are expressed as percentage point changes in response to the
intervention. Intervention statuses associated with fewer than 10 schools are suppressed. Intervention statuses associated with fewer
than 10 schools are suppressed.
Figure 11Error! Reference source not found. plots the marginal effects from the enrollment, attendance
rate, and promotion rate regression results along with 90 percent confidence intervals for each estimate.
Schools that received the WASH intervention in both the pre- and post-June 2014 periods experience a 23
student increase (effect size of 22.4 percent of a standard deviation) relative to similar schools that never
received the WASH intervention and did not receive any other interventions prior to June 2014. We also
see a statistically significant 15.4 student increase (effect size of 10 percent of a standard deviation) in
enrollment among non-UNICEF WASH schools relative to similar schools in the comparison group. We
find almost no effect on WASH schools that have had the intervention only recently, suggesting that the
intervention may require additional time for its effects to materialize. Schools that implemented WASH
along with CFS exhibit a positive increase in enrollment. However, this estimate is not statistically
significant.
We find a positive, though statistically insignificant, effect of WASH on attendance rates among schools
that received the treatment in two periods by 11.1 percentage points. WASH schools that received the
treatment in the most recent period and in conjunction with CFS exhibit no differences in attendance with
comparable non-WASH schools. The estimated treatment effect among non-UNICEF WASH schools is
also not statistically different from zero. Lastly, when examining the impact of the WASH intervention on
grade promotion rates, we find modest positive effects across all types WASH schools by between 1.5
percentage points and 4.3 percentage points. However, these estimates are not significantly different from
zero.
Challenges in implementation and monitoring of WASH interventions. Across the literature, we know that there are a number of constraints in sustaining WASH interventions. To
understand the constraints that UNICEF-supported currently face in maintaining WASH facilities, we asked
school Head Teachers a series of questions related to these challenges. Error! Reference source not found.
Figure 12 shows the percentage of head teachers reporting challenges in initial installation or construction,
maintenance or upkeep of facilities, and usage of facilities by students, teachers and staff, by type of school.
25
As can be seen, schools working with non-UNICEF WASH implementers report slightly more challenges,
particularly in ongoing maintenance of facilities. We can also get some idea of how head teachers view the
causes of these challenges. Error! Reference source not found.2 also shows the percentage of schools
citing a lack of funding, no training in how to use the facilities, and poor quality of the service provider as
primary causes of challenges in implementation.
Figure 12 - WASH challenges in implementation, as reported by Head Teachers
Findings from focus groups As discussed at the beginning of this section, increasing the equitable access to and use of safe water and
basic sanitation services improves health and education outcomes. Numerous reports confirm the
importance of the link between WASH and education (UNICEF, 2006; Gleaton, 2012; Jasper, Le, and
Bartram, 2012). These studies also indicate that issues of corruption in the water committees and delayed
repair of damaged infrastructure often result in erratic service and the lack of financial support, and
community engagement, were major limitations to sustaining and improving WASH interventions.
Access to clean water and hygiene facilities was of high importance to all of the parents, community
members and students who participated in the focus group discussions. These different groups noted that
having water at the schools allowed students to bathe, cook, and wash their clothes. Access to water made
it easier for to justify coming to school; and many parents noted the particular importance that the hygiene
training has had on reducing diarrhea, vomiting, and other water borne diseases. Several parents across all
counties noted:
“There are many pupils that are boarding in the school so this water is of great help because they use it
for cooking their food, bathing and washing their clothes. It is important for them to have water when at
the school” Parent FGD, Garissa.
“We have handwashing facilities which have been place next to the toilets and also if you look at
this basins, you see this one is filtered so you put water there and it’s safe for the children to drink
water.” Community Member FGD, Marsabit
Parents often noted that they were made aware of the WASH practices through meetings at the school and
that organizations such as UNICEF not only held workshops, but handed our materials to teach them about
the importance of hand washing and sanitation. One parent noted, “there was a meeting in this school where
all the schools from around here attended and we were taught how to wash our hands and were also given
leaflets with this information. The information was also written on buildings - wash hands after using the
toilets. As the pictures below show, many of the schools we visited had colorful walls painted with
instructions on good hygiene for students to follow.
63.663.3
74.8
85.2
48.6
59.0
05
01
00
(%)
Initial Maintenance Use
Percent Reporting Challenges in Implementation
UNICEF Non-UNICEF
50.9 54.1
29.537.7
20.524.6
05
01
00
(%)
Funding No Training Service
Percent Reporting Causes of Challenges
UNICEF Non-UNICEF
26
One additional issue that was raised in the focus groups centered around the way many of the WASH
facilities had been modified to ensure that physically challenged students could also use the facilities. One
of the areas that UNICEF specifically supported was making modifications to make WASH facilities more
child friendly – particularly for children with disabilities. Therefore, hearing the issue discussed among
parents in Marsabit reinforced the fact that the interventions were seen as important by the communities.
Finally, parents and students also mentioned that having the water and hand-washing facilities on the school
grounds reduced the amount of time that students had to take away from classes or studying, thus, increasing
the students time to focus on learning.
While much of the discussion was focused on the positive aspects of the WASH interventions, the
moderators were asked to also discuss the types of challenges that are faced in the school, vis a vis the
WASH intervention. As expected, parents noted that this year there have been more challenges related to
the access to clean water than in previous years. The drought was mentioned by nearly 30 percent of focus
group participants. They indicated that bore holes had been drying up at the schools or that there was no
longer access to clean water for hygiene purposes as a result of the drought. The following two quotes from
focus group discussions highlight the importance communities do place on the provision of clean water:
“Our school had a bore hole but it got dry, now students do not have water”.
Another participant notes that in some cases, schools end up relying on rainwater to fill the tanks because
the volume of children drinking the water exceeds the tank size. He noted, “…the water that we have….the
toilets are okay, but the water tank is too small to store enough water for all the children, the water will
not be enough because the children are many, so the water does not last long when they open school….water
does not last even a month, then we must depend on rainwater.”
The size and upkeep of the infrastructure posed a particular problem for many communities. Parents and
students noted, “….the toilets are okay but the water tank is small to store enough water for all the students,
the water will not be enough because the children are many. The water does not last long when they open
school the water does not last even a month….the, students go home or have to look for other sources of
water”.
The types of challenges that parents and students mentioned mirror both the literature and the perceptions
surveys conducted during this evaluation. The triangulated data shows that schools lack the resources to
maintain the infrastructure and that having sufficient water is often a key issue. Parents also indicated that
they continue to have problems with pipes bursting at the schools, which leads to water wastage and
subsequently, insufficient water for students. It was noted in several focus groups that,” …The toilets should
be kept clean and there is no water and soap….[communities need to support, but they often don’t]”.
While parents indicated that things have gotten better as a result of UNICEF WASH interventions and both
groups noted that having access to clean water does make a difference in attendance, this is a challenge that
UNICEF should consider supporting in the future. Parents in a focus group in Marsabit noted that, “If we
are provided with water, our children given food and the school is fenced we will keep them here. Nowadays
we are enlightened not like there before, we want our children to be educated – water and other things will
help keep them in school.”
27
Indicative Results of Solar-Power Interventions The provision of solar panels for lighting in schools was intended as a way of strengthening the schools
ability to provide safe study spaces after dark,
reduce the costs of power for schools, and
ultimately improve learning opportunities for
students. As part of this intervention, UNICEF
worked with Power Point Systems East Africa to
install solar lighting in two dormitories (boys and
girls), in ablution facilities, and in two upper
primary classrooms, in each school. To
understand the extent to which schools in this
sample have lighting, we collected data from head teachers, teachers and students at the school and
conducted school observations to verify the status of the infrastructure.
Status of Solar interventions For the solar intervention section, we look at UNICEF solar intervention schools and non-UNICEF solar
schools (most of which are supported by the GoK program). Non-solar schools mostly have regular
electricity. Approximately 198 of the 322 schools visited during the endline data collection process have
solar lighting. The solar lighting was provided to schools by many different actors including UNICEF, local
governments, and other international organizations. The solar interventions provided to schools by
UNICEF are located in dormitories, grade 7 and 8 classrooms, or the highest grade levels in the school,
latrines, and in some cases in the library. The objective of providing the solar panels is to increase the
amount of time students have available to study as well as provide additional security to the schools at
night. Error! Reference source not found.Figure 1312 shows the percentage of all schools by their source
of power, while Figure 1413 shows the number and proportion of solar schools by their implementing
organization. The largest implementer of solar power in ASAL schools in the study was reported to be the
government, followed by UNICEF.
Figure 13 – Solar: type of power
Figure 14 – Solar: UNICEF and Non-UNICEF
How effective are the solar interventions for outcomes of interest? Increasing the amount of study time for students is one of the key intended outcomes of the UNICEF has
supported solar power interventions (see Annex D). The belief is that if schools have power in places like
the dormitories and classrooms, that students – particularly those that board – will have more time to study
12 These percentages do not add to 100 because of some missing data on this response. 13 Although some schools have both UNICEF and government as implementers of solar intervention, a school is considered
UNICEF if it has both.
77.6
15.66.5
05
01
00
(%)
Type of Power
Solar Electricity None
78
120
UNICEF Other
Power Point Systems East Africa: In June 2014,
UNICEF contracted with Power Point to install
solar panels in approximately 100 schools across
all 8 counties in the study. At the time of
installation, two school staff members then
received training on use and maintenance of the
system.
28
since they can now study later into the evening and in the mornings when it is dark. Annex D shows the
head teachers’ evaluation of reported sufficient light in key areas of the school. UNICEF schools that are
implementing the solar intervention reported sufficient lighting in their classrooms at a higher rate than
other types of schools (63 percent of head teachers report sufficient light in the classroom in schools that
installed solar lighting compared to 39 percent where the intervention is not present at all).
Despite this, students report studying after dark in only about 50 percent of UNICEF solar schools,
compared with 52 percent of non-UNICEF solar schools and almost 62 percent of non-solar schools (Annex
D). Error! Reference source not found. Figure 15 shows the mean percentage of students who report
studying in the evening, per school. Both UNICEF and non-UNICEF solar schools appear at a disadvantage
compared to non-solar schools in facilitating the ability of students to study in the evening. Further, lighting
does appear to be a key impediment in the ability of schools to promote learning. While most of the schools
in the “no solar” category have electricity, Figure 15 shows the percentage of students who report being
unable to study at night14 due to poor lighting, across the same category of schools.
Figure 15 – Solar: differences in evening study
While it appears that solar schools overall struggle to provide opportunities for night study, it does appear
that a greater percentage of students in solar schools study for a longer period of time, per day. Error!
Reference source not found. Figure 16 shows the percentage of students by the number of hours they
report studying per day, by solar and non-solar schools. There is a trade-off between students studying less
than 2 and 2-4 hours per day, so that students in non-solar schools are more likely to study less than two
hours per day while students in solar schools are more likely to study 2-4 hours per day. A similar
percentage of pupils study more than 4 hours a day, regardless of intervention or implementing
organization.
Figure 16 – Hours of study
14 Students who indicate they typically study during the day (and not also during the evening) are asked why they do not study at
night.
24.4
18.2 18.9
01
02
03
0
(%)
Mean % of Students Studying in Evening per School
No Solar UNICEF Non-UNICEF
8.6
17.9
13.3
05
10
15
20
(%)
% Schools With Lighting Impediments to Night Study
No Solar UNICEF Non-UNICEF
36.529.8
22.7
49.6
59.163.9
13.911.113.3
03
57
0
(%)
Less Than Two Two to Four More Than Four
Mean % Students by Hours of Study per School
No Solar UNICEF Non-UNICEF
29
Similar to other interventions, head teachers were slightly more positive about the interventions impact on
students. Figure 17 shows the percentage of head teachers who agree or strongly agree that the named
intervention positively influenced a specific outcome. Schools where the solar intervention was
implemented by UNICEF saw slightly higher rates of agreement than schools where solar was implemented
by a different organization.
Figure 17 – Solar: head teacher assessment
Solar regression analysis We disaggregate schools’ treatment status to identify schools that received the Solar from UNICEF and
from a non-UNICEF organization. We do not distinguish between Solar schools that implemented the
intervention in the pre- and post-June 2014 periods since there were only four schools that did so. As such,
we consolidate all UNICEF schools as a single group. We still interact Solar treatment status with receiving
other treatments at the same time in the post-June 2014 period. We present graphically the marginal effects
of receiving the Solar intervention on student enrollment growth, growth in attendance rates, and growth in
grade promotion rates relative to comparable schools that never received the Solar intervention and did not
receive any other treatments prior to June 2014. Figure 18, below, plots the marginal effects associated with
various Solar treatment statuses along with their corresponding 90 percent confidence interval.
Figure 18 - Margins plot of Solar outcomes Growth in Enrollment
Growth in Attendance Rates
Growth in Promotion Rates
Notes: Marginal effects associated with enrollment are expressed as changes in enrollment in response to the intervention, in counts.
Marginal effects associated with attendance and promotion rates are expressed as percentage point changes in response to the
intervention. Intervention statuses associated with fewer than 10 schools are suppressed.
UNICEF and non-UNICEF implementing schools both show positive gains in enrollment as a result of the
Solar intervention by 6.99 and 5.74 students, respectively. The equivalent effect size is 7 percent of a
91.085.8
79.5
69.2
87.2 85.8
05
01
00
(%)
Attendance Dropout Enrollment
Agree or Strongly Agree
UNICEF Non-UNICEF
30
standard deviation for UNICEF Solar schools and 5.3 percent of a standard deviation for non-UNICEF
Solar schools. The estimated effects of the Solar intervention, however, are associated with relatively wide
confidence intervals. We find no effect among schools that implemented both the Solar and CFS
interventions in the post-June 2014 periods. However, we find a statistically significant 21.98 student
increase in enrollment among schools that implemented both the Solar and C4D interventions. We estimate
that the Solar intervention had little to no effect on attendance rates among schools that received the
intervention from UNICEF as well as from non-UNICEF organizations. Similar to the effects seen in
enrollment, schools that implemented both the Solar and C4D interventions exhibit a 46.5 percentage point
increase in attendance rates relative to similar schools in the comparison group. Grade promotion rates are
largely unaffected by the Solar intervention with estimated effects ranging between -0.81 and .72
percentage points, these estimates are not statistically significant at the conventional levels.
Lastly, Figure 19 displays the marginal effects of the Solar intervention on student reported study frequency.
In this case, we estimate the change in the likelihood of students who report studying at least once a day
relative to similar students in control schools who did not receive the Solar intervention. We find that
students attending UNICEF Solar schools are 13.2 percent more likely to report studying at least once a
day than similar students in non-Solar schools. Students in schools that implemented both Solar and WASH,
and Solar and C4D are 8.8 percent and 14.8 percent more likely to report studying at least once a day than
their counterparts. These estimates are marginally significant at the 80 percent confidence level.
Figure 19 - Margins plot of Solar effect on frequency of studying
Challenges in implementation and function of solar power As with WASH facilities, maintaining the solar infrastructure presents a challenge to the sustainability of
the intervention. It is not sufficient to implement the panels. Schools and communities need to monitor and
maintain the infrastructure to ensure long term use. When the head teachers were asked about the challenges
in monitoring and maintaining the solar panel, three main challenges emerged in the UNICEF supported
schools: Insufficient funding (47 percent); No training on the maintenance of the panels (41 percent); and
service quality (25 percent) (see Figure 20). The results were similar in the schools supported by other
organizations, with higher percentages of head teachers reporting these three areas as a challenge.
31
Figure 20 – Solar challenges in implementation
On the whole, challenges in implementation and monitoring appear to be correlated with outcomes of
interest (percent of students who study at night) to a slightly higher degree than with the WASH challenges,
particularly in training in use and quality of service. This would seem to indicate that challenges in
maintenance of the solar interventions represent a serious challenge, and that UNICEF may consider more
regular support to these schools in improving the delivery of education.
Findings from focus groups The implementation of solar power was intended to help improve student’s safety and their ability to study
for longer periods of time. As the data has shown, the intervention is both cost-beneficial in the long term
and educators perceive that it does support improved learning. The results of the focus groups can be broken
down into three main themes: more time for teaching and learning; increased school safety; and indirect
benefits.
Increased time for teaching and learning The importance of having focused time to study is well documented in the literature (Bruns et al, 2014;
Destefano et al, 2011). In the ASAL schools, the contribution of solar power to increased time on teaching
and learning was mentioned approximately 90 percent of the time across the 23 focus group discussions.
Parents and community members noted that students are now able to study in the morning and in the evening
- “It [solar power] has helped because nowadays children can go to school early in the morning to do
morning studies but before the installation of solar it was impossible because of the dark. So it has helped
so much.” Parent FGD, Marsabit
They also noted that teachers have more time to do their prep work as well as work with students who board
at the school in the evening. Parents and community members noted that, “These days we have night preps
and also morning preps. The teachers who are residing in the school are also able to teach at night for
those children who are around or those who are in the boarding.”
Students in the FGDs agreed that having solar power increased their ability to study for extended periods
of time. Students also extensively discussed the ability to study at night or early in the morning; read text
more clearly at night as a result of having lights; and being able to study in various locations. When asked
how many additional hours of study time they believed having solar power provided, at least six students
indicated that they had about 2-3 hours of extra study time now available that previously did not exist. The
following quote captures the collective sentiment of students related to the issues of learning: “We see well
when reading our books so we are able to read in the evening even when it’s dark for a longer time than
before.” Student FDG, Tana River.
38.5 38.7
70.578.3
39.749.2
05
01
00
(%)
Initial Maintenance Use
Percent Reporting Challenges in Implementation
UNICEF Non-UNICEF
47.4
55.0
41.046.7
25.6 24.2
02
04
06
0
(%)
InsufficientFunding
No Training Service Quality
Solar: Challenges in Monitoring and Implementation
UNICEF Non-UNICEF
32
Finally, while the FGD do not provide concrete evidence that increased lighting leads to improved reading
results, several parents/community members did feel that having more lighting was directly linked to the
learning outcomes of the school. One of the parents from Marsabit noted, “It [solar power] has improved
student learning because last year’s result of Kenya Certificate Exams this school did better than all other
public schools - it was number 1 in Moyale sub-county, Marsabit County.”
School Safety Multiple counties in the ASAL region are currently facing issues related to school safety. Schools are
occasionally attacked by members of armed groups and communities often feel unsafe sending their
children to school. The issue of school safety was brought up as a theme throughout the 23 focus groups.
Parents, community members and students indicated that having solar power allowed the schools to ensure
there was sufficient lighting in the school compound, which they felt, worked as a deterrent to crime and
made students feel safer. One student in Samburu stated that, “Lighting has improved security as criminals
do not come in school.” Students also noted that having lighting kept students from getting hurt at school.
“At night when going to the dormitory, we are able to see our way clearly and avoid dangers or injury –
for example, there could be a snake passing, a thorn on the ground, or something else that could hurt us.”
Finally, students in the FGDs also noted that having lighting at the school contributed to better hygiene
practices. Several students commented that before the school had lighting, many students would just “go
around the corner” to relieve themselves at night. But now, having the lights around the latrine facilities
enable students to use the latrines at night and is helping to keep the school cleaner.
Indirect Benefits The focus group discussions re-emphasized the elements of adding solar power that one would expect –
increased time for teaching and learning as well as school safety. But there were two additional indirect
benefits that parents and community members mentioned during the discussions. First, a parent in Garissa
County noted that while their school was only a primary school, that in the evenings students from the
nearby secondary school would come over to study in the evenings. While the indirect benefit of having
students from other schools use the lighting was only mentioned in Garissa, it is an important extended
benefit that should be examined in a future study.
The second indirect benefit is linked to teaching and learning, but relates specifically to class 7 and 8
students. Solar lighting was added to class 7 and 8 classrooms. Parents, community members and students
all mentioned that having lighting in these classrooms specifically helped students in these grades study for
their exams by giving them a more isolated area to study.
Challenges of Solar power While the FGDs mainly focused on the benefits that communities feel have accrued to their students, there
were a few important challenges that parents raised during the discussions. First, having the resources to
sustain the solar panels was a challenge to communities. Several focus group participants mentioned the
lack of resources to fix and maintain the solar panels. This challenge was also re-enforced in the head
teacher survey. Second, parents in the Garissa FGD mentioned that one of the buildings with solar power
had burned down and had to be rebuilt, which the community was able to eventually finance and complete.
However, it was not clear from the discussion whether the fire was related to the solar infrastructure, or
some other reason. Finally, parents and community members mentioned that in the schools where there are
no security fences, the solar panels were often damaged, or destroyed by “criminals”. While this issue was
only mentioned two or three times, it is another challenge that UNICEF should consider and further explore
with communities.
33
Indicative Results of Child-Friendly Schools Interventions UNICEF characterizes a child friendly school as one that is inclusive, healthy and protective for all children,
and positively engaged with families and communities with the aim of promoting safe and healthy
educational delivery at the school level. Planned project interventions involve capacity development
training aimed at enhancing teacher skills in child-centered pedagogy that focus on notions of inclusiveness,
including gender sensitivity and sensitivity towards students with disabilities. The CFS approach includes
the following activities: (1) how to develop teaching and learning materials; and (2) how to improve school
management through the use of tools such as a CFS monitoring toolkit, action research, and a mobile phone-
based school data monitoring and evaluation system. This intervention attempted to establish children’s
governance groups to enhance student leadership, participation, and empowerment in school decision
making.
How effective are the CFS interventions for outcomes of interest? The UNICEF intervention is intended to boost attention and retention in CFS schools, and to improve the
application of student-centered instruction and time use in the classrooms. In addition to the regular head
teacher, student and school instruments, the CFS intervention was assessed using a classroom observation
instrument following the Stallings methodology. Similar to the WASH intervention, we are able to separate
schools that have been implementing the intervention for a longer period (prior to June 2014), compared to
schools that have been implementing for a shorter period and schools that have not implemented the
intervention. As is clear from Figure 21, the change in girls’ outcomes does not appear positive. Although
overall attendance and promotion rates are relatively high (see Annex D), the changes are only slightly
positive on attendance in schools implementing CFS for a shorter period, and there does not appear to be a
difference between schools that have been implementing the intervention for a longer period. As can be
seen in Figure 21, however, the impact on boys is mixed, with a clear positive effect for retention (for both
longer and shorter reported duration) and a negative effect with attendance rates.
Figure 21 – CFS: Changes in key outcomes
-13.3 -13.1
8.0
-20.0
0.0
20.0
% c
han
ge
Other CFSpre & post 6/14
CFSpost 6/14
Girls Attendance Rate Change, Grades 5-6
-9.2-11.2
-0.4
-20.0
0.0
20.0
% c
han
ge
Other CFSpre & post 6/14
CFSpost 6/14
Boys Attendance Rate Change, Grades 5-6
-1.8
-7.6-10.6
-20
.0
0.0
20
.0
% c
ha
ng
e
No CFS Prior Post
Girls Promotion Rate Change, Grades 5-6
-0.3
14.5
8.8
-20
.0
0.0
20
.0
% c
ha
ng
e
No CFS Prior Post
Boys Promotion Rate Change, Grades 5-6
34
Classroom Observation - Background To capture differences in student-centered instruction and time use at the classroom level, we implemented
a classroom observation tool following the Stallings methodology. The CFS intervention is designed to
promote changes in the style of teaching practice, with teachers trained in child-centered pedagogy, and
urged to move away from teaching through rote memorization. Classroom observations were conducted in
54 (17 percent) of the 321 schools reached in the endline (Figure 22). 13 of these schools received the CFS
intervention and 41 did not.
Figure 22 – Number of schools in which classroom observations were conducted15
Academic and non-academic activities The Stallings tool differentiates class time into that devoted to academic and non-academic activities. Figure
23 shows teacher engagement in different academic activities. Academic activities include copying,
debating, desk work, exposition and demonstration, practice and drilling, and reading out loud. Note that
these numbers do not add to 100 percent because not all of class time is devoted to academic activities. The
graph shows the percentage of class time (average) devoted to these different activities for CFS and non-
CFS schools. As can be seen, there are similar percentages across both types of schools in terms of the
percentage of class time devoted to each activity. Most class time is devoted to exposition and
demonstration (18-22 percent), debate (12-13 percent), followed by practice and drilling (6-8 percent), and
reading aloud (5-8 percent).
Exposition and demonstration includes lecturing and showing students how to do something new16. While
this activity is less likely to be done in a collaborative manner, debate is identified as an activity that
involves the collaborative exchange of ideas and discussion between teachers and students. While there are
not large differences between CFS and non-CFS schools in terms of class time devoted to academic
activities, both classes include a mix of these types of activities. Of particular interest, it appears that in
both CFS and non-CFS schools the proportion of class time devoted to copying is zero. This is important,
as an objective of the CFS intervention is to move teachers away from using rote memorization, and
stimulate active participation in learning among students.
15 The number of schools per county is provided in parentheses, for reference 16 The CFS training manual that was used can be made available upon request
16
12
8
6
4
3
3
2 (11)
(14)
(51)
(32)
(52)
(35)
(22)
(104)
0 10 20#
Turkana
Mandera
Garissa
Marsabit
Wajir
Samburu
Isiolo
Tana River
35
Figure 23 – Percentage of class time devoted to academic activities
Figure 24 – Percentage of class time spent on nonacademic activities17
Figure 24 shows the percentage of class time devoted to non-academic activities. Note that the total class
time between the two graphs does not add up to 100% because of a small number of missing observations
in one school. Classroom management is defined as activities that include passing out papers, cleaning the
board or putting away materials. A significant proportion of non-academic class time is devoted to
classroom management, although note that classroom management tends to be more collaborative in CFS
schools.
At the end of the classroom observation, enumerators were asked to make a series of evaluative judgments
about the frequency of CFS-related content delivered by teachers in the classroom. Table 3 shows the
percentage of class time that the teacher devoted to specific tasks. CFS schools tended to have a greater
percentage of class time devoted to activities that would be consistent with participation in a CFS training.
Teachers were more likely to ask probing and open-ended questions, to ask students to expand and explain
the meaning of concepts during class discussion, and to apply their understanding of topics to relevant life
experiences. The exception is with gender and disadvantaged students, where CFS schools were as likely
as non-CFS schools to discuss this content. Of note is that across all 54 schools, enumerators observed both
CFS and non-CFS classrooms did not discuss gender or other biases against disadvantaged students.
Table 3 - Highlights of the classroom observation Status Proportion of
Class Time Teacher…
...asks probing and open-ended questions such as how and why
...asks students to expand and explain in class discussion
...asks students to apply understanding to relevant life experiences
...talks about gender or other biases against disadvantaged students
...integrates gender-sensitive material into pedagogy
Non-CFS 1 or 2 times
29% 20% 24% - 12%
CFS 46% 38% 38% - 8%
Non-CFS About half 24% 20% 2% - 2%
17 Reading out loud does mean aloud and not loudly. Classroom management refers to time spent by the teacher
ensuring that that lessons are being delivered as planned and managing student behavior.
26
21
1513
7
11
78
2
6
01
02
03
0%
Exposition& Demonstration
Debate Copying Practice& Drilling
DeskWork
Non-CFS CFS
15
10
15
119
7
4
12
01
02
03
0%
ClassroomManagement
Alone
Teacher Outof Class
ReadingOut Loud
ClassroomManagement
Non-CFS CFS
36
CFS 8% 15% 8% - 0%
Non-CFS Entire
5% 5% 7% - 5%
CFS 15% 0% 0% - 8%
Non-CFS Did not observe
41% 56% 66% 100% 80%
CFS 31% 46% 54% 100% 85%
CFS regression analysis Following the same structure for the regression analysis, we assess the impact of the CFS intervention on
growth in student enrollment, attendance rates, and grade promotion rates. Similar to the analysis of WASH
schools, we are able to estimate the impact of the CFS intervention among schools that implemented the
intervention prior to and after June 2014, schools that only implemented the intervention after June 2014,
and schools that received the intervention from organizations other than UNICEF (some head teachers
reported receiving the CFS intervention from KEPSHA and/or the Kenyan Ministry of Education and not
through UNICEF). We further disaggregate CFS schools that implemented only in the post-June 2014
period to identify those that also implemented WASH, Solar, and C4D over the same period. Error!
Reference source not found. Figure 25 presents the marginal intervention effect across different
intervention statuses on growth in student enrollment, attendance rates, and grade promotion rates relative
to similar schools that never implemented CFS and did not receive any other intervention prior to June
2014.
Figure 25 – Margins plot of CFS outcomes Growth in Enrollment
Growth in Attendance Rates
Growth in Promotion Rates
Notes: Marginal effects associated with enrollment are expressed as changes in enrollment in response to the intervention, in counts.
Marginal effects associated with attendance and promotion rates are expressed as percentage point changes in response to the
intervention. Intervention statuses associated with fewer than 10 schools are suppressed.
Schools that implemented CFS in both the pre- and post-June 2014 periods display an increase in their
enrollment, relative to similar schools in the comparison group, by an average of 9.2 students (effect size
of 15.3 percent of a standard deviation). However, this estimate is not statistically significant. Schools that
implemented the intervention only after June 2014 exhibit a statistically significant 11.4 student increase
in student enrollment relative to similar non-CFS schools, equivalent to an effect size of 6.9 percent of a
standard deviation. Schools that implemented CFS through non-UNICEF organizations exhibited the
largest increase in student enrollment by 16.1 students, relative to comparable non-CFS schools. CFS
schools that also implemented other interventions displayed modest positive effects in some cases and
negative among schools that also implemented C4D. The CFS interaction effects are all statistically
insignificant.
37
We find mixed and insignificant results of the effect of CFS on attendance rates among all CFS schools
except those schools that implemented CFS and C4D where we see a 25 percentage point increase relative
to similar non-CFS schools. Lastly, we find small positive effects of CFS on grade promotion rates between
0.6 and 3.7 percentage points across all types of CFS schools, although these estimates are not statistically
significant.
Findings from Focus Groups One of the goals of CFS is to ensure that all children are included in the teaching and learning process and
feel that they have an opportunity to learn. For children with disabilities, the challenges of learning in
developing contexts can be even more challenging as they are often excluded from schools – or the physical
environment does not facilitate their school attendance. CFS is trying to change that. Parents, community
members and students discussed how their schools are working to integrate disabled students into the
learning environment. Several parent focus groups mentioned that UNICEF had improved WASH facilities
to facilitate their use by physically disabled students. Parents also noted that, “..there are students who
have been appointed to support the students with disabilities and they also help others appreciating them.
Like last week there was a function where this group of student volunteers was introduced to the
missionaries as care givers of the pupils with disability.
Parents also noted that, “Teachers also go out of their way to help the physically impaired children. They
even go to the extent of hiring motorbike using their money to take these students home. This has also
contributed to children coming to school.” Teachers also teach students to. “…accept them [disabled
students] and respect them the way they are. They should not laugh at them or abuse them. They now
understand them and consider them as part of the community.”
Indicative Results of C4D Interventions Communication for Development (C4D) has played an integral part in international development efforts
over the past five decades. For UNICEF, C4D is considered one of the core cross-cutting strategies essential
to advancing its program goals, particularly to ensure that the rights of children are realized. C4D aims to
build community capacity to meaningfully engage and participate in analyzing challenges and barriers and
identify priority solutions for education access and quality issues in schools. The approach to C4D
encompasses a range of interventions from advocacy to mobilization of communities to changes at both the
individual and community levels facilitated through many diverse communication channels and processes.
The intervention uses local media channels to communicate and promote understanding of the legal
framework of education governance. Community members were inducted in CFS principles and disaster
risk reduction (DRR), and engaged parents, teachers, and pupils together in school-based change.
Integrating C4D principles and approaches is a way to create an enabling environment for the empowerment
and engagement of individuals and communities.
How effective are the C4D interventions for outcomes of interest? The C4D intervention is designed to improve enrollment, attendance, and retention. In particular, by
targeting out of school children and promoting the importance of schooling through media and other forms
of communication to the wider community. The hope is that education can be seen as a more attractive
investment for families. As the C4D intervention implemented by UNICEF may be similar to community
participation interventions implemented by other organizations, we sought to compare interventions
implemented by UNICEF to similar ones implemented by other organizations. Table 4 – Changes in
enrollment, C4D status shows enrollment changes over the intervention period, and we construct
comparison groups based on whether a C4D intervention was implemented by UNICEF partner, a different
organization, or whether the school did not receive a C4D intervention. While schools without a C4D
intervention saw enrollment decreases, those implementing a C4D intervention saw enrollment increases
38
for boys. The enrollment level for girls in UNICEF C4D schools stayed almost exactly the same over the
period, while for boys it increased by 14%.
Table 4 – Changes in enrollment, C4D status C4D Status Girls Boys % change
2013 2015 2013 2015 Girls Boys
No C4D 327 286 343 322 -13% -6%
UNICEF 477 472 654 742 -1% 14%
Non-UNICEF 282 242 345 358 -14% 4%
UNICEF supported C4D schools also saw increases in attendance over the implementation period. Figure
26, on the following page, shows attendance rate changes for girls, which increased 14.1 percent points
over the period. Both non-UNICEF C4D interventions and schools without C4D interventions saw
decreases. Figure 26 illustrates a similar result where attendance rates increased by just under 10 percent
while the other two categories saw decreases of 9-10 percent for C4D interventions.
Figure 26 – Changes in attendance, C4D status
Retention does not show quite as positive a change across the intervention categories. Figure 27 shows
promotion rates for girls, which decreased over the intervention period in all three types of schools. The
promotion rate for boys actually decreased to the largest extent in UNICEF schools (-9.8%) compared to
1-4% drops in non-UNICEF C4D schools.
Figure 27 – Changes in promotion, C4D status
Figure 28 shows the percentage of schools that receive training in advocacy, community outreach and social
mobilization across different types of schools. UNICEF schools implementing C4D receive substantially
more training of this type compared to schools that have not received the C4D intervention.
-9.6
14.1
-1.2
-20
.0
0.0
20
.0
% c
ha
ng
e
No C4D UNICEF Non-UNICEF
Girls Attendance Rate Change, Grades 5-6
-9.1
9.8
-10.4
-20
.0
0.0
20
.0
% c
ha
ng
e
No C4D UNICEF Non-UNICEF
Boys Attendance Rate Change, Grades 5-6
-2.0
-11.9
-16.0
-20
.0
0.0
20
.0
% c
ha
ng
e
No C4D UNICEF Non-UNICEF
Girls Promotion Rate Change, Grades 5-6
-3.6
-9.8
1.1
-20
.0
0.0
20
.0
% c
ha
ng
e
No C4D UNICEF Non-UNICEF
Boys Promotion Rate Change, Grade 5-6
39
Figure 28 – Has received C4D training
Figure 29 shows schools that report a communication plan to attract out of school children. C4D schools
report these at a similar rate to other types of schools. Figure 29 shows the reported components of out of
school children communication plans. Posted messages at school and the community are the least (7.9%)
common forms, compared to media campaigns (13.2%), targeted conversations with out of school children
in the community (50.0%) and community meetings (68.4%).
Figure 29 – OOSC communication plan
Figure 30 shows the percentage of schools that reported engaging in advocacy and social mobilization
around school participation in the past year. As can be seen, UNICEF schools implementing C4D are the
highest at 87 percent, followed by non-UNICEF schools and those not reporting any C4D intervention.
18.9
71.2
42.6
05
01
00
(%)
Has Received Advocacy and Outreach Training
No C4D UNICEF Non-UNICEF
68.676.4 77.5
05
01
00
(%)
Has OOSC Communication Plan
No C4D UNICEF Non-UNICEF
7.9 7.913.2
68.4
50.0
05
01
00
(%)
UNICEF
OOSC Plan Components
School Community
Media Meetings
Conversations
40
Figure 30 – C4D: Engaged in advocacy or social mobilization
Figure 31 shows the percentage of C4D schools that believe C4D activities have had an impact on the
school. 94.6 percent and 83.8 percent believe that communication activities at the school have been
successful in improving stakeholder perceptions of the value of education, and have been successful in
increasing the intake of out of school children back into the school, respectively. Still, many believe that
the barriers to schooling are too great to be overcome with communication activities, with a higher
percentage (56.8 percent) in UNICEF schools compared to non-UNICEF ones.
Figure 31 – C4D: Head teachers’ assessment
C4D regression analysis The regression results for the C4D intervention are presented below in Figure 32. We differentiate between
schools that received the intervention UNICEF and from non-UNICEF partners. Similar to the analysis of
the Solar intervention, we do not observe enough schools who implemented the intervention prior to June
2014 to distinguish UNICEF C4D schools. We do, however, stratify C4D recipients among those schools
that received a combination of C4D and CFS.18 However, since there were a relatively small number of
schools with sufficient data that received the C4D intervention alone, we are unable to confidently estimate
the intervention effect for these schools. Further, the majority (22 out of 32 UNICEF C4D schools) received
C4D in conjunction with CFS, and more than half of all C4D schools received the intervention from non-
UNICEF organizations, which are the only two types of C4D schools that we are able to identify the
intervention effect for.
18 There were fewer than 10 schools implementing C4D and WASH and/or C4D and Solar, the results of which are suppressed
due to lack of statistical power.
59.6
87.0
75.0
05
01
00
(%)
Engaged in Advocacy or Social Mobilization
No C4D UNICEF Non-UNICEF
83.892.3
56.847.5
94.6 94.9
05
01
00
(%)
Attendance Barriers Perceptions
C4D: Agree or Strongly Agree
UNICEF Non-UNICEF
41
Figure 32 – Margins plots of C4D outcomes Growth in Enrollment
Growth in Attendance Rates
Growth in Promotion Rates
Notes: Marginal effects associated with enrollment are expressed as changes in enrollment in response to the intervention, in counts.
Marginal effects associated with attendance and promotion rates are expressed as percentage point changes in response to the
intervention. Intervention statuses associated with fewer than 10 schools are suppressed.
We find a positive and significant effect of C4D among non-UNICEF implementers on growth in student
enrollment, by about 16.6 students (effect size of 24.7 percent of a standard deviation), while UNICEF C4D
schools that also implemented CFS exhibit a negative but insignificant effect relative to similar comparison
schools. On the other hand, we find a statistically significant 27.6 percentage point increase in attendance
rates among C4D schools that also implemented CFS relative to comparison schools, while non-UNICEF
C4D schools do not show any statistically significant improvements. Finally, we estimate small and
statistically insignificant increases in grade promotion rates among all C4D schools relative to the
comparison group.
Findings from Focus Groups During the FGDs, parents, community members, and students were asked to discuss strategies, training,
and activities related to helping get out of school children back in school. The findings from these FGD
varied greatly from one school to the next and should be taken with caution. The main themes arising in the
community and parent focus groups related to C4D include: the role of the school committees and student
government at the school; and the extent to which UNICEF was recognized for the training and activities
taking place in the schools.
Community activities Parents and community members who participated in the FGDs clearly understood the importance of having
students attend school. However, the actions taken by communities and their understanding of best practices
for recruiting and retaining children in school varied widely across communities. Parents talked about how
the school committees travel out to remote areas and talk to parents about the importance of education.
They also mentioned that the school director and members of their BoMs had participated in meetings that
focused on strategies for recruiting out of school students. A parent in the Isiolo focus group captured the
role parents understand that they should be playing through the following quote, “… the school committee
sometimes goes round the village telling women that the government want every child to go to school. Also
as a committee member you are asked to pass this information to all parents in the village and that is what
we usually do.”
They also noted that, “we [the SMC] help by reporting all the children that have dropped to the chief. We
also follow-up with the parents to know why their children are not going to school and insist that they
should take them back to school.”
42
Parents also indicated that they have been instructed (i.e. trained by either the chief or someone else) – and
often meet with the teachers to understand how students are performing in school and help the school
improve attendance by visiting the homes of students who fail to come to school. These activities are all
positive and in line with the types of activities that the C4D training hopes to instill in communities.
It was clear from the discussions that the Chief plays a significant role in organizing and ensuring that
children go to school. Parents and community members tended to work with and through this person, which
aligns with the culture and practices of the nomadic tribes. However, it was less clear whether the techniques
used by the communities were also in line with what UNICEF and other organizations teach in terms of
recruiting new students. One disconcerting theme that arose in several focus groups was the idea that parents
are arrested and prosecuted for not sending their children to school. A parent in Isiolo noted, “We were told
that if a parent does not take her children to school it’s an offence and [they] can be arrested and
prosecuted. This has made the number of children in this school increase very much and this is due to the
fear of being arrested and also because parents have now know that there is a law that protect the children.”
The context of the discussion centered around parents discussing the rights of children, but in many cases
– and across several communities, the issue seems to have been taken to the extreme. UNICEF and other
organizations working on awareness and mobilization need to understand the extent to which these type of
fear tactics are used and focus their training to ensure communities use more positive recruiting methods
moving forward.
C4D training workshops It was clear from the focus group discussions that communities – particularly the head teacher and chief
had participated in meetings and training workshops that improved their knowledge and awareness of the
importance of education. However, it was less clear the extent to which any of the activities can be traced
back to UNICEF support verses other organizations. During the focus group discussions, parents and
community members were asked who provided the community mobilization training and whether they were
familiar with the Elimu Yetu Coalition EYC). No participants recognized EYC as having provided
training,19 nor did they attribute the training and activities to UNICEF. Rather, the communities referenced
groups such as Youth trust, TUSOME and the government as the organizations most linked to improving
their knowledge and awareness around the importance of education. Additionally, the results from the focus
groups around the C4D topic were fairly generic. Parents and communities rarely specified activities
beyond going to homes and talking to parents about getting their children in school. There was no mention
of radio messages, community advertising/marketing or additional local training workshops – even when
participants were specifically asked to elaborate on other types of activities the community engaged in
throughout the year.
Raising community awareness does take time and parents noted that, “… we have just started [the process]
last year - you remember we are nomads and it is very hard for us to take the messages at first, so it’s better
if we are given time to catch up because there’s a lot of illiteracy in our community so it not the way you
expect the results very fast, they still need time.” So while the results of this evaluation appear to be less
promising for C4D in this first year, the types of activities and the length of time and intensity of training
required for strong uptake likely affect our ability to see any results. The FGDs seem to indicate that training
was not consistent, short in duration and that there was no on-going support to the communities for
conducting different types of activities in support of education. As with some of the previous interventions,
UNICEF needs to look carefully at the training approach used to improve community awareness and
perhaps strengthen both the delivery in communities and on-going support for implementation. The issue
of duration and dosage are further elaborated in the discussion section.
19 The primary target of C4D training was the head teacher.
43
COST-BENEFIT ANALYSIS Cost-benefit, cost effectiveness and other economic analyses are often used to guide policymakers and
others interested in understanding the types of interventions that have the greatest impact on specified
outcomes (Flay et al., 2005; Spoth, Greenberg, & Turrisi, 2008). Cost-benefit analysis allows
policymakers to determine if their investments are sounds; and provides a basis for comparing the results
to other alternatives to determine which of the investments provides the greatest returns. In this study we
present the cost-benefit analysis of three UNICEF interventions: WASH, Solar power, and CFS – each of
which [together and individually] seek to help retain students in school. We examine whether the
interventions, when implemented in low cost boarding schools in the ASAL region of Kenya, are a good
investment of donor dollars.
Interventions such as WASH, solar power and CFS attempt to address issues of enrollment, retention and
completion in schools. As this report has shown, the UNICEF interventions have had different levels of
success when compared to schools that had similar interventions by other organizations – or no
interventions. While this evaluation looks at results after only one year, over time, it is expected that
UNICEF will see greater improvements in enrollment and retention. The following analysis will show the
length of time that it will take UNICEF to attain returns on their investment. It is important to note that the
cost-benefit model (CBA) that is presented in this section serves as a means of modeling the potential long-
term benefits that may accrue to society based on the potential for better future employment. Caveats and
limitations of these findings are discussed at the end of the section.
Assumptions To build this investment ‘simulation’ model, we made the following simplifying assumptions.
1. Intervention costs are provided to us by each implementing organization. The benefit-cost analysis
is conducted separately for WASH, Solar power, and child friendly schools to determine the
monetary returns to each intervention.
2. We do not include the costs of running the schools since these are provided by the government and
essentially equivalent across the ASAL schools.
3. To determine the number of students and schools affected by each intervention, we rely on our
enrollment regression results for the WASH, Solar power, and CFS interventions. The analysis
examined 31 WASH only schools, 23 schools that received solar power, and 48 schools that
implemented CFS. This analysis does not include C4D schools due to the lack of positive results
observed in those schools, i.e. we found no measurable benefits to enrollment as a result of C4D.
Table 5 – Estimated enrollment benefits for WASH, Solar, and CFS Cell size Marginal effect Standard error
WASH 31 +22.99 (10.36)
Solar 23 +6.99 (16.32)
Child friendly schools 48 +10.66
66
(7.97)
Note: Cell size refers to the number of treated schools included in the regression analysis to identify
the marginal effects.
4. Benefits to society are calculated based on the net present value of increased salaries and tax
revenue paid to society over time in 2015 US Dollars (USD) using a standard CBA approach (Levin
and McEwan, 2001). Specifically, the benefits are the net present value of those taxes paid due to
increased earnings as a result of increased educational attainment over and above the taxes they
would have paid from not benefitting from the interventions. We group students into the following
44
categories: No education (i.e. dropped out prior to grade 5), attained grades 6 through 8, and
completed grade 8.20
5. The mean wages used to calculate the tax gain to society are drawn from the World Bank STEP
surveys. We use these wage data to construct the age-earnings profiles of individuals with different
levels of educational attainment. We assume that students begin to earn wages at age 18 (3 years
after the intervention year)21. The age-earnings profile data enable us to compute total lifetime
earnings for different types of individuals in Kenya. As Figure 33 below shows, the returns to
education to those students completing grades 9-12 is approximately 69 percent. For those students
who complete grades 9-12 their annual earnings start at nearly USD 2,007 at age 18 and continue
to increase over the next 36 years. For students who attain grades 6-8, earnings typically start at
USD 978, reach a peak of USD 1,634 when the student is in their mid to late 30s and then declines
over time. For students with no education, the line remains relatively flat over their life time.
Figure 33 – Age-earnings profile by educational attainment – Kenya, 201322
Source: The STEP Skills Measurement Program 2013, the World Bank.
6. Enrollment data for students in intervention schools are not currently available for each individual
grade level. We measure the benefits for the current cohort of 5th and 6th grade students over their
labor market life-cycle. Further, we use the following grade level estimates to calculate the number
of students that move on to the higher earning brackets.
Table 6 – Grade promotion, dropout, and repetition rates – Kenya, 2014 Promotion rate Dropout rate Repetition rate
Grade 6 89.9% 5.6% 4.5% Grade 7 78.7% 7.8% 13.6% Grade 8 75.2% 2.0% 23.1% Grade 9 100.7% 0.6% -1.3%
20 Refer to the Annex for a detailed description of the net present value calculations. 21 We acknowledge that students who drop out of the education system are likely to work in the informal sector, or as herdsmen,
helping their parents, but the income is minimal [or in lieu of parental income] and therefore not included in the analysis.
22 The Kenyan Shilling to US Dollar exchange rate used is 102.17 KES to the dollar.
45
Grade 10 99.8% 1.2% -1.0% Grade 11 90.1% 2.3% 7.5% Grade 12 -- 3.2% -- Source: Calculations made by FHI 360 Education Policy and Data Center, using data from the
Demographic and Health Surveys in Kenya, 2009.
Based on the mean enrollment in the WASH and Solar schools and the promotion rates in Table 7, we
estimate the following number of students to have benefitted from the interventions directly. We show that
713 students attain at least grade 6 as a result of WASH, 161 students in Solar, and 512 students in CFS
who would not have otherwise.
Table 7 – Increased enrollment and grade attainment, by intervention WASH Solar CFS
Number of schools 31 23 48
Mean enrollment 114.9 116.9 124.1
Total enrollment 3,562 2,689 5,957
Total students affected +713 +161 +512
Students progressed through grade 8 +334 +75 +240
Students completed grade 8 +379 +86 +272
Results of the benefit-cost analysis Below, we plot the cumulative lifetime benefits associated with the WASH and Solar interventions under
different discount rate assumptions. We incorporate different discount rates in increments of two percentage
points to provide a range of possible net present values of the cumulative lifetime benefits of the
interventions. The benefits are presented in the form of the future tax payments made as a result of the
interventions over and above the tax payments from a scenario where the intervention never took place.
Table 8 displays the total and per pupil cost of each intervention in US Dollars.
Table 8 – Costs per school, per student, and total, by intervention
WASH Solar CFS
Total intervention cost $1,646,486 $315,582 $231,424
Total cost for schools in the analysis $505,357 $111,667 $112,206
Cost per school $16,302 $4,855 $2,338
Cost per student $142 $42 $19
We plot the cumulative benefits of WASH, Solar, and CFS separately, over the 36 years post-
implementation in Figure 34. The time horizon for the cost-benefits simulation represents the time needed
for the current student beneficiaries to be of employment age as well as the time covered by their labor
market lifecycle, i.e. from ages 18 through 50 years.
46
Figure 34 – Cost-Benefit simulation results – NPV of cumulative taxes paid over lifecycle WASH
Solar
Child Friendly Schools
Note: The horizontal red line represents the total intervention costs for the schools present in the analysis, i.e. 31 WASH schools,
23 Solar schools, and 48 child friendly schools.
Based on this simulation, it is clear that the cost-benefit simulation with the lowest discount rates yields the
highest benefits, and vice versa. However, we can see that over an entire lifecycle the WASH intervention
may generate between USD 602,000 and USD 1,944,000 in benefits, the solar intervention may yield
between USD 136,000 and USD 440,000 in benefits, and the CFS intervention is expected to generate
between USD 432,000 and USD 1,395,000 in benefits.
Using the results shown in Figure 34Error! Reference source not found., we compute the number of years
required for each investment to be returned in full, i.e. the number of years required for the benefit-cost
ratio to reach 1. For WASH, the initial investments will be completely returned by between 11 years and
21 years, depending on the discount rate used. The Solar investments will be returned in full after 11 to 20
years following implementation of the intervention. Finally, the simulations show that the CFS investments
yield the highest returns, relative to their initial investments. Depending on the discount rate, CFS will
return all of its initial investment within 5 years of implementation.
Over an individual’s labor market lifecycle, the WASH intervention yields a benefit to cost ratio of 1.19 to
3.85 per US Dollar invested, the lowest benefit to cost ratio denotes the highest discount rate of 10 percent
while the highest ratio denotes the lowest discount rate of 2 percent used in the simulation model. The
investments in Solar power produce a benefits to cost ratio ranging from 1.27 to 4.17 per US Dollar invested
over a lifetime. Lastly, investments made in the CFS intervention yield the highest benefit to cost ratios
over a lifetime between 4.0 and 12.5 per US Dollar invested.
Table 9 – Lifetime benefit-to-cost ratios per intervention, by discount rate23 Discount rate r = 2% r = 4% r = 6% r= 8% r = 10%
WASH 3.85 2.70 2.00 1.52 1.19
Solar 4.17 2.86 2.13 1.61 1.27
Child friendly schools 12.50 9.09 6.67 5.26 4.00 Note: the benefit-to-cost ratios are calculated as the ratio of cumulative benefits after 36 years
over the initial cost of the intervention for the schools included in the simulation model.
Limitations of the Benefit-cost model The benefit-cost analysis has four main limitations that restrict inferences about economic benefits. First,
the analysis assumes that the infrastructure remains intact and that schools have the required resources to
repair and refurbish the WASH and solar power infrastructure as needed over time. If schools do not invest
23 For the full list of benefit-cost ratios over time and for each intervention under different discount rate scenarios, please refer to
the Annex.
47
in the upkeep of the facilities, it is likely that they will depreciate and would likely be an associated decline
in enrollment and retention.
A second limitation is that these estimates are based on projected versus actual benefits as well as drawn
from national surveys rather than earnings data specific to the ASAL region and the counties included in
this study. We estimate lifetime beginning at age 18 and assuming that the students work continuously over
the next 36 years. Although this is a standard approach, actual earnings from communities in these counties
may provide a more accurate accounting of economic status and increased tax revenues.
The third study limitation is that there are likely to be unmeasured economic benefits. For example, fringe
benefits were not estimated as part of lifetime earnings (i.e. health insurance). Benefits for parents, other
family members, or the community more broadly were not estimated.
Finally, the estimations assume a fixed number of students that move through the system with different
levels of education. The estimates do not account for future cohorts of students or additional positive
changes in enrollment and retention that would impact the overall benefits and reduce the time it takes for
UNICEF to gain back its initial investments. This final assumption implies that the results of our simulation
model are conservative and may serve as a lower bound given our model assumptions.
Concluding remarks for cost-benefit analyses
The cost-benefit analyses show that, over the long-run and regardless of the discount rate used, investing
in child friendly schools yields the highest returns in the form of additional taxes paid over a lifetime. Every
shilling invested in CFS is expected to yield a return between 4.0 and 12.5 after 36 years. Further, CFS
investments are returned in full faster than investments in WASH or Solar power. Our simulation model
predicts that the initial investments made in CFS will be returned in full after 4-5 years since implementing
the intervention. As for WASH and Solar power interventions, the lifetime return yields are between 1.19
and 4.17 for each shilling invested. It is important to note that initial investments are returned in full between
10 and 21 years post-implementation.
Although it appears that the initial investments in WASH, Solar power, and CFS are returned after a
substantial period of time, it is important to note that this simulation model includes the benefits from only
one cohort of 5th and 6th graders that we were able to observe. Ideally, with more data, we would observe
multiple cohorts affected by the same initial investment before the need to reinvest to update and maintain
current physical capital (WASH facilities, solar panels, etc.) as well as human capital (professional
development and training for new teachers). As such, this simulation model provides UNICEF with an
informed notion of the relative cost-efficiency of each investment under current data constraints.
DISCUSSION This section will draw on the results to discuss the operational hypothesis that higher duration and dosage
of interventions (or specific combinations of interventions) have a greater impact on outcomes (i.e.
enrollment, attendance, study time, outreach). The section begins with an elaboration of our dosage,
duration and enabling environment framework. It is followed by a (1) discussion of whether the duration
and dosage of training was sufficient to begin to affect changes in behavior among communities, parents
and teachers; (2) the extent to which the enabling environment contributed to differences or similarities in
findings; and (3) whether and how different combinations of interventions impacted the outcome variables.
Understanding Dosage, Duration and Enabling Environment When scaling up education reforms, implementing an intervention in thousands, even millions of
classrooms where teachers may lack the proper skills is a different kind of challenge (Thompson and
William, 2007). The challenges of successfully implementing education reforms and seeing an impact on
48
educational outcomes include the sheer number of classrooms, the complex system in which these
classrooms often function (i.e. different environments even within the same country), the separateness of
the classrooms, the private nature of teaching, and the type of pedagogical support teachers receive on a
regular basis (Thompson and Wiliam, 2007). For an intervention to show impact, it means that every
teacher, every head teacher, and every parent and community member has to understand and commit to the
intervention, on their own. Projects can have great interventions and theories of action – but the enabling
environment to move an action plan forward can reduce the impact of interventions and slow progress. The
projects can be well designed, but if the teachers cannot deliver the interventions with a high level of quality
with the appropriate intensity, then the implementation effort is wasted (Thompson and Wiliam, 2007).
This is the challenge we face in closing the learning gap.
In 2014, the Basic Education Coalition Working Group for Monitoring and Evaluation wanted to
understand whether different combinations of interventions had more or less of an impact on improved
reading outcomes. Using data from a multitude of USAID-funded programs, the working group devised a
framework for analyzing the effect sizes of these interventions with the purpose of being able to inform
expectations of what it really takes to improve reading outcomes, particularly at scale. This framework,
comprised of three components, is used to frame the discussion around the findings of this evaluation.
Duration Duration is defined as the time period of implementation, or how long the intervention is applied. Similar
to the concept of dosage (below), duration refers to the exposure of an intervention over time – or, the
length of time that interventions are provided to participants. Through our analysis, we were able to separate
out schools that had longer periods of the interventions (pre June 2014) and compare the outcomes at those
schools with the results from schools that had only post 2014 interventions. The results for the WASH and
solar schools are discussed below. The results for CFS and C4D are discussed in the section on dosage.
Schools that received the WASH intervention in both the pre- and post-June, 2014 periods experienced an
average increase in student enrollment of 23 relative to similar schools that had never received WASH and
did not receive any interventions prior to June, 2014. We also saw an increase of 15 students, on average,
among non-UNICEF WASH schools relative to similar schools in the comparison group. This finding tells
us that WASH does have some relationship to increases in student enrollment. Interestingly, for the schools
that have had WASH implemented only recently, there was almost no effect on enrollment, suggesting that
duration is an important factor in affecting enrollment and attendance outcomes.
As a comparison, the introduction of a latrine/Wash interventions in India increased schools’ eighth-grade
student enrollment by only 1.5 students, with a standard error of 0.755 (Aduki, 2014). A review of eleven
articles by Joshi and Amandi (2013) also showed that the child’s gender, age, and access to water, sanitation
and hygiene training over an extended period of time were positively associated with education outcomes.
The positive impact of WASH interventions on school enrollment in 42 primary schools in Kenya after a
year of interventions is also supported by Patel et al. (2012).
In terms of the solar power interventions, it is important to note that we had only three schools that
implemented solar power both pre and post 2014, so the results cannot be extrapolated to other schools.
However, in the three schools that had the longest duration of the intervention, the schools exhibited an
increase of about 40 students compared to schools that had not received solar power.24 We also note that
schools that implemented both Solar and C4D exhibited significant growth in enrollment by 22 students.
24 In the final regression analyses of Solar power interventions, we consolidate schools that received Solar power from UNICEF
regardless of duration to maintain statistical power.
49
Dosage Dosage is the frequency and time that is spent implementing technical interventions. Implementation
dosage refers to the intensity with which interventions are carried out with fidelity. This intensity includes
the amount of training that participants receive in preparation to deliver interventions; the amount of time
that coaches spend working with teachers on an intervention; or the amount of time teachers receive training
on the use of new materials (OPRE, 2013), Understanding what dosage of an intervention is needed is
important because it affects the outcomes as well as the costs, staffing, replication and scale-up efforts that
follow.
The existing research on dosage is clear that one dose of an intervention is usually not enough (Boller et
al., 2004; Winton, 2008; Joyce & Showers, 1980). However, more is not always better (OPRE, 2013). As
shown by Boller et al. (2004); Joyce & Showers (2002); Raikes, et al. (2006); and Winton & McCollum
(2008). One-day workshops for teachers do not provide the necessary dosage to affect teacher learning or
change and improve their delivery of classroom practices in the long term. Professional development
interventions specifically, need to be delivered more intensely, usually with longer duration or frequency
to make a difference (Halle, Zaslow, Tout, Starr, Wessel, & McSwiggan, 2010). However, there is little
research that sheds light on the optimal amount.
In Kenya, the duration of training workshops and its effect on CFS uptake seems to be linked, but not
necessarily in the way we would have expected. As previously discussed in this report, regression results
show that schools that implemented CFS in both the pre- and post-June 2014 periods (i.e. a longer duration)
display an increase in their enrollment by an average of nine students. Schools with lower duration of
implementation (i.e. post June 2014 only), exhibit a statistically significant 11 student increase in student
enrollment relative to similar non-CFS schools. When we look at the dosage of training, the teachers in the
UNICEF schools received a higher dosage of general professional development training (nine days)
compared to the non-UNICEF schools (seven days) and the schools with no CFS interventions (eight days).
Similarly, when we looked at the reported number of days of training on CFS specific content, teachers at
the UNICEF-supported schools reported receiving eight days of training compared to five days in the non-
CFS schools and seven days in schools that received CFS training from another organization. The standard
errors in both cases ranged between 0.4 and 0.6 – similar to what the literature has found in other training
cases.
So does the dosage and duration of training matter? In the case of CFS in Kenya, it is difficult to say after
only one year. It is clear, however, that teachers received more than one day of training, and that seems to
have had a positive effect on enrollment and attendance. The schools that had CFS training also had higher
levels of enrollment and attendance when compared to schools that had no CFS training. Additionally,
FGD, student surveys, classroom observations and head teacher surveys further suggest that the schools
where CFS has been implemented (whether by UNICEF or someone else) are more welcoming and there
is a positive perception about the environment (learning, security, and inclusion) for all children. These
items all suggest that the dosage and duration of the interventions is perhaps sufficient, though we would
have expected the schools that had received two doses of training (pre and post) to have had the highest
enrollment and attendance results. This result likely indicates that other factors such as the enabling
environment, teacher migration and attendance or other factors may also be influencing the results within
the schools that have received CFS training.
In terms of the C4D interventions, anecdotal information obtained through FGDs and surveys suggest that
there is a lot of inconsistency in how the C4D interventions were implemented. While the C4D schools that
received the intervention from UNICEF in conjunction with implementing CFS had higher attendance rates
than similar non-C4D schools by approximately 27 percentage points. We further found that the growth in
enrollment over the course of the year was only statistically significant in schools that received the
intervention from non-UNICEF organizations (i.e. the length of training data is not available to FHI360).
50
What we learned through the FGDs lead us to believe that the dosage and duration of C4D was not sufficient
to see impacts on student outcomes. The training, done through a cascade model, helped communities
understand the importance of education, but it is less clear that the communities learned the different skills
needed to properly mobilize and help retain students in school (e.g. use of fear tactics, no radio or public
messaging mentioned in FGD). It is likely that interventions such as C4D need more time and support in
the implementation phase to ensure that communities are able to implement the interventions according to
the design (fidelity to implementation) and use different types of activities depending on the contexts to
influence community perceptions on education.
Enabling environment An enabling environment is defined as a set of interrelated conditions (i.e. political, institutional, technical,
and cultural) that impact the capacity of actors (i.e. teachers, parents, communities, donors, and
governments) to engage in development processes in a sustained and effective manner. The challenge often
faced in implementing education reform and taking the reform to scale is that the existing situation in a
country is not an accident. There are often well-entrenched interest groups who want to ensure that
institutional arrangements do not change since they often directly benefit from the arrangement (Crouch
and DeStefano, 2006; Moe 2003; Hess, 2004). Often proposed school level reforms or pilot programs are
seen as only affecting a small part of the system, so these groups do not feel threatened. However, when
these changes (albeit policy or scale up of interventions) begin to change the resource allocations, interest
groups who may not benefit from the change respond – and often forcefully (Crouch and DeStefano, 2006).
In the Kenya ASAL region there are several “environmental” issues that are also at play. While we are not
able to statistically measure those at this time, the issues are worth mentioning because they likely affected
the outcomes of this evaluation in different ways.
Security and Politics In several of the ASAL counties (i.e. Garissa, Mandera, and Wajir) security issues have plagued both the
implementation of UNICEF interventions as well as data collection for this impact evaluation. Certain
armed groups are known in Kenya for attacks on various public areas, including schools and universities.
The activities and threats from these groups delayed implementation of several UNICEF interventions
(reducing the duration of the interventions) and prevented the data collection teams from reaching
approximately 34 schools in Mandera, Garissa and Wajir.
In the best of circumstances, education reform efforts need stable environments and support from the
political (government), institutional (teachers, head teachers, inspectors) and technical (donors and
technical assistance) agents to succeed. Through discussions with parents and community members as well
as meetings with the implementing organizations and data collectors, we know that security issues and fear
of attack have affected the ability of students to attend school. Fear of attacks have increased parents
wariness to send their children to school; the lack of fencing – or even long distances students and teachers
may have to walk have affected their desire and ability to attend the school; and the danger associated with
the unrest made it difficult for data collectors to reach the schools to even see if the interventions had made
a difference. These security issues greatly impact and work against creating an enabling environment for
learning and improvement – and thus, must be taken into consideration when understanding the results of
this evaluation.
The Drought In many cases, understanding the enabling environment means that we look at different political,
institutional and technical aspects of interventions and try to understand who the winners and losers are –
and how each group may have affected the results. But, sometimes, natural disasters also affect the
environment and can impact the results we see in schools. Throughout the focus group interviews, parents
and community members mentioned the challenges the region has faced as a result of the drought. While
51
the communities recognize that organizations such as UNICEF have worked hard to improve or establish
water sources at the schools, many communities noted that bore holes have dried up, or that the tanks at the
schools did not have sufficient water (i.e. community use vs. student use) as a result of the drought. While
the evidence is anecdotal, the lack of water may have also reduced the impact of the interventions during
this year. Families have lost their herds and need children to work in the informal sector; there is no water
at the school so students stay home; and in schools where there is piped water, communities often share the
water source possibly reducing the availability of water for students.
Community Commitment In closing, it is important to note that many of these counties and schools do have a positive enabling
environment when it comes to education. Throughout the FGDs we saw an incredible commitment of the
parents and community members to their school. They recognized the importance of education. They were
trying to use strategies they had been taught to bring more children into the schools; and the chiefs were
committed to ensuring all children received education. This enabling environment creates an opportunity
for UNICEF to continue to work with and through these communities to implement hygiene, CFS and C4D
training. There is a base of institutional capacity (i.e. people power) that has been built and in many counties
a great deal of political will (i.e. support from the chiefs, elders, parents) to improve education. More
training and direct support over a longer period of time is needed to ensure that communities implement
CFS and C4D properly, but the foundation has been laid and UNICEF can build on that enabling
environment.
52
REFERENCES Adukia, A. (2014). Sanitation and Education. Harvard University
Ashish Joshi and Chioma Amadi (2013). Impact of Water, Sanitation, and Hygiene Interventions on
Improving Health Outcomes among School Children. Center for Global Health and Development, College
of Public Health, UNMC, Omaha, USA
Birdsall, N., Levine, R. & Ibrahim, A. (2005). Towards Universal Primary Education: Investments,
Incentives, and Institutions. Sterling, Va. Stylus Publishing.
Chapman, D.W. and Quijada, J. J., (2008). An analysis of U.S assistance to basic education in the
developing world, 1990-2005, International Journal of Educational Development: 29(3), 268-280.
Crouch, L. & DeStefano, J. (2015). A Practical Approach to In-Country Systems Research. This paper was
prepared for the Research on Improving Systems of Education (RISE) Program Launch Conference,
Washington, DC, June 18–19, 2015.
Education for All Global Monitoring Report (2014). Teaching and Learning: Achieving Quality for All.
UNESCO Publishing.
Flay BR, Biglan A, Boruch RF, Castro FG, Gottfredson D, Kellam S. Standards of evidence: Criteria for
efficacy, effectiveness and dissemination. Prevention Science (2005); 6:151–175.
Hanushek and Woessman (2012). Do better schools lead to more growth? Cognitive skills, economic
outcomes, and causation. Journal for Economic Growth. 17:267–321
Levin, H. and Lockheed, M. (1991). Effective Schools in Developing Countries. International Bank for
Reconstruction and Development, The World Bank.
Lockheed, M. & Verspoor, A. (1992). Improving Primary Education in Developing Countries. Oxford
University Press.
Ministry of Education, Science and Technology. (2014). Basic Education Statistical Booklet. Retrieved
from http://www.mygov.go.ke/.
Patel, M., Julie R. Harris, Patricia Juliao, Benjamin Nygren, Vincent Were, Steve Kola, Ibrahim
Sadumah, Sitnah Hamidah Faith, Ronald Otieno, Alfredo Obure, Robert M. Hoekstra, and Robert Quick.
(2012). Impact of a Hygiene Curriculum and the Installation of Simple Handwashing and Drinking Water
Stations in Rural Kenyan Primary Schools on Student Health and Hygiene Practices. American Journal of
Tropical Medicine and Hygiene. 87(4): 594–601.
Spoth R, Greenberg M, Turrisi R. (1980) Preventive interventions addressing underage drinking: state of
the evidence and steps toward public health impact. Pediatrics 121(Suppl 4): S311–336.
Stallings, J. (1980). Allocated Academic Learning Time Revisited, or Beyond Time on Task.”
Educational Researcher: 9 (11): 11–16
Thompson, M., Wiliam, D. (2007). Tight but Loose: A Conceptual Framework for Scaling Up School
Reforms. Paper presented at the annual meeting of the American Educational Research Association (AERA)
April 9, 2007 - April 13, 2007; Chicago, IL.
53
UNESCO. (1990). Meeting basic learning needs: A vision for the 1990s. Background Document: World
Conference on Education for All. Retrieved from http://www.unesco.org/education/pdf/11_92.pdf
UNICEF. (2014). Annual Report 2014 – Kenya.
http://www.unicef.org/about/annualreport/files/Kenya_Annual_Report_2014.pdf
United Nations (2015). Sustainable Development Goals.
http://www.un.org/sustainabledevelopment/sustainable-development-goals/
Wasik, B. A., Mattera, S. K., Lloyd, C. M., & Boller, K. (2013). Intervention dosage in early childhood
care and education: It’s complicated (OPRE Research Brief OPRE 2013-15). Washington, DC: Office of
Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health
and Human Services. doi: 10.4269/ajtmh.2012.11-0494 PMCID: PMC3516306
54
ANNEXES
Annex A: Supplemental Methodology Information
Annex table 1 – Number of schools by reported treatment status Treatment Status N
C4D Not Reported 271
C4D UNICEF 38
C4D Other Org 40
CFS Not Reported 201
CFS UNICEF 117
CFS Other Org 31
Solar Not Reported 151
Solar UNICEF 78
Solar Other Org 120
WASH Not Reported 176
WASH UNICEF 112
WASH Other Org 61
55
Instruments and Key Data Sources
Outcome indicator
Definition Source Sample size Timing
Duration, dosage,
enabling environment
Amount and intensity of
training interventions;
presence of other (third
party) contributing
interventions
Head teacher
questionnaire
All schools Endline
Attendance (boys and
girls)
Percentage of students
from two consecutive
grades present on the day
of data collection,
disaggregated by gender
Primary data collection:
school attendance
registers at Day &
Boarding Schools
All schools Baseline
and
endline
Retention Number of students
promoted to the next
grade as a proportion of
students enrolled in the
previous grade the
previous year
Primary data collection –
data on pupils, repeaters
and new entrants by year
by grade
All participating
schools with data
availability
Endline
only
Solar power effect on
studying habits
Percentage of students
reporting studying on
school premises after dark
Student survey
Student focus groups
All participating
schools
Endline
only
Ongoing activities
supporting enrollment
drives
Percentage of schools
where enrollment
promotion activities took
place
Head Teacher survey
Teacher focus group
Student focus group
Endline
only
Community
perceptions of the
value of interventions
and their impact
Parent and community
perceptions on the value
of interventions and
persistent barriers to
school participation
Primary qualitative data
collection: focus group
discussions and key
informant interviews in
communities around
selected schools;
All participating
schools
Baseline
and
endline
Student perceptions Student reflections on the
value of interventions and
persistent barriers
Focus group discussion
for std. 6 pupils in
selected schools
Student rapid survey
All participating
schools
Endline
Effective use of
student-centered
instruction in class
Percentage of time that is
used for the application of
student-centered methods
promoted by CFS training
Primary data collection
(classroom observation)
At least 1
classroom per
school
Endline
Cost-effectiveness
Cost per student of each
intervention, in constant
2014 currency, relative
the change observed in
attributable lower-level
outcomes
Implementing org. will
provide their school-
level budget information
for each intervention.
Will also request per
student costs per district.
All
implementing
organizations, by
county
Endline
only
56
Hypotheses to be analyzed in the observational study:
The research hypotheses for this evaluation are based on the UNICEF and DFID Theory of Change which
states:
Interventions that improve water and sanitation systems will increase school attendance, particularly among
girls. As a result, the intervention will increase their overall exposure to instruction leading to higher
academic performance. The lower-level outcome for this intervention will be measured by changes in both
enrollment and attendance, particularly among girls.
Installation of solar-powered electricity will expand the effective time available for instruction, including
homework, leading to higher academic performance. The lower-level outcome measure is evidence of
students using school facilities for study during evening hours (after dark).
Community involvement interventions will raise the public awareness of the value of education and
enhance the positive perception of schooling for boys and girls. The lower-level outcome for this
hypothesis is evidence of the community involvement activities taking place and communities reporting
changes in attitudes and behaviors with respect to school enrollment. The outcome measure related to this
intervention are attendance and retention in the affected schools.
Teacher training for CFS will increase the effectiveness of instruction, ensure child friendly and inclusive
classrooms; safe schools; schools that promote health, nutrition and equitable education; and enhanced
community linkages and partnerships. The hope is that by bringing together these elements, it will
contribute to increasing the pupils’ academic performance. While the ultimate outcome – academic
performance – will not be measured for this observational study, a lower-level outcome will involve the
extent to which the student-centered instruction is applied at the classroom level. .
The null hypotheses to be tested related to the causal chain include the following:
i. H0: No effect of receiving WASH interventions on attendance and retention
ii. H0: No effect of receiving Solar Power on student studying patterns
iii. H0: No effect of receiving CFS training on application of student-centered instruction and time
use.
iv. H0: No effect of receiving CFS teacher training on attendance and retention
v. H0: No effect of receiving training on community engagement on attendance, retention and
sustainability of enrollment promotion
vi. H0: No effect of the combined interventions on attendance, retention and learning
environments.
Each of these null hypotheses is plausible for this evaluation. First, the duration of the interventions may
not be sufficiently long or intensive for the effects in enrollment and retention to manifest themselves.
Where expected effects are particularly modest, measurement error may sometimes crowd out the effect –
which is especially plausible when outcomes are self-reported as is the case with observational tools and
self-reported practices. Further, enrollment, attendance, and retention rates are vulnerable to demographic
shifts (for example, due to security concerns, climate or other external factors) that may affect different
counties differently.
Finally, and crucially, most of the schools already had some interventions taking place at the time of the
baseline, including the control schools. Nearly all schools reported having more than one intervention in
the two years prior to the baseline. Due to the high level of contamination and the lack of a clear start date
57
for the interventions, the estimates of the effect between baseline and endline in this study may
underestimate the true causal effect of interventions.
For these reasons, the observational study approach, with the focus on lower-level outcomes and the
implementation of interventions using the duration-dosage-enabling environment, is the appropriate design
for the endline study. In addition, an in-depth look at the “how” and the “why” of the interventions and
their effects on lower-level outcomes will be made possible through focus groups. Finally, the focus groups
will also provide a gauge of the remaining barriers for school participation and learning, and allow for better
understanding of the appropriateness of interventions to the real needs and barriers facing communities in
the ASAL regions.
Variables and analyses Qualitative analysis In terms of qualitative research, we conducted focus groups with students and teachers at each school;
community members; school and classroom observations; as well as interviews with key informants from
the implementing organizations. These various qualitative techniques allowed us to understand the
challenges and contributions of the interventions from various perspectives and document how the
interventions were implemented. The classroom observations – newly added for the endline – allowed us
to determine the extent to which training content on CFS and C4D was implemented at the school level.
We explain both the instruments and methodology for collecting and analyzing the data under each
instrument.
1. Focus Group Protocols: The endline data collection team used slightly revised and refined focus
group protocols to gather data on the effect of the interventions on the experience of students,
teachers and the communities. One focus group with each stakeholder group will be conducted
during the school visit.
2. Classroom Observation Protocol: The Stallings classroom observation instrument was used to
measure time on task and document whether the techniques taught to teachers during CFS and C4D
training workshops is translating to the classroom.
We conducted 25 focus groups across schools and communities. Classroom observations were conducted
in 54 classrooms across 54 schools, primarily those with CFS interventions. The focus groups were recorded
using the tablets, transcribed and upload to NVIVO for coding and analysis. Once analyzed, we estimated
the level of agreement using Cohen’s Kappa. We found that the coefficient was high, and therefore, we did
not feel that collecting additional data would be useful
Changes to the data collection and analysis process from baseline to endline Based on a review of the processes, procedures, and outcomes of the baseline, FHI360 made the following
changes to the data collection process to ensure quality control and meaningful outcomes for the overall
study.
1. Reviewed the instruments from the baseline and based on conversations with UNICEF Kenya,
made revisions in the protocols to ensure consistency, minimize missing data, and provide more
in-depth data and responses related to the interventions.
2. Refocused existing instruments and analysis towards the intended lower-level outcomes of the
interventions.
58
3. Changed the methodology to an observational study with the understanding that effects from the
interventions may not always be solely attributable to interventions, but must serve as initial gauges
of likely future outcomes.
4. Provided extended training to enumerators on the use of the newly revised instruments.
5. Worked with the field team to provide additional training on data cleaning so improve the quality
of the data in the system.
6. Strengthened the management of data collection on the ground.
7. Attempted to gather missing data from the baseline to better account for the enabling environment
and possible effects of third party interventions.
8. Added a classroom observation component in a sub-sample of schools to gather data on whether
and how teachers demonstrate skills received during CFS and C4D training.
9. Added a short student survey to collect data on student study habits.
10. Used tablets to collect data. The use of tablets increased the accuracy with which data was collected;
allowed FHI360 staff to spot check data daily and identify any problems with data quality; and
conduct the analysis more quickly.
59
Annex B: Computation of Net Present Value in Cost-Benefit Analysis The net benefit from WASH, Solar, and CFS is calculated as the (net) present value of taxes paid over the
life-cycle for students progressing to at least grade 6 over and above the taxes they would have paid had
they halted their educational progression in grade 5, less the initial cost (investment) of the intervention.25
Formally, we compute the net present value of the benefits accrued to each individual benefitting from the
WASH or Solar intervention as follows:
𝑁𝑃𝑉𝑡𝑏 =
∑ (𝑌𝑡𝑏 − 𝑌𝑡
0)𝑇𝑡=1
(1 + 𝑟)𝑡− 𝐶0
𝑁𝑃𝑉𝑡𝑏 is the net present value of the cumulative taxes paid by individuals with education attainment b less
the taxes paid had they not benefitted from the program, at time T. r is the discount rate for the value of
money in the future and 𝐶0 is the initial cost/investment of the WASH, Solar, or CFS intervention. For the
simulation model, we present the cost-benefit analyses under a range of possible discount rates. Finally, the
total net benefit of the interventions is the NPV associated with educational attainment b multiplied by the
number of students with education attainment level b.
Annex table 2 – Benefits-to-cost ratios for WASH, Solar, and CFS
r = 2% r = 4% r = 6% r= 8% r = 10%
Years post-implementation WASH
5 Years 0.39 0.37 0.35 0.33 0.31 10 Years 0.90 0.81 0.72 0.65 0.59 15 Years 1.49 1.27 1.09 0.93 0.81 20 Years 2.13 1.69 1.39 1.16 0.97 25 Years 2.70 2.08 1.64 1.32 1.08 30 Years 3.23 2.44 1.82 1.43 1.15 36 Years 3.85 2.70 2.00 1.52 1.19
Solar
5 Years 0.42 0.40 0.37 0.35 0.33 10 Years 0.96 0.86 0.77 0.69 0.63 15 Years 1.59 1.35 1.15 1.00 0.87 20 Years 2.27 1.82 1.49 1.23 1.04 25 Years 2.86 2.22 1.75 1.41 1.15 30 Years 3.45 2.56 1.96 1.52 1.22 36 Years 4.17 2.86 2.13 1.61 1.27
Child Friendly Schools
5 Years 1.33 1.25 1.19 1.12 1.06 10 Years 3.03 2.70 2.44 2.22 2.00 15 Years 5.00 4.35 3.70 3.13 2.78 20 Years 7.14 5.88 4.76 4.00 3.33 25 Years 9.09 7.14 5.56 4.55 3.70 30 Years 11.11 8.33 6.25 4.76 3.85 36 Years 12.50 9.09 6.67 5.26 4.00
25 We cap our benefits at attaining the final year of secondary school in order to produce a conservative estimate that could serve
as a lower bound for the benefits associated with each intervention.
60
Annex C: Additional Focus Group Results In this section, we highlight additional findings from the focus group discussions with parents, students,
and community stakeholders. These FGD findings highlight challenges still facing parents and students that
donors and UNICEF can provide support for in the schools.
School fees are an issue among the schools participating in the study as costs of uniforms and other materials
for school are expensive. Some parents indicate that the community may provide financial support to help
a family buy uniforms to get their children in school, however fees are still brought forward as a non-
negligible issue. It was mentioned across many of the FGD by both parents and students. Several parents
mentioned that providing scholarships, especially for orphaned children, may prove helpful in encouraging
enrollment.
“If the parents of that child cannot afford to buy for this child school uniform or books they will assist him
by contributing for him money so that he can buy for him.”
“one day when Iwas walking around the town came across a girl who had never enrolled to school and
when I asked the parents why they had not taken the child to school they said that they did not have money
to buy for her uniform and books. So, I went to the shop and bought for her uniform, shoes and books and
gave them to the parents and asked her to take the child immediately to the school and now the girl is in
school.”
“So most of the people are now in the town and if we come across any children not in school we find out
what is wrong and if it is because they do not have uniform or books we go to the UN office and ask them
for assistance, which they often provide and the books are provided by the government. We have done this
several time and almost all the children are in school although these uniforms are now torn and the
organization is no longer helping because there is peace in this region. The UN helps the region when there
is insecurity but when security improves they move to another region.”
“When our students do not have uniform and we are not able to provide for them we go knocking to all
NGOs doors in this region asking them to help our kids”.
School feeding programs and the need to ensure children have food at school was the topic most mentioned
in the focus groups when parents were asked what more could be done for the schools. They indicate that
the drought has severely impacted the households both in terms of access to water and food. Parents also
mention fairly frequently that when children get meals at the school, they are more likely to attend and stay.
Below, are some quotes from the FGD, where provision of meals at school was mentioned in every focus
group (all 23).
“children report to school at 6am and they don’t take breakfast so if schools would provide milk at break
time it would be very good. Also as we said, the school is not provided with enough food so the children
are served with very little food and they are not satisfied, so the government or others should add enough
food so that the children can comfortably stay in school.”
“I think the diet should be improved, apart from the one which is given by the school feeding program or
school meals program, at least even supplements should be provided because if you look at the diet which
is given by the school meals, there are no vitamins only beans, maize and oil, vegetables oil, but some fruits
should be provided for the students.”
61
“Children in boarding school should be given good food and those going back home should also be given
food – they cannot study when they are hungry. Water could also be of great help to the community and
build wash facility for the children to use after going to the toilet.”
Parents in Garissa and Wajir especially mentioned the need for security fences around the perimeter of the
schools. Here is what some of the participants said related to this issue:
“The school is not properly fenced and because of insecurity when the teacher sees the school is not
completely fenced they feel not secured and so they don’t come to school. The fence should go up so that
even the wild animals cannot pass through.”
“On the issue of security, they [the organizations] should also build fence around the school because of
security issues. Since the school is not fenced they [the students] are not safe.”