monitoring, evaluation and learning (mel) frameworkthis document presents brac‟s monitoring,...
TRANSCRIPT
1
Monitoring, Evaluation and Learning (MEL) Framework
Community Based Education for Marginalised Girls in Afghanistan, BRAC Afghanistan
Girls Education Challenge – Transition Window (GEC-T): 5085
2
Table of Contents 1. Introduction ............................................................................................................................................. 4 2. Learning from GEC 1 ............................................................................................................................. 8 3. Monitoring ............................................................................................................................................ 11 4. Key evaluation questions ...................................................................................................................... 17 5. Evaluation approach ............................................................................................................................. 18
5.1 Research design .............................................................................................................................. 18 5.2 Measuring outcomes ....................................................................................................................... 19
5.2.1 Sustainability ............................................................................................................................. 32 5.3 Ethical protocols ............................................................................................................................. 35
5.3.1 Ethics ......................................................................................................................................... 36 5.3.2 Child protection ......................................................................................................................... 41
6. Sampling framework ............................................................................................................................ 45 6.1 Target groups .................................................................................................................................. 45 6.2 Control groups / Counterfactual scenario .................................................................................... 47 6.3 Cohort tracking .............................................................................................................................. 49
6.3.1 Learning cohort ......................................................................................................................... 49 6.3.2 Transition cohort ....................................................................................................................... 51 6.3.3 Replacement strategy ................................................................................................................ 52
6.4 Power calculations and sample sizes ............................................................................................. 53 6.5 Benchmarking ................................................................................................................................. 57
7. Baseline study ........................................................................................................................................ 59 8. Evaluation governance ......................................................................................................................... 60
8.1 Evaluation steering group ............................................................................................................. 60 8.2 External evaluator .......................................................................................................................... 60 8.3 Data validation ................................................................................................................................ 61
9. Data quality assurance ......................................................................................................................... 61 9.1 Training ........................................................................................................................................... 61 9.2 Piloting ............................................................................................................................................. 63 9.3 Data cleaning and editing............................................................................................................... 63
10. Risks and risk management ............................................................................................................... 64 11. Learning ............................................................................................................................................... 67
11.1 Learning strategy .......................................................................................................................... 67 11.2 Stakeholder engagement, dissemination and influencing ......................................................... 69
12. Evaluation work plan ......................................................................................................................... 70 12.1 Timetable ....................................................................................................................................... 70 12.2 Responsibilities ............................................................................................................................. 71
Annex A: Logframe .................................................................................................................................. 72 Annex B: Draft evaluation tools .............................................................................................................. 72 Annex C: Draft sampling framework ..................................................................................................... 72 Annex D: ToR for evaluators ................................................................................................................... 73
3
4
1. Introduction
This document presents BRAC‟s Monitoring, Evaluation and Learning (MEL) framework for the
Community Based Education for Marginalised Girls in Afghanistan project, an eight year project funded
by UK Aid under the Girls Education Challenge – Transition (GEC-T) window.
The overall aim of BRAC‟s project is to improve the life chances for marginalized girls in Afghanistan,
with the following three objectives being central to achieving this aim:
1. Supporting marginalized girls to learn the skills they need to be empowered, valued and
productive members of their communities,
2. Supporting marginalized girls to transition effectively from BRAC community-based education
(CBE) primary schools into government schools, CBE secondary schools or TVET training,
3. Ensuring the sustainability of marginalized girls‟ educational outcomes.
BRAC‟s project has been designed based on a number of key contextual factors. Afghanistan is one of the
poorest and most conflict-affected countries in the world, with four decades of political and civil conflict
and war leading to poor development outcomes. One of these outcomes is a large deprivation in
educational opportunities for children, particularly for girls. There is a range of supply and demand
factors that continue to pose challenges to girls‟ access to education in Afghanistan. For instance, supply
factors include: lack of accessible government schools, particularly for those girls living in remote areas
who must travel long distances to reach the nearest school; lack of female teachers; poor quality of
teaching; inadequate learning resources, school facilities and infrastructure; and a lack of safe
environments for girls in schools, including prevalence of corporal punishment by teachers. Demand
factors include: poverty, which is linked to requirements for child labour (particularly domestic labour for
girls); violence and insecurity; conservative cultural norms that restrict women‟s and girls‟ mobility,
linked to concerns about honour and chastity; negative perceptions of girls‟ education and participation in
other activities such as employment; and early marriage and subsequent pregnancy of girls.
Supply and demand barriers to girls‟ education in Afghanistan intersect in complex ways, and have
shifting impacts on girlsas they get older and transition through different stages of schooling, particularly
from primary to secondary school. As girls are preparing to transition into secondary school, they enter
adolescence and puberty, leading to family concerns about girls‟ chastity and honour (exacerbated by lack
of female teachers), more focus on marrying girls (linked to poverty and the need to secure bride price or
offset household expenditure), and increasing requirements for girls‟ domestic labour (including caring
for younger siblings and older or sick family members). A shortage of secondary schools, particularly for
girls, also exacerbates concerns about girls‟ safety when traveling long distances, with distances to
5
secondary schools from remote areas often greater than for primary schools.
In light of the barriers to girls‟ education outlined above, BRAC‟s GEC-T project includes a range of
activities that sit within four broad packages.
1. Provision of primary and secondary education and TVET vocational training for girls. BRAC
will support girls‟ primary, secondary and vocational education through three types of schools: CBE
primary schools (established in GEC 1) that will continue into secondary school grades; government hub
schools (primary and secondary) to which BRAC CBE students have transitioned since GEC 1; and
newly established TVET centres aimed at increasing girls‟ vocational skills. This provision of education
will include the development of gender sensitive curricula, extra-curricular and life skills activities,
distribution of stipends to highly marginalized girls (including poor, disabled and orphaned girls)
attending government hub schools, provision of chaperones (khalas) for girls attending government hub
schools, transport assistance to girls attending TVET centres, and a girls‟ mentoring and co-curricular
program implemented in government hub schools.
2. Teacher training. BRAC will deliver a package of teacher training at all three school levels, including
subject-based, gender awareness and child protection/safeguarding training. This will include refresher
and grade changing teacher training.
3.Community engagement and stakeholder advocacy. BRAC will engage and raise the awareness of a
range of stakeholders at the community level, including parents and other community members, in
relation to the importance of girls‟ educational and employment outcomes. This will include the
implementation of advocacy meetings with community members and school management shuras (SMSs),
community training orientations (e.g. on sanitation, corporal punishment and gender awareness) and
mother‟s forums.
4. School management strengthening. BRAC will strengthen school management through a range of
activities, including: conducting advocacy workshops with school and government stakeholders;
implementing training on gender awareness, corporal punishment and school management for SMSs,
school staff, community development councils (CDCs) and other relevant actors; supporting the linkage
between school child safeguarding standards and codes of conduct, and community and local authority
child protection mechanisms and systems; piloting sustainable CBE models; and sharing learning
products with school, government actors and other stakeholders.
BRAC‟s Theory of Change (see Figure 1) stipulates that the activities outlined above will contribute to
five intermediate outcomes: attendance, teaching quality, girls‟ life skills, improved attitudes and
perceptions, and improved school governance. By ensuring that all girls can access government schools,
CBE schools or TVET centres with qualified teachers that provide supportive and learning environments
and co-curricular activities, BRAC can provide high quality education for girls. By raising awareness of
6
and promoting girls‟ and women‟s rights, BRAC will improve attitudes and perceptions of girls‟ and
women‟s participation in education, employment and income earning activities, and, by providing girls
with important life skills, BRAC will ensure that girls are prepared and empowered to engage in these
activities. Supportive schools, communities and effective school management in accessible locations will
ensure attendance. Working with the community and government to improve school governance and
ensure adherence to international child safeguarding standards will boost attendance, retention and
completion and also work towards creating safe, efficient and sustainable school infrastructure and
knowledge in communities. Overall, an holistic, supportive and quality education, accessible to all, will
lead to good learning outcomes and ensure transition,while effective community engagement, the
strengthening of school governance and strengthened links between government bodies, government
schools and CBE schools will improve sustainability of infrastructure and project results. The Theory of
Change is based on several key assumptions: community, government and other relevant stakeholders
continue to support girls‟ education programming and participate in project activities;no external factors
such as conflict, natural disasters or political unrest impact negatively on project operations; community
members are receptive to gender awareness and engagement; and school stakeholders at all levels
collaborate to strengthen child protection and safeguarding.
BRAC will work in 10 provinces: Baghlan, Balkh, Hirat, Jawzjan, Kabul, Kapisa, Kunduz, Nangarhar,
Parwan and Samangan. The main beneficiaries of the project are girls attending three different kinds of
educational institutions: CBE schools at the community level (girls continuing their CBE from GEC-1),
government hub schools (girls who have transitioned from CBE in GEC-1 to government schools), and
TVET centres (girls from CBE communities in GEC-1 who dropped out of BRAC classes or did not
transition from CBE to hub schools, and will attend vocational training at the district level).The project
will work with other beneficiaries and stakeholders at the local, regional and national levels. At the local
level BRAC will work with parents, community members, community leaders (community shura leaders
and hub school shura leaders), Community Development Councils (CDCs), teachers, student organizers
(khalas) and education volunteers through school committees, mother‟s forums and parent meetings.
BRAC will also work closely with Child Protection Action Network (CPAN) groups and local
government officials to ensure cooperation and collaboration in relation to child protection. At the
regional levels, the project will engage with provincial and district level MoE officials, vocational skills
associations (e.g. ICT and midwifery) and partner NGOs for improving the overall performance of
schools and TVET centres. At the national level, BRAC will engage national bodies such as the MoE,
Ministry of Women‟s Affairs (MoWA), Ministry of Labor, Social Affairs, Martyrs and Disabled
(MoLSAMD), the Directorate of TVET, the Afghan Midwifery Association and national and international
development and research partners.
7
Figure 1: BRAC Theory of Change: GEC-T
8
The primary aim of this document is to provide a framework and guiding structure for the monitoring,
evaluation and learning of BRAC‟s GEC-T project. It should be used by: BRAC programming staff as a
cross-referencing tool to ensure that MEL objectives and activities are aligned with programming across
the life of the project; BRAC‟s monitoring and research staff, who will be responsible for conducting high
quality monitoring, overseeing the quality and ethical implementation of external evaluations, and
conducting issue-based research projects linked to GEC-T; and external evaluators, who will find the
document useful in guiding and designing evaluation activities, methods, processes and outcomes based
on BRAC‟s expectations. Across all these stakeholders, the MEL framework should be used as a learning
tool that can be updated accordingly over the life of the project based on how monitoring and evaluation
results are fed back into programming.
2. Learning from GEC 1
The BRAC GEC-1 endline evaluation identified a number of key challenges in the MEL approach that
will be addressed and mitigated in GEC-T. These are outlined below.
Monitoring
Several GEC-1 output indicators were not easily measurable in terms of number or percentages, with
strong reliance on descriptive qualitative data in evaluation methods and unclear pre-planning on how
outputs should be quantified. In GEC-T, the logframe will be designed to measure output indicators with
both quantitative and qualitative data (where relevant) with clearly labeled data sources.
The endline evaluation suggested that there might be some problems ensuring that stipends were
distributed to the most marginalized government school girls. In GEC-T, stipend recipients will be highly
marginalized CBE school girls who have transitioned to government hub schools. Given some evidence
to suggest that transition of girls from CBE to government hub schools may be more common in higher
income households, BRAC will ensure that stipend recipients are genuinely marginalized, with low
household income being an important measure of marginalisation in this context. Other measures of
marginalization include disabled and orphaned girls. BRAC will continue to appoint a stipend selection
committee as in GEC-1; however, committee members will additionally comprise community
representatives from BRAC CBE communities where girls reside, including parents (e.g. from the
Mothers Forum) and school and community shura members.
Evaluation
Tracking and recontacting longitudinal cohorts of girls was a challenge in all GEC-1 evaluation phases,
particularly for out-of-school girls. There were a number of reasons for these challenges, primarily, the
lack of a comprehensive tracking system with inconsistent ID numbers used at different evaluation waves,
and limited tracking data recorded for girls, their family members, alternative community contacts and
9
household addresses/locations. The lack of a comprehensive tracking system was complicated by the
sampling design, which drew from school-based sampling for CBE girls and subsequent household
sampling for girls‟ household members, and household-based sampling for out-of-school girls.
Furthermore, the tracking system was not effectively calibrated to track in-school girls who had dropped
out between evaluation waves. In GEC-T, these challenges will be mitigated by ensuring that effective
tracking, recontacting and sampling systems and approaches are designed (see section six of the MEL
framework). Furthermore, unlike in GEC-1, the GEC-T evaluation tracking system will be linked to
BRAC‟s beneficiary tracking database for all three groups of intervention girls (CBE, government and
TVET).
In GEC-1, BRAC‟s independent evaluator only conducted household surveys with girls‟ mothers or
female primary caregivers, and not with male household members (e.g. fathers, or older brothers or uncles
in single-mother households). This meant that KAP survey questions (i.e. on attitudes and perceptions
towards girls‟ and women‟s education, employment and participation in shuras) appended to the
household survey, were only answered by women, limiting the ability to quantify male household
members‟ changes in attitudes over time. In GEC-T, any KAP survey questions included in the household
survey will be asked of both male and female caregivers of girls.
The endline evaluation found some limitations in the methods for collecting attendance data. These
included the inability to conductunannounced attendance spotchecks, some gaps in attendance data
collection tools (i.e. no checklist to record actual same-day attendance recorded by teachers on days when
spotchecks were conducted), and oversampling attendance data in government schools (potentially
leading to a reduction in quality of attendance and spotcheck data where enumerators are requested to
record attendance for ALL classes in a school, but with insufficient budget or time allocated to ensuring
this can be realistically done). Challenges conducting unannounced spotchecks are difficult to mitigate
given the security context and BRAC‟s requirements to facilitate enumerators‟ contact with schools and
communities. Although unannounced spotchecks are not possible, BRAC field staff will support a semi-
announced approach, whereby school staff and community members will be advised of a one week
window in which data collectors will visit schools, with the actual day not to be announced. Gaps in the
attendance data collection tool will be corrected such that checklist options are included for the teachers
record of attendance on the day of the spotcheck and enumerators‟ spotcheck head counts. Finally, unlike
in GEC-1, in which government hub school attendance was recorded for all classes from grades four to
nine, in GEC-T attendance will only be recorded for the classes within BRAC‟s cohort grades (i.e. grades
five and six at baseline, and subsequent cohort grades at midline and endline evaluation waves).
The endline evaluation found that there were a number of data entry errors that were not corrected at
baseline or midline. These included EGRA EGMA time being included as a fraction rather than an integer
reflecting whole seconds left on the clock. There were also data entry errors at baseline and midline in
relation to incorrectly recording Persian or English numerals that may be different from one another but
10
look similar or the same. This impacted mostly on recording of age, which made it difficult to match girls
across longitudinal cohorts. BRAC will brief the external evaluator on these errors early in the evaluation
process to ensure that: tools are properly designed to avoid data problems; enumerators are trained in
avoiding data errors; and data entry staff are aware of possible sources of error and ways to avoid them. If
possible, and depending on the available resources of the external evaluator, BRAC will encourage digital
data collection to sideline data entry based on paper surveys.
A number of problems with EGRA and EGMA tools provided by the fund manager were found at
baseline, midline and endline. These included: formatting problems when using Dari and Pashto
(particularly the latter) in Microsoft Word; incorrect reading orientation of subtask tables; translation
errors; omission of mathematics equations; and structural problems related to the calibration of tool use in
Afghanistan (particularly in relation to ensuring that Dari and Pashto tools were equally calibrated, with
Pashto learners disadvantaged in a number of subtasks). Given that the external evaluator is expected to
design EGRA and EGMA tools for GEC-T, with the new addition of SeGRA and SeGMA tools for
secondary school grades, a number of procedures must be put in place when designing such tools to avoid
the errors found in GEC-1. The external evaluator should ideally be familiar with literacy and numeracy
tool design and proper calibration according to language, particularly in Afghanistan. Such an evaluator
may be difficult to find, in which case BRAC will seek additional support from the fund manager in the
design of tools. All literacy and numeracy tools will also be piloted prior to baseline data collection.
In GEC-1, only half of BRAC‟s target provinces were sampled for qualitative data collection, limiting the
ability of the external evaluator to draw from qualitative findings to contextualize and expand learning
based on the results of the quantitative data, particularly for learning, attendance and teaching quality
data. In GEC-T, qualitative data will be collected in all 10 provinces receiving BRAC interventions.
In GEC-1, qualitative interviews and focus groups with girls drew from the same methods and approaches
used with adults (a question and answer approach with follow up, and facilitated discussion in focus
groups). In GEC-T, the external evaluator will be encouraged to design and adopt more child friendly and
interactive qualitative methods, which may include playing games, drawing/painting, or role playing etc.
For example, rather than simply asking girls about teacher quality and classroom practices, girls could be
asked to do a role play where they enact a typical day in the classroom. BRAC also encourages the use of
methods such as auto-ethnography and life history mapping to examine qualitative data longitudinally.
Learning
In GEC-1, BRAC made few programming modifications as a result of evaluation findings, suggesting
that there was a lack of learning systems in place to ensure that evaluation and monitoring findings fed
back meaningfully into project design.Challenges are always shifting and changing in an unpredictable
context such as Afghanistan, and these challenges should be closely monitored and relevant changes to
11
MEL design and implementation made accordingly. In GEC-T, BRAC will ensure that learning feedback
mechanisms are in place, as outlined in section 11 of the MEL framework.
3. Monitoring
BRAC‟s monitoring department will monitor key project outputs and activities across the life of the
project. A Real Time Monitoring (RTM) system will be implemented in order to deliver timely feedback
to programming staff, and will be administrated via a web application and mobile devices. The project
will use three technology-based methods and tools for monitoring outputs and activities: (1) monthly
progress tracking ICT-based MIS tools; (2) monthly program quality monitoring using event-based
checklists and assessment forms; and (3) quarterly output monitoring.
The independent monitoring unit will appoint 10 Monitoring and MIS (MMIS) officers in 10 provinces to
ensure timely, appropriate and accurate data collection and broader monitoring tasks. The independent
monitoring team will randomly visit different intervention locations on a weekly basis to monitor the
quality of program activities. Following weekly visits, the MMIS officer will meet with the project team
members and develop an action plan based on the findings. A quarterly report will be created centrally
using the accumulated quality monitoring data and shared in the quarterly progress review meetings.
Outputs will be tracked on a quarterly basis following the logframe and both quantitative and qualitative
data will be collected to assess progress against targets, summarized in target versus achievement reports
as per BRAC‟s monitoring policy. A variety of tools and methods will be used to track progress against
each output (see Table 1). Analysis will be done using software such as Excel, SPSS and STATA, with
qualitative data to be analysed manually through thematic coding. All data will be disaggregated
accordingly, including by province, gender and other demographic factors. Quarterly reports will be
generated and shared with management, headquarters and donors, and will include a summary of findings
and recommendations.
Project monitoring will also involve a number of stakeholders external to BRAC. As per government
policy, the MoE will conduct joint monitoring of the project through coordinated and routine visits to
beneficiary schools by PED and DED monitoring officers in collaboration with BRAC. Joint monitoring
will also involve community members, including school shura members, parents and other local
stakeholders. The monitoring department will maintain close links with community stakeholders
throughout the project in order to track and quickly respond to any local challenges in project
implementation.
Monitoring data will be primarily used to inform project management. The head of the monitoring
department will be responsible for checking the validity of all reports compiled by monitoring staff and
giving feedback to the project implementation team to strengthen project performance and achieve
12
milestones and targets. Feedback will take place through regular monitoring results meetings with the
GEC-T project manager.
Monitoring results will also be used to inform the project evaluation. Although the independent evaluator
will not report against outputs, BRAC will provide summaries of monitoring data to enable the
independent evaluator to explore how effectively activities are feeding into outputs, intermediate
outcomes, and primary outcomes.
Table 1 contains a summary of each output indicator included in the project logframe, with corresponding
information on how outputs will be measured, relevant data collection tools and methods, rationale behind
BRAC‟s approach and the frequency of data collection for each output indicator.
Table 1: Outputs for measurement
Output Level at
which
measurement
will take place
Tool and mode of data
collection
Rationale Frequency of
data collection
Output 1.1: # learners
received the mentoring
and lifeskills training
(disaggregated by
mentor and mentee)
School
Monitor site visits, mentor and
mentee profiles,
activity/training observation,
life skills training enrolment
and attendance register, mentee
and mentor satisfaction and
program evaluation surveys,
interviews with mentors and
mentees, life skills and
knowledge survey
Two groups of beneficiary are
defined in the monitoring
approach, mentors and mentees.
Mentors will mentor large
groups of girls (mentees) who
comprise a cascaded knowledge
group. Both groups can be
monitored using the same
general methods (site visits and
observations of training and co-
curricular activities, satisfaction
surveys), however an additional
survey with knowledge- and life
skills-related questions should
take place with mentees to
measure how much knowledge
has effectively cascaded from
mentors to mentees.
Quarterly
Output 1.2: Number of
learners received co-
curricular activities
(disaggregated by
categories i.e. story
writing, wall magazine
etc.)
School
Co-curricular activity
enrolment and attendance
register, learner satisfaction and
program evaluation surveys
This output will focus only on
the measurement of enrolment
and participation in co-curricular
activities, and perceived quality
of these activities. Life skills
developed as a result of the
activity will be measured at the
IO level.
Quarterly
Output 1.3: Number of
extremely
marginalized girls who
received incentives to
stay in school
(stipends etc)
School,
community
Minutes of stipend selection
committee meetings, stipend
recipient lists, interviews with
girls receiving stipends and
parents
Endline concerns about whether
all girls receiving stipends were
genuinely marginalized mean
that community members and
parents will be more involved in
the stipend selection committee
meetings. Minutes of these
Annually
13
meetings are a key way of
ensuring that stipend selection is
transparent, and can be
complemented with qualitative
interviews with girls and parents
Output 2.1: # of Govt.
school teachers, CBGS
teachers and MTs
received training on
basic teaching and
subject base learning
and active pedagogy
School
Teacher and MT training
enrolment and attendance lists,
teacher and MT training
satisfaction and evaluation
forms, qualitative interviews
with teachers and MTs
Teacher and MT training should
take place annually and thus
should be monitored annually.
Although several tools and
approaches can be useful to
measure this output, teacher
observations are not necessary as
they are being conducted at the
IO level for teacher quality.
Annually
Output 2.2: Number of
CBGS teachers and
TVET instructors
received refresher and
grade changing
training
School
Teacher training enrolment and
attendance lists, teacher
training satisfaction and
evaluation forms, qualitative
interviews with teachers
Teacher refresher and grade
changing training should take
place annually and thus should
be monitored annually.
Annually
Output 2.3: Number of
POs, MTs and SO
trained and involve
them in their role
School
Training enrolment and
attendance lists, training
satisfaction and evaluation
forms, staff performance
reviews, qualitative interviews
with POs, MTs and SOs, and
the stakeholders they support
(i.e. teachers and girls)
The output indicator must
measure two levels: (1) whether
stakeholders are being trained
effectively and (2) whether they
are fulfilling their roles. Thus,
monitoring will require more
general monitoring tools used
during training (attendance lists,
training satisfaction forms) but
these should be complemented
with external evidence,
including annual staff
performance appraisals, and
qualitative interviews with the
stakeholders that staff will
support (e.g. girls supported by
khalas, and teachers supported
by master trainers)
Annually
Output 3.1:# of
teachers, staff and
education stakeholders
received training on
SRGBV and using the
knowledge to make
school environments
safe
School
Training enrolment and
attendance lists, training
satisfaction and evaluation
forms, qualitative interviews
with teachers and school staff,
gender, CP and GBV KAP
survey
Although monitoring will
require more general monitoring
tools used during training
(attendance lists, satisfaction
forms), it should also include the
measurement of change in
knowledge, attitudes and
practices at the school level.
BRAC will design a mini-KAP
survey to this end, with
questions related to CP,
harassment of girls (at school
and on the way to school) and
school practices to mitigate
violence and harassment.
Annually
Output 3.2:# of
mothers of GEC
learners are
School
Shura/council memberships and
meeting participation lists,
minutes of shura meetings,
This output will be measured
through tracking women‟s
membership in shuras and their
Quarterly
14
participating in
community
representative
structures (i.e. school
Shuras, local
education council,
district education
council etc.) and
supporting to enhance
girls education
attendance in meetings, and
measuring change over time. It
is known that women‟s
membership and participation
can often be tokenistic and
symbolic in nature, rather than
meaningful and engaged
participation leading to decision-
making. This aspect of
meaningful participation will
measured at the IO level (school
governance) – see further below.
Output 3.3: Men and
boys involved in "X "
project event, i.e.
community events,
school management,
reporting plan and
local level advocacy
School,
community
Project event and activity
registration and attendance lists
(disaggregated by gender, age
and type of event/activity),
KAP survey with men and boys
It is important to monitor how
many events have participation
from boys and men; however, it
is also important to understand
which events and activities they
are participating in, and if this
participation is leading to change
in knowledge, attitudes and
practices (e.g. if boys and men
are participating in school
management but not in
discussion groups about girls
harassment, this would not be
likely to change the status quo).
BRAC will conduct the same
KAP survey included in the
household survey with a
proportion of men and boys
participating in events and
activities and will track change.
Quarterly
Output 4.1:# of
mothers/parents
forums organized and
raising cumulative
voice to claiming the
rights of girls
education
Community
Mothers forum attendance
records and minutes (if
available), monitor observation
of mother‟s forum, case study
interviews with mothers
Mothers may not be literate and
may not be able to take meeting
minutes, however, if there are
literate/numerate or partially
literate/numerate members, they
will be asked to record at the
very minimum the number of
forum meetings per month and
the number of participants per
forum. This will be crosschecked
by monitor site visits to observe
forum meetings. Mothers will
also be interviewed to gauge any
examples of decision making or
change resulting from the forum
and these will be captured in
case studies.
Monthly forums
Quarterly
monitoring
visits
Output 4.2: # of
community
sensitization events
(i.e. meetings,
community dialogues,
open forums etc)
organized on gender,
Community
Community sensitization event
attendance records and minutes
(if available), monitor
observation of community
sensitization events, interviews
with community stakeholders
Given that community events
related to children‟s and girls‟
rightsoften have school staff
present (e.g. teachers,
principals), there should be
literate members who can take
minutes of meetings. Minutes,
Quarterly
15
children‟s rights&
violence, girls
education and
empowerment of girls
monitor observations of
meetings and interviews with
stakeholders will all contribute
to tracking stakeholder support
for girls‟ education and
empowerment and any obstacles
to this support that need to be
addressed in the quarterly
monitoring cycle.
Output 4.3: # of joint
monitoring conducted
involving government,
other agencies and
donors
Community
Government monitoring
reports, minutes of joint
monitoring meetings
GEC-T will see the continuation
of joint BRAC and government
monitoring of BRAC activities,
but with a strengthened
monitoring presence from the
FM country team. Joint
monitoring reports and meeting
minutes will provide an
objective measure of any
problems arising in joint
monitoring.
Quarterly
Output 5.1: # of
schools identify and
establish effective
reporting mechanisms
for mitigating GBV in
schools or on the way
to school
School
Hard copy referral reports,
interviews with teachers and
SMSs, CPAN meeting minutes,
CPAN referral closure reports
A key issue to track is whether
cases are being successfully
resolved after reporting and
referral. This will be done
primarily through CPAN‟s
closure reports (indicating
successful or unsuccessful
closure of a report)
Quarterly
Output 5.2: # of issues
identified in CP audit
and take measure to
address the issue in
time and appropriate
standard
School
Safeguarding audit reports
reviewed by BRAC
safeguarding committee, list of
recipients of reports (e.g. email
lists), minutes of CPAN
meetings
Although the production of
audits is important,
dissemination needs to be
tracked to ensure that audits are
accountable and useful to
stakeholders. It is important to
understand whether audit reports
are leading to building of
commitment in relation to CP
and safeguarding. This can be
measured through CPAN
minutes of meetings, by tracking
reference to audit reports and
actions.
Annually
Output 5.3: # of staffs,
teachers and others
affiliated to GEC-T
project endorsed and
following the child
protection principals
and standards
School
Standard CP assessment toolkit,
class observations, staff
performance reviews,
interviews with students
Monitoring should not focus on
staff and teachers‟ own
statements of project
endorsement, but rather any
evidence of change in behaviour
or knowledge. These
measurements require the use of
tools such as CP assessment
toolkits and interviews with
students to capture behaviour
and knowledge change.
Annually
Output 6.1: # School
Shuras trained and
made functional to
School,
community
Training enrolment and
attendance lists, training
satisfaction and evaluation
Standard tools and approaches,
including training enrolment
lists, participant satisfaction
Annually (SMS
capacities
checklists to be
16
effectively school
management and
governance
forms, qualitative interviews
with SMS members and
government officials, school
shura capacities checklist
surveys etc, can tell us about
how many SMSs or other
stakeholders are trained and
what they think of the training.
However, the same SMS
capacities checklist designed for
the evaluation will be
implemented through BRAC
monitoring to ensure timely
response to obstacles to capacity
building.
administered
every two
years, not in an
evaluation year)
Output 6.2: # Number
of advocacy events
conducted at district,
province and national
level to reduce the
girls education
challenges
Community,
government
Enrolment and attendance lists
of advocacy events, monitor
observation of advocacy events,
interviews with event
participants, advocacy event
reports
Although the output should be
measured quantitatively through
enrolment and attendance lists
for events, events should also be
observed by monitors who may
detect continuing barriers to
addressing girls‟ education
challenges, and who may need to
follow up with formal or
informal interviews with
advocacy stakeholders
Quarterly
Output 6.3: # of
communities that have
successfully mobilised
resources to support
schools
Government
Financial and time records for
in-kind and monetary
contributions, interviews and
focus groups with school
shuras, school staff, teachers
and PED/DED officials
It is expected that self-financing
will not occur suddenly and will
likely be preceded by smaller
indications of in-kind or
monetary contributions.
Tracking these contributions will
be an important part of
establishing the scale of self-
financing of CBE(see output
7.1). However, it is also
important to track successful
resource mobilization at hub
school levels. Qualitative data
from school shuras and school
staff will enable a deeper
analysis of enablers of and
challenges to successful
mobilization, with special
attention to examples of success
and what factors contributed to
this success.
Quarterly
Output 7.1: # Self
financing CBGS
demonstrated and
lessons documented
Community
Financial and time records for
in-kind and monetary
contributions, interviews and
focus groups with community
members (e.g. parents), SMSs
and other community
stakeholders
Evidence of successful
mobilization of resources is
captured under output 6.3.
Output 7.1 is more related to the
community ability to take over
CBE classes. This output will
draw from financial and time
record for contributions, but will
largely be measured qualitatively
through interviews and focus
groups with community
members. These interviews will
track key issues annually,
Annually
17
including enablers and barriers
to self-financing, entry points to
mobilizing self-finance, and the
efficacy of community and SMS
mobilization in generating
support for community financing
models.
Output 7.2: # of ICT
based learning hubs
established for
providing distance
support to learners
Community,
ICT hub
Log of ICT learning
correspondence, interviews
with ICT support teachers,
interviews with girls receiving
ICT support
By virtue of its virtual platform,
number of correspondence
communication can be tracked
digitally. However, the quality
of the correspondence and
support should additionally be
measured through interviews
with ICT support teachers and
girls.
Quarterly
Output 7.3: # of
studies conducted,
lessons and best
practices documented
and disseminated
BRAC office
ToRs for research studies, list
of recipients of disseminated
reports and best practices (e.g.
email lists)
Key to measuring this output is
tracking dissemination, for
instance through email lists of
recipients. This will further
allow BRAC to follow up with
recipients to see if any research
uptake has occurred.
Annually
4. Key evaluation questions
In line with guidance from the GEC Fund Manager (FM), BRAC‟s MEL framework is based on a set of
key evaluation questions at the program level, and a set of sub-questions at the project level. Program-
level questions, rooted in the key GEC primary outcomes (transition, learning and sustainability), are:
1. Was the GEC successfully designed and implemented? Was the GEC good Value for Money?
2. What impact did the GEC funding have on the transition of marginalized girls through education
stages and their learning?
3. What works to facilitate transition of marginalized girls through education stages and increase
their learning?
4. How sustainable were the activities funded by the GEC and was the program successful in
leveraging additional interest and investment?
Project evaluation questions, based on the intermediate outcomes (attendance, teaching quality, life skills,
attitudes and perceptions, and school governance) and broader evidence of efficacy, are:
5. To what extent did the intervention contribute to a change in girls‟ attendance?
6. What impact does BRAC teacher training have on improving teachers‟ pedagogical skills,
gender- sensitive teaching practices, and safe and protective learning environments for girls?
7. What is the impact of mentoring and co-curricular activities on developing girls‟ life skills, and
18
how do these skills interact with primary outcomes (learning and transition)?
8. What impact has the project had on community members‟ attitudes towards and perceptions of
girls‟ safety, and girls‟ and women‟s education, employment and participation in civic activities?
9. How effective are school governance activities and capacities in creating safer environments and
more sustainable education for girls?
10. Was the project‟s theory of change supported? What necessary adjustments (if any) should be
made to the theory of change to ensure more effective programming?
11. What external contextual factors explain/mitigate outcome results?
5. Evaluation approach
This section of the MEL framework outlines the overall evaluation approach, including the research
design, strategies for measuring primary and intermediate outcomes, and the ethical protocols that must
be in place in order to conduct a rigorous and ethical evaluation, including ensuring child protection and
safe guarding procedures are in place.
5.1 Research design
A randomised control trial (RCT) is not feasible in Afghanistan for a number of reasons. Continuing and
deepening insecurity, and local disillusionment and distrust related to inadequate reach of development
projects, pose serious ethical challenges to an RCT approach, which would require explicitly randomizing
communities to receive interventions or not. Given key timelines and transition objectives of the project
(e.g. to transition girls through primary into secondary school and complete full cycles of education over
eight years), more ethical RCT approaches such as staggering entry of intervention groups (i.e. where
control groups receive interventions at later stages of the project cycle) are not feasible. Consequently, the
research design will be two pronged: (1) a quasi-experimental design tracking two cohorts of girls in
intervention groups (CBE and government hub school), and corresponding cohorts of girls in control
schools, and (2) pre-post test design for girls enrolled in TVET vocational education classes. Further
details of control groups and establishing a counterfactual, including the rationale for not including a
control group for TVET girls,are included in section6.2 of the MEL framework.
The project evaluation will take place over four waves: baseline, first midline, second midline, and
endline. The justification for two midlines is due to the length of the intervention (eight years) and
requirements to track progress and impact consistently and regularly across the intervention.
The research design will include four approaches to ensuring the rigour of the evaluation and establishing
causality of project impacts. First, the evaluation will compare outcomes for treatment and comparison
groups with a difference-in-difference approach, with change occurring in the comparison group
19
providing the counterfactual scenario to interventions. Second, the project will use longitudinal cohort
tracking to follow cohorts of girls receiving interventions across the life of the project. Longitudinal
tracking will be used primarily to track cohorts across two primary outcomes, learning and transition, and
these cohorts will be linked. Longitudinal cohorts will not be the same as in GEC-1 and will be newly
sampled at GEC-T baseline. Third, the evaluation will draw from both quantitative and qualitative
methods, comprising a mixed-methods evaluation that will allow the triangulation of various data sources,
and a more nuanced interpretation of what project change has occurred and why. Fourth, the evaluation
will integrate primary and intermediate outcomes. This will be done by collecting data at the school,
community and household levels in the same sampling clusters, matching different levels through school
and community numerical identifiers, and merging quantitative and qualitative findings through
qualitative and quantitative summaries on key outcomes and intermediate outcomes for each sampling
cluster.
5.2 Measuring outcomes
This section outlines key information on the primary and intermediate outcomes that will be evaluated
and the corresponding approaches and methods proposed to measure these outcomes. This information is
summarized in Table 2 and described in detail further below.Primary outcomes will be measured for both
treatment and control groups; however, intermediate outcomes will only be measured for treatment
groups.
Table 2: Outcomes for measurement
Outcome
Level at which
measurement will take
place
Tool and mode of data
collection Rationale
Frequency of
data collection
Primary Outcomes
Literacy School
EGRA/SeGRA,
qualitative interviews
and focus groups with
girls, teachersand
school shuras
EGRA/SeGRA will be tested at the
school level for CBE and government
school girls receiving BRAC
interventions. This will take place to
ensure that tested girls are enrolled in
BRAC supported schools. Literacy
tests will be complemented by
qualitative interviews and focus
groups with girls, teachers and school
shuras in order to contextualize
learning results. These interviews will
also take place at the school level.
Per evaluation
point
Numeracy School
EGMA/SeGMA,
qualitative interviews
and focus groups with
girls, parents, and
teachers
EGMA/SeGMA will be tested at the
school level for CBE and government
school girls receiving BRAC
interventions. This will take place to
ensure that tested girls are enrolled in
BRAC supported schools. Numeracy
tests will be complemented by
qualitative interviews and focus
Per evaluation
point
20
groups with girls, teachers and school
shuras in order to contextualize
learning results. These interviews will
also take place at the school level.
Vocational and
life skills for
TVET learners
School (TVET centre),
household
Vocational
proficiencies and
knowledge survey,
Rosenberg Self-Esteem
(RSE) Scale and
General Self-Efficacy
Scale (GSE), qualitative
interviews with teachers
and focus groups with
girls
The vocational proficiencies and
knowledgeof girls participating in
TVET training will be tested at the
school level (TVET centre) at baseline
and second midline (selection of new
cohorts of girls), and at the household
level at first midline and endline
(follow up of cohort). School-based
sampling is necessary when first
sampling girls in order to ensure that
enrolled girls are participating in the
evaluation. However, household
sampling is necessary for follow up as
TVET girls will no longer be studying
and will not be trackable at the school
level.The EE will need to create a
vocational survey tool based on the
core competencies and knowledge
expected for each type of vocational
course. Self-esteem and self-efficacy
measures (Rosenberg Self-Esteem
Scale and General Self Efficacy scale)
will be added to the survey to test
whether TVET training increases
girls‟ self esteem, confidence and
efficacy. The rationale for using the
above scales is in order to use
standardized measures and attempt to
apply these in the Afghan context.
Qualitative interviews with TVET
teachers and focus groups with TVET
girls will be conducted to complement
the survey data.
Per evaluation
point
Transition Household
Household
survey,qualitative
interviews and focus
groups with girls,
parents, teachers,
school shuras
Measuring transition will need to take
place at the household level as girls
may have dropped out of school or
completed vocational training between
evaluation waves and may not be
trackable at the school level.
Transition data from the household
survey will be complemented with
qualitative data collection in order to
understand pathways and barriers to
transition, and contextualize reasons
for successful and unsuccessful
transition.
Per evaluation
point
Intermediate Outcomes
Attendance School, household
School register, spot
checks, checklists
appended to
EGRA/SeGRA and
EGMA/SeGMA tools,
Attendance data from school registers
and spot checks will need to take
place at the school level. However,
additional data should be collected at
other levels in order to triangulate the
Per evaluation
point
BRAC will
additionally
21
household survey,
qualitative interviews
and focus groups with
girls, parents, teachers
and school shuras
core attendance data. Questions about
girls‟ reported attendance and
perceptions about attendance
(including barriers and enablers) will
be appended to EGRA/SeGRA and
EGMA/SeGMA tools. Questions
about parental perceptions of girls‟
attendance (including barriers and
enablers to attendance) will be
included in the household survey,
which will enable triangulating
different data sources and
disaggregating girls‟ attendance by
household and demographic factors.
Perceptions of attendance will also be
collected through qualitative
interviews and focus groups to enable
a more nuanced understanding of how
the project activities have impacted on
attendance, and enablers and barriers
to attendance.
conduct
spotchecks and
record school
register
attendance
biannually
Teaching
quality School, household
Classroom
observations, Active
Pedagogy Inventory
(API) tools, Teacher
Value-Added (TVA)
modeling, checklists
appended to
EGRA/SeGRA and
EGMA/SeGMA tools,
household survey,
qualitative interviews
and focus groups with
girls, parents, teachers
and School shura
members
Teaching quality should be measured
primarily at the school level through
classroom observations, and questions
appended to girls‟ EGRA/SeGRA and
EGMA/SeGMA tools. BRAC
encourages the EE to adopt innovative
methods such as API and TVA. At the
household level, questions will be
included in the household
questionnaire to capture parental
perceptions of change in teaching
quality. Inclusion of teacher quality
questions in literacy/numeracy tools
and household surveys will enable the
integration of outcomes and IOs,
particularly how teaching quality
intersects with learning and transition.
Qualitative data will assist in
collecting and interpreting
stakeholders‟ perception of change in
teaching quality over time.
Per evaluation
point
Life skills School, household
Rosenberg Self-Esteem
(RSE) Scale and
General Self-Efficacy
Scale (GSE) appended
to EGRA/EGMA and
SeGRA/SeGMA tools,
qualitative interviews
and focus groups with
girls (mentors and
mentees), teachers, and
parents
Life skills will be primarily measured
through a survey appended to
EGRA/EGMA and SeGRA/SeGMA
tools. BRAC recommends usingRSE
and GSE scales, as per the learning
outcome for TVET girls. Qualitative
data will assist in interpreting how life
skills interact with other outcomes
(including learning and transition).
Per evaluation
point
Attitudes and
perceptions Household, community
KAP survey questions
included in the
household survey (for
A mini KAP survey included in or
appended to existing tools (i.e.
household survey, learning tests or
Per evaluation
point
22
parents) or appended to
learning test tools (for
girls),or appended to
school shura
management checklist
(for shura members),
qualitative interviews
and focus groups with
teachers, parents, girls,
school shuras, and
khalas
school shura management checklist) is
an efficient way of quantifying shifts
in attitudes and perceptions over time,
particularly if using an ordinal scale
such as a Likert scale. However, it
needs to be complemented with
qualitative data to contextualize shifts
in attitudes and perceptions,
particularly since there are a range of
activities contributing to this IO.
School
governance School, community
School shura
management checklist,
focus groups with shura
members, interviews
with PED and DED
officials
School shura members will complete a
checklist with results aggregated
across the shura and then
disaggregated by gender. The
checklist will thus be based on self-
reported perceptions of capacity and
skills, and knowledge related to
BRAC training. The results of the
checklist will be triangulated with
focus group data with shura members.
Central to the GEC-T‟s GESI
approach is understanding how gender
inclusive school shuras are and how
BRAC‟s activities are promoting
gender equality. These questions will
be answered through cross-referencing
the results of the shura checklist with
qualitative data. Interviews with PED
and DED officials will provide
additional information on how
government has supported school
governance, including through
funding and resourcing, and what
remaining challenges there are to CBE
hand over or self-sustainability.
Per evaluation
point
Learning
The outcome indicator for learning is: Number of marginalized girls supported by GEC with improved
learning outcomes. For BRAC‟s project, learning outcomes will be measured at three levels: literacy,
numeracy and vocational skills.
Literacy and numeracy will be measured for two BRAC intervention cohorts, girls attending CBE schools
and government hub schools, and will not be measured for TVET girls as vocational education will not
include literacy and numeracy skills training. Literacy and numeracy will be measured through EGRA
and EGMA tools (for primary grade students) and SeGRA and SeGMA tools (for secondary grade
students), and this will be done at the school level through random sampling of girls‟ enrolment lists.
This is to ensure that all girls sampled are in fact enrolled in a BRAC intervention, with household
23
sampling running the risk of selecting girls enrolled in a non-BRAC school. While this will be more
straightforward at the CBE level (as we know that all girls enrolled in CBE classes are BRAC
beneficiaries), care must be taken at the government hub school level to ensure that selected girls were
previously enrolled in a BRAC CBE intervention in GEC-1 (rather than always enrolled in a government
hub school or other type of non-BRAC school intervention). Consequently, enrollment lists used for
random sampling will be provided by BRAC, not by the government hub school. The primary
measurement for learning will be whether girls have achieved a 0.25 SD increase above the benchmarked
mean at each evaluation wave.
Given that literacy and numeracy tests may be administered to disabled girls (one of BRAC‟s
marginalization target groups), efforts must be made to ensure that such tools are accessible and do not
further marginalize girls by excluding them from participation in the evaluation. According to BRAC‟s
recent assessment of special needs of girls, a large proportion of girls with disabilities have learning
disabilities (including autism) and physical disabilities (including hearing problems, speaking problems,
difficulties walking and difficulties seeing). The external evaluator will be expected to incorporate
inclusive approaches to collecting learning data from disabled children. These may include printing
literacy and numeracy stimuli in large fonts, speaking loudly, slowly and clearly, ensuring that tests are
taking place in a location that physically disabled children can access, and considering applying more
time in timed subtasks for girls who have difficulties speaking.
Literacy and numeracy test results will be complemented by qualitative interviews with teachers and
focus groups with girls, parents and school shura members. Qualitative interviews and focus groups with
teachers and girls will be conducted at the school level, focus groups with parents will be conducted at the
community level, and focus groups with school shura members will be conducted at the school level in
government schools and at the community level in CBE communities. Qualitative data will assist in
contexualising the results of learning tests. Key questions to be addressed through the qualitative data
may include:
How do girls‟ learning experiences and perceptions of teaching quality intersect with learning
results?
How do teachers‟ perceptions of their own classroom practices intersect with girls‟ learning
results?
How do household attitudes and practices (e.g. support for girls‟ education, pressure for girls‟ to
conduct household labour, parental literacy, and support of girls‟ learning and homework etc.)
impact on learning?
How do school shura practices (e.g. classroom monitoring, parent engagement etc.) impact on
learning?
What are the barriers to girls‟ learning and how can BRAC address these barriers through
programming modifications?
24
The third learning outcome, vocational and life skills, will only be measured for girls who are enrolled in
TVET training (life skills as an intermediate outcome will be measured for government hub school girls
engaging in mentoring and co-curricular activities). This learning outcome will be measured by
administering a survey testing girls in key competencies, skills and knowledge related to the specific
vocational courses they are enrolled in. Self-efficacy and self-esteem scales (Rosenberg Self-Esteem Scale
and General Self-Efficacy Scale) will be included in the survey. As noted elsewhere in this document, two
different cohorts of TVET girls will be tracked for the evaluation: cohort one, which will be sampled at
baseline and followed only up until first midline; and cohort two, which will be sampled at second
midline and followed up at endline. Vocational skills will be tested at the school level (TVET centre) at
baseline and second midline, and at the household level at first midline and endline. This is due to initial
cohort sampling being more efficient and straightforward at the school level, but with follow up
evaluation waves having to occur at the household level due to TVET courses only running for six
months, with girls unable to be tracked longitudinally at the school level. The EE will be expected to
design a short survey with knowledge- and skills-based questioning related to each of the core
competencies expected for each type of vocational course. Girls will only answer the questions related to
the core competencies of the training they have participated in (i.e. girls doing beauty parlour training
would not be asked questions about carpet weaving).
The vocational and life skills survey will be complemented by qualitative interviews with TVET teachers
and focus groups with TVET girls. Qualitative data will be primarily directed towards exploring girls‟
perceptions of the quality of TVET curriculum and teaching, their own confidence in their new
knowledge and proficiencies, and perceptions about future prospects based on newly acquired skills.
Transition
The outcome indicator for transition is: Number of marginalized girls who have transitioned through key
stages of education, training or employment (primary to lower secondary, lower secondary to upper
secondary, training, employment or other).
Successful transition is defined in different ways across the three different types of cohorts.
For primary school girls, completing primary school and entering into and completing lower-
secondary school
For lower-secondary school girls, completing lower-secondary school, and entering into and
completing upper-secondary school, or entry into safe, fairly paid employment (if age
appropriate)
For upper-secondary school girls, completing upper-secondary, and enrolling in post-secondary
education (e.g. tertiary), TVET, or entry into safe, fairly paid employment
25
For TVET girls, completing TVET training, sufficient acquisition of vocational skills, or entry
into safe, fairly paid employment (if age appropriate).
For cohort girls in primary CBE and government hub schools, successful transition is defined solely as
entry into and completion of lower-secondary school. Employment is not thought to be successful
transition as employment among primary school children would usually be an indication of child labour
rather than the employment expected after successful transition through secondary school. It should be
noted that girls enrolled in CBE or government hub schools who drop out of primary or lower-secondary
school will not be eligible for BRAC TVET training.
For lower-secondary school girls, successful transition involves completing lower-secondary school, with
two possible transition pathways, entry into and completion of upper-secondary school, enrollment in
TVET, or entry to safe and fairly paid employment. BRAC will not provide TVET training for girls in
this category, hence, entry into TVET would not be linked to the intervention (i.e. girls may enter into
TVET implemented by another governmental or non-governmental organisation). Entry into employment
is in part dependent on girls‟ age. Although employment before the age of 18 is classified as illegal in
Afghanistan, it is not realistic to think that girls cannot enter into safe and fair employment before the age
of 18. For instance, girls completing lower-secondary school may be recruited to teach CBE classes,
particularly in remote areas where accessing female teachers is challenging. Hence, although age is a
defining feature of whether girls should be employed, there should be some flexibility around determining
if this is successful transition (or child labour). This will be done by adding appropriate questions in the
household survey tool (see below).
For upper-secondary school girls, successful transition is defined as completion of upper-secondary and
entry into post-secondary education (e.g. tertiary), TVET, or safe and fairly paid employment. As per
lower-secondary school girls, BRAC will not provide TVET training for girls who have dropped out of or
completed upper-secondary school, and so any entry into TVET would not be linked to the intervention. It
is assumed that on completion of upper-secondary school, girls are at an eligible age for employment.
For all girls in primary and secondary schools, if they remain enrolled but fail a grade and repeat it, this is
classed as unsuccessful transition, although the same girls may transition in future evaluation waves by
progressing on to the next grade.
In relation to TVET girls, it should be noted that BRAC is not conducting any direct support activities
that would lead to TVET girls finding suitable employment; however, BRAC will conduct indirect
activities, such as sharing information about TVET graduates with district and province level skills
associations and the Directorate of TVET with the aim of establishing links between vocationally trained
girls and government and non-government groups that may assist with their future employment. Although
employment in safe and fairly paid employment may be one type of transition for TVET, many girls will
26
be too young to legally work. Consequently, BRAC additionally defines successful transition where girls
complete TVET and acquire sufficient vocational skills, with the view that girls may be able to use these
skills in paid and safe employment when reaching a suitable age.
Measuring transition will take place at the household level. For CBE and government school girls,
learning and transition cohorts will be the same at baseline. At subsequent follow up evaluation waves,
girls in the learning sample must be replaced if they have dropped out of school. However, girls in the
transition sample (sampled at baseline) must be followed up at all evaluation waves, regardless of their
enrollment status, in order to capture an accurate understanding of their different transition pathways over
time. Due to the unpredictability of drop out rates, an appropriate attrition buffer will need to be
established to ensure that a sufficient number of girls in the transition sample are retained throughout the
project (see section 6.4 of the MEL framework). As TVET girls enroll in vocational training for six
months, and those sampled at baseline will not be enrolled in TVET at midline or endline evaluation
waves, transition of TVET girls must also be measured at the household level. TVET girls are not as
susceptible to drop out due to the short course implementation time; however, attrition buffers must still
be established due to other possible reasons for attrition (see section 6.4 of the MEL framework). In
contrast to the transition sample for CBE and government school girls, which will be tracked and
followed from baseline to endline, TVET girls will only be tracked for one evaluation wave.
Consequently, TVET girls will be sampled at baseline and tracked and surveyed at the first midline
evaluation point. At the second midline evaluation point, a new cohort of TVET girls will be sampled and
then tracked and followed up on at endline.
Transition for all three cohorts of girls will be measured through a household survey tool. Questions will
include: current enrolment status and type of school, past year enrolment status and type of school, grade
(for CBE and government hub school girs), current employment status, past year employment status and
type of employment. Questions should also be asked about the quality of current employment and income
earning activities, whether girls are remunerated for their work, and whether they feel safe and respected
in their employment environment. For all three cohorts of girls, transition will be measured quantitatively
through a nominal variable (yes or no) indicating successful transition or not. The transition figure entered
into the logframe will be the total beneficiary number (total N) multiplied by the transition rate (% of
sampled beneficiaries who have transitioned). The independent evaluator will also include sub-reporting
on the frequency of different types of transition pathways.
Measuring transition will also require adopting qualitative methods in order to build a deeper
understanding of mediating pathways and barriers to girls‟ transition. Qualitative interviews and focus
groups will be conducted with girls (both those who have successfully transitioned and those who have
not), parents (mothers and fathers) of girls who have transitioned or not, teachers (for all three types of
school/training), and school shura members (for CBE and government hub schools). Qualitative data will
be collected from in-school girls, teachers and government hub school shuras at the school level, and from
27
dropped-out/non-transitioned girls, parents and CBE shuras at the community level. Key questions to be
addressed through the qualitative data may include:
Where successful transition has taken place, which project activities account for different types of
transition?
How have project activities reduced barriers to girls‟ transition?
What are the reasons for girls not succeeding in transition and how can BRAC address these
barriers through programming modifications?
What are the intersections of different categories of marginalization with transition?
Are there other types of successful transition not expected or included in the transition pathway
plan (for instance, re-enrollment of TVET girls into madrasas or formal schools)?
For girls who have entered employment, is this considered child labour, or are girls working in
safe, fairly paid employment? What does a safe working environment look like?
Attendance
The outcome indicator for attendance is: percentage improvement in girls' attendance.
Attendance will be measured for two BRAC intervention cohorts, girls enrolled in CBE schools and
government hub schools, and will not be measured for TVET girls as vocational education courses are
only six months in length. Attendance will be measured through attendance data collection forms
implemented at two levels: attendance recorded in school registers, and spot check attendance. For school
register attendance, enumerators will collect: total class enrolment, number of students present on the six
days in the week immediately preceding the data collection (i.e. six attendance figures collected). For
spotcheck attendance, enumerators will do a head count of sampled schools on the same day as collecting
school attendance register data. In addition to doing a head count, enumerators will record data for total
class enrolment, and the same day attendance number recorded by the teacher. BRAC will also do two
annual spot checks of attendance using the same method. In CBE schools, independent evaluator
attendance and spot check data will be collected for all sampled schools. In government hub schools,
attendance and spot check data will be collected for all grade 5 and 6 classes at baseline and subsequent
cohort grades at midline and endline.
There are a number of possible challenges in collecting attendance data. As noted in section 2 of the MEL
framework, conducting announced spotchecks is highly challenging in Afghanistan, where security risks
to enumerators must be mitigated by BRAC facilitating their entry into target communities and schools.
However, BRAC field staff will support a semi-announced approach, whereby school staff and shuras
will be advised of a one week window in which data collectors will visit schools, with the actual day not
to be announced. Another challenge is in assuming that the attendance registers kept by teachers are
accurate, particularly if they are aware that enumerators will be visiting to collect attendance data. This
28
challenge will be partially mitigated by collecting 6-day attendance in the week prior to data collection
and doing semi-announced spotchecks in the following week. Data will also be collected in classroom
observation checklists on whether teachers correctly call the attendance roll in order to triangulate
attendance data and determine its likely accuracy.
Self-reported quantitative attendance data will also be collected from girls and carers. Questions about
girls‟ perceptions of attendance, including self-reported attendance and reasons for non-attendance, will
be added to surveys and checklists appended to EGRA/EGMA and SeGRA/SeGMA tools. Questions
about carer reports of girls‟ attendance will also be included in the household survey. Although girl-
reported and carer-reported attendance are not the ideal measures of attendance, they can complement the
primary attendance measures by allowing more complex disaggregation of data (e.g. according to key
demographic characteristics).
Quantitative data will be complemented by qualitative data, including in-depth interviews with teachers,
and focus groups with girls, parents, and school shura members. Qualitative interviews and focus groups
with teachers, girls and government hub school shuras will be conducted at the school level, and focus
groups with parents and CBE school shuras will be conducted at the community level. Qualitative data
will assist in contexualising the results of the attendance data. Key questions to be addressed through the
qualitative data may include:
What are the enablers of girls‟ school attendance and to what extent have BRAC project activities
(e.g. stipends, transport incentives, mothers‟ forum, parents‟ meetings, mentors, khalas, school
shura management, community engagement) supported these?
What are the persisting barriers to girls‟ school attendance and how can BRAC strengthen its
programming to respond to these barriers.
Teaching quality
There are two outcome indicators for teaching quality:(1) percentage of teachers demonstrating improved
capacity in subject-based learning, gender-sensitive pedagogy and child-centred learning, and (2)
percentage of schools that have established child-friendly learning environments and following updated
teaching-learning approaches.
Teacher quality will be measured for teachers in all three treatment groups (CBE schools, government
hub schools and TVET centres), primarily through classroom observations and checklists. When
designing classroom observation checklist tools, the independent evaluator should be familiar with
BRAC‟s teacher training packages and ensure that tools are properly calibrated to capture the key teacher
skills, pedagogy and practices that are included in training.Observation checklists will also measure
teacher corporal punishment and respectful, child friendly behaviours in the classroom. At each
29
evaluation wave, the teaching quality indicator achievement will comprise a % of checklist items
observed in classrooms for child-centred and gender-sensitive pedagogical practices, and appropriate
child protection practices (i.e. non-violent disciplinary methods).In addition to standard classroom
observation methods, BRAC encourages the independent evaluator to include more innovative methods,
such as Active Pedagogy Inventory tools, and Teacher Value-Added (TVA) modeling.
Improved capacity in subject-based learning cannot be reliably measured through a classroom observation
as it will be challenging to control which class subject will be observed, and all subjects would ideally
need to be observed, which is not feasible from a time and budget perspective. Furthermore, enumerators
would theoretically be required to understand the key skills and observed behaviours that would comprise
adequate subject-based teaching, which is beyond the scope of the evaluation. Hence, subject-based
learning will be measured through qualitative methods (see further below).
Teaching quality will also be measured through surveys appended to the EGRA/EGMA and
SeGRA/SeGMA tools in order to capture girls‟ perceptions of teaching quality (and improvement in
teacher quality), and questions in the household survey for carer perceptions of teaching quality (and
improvement in teacher quality). Questions asked of both girls and carers will also address teacher
absence and corporal punishment.
Qualitative interviews with teachers, and focus groups with girls and school shuras (all at the school
level), will be conducted to assist in triangulating and contextualising the data from classroom
observations. Key questions to be addressed through the qualitative data may include:
What are teacher perceptions of the quality of their own teaching and their confidence in teaching
and how have these changed over time?
What are girls‟, parents and school shura members‟ perceptions of teaching quality and how have
these perceptions changed over time?
What other factors may explain teaching quality?
Has subject-based training (e.g. maths) impacted on teacher confidence and girls‟ learning? How?
Has grade change training adequately equipped teachers to deliver quality education at higher
grade levels?
Life skills
There outcome indicator for life skills is: percentage of learners with improved level of self esteem or self
confidence.
This outcome is linked to BRAC‟s mentoring program and co-curricular activities (e.g. debating, story
telling, drama, reading competitions and wall magazines). All the activities are implemented in
government hub schools only,and are hypothesized to lead to girls‟ improved life skills, defined as
30
enhanced self-esteem, self-confidence and self-efficacy. Therefore, the outcome is measured only at the
government hub school level. Life skills will be measured through a survey appended to EGRA/EGMA
and SeGRA/SeGMA tools, with appropriate measures to be determined by the independent evaluator. For
self-esteem, self-confidence and self-efficacy, BRAC suggests using scales that have been standardized in
other contexts such as the Rosenberg Self-Esteem Scale and General Self-Efficacy Scale (GSE), with
attention paid in piloting to the cultural and linguistic appropriateness of these scales in Afghanistan.
Qualitative methods will include focus groups with girls (mentors and mentees), and teachers, all at the
school level, and focus groups with parents. Qualitative research questions may include:
How effective has the mentoring program been in increasing girls‟ life skills, learning and
attendance? Why/why not?
How do girls‟ life skills contribute to other key outcomes, including learning, attendance and
transition?
Have girls‟ life skills had any impact on family attitudes and perceptions about girls‟ education,
future employment and broader civic participation?
Attitudes and perceptions
There are two outcome indicators for attitudes and perceptions:(1) percentage of parents reporting
improved attitudes towards girls education and employment, and (2) percentage of girls reporting that
they feel confident and safe while attending and travelling to and from schools and TVET centres.
This outcome is linked to a variety of BRAC activities, and can be broadly separated into two key areas:
(1) supportive attitudes towards girls‟ and women‟s education, employment and civic participation
(linked to advocacy meetings with community members and school management shuras, community
training orientations covering gender awareness, mother‟s forums, and gender awareness training with
school shuras); and (2) perceptions of girls‟ safety and comfort traveling to school or TVET (linked to the
provision of stipends and travel incentives, and school chaperoning by khalas). Due to the various
stakeholders whose attitudes and perceptions may change, BRAC recommends measuring this
intermediate outcome with parents and girls (both for perceptions of safety and supportive gender
attitudes), and with school shuras (for supportive gender attitudes). This will be done by appending the
same KAP survey to the household survey (for parents), learning tests (for girls) and school shura
management checklists (for school shuras).
Although KAP survey questions will allow the primary measure of a percentage of stakeholders with
positive attitudes and perceptions, qualitative data will allow for a more nuanced understanding of how
and why change has happened, and how much this change can be linked to BRAC‟s activities or broader
contextual issues.For instance, girls‟ and parental perceptions of girls‟ safety declining could be linked to
31
broader insecurity, or linked to BRAC‟s activities not being implemented sufficiently (e.g. khalas not
accompanying girls to school anymore). Furthermore, given the range of activities linked to this IO,
qualitative data will assist to identify why change is happening and how it can (or cannot) be attributed to
BRAC activities. Qualitative interviews will be conducted with teachers at the school level, and focus
groups will be conducted with girls and (gender segregated) school shuras at the school level and parents
at the community level. Qualitative research questions may include:
What is the strongest driver of change in attitudes and perceptions?
Which BRAC activity has contributed most to changing attitudes and perceptions?
School governance
The two outcome indicators for school governance are: (1) Responsiveness and accountability of school
shura members improved in “X” percentage from the baseline, and (2) number of schools with further
funding/government support/other and serving the education demands of “X” number of students.
School shuras will be established and/or supported in CBE and government hub schools. TVET centres
will not have shuras and, thus, this outcome indicator is linked only to CBE and government hub schools.
This outcome indicator will be primarily measured through a school shura management checklist to be
filled out by school shura members. The checklist will be developed by the independent evaluator, and
will include items related to BRAC‟s shura capacity and knowledge training, such as shura advocacy
skills, school monitoring and auditing skills, knowledge and observation of child protection standards and
codes of conduct, and knowledge of gender-sensitive school development plans. Responses to checklist
items will be aggregated across all responding shura members to form a % score indicating shura skills
and capacities. This score will be disaggregated by gender in order to track capacity for both male and
female members over time.
The second indicator, schools obtaining further funding or other types of contributions, is included as
output 6.3 in the monitoring plan, and will be primarily measured by BRAC through financial and time
records for in-kind and monetary contributions (this will require BRAC to develop a monetised value
system for in-kind contributions). Although this data will not be collected at outcome level, the EE will
draw from monitoring output data and complement it with qualitative data aimed at exploring the
opportunities and challenges for self-financing or advocating for school funding (see below).
Qualitative data will be collected through (preferably gender segregated) focus groups with school shura
members, and in-depth interviews with PED and DED officials. Qualitative research questions may
include:
How effective has shura training been in preparing shuras to effectively manage schools? What
more could be done to support shuras?
32
Has the shura been able to mobilise in-kind or monetary contributions for the school? What are
the barriers and enablers of doing this?
Has the shura been able to mobilise support for community take-over of CBE classes? What are
the barriers and enablers of doing this?
To what extent is school shura membership gender inclusive? Do women participate
meaningfully in oversight and accountability processes? What are the barriers to their
participation?
What is the relationship between shura attitudes towards girls‟ and women‟s participation (see
previous IO) and women‟s meaningful participation in school governance?
How can BRAC programming promote more meaningful participation of women in school shuras
and reduce barriers to their participation?
What kind of support has the PED and DED provided for the continuity of schools, and has this
support changed over the life of the project? Why/why not?
5.2.1 Sustainability
The GEC-T outcome for sustainability is that the project can demonstrate that the changes it has brought
about which increase learning and transition through education cycles are sustainable. In line with GEC
guidance, sustainability will be measured at three levels (school, community and system), against a
Sustainability Scorecard with ratings between 0 and 4 for each level. The ratings are defined as below:
Score of 0: Negligible sustainability (null or negative change)
Score of 1: Latent sustainability (knowledge development and change in attitudes)
Score of 2: Emerging sustainability (changes in behaviour)
Score of 3: Sustainability becoming established (critical mass of behaviour change)
Score of 4: Established sustainability (changes are institutionalized)
The independent evaluator will determine ratings at each evaluation wave based on progress on key
indicators.
BRAC has developed a set of indicators for each of the three sustainability levels.
School
Indicator 1: Percentage of schools with established child-friendly learning environments and following
updated teaching-learning approaches.
Indicator 2: Percentage of trained teachers and school management shura members continuing their
services.
33
Indicator 3: Percentage of CBE schools operating with community financing.
Community
Indicator 1: Percentage of communities actively involved in government schools, CBE secondary schools
and vocational schools.
Indicator 2: Percentage of communities actively involved in negotiation with the government and
mobilizing resources to continue girls’ education.
Indicator 3: Parents’ perceptions and attitudes changed X% from baseline.
System
Indicator 1: Percentage of CBE students and teachers absorbed by the government education department.
Indicator 2: Policy makers are responsive and providing necessary support to improve girls’ education.
Indicator 3: Responsiveness and accountability of SMS members improved in “X”% from the baseline.
Table 3 indicates the types of measurement, sources, methods, rationale and frequency of data collection
required to measure indicators across the three key sustainability levels.
Table 3: Sustainability outcome for measurement
Sustainability
Level
Where will
measurement take
place?
What source of
measurement/ verification
will you use?
Rationale – clarify how you
will use your qualitative
analysis to support your
chosen indicators.
Frequency
of data
collection
School School
School
School, community
1. School curriculum checklist
item, school CP and child
safeguarding policy checklist
item, qualitative interviews
with school staff, teachers and
SMSs
2. Teacher and SMS
registration lists (BRAC
monitoring), qualitative
interviews and focus groups
with teachers and shuras
3. Financial and time records
for in-kind and monetary
contributions (available from
BRAC monitoring), interviews
and focus groups with school
staff, SMSs, community
members (e.g. parents) and
There is a combination of
quantitative and qualitative
methods proposed to measure
sustainability at the school level.
School curriculum and
CP/safeguarding checklist items
can be appended to a school
survey to be implemented along
side teacher observations.
Teacher and SMS registration
lists and financial and in-kind
contributions will be obtained
from BRAC monitoring data.
Qualitative interviews and focus
groups at the school and
community levels will be central
to measuring enablers of and
barriers to sustainability and
Per
evaluation
point
34
other community stakeholders
measuring the scale of
sustainable outcomes according
to the scorecard ratings.
Community Community
Community,
government
Household,
community
1. Qualitative interviews and
focus groups with community
members (parents, community
leaders, religious leaders)
2. Financial and time records
for in-kind and monetary
contributions (disaggregated
by type of contributor),
qualitative interviews and
focus groups with community
members (parents, community
leaders) and government
officials
3. KAP survey on gender
attitudes and perceptions
(appended to household
survey), qualitative interviews
and focus groups with parents
Records of financial and time
contributions that will be used to
measure other outcomes and
outputs will be key for
understanding how communities
and community members are
specifically contributing,
requiring disaggregation of data
by type of contributor.
Qualitative data will be central
to measuring sustainability at the
community level. While KAP
surveys can measure change
over time, qualitative data will
assist in understanding how
sustainable these changes are
and the enablers of and barriers
to sustainability (e.g. perceptions
may improve over time but
qualitative data may suggest that
these improvements are linked to
schools remaining open or
employment availability).
Qualitative methods will also
explore the types of community
engagement with schools, and
how this engagement could
continue in the absence of the
school (e.g. if TVET centres
close at the end of the project).
Interviews with government
officials will assist to triangulate
data on community engagement,
including government
perceptions of the extent and
efficacy of this engagement.
System School,
government
Policy level
School
1. BRAC student tracking
database, CBE teacher
tracking database, qualitative
interviews with teachers,
school staff, SMSs, and
PED/DED officials
2. Policy activity (e.g.
workshop) reports, and policy
outputs
3. School shura management
and capacity checklist, focus
groups with school shuras
Although BRAC‟s student
tracking database will contain
information on CBE student
transfers to government hub
schools, BRAC will also require
a CBE teacher tracking database
to understand the trajectory of
CBE teachers after project
completion. The extent to which
students and teachers have been
absorbed into the hub school
system, and the enablers of and
35
challenges to this occurring will
be measured through qualitative
methods. At the policy level,
measuring sustainable systems
will require a review of policy
documents, which may include
reports from events and activities
which BRAC has contributed to
(e.g. donor or MoE advocacy or
policy workshops) or policies
BRAC has contributed to (e.g.
CBE policy). Central to
understanding sustainability is
whether clear actionable points
from event reports or policy
documents have been produced
and followed up on. At the SMS
school level, while the shura
checklist will measure change in
capacities and competencies
over time, focus groups will
measure shura perceptions of
how much these capacities and
competencies have been
institutionalized and whether
internal processes have been put
in place for the continuity of
capacity and management
support.
5.3 Ethical protocols
The basis for BRAC‟s ethics framework is a set of four internationally recognized ethical principles for
research conduct (respect, beneficence, justice, research integrity), outlined below.
Respect for persons
Researchers must respect the dignity of all individuals involved in research and recognize their rights to
autonomy and self-determination, such that they can make their own choices and decisions to participate
in research (or not) based on informed consent and with no coercion. Respect for persons also involves
respect for their privacy, which requires ensuring confidentiality or anonymity, and guaranteeing
protection of data.
Beneficence
36
Beneficence in research ethics refers to an obligation to maximize benefits and minimize harm, by
anticipating any potential negative consequences of conducting the research and ensuring that procedures
for mitigating harm are in place.
Justice
Justice in research means to distribute the risks and benefits of research fairly and refrain from research
practices that reproduce injustice, or take advantage of or exclude marginalized populations.
Research integrity
The principle of research integrity is related to ensuring the quality, rigour, professionalism, transparency
and validity of research. Conducting research that is of poor quality should be seen as unethical and,
furthermore, an ineffective and inefficient use of resources.
5.3.1 Ethics
This section describes the ethical protocols that should be put in place to ensure that all BRAC GEC-T
evaluation activities are conducted according to high ethical standards. We present a broad approach that
links key ethical principles, standards and risk mitigating procedures at each evaluation stage (pre-data
collection and planning, data collection, data analysis, storage and report writing, and dissemination of
results). Some of these risk mitigation procedures are outlined in more detail in section 10 on risk
management. In the following section, we focus on a specific sub-component of BRAC‟s ethical protocol,
child protection, in order to outline how children‟s safeguarding standards will be ensured.
While this section outlines key standards and mitigating procedures, the external evaluator will be
expected to develop a comprehensive risk register detailing full details of research risks and mitigations in
line with their operational procedures. The external evaluator should also ensure there is an individual on
the evaluation team who is responsible for ensuring ethics protocols are systematized and implemented
throughout all evaluation waves.
In relation to national research ethics protocols in Afghanistan, there are no compulsory norms or policies
for government or private regulation of research or evaluation. The Ministry of Public Health does have a
recently established Institutional Review Board (IRB) that governs research ethics for health related
projects (primarily medical research with humans); however, there is no corresponding IRB in the
Ministry of Education or other ministries currently in place. BRAC‟s Research and Evaluation Unit will
thus takeresponsibility for ethics oversight and ensure that risk mitigation procedures are being
implemented by the external evaluator. The independent evaluator will be asked to provide BRAC with
regular (e.g. monthly) ethics reports detailing any ethical challenges and how these were addressed. In
37
relation to child safeguarding, and evaluator staff observing or participating in any type of violence or
harm, specific protocols will be in place (see further details in the following section on child protection).
Pre-data collection and planning
Prior to conducting data collection, the external evaluator in consultation with BRAC must ensure that all
research design and planning is conducted with a clear set of risk mitigation procedures in place.A key
risk mitigation procedure in the starting phase of the project is for BRAC to ensure the selection of an
adequately skilled and experienced external evaluator. Another vital risk mitigation procedure that cuts
across multiple ethical standards is the implementation of comprehensive training of evaluation staff,
including enumerators and interviewers (see Table 4, and section 9.1 of the MEL framework on training).
The design of tools and methodological approaches should also be ethically implemented through a range
of standard practices, including by piloting to ensure that tools and methods are appropriate and sensitive
to various forms of diversity and cultural context, and making any necessary adjustments.
In the pre-data collection and planning stage, the principle of justice is particularly important.
Marginalised populations should be included in the evaluation where relevant, and adequate
methodological and sampling procedures designed to ensure they are not excluded. For instance, in a
context where community leaders and other powerful persons often dominate discussions in focus groups,
sampling and methodological procedures should be mindful of ensuring that less powerful voices are
heard. Furthermore, where required, focus groups should be gender segregated due to conservative norms
in Afghanistan and to ensure that women‟s voices are heard. Given that BRAC‟s intervention in GEC-T is
targeting disabled children, procedures must be in place for the meaningful inclusion of these children, for
instance, by locating adequately accessible locations for interviews with physically disabled children, and
ensuring that adequate pace and clarity of interview processes are put in place for children with learning
or attention disabilities. Although the participation of marginalized and vulnerable groups in the
evaluation should be facilitated, this should always be balanced with an understanding of how
participation may burden these groups. When designing tools, the external evaluator should consider
which methods are essential and with whom, and avoid oversampling, collecting unnecessary data or
burdening participants with time intensive and obtrusive methods, particularly with the knowledge that
such methods may remove participants from important daily activities, including income-earning, school-
related and household duties and activities. For example, ethical research at the school level should avoid
taking children out of a learning context (i.e. taking children out of an active classroom) or taking
teachers away from their duties (i.e. conducting an interview during class times, thus leading to teacher
absence or the cancelation of class).
Table 4: Ethical standards and mitigating procedures in pre-data collection and planning procedures
Ethical
principle
Ethical standard Mitigating procedures
38
Respect
Respect the rights and dignity of all
people involved in the research,
regardless of age, ethnicity, gender,
religion, disability or other forms of
diversity
Train enumerators in non-discriminatory, culturally
sensitive and respectful behaviours when conducting
research.
Recruit female enumerators to interview women and
girls, and male enumerators to interview men and boys.
Train enumerators in how to obtain informed and
voluntary consent with no coercion, and how to respect
confidentiality and privacy of participants at all times.
The external evaluator should ensure that research tools
and methods have been piloted and are appropriate and
sensitive to age, gender, and other forms of diversity.
Beneficence
Do no harm and avoid any practices that
may harm, abuse, discriminate or exploit
participants.
Ensure that duty of care and any incident protocols and
reporting procedures are in place prior to data collection
and that field staff are properly trained in how to use
them.
Protect the wellbeing and security of
researchers.
The external evaluator should present adequate safety
and security plans for field staff, including any necessary
security training
Justice
Ensure that marginalized populations are
included in the research, where relevant.
Ensure that clear categories of marginalization are
identified, and that sampling approaches and methods
are inclusive of marginalized groups where relevant.
Avoid disproportionately burdening
marginalized populations.
Consider which methods are essential and with whom,
and avoid oversamplingor burdening participants.
When conducting focus groups or other time and energy
intensive data collection procedures, offer refreshments.
Research
integrity
Ensure that researchers have the
necessary skill and experience to conduct
the research
Implement robust selection procedures when choosing
an independent evaluator.
Implement comprehensive training of enumerators and
research staff.
Data collection
During data collection, the principle of respect is paramount to ensuring that research is conducted
consensually, privately and confidentially. Obtaining informed consent first involves ensuring that
participants are provided with sufficient information about the evaluation and their participation in it, and
that their consent is given with no coercion or pressure to participate. In Afghanistan, given very low
literacy rates, particularly among the marginalized populations targeted by BRAC interventions,
information should be provided and consent obtained verbally rather than through written information
sheets and signed consent forms.
Ensuring confidentiality and privacy of participants during data collection can be complex, particularly in
school settings, where there may be insufficient private settings, particularly in BRAC CBE schools
where there may only be one room where a class is taking place. Finding a private space in households
may also be difficult where other family members may insist on being present during data collection
procedures. Such challenges should be considered in the pre-data collection and planning stages, and
necessary mitigating strategies developed. For instance, enumerators should be trained in how to politely
request bystanders for privacy, and ways to conduct interviews (or terminate them) if privacy cannot be
39
established. During qualitative focus groups, ensuring complete individual privacy and confidentiality is
clearly not possible as there are multiple participants present. The limits of confidentiality should be
stated clearly in these circumstances.
Research integrity is also vital to ensuring that data collection is implemented rigorously and to a high
quality.External evaluators must have comprehensive monitoring systems in place to ensure that data has
been collected honestly and without fabrication, sampling and data collection procedures have been
conducted without bias, and data outputs are of a sufficiently high standard. In cases where data outputs
are poor or where there are doubts about whether data has been fabricated, the external evaluator must
have clear follow up procedures, including additional enumerator training and re-fielding of data
collection if necessary. In cases where enumerators are unable to implement sampling or data collection
protocols due to unavoidable or difficult to mitigate challenges (e.g. pressures from community members
to sample certain groups of people), these cases should be reported to the external evaluator and BRAC
and discussed in any evaluation outputs.
Table 5: Ethical standards and mitigating procedures in data collection
Ethical
principle
Ethical standard Mitigating procedures
Respect Obtain informed and voluntary consent
without coercion
Provide sufficient information to participants before
obtaining consent, including details of the evaluation,
what participation entails, any benefits or risks of
participation, how the data will be used, and the limits of
confidentiality.
Participants should at no time be coerced or put under
pressure to consent to participate.
All participants should be allowed to withdraw their
consent at any time.
Consent should be obtained before recording
participants with any audio or video devices, or before
taking photos.
Ensure privacy and confidentiality Enumerators should respect the privacy of individuals,
including respecting participants‟ refusal to answer any
sensitive questions.
Enumerators must respect the confidentiality of
participants by not sharing any information obtained in
interviews with others.
Research
integrity
Ensure that research is conducted with
transparency, honesty and accountability
Enumerators must conduct all data collection
transparently and honestly, and must not falsify or
fabricate data.
The external evaluator must ensure that adequate field
monitoring systems are in place to monitor the quality
and legitimacy of data collection processes and outputs.
Data analysis, storage and report writing
40
Ethical procedures should be put in place for all post-data collection processes, including data cleaning
and analysis, storage and report writing. There are two key ethical principles that are central to ethical
conduct in these post-data collection processes. Respect and ensuring privacy and confidentiality of
participants requires following appropriate protection procedures related to storage, transfer and
destruction of data.For instance, external evaluators and BRAC will store contact details and other
identifying features of participants in a separate file, linked to the main dataset through a unique
numerical identifier. External evaluators and BRAC will also establish clear timelines and processes for
the transfer, retention or destruction of hard copy and/or soft copy data sources. To ensure participant
confidentiality, all quantitative and qualitative data sets will be de-identified before sharing with external
stakeholders (including the fund manager). All data included in reports will also be de-identified. If this is
challenging (e.g. when reporting data from a highly recognizable individual such as a highly ranking
government official), the external evaluator will make sure the limits of confidentiality where explained
during data collection and consider modifying or omitting identifiable details from reporting (such as
province of interview).
Research integrity is also vital during post-data collection processes. Data cleaning should be transparent
and where any associated procedures have involved the modification of raw data that has not contained
errors, justification for this by the external evaluator should be provided to BRAC.Data analysis
procedures should be as independent and objective as possible, and, where possible, subjected to inter-
rater reliability processes (e.g. independent coding of a proportion of qualitative data sets by two different
research staff) and peer review (e.g. feedback from more senior research staff in the external evaluator
team).Reporting should also be as independent and objective as possible, and external evaluators should
ensure that any limitations or biases in the findings are presented in reports and necessary caveats
included.
Table 6: Ethical standards and mitigating procedures in data analysis, storage and reporting
Ethical
principle
Ethical standard Mitigating procedures
Respect
Ensure privacy and confidentiality Follow appropriate protection procedures related to
storage, transfer and destruction of data.
De-identify all datasets (quantitative and qualitative)
before sharing with stakeholders, including the FM.
De-identify all data included in reports.
Research
integrity
Ensure that research is conducted with
transparency, honesty and accountability
Data cleaning, analysis and reporting processes should
be transparent, and as independent and objective as
possible.Inter-rater reliability and peer review are key
processes that can support this transparency and
objectivity.
Dissemination of results
41
Sharing and disseminating the results of research and evaluation to key stakeholders is important so that
others may benefit, and should always be done unless the risks of doing so outweigh the benefits. There
are a number of ways to mitigate these potential risks. It is assumed that respectful practices such as
maintaining privacy and confidentiality through de-identifying data will already have taken place when
writing evaluation reports. However, further protocols should be in place to ensure that the dissemination
of results and reporting outputs will not harm research stakeholders, particularly marginalized and
vulnerable groups. Where necessary, BRAC will consider omitting any potentially harmful findings from
publically shared documents. BRAC will also ensure, as much as possible, that dissemination is inclusive
of participants and communities, and that marginalized groups (e.g. women, disabled persons, children)
are not excluded from dissemination activities. This will require some thinking of the best mediums,
settings and processes for dissemination, for instance: disseminating findings verbally rather than in
written form in order to be inclusive of illiterate persons; organizing separate dissemination activities for
men and women in more conservative settings where women may be discouraged from sharing spaces
with men, and ensuring that female staff are disseminating findings to women; creating accessible spaces
and dissemination mediums to encourage the participation of disabled persons; and ensuring that
dissemination activities take place with children and are child-friendly (see next section for further
details).
Table 7: Ethical standards and mitigating procedures in dissemination of results
Ethical
principle
Ethical standard Mitigating procedures
Beneficence
Do no harm and avoid any practices that
may harm, abuse, discriminate or exploit
participants.
When disseminating reports, BRAC will be mindful of
any findings that may inadvertently harm participants
and, where necessary, omit these from any publically
shared documents.
Justice Ensure that marginalized populations are
included in the research, where relevant.
Ensure that findings are shared with research
participants and communities, and dissemination
strategies and practices are inclusive of marginalized
groups.
5.3.2 Child protection
Overall, the ethical protocols outlined in the previous section are applicable to both adults and children;
however, there are some more specific protocols that are relevant to ensuring protection and safeguarding
of children throughout the different stages of the evaluation process, particularly during pre-data
collection and planning, data collection and dissemination of results.These protocols will be overseen by
BRAC‟s Child Safeguarding Committee throughout the project.
Pre-data collection and planning
When bidding out and selecting an external evaluator, BRAC will require bid submissions to demonstrate
42
that external evaluators have extensive experience conducting research with children. If not already in
place, the external evaluator will be expected to develop a child protection policy, adhere to this policy
throughout the evaluation and ensure all research staff, including data collectors, have been briefed and
trained on the child protection policy and related practices and expectations. Due to poor or non-existent
governance structures in place at government levels to track and register child offenders, thorough
screening of data collection staff for evidence of past abusive practices towards children is unrealistic.
However, the external evaluator will be expected to do everything possible to ensure that recruited staff
do not have a history of child abuse or violence (e.g. by refusing to hire enumerators known to have acted
inappropriately with children in previous research projects).
When designing research methods, tools and sampling procedures, the external evaluator must ensure that
these are safe. For instance, questions in research tools should be sensitive and age appropriate to
minimize distress to children, and data collectors must understand how to deal with any possible distress
or trauma that may arise (e.g. by terminating an interview or providing referral services to children).
Prior to data collection, as part of the child protection policy, the external evaluator will be expected to
develop a code of conduct and incident protocols to deal with (1) any enumerator observation of violent
or inappropriate conduct, and (2) any enumerator participation in violent or inappropriate conduct. In the
first case, data collection staff may observe violence, for example, observing corporal punishment by
teachers in a classroom setting. Incident protocols should involve enumerators reporting incidents to
evaluator fieldwork supervisors or managers, who should then report them to directly to BRAC. It is also
possible that enumerators do not directly observe violence, but are told of violent practices or experiences
by children during interviews. This is a highly sensitive issue and must be treated carefully; however, if
children request assistance, research staff should be prepared with incident protocols and possible referral
contacts. Although referral services for children are limited in more remote areas, BRAC‟s collaboration
with CPAN (Child Protection Action Network) groups in target provinces could serve as an important
referral pathway.In the second case, data collection staff may breach the external evaluator child
protection policy by acting inappropriately, disrespectfully or violently towards a child. The external
evaluator‟s child protection policy must include mitigating strategies and, importantly, strategies for
monitoring enumerator behaviour and conduct with children. Any breaches found must be dealt with
accordingly, including by directly reporting such breaches to BRAC staff (who may refer such cases to
CPAN groups), immediately terminating employment of the enumerator in question (and taking any
punitive measures necessary), and ensuring the wellbeing of the child (referring to any health, justice or
other services if necessary).
Data collection
As outlined in section 5.3.1, data collectors must obtain informed consent before conducting any data
collection activities. When interviewing children, in most cases children‟s parents‟, guardians or
43
caregivers must also provide informed consent. In BRAC‟s GEC-T evaluation, this will be highly
challenging due to the proposed sampling procedures, which involve sampling children at the school level
and then visiting homes to interview parents or other household members. Obtaining individual consent
from all parents prior to doing school-based sampling would be very labour, time and cost intensive,
particularly for obtaining parental consent for girls enrolled in government schools or TVET centresthat
are far from CBE communities. A more viable alternative would be the following procedure (although
BRAC will welcome alternative solutions from the external evaluator). Prior to any data collection,
BRAC field staff will include an evaluation briefing session with parents in CBE communities (i.e.
parents of girls attending CBE schools, government schools or TVET centres) to share information about
the evaluation and obtain group consent. These briefing sessions could be linked to existing BRAC
activities, including Mothers Forum meetings, parent meetings or community advocacy meetings, and
BRAC field staff should take care not to exert undue influence or coercion when seeking parental
consent. If any parents refuse to consent, BRAC will record the details of the relevant girls (including
name, age, father‟s name etc.) and these details will be shared with the external evaluator, who will
exclude girls from the school enrolment lists before conducting random sampling. At the school level,
girls must also individually consent to participate in any evaluation data collection, and may refuse to do
so even if their parents have provided consent.
Although the above procedures are viable for obtaining the consent of girls‟ parents in CBE (i.e.
intervention) communities, they will be much more difficult to implement in government school control
communities, for a number of reasons. Girls may live in a number of different communities, near or far
from government control schools. BRAC staff would not realistically be able to find girls‟ households
and may not be able to securely access these communities. A possible alternative might be to have
government schools share information about the research with parents through evaluation information
sheets, request written consent, and only randomly sample participants from lists of students who have
parental consent. This latter option would likely have strong limitations. It may be unethical to place time
and resource burdens on already under-resourced government schools, particularly schools that are
receiving no intervention benefits from BRAC. Furthermore, sharing information and requiring consent in
written form would likely be highly inaccessible for illiterate parents and would likely lead to strong
sampling biases (i.e. students permitted to participate in the evaluation would be more likely to come
from more educated and literate households). A more realistic alternative could be to seek consent only at
the school level, from girls and teachers, assuming teachers can act as appropriate guardians.
Much like when obtaining consent, ensuring children‟s privacy can sometimes be more complex than for
adults. Although children‟s privacy should be encouraged, there are limitations to this that the external
evaluator and enumerators need to be aware of. At the school level, as noted in section 5.3.1, finding a
private setting may be challenging, particularly in smaller schools, such as BRAC CBE schools, where a
classroom may be the only available space and may be occupied with other students. External evaluator
44
experiences in GEC-1 suggest that students and even teachers may be curious about surveys and
interviews and may try to observe or interfere (for instance, by helping students when conducting literacy
and numeracy tests). At the household or community level, parents or guardians may insist on being
present during children‟s interviews, and it is within their right to do so.Having others present during an
interview may influence how children respond to some questions. It may also place children at risk when
asking sensitive questions,for instance, questions about children‟s experience of violence at school or in
the community (i.e. asking a student about school corporal punishment when a teacher is present may lead
to further corporal punishment being enacted against the child in the future). Enumerators spending long
periods of time alone with children in private interview settings, in both school and household settings,
may also lead to concerns about child safety, particularly among girls in a cultural context such as
Afghanistan where honour and chastity and concerns about sexual violence are prominent.
The potential scenarios outlined above raise a number of concerns about both data quality and integrity,
and possible risks to child safety. Safety to children should always take precedence; however, often both
data quality and child safety can be balanced given the proper preparation and considerations. For
instance, enumerators should be trained in how to ensure appropriate privacy and confidentiality of
children. The external evaluator should be mindful of possible risks when designing tools, and if
necessary minimize highly sensitive questions. The external evaluator may also consider including
questions in data collection tools about who was or how many people were present during the interview in
order to contextualize results. Ensuring that all girls are interviewed only by female enumerators is an
important way to mitigate risks related to concerns about girls‟ safety and honour; however, it is not
necessarily sufficient as female enumerators may also act disrespectfully, violently or inappropriately
with girls. It may be necessary for enumerators to shift traditional notions of privacy from physical
privacy to auditory privacy, and attempt to conduct interviews out of earshot of others but where the
interview can always be observed and seen by others.
One anticipated challenge in BRAC‟s GEC-T evaluation is how to manage child protection when
transferring sampling from the school to the household level (only relevant for girls in the evaluation
intervention group, not the control group which will not include household-based sampling). According to
the proposed sampling procedures, girls living in CBE communities will be sampled first at the school
level (either in a CBE school, government school or TVET centre) and then enumerators teams will travel
to girls‟ households to interview household members. The ways in which enumerators‟ movement
between schools and households will be facilitated could have some ethical consequences for children if
they are traveling with enumerators between the school and household. If girls are attending CBE schools
in their own community, it is possible for enumerators to enlist the help of teachers, BRAC staff and
community leaders to help find girls‟ households and minimize or eliminate enumerator time spent alone
with girls. However, this process is much more complex for those girls attending a government hub
school located far away from their community. The external evaluator should design an approach to
45
overcoming this challenge, which may include collecting girls‟ information (name, age, father‟s name,
mother‟s name, phone number, and household address or information about any landmarks to assist in
finding households), then traveling alone to communities on a separate day and enlisting the help of
community leaders and BRAC staff in findings girls‟ households. Any approach proposed by the external
evaluator should be both realistic and ethical, and should take into consideration potential time and
budgeting implications.
Dissemination of results
Dissemination of results to children should be inclusive and child-friendly. Different categories of
children, including younger and older children, literate and illiterate children, abled and disabled children,
married and non-married children, and poor and less poor children, may require different dissemination
practices and mediums. BRAC may not have the financial and human resources to target different
dissemination activities to multiple categories of children (as well as adults as outlined in section 5.3.1);
however, BRAC will attempt to balance any requirements based on child diversity with overall project
requirements based on budgeting and resources. BRAC will also ensure that the dissemination of results
will be child friendly. This may require avoiding written outputs given literacy rates among some girls
may be limited, and should lean towards verbal presentation of findings in creative and interactive ways
(e.g. through role play). BRAC may also consider linking dissemination with, and thus strengthening,
project activities, for instance: supporting teachers to disseminate research results with their students
through the use of classroom techniques included in BRAC teacher training (e.g. role play, group
discussion); or supporting mentors and mentees in government hub schools to incorporate the
dissemination of key research findings into extra-curricular activities such as public speaking or debating.
6. Sampling framework
BRAC‟s sampling framework is included as Annex C to the MEL framework and includes information on
sampling clusters (treatment schools and communities), matched control schools and communities, and
how target groups are represented within clusters.
6.1 Target groups
In Afghanistan, girls are marginalized from educational outcomes due to a range of factors, including
their location (a large proportion of the country‟s population live in remote or rural locations), poverty,
conservative attitudes about girls‟ education, and conflict- and disaster-induced displacement. BRAC
identifies three levels of marginalization among its girl beneficiaries: (1) girls are marginalized due to
contextual, economic and social factors, including distance from schools, poverty, and conservative
communities that discourage girls‟ education, (2) girls are marginalized due to poverty and embedded
gender norms, live in remote areas affected by conflict and are likely to face harassment traveling to
schools, and (3) girls are highly marginalized and live in the most remote and conflict-affected areas and
46
face the highest levels of poverty and gender discrimination. According to BRAC program learning,
approximately 60% of its beneficiaries face level one marginalization, 13% face level two marginalization
and 27% face level three marginalization. Across all three levels of marginalization, there are beneficiary
girls in additional categories of marginalization, including girls with disabilities, girls who are orphaned,
girls from ethnic minorities, and girls who experience violence or abuse, who face additional challenges
such as lack of mobility, discrimination and social stigma. Furthermore, once girls enter adolescence, they
often become increasingly marginalized from educational opportunities due to concerns about honour and
chastity, and family preferences for marriage or work over education, frequently restricting their
transition to secondary school. All BRAC girls will fit into this category of marginalization.
Consequently, BRAC defines its primary marginalization group as girls living in remote, rural or un-
served urban areas. Sub-categories of marginalization include:
Girls living in extreme poverty
Girls with disabilities
Orphaned girls
Girls who experience violence or abuse
BRAC is aware that some girls face multiple layers of marginalization and is currently undertaking
analysis of data on its target beneficiaries to identify those girls who fall into marginalisation subgroups.
As noted elsewhere in the MEL framework, there are three key treatment groups in GEC-T: (1) girls
enrolled in CBE schools, (2) girls who completed BRAC GEC-1 CBE education and transitioned to
government hub schools, and (3) girls who dropped out of BRAC GEC-1 CBE schools (or did not
transition to hub schools) enrolling in TVET. All girls across the three treatment groups can be classified
as marginalized according to the primary category of marginalisation (living in remote, rural or un-served
areas); however, treatment groups 2 and 3 may experience heightened barriers to learning and transition
due to the need to travel to hub schools or TVET centres (the latter located in district centres).Girls
belonging to the sub-categories of marginalization will be found in all three target groups; however, these
marginalized girls may also face distinct barriers to learning and transition across the key target groups.
Poor, disabled, orphaned and adolescent girls may face particular barriers due to traveling distance and
lack of mobility (for physically disabled girls), lack of safe, accessible and affordable transport (for poor,
disabled and orphaned girls), requirements for child labour (for poor girls), and family concerns about
honour and chastity (for adolescent girls).
Under each treatment group, it is possible to define target groups that will benefit from specific BRAC
intervention activities. These are listed below.
Treatment group 1:
47
Target group 1.1: Disabled girls receiving special physical needs assistance (e.g. ramps, seating
and large text books)
Target group 1.2: Girls who have experienced violence or abuse
Target group 1.3: All other in-school girls
Treatment group 2:
Target group 2.1: Poor, disabled and orphaned girls receiving stipends
Target group 2.2: Girls receiving mentoring and participating in co-curricular activities
Target group 2.3: Poor, disabled and orphaned girls receiving transportation assistance from
school organizers (khalas)
Target group 2.4: All other in-school girls
Treatment group 3:
Target group 3.1: Girls receiving transportation incentives/stipends
Target group 3.2: All other girls in TVET
A community mapping exercise has not been completed at the time of drafting the MEL framework.
However, BRAC does not expect any significant sampling bias for transition of girls enrolled in CBE
schoolsas they are all located in girls‟ communities. There may, however, be a possible sampling bias for
girls in GEC-1 CBE communities enrolled in government hub schools as the distance of CBE
communities from hub schools varies. Distance in Km between girls‟ communities and hub schools, and
the location of government hub schools (rural, semi-urban and urban), are included in hub school lists,
allowing for a stratified sampling of schools. Data is not currently available for distance between GEC-1
CBE communities and district centres for TVET girls, and this will be difficult to attain prior to baseline
data collection given that girls will be traveling from a large number of catchment areas to district centres.
However, BRAC will provide general information on average distances, allowing for stratified sampling
that will include, at the province level, a balance of remote and less remote district-level TVET centres.
6.2 Control groups / Counterfactual scenario
The research design is two pronged, including a pre/post design for TVET girls, and a quasi-experimental
design for CBE and hub school girls. One comparison group consisting of non-intervention government
school girls will serve as the control group for both CBE and government hub school girls receiving
interventions (see Table 8). The reason for omitting a control group for TVET girls is due to the two
primary outcomes for TVET girls being specific to the intervention and linked to one another. TVET girls
are not receiving any literacy and numeracy training and so will not conduct learning tests, but they will
sit learning tests related to vocational skills development and knowledge. Transition for TVET girls is
also quite specific as employment is not necessarily a realistic transition pathway given BRAC‟s
48
programming does not directly assist girls to find employment. Consequently, one transition pathway is
the acquisition of vocational skills and knowledge, with the understanding that these skills may lead to
girls to find safe and fairly paid employment when they reach an acceptable age for employment.
Controlling for vocational skills and knowledge is not realistic and thus a pre-post test approach is more
appropriate.
Table 8: Treatment and corresponding control groups
Type of intervention Intervention cohort Control cohort
CBE education Girls enrolled in BRAC CBE classes
Girls enrolled in government schools NOT receiving
BRAC interventions Government hub
school education
Girls who have transitioned from BRAC CBE
classes in GEC-1 to government hub schools
TVET education Girls who have dropped out of BRAC CBE
classes in GEC-1 and enrolled in BRAC
vocational training
No control group (pre-post test design)
The comparison group will be determined through the identification of non-intervention government
schools matched to BRAC intervention CBE and hub school communities. BRAC will consult with PED
and DED officials to identify appropriate matched schools. Matching will be done primarily at the district
level to increase the likelihood of cultural, ethnic and linguistic homogeneity. Data for additional
matching criteria may be difficult to obtain at the community level (such as number of households and
population size); however, key data will be accessible at the school level, including rural/urban location,
gender composition (mixed-gender or girl-only), and whether existing interventions (non-BRAC) are
currently being implemented. Attempts will be made to match control schools where no existing
education intervention is taking place in order to minimize contamination of the control sample.
Access to government control schools will be facilitated through permissions with the corresponding
PEDs and DEDs. Although BRAC does not anticipate any challenges in obtaining PED and DED
permissions, it is possible that school staff may be resistant to having data collected in schools due to lack
of knowledge of BRAC and due to schools not currently receiving any BRAC interventions. Given that
BRAC staff do not work in control schools, their ability to facilitate independent evaluator access will be
more limited than for treatment schools. BRAC will coordinate closely with government officials to
facilitate independent evaluator access, including formal introductions to government school staff. At the
community level, there may be some challenges in collecting household data to measuretransition.
Community members may be suspicious of unknown outsiders entering the community, particularly when
asking questions about girls in the absence of any intervention. Thus, BRAC will seek additional
assistance from the PED and DED in contacting community shuras, CDCs or community leaders in order
to facilitate community access.
49
Additionality will be estimated through the difference-in-difference approach, with change occurring in
the comparison group providing the counterfactual scenario to CBE and hub school interventions. The
first assumption of a difference-in-difference approach, „the outcome trajectories of the intervention and
comparison groups would be parallel in the absence of the intervention‟, will be partially tested by
populating the sampling framework with secondary data on geographical and educational characteristics.
As noted above, it will likely be difficult to obtain more complex secondary data such as population and
demographic data; however, such data can be collected ex-post throughout the evaluation waves in order
to verify the first assumption. The second assumption of a difference-in-difference approach, „all
individuals in the target group are reached by the same intervention or interventions at the same time and
no individual of the comparison group received any of the interventions‟, will be partially verified ex-post
by analyzing treatment and control group exposure to interventions. Further, as noted above, selection of
control schools and communities will be based on a prior mapping of any non-BRAC interventions being
implemented, with updated mapping of possible contaminating interventions being conducted throughout
the evaluation waves.
6.3 Cohort tracking
This section provides information on how girls in treatment groups and the control group should be
tracked longitudinally for both learning and transition. All girls in learning cohorts (CBE and government
hub school treatment groups and government control group) should be tracked at the school for the entire
length of the project and should be replaced if unsuccessfully tracked at any evaluation point. In contrast,
the transition samples for all treatment and control groups need to be tracked at the household level.
Furthermore, in order to successfully track transition over time, CBE, government hub schools girls and
control government school girls must not be replaced if they have been found to have dropped out of
school (i.e. when attempting to track learning cohorts at the school level). Consequently, all CBE and
government hub schools girls, and corresponding government control group girls sampled for the learning
and transition cohorts at baseline must be followed up throughout the project until endline. They should
not be replaced at midline or endline, as this would jeopardise the project‟s ability to learn about
unsuccessful transition pathways and thus remaining barriers to transition. For TVET girls, tracking
transition over time will differ. TVET girls will only be tracked for one evaluation wave. They will be
sampled at baseline and tracked and surveyed at the first midline evaluation point. At the second midline
evaluation point, a new cohort of TVET girls will be sampled and then tracked and followed up on at
endline.
6.3.1 Learning cohort
The learning cohort of CBE girls and government school girls will be sampled and tracked at the school
level. Random sampling will be conducted through BRAC enrollment lists in order to ensure that girls are
BRAC beneficiaries. These girls will be tracked through the design of a comprehensive tracking system,
50
with information about girls‟ names, age, grade, father‟sname,mother‟s names, name of community in
which she lives, and location of her household in the community (if known). Community and location of
household will facilitate visits to girls‟ household to conduct household surveys (see section on transition
below). This tracking system will be linked to BRAC‟s programming tracking system through a unique
identifier number. This will enable effective school-based tracking at midline and endline waves, as
crosschecks with BRAC‟s tracking system will provide information on girls‟ grade and enrollment status.
BRAC does not anticipate having any problems tracking girls moving from primary to secondary school
grade at the CBE level, as these schools will be in the same location. At the government hub school level,
tracking girls who are moving between primary and secondary may be challenging in cases where
primary and secondary, or lower-secondary and upper-secondary, schools are in different locations.
According to BRAC‟s field assessment of government hub schools, only three schools have education up
to 6th grade only, suggesting that tracking learning samples between primary and secondary school
transition will not be a large challenge. A total of 81 secondary schools in the hub school sample provide
education up until grade nine only (all remaining beneficiary hub schools provide education until 12th
grade). Presumably, when girls graduate from schools that do not provide 12th grade education, they will
have the opportunity to transition to lower-secondary or upper-secondary schools in accessible locations
in the same communities or districts. These types of shifts between different schools should be trackable
through the BRAC programming tracking system, and BRAC will share details about the location of
corresponding secondary schools.
The process for tracking the learning cohort (for both CBE and government hub school girls) will thus
involve the independent evaluator: (1) reviewing cohort girls listed in BRAC‟s tracking system; (2)
identifying whether girls have failed/repeated a grade or whether they have officially dropped out; (3)
approaching the school and providing cohort lists to school administrations who will assist in identifying
girls‟ classes (in government hub schools); (4) and sampling girls.
Although the linkage of the learning cohort tracking system to BRAC‟s tracking system (through unique
identifiers) is feasible for treatment groups, this will clearly not be possible for the government school
control group. Theoretically, the processes outlined in the previous paragraphs would still apply to control
groups, minus the linkage to BRAC‟s program tracking system. Girls will thus be tracked through the
design of a comprehensive tracking system, with information about girls‟ names, age, grade, father‟s
name and mother‟s names, as for treatment groups. They will then be tracked at midline and endline
follow up evaluation points by drawing from school assistance in identifying girls‟ classes and sampling
girls. Control girls may move from primary to lower-secondary or lower-secondary to upper-secondary
schools between evaluation waves. There are a number of ways in which BRAC and the independent
evaluator can handle this situation. First, when completing the sampling framework before baseline,
attempts should be made to record the names of other schools in the area to which girls may transition.
Second, enumerators will approach control schools sampled in the previous evaluation wave and request
51
assistance in identifying if tracked girls have transitioned to other schools (government schools should
record this information).
Girls participating in TVET training will not be tested for literacy and numeracy as their vocational
courses will not involve any learning related to literacy and numeracy. Hence, we would not expect to see
any significant improvements in reading and maths over time within this cohort. Instead, they will be
tested for vocational skills acquisition. As noted previously, TVET girls will be sampled in two separate
cohorts with each cohort only being followed up for one evaluation wave. Although the first learning data
collection wave for each cohort (baseline or second midline) will be implemented at the school level (i.e.
TVET centres), follow up (first midline or endline) will be done at the household level as TVET girls will
have completed their vocational training. In the first learning data collection wave, information for a
tracking system will be obtained, including girls‟ name and age, father‟s and mother‟s name, community
name, household location/address and telephone number (if available). Enumerators will then go to girls‟
community to obtain tracking data (see below). At follow up evaluation waves, girls in both the learning
and transition cohorts will be tracked at the household level.
6.3.2 Transition cohort
For CBE and government school girls, and government school control girls, at baseline the transition
cohort will be the same as the learning cohort. After conducting learning tests at the school level and
collecting girl details (including community name and household location, if the latter is known), on a
separate day enumerators will go to girls‟ communities (if different to the community in which the school
is located, which will be the case for girls attending government hub school) and track girls‟ households.
Given that girls‟ reports of household location may not be sufficient to find households, BRAC staff and
community leaders and shuras will assist enumerators to find girls. Enlisting the help of community
leaders and shuras will be particularly important in government control schools, where BRAC staff will
be unfamiliar with the community and will be unlikely to be able to assist.
At the first midline follow-up, CBE, government hub school and government control transition cohorts
will be tracked at the household level. Enumerators will first attempt to track girls at the school level for
learning tests. All girls sampled at baseline will then be tracked at the household level as part of the
transition sample, regardless of whether they were present in school on the day of the test or not, or
whether they have dropped out. In some cases, making this distinction will not be clear as girls may not
have formally dropped out but may not have attended school for many days or months. If girls who are
absent on the day of learning tests are tracked at the household level and found to be still enrolled in
school, learning tests should be administered to them at the household level. At second midline and
endline, girls in the transition sample will be tracked directly at the household level with no visits to
schools required.
52
For TVET girls, at baseline and second midline, transition cohorts will be sampled at the household level
from the same cohorts sampled at the school level for learning. Tracking from the school to the
community will take place according to the same processes outlined above for CBE and government
school girls, i.e. collecting information for the tracking system and enlisting the help of BRAC field staff
and community leaders and shuras to find girls‟ households. At first midline and endline (the two follow
up points for the two different cohorts of TVET girls sampled at baseline and second midline), TVET
girls will be directly sampled at the household level according to the same standardized procedures
outlined in this section for other cohorts.
Learning and transition cohorts will be matched through unique identifiers attributed to the same girls in
the cohort tracking system. However, it should be noted that any girls in the learning cohort that have
been replaced at midline or endline will not be included in the original transition cohort and thus cannot
be matched to this cohort. Throughout evaluation waves, attrition of girls from the learning cohort due to
drop out or other reasons is expected, as is attrition of the transition cohort. Consequently, as the project
progresses, we would expect to see a reduction in the collection of data from learning and transition girls
matched at baseline.
6.3.3 Replacement strategy
The independent evaluator will design a detailed replacement strategy for girls who cannot be sampled at
follow up evaluation points. These strategies will be different for different cohorts of girls. A
recommended approach is outlined below.
Learning cohort CBE, government hub school, and control government school girls will all be replaced if
unable to be tracked at follow up evaluation points. Due to original sampling of learning cohorts in the
treatment groups happening directly from BRAC enrolment lists, any replacements will be done by
conducting a random sample of the same class lists, including only beneficiary girls who are enrolled in a
corresponding grade to the girl who has been replaced. This would be done prior to post-baseline
evaluation data collection waves with assistance from BRAC so that enumerators are prepared with
possible girl replacements prior to data collection. In the case of government school control girls, this
process of random selection of enrollment lists will need to happen at the school level on the day/s of data
collection. Tracking data must be collected for all replaced girls.
As noted previously, if girls are not in school on the day of learning tests and are still enrolled, household
survey data collection will reveal this when enumerators collect transition data. In these cases, girls
should be administered learning tests at the household level. This means that in some cases, although
likely few, we will have a substitute girl at the school level but with the girl who has been replaced
retained (as they are still enrolled in school). In such cases, substitute girls should be eliminated from the
sample.
53
Transition cohort CBE, government hub school and control government school girls will not be replaced
and will be followed up across all evaluation waves. TVET girls will not be replaced as they will only be
tracked for one evaluation wave.
Due to unpredictable security across many of the provinces in which BRAC works, it is possible that
schools and communities will have to be replaced, for instance, if BRAC schools are closed or if
enumerators cannot securely access schools or communities. In such cases, whole communities will need
to be replaced. This will be done by randomly selecting a new cluster (school or community) from the
same province, with additional stratification to maintain a similar cluster composition as the cluster that
was replaced.
6.4 Power calculations and sample sizes
All sample size calculations have been done assuming a national level minimal detectable effect (MDE)
of 0.25 SD for learning and 10% for transition. The MDE would need to be higher for province-level
estimates. The below parameters are currently used for treatment 1 and treatment 2 for literacy and
numeracy outcomes and transition outcomes (the significance level and power defined are the default
values and therefore are not listed in the lines of code in this section).
Literacy and numeracy
Minimum detectable effect (MDE) = 0.25 standard deviations
Sig level = 5%
Power = 80%
Attrition = 30%
Intra-cluster correlation (ICC) = 0.1
Transition
Minimum detectable effect (MDE) = 10%
Sig level = 5%
Power = 80%
Attrition = 30%
Intra-cluster correlation (ICC) = 0.1
Power calculations and sample sizes were conducted using STATA 13. To assist with determining sample
sizes, the add-on Stata command “clustersampsi” was installed. Options within this procedure which are
used in this analysis are as follows:
clustersampsi, <options>
rho = intraclass correlation coefficient
alpha = the significance level of the test; default is alpha(0.05)
beta = the power of the test; default is beta(0.8)
54
k = number of clusters in each arm
m = average cluster size
mu1 = mean for sample 1
mu2 = mean for sample 2
sd1 = standard deviation of sample 1; default is sd1(1)
sd2 = standard deviation of sample 2; default is sd2(1)
Clusters, also referred to as primary sampling units (PSUs), are used in the analysis. At the CBE level,
the PSU is a community given that some communities have more than one CBE class and more than one
grade. At the government school level, the PSU is a school. At the TVET level, the PSU is a TVET
centre. A two-stage cluster sampling technique is used to first choose the PSU and then the sampling units
within the PSU. Sampling units within a PSU are the individual girls who will be assessed for learning
and transition.
Control group 1 is the same for treatment 1 and treatment 2. This is similar to a three-armed study, as it
will be of interest to the project to examine change within the two treatment groups against change in the
control group, but also examine any differences in change between CBE and hub school treatment groups.
Hence, it is recommended to use the same number of clusters in each arm; however, the size of the
control group may be reduced after analysing the control schools and communities in the sampling
framework (see further below).
Literacy and numeracy
The base-cases (see Table 9) have been run using the specified parameters and estimated clusters for an
initial view of the command. A sample of 20 clusters in this case is not allowed, and a number of clusters
below 40 is not feasible from a budget perspective due to large corresponding sample sizes, and an
insufficient mean number of students in CBE clusters to achieve the target sample (i.e. there is a mean of
35 students in each CBE cluster, and a mean of 27 students in each class).Given that clusters will be
further stratified at several levels, including province (10 provinces), district (average of four districts per
province for both CBE and hub schools) and grade (grades 5 and 6), it is recommended to select 70
clusters per group to allow for more representative sampling while retaining the lowest possible sample.
This would allow for a sample size of 490 girls doing learning tests in CBE (treatment group 1) and 490
in government hub schools (treatment group 2) after attrition buffers are applied (see Table 10).
Table 9: Base-case ‘clustersampsi’
clustersampsi, Number of
clusters/PSUs
Cluster/PSU
size
Sample
Size
k(20) rho(0.1) mu1(0) mu2(0.25) Infeasible with cluster randomisation
k(27) rho(0.1) mu1(0) mu2(0.25) 27 256 6,912
k(30) rho(0.1) mu1(0) mu2(0.25) 30 59 1,770
55
k(40) rho(0.1) mu1(0) mu2(0.25) 40 17 680
k(50) rho(0.1) mu1(0) mu2(0.25) 50 10 500
k(60) rho(0.1) mu1(0) mu2(0.25) 60 7 420
k(70) rho(0.1) mu1(0) mu2(0.25) 70 5 375
k(80) rho(0.1) mu1(0) mu2(0.25) 80 5 400
Table 10: Estimated sample with 30% attrition buffer
Number of
clusters
Cluster size
with attrition
(rounded)
Sample size
with attrition
27 333 8991
30 77 2310
40 22 880
50 13 650
60 9 540
70 7 490
80 7 560
A list of control government schools was not available at the time of developing the MEL framework, so
it is not possible to examine the number of available clusters, their location (province and district) and the
population of students. Consequently, it is recommended to sample the same number of clusters for
control government schools as for treatment clusters, and then the feasibility of number of clusters and
sampling sizes can be verified when control cluster lists are available. This would mean that 70 CBE
clusters, 70 government hub clusters and 70 control clusters would be sampled, with seven girls per
cluster receiving learning tests.
Transition
The current sample size of learning outcomes will satisfy the desired parameters for transition outcomes
at the national level, for both CBE and government treatment groups, where the mean size of CBE
clusters is 35 and the mean size of government hub clusters is 119 (although note that the figure 119 is the
mean number of GEC-1 CBE girls per cluster who should be transferred to the hub school, not the actual
mean enrolment number, which is not available). This is inclusive of 30% attrition. For both CBE and
hub school treatment groups, the learning outcome sample sizes will detect a difference of 10percent (the
MDE for transition). The first line of STATA code below is for CBE and the second is for hub schools.
The minimum sample will more than suffice to statistically cover the required precision at the national
level.
clustersampsi, binomial detectabledifference k(70) m(27) rho(0.1) p1(0.7)
clustersampsi, binomialdetectabledifference k(70) m(119) rho(0.1) p1(0.7)
TVET girls
56
Developing sample estimates for treatment 3 (TVET girls) through power calculations is not advisable
given that TVET girls will not have a MDE applied for either learning or transition. Given there are 40
clusters (TVET centres), four per province (at the district level), it is feasible to sample 20 clusters, two
per province. It is recommended to sample one central and one more remote district in order to capture
barriers and impact of BRAC activities in different kinds of locations. It is difficult to estimate how many
girls should be sampled in each cluster until more information is provided about how many girls will
enrol in each cluster; however, a minimal sample of 10 girls per cluster will be established until more
information is available.
Proposed sample
Table 11 contains details of the baseline sample for three treatment groups and one control group, with
the same sample to be followed up on at both midlines and endline. Table 12 contains further details of
the sample required for each type of primary data collection tool.
Table 11: Estimated total sample
Clusters in
Population
Sampled clusters Sampled units/PSU
(girls/school)
Sample Size
(# of girls)
Treatment 1 204 70 7 490
Treatment 2 300 70 7 490
Treatment 3 40 20 10 200
Control 1 Unknown 70 7 490
Total > 544 230 1670
Table 12: Sample of primary data collection types
Learning tests Household
survey
Attendance Classroom
observations
Treatment 1 490 490 70 70
Treatment 2 490 490 70 70
Treatment 3 N/A 200 N/A 20
Control 1 490 490 N/A N/A
Total 1470 1670 140 160
Qualitative sample
In GEC-T, qualitative data will be collected in all 10 provinces in one community per treatment group per
province, except where certain types of FGDs or IDIs are not required (for instance, mentor and mentee
FGDs are only required in hub schools) (see Table 13 and 14). A total of 120 FGDs and 65 IDIs will be
done at baseline and subsequent follow up evaluation waves.
57
Table 13: FGD qualitative sample
FGD
mothers
FGD
fathers
FGD SMSs FGD girls FGD
mentor and
mentees
Treatment 1 10 10 10 10 ---
Treatment 2 10 10 10 10 10
Treatment 3 10 10 --- 10 ---
Total 30 30 20 30 10
Table 14: IDI qualitative sample
IDI teachers IDI Stipend
recipients
PED official DED
official
Khala
Treatment 1 10 ---
10 10
---
Treatment 2 10 5 5
Treatment 3 10 5 ---
Total 30 10 10 10 5
6.5 Benchmarking
Learning
Learning benchmarking for CBE and government hub school girls will take place in BRAC government
hub schoolsincluded in the baseline sample. Benchmarking will take place with girls in grades 7, 8, 9, 10,
11 and 12 (grade six girls don‟t have to be benchmarked separately as they already comprise a treatment
group that will receive learning tests). Learning will be benchmarked for 50% of clusters sampled at
baseline in BRAC government hub schools. Two girls from each benchmark grade in each cluster will
conduct SeGRA and SeGMA learning tests, comprising a total benchmark sample of 480 girls (2*6*40).
Girls in each grade will be randomly selected from class enrolment lists. As girls included in
benchmarking will not comprise BRAC beneficiaries, school enrolment lists rather than BRAC cohort
lists will be used for sampling. Targets will be calculated for treatment cohorts based on a 0.25 standard
deviation increase in learning scores above the benchmark for the corresponding grade.
Transition
Transition will be benchmarked at the household level in communities where BRAC beneficiaries live. It
should be noted that all BRAC girl beneficiaries live in CBE communities. These are communities in
which CBE classes were located in GEC-1, some classes that have continued and others that have closed.
For those CBE classes that have closed, some beneficiaries are traveling to attend government hub
schools and others are attending TVET centres. However, transition benchmarking will not take place in
communities in which government hub schools are located, nor within communities in which TVET girls
58
reside. In the first case, this is because the girls in government school communities are not theoretically
receiving direct BRAC interventions and are thus not comparable to girls living in CBE communities.In
the second case, given that TVET transition is primarily defined as learning vocational skills,
benchmarking is not required.
The benchmarking sample of CBE communities in which CBE schools are active will be 50% of baseline
learning and transition sampling (n=40). Determining the total sample of CBE communities in which girls
are living but attending government hub schools is difficult given that each government hub school may
have taken over girls from multiple different CBE communities in a catchment area. According to BRAC
data, each treatment government hub schools has received transitioned girls from a mean of 5.36 CBE
communities. Given this complexity, the number of these communities to be sampled could be up to 80*5
(400), although this is not possible to establish prior to baseline data collection. It is recommended that
transition benchmarking take place in 40 communities in which government school treatment girls
sampled for the baseline reside. The process for selecting these communities will be managed by the
external evaluator, but will likely involve a combination of sampling strategies (e.g. stratified and random
sampling of communities) that will be possible once BRAC has more information on the communities in
which hub school girls reside.
Hence, transition benchmarking will take place in 80 communities. Transition benchmarking will take
place among 10- to 20-year-old girls/young women, with 5 girls sampled per community (total of 400
girls). All girls/women sampled for benchmarking must not be enrolled in a BRAC school intervention
(they cannot be attending a CBE class or grades 5 or 6 in a government hub school that BRAC is
supporting). The benchmarking exercise will involve administering a very small survey that will include
demographic questions on a household girl, and her current enrollment status, past year enrollment status,
and type of school and grade (if the girl is in school). BRAC recommends also including questions related
to employment and quality of employment, with details of age-appropriate work transition described
previously in section 5.2. It is envisaged that the transition benchmarking would only take approximately
10 minutes to conduct per household.
Sampling at the household level for transition benchmarking will be random (i.e. random walk), with a
screening strategy developed to (1) identify if there is any household girl not enrolled in a BRAC
intervention, and (2) if there are any girls in the household aged 10 to 20 (the estimated age range for
BRAC beneficiaries across the eight years of the project). Households must meet both criteria to be
sampled, with an additional random sample of eligible girls then taking place (e.g. through a kish grid).
BRAC anticipates some challenges with this process. In communities in which CBE classes are active,
screening should be straightforward by asking if any girls are enrolled in the local CBE class (under the
assumption that BRAC is not engaging in any project activities in these communities to transition girls to
government hub schools). For communities in which previous CBE girls have transitioned to government
hub schools, the process is slightly more complex as there may be some girls attending the government
59
school who are not direct beneficiaries (i.e. in grades 5 or 6). Furthermore, some parents may not know
whether BRAC is supporting the government school where girls are enrolled (for instance, there may be
more than one hub school in the area). Consequently, screening at the household level needs to occur
according to the same screening criteria listed above; however, additional information might need to be
shared with household members, including the name of the hub school that BRAC supports.
Total sample for benchmarking
The total benchmarking samples for learning and transition are listed in Table 15.
Table 15: Learning and transition benchmarking samples
Type of cluster Sampled clusters Sampled units per
cluster
Sample Size
Learning School (Gov hub) 40 12 480
Transition Community (CBE) 80 5 400
Total 120 880
7. Baseline study
Information about requirements for the baseline study and how these differ to requirements for midline
and endline evaluation waves is presented throughout the MEL framework. However, for clarity, key
requirements and expectations are briefly listed here.
The baseline will include piloting of all tools and benchmarking of learning scores and transition rates;
however, these processes are not expected to occur in future evaluation waves. One exception for piloting
is if completely new tools are designed in future evaluation waves, or if tools used at baseline require
substantial modification (for instance learning tools).
At baseline, all treatment CBE and hub school girls, and control government school girls, will be in
grades 5 and 6 grade. At future evaluation waves, cohort girls will be tracked at their corresponding
higher grade levels, except in cases where cohort girls have repeated a grade, in which case they will be
tracked at their current grade level rather than the expected grade level.
New cohorts of girls will be sampled at baseline (the evaluation will not track cohorts from GEC-1) and
these must be tracked across the project. New cohort girls should only be selected in future evaluation
waves for learning cohorts in the case where girls cannot be found or followed up on from previous
evaluation waves, or where whole communities must be substituted and completely new learning and
transition cohorts must be selected. One exception is for TVET girls, who will be sampled in two groups:
baseline cohort (tracked only until first midline), and a newly selected cohort at second midline (tracked
until endline).
60
Cohorts of girls selected at baseline will be tracked throughout the eight years of the project (for treatment
CBE and government hub school girls, and control government school girls), and learning and transition
cohorts will be matched. However, throughout evaluation waves, attrition of girls from the learning
cohort due to drop out or other reasons is expected, as is attrition of the transition cohort. Consequently,
as the project progresses, BRAC expects an overall reduction in the amount of data that can be collected
from the same learning and transition cohort girls matched at baseline.
8. Evaluation governance
8.1 Evaluation steering group
BRAC‟s Research and Evaluation Unit has primary oversight of the evaluation and is responsible for
ensuring that all evaluation waves are completed at a high quality. This will involve review of all
evaluation outputs, including any inception outputs, research and tool design, datasets (including
quantitative and qualitative datasets), and final reports. BRAC‟s Monitoring Department will also
contribute to independent monitoring of all fieldwork, enlisting the assistance of the provincial office
heads and monitoring officers to conduct random monitoring of fieldwork.
A Research Steering Committee will be formed for providing policy guidance and expert advice on major
M&E activities. The Committee will be formed by BRAC GEC-T Project Manager, GEC M&E Focal
person (BRAC AFG Research Unit), Representative from External Evaluator, Representative from GEC
project co-implementing organizations/M&E Focal point and an Education Research Expert from BRAC
International HQ. The committee will be chaired by BRAC AFG Country Representative.
The committee meeting will be held once in a year. The committee will review M&E Plan, take account
of M&E activities, and endorse all research reports.
8.2 External evaluator
BRAC will engage in a competitive tender for independent evaluators. A ToR for the tender is included in
Annex D of the MEL framework. The ToR will be announced publically and additionally shared with
known research organisations in Afghanistan, including the GEC-1 independent evaluator. Interested
organizations will be requested to submit technical and financial proposals to BRAC. A committee,
consisting of the GEC-T project manager, the head of the Research and Evaluation Unit, the procurement
and accounts department, under the guidance of the Country Representative, will review all proposals on
the basis of BRAC‟s selection criteria and due diligence processes. These criteria include: capacity of the
organization to plan and conduct the evaluation, fulfil scope of work requirements, relevant experience
and expertise, approach to planning and implementing the evaluation, risk management strategies, quality
61
assurance, research ethics, and proposed budget. Each proposal will be assessed against the criteria and a
composite score allocated. The highest technical scoring and best value for money evaluator will be pre-
selected by the committee. The BRAC country representative will interview the evaluator, and BRAC‟s
team will meet the principal researcher in order to gauge the organisation‟s suitability in undertaking a
complex quasi-experimental evaluation and its experience in both qualitative and quantitative research
design and data analysis (i.e. using SPSS/STATA and Nvivo software). Once a selection is made for the
baseline evaluation, BRAC reserves the right to retain the same evaluator in future evaluation waves
subject to high quality outputs, or re-tender the evaluation in future evaluation waves.
8.3 Data validation
BRAC will require the independent evaluator to establish a rigorous monitoring system of all evaluation
processes, including training, data collection, data entry, translation, data cleaning and data analysis (see
section 9 for more information). Details of this system (including any established monitoring systems and
how these will be adjusted or enhanced for the GEC-T evaluation) will be requested in the ToR for
external evaluator proposals and will be further developed in consultation between BRAC and the
selected independent evaluator. At a minimum, BRAC expects clear monitoring strategies to be
implemented at all key phases of the evaluation, including pre-data collection, during data collection, and
post data collection. Pre-data collection must include comprehensive training of research staff. During
data collection, monitoring and governance could include unannounced monitoring field visits to data
collection sites, back checks of previously sampled communities and data collection observations. At the
data processing level, data entry and translation should be checked by the EE (e.g. through double entry
or a % of re-entry, duplicate analysis of databases, and other checking procedures). BRAC staff will also
conduct monitoring of EE data collection, data entry and data products (e.g. transcripts). BRAC will
request weekly monitoring updates during the fieldwork and data entry, and will request all syntax for
data analysis in order to ensure accurate and rigorous analytical techniques have been employed.
9. Data quality assurance
9.1 Training
Experience from GEC-1suggests that comprehensive training of research and evaluation staff is required
to successfully implement the range and complexity of monitoring and evaluation data collection tools
and approaches. The external evaluatorwill be asked to provide recommendations on requirements for
training based on the existing skills and experience of enumerators. However, BRAC has the following
expectations for training on the evaluation background, literacy and numeracydata collection, qualitative
research, sampling, and ethics and safety.
Evaluation background. All enumerators should be fully briefed on BRAC‟s intervention and the
evaluation, including a full description of all BRAC project activities, evaluation objectives and research
62
questions.
Literacy and numeracy.EGRA/EGMA and SeGRA/SeGMA training should last for a minimum of five
full days and could include the following components: review of tool subtasks; overview of possible
challenges to collecting reliable data (e.g. working in noisy and possibly disruptive settings, providing
students with too much time on timed subtasks, wanting to assist students with subtasks) and how to
mitigate these challenges; practice using the tools; role plays and inter-rater reliability exercises; and
quizzes/tests. The external evaluator should ensure that all trainers have been sufficiently trained in the
tools and issues to be covered in the enumerator training, and will be responsible for checking the
progress of enumerators and correcting any errors that arise in the training.
Qualitative research. The quality of qualitative data in Afghanistan can be limited unless proper training
procedures are implemented. Qualitative training for GEC-1 varied in length between five and seven days
depending on the number of qualitative tools to be used (up to 10 different in-depth and focus groups
interview tools). Training should include:overview of qualitative research, review of tools, practice and
role plays, and qualitative tips (e.g. probing, follow up, and establishing rapport).
Sampling, recontacting and replacement.GEC-T sampling, recontacting and replacement procedures
are complex and operate at multiple levels. These procedures should be covered in detail in enumerator
and interviewer training. BRAC also recommends that the external evaluator produce an
enumerator/interviewer guidance manual to provide clear instructions on sampling, recontacting and
replacement for intervention and control group girls, with linked instructions on how to deal with
complexities and challenges that may arise.
Ethics and safety. The external evaluator is expected to conduct ethics and safety training with all
enumerators and qualitative interviewers collecting data for the GEC-T. Ethics and safety training should
include the following topics: how to obtain informed consent, including respecting the rights of
participants to refuse to participate in any data collection procedures; how to deal with sensitive issues
and questions, and cultural sensitivities (e.g. male enumerators asking men about their female family
members); how to maintain confidentiality; how to conduct ethical research with children; how to
mitigate ethical challenges; and child safeguarding procedures, including when and how to report the
observation of a breach in child safety (e.g. when observing corporal punishment in schools, or any type
of violence perpetrated against a child).
Additional training will be required for other types of research tools not listed above (e.g. household
surveys, classroom observation tools, attendance data collection tools, etc.). The length and content of
these training components is at the discretion of the external evaluator, but should include tool review,
practice, challenges in data collection, and ways to mitigate challenges.
63
Although training will be an essential part of ensuring that enumerators and other research staff are fully
prepared to collect high quality data, it will also provide an opportunity for the external evaluator to
recognize and correct any potential challenges or problems with tools, samplingdesign and data collection
approaches not identified in the piloting (see below).
9.2 Piloting
Some tools used in GEC-1 will be reproduced in GEC-T with some minor or major modifications (e.g.
household survey, attendance checklists). Consequently, the external evaluator will be required to pilot
quantitative and qualitative evaluation tools at baseline to ensure that questioning, translations and length
of tools are functional and adequate for the research objectives. EGRA/EGMA and SeGRA/SeGMA and
any other school-based tools (e.g. attendance checklists, classroom observations, qualitative interviews
with teachers) should be piloted in BRAC intervention schools not sampled for the evaluation, with
corresponding household-based tools (e.g. household survey, qualitative interviews and focus groups with
girls and parents) being piloted at the household level. Piloting could take place in one or more of
BRAC‟s intervention schools/communities in Kabul province for ease of access; however, the language
of tools should be taken into account (i.e. if no BRAC intervention schools in Kabul have Pashto as the
primary language of instruction, piloting may also need to take place elsewhere such as Nangarhar).
The external evaluator will be responsible for identifying any errors or problems with the data collection
after debriefing with enumerators and research staff, and will make necessary adjustments in consultation
with BRAC. If the external evaluator recommends major modifications to tools, BRAC will refer these
recommendations to the FM and other GEC-T stakeholders if necessary, particularly in relation to tools
that have been mainstreamed across the GEC-T portfolio of projects (e.g. the household survey).
Piloting of tools will only need to take place before baseline data collection and will not take place before
the implementation of other evaluation waves, unless new tools are added at midline and endline waves.
In this case, all new tools should be piloted.
9.3 Data cleaning and editing
Data checking, cleaning and de-identification will primarily be the responsibility of the independent
evaluator. However, BRAC will review all quantitative and qualitative data sets, including checking for
duplicates and internal consistency, ensuring that adequate cleaning has been conducted, and checking
that data has been sufficiently de-identified. In quantitative data sets, any references to names should be
removed, and household contact information, including addresses and telephone numbers, will not be
shared with the FM or any other project stakeholders. In qualitative data sets, any identifying details of
participants should also be removed. It may be necessary to remove any sensitive information that may
cause harm to participants if shared (see further below). Decisions about if this is necessary should be
64
made by the independent evaluator (the staff member in charge of overseeing research ethics and child
safeguarding) in consultation with BRAC.
10. Risks and risk management
Table 16 outlines potential risks that may impact on BRAC‟s ability to ensure a robust and rigorous
evaluation of GEC-T, along with recommended risk mitigation strategies. Some of these risks are linked
to section 5.6 on ethics protocols.
65
Table 16: Risks and mitigations
Potential risks Probability of
risk occurring
over the course
of the project
Potential
impact on
project’s
success
Proposed actions to mitigate risks that have both significant
probability and impact/importance
Necessary permissions are
not secured
Low High BRAC has obtained permissions from the MoE to implement
GEC-T programming, including conducting project monitoring
and evaluation in government hub schools receiving BRAC
interventions. Necessary permissions must also be obtained and
maintained to conduct evaluation data collection in government
control schools. Although the probability of this risk occurring
is small due to BRAC‟s strong relationship with the MoE, if it
should occur this could jeopardise the inclusion of control
groups in the project evaluation, and thus the proposed quasi-
experimental approach. This risk will be mitigated by BRAC
securing permissions well in advance of the baseline data
collection, and ensuring timely renewals of all necessary
permissions over the course of the project.
Insecurity leads to closure
of schools in BRAC target
communities
Low Medium There was no closure of BRAC schools in GEC-1; however,
with an increasingly unstable security situation in Afghanistan,
this risk is possible. An outcome of this risk to the project
evaluation could be the loss of cohort girls from the longitudinal
study. This may lead to a sampling bias (i.e. if only girls living
in secure communities are recontacted). When producing the
sampling framework and community/school selection for the
baseline study, the EE will identify possible replacement
communities (e.g. through intra-cluster random selection). Any
resulting sampling bias will be discussed and contextualized in
evaluation reports.
Insecurity limits ability of
EE to sample target
communities
High Medium An increasingly unpredictable security context may limit
enumerator access to insecure communities, even when BRAC
schools remain open. As noted above, this may lead to the loss
of cohort girls from the longitudinal study. This risk will be
mitigated by: BRAC coordinating enumerator access to
communities; the EE identification of possible replacement
communities; and any sampling biases discussed in evaluation
reports.
The safety of evaluator
staff is compromised
Medium High Insecurity in Afghanistan poses a large risk to the safety of
evaluation staff, including evaluators, fieldwork supervisors and
field monitors. This risk can lead to the injury, kidnapping or in
extreme circumstances death of field staff. BRAC will mitigate
this risk by: facilitating enumerators‟ access to target
communities; ensuring that the EE has key security protocols for
66
data collection (including any necessary security training); and
ensuring that enumerators are recruited from the province or
district where data collection takes place.
Survey fatigue after
multiple evaluation waves
Medium Medium In GEC-1, the external evaluator reported some survey fatigue
among community members who had been sampled multiple
times over evaluation waves, potentially accounting for some
participation refusals in the endline evaluation. Such survey
fatigue may, over time, lead to the loss of cohort girls and their
households from the longitudinal study, and may impact
negatively on community relations with BRAC staff. This risk
will be mitigated by: attempting to minimize the length of
survey/interviews tools (to be tested during piloting); providing
refreshments to qualitative interview and focus group
participants; and ensuring that sampled girls and household
members fully understand that they will be recontacted in future
evaluation waves before consenting to participate in data
collection.
Data fraud High High Data fraud is an important and frequent challenge in
Afghanistan, and can occur at different levels of the evaluation
process, including during data collection, and data entry and
processing (including translation). At the field level, data fraud
may occur partly due to insecurity leading to enumerators being
frightened to access insecure areas and fabricating data, and
insecurity also limiting the ability of evaluators to monitor data
collection in the field. At the data entry and processing level,
data entry staff and translators may also fabricate data (e.g. by
copy and pasting transcripts or data entry cases) to shorten data
processing timelines, or for other reasons. Fraudulent data
would render evaluation findings invalid and could lead to
subsequent changes in project design being made based on false
information. This risk will be mitigated by ensuring that the EE
has a strong data monitoring and data governance system in
place (see section 8.3). The EE should also, to the extent
possible, recruit field staff from the provinces and districts
where data collection will take place to enhance enumerator
access.
Low capacity of research
staff
Medium High Low capacity of research staff is a common challenge in
Afghanistan and can lead to poor quality of data, subsequently
harming the quality of evaluation results. This risk will be
mitigated by BRAC carefully selecting the EE based on past
experience and existing skills, and ensuring that the EE
implements intensive enumerator training, conducts
comprehensive monitoring of data collection, and feeds
monitoring results into fieldwork practices (e.g. conducting
additional training or re-fielding data collection if necessary).
Incorrect sampling Medium Medium Sampling procedures such as random sampling are sometimes
67
procedures difficult to ensure, particularly in a complex and conflict-
affected environment such as Afghanistan where ethnic
divisions and intra-community power relations may interfere
with robust implementation of sampling protocols. A number of
challenges could arise, including: enumerators bypassing
complex sampling procedures in order to facilitate easier access
or reduce timelines; community members interfering in
sampling procedures (e.g. community leaders attempting to
influence participant recruitment); and BRAC staff interfering in
sampling procedures (e.g. teachers attempting to encourage
recruitment ofmore advanced girls in to learning tests). Incorrect
sampling procedures would likely lead to a range of possible
sampling biases. This risk will be mitigated by ensuring that:
enumerator monitoring includes verification of sampling
approaches; enumerators are sufficiently trained on sampling
protocols (including on possible challenges and ways to mitigate
them); and enumerators report any pressures to deviate from
sampling protocols to fieldwork supervisors to enable evaluator
contextualization of evaluation results (and include any relevant
discussion in evaluation reports).
11. Learning
As noted in section 2 of the MEL framework, in GEC-1 BRAC lacked a structured learning system to
ensure that M&E findings were fed back meaningfully into project design, partly due to the GEC-1 focus
on monitoring and evaluation, with less emphasis on learning feedback loops and broader GEC
learning.This section describes the structures BRAC will put in place to ensure that learning is
mainstreamed across the project in GEC-T.
11.1 Learning strategy
BRAC‟s learning strategy is based on two levels of learning: internal and external.
Internal learning
Internal learning will be primarily based on the results of monitoring and evaluation results. There are a
number of activities that were shown to be critical to the success of GEC-1. Although BRAC is interested
in learning about how the efficacy of these activities continues through GEC-T, the focus of learning will
be on how new activities impact on outcomes, intermediate outcomes and outputs, and any implications
for the theory of change. Learning issues include:
The role of TVET in contributing to girls‟ vocational and life skills
The role of mentoring and co-curricular activities in increasing girls‟ life skills
68
The efficacy of mainstreaming child safeguarding standards, auditing and policies on corporal
violence and the creation of safe learning environments
How engaging men and boys can lead to change in gender norms and support for girls‟ and
women‟s education, employment and broader participation in social and civic activities
(including school management)
The efficacy of gender-sensitive curriculum and teaching and how this may lead to change in
gender norms and teaching practices
How inclusive education activities, particularly for highly marginalized girls (e.g. disabled girls,
orphaned girls, and girls who experience violence), can improve learning and transition
Sustainable pathways in girls‟ education, particularly the efficacy of piloting sustainable CBE
models.
BRAC is also interested in learning about how effective new methodologies are in tracking project
outputs and outcomes. Such methodologies include real-time monitoring, auto-ethnography and life
history mapping, use of scales to measure life skills (Rosenberg Self-Esteem Scale and General Self-
Efficacy Scale), Active Pedagogy Inventory (API) tools, and Teacher Value-Added (TVA) modeling.
Learning issues above will be integrated into GEC-T through a range of internal mechanisms. BRAC‟s
monitoring department will engage in informal meetings with project management staff where key
learning opportunities arise. However, more structured learning mechanisms will include BRAC quarterly
meetings after the submission of quarterly reports, to review monitoring results and discuss how findings
should be fed back into project design and implementation. At the evaluation level, BRAC will request
the external evaluator to present the findings and recommendations from each evaluation wave through a
formal presentation to the BRAC GEC-T team, Research and Evaluation Unit, Monitoring department,
and other internal staff where relevant. Following these presentations, BRAC will hold an evaluation
steering committee meeting to discuss any implications for modifications in programming, and design
clear steps on how these modifications will be implemented. These evaluation meetings should also re-
examine the theory of change to interrogate whether it needs to be modified based on evidence from the
evaluation.
Internal learning from M&E will be complemented by small issue-based research projects contracted or
implemented by BRAC‟s Research and Evaluation Unit. Research will address key question related to
inequalities and extreme marginalization.
A timeline for BRAC‟s internal learning activities and deliverables is included in Table 17.
69
Table 17: Timelines for BRAC internal learning activities and deliverables
Activities and deliverables Expected dates
Quarterly monitoring meetings Quarterly
External evaluator presentation In March/April following each evaluation wave
BRAC evaluation review meetings Following each external evaluator presentation
Evaluation report March 2018, 2020, 2022 and 2024
Issue based research report 1
Issue based research report 2
External learning
External learning activities will take place primarily at the GEC-T level, through a range of activities to
promote cross-project learning at the national and global levels. At the country level, BRAC will
collaborate on learning activities with STAGES, the other GEC-T project in Afghanistan. This could take
place through quarterly learning meetings with STAGES consortium partners and the FM in-country
team. Topics of discussion could include challenges, enablers, and project experiences to share best
practices, particularly in relation to new GEC-T focus areas, including transition from primary to
secondary, and CBE sustainability models. BRAC will also participate in learning activities sponsored by
the in-country FM team, including workshops and evaluation events.
At the global level, BRAC will engage in learning activities to promote learning across the GEC-T
portfolio. BRAC will contribute to………….BRAC to list which learning clusters they would like to
contribute to and learn from, and why.What specifically can BRAC contribute to the learning themes?
BRAC will also actively participate in quarterly learning cluster webinars.
11.2 Stakeholder engagement, dissemination and influencing
BRAC‟s dissemination strategy for GEC-T will be based on three levels: accountability, learning and
advocacy.
Monitoring and evaluation outputs, including quarterly monitoring reports and evaluation reports, will be
disseminated to the donor, FM, external evaluator (EM) and other relevant stakeholders, primarily for
accountability, to demonstrate the impact of GEC-T funding.
To promote learning, monitoring quarterly reports and evaluation reports will not be disseminated in their
original forms, partly due to the need to disseminate concise and accessible M&E outputs. Instead, a
series of smaller, more concise outputs will be produced. Quarterly monitoring reports and evaluation
70
reports will be the basis for the production of case studies and summary documents (e.g. briefing notes)
on what works in girls‟ education programming in Afghanistan. The executive summaries of evaluation
reports will also be published as stand alone documents. Furthermore, BRAC‟s issue-based research
projects will be disseminated in their entirety. These outputs will be disseminated to key stakeholders,
including government ministries and intra-ministry units (e.g. the MoE and other key bodies engaged in
the project such as the Ministry of Women‟s Affairs, and the TVET Directorate), other NGOs working on
girls‟ education programming and CBE education, UN stakeholders including UNICEF and UNESCO,
and other donors participating in dialogues and funding of CBE education and girl‟ education.
As noted in section 5.3, BRAC will ensure that dissemination of learning outputs is inclusive of
beneficiaries and communities, and that marginalized groups (e.g. women, disabled persons, children)
will be included in dissemination activities. Due to high rates of illiteracy, dissemination of printed
outputs will be complemented by the verbal presentation of project results and learning. This will happen
at the community and district levels to ensure inclusion of marginalized groups. BRAC will organize
gender-segregated presentations in more conservative areas to ensure women are not excluded and will
attempt to ensure that presentations for women are delivered by women. BRAC will also consider
inclusive locations for presentations to encourage the participation of disabled persons. Dissemination of
results to children will be inclusive and child-friendly, and BRAC will link dissemination practices with
project activities. This may include, for instance, supporting teachers to disseminate research results with
their students through the use of classroom techniques included in BRAC teacher training, or supporting
mentors and mentees in government hub schools to incorporate the dissemination of key research findings
into extra-curricular activities such as public speaking or debating (see section 5.3.2 of the MEL
framework).
At the advocacy level, BRAC will use evidence and learning to support dialogue with the MoE and other
government organisations and programs (e.g. the Citizen‟s Charter), including through advocating for the
sustainability of CBE education and the sign off of the CBE policy, and other relevant strategic outputs.
12. Evaluation workplan
12.1 Timetable
Evaluation timelines are listed in Table 18for baseline, and with the same timelines projected for both
midlines and endline (albeit in subsequent years) in Table 19. The baseline evaluation will take place in
2017/2018, first midline in 2019/2020, second midline in 2021/2022, and endline in 2023/2024 (with
project closure expected in 2024). School term dates will not impact on the evaluation timelines;
however, school examinations generally begin at the end of November or early December and all data
collection should be completed prior to these dates. After submission of reports to the FM, learning
activities will take place at the end of each evaluation cycle (see section 11).
71
Table 18: Timelines for baseline evaluation
July 17 Aug 17 Sept 17 Oct 17 Nov 17 Dec 17 Jan 18 Feb 18 Mar 18
EE procurement
Research & tool design
Piloting
Baseline data collection and
benchmarking
Data processing (entry and
translation)
Data cleaning and analysis
Report writing
Submission of draft report to FM
Table 19: Timelines for midline and endline evaluations
Aug Sept Oct Nov Dec Jan Feb Mar
Research design & tool revisions
Baseline data collection
Data processing (entry and
translation)
Data cleaning and analysis
Report writing
Submission of draft report to FM
12.2 Responsibilities
This section outlines the MEL responsibilities of key actors, both within and externally to BRAC,
summarized in Table 20.
Table 20: MEL responsibilities
MEL responsibility Person/unit/organisation responsible
Procurement of external evaluator BRAC EE selection committee
Internal BRAC monitoring BRAC Monitoring Department and 10 Monitoring officers
External evaluation External evaluator
Evaluation ethics and safeguarding Key staff member selected in the external evaluator team
BRAC‟s Child Safeguarding Committee
Overall oversight of external evaluations,
including quality assurance
BRAC Research and Evaluation Unit
Evaluation Steering Committee???
Oversight and/or conduct of issue-based
research
BRAC Research and Evaluation Unit
Learning fed into project design and
implementation
BRAC Monitoring Department and BRAC Project Manager
Contribution to learning clusters BRAC GEC-T Project Manager, Research and Evaluation
Unit, and Monitoring department
72
Annex A: Logframe
The project logframe is a separate Excel document that should be read in conjunction with the MEL
framework.
Annex B: Draft evaluation tools
All evaluation tools will be designed by the external evaluator in consultation with BRAC prior to
baseline data collection, and thus are not included in the draft MEL framework.
Annex C: Draft sampling framework
A draft sampling framework with details of all community and school clusters is attached to the MEL
framework in Excel format.
73
Annex D: ToR for evaluators
BRAC to insert