Improving School Quality in East Africa:Randomized Evaluation of Policies to Create Local Accountability under Free Primary Education in Kenya and Uganda

Germano MwabuUniversity of Nairobi, Kenya

Frederick MugishaEconomic Policy Research Centre, Uganda

RESEARCH PROPOSALPresented to

PEP-AusAid Policy Impact Evaluation Research Initiative

Overview Randomized controlled trial of institutional interventions in

the Kenyan and Ugandan primary education sector Project will take place in close collaboration with the Ministries

of Education (MoE) in both countries Both countries characterized by large increases in enrollment

following introduction of free primary education and (perceived) decline in educational ‘quality’

Research will focus on two key interventions Hiring local contract teachers to address shortage of

government teachers in Kenya, and funding locally determined projects in Uganda.

Community-Based Monitoring System (CBMS) Interventions correspond to the two challenges in FPE – input

shortages and lack of parental ownership.

Research Question Expansion of the primary education sector under FPE has

led to influx of new students creating shortage of teachers undermined the ability of schools to raise funding locally due to the

abolition of fees weakened community based monitoring of schools. (FACTS)

Main question: does local accountability require local fundraising or is strengthening CBMS sufficient? (question: what is local accountability)

Hypotheses in Detail Hiring additional contract teachers in Kenya will have a

significant effect on student learning achievements. (Either by lowering teacher-pupil ratios or changing the nature of the contract.)

Providing school lunches in Uganda will have a significant effect on student attendance and learning.

Giving parents a financial stake in the school (soliciting co-funding to hire contract teachers) will increase local accountability and community monitoring relative to pure top-down funding.

Informing school management committees of parents’ rights and teachers’ responsibilities and providing them with tools to monitor school performance will improve school management and student performance.

Community-based monitoring systems are particularly effective when parents have a financial stake in the school – i.e., there are complementarities between the interventions.

Description of Interventions School Management Committee Score Cards

completed by SMC on a termly basis, gathering information on…

teacher performance: teacher attendance, preparation of schemes and lesson plans, class room activities measured via direct class room observation and spot checks

financial administration: correspondence between budgeted and actual expenditures, appropriate input purchases, perceived wastage or inefficiency in budget use

School facilities and maintenance: hygiene and sanitation, classroom conditions and repairs

School meals (Uganda only) matching grants for locally funded school lunches Create cooperative decision-making across geographically

proximate schools

Description of Interventions Contract Teachers (Kenya only)

Provide co-funding to SMCs to hire contract teachers Addresses two issues:

Acute teacher shortage due to constraints placed on local fundraising by Free Primary Education

providing for contract teachers will test the importance of local control over teaching staff in creating accountability and quality service provision

Hypothesis: local contract teachers -- hired on a fixed term, renewable contract in which the SMC has control over hiring and firing -- will be more responsive to local demands

Research design: pop & outcomes Population to be studied:

196 Kenyan schools with a total of 3299 students, nationally representative, based on Southern and Eastern African Consortium for Monitoring Education Quality Survey (SACMEQ)

Ugandan sample is representative of rural primary school students in four districts selected to represent poor-performing districts of the 4 regions

Outcome of interest: Student achievement 9 months – 1 year after start of the

intervention: measured by national exams as well as custom-made literacy and numeracy exams (in conjunction with Kenya Institute of Education in Kenya project, Uganda National Examinations Board)

Changes in enrolment and transition to secondary school 9 months after start of the intervention.

Research design: randomization Control schools chosen randomly and simultaneously with the

treatment. In Kenya, use SACMEQ sample stratification so treatment

and control schools have equal representation from urban and rural areas. Successful randomization will be tested using pre-intervention information.

In Uganda, sample from primary data collected at district level. Schools representative of rural sub-counties, with ‘blocking’ of the sample based on pre-intervention test results to ensure representation of low-performing schools.

Use cross-cutting design: treatment group for each intervention assigned orthogonally

Difference-in-differences model exploiting the randomized design of the intervention, comparing mean changes in academic performance between treatment and control schools.

Research design: analytical framework Cross-cutting interventions to test hypotheses about

interactions For instance, is a CBMS program enough to increase accountability,

or is it primarily effective when parents have more control over teachers (through contract &/or fundraising)

If Y is individual exam performance, i indexes individuals and j indexes schools, TEACH and CBMS are treatment dummies then:

The Bx parameter will test the interaction between local ownership and community monitoring.

We plan to go further and test two levels of funding as follows:

ijtjtjtjtXjtCjtTijt wvCBMSTEACHCBMSTEACHY )(

Similarly for Uganda:No meals intervention

Meals intervention

No co-funding (full)

Co-funding(partial)

No SMC intervention 17(no MealsNo SMC),Pure Control

8 8

SMC score-cards

Bottom-up accountability

17 8 X X 8 X X

Top-down accountability

17 8 X X 8 X X

Minimum detectable effects for alternative outcome measures and designs, Uganda

(1) (2) (3)

Design parameters

j=100, n=30, P=.5

j=400, n=30, P=.125

j=100, n=50, P=.5

Controls district, gender

District, gender

district, gender

Outcome English 0.69 (0.36) 0.52 (0.27) 0.68 (0.35)

Math 0.53 (0.34) 0.40 (0.25) 0.52 (0.33)

Avg score 2.46 (0.37) 1.84 (0.28) 2.43 (0.37)

Pass indicator

0.12 (0.30) 0.09 (0.22) 0.12 (0.29)

Note: Minimum detectable effect sizes in parentheses.

Minimum Detectable Effect Sizes for alternative outcome measures and test parameters, Kenya

SAQMEQ

Parameter estimated (assumptions)

Estimated intra-class correlation

MDES (P=.5, j=196, n=28)

MDES (P=.25, j=196, n=28)

Outcome Reading .38 .28 .32

Math .46 .25 .29

Average .45 .27 .32

Data Collection/SourcesKenya:

Admin data:use of Education Management information System, Kenya Certificate of Primary Education exam data, Southern and Eastern African Consortium to Monitor Education Quality (SACMEQ) II and III; the Kenya Integrated Household Budget Survey (KIHBS).

Follow-up to SACMEQ II to be conducted with MoE

Uganda: Admin data: use of Primary Leaving Examination results and

administrative data on school funding and district-level inputs.

Survey of school management and pupils to be collected by EPRC researchers in four districts. Tests prepared by UNEB (National Assessment for Planning in Education) to be administered to a sample of students

Example of MoE Database: nationwide coverage over time

BARINGO

BOMET

BUNGOMA

BUSIA

ELGEYO-MARAK

EMBUGARISSAHOMA BAY

ISIOLOKAKAMEGA

KERICHO

KIAMBU

KILIFI

KIRINYAGAKISII

KISUMU

KITUI

KWALE

LAIKIPIA

LAMU

MACHAKOS

MAKUENI

MANDERA

MARSABIT

MERU

MIGORI

MOMBASA

THIKA

NAIROBI

NAKURU

NANDI

NAROK

MERU SOUTH

NYAMIRANYANDARUANYERI

SAMBURU

SIAYA

TAITA TAVETA

TANA RIVER

TRANS-NZOIA

TURKANA

UASIN GISHU

VIHIGA

WAJIR

WEST POKOT

Change in Log Enrolment(.6,1.2](.4,.6](.2,.4](0,.2](-.2,0](-.4,-.2](-.6,-.4][-1.2,-.6]No data

Values represent the change in the log of public school enrolment

Change in Enrolment since FPE

BARINGO

BOMET

BUNGOMA

BUSIA

ELGEYO-MARAK

EMBUGARISSAHOMA BAY

ISIOLOKAKAMEGA

KERICHO

KIAMBU

KILIFI

KIRINYAGAKISII

KISUMU

KITUI

KWALE

LAIKIPIA

LAMU

MACHAKOS

MAKUENI

MANDERA

MARSABIT

MERU

MIGORI

MOMBASA

THIKA

NAIROBI

NAKURU

NANDI

NAROK

MERU SOUTH

NYAMIRANYANDARUANYERI

SAMBURU

SIAYA

TAITA TAVETA

TANA RIVER

TRANS-NZOIA

TURKANA

UASIN GISHU

VIHIGA

WAJIR

WEST POKOT

Change in KCPE points(5,20](0,5](-5,0][-20,-5]No data

KCPE scores measured by sum of English, Math and Science exams

Change in KCPE Scores since FPE

ConsultationsWhat have we done to include stakeholders in the planning

process? Kenya

Meetings held with Teacher Service Commission, Kenya Institute of Education and MoE (Quality Assurance and Basic Ed Directorates)

Focus groups discussions with teachers and head teachers plus church sponsor

Training workshop with MOE staff to bring them into the project as active participants

Uganda Preliminary meetings with District Education officials in each of

the 4 districts, plus MOE staff in Kampala. School visits to meet with teachers & head teachers in each

district 3-day intervention design and training workshop in Kampala

with MoE staff, UBOS, DEOs., and civil society organizations.

Dissemination Plan Standard research model: get data, do

analysis, try to convince policymakers you’re right

Our approach in this project: Train government officials in methods of analysis Rely as much as possible on administrative data

sources Involve gov officials in the implementation and

analysis Final results are fully owned by policymakers

as their own collaborative product

Project staff

Lead Institutionsa. University of Nairobi, School of Economics, Kenya

Germano Mwabu, Faculty memberMs Mumia Phillis Machio, Graduate studentMs Racheal Nakhumicha Musitia, Graduate studentMs Alice Muthoni Nga’ng’a, Graduate student

b. Economic Policy Research Centre (EPRC), Uganda

Fredrick Mugisha, Senior Research FellowLawrence Bategeka, Research FellowMs Madina Guloba, Assistant Research Fellow

c. Ministry of Education, KenyaMinistry officials d. Ministry of Education and SportsMinistry officials

Project staff…

Collaborating InstitutionsUniversity of Connecticut, Economics Department: Samson

Mwangi Kimenyi

University of Oxford, Centre for the Study of African Economies:Tessa Bold, Justin Sandefur, Roxana Gutierez, Andrew Zeitlin, Stefan Dercon

---All the above researchers are involved in a larger research

project funded by DFID to improve educational outcomes in Africa and Asia

Budget

Two thirds of the requested resources will fund project interventions directly

Thank you