november 5, 2009
DESCRIPTION
The PNPM- Generasi Project One-Year Impact Evaluation Preliminary Findings Presented by: Susan Wong, EASER, The World Bank Ben Olken, M.I.T. Department of Economics. November 5, 2009. Structure of Today’s Presentation. Indonesian context Description of PNPM-Generasi project design - PowerPoint PPT PresentationTRANSCRIPT
The PNPM-Generasi Project One-Year Impact
Evaluation Preliminary Findings
Presented by:Susan Wong, EASER, The World Bank
Ben Olken, M.I.T. Department of EconomicsNovember 5, 2009
2
Structure of Today’s Presentation
• Indonesian context • Description of PNPM-Generasi project design• PNPM-Generasi implementation update• Impact evaluation design and analysis• Preliminary findings of one-year PNPM-
Generasi implementation– Impact on health indicators– Impact on education indicators– Effect of community incentives– Heterogeneity
• Conclusion, Further Analysis, and Next Steps
3
Indonesian Context• Remarkable progress in poverty reduction and key human
development indicators over past few decades. • Economic growth, increased access to education and
health services, expanded anti-poverty programs, and improvements in infrastructure have all helped to reduce poverty.
• Poverty headcount is 14.1% in 2009.• However, 32.5 m Indonesians still live below poverty line &
one-half of all HHs remain clustered around national poverty line. High vulnerability.
• Regional disparities with Eastern Indonesia lagging behind other parts of country, esp. Java.
• Human development areas require more attention: child malnutrition, infant and maternal mortality, primary to secondary school transition, access to safe water and sanitation. Quality of services also a major concern.
4
Two Pilot ProjectsIn 2007, GoI started two pilot projects:
• Household CCT – the traditional model– Quarterly tranches of cash transfers – Statistically identified 633,000 poor households with
children – Currently in 13 provinces, 70 districts, 629
municipalities– Annual budget of IDR 1.2 trillion (@USD 120 m)
• PNPM-Generasi Community Block Grants – Addresses the same health and education indicators,
but at the community level– 5 provinces, 21 districts, 178 subdistricts – Covering approx. 3.1 million beneficiaries– Total budget from 2007-2009 of @USD 107 m
5
WB Support Role
• Collaboration between PREM, HD, and Social Development Units in Indonesia
• Provide TA for design, implementation and evaluations of two pilots.
• Portion of KDP/PNPM WB loan funds support the PNPM Generasi pilot in 5 provinces.
6
The PNPM-Generasi Project
• Objectives: Accelerate the achievement of MDGs– Reduce child mortality– Reduce maternal mortality, and – Ensure universal coverage of basic
education• Conditionalities: Places incentives on
communities to identify problems and seek solutions to improving 12 health and education indicators
7
The PNPM-Generasi Project
• Community incentives: – Version A: 20% of year 2 allocation depends
on previous year’s village performance– Version B: Village performance not linked to
fund allocation. Otherwise identical to Version A.
• Implemented through KDP/PNPM-Rural with:– Same management structures at the national,
provincial, and district levels as PNPM-Rural– Facilitated by 2 subdistrict facilitators
8
PNPM-Generasi Project Design
12 indicators: communities are required to work on the same indicators as HH-CCT (Program Keluarga Harapan)
Health:1. Four prenatal care visits during pregnancy2. Taking iron tablets during pregnancy3. Delivery assisted by trained professional4. Two postnatal care visits5. Complete childhood immunization6. Ensuring monthly weight increases for infants7. Regular weighing for under-fives8. Taking Vitamin A twice a year for under-fives
Education:9. Primary school enrolment (7-12 year olds)10.Regular primary school attendance >85%11.Junior secondary school enrolment (13-15 year olds)12.Regular secondary school attendance >85%
9
PNPM-Generasi Design
Socialization stageWhat is Community CCT?What are the 12 indicators?
Villageplanning stage
Village implementation
stage
Performancemeasurement
Social mapping;Village council
election;Women’s focus
groups;Inter-village
meetings;Workshop with
providers;Prioritization;
Decision-making
Implementation of village activities;Monitoring of 12 indicators;Village Council monthly review meetings;
Cross-village audits;External audits
PNPM-GenerasiImplementation Cycle
Fund allocation for following year
10
PNPM-Generasi Project Implementation
• Geographical coverage:– 178 subdistricts in 21 districts, five provinces– Approx 3.1 million beneficiaries
• Block grant amounts:– 2007 average per village amount USD 8,400 – 2008 average per village amount USD 11,600 – 2009 average expected per village amount USD 14,400
• Timeframe:– First block grant disbursed to villages in Oct-Dec 2007– Second year disbursement to villages in Oct-Dec 2008– Third year disbursement to villages in Oct-Dec 2009
11
Village Fund Allocation in 2007
• 56% of block grants on education:– School materials, equipment and uniforms (59%)– Financial assistance and school fees (31%)– Infrastructure (satellite classrooms and access roads) (5%)– Financial incentives for honorarium teachers (4%)– Training and behavior change communication (1%)
• 44% of block grants on health activities:– Supplementary feeding activities (40%)– Financial assistance for pregnant mothers to use services
(30%)– Infrastructure (13%)– Facilities and equipment (11%)– Training and behavior change communication (3%)– Incentives for health workers (3%)
12
Impact Evaluation Design• Uses a randomized evaluation• Subdistricts allocated by lottery into
three groups: – with performance incentives, – without performance incentives, and – controls
• Subdistrict level randomization addresses spillovers and crowding out
13
Impact Evaluation Design• Three rounds of surveys:
- Baseline/Wave I (2007): PNPM-Generasi & PKH (HH CCT)– Wave II (2008): PNPM-Generasi only– Wave III (scheduled to begin Nov 2009): PNPM-Generasi &
PKH
• Survey design:– 12,000 households per wave spread over 300 subdistricts
including• Anthropometric measurements of children <3• Math and Indonesian tests administered to school-aged
children (Baseline and Wave III)– School and health provider interviews to track supply-side
effects– Qualitative studies to understand bottlenecks in use and
provision of services (Baseline and Wave III)
14
Impact Evaluation DesignSample size per survey for PNPM-Generasi:33,000 total respondents - 12,000 households
– 10,800 married women in reproductive age • 4,850 pregnancies (2 years prior to the survey)
– 9,500 school-aged children (7-15 years old)– 4,750 children under-three
• 2,313 villages• 300 subdistrict health centers (puskesmas)• 1,157 midwives• 2,391 village health posts (posyandu) (Waves
II & III only)• 847 junior secondary schools• 1,065 primary schools (Waves II & III only)
15
Mid-Term Impact Analysis• Regressions run for:
– PNPM-Generasi vs No PNPM-Generasi– Incentivized (version A) vs Non-Incentivized
(version B)
• Regression specifications: – Uses baseline data as control variables
• Subdistrict average• Individual baseline values for panel respondents (0 for
non-panel)
– District fixed effects– Province * previous KDP experience fixed effects– HH sampling category dummies
16
Overview of Preliminary Findings
• Substantial improvements in health indicators• No impact on education indicators• Performance-based incentives lead to
consistently better outcomes• Substantial regional heterogeneity with
strongest improvements in Sulawesi• Provider effort, especially for midwives in
incentivized locations, increased substantially.• Greater community engagement particularly
through service provision at the village health posts.
17
Preliminary Findings: Health• Strongest improvements on health services
coverage:– Participation in growth monitoring– Deliveries assisted by doctors or midwives,
particularly in Java and Sulawesi– Large increase in village health post participation
• Long-term health outcomes:– Large reductions in neonatal and infant mortality
(although some small differences noted at baseline)
– Some reductions in malnutrition (<2SD weight-for-age) among children under-three in NTT and Sulawesi
18
Generasi Project Effects on Health Service Coverage
0.0342 Standard Deviations
5%
4.69%
2.30%
5.08%
-1.92%
-0.19%
-1.72%
-0.17%
-2.00% 0.00% 2.00% 4.00% 6.00%
ANC
Iron tablets
Delivery
PNC
% immunization complete
Weight Check
Vit A
Malnutrition
Average StandardizedHealth Effects
Percentage change
- Increased participation in growth monitoring- Average improvement across all health indicators
7.52 visits
1.975 sachets
75%
2.91 visits
70%
2.18 times
1.62 Vit A
21%
Control avg.
Notes on the figures
Control group averages
Key messages of the figure
Bars in solid colors are statistically significant
Each bar represents the percentage change in the indicator in Generasi treatment areas compared to control areas
Zero percent is the control group average. Bars to the right indicate increase while bars to the left indicate reduction.
Bars in patterns are not statistically significant
Generasi Project Effects on Health Service Coverage
0.0342 Standard Deviations
5%
4.69%
2.30%
5.08%
-1.92%
-0.19%
-1.72%
-0.17%
-2.00% 0.00% 2.00% 4.00% 6.00%
ANC
Iron tablets
Delivery
PNC
% immunization complete
Weight Check
Vit A
Malnutrition
Average StandardizedHealth Effects
Percentage change
- Increased participation in growth monitoring- Average improvement across all health indicators
7.52 visits
1.975 sachets
75%
2.91 visits
70%
2.18 times
1.62 Vit A
21%
Control avg.
20
Generasi Project Effects on Services at Posyandu
20.28%
42.21%
-2.9%
19.75%
48.08%
26.55%
39.77%
-10.00% 40.00%
Weighing
Supplementaryfeeding
Immunized
ANC
Iron pills
Vit A
FP injections
Percentage change
Large increase in Village Health Post participation :•Numbers of children weighed; receiving supplementary feeding; immunized; receiving Vit A •Numbers of pregnant mothers receiving antenatal care; iron pills•No increase in non-targeted village health post services
41.21 children
34.99 children
11.85 children
4.54 mothers
4.79 mothers
44.61 children
2.83 mothers
Control avg.
Generasi Project Effects on Health Outcomes
-65.00%
-45.00%
-2.62%
-10.17%
-1.92%
-70.00% -50.00% -30.00% -10.00%
Neonatal mortality duringprevious 18 months
Infant mortality duringprevious 24 months
ARI/diarrhoea in past onemonth among children <3
years of age
Malnutrition (<2SDweight-for-age) for <3
years of age
Severe malnutrition(<3SD weight-for-age)
for <3 years of age
Percentage Change
•Reduction in neonatal mortality and infant mortality (although some small differences noted at baseline)•Similarly large infant mortality reductions found in other randomized community health programs in Uganda and Bangladesh
Control avg.
5.9%
20.8%
34.3%
18 deaths per 1000 live births
8 deaths per 1000 live births
22
Preliminary Findings: Education
• No overall project impact on education– Negative impact on enrollment and
attendance of 13-15 year olds, primarily among those 13-15 year olds who would have been in primary
– No impact on primary or jr. secondary net enrollment
• Overall jr. secondary school enrollments increased in both treatment and control over this period
23
Generasi Project Effects on Education Indicators
0.11%
-0.43%
-7.16%
-12.50%
-1.67%
0.20%
-4.48%
-14.00% -12.00% -10.00% -8.00% -6.00% -4.00% -2.00% 0.00%
SMP gross enrollment
13-15 enrollment in otherschools
13-15 enrollment in SMP
13-15 total enrollment
7-12 total enrollment
7-12 enrollment in SD
SD gross enrollment
Percentage Change
- Reductions in 13-15 total enrollment driven largely by lower enrollment in SD- Some evidence of slightly lower SMP attendance (1% lower)
Control avg.
93.7%
89.1%
97.9%
90.7%
66.0%
24.8%
97.8%
24
Some Hypotheses on Education Findings
• Primary school enrollment already high at 95%.• Jr. secondary gross enrollment increased overall in
treatment and control areas.• Great deal of fluctuation over last couple of years.• Generasi targets only 13-15 years old for jr. secondary,
so communities may have interpreted this age conditionality strictly.
• Program missed the school enrollment period of June-July.
• Communities seemed to be favoring more assistance towards children already in school vs. focusing on out-of-school children. “Help the greatest number vs. the few” mentality.
25
Preliminary Findings: Community Incentives
• Version A, incentivized version outperformed non-incentivized version in improving health service coverage– Prenatal care– Postnatal care– Growth monitoring
• Version A had larger impact on long-term health outcomes:– Acute morbidity (ARI or diarrhea)– Malnutrition
• Version A was more effective in increasing midwives’ work efforts in:– Outreach services– Time spent seeing patients in their public capacity (reduced
time for private practice)
26
Effect of Community Incentives - Health Indicators
6.58%
3.86%
1.01%
-4.24%
4.56%
4.40%
0.0643
3.45%
-7.69%
-0.62%
7.35%
2.84%
0.25%
2.53%
5.09%
-8.56%
4.33%
0.003
-0.1 -0.05 0 0.05 0.1
Standardized AverageEffect
Malnutrition
Immunized
Vit A
*Weight checks*
*PNC*
Delivery
Iron pills
*ANC*
P e rc e n ta g e C h a n g e
Incentivized (A) Non-Incentivized (B)
•Incentivized version outperformed non-incentivized in improving health service coverageControl avg.
7.52 visits
1.975 sachets
74.6%
2.91times
2.18 times
1.63 times
69.5%
20.8%
27
Effect of Community Incentives - Health Outcomes
-7.69%
-8.16%
-70.00%
-43.89%
0.0578
-13.56%
-46.11%
0.026
-62.50%
-8.48%
2.62%
4.33%
-0.8 -0.7 -0.6 -0.5 -0.4 -0.3 -0.2 -0.1 0 0.1
Standardized Average Effect
Neonatal mortality during previous 18 months
Infant mortality during previous 24 months
ARI/diarrhoea in past one month among children <3years of age
Malnutrition (<2SD weight-for-age) for <3 years ofage
Severe malnutrition (<3SD weight-for-age) for <3years of age
P e rc e n ta g e C h a n g e
Incentivized (A) Non-Incentivized (B)
•Incentivized version outperformed non-incentivized in improving health outcomesControl avg.
5.9%
20.8%
34.3%
18 deaths per 1000 live births
8 deaths per 1000 live births
28
Preliminary Findings: Heterogeneity
• Regional heterogeneity– Sulawesi - strongest and consistent effects– Java - some positive impacts– NTT – smallest effects
Average Standardized Effects:Main Health Indicators
0.041
0.0868
0.051
0.1731
0.025
0.0643
0.03
0.014
0.003
-0.024
0.0342
-0.0759
-0.1 0 0.1 0.2
Reg
ion
s
Standard Deviations
Avg Health Effect - Generasi Avg Health Effect - A Avg Health Effect - B
All
NTT
Sulawesi
Java
•Sulawesi - strongest and consistent effects•Java - some positive impacts•NTT – smallest effects
30
Effect of Community Incentives - Standardized Average Health Outcomes
0.105
0.042
0.009
0.052
0.058
0.099
0.034
0.072
-0.017
0.036
0.110
0.026
-0.020 0.030 0.080
NTT
Sulawesi
Java
All
P e rc e n ta g e C h a n g e
All Incentivized (A) Non-Incentivized (B)
•Incentivized version had larger impact on average health outcomes overall and in NTT
Average Standardized Effects: Main Education Indicators
-0.044
-0.1479
-0.024
-0.0649
-0.068
-0.034
-0.055
-0.018
-0.2231
-0.037
-0.0742
0.022
-0.3 -0.2 -0.1 0 0.1
Reg
ion
s
Standard Deviations
Avg Education Effect - Generasi Avg Education Effect - AAvg Education Effect - B
All
NTT
Sulawesi
Java
Negative impact on education largely seen in Sulawesi
32
Conclusions• PNPM-Generasi has:
– Improved health service coverage mainly through increased village health post (posyandu) participation
– Reduced infant mortality, acute morbidity and malnutrition– Increased number of hours midwives spend on outreach and
services in their public capacity
• PNPM-Generasi has not yet improved formal education indicators
• Community incentives ensures better outcomes with the same project funds and design
• Certain heterogeneity in outcomes were observed:– Regional: Sulawesi strongest, positive in Java, small in NTT
• Increased community participation and engagement, especially through service provision at village health posts.
33
Future Analysis• Cost-benefit analysis• Where/for whom PNPM-Generasi
works best• Spillovers• Details of community incentives• Changes in prices• Targeting• What communities spent their funds
on
34
Next Steps
• PNPM-Generasi will likely expand in 2010 to an additional 1-2 provinces.
• Adjustment of some of the education indicators
• Conduct wave III survey in 660 kecamatan jointly with HH CCT (Nov 2009– Jan 2010)
• Finalize operations paper on lessons learned thus far