1

Validating Ex Ante Impact Evaluation Models: An Example from Mexico

Francisco H.G. FerreiraPhillippe G. Leite

Emmanuel SkoufiasThe World Bank

PREM Learning Forum-April 22, 2008

Introduction

Conditional Cash Transfer (CCT) programs are becoming an important element of social policy in LAC

Distinguishing characteristic of CCTs: social accountability supported by

rigorous impact evaluation (IE)

Introduction Alternative IE designs: Experimental design: hh randomly

assigned to T and C groups, prior to implementation of program. Typically hh surveyed in baseline and for 1 or more rounds after the start of the program.

+: provide most reliable estimates (gold standard) of program impacts

-: costly, likely to be of small scale -: large time lags involved

Introduction Quasi-experimental designs: typically

comparison/control hh are obtained ex-post (after the start of the program), attempting to equalize selection bias between treatment and control groups

+: less costly -: lack of baseline data (and/or pre-program

differences)

Overall, ex-post methods do not provide ANY information about the possible effects of the program prior to its implementation.

Introduction Ex ante methods: simulate the effects of the

program on the basis of a structural (or reduced form) model of household behavior Easily implemented using a representative hh

data set (e.g. BFL, 2003) Expand the set of policy-relevant questions that

can be addressed, e.g. useful in designing the program, size of transfer, etc.

Based on the concept of treatment and comparison/counterfactual group

However, require some strong assumptions about:

Functional formPerfect implementation of the programAbsence of time or trend effects.

Introduction This paper is one of the first to provide a

validation test of the ex-ante evaluation methodology

Approach: Use household survey data from two CCT programs (PROGRESA in Mexico and BDH-Bono de Desarollo Humano in Ecuador) where experimental designs were employed to (ex post) evaluate program impact

Use the baseline data from each survey to apply ex-ante evaluation methods to predict program impact.

Introduction

Compare the impact predictions obtained with the ex-ante method to the impact estimates obtained using the experimental (ex-post) methods.

Some Background on PROGRESA

What is PROGRESA? Targeted cash transfer program conditioned on

families visiting health centers regularly and on children attending school regularly.

Cash transfer-alleviates short-term poverty Human capital investment-alleviates poverty

in the long-term Started in 1998. By the end of 2004: program

(renamed Oportunidades) covered nearly 5 million families, in 72,000 localities in all 31 states (budget of about US$2.5 billion).

Transfers given to mothers: 20% of hh consumption expenditure

Some Background on PROGRESA

Two-stage Selection process: Geographic targeting (used census data to

identify poor localities) Within Village household-level targeting

(village household census)Used hh income, assets, and demographic

composition to estimate the probability of being poor (Inc per cap<Standard Food basket).

Discriminant analysis applied separately by regionDiscriminant score of each household compared to a

threshold value (high DS=Noneligible, low DS=Eligible)

Initially 52% eligible, then revised selection process so that 78% eligible. But many of the “new poor” households did not receive benefits

10

Ex ante model: BFL

Why BFL instead of Attanasio, Meghir et Santiago (2005) ou Todd et Wolpin (2005)? Simplicity since dynamic Ex ante models as AMS

and TW are data intensive depending on panel data.

Is a behavioral model based on four key assumptions:Do not model household behavioral, i.e., do

not debate who makes child’s decision;Adults are unafected by children’s choice;Siblings interaction are ignored;Household composition is exogeneous

11

Ex ante model: BFL

The model Child’s occupational choice

(0) Not going to school;(1) Going to school and paid work;(2) Going to school and non-paid work

i22i2i2ii

i11i1i1ii

i00i0i0ii

v)y(YZ(2)U

v)y(YZ(1)U

v)y(YZ(0)U

12

Ex ante model: BFL

The model Child’s contribution to income in each state 0,

1 and 2

)exp(

)1(

;; 210

mM

where

uSIndmXwLog

Then

wDywMywy

iiii

iiiiii

13

Ex ante model: BFL

The model Child (household) i chooses the alternative

that yields the highest utility

Ui (0)Zi 0 Y i 0 wi 0 vi0

Ui (1)Zi 1 Y i 1 wi 1 vi1

Ui (2)Zi 2 Y i 2 wi 2 vi2

where

0 0 ; 1 1 M; 2 2 D

14

Ex ante model: BFL

The model Child (household) i chooses the alternative

that yields the highest simulated utility

Ui*(0)Zi ˆ 0 Y i ˆ 0 ˆ w i ˆ 0 vi0

Ui*(1)Zi ˆ 1 (Y i T )ˆ 1 ˆ w i ˆ 1 vi1 if potential benef

Ui*(1)Zi ˆ 1 Y i ˆ 1 ˆ w i ˆ 1 vi1 otherwise.

Ui*(2)Zi ˆ 2 (Y i T )ˆ 2 ˆ w i ˆ 2 vi2 if potential benef

Ui*(2)Zi ˆ 2 Y i ˆ 2 ˆ w i ˆ 2 vi2

where ˆ 0 ˆ 0 ; ˆ 1 ˆ 1 M; ˆ 2 ˆ 2 D

15

Ex ante estimator

Average Intent to Treat effect (AIT) which provides an estimate of the average impact of the availability of the program to eligible households (in treatment communities) by simulating impact of the program on the sample of eligible age group of children;

Assumes good implementation of program Attention: Ex ante model is static, i.e., no time or

trend effects. So, it is best to compare AIT (ex ante) with AIT (ex

post) obtained using 2DIF (which removes the trend effect from the estimated impact) whenever is possible.

16

Results: PROGRESATable 1: Children's Occupational Choices in MexicoActual and Counterfactual Enrollment Rate for Target Population

Observed

8-17 years-old 74.5% 4.0% **

0.4%

8-11 years-old 93.8% 1.8% ** 0.0%0.7% 0.4%

12-17 years-old 57.5% 5.8% ** 5.9% **

2.1% 0.8%

8-17 years-old 69.4% 4.3% **

0.7%8-11 years-old 93.9% -0.3% -0.2%

1.0% 0.4%

12-17 years-old 47.9% 9.5% ** 6.6% **

2.2% 0.8%Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculationNote:1: Results from Skoufias and Parker (2001) - tables 6.** Significant at 5% level; * Significant at 10% level.Standard Deviation computed by bootstrap method.

Ex Ante ITT

Boys

Girls

-

-

Ex Post ITT1

17

Results: PROGRESATable 2: Children's Occupational Choices in MexicoActual and Counterfactual Child Labor Rate for Target Population

Observed

8-17 years-old 23.0% -2.6% **

0.3%

8-11 years-old 6.7% -1.1% 0.3%

0.0% 0.4%

12-17 years-old 37.3% -4.7% ** -3.7% **

0.0% 0.5%

8-17 years-old 9.7% -0.7% *

0.3%

8-11 years-old 4.2% 0.0% 0.4%

0.0% 0.3%

12-17 years-old 14.6% -2.3% * -1.3% *

0.0% 0.5%Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation.Note:1: Results from Skoufias and Parker (2001) - tables 5.** Significant at 5% level; * Significant at 10% level.Standard Deviation computed by bootstrap method.

Girls

Ex Post ITT1 Ex Ante ITT

Boys

-

-

18

Results: PROGRESA

Table 3: Poverty Index: Observed and Simulated

% % %

"Mexico"Treatment Comunities - Baseline 60.9% 1.1% 30.8% 1.1% 21.5% 1.1%

Ex Post: Skoufias and Di Maro (2006) -16.5% 1.6% -24.3% 1.5% -29.2% 1.4% Ex Ante: SimulatedSource: Baseline Survey 1997 and Rounds 1-4;Authors' calculation;Skoufias and Di Maro (2006) table 4.Note: Prices 11/99; PesosPoverty Line = Mean of Nov 98 Consumption per Capita. Obtained from Skoufias and Di Maro (2006)

-13.1% -26.6% -33.1%

FGT(0) FGT(1) FGT(2)

19

Results: PROGRESA

Table 4: Estimated and Observed Progresa's Cost

BFL Simulation1,2 January 20021,3

Number of Families with Children Receiving Benefits:

4,567 1,681,254

Average Monthly Transfer for Families with Children

$301 $300

Estimated Total Annual Transfer $16,517,580 $6,061,221,243

Scaling Up Total Transfer: Simulated average times families in 2002

$6,080,632,364 n.a.Source: Baseline Survey 1997 and Rounds 1-4;Authors' calculation;Skoufias and Di Maro (2006) table 4.Note: 1: 11/1999 prices 2: Estimated based on the Baseline survey of 1997 3: National oficial numbers: http://www.oportunidades.gob.mx/indicadores_gestion/ene_feb_02/indice.htm

20

Conclusion

Ex Ante model analysis indicates so far that they can be very useful as well as powerful in predicting program impacts.

But work is still in progress. Useful for simulating the design or re-design of

a transfer program. Increasing demand from governments as

Panama, Jamaica and Ecuador