c10 zero grazing project - partos · c10 – zero grazing project marrit van den berg1, elisabetta...

C10 – Zero Grazing Project

Marrit van den Berg1, Elisabetta De Cao

2,3, Steffen Eriksen

3, Chilot Y. Tizale

4, Tefera Wondwosen

5

1 Wageningen University, Netherlands

2 University of Oxford, United Kingdom

3 University of Groningen, Netherlands

4 Ethiopian Institute of Agricultural Research

5 International Food Policy Research Institute - Addis Ababa

1

1. Introduction

This paper provides an evaluation of the Zero Grazing of the Oromia Self Reliance Association (OSRA)

financed by ICCO within the MFSII framework for the Dutch Ministry of Foreign Affairs. In this evalua-

tion, we try to answer the following questions:

1. What are the changes under MDG1 -to eradicate poverty and hunger, during the 2012 – 2014

period and what are the differences between the target group and comparable households in

2012?

2. To what degree are these changes and differences attributable to the development

interventions of OSRA (i.e. measuring effectiveness)?

3. What is the relevance of these changes?

4. Were the development interventions of OSRA efficient?

5. What factors explain the findings drawn from the questions above?

In addition to this, we did an evaluation of the first phase of the project, which ended in 2012. We

observe limited impact of both phases. During phase 2, we observe an increase in the average amount

of fodder produced, which could directly result from the project. Also total expenditure increased in the

project areas compared to the control areas. However, we cannot explain this from the project

activities, as we hardly observe dairy sales and increased self-consumption of dairy (which we cannot

statistically prove) cannot realistically be large enough to explain this difference. We do not see

adoption of zero grazing or a change in herd composition towards improved cattle. The limited impact is

likely to be due at least partly to the lack of a marketing component in the project: if farmers do not

have access to a market for dairy products, incentives for productive investments are very limited.

The structure of the paper is as follows. We first present the context of smallholder livestock production

in the target districts and give a description of the project. Subsequently, we present the methodology

and data. This is followed by an analyses of the results, an discussion, and a conclusion.

2

2. Context1

After experiencing severe country-wide famines in both the 1970s and 1980s, Ethiopia seems to have

ventured on a pathway towards development and food security. Since 1992, the Ethiopian Government

has adopted a strategy of Agricultural Development-Led industrialization, which involved substantial

liberalisation of the economy. Per capita incomes increased by over 50 percent from 2001 to 2009, and

poverty rates declined by 33 percent from the mid-1990s to 2011 (www.WorldBank.org). Yet the pov-

erty rate is still high at 30% and the food security situation is precarious. In the past two decades there

have been several major, though localized, food production shortfalls, and even in normal years an es-

timated 44 percent of the population is undernourished (Schmidt and Dorosh, 2009). Further broad-

based development is thus needed to improve the situation of the remaining poor and food insecure.

Though growth in industry and services has outpaced agricultural growth, the latter has made a major

contribution to overall growth and has been essential for poverty alleviation.

A recent study on strategic priorities for agricultural development in Eastern and Central Africa con-

cludes that milk would be the most important commodity subsector for growth-inducing investment and

that milk is especially important for Ethiopia, Eritrea, and Sudan (Omamo et al, 2006). Ethiopia’s dairy

sector holds a large potential for development (Ahmed et al, 2004; Negassa et al, 2012). Income and

population growth are expected to lead to substantial increases in the demand for dairy. On the supply

side, the country holds the largest livestock population in Africa, and the climate is suitable for dairying

(Ahmed et al, 2004. Yet compared to the neighbouring countries, the government has done little to im-

prove the dairy sector and its productivity is low (Negassa et al, 2012).

Development of the dairy sector may positively affect the lives of many people, as production is spread

widely over the rural population. The traditional smallholder system produces 97 percent of Ethiopia’s

total milk production (Ahmed, et al 2004). Most milk is produced in the highlands on farms with mixed-

crop livestock production systems and, increasingly scarce, communal grazing lands. Milk is mainly used

for home consumption, and the marketed surplus is small. Households on average own two to four cat-

tle, of which 45 percent are draft cattle and 25 percent is used for dairy production (Negassa et al,

1 This section is partly based on an early paper based on the baseline data: Elisabetta De Cao, Chilot Y.

Tizale, Marrit van den Berg, and Tefera Wondwosen (2013). Paper presented t the CSAE Conference

2013: Economic Development in Africa, Oxford , UK, 17th - 19th March 2013.

3

2012). Most cattle are of indigenous breeds, with low production levels compared to crossbreds or exot-

ic breeds.

Not just the production, but also marketing and processing is generally informal and small scale. Only a

very small portion of the production is industrially processed. The remainder is administered by cooper-

atives and smallholders. These cottage dairy products and the fresh fluid milk are sold and consumed

locally. Even in the dairy market in Addis Ababa, the majority (75%) of the products sold come from

traditional processing; 17% are process in local industry and 8% is imported (Francesconi, 2009).

The OSRA Zero Grazing project targets dairy production in ten rural kebeles in four districts in the neigh-

boring Special Zone and East Shewa Zone of Oromia National Regional State: Akaki, Ada’a, Gimbichu,

and Sebata. These districts are located close to the capital, Addis Ababa (35-45 km), but infrastructure is

poorly developed. The main economic activity in the area is agriculture, as is common in the Ethiopian

highlands. The area is characterized by black cotton soils, and rainfall is adequate and evenly distributed.

The principal crops produced are teff, chickpea and wheat. As a secondary activity, the farmers engage

in livestock rearing for draught power, food, income and asset accumulation. Mostly, farmers rear dif-

ferent types of local livestock, and the returns from the rearing activities are low and declining. As the

majority of the farmland has been allocated to crop production, only small pieces of land, such as road

sides, are left for grazing. As a consequence, crop residuals are the main source of food for the animals,

accounting for more than 50% of the annual requirements.

In the years before the project, there have been several attempts to boost dairy production and in-

comes, but these have not resulted in substantial changes in the sector. In collaboration with district

agricultural offices, Debreziet Agricultural Research Center (DZARC) of the Ethiopian Institute of Agricul-

tural Research (EIAR) and the International Livestock Research Center (ILRI), have promoted crossbred

cows, improved feeds, and improved health management. In addition, the extension service has since

long t promoted Artificial Insemination (AI).

3. Project description

Within this context, OSRA has executed a program to stimulate income from the production of milk and

milk products through the introduction of improved breeds and the intensification of management

4

through zero grazing. After a one-year pilot in 2008-2009, the project was intended to start Jan 1, 2010.

However, there was a delay in the approval of the proposal, which in turn delayed actual

implementation such that project activities only started in the last quarter of 2010. Due to this delay,

the end-date of the project was postponed from December 31, 2011 to November 31, 2012. The overall

budget was €104,264.50.

The ultimate objectives of the project were:

1. Improvement in family income from sales of milk and milk products.

2. Reduction in the pressure on grazing land through promotion of zero grazing.

3. Improvement of the capacity of the target households on zero grazing and improved diary pro-

duction and management.

4. Improved access to input and output markets, extension support and technical services in the

area of artificial insemination (AI), feed and health management, and finance that are relevant

to promote zero grazing.

The project activities were diverse. A central nursery site was established to gather different multipur-

pose seedlings used for animal feed. This was done to lower the pressure on the already sparse grazing

land and to improve the availability of feed. In order to improve the AI services, AI technicians were

trained. In addition, AI crushes used for restraining cows during AI were constructed, along with various

AI materials. Furthermore, motor bikes were provided to the district livestock development, health and

marketing agencies, to improve access. Farmer access to support services and markets was promoted

through the organization of workshops with community representatives and service providers. In addi-

tion, the capacity for zero grazing and improved livestock management was stimulated through training

of experts, developments agents and farmers and through the facilitation of experience exchange

among farmers.

The project beneficiaries were cattle-holding farm households in ten selected kebeles of Akaki and Ada’a

districts. The selection of the target kebeles was based, among others, on: the number of popula-

tion/households in the area; the cattle population, especially improved and/or Borena breeds in the

localities; and the accessibility of the localities so that technical and other supports be provided to target

groups. Within the kebeles, a total of 1,700 farmers where selected for participation in training and rep-

resentatives where invited to workshops with service providers. The selection of these direct project

beneficiaries was conducted jointly with the Livestock Development, Health and Marketing Agency, de-

5

velopment agents, kebele administrations and the representative community members based on the

interest of the community member to participate, those who have local breeds or exotic breeds, acces-

sibility to road infrastructure, female headed households, and experience in promoting improved forage

at backyard or farm boundaries.

The project assumes that the results of all project activities spill over to the remainder farmers in the

community. Knowledge and experience is supposedly shared, and improved services and access to AI

benefit all farmers in the kebeles. However, the Livestock development, Health and Marketing agency

was unable to assign trained AI staff to Artificial Insemination. Besides, the office transferred the trained

AI to other locations.

By the end of the initial project, a second phase was approved. The extension started with a small delay

in April 2013 because the government needed to evaluate the first phase and appraise the proposal. The

second phase is planned to finish in 2015. While in first phase was targeted to ten kebeles, the project is

now scaled up to twenty kebeles. The new phase continued with the introduction of improved breeds,

distribution of fodder seeds, training (another 2000 farmers), and improvement of AI services (capacity

building support). According to the project implementers, during 2013 a total of 410 cows were in-

seminated out of which 189 have given birth to calves. Likewise in 2014 620 cows were insem-

inated by the AI technicians in both districts out of which 260 cows have given birth to calves.

New activities were milk processing and the introduction of bio-gas technologies. The latter activity has

been included because to improve overall welfare by using waste to produce energy. OSRA provides

materials like cement. The plan was to reach 200 households in two years. Yet, as the costs were higher

than envisaged, only 83 instead of 100 households were reached in 2013. About 10-15 percent of these

households had started to use the installations in the summer of 2014. Because of the high costs, OSRA

has sought collaboration with the government.

In addition, the second phase involved the distribution of heifers, contrary to the first phase. The identi-

fication of the suppliers was problematic. The project initially planned to distributed heifers of the local

breed Boren, but had to change into the improved Frisians. In addition, the rainy season delayed distri-

bution. Ultimately, the project reached 67 households out of the 100 planned. They all received the

improved breed, but had to pay 40 percent of the costs. Reportedly, so far no milk has been marketed

but production has increased from 1 to 3 liters per day. The milk is shared with neighbors.

6

Reported challenges are the accessibility of the farmers and government bureaucracy. During our visit in

May 2014, project staff indicated that they were able to do 60-70% of activities planned in 2013, and

45%-50% of the planned 2014 activities so far. Some activities, like training would continue until mid-

September.

4. Methodology

The objective of this report is to describe changes in wellbeing that could be attributed to participation

in the Zero Grazing project. The challenge in any impact evaluation is to isolate the causal role of the

intervention from other determinants of wellbeing (Armendariz (2010)). When comparing the situation

of beneficiaries before and after an intervention, the observed changes in outcome variables cannot

solely be attributed to the intervention, as there are many other changes in the environment of the re-

spondents during the project period. Comparing beneficiaries with a control group of non-beneficiaries

does not automatically provide the solution. Beneficiaries could for instance have been wealthier than

non-beneficiaries when the program started or vice versa. This and other factors can make some people

more likely to choose to participate in the program, thus causing self-selection bias. Besides self-

selection bias, program placement bias is also frequently observed in evaluation studies. NGOs target

their projects purposely at specific, often disadvantaged, areas. If the control group's physical, economic

and social environment does not match that of the beneficiaries, this will result in differences not

caused by the intervention and thus in biased estimates of impact.

The standard approach to overcome these issues is to conduct a randomized controlled trial (RTC). Be-

fore the program starts two groups will be created at random. One group will receive the treatment

(treatment group) while the other group will act as a benchmark (control group). Because of the ran-

domization the members in the control group represent the members in the treatment group. After the

treatment has taken place, both groups can be compared on the outcomes of interest. Differences in

outcomes can now solely be attributed to the treatment. However, creating two groups at random, a

necessary step for RTCs, is not always possible because of the program implementation or ethical rea-

sons.

7

As we had no influence on the design of the zero grazing program, we use an alternative to RCTs: the

propensity score method (PSM) (Rosenbaum(1983). The basic idea is to match beneficiary households to

households from the control group who are observationally similar in terms of characteristics that are

not affected by the intervention. These include stable characteristics of the household head and the

household. In practice, PSM involves running a binary regression of the project participation indicator on

the relevant set of characteristics. The result is the propensity score: the estimated probability that a

household is a beneficiary. For further analysis, we study only households with propensity scores in the

range in which both beneficiaries and non-beneficiaries are observed (the common support): non-

beneficiaries with a very low propensity score and beneficiaries with a very high propensity score are

discarded. For these households, we conducted a balancing test. Although debated in the literature, a

balancing test is generally used to determine if the observable controls are distributed similarly between

the two groups in question.2 If any significant difference exists between the two groups, a balancing test

should pick up this difference and indicate that the current composition of the data could lead to a bi-

ased estimate of the treatment effect. The balancing test is conducted as a set of OLS regressions in

which each control variable is individually regressed on a constant and the treatment dummy. The

standard errors in the regressions are clustered at the kebele level to account for intraclass correlation.

The coefficient of the treatment dummy now gives us an unbiased estimator of the difference between

the control and treatment group at the time of the baseline survey, provided that there is only selection

on the observed characteristics. A similar set of OLS regressions is used for the impact indicators. The

coefficients of the treatment dummy now give an unbiased estimator of project impact, again provided

that the assumption of selection on observed characteristics only holds.

When data on beneficiaries and controls are available for two points in time, a so-called double differ-

ence (DD) model can be used. Ideally, data are available from before (baseline) and after (end line) the

intervention, so that the complete change due to the project can be measured. For the current evalua-

tion, we had information for two periods in time of which the first was almost at the end of the original

project, while the second was after the extension. The DD estimates will therefore measure the addi-

tional impact of the project extension.

While the RTC methodology ensures that the only difference between members in the treatment and

control group is the treatment, the DD method only eliminates differences between members of the

two groups that do not change over time. So the disadvantage of the DD method over the RTC method

2 For a summary of the literature and further discussion of balancing tests see Kleyman (2009)

8

is that it does not automatically distinguish between the impact of time-varying characteristics between

members of the two groups and the impact of the treatment. A (partial) solution to this problem is to

include as many time varying characteristics into the DD model as possible so that only the influence of

unobserved time-varying characters remains. As the DD method works best if the treatment and com-

parison group are as comparable as possible, we apply the DD method to households within the com-

mon support of the PSM only.

Econometrically speaking the double difference estimator is given by the following expression:

𝑌𝑖𝑗𝑡 = 𝛼 + 𝛽1𝑃𝑜𝑠𝑡𝑡 + 𝛽2𝐷𝑗𝑇 + 𝛽3𝑃𝑜𝑠𝑡𝑡𝐷𝑗

𝑇 + 𝛾𝑋𝑖𝑗𝑡 + 𝜀𝑖𝑗𝑡 (1)

Where 𝑌𝑖𝑗𝑡 denotes an outcome variable for respondent i in group j at time t, 𝐷𝑗𝑇is a dummy variable

equal to one if the respondent belonged to the treatment group, 𝑃𝑜𝑠𝑡𝑡 is a binary variable that takes

the value one if the observation corresponds to the post-treatment time period, 𝑋𝑖𝑗𝑡 is a set of con-

trols3, and 𝜀𝑖𝑗𝑡 denotes the error term. Then, 𝛽3 in (1) is the treatment estimate of the intervention's

impact on outcome Y. That is, 𝛽3 measures the difference between the treatment and control group in

the growth of outcome Y, and is an unbiased estimate of the average impact on the dependent variable

Y of being assigned to the treatment group provided the assumption of no selection on unobservables

holds.

Some of the outcome variables considered in the analysis are binary. In these cases, we estimated a

linear probability model (LPM) and report the marginal effect of 𝐷𝑖𝐶 for the impact of the sustainable

energy project on outcome Y.4 In all models the standard errors are clustered at the kebele level. Clus-

tering at the kebele level provides a relatively low amount of clusters (18), but big enough clusters. It is

important to cluster the standard errors, as the data might be subject to intraclass correlation, that is,

households in the same kebele are likely to be more similar on a wide variety of measures than house-

holds that are not part of the same kebele. The higher intraclass correlation, the less unique information

3 Additional covariates are included to control for various characteristics of the households. This not only helps

with the precision of the estimates, but also reduces the problem of omitted variable bias. This is necessary to obtain unbiased measures of the treatment effects. 4 In recent literature, Puhani (2012) shows that in a nonlinear difference-in-difference, such as the one used in this

study, the cross difference is not equal to the treatment effect. Instead the treatment effect comes from the cross derivative (or cross difference) of the conditional expectation of the observed outcome minus the cross derivative of the conditional expectation of the potential outcome without treatment. Although this calculation of the treat-ment effect is appealing, this study will follow common practice in the field and report the estimate of 𝛽3 in the case of a LPM.

9

each household provides. This has to be taken into account when running the regressions by inflating

the standard errors.

It is often the case that there is some level of attrition (or dropping out) and/or new entry in the dataset.

The DD model presented in (1) can be estimated using an unbalanced panel, keeping in all observations,

including the ones which we only observe in one time period. It can also be estimated using a balanced

panel, dropping observations which we only observe in one period. Estimating (1) using an unbalanced

panel increases the power, but as the sample becomes more heterogeneous the results might be dis-

torted. As in this case, the number of new entries and drop outs are very low, as will be seen in the fol-

lowing section, all estimations will be done on the unbalanced panel.

Although a combination of PSM and DD can solve many potential biases in the data, some bias may

remain. As PSM assumes selection on observables, bias can still be caused by selection on unobserva-

bles. Adding DD to PSM helps picking up the time- invariant heterogeneity, but bias can still remain due

to time-variant unobservables. Yet, a positive significant effect in the PSM and especially the DD model

is a strong indication of an influential intervention. An insignificant effect -or even a significant negative

effect, however does not necessarily imply that the intervention does not work -or even does harm, as

in our analysis we only measure short-term effects.

5. Data

Data for both the baseline survey and follow-up were collected in four districts in the neighboring Spe-

cial Zone and East Shewa Zone of Oromia National Regional State: Akaki, Ada’a, Gimbichu, and Sebata.

The baseline survey was conducted in September 2012 –so just before the end of the first phase of the

project, and included 495 farm households equally divided over three groups: farm households with

direct participation in the project; farm households with dairy animals with indirect participation

through spillover effects at the district level (but not in the same kebele), and farm households with

dairy animals outside the project districts. The first group was randomly selected from project partici-

pants in four project kebeles (all project kebeles not involved in the pilot), and the other households

were randomly selected from farmers with dairy livestock in kebeles comparable to the project kebeles

in terms of soils, rainfall, farm size, crops, role of livestock, infrastructure and other relevant characteris-

tics. The follow-up survey was collected in Mid-May to Mid-June 2014, a few months before the end of

10

the Second Phase, in a similar manner as the baseline survey and revealed a very low amount of drop

outs and new entries (4 and 1 respectively).

The questionnaire contains general questions on household composition and housing conditions,

household expenditures and food security, crop production and consumption, land and livestock en-

dowments, and detailed questions related to dairy cattle and production, involving grazing, fodder pro-

duction, health, milk production, production costs, marketing, and home consumption.

We found no significant differences between the control and the “spillover” groups in household char-

acteristics and technology adoption in the baseline survey. This suggests that there were no significant

spillover effects of the intervention. Such effects were expected because households in the “spillover”

group would benefit from the improvements in the AI service. Unfortunately, these improvements were

not realized for reasons explained above: Trained AI staff was not assigned to AI, and some were trans-

ferred to other locations. This means that although trainings were given, this has not resulted in better

trained AI staff in the treatment kebeles. As a result, the spillover and control group were merge into

one large non-treatment group. The results presented in this project thus only distinguish between two

groups.

5.1 Descriptive statistics and balancing test

The descriptive statistics and balancing tests presented here are based on the common support as a

result of the estimated propensity score. The set of covariates chosen for the estimation of the propen-

sity score is the same as the set of controls used in the DD regressions. It is important to note that while

these tests are conducted on the baseline survey, the survey was conducted after the intervention was

implemented. We therefore do balancing tests only or those variables that we expect not to be influ-

enced by the intervention –the controls.

Figure 1 below displays the distribution of the propensity score in the treatment and control group. Plot-

ting the distribution of the propensity score is helpful to see if there are any problems in the common

support. The distributions seem to have a wide area from which they overlap, thus indicating that only a

few observations are dropped from the sample. Specifically, the region of the estimated common sup-

port is given by [.1457, .7317] given a total number of 12 observations who are dropped from the sam-

ple.

11

Figure 1: Distribution of the propensity scores in the control and treatment group

Most of the household heads in the sample are middle-aged Oromo men with farming as primary occu-

pation (Table 1). They own on average 2 hectares of land, about three oxen and four sheep/goats for a

family of 7. Education of the household head is very low at less than three years on average. Overall the

balancing tests reveal a good balance between the two groups in terms of the controls variables. The

only significant difference is between the two groups is found in the education variable but this amounts

to less than a year.

12

Table 1: Summary statistics and balancing tests for controls

Summary Statistics Balancing tests

Control Treatment

Dependent variable No. Mean No. Mean

Treatment

(1) (2) (3) (4)

(5)

Characteristics of the household head

Sex (male=1) 308 0.88 158 0.90

0.019

(0.037)

Age (years) 308 45.06 158 43.65

-1.410

(1.756)

Material (1=married) 308 0.87 158 0.86

-0.009

(0.026)

Education (years) 308 1.90 158 2.69

0.790*

(0.388)

Ethnicity (Oromo=1) 308 0.91 158 0.84

-0.077

(0.047)

Job (farmer=1) 308 0.93 158 0.96

0.027

(0.023)

Household characteristics

Members (#) 308 6.77 158 6.62

-0.152

(0.520)

Dependency ratio 308 94.12 158 90.76

-3.358

(7.366)

Land (ha) 308 2.13 158 1.93

-0.199

(0.168)

Oxen (#) 308 2.95 158 3.06

0.108

(0.189)

Goat & Sheep (#) 308 3.67 158 4.27

0.600

(0.700) Notes: Column 2 and 4 presents the mean of the control and treatment group respectively. Column 5 displays the coefficient

from a separate OLS regression. Standard errors are clustered at the kebele level. The summary statistics and balancing tests

are based on the common support as given by the propensity score. Robust standard errors in parentheses ***Significant at

the 1 percent level. **Significant at the 5 percent level. *Significant at the 10 percent level.

13

6. Results

6.1 The initial project (2010-2012)

Overall, the beneficiary and control households do not differ much with respect to the outcome indica-

tors. We found some small but significant differences. However, these seem to point at mild selection

bias rather than at impact. Beneficiary households own nine indigenous cows on average compared to 8

for those in non-project kebeles. The number of crossbred and exotic cows is low: less than one on av-

erage. While the average number of crossbreds per household does not differ significantly between

beneficiaries and non-beneficiaries, the number of exotic cows is slightly higher for the controls: 0.08 for

the control group and 0.01 for the treatment group. Yet this difference is too small to be meaningful.

Milk production is low for both groups at about 200 liters per cow per year (slightly higher for the con-

trol group) with about 10 kg of milk or derived products consumed per household in the seven days

before the survey. Only three farmers (in the non-project kebeles) sold milk (not included in the PSM).

AI is practiced slightly more by the non-project farmers: by ten percent compared to three percent of

project beneficiaries. Usage is too low to detect differences in costs. On average, cows graze almost 11

months of the year, implying very little adoption of zero grazing. Still, the cows of the project beneficiar-

ies graze a bit less on average than those of the controls: 10.2 months compared to 10.9 months. Also,

project farmers are more likely to grow fodder, but average fodder production is the same between

treatment and control farmers. Overall, the differences between project beneficiaries and controls are

very small and not consistently pointing at positive project impact.

Table 2: Summary statistics and impact tests for outcomes

Summary Statistics Impact tests

Control Treatment

Dependent variable No. Mean No. Mean

Treatment

(1) (2) (3) (4)

(5)

Expenditures/capita 307 2,035 158 2,401

364.979

(Birr)

(400.077)

Yearly milk production 253 217.42 132 199.30

-18.983

/cow (liters)

(20.641)

Home consumption of 275 11.63 156 9.26

-2.419

14

dairy/week (kg)°

(1.693)

#grazing months (#) 307 10.93 158 10.18

-0.755*

(0.369)

Fodder production 307 0.39 158 0.56 0.162**

(yes=1) (0.069)

Fodder quantity (kg) 307 1.887 158 1.138 -748.763

(514.295)

Indigenous cows (#) 307 1.23 158 1.31

0.073

(0.078)

Crossbred cows (#) 307 0.03 158 0.05

0.018

(0.021)

Exotic cows (#) 307 0.02 158 0.00

-0.019

(0.016)

Indigenous calve (#) 307 0.84 158 0.96 0.122*

(0.062)

Crossbred calve (#) 307 0.02 158 0.04 0.018

(0.022)

Exotic calve (#) 307 0.01 158 0.00 -0.010

(0.007)

Use of AI (Yes=1) 307 0.10 156 0.03

-0.072**

(0.033)

Costs of effective AI °° 27 17.41 4 44.25

26.843

(35.670) Notes: Column 2 and 4 presents the mean of the control and treatment group respectively. Column 5 displays the coefficient

from a separate OLS regression. Standard errors are clustered at the kebele level. The summary statistics and balancing tests

are based on the common support as given by the propensity score. °All dairy products are converted to kgs of fresh milk. °°The

total cost for the insemination times the number of cows that got pregnant divided by the total number of cows inseminated.

Robust standard errors in parentheses ***Significant at the 1 percent level. **Significant at the 5 percent level. *Significant at

the 10 percent level.

6.2 Phase 2 (2013-2014)

Table 3 reports all the estimates with the respective statistics from all the different approaches used in

this project. The results are mostly robust to the approach selected. Beneficiaries households increased

their expenditures (including home consumption of crops and animal products) by 790 birr more than

the control household during Phase 2 of the project. This is quite substantial compared to the overall

increase between baseline and endline of 2,334 birr (year effect), which is similar to baseline income

and at least partly explained by inflation. However, our results suggest that this was not due to a chang-

15

es in milk production or consumption, for which no significant difference is observed. Moreover, milk

sales are still very rare. Yet, we do observe a relative increase in fodder production levels for farmers in

the treatment kebeles, though the share of farmers growing fodder did not change significantly. Howev-

er, though we observe an overall decrease in the average number of months that cows are grazing –and

hence an increase in zero grazing, this is not larger for the project area. We find no changes in the use of

AI or the effective costs of AI for ether project farmers or controls. Relatedly, we find no significant dif-

ference in the size and composition of the herd, except for a small increase of the number of indigenous

cows for the treatment group, but this effect disappears when we use a combination of DD and PM.

The question remains whether we find few evidence of project results because of lack of impact or be-

cause our sample was too small to have sufficient power. We therefore computed the minimum effect

size that we could detect with our sample given the heterogeneity of the sample. For herd composition,

we could generally detect a difference of less than 0.1 animal. Only for indigenous calves this was high-

er: 0.23. For grazing months, we could detect an average difference of 0.94 months, which is quite small.

We can thus conclude that our analysis is quite accurate with respect to herd composition and grazing.

For home consumption of dairy products and milk production the numbers are much higher: 8.93 kg per

week for consumption –which is an additional increase of almost the same magnitude as the average

weekly consumption in the baseline survey, and 65.65 liters per year per cow for production –which is

equivalent to about one third of average baseline production. These minimum effect sizes are so high

because of the large differences between households. The observed (insignificant) effect on milk pro-

duction is about 32 liters per year and the observed (insignificant) impact on dairy consumption is about

3.7 kg per week, which are both somewhat substantial but much lower than the increases of 2-5 liters

per day mentioned in FGDs during an evaluation of the initial project commissioned by OSRA (Gudeta,

2013). Yet, we can be sure that if the project resulted in increased milk production, this has not led to

sales of milk and increased incomes at least not in our sample. Similarly, the impact coefficients for AI

point in the right direction and are relatively high. Our sample is not sufficient to tell whether these are

project impacts or statistical artefacts. However, we do not find evidence of an increase in the number

of crossbred calves, and this finding is quite accurate.

Table 3: Estimation results

(1) (2) (3) (4)

Variables DD DD PSM+DD PSM+DD

16

- with controls - with controls

Indigenous cow SI SI SI NI

Crossbred cow NI NI NI NI

Exotic cow NI NI NI NI

Indigenous calve NI NI NI NI

Crossbred calve NI NI NI NI

Exotic calve NI NI NI NI

AI NI NI SI NI

Effective costs AI°° NI NI NI NI

#grazing months NI NI NI NI

Milk production NI NI NI NI

/cow (liters/year)

Home consumption NI NI NI NI

of dairy (kg/week) °

Fodder production NI NI NI NI

Fodder quantity SI SI SI SI

Total expenditures per capita

SI SI NI SI

Notes: NI: no impact; NM: Not measured; SD: Significant decrease; SI: significant increase. Controls include: Sex, age, marital

status, members, ethnicity, job, land, pack animals, and goat & sheep. See detailed estimation results in the appendix.

Our analysis suggests that the impact of the project has been limited. Though we cannot completely be

sure that this is not due to the limited power of our sample, we feel confident to say that the project did

not result in a widespread adoption of zero grazing, sales of dairy, or an increase in ownership of im-

proved cattle among participating households. It is possible that the project participants in our sample,

17

which were selected based on participation in the initial project, did not participate in the second pro-

ject. However, this is not because they already benefited sufficiently in the 2010-2012 project: we do

not find impact for this project either.

6.3 Efficiency analysis

To assess the efficiency of the project, we computed unit costs for selected activities and compared

these to values found in the literature. The cost of the training of AI technicians was only 61% of that of

a similar, but somewhat shorter, training in Indonesia. Also the farmer trainings were much lower than

data found in the literature: Int$50 per trained farmer compare to Int$521-819 in Bangladesh. However,

the large difference in values suggests different training lengths. More money was spent per farmer for

the distribution of seedlings and seeds than found in the literature. However, both the quantity and type

of seeds are different, so these numbers are hard to compare. Overall, we find it difficult to compare

the unit costs computed for the project with unit costs from the literature. The most reliable comparison

seems that for the AI technician training, which suggests the project was quite efficient.

Table 4 Efficiency estimates (based on the 2010/11 financial reports)

Activity Unit costs Unit cost benchmark Source benchmark

Forage seeds and planting materials distributed

Int$19 for seedlings and seeds Int$ 9.70 for seeds and other inputs

Teweldmehidin, M. Y., & Conroy, A. B. (2010). (Namibia)

Livestock training Int$50 (training on zero graz-ing & improved heifer man-agement, business manage-ment skill, book keeping)

Int$ 521.00-Int$ 819.00 (vocational educa-tion)

Dohmen (ILO, 2009) and Dar, A. et al. (2006) (Bangladesh)

Training of AI technician (per person trained)

Int$ 1,672 (45 days training)

Int$ 2,726.00 (one month training)

Subagiyo, 2010 (Indonesia)

Note: See Table A5 in the appendix for unit costs calculations

7. Discussion

Previous research in Ethiopia shows that the adoption of improved dairy technology results in higher

per capita incomes and higher intake of calories, protein, and iron (Ahmed et al, 2004). Yet adoption is

constrained by increasing fodder scarcity and a lack of economic incentives to produce marketable sur-

plus (Lemma et al, 2008a). The demand for milk and milk products has increased, putting an upward

18

pressure on prices, but marketing systems are not well-established (Lemma et al, 2008b). Also the lack

of health infrastructure and veterinary services are a disincentive for acquiring improved breeds (Negas-

sa et al, 2012). Improved dairy technologies related to housing, feeding and healthcare largely improve

milk production performance for crossbred cows, but have only a limited effect on the productivity of

local cows (Mekonnen et al, 2010). Sustainable commercialisation of smallholder dairy in Ethiopia there-

fore requires an integrated approach involving technological as well as institutional innovations.

The project activities correspond to part of these observations. Yet, the focus is completely on technol-

ogy and the output market is ignored. As indicated above, a lack of economic incentives due to an un-

derdeveloped marketing system will limit the adoption of improved technologies. In fact, during our

discussion with OSRA, the organisation indicated that those farmers who received subsidized improved

heifers substantially increased their production, but did not yet sell milk. Instead, they shared it with

their neighbours. While in itself this could result in higher food and nutrition security, it suggests limited

incentives for further investments. Relatedly, the evaluation of the initial project commissioned by OSRA

indicated that farmers show no interest taking credit for purchasing improved cattle (Gudeta, 2013).

Future projects should therefore include a component of improving linkages to output markets. Espe-

cially given the location of the kebeles close to Addis Abeba this is a missed opportunity In the present

project.

8. Conclusion

Our evaluation shows an increase in total expenditure in the project areas compared to the control

areas. In addition, we observe an increase in fodder production. While the latter could well be a direct

result of the project, it is unlikely that this is the case for the increase in expenditures: Milk sales are

extremely rare and a possible increase in milk consumption (that we do not observe due to limited

power of the sample) cannot explain the relatively large increase. Gazing decreased between the

surveys, but equally in treatment and control areas.

Given the low levels of income in the region and the scarcity of fodder, the observed changes are highly

relevant. However, for the project to be more effective, it should have included a market component to

increase the profitability of dairy production and generate incentives for investment. Due to lack of

19

good benchmarks, we do not want to give a general evaluation of project efficiency. However, the AI

training seemed highly efficient.

Table 5 Evaluation conclusions

Statement Rating1 Comments

The project was well designed 5 Integrated technological innovations, but lack of attention for output markets

The project was implemented as designed 7 Costs and bureaucracy were underestimat-ed, which resulted in some delays, downscal-ing and cost recovery from participants.

The project reached all its objectives 5 Some evidence of impact on fodder produc-tion, slight decrease in grazing.

The observed results are attributable to the project interventions

6 The results for phase 1 are based on the baseline only and could result from other factors than the project. The results for phase 2 are based on the DD method, which is relatively reliable. Yet, we cannot explain the increase in expenditures from the pro-ject due to the lack of impact on sales and consumption. For some variables, standard deviations are high. Cross-variable compari-son of impact estimates is helpful .

The observed results are relevant to the project beneficiaries

8 Increased expenditures and fodder produc-tion are important. Zero grazing can benefit the area

The project was implemented efficiently 8 Difficult to say due to lack of good bench-marks, but the (AI) trainings seem highly efficient.

1 Our agreement on a scale for 1 (not at all) to 10 (completely)

References Ahmed, M. A., S. Ehui, et al., 2004. Dairy development in Ethiopia. EPTD Discussion Paper, International

Food Policy Research Institute. 123

Armendáriz, Beatriz and Jonathan. Morduch, 2010. The Economics of Microfinance. The MIT Press, sec-

ond edition.

Dar, A.; Abrahart, A.; Ahmed, S.; Rashed Al-Zayed, S.; Khan, Q.; Stopnitzky, Y.; Savachenko, Y.; Tan, H. (2006) The Bangladesh vocational education and training system: an assessment. Washington, DC:

20

World Bank. http://documents.worldbank.org/curated/en/2006/11/10151997/bangladesh-vocational-education-training-system-assessment Dohmen, D. (2009). Financing of Technical and Vocational Education and Training (TVET) in Bangladesh.

For the EC/ILO Executed TVET Reform Project in Bangladesh.

Dorosh P. and Rashid S., 2012. Introduction. In: Food and Agriculture in Ethiopia. Progress and Policy

Challenges. Paul Dorosh and Shahidur Rashid Editors. PENN Press.

Francesconi, G.N. 2009. Cooperation for Competition: Linking Ethiopian Farmers to Markets. Interna-

tional Chains and Network Series, vol. 5. Wageningen, the Netherlands: Wageningen Academic Publish-

er.

Kleyman, Yevgeniya N, 2009. Testing for covariate balance in comparative studies. Ph.d. dissertation,

University of Michigan.

Lemma, T,. Azage Tegegne, Ranjitha Puskur and Dirk Hoekstra. 2008 a. "Moving Ethiopian Smallholder

Dairy Along a Sustainable Commercialization Path: Missing Links in the Innovation Systems."

Lemma, T., Ranjitha Puskur Dirk Hoekstra, and Azage Tegegne, 2008b. Exploring Innovation Capacity in

Ethiopian Dairy Systems. Paper presented at the international conference “Knowledge for enhancing ag-

ricultural development in developing countries’, IFPRI, 7-9 April 2008, Addis Ababa, Ethiopia

Mekonnen, H; G Dehninet and B Kelay. 2010. "Dairy Technology Adoption in Smallholder Farms in “De-

jen” District, Ethiopia." Tropical animal health and production, 42(2), 209-16.

Negassa A., Rashid S., Gebremedhin, B. and Kennedy, A. (2012) Chap. 6 “Livestock Production and Mar-

keting” in Food and Agriculture in Ethiopia. Progress and Policy Challenges. Paul Dorosh and Shahidur

Rashid Editors. PENN Press.

Omamo, SW; X Diao; S Wood; J Chamberlin; L You; S Benin; U Wood-Sichra and A Tatwangire. 2006.

"Strategic Priorities for Agricultural Development in East and Central Africa," Research Report 150.

Washington, DC: International Food Policy Research Institute (IFPRI).

Rosenbaum, P.R. and D.B. Rubin (1983) The central role of the propensity score in observational studies

for causal effects. Biometrika, Vol. 70, No. 1. pp. 41-55.

21

Puhani, Patrick A., 2012, The treatment effect, the cross difference, and the interaction term in nonline-

ar ’difference-in-differences’ models. Economics Letters.

Subagiyo, I. (2010) Indonesia-Japan Training on Artificial Insemination of Dairy Cattle: Case Study. Task

Team on South-South Cooperation.

Tegegn Gudet (2013) Final evaluation report on “Promotion of zero grazing and improved dairy produc-

tion project”

Teweldmehidin, M. Y., & Conroy, A. B. (2010) The economic importance of draught oxen on small farms

in Namibia’s Eastern Caprivi Region. African Journal of Agricultural Research, 5(9), 928-934.

22

Appendix

Table A1: Double Difference estimates (no controls)

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14)

VARIABLES Total ex-

penditures

Yearly milk

production/cow (liters)

Home

consumption of

dairy/week

(kg)

#graze

months

Fodder

production

Fodder

quantity (kg)

Local cow Crossbred

cow

Exotic

cow

Local calve Crossbred

calve

Exotic

calve

AI AI effect

Year 1,975.776*** -18.047 7.833*** -4.294*** -0.128 -1,338.221*** -0.228*** 0.034* 0.018 0.054 0.031 0.009 0.085 3.135

(306.895) (20.915) (2.478) (0.302) (0.089) (416.019) (0.046) (0.018) (0.022) (0.053) (0.019) (0.011) (0.051) (10.737) year*treatment 627.262* 29.044 5.647 1.030 0.146 1,611.880** 0.195** 0.028 0.031 0.007 0.037 0.022 0.133 -38.638

(340.161) (33.786) (3.884) (0.663) (0.100) (579.959) (0.078) (0.025) (0.043) (0.084) (0.030) (0.030) (0.079) (40.511)

Constant 2,046.715*** 214.360*** 11.444*** 10.936*** 0.400*** 1,829.018*** 1.234*** 0.030*** 0.018 0.836*** 0.018** 0.009 0.095*** 17.143*** (153.635) (8.895) (1.113) (0.134) (0.066) (398.451) (0.046) (0.006) (0.015) (0.053) (0.007) (0.006) (0.032) (4.091)

Observations 984 789 951 868 893 986 985 985 985 985 985 985 889 108 R-squared 0.140 0.001 0.025 0.284 0.062 0.026 0.021 0.010 0.006 0.007 0.012 0.004 0.047 0.020

Notes: All the columns present the coefficient estimate of the Double Difference regressions. The model regresses the outcome variable on a treatment dummy (equaling 1 if the house-hold belongs to the treatment group and 0 otherwise), a year dummy (equaling 0 for the recall period and 1 for the time of the survey) and an interaction of those. All regressions have

the standard errors clustered at the kebele level and are estimated with controls, which include: Sex, Age, marital, members, educ, ethnic, job, depratio, land, oxen, and goatsheep. Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

Table A2: Double Difference estimates (with controls)

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) VARIABLES Total ex-

penditures

Yearly milk

production/cow

(liters)

Home

consumption

of

dairy/week

(kg)

#graze

months

Fodder

production

Fodder quanti-

ty (kg)

Local cow Crossbred

cow

Exotic

cow


calve

Exotic

calve

AI AI

effect

Year 2,198.082*** -10.015 8.072*** -4.416*** -0.096 -1,187.095*** -0.208** 0.062** 0.021 0.039 0.062* 0.011 0.102* 4.635 (352.834) (21.183) (2.620) (0.280) (0.084) (338.890) (0.071) (0.027) (0.024) (0.061) (0.035) (0.014) (0.055) (13.103)

year*treatment 892.857** 26.322 5.287 0.917 0.159 1,502.161** 0.172* 0.023 0.032 -0.027 0.032 0.019 0.115 -56.280

(358.249) (36.938) (3.678) (0.672) (0.094) (527.837) (0.091) (0.024) (0.047) (0.093) (0.034) (0.033) (0.079) (36.306) Constant 5,108.381*** 162.464 -12.160** 8.430*** 0.245* -224.793 0.144 -0.017 -0.035 -0.248 -0.023 -0.000 0.019 -30.171

(436.902) (93.043) (4.257) (0.713) (0.116) (695.354) (0.222) (0.019) (0.066) (0.208) (0.031) (0.041) (0.098) (34.931)

Observations 944 756 909 834 856 944 944 944 944 944 944 944 853 104

R-squared 0.247 0.015 0.074 0.313 0.079 0.067 0.217 0.049 0.030 0.164 0.066 0.015 0.072 0.124

Notes: All the columns present the coefficient estimate of the Double Difference regressions. The model regresses the outcome variable on a treatment dummy (equaling 1 if the house-hold belongs to the treatment group and 0 otherwise), a year dummy (equaling 0 for the recall period and 1 for the time of the survey) and an interaction of those. All regressions have the

standard errors clustered at the kebele level and are estimated without controls. Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

23

Table A3: PSM with Double Difference estimates (no controls)

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14)

VARIABLES Total ex-

penditures

Yearly milk

production/cow (liters)

Home

consumption of

dairy/week

(kg)

#graze

months

Fodder

production

Fodder quantity

(kg)

Local cow Crossbred

cow

Exotic

cow


calve

Exotic

calve

AI AI effect

year 2,053.840*** -23.584 7.970*** -4.260*** -0.132 -1,401.216*** -0.212*** 0.030 0.020 0.061 0.030 0.010 0.069 5.573

(291.943) (23.258) (2.445) (0.326) (0.089) (442.333) (0.043) (0.017) (0.024) (0.053) (0.021) (0.012) (0.048) (12.729) year*treatment 516.027 34.562 4.444 0.993 0.157 1,677.808** 0.192** 0.029 0.032 0.015 0.036 0.022 0.154* -42.877

(333.624) (34.178) (3.716) (0.705) (0.098) (608.200) (0.079) (0.026) (0.045) (0.083) (0.029) (0.031) (0.082) (40.664)

Constant 2,035.367*** 218.282*** 11.670*** 10.928*** 0.397*** 1,899.577*** 1.235*** 0.033*** 0.020 0.840*** 0.020** 0.010 0.098*** 17.407*** (159.548) (10.836) (1.200) (0.142) (0.066) (433.326) (0.044) (0.007) (0.016) (0.052) (0.007) (0.007) (0.031) (4.339)


Notes: All the columns present the coefficient estimate of the Double Difference regressions after first matching the baseline characteristics on their propensity score. The estimation is then done for observations on the common support. The model regresses the outcome variable on a treatment dummy (equaling 1 if the household belongs to the treatment group and 0

otherwise), a year dummy (equaling 0 for the recall period and 1 for the time of the survey) and an interaction of those. All regressions have the standard errors clustered at the kebele level and are estimated with controls, which include: Sex, Age, marital, members, educ, ethnic, job, depratio, land, oxen, and goatsheep. Robust standard errors in parentheses *** p<0.01,

** p<0.05, * p<0.1

Table A4: PSM with Double Difference estimates (with controls)

(1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) (12) (13) (14)

VARIABLES Total ex-penditures

Yearly milk production/cow

(liters)

Home consumption

of

dairy/week (kg)

#graze months

Fodder production

Fodder quantity (kg)

Local cow Crossbred cow

Exotic cow

Local calve Crossbred calve

Exotic calve

AI AI effect

year 2,334.100*** -13.459 8.152*** -4.301*** -0.099 -1,174.229*** -0.168** 0.061 0.026 0.066 0.065 0.015 0.081 9.964

(348.976) (23.900) (2.647) (0.305) (0.088) (394.096) (0.066) (0.036) (0.028) (0.070) (0.047) (0.017) (0.054) (16.928) year*treatment 790.101** 31.933 3.726 0.870 0.155 1,551.552** 0.149 0.027 0.032 -0.056 0.033 0.019 0.136 -61.286

(369.904) (37.454) (3.527) (0.703) (0.097) (557.534) (0.086) (0.025) (0.049) (0.092) (0.033) (0.034) (0.080) (38.924)

Constant 5,151.897*** 181.301* -11.436** 8.492*** 0.262** -86.272 0.173 -0.005 -0.035 -0.191 -0.011 0.002 -0.005 -34.639 (480.376) (99.138) (4.459) (0.725) (0.112) (727.239) (0.242) (0.019) (0.070) (0.200) (0.033) (0.044) (0.092) (43.996)


Notes: All the columns present the coefficient estimate of the Double Difference regressions after first matching the baseline characteristics on their propensity score. The estimation is then done for observations on the common support. The model regresses the outcome variable on a treatment dummy (equaling 1 if the household belongs to the treatment group and 0 otherwise), a year dummy (equaling 0 for the recall period and 1 for the time of the survey) and an interaction of those. All regressions have the standard errors clustered at the kebele

level and are estimated without controls. Robust standard errors in parentheses *** p<0.01, ** p<0.05, * p<0.1

24

Table A5 Unit cost calculations

year Units total costs (Birr) PPP $ cost($)

cost (int$) unit costs (int$)

Training of AI technicians (45/days) 2010 2 11180 4.18 14.41 775.8501 3243.053

2011 2 11,840.00 4.92 16.9 700.5917 3446.911 1672

Training of farmers (on zero grazing & improved heifer management, business management skill, book keeping)

2010 0 0 4.18 14.41 0 0 2011 1693 289,990.87 4.92 16.9 17159.22 84423.38 50

Fodder and forage plants provision 2010

52,048.08 4.18 14.41 3611.942 15097.92

2011 2866 138,464.79 4.92 16.9 8193.183 40310.46 19

c10 zero grazing project - partos · c10 – zero grazing project marrit van den berg1, elisabetta...

Documents