LECTURE 8.EVALUATION IN AGRICULTURE AND RURAL DEVELOPMENT

MSc Economic Policy StudiesEC8011: Welfare Economics, Project and Programme Appraisal and EvaluationAlan Matthewsalan.matthews@tcd.ie25 Nov 2011

Outline

The policy context for EU and Irish agricultural policy

Evaluation experiences Evaluation specificities Case studies

Arterial drainage, agricultural research, rural development

Forestry

THE POLICY CONTEXT FOR AGRICULTURE AND RURAL DEVELOPMENT

Agricultural and rural development policy

Multiple objectives Ensuring food security and food at

reasonable prices Regulating food safety Supporting farm incomes Stabilising prices Promoting sustainable agriculture Achieving greater territorial cohesion Encouraging growth in agricultural output

through provision of public goods

The policy framework

Market measures and direct payments provided through Pillar 1 of the CAP

Rural development measures (improving agricultural competitiveness, sustainable land management and territorial cohesion) promoted through Pillar 2 of the CAP

Significant state expenditure through co-financing EU schemes as well as independent schemes and regulatory agencies

Specificities of evaluation in the agricultural and rural development context Covers both policies and programmes Address both socio-economic and

environmental objectives Complexity

Objectives are often very general in nature Difficulties in identifying impacts

attributable to specific measures in the context of multiple intervening factors

The challenges of establishing the counterfactual of what would have happened in the absence of the programme

EVALUATION EXPERIENCIES

Role of the Office of Public Works

Began CB analysis of arterial drainage schemes in 1970

Methodology evaluated by Bruton and Convery (1982) “The Office of Public Works stands alone among

public agencies in Ireland in its willingness to publish its investment appraisal procedures. We have accepted the invitation implicit in this act and have critically analysed the procedures so described. In doing so we are reviewing techniques of analysis which in many – perhaps almost all – cases are no doubt better than those employment elsewhere in government in Ireland. We salute the members of the Office of Public Works who have had the courage and confidence to subject their work to public scrutiny: this volume is dedicated to them.”

Department of Agriculture and Food

Analysis and Evaluation Unit in the 1990s Supported by EU Structural Funds

Value for Money Reviews

Department of Agriculture, Fisheries and Food Beef Classification Scheme, 2005 Compensatory Allowance Scheme, 2006 BSE eradication, 2006 Laboratory Testing for Plant and Animal Diseases,

2007 Farm Waste Management Scheme, 2007 Dairy Hygiene Scheme, 2008 Food Industry Research Measure, 2008 Processing investment aids, 2008 Bovine TB eradication, 2008 Young Farmers Installation Scheme, 2009 Forest Roads Scheme, 2010

Agricultural policy covered by EU standard guidelines for evaluation and IAs Evaluation plan Impact assessment of proposed policy

changes e.g. CAP reform post-2013 Public quality statement on the evaluation

produced ex-post by DG Agri DG Agriculture and Rural Development

web page In addition, the European Court of Auditors

(ECA) periodically undertakes evaluations of particular programmes

EU rural development programmes

In the EU, a formal system of evaluation is in place.

EU member states are obliged by EU legislation to assess the performance of their Rural Development Programmes (RDP).

The EU Rural Development Regulations of 1999 and 2005 require that each RDP undergoes ex ante, mid-term and ex post evaluations according to a specified timetable.

Innovations in evaluation 2007-2013 A more strategic approach concerning the definition

of rural development programmes; A common framework for the monitoring and

evaluation of all measures funded through the European Agricultural Fund for Rural Development;

The establishment of an “ongoing evaluation system” in order to: i) better linking monitoring activities with evaluation needs in terms of data collection; ii) establishing and quantifying baseline indicators and target levels in a timely manner; and regularly assessing the progress of the programmes in achieving quantified goals against baselines;

Setting up of a European Evaluation Network to help establishing good practice and capacity building in the evaluation of Rural Development Programmes.

EU rural development evaluation requirements

Annual MS progress report containing summary of on-going evaluation activities.

In 2010, ongoing evaluation will be in the form of a separate mid-term evaluation report.

In 2015 it will be in the form of a separate ex-post evaluation report.

Also, in 2010 and each year thereafter, each member state will be required to submit to the European Commission a summary progress report implementing its national strategy plan and objectives, and strategic guidelines.

A summary of the midterm and ex post evaluation reports prepared by the member states will be made, under the responsibility of the European Commission, by end-2016.

The evaluation system16

Period 2007-2013

“Ongoing evaluation”

Programming

Ex-ante evaluation

Mid-term evaluation

Policy implementation

Ex-post evaluation

Evaluation network for RD HelpdeskEvaluation expert committee

CMEF

17

1. Community strategic guidelines define EU priorities of each of the axes

2. National Strategy Plans translate EU priorities into strategies in light of national specificities

3. National or regional programmes of measures on the basis of needs assessments and the national strategies

4. Implementation of programmes: monitoring and ongoing evaluation based on a Common Monitoring and Evaluation Framework

EU strategic approach for rural development programming 2007-2013

CMEF indicators

Common baseline indicators (59)

Objective-related indicators + targets Context-related indicators

Common monitoring indicators (95)

Financial indicators Output indicators Result indicators

Common impact indicators (7)

Additional programme-specific indicators

Handbook on the CMEF (Guidance document with 4 annexes consisting of 15 chapters)http://ec.europa.eu/agriculture/rurdev/eval/index_en.htm

ROLE OF MODELS IN AGRICULTURAL POLICY EVALUATION

Model simulations - applications

Implications of moving to decoupled payments

Implications of removing milk quotas Implications of concluding a Doha Round

agreement Implications of meeting climate change

targets

Models as experimental laboratories

Types of economic models Partial equilibrium commodity models

Single-commodity models (food safety regulation, labelling..)

Multi-commodity models (FAPRI-Ireland….) Calibrated or econometric

Partial equilibrium programming models Computable general equilibrium models Microsimulation models (tax-benefit..)

Model characteristics

Time series econometric vs. market equilibrium models

Static vs. dynamic Level of aggregation Establishing the baseline Defining scenarios

Pros and cons of models

Validation Credibility gained through repeated use

(HERMES model…) Expensive to construct and maintain More complex models can become ‘black

boxes’ – difficult to communicate to policy makers (SEAMLESS…)

EXAMPLEARTERIAL DRAINAGE INVESTMENTBRUTON AND CONVERY 1982

Arterial drainage case study

Background Arterial drainage – artificial widening and

deepening of main rivers to increase their effectiveness at draining their catchment areas

Field drainage – the activities necessary to remove surplus water from fields

Interdependent Long history in Ireland Undertaken by the Office of Public Works

Arterial drainage case study

Ireland suffers from drainage problem due to its topography

By preventing flooding, main benefit is to increase the productivity of affected lands

Potential environmental implications for: Water supply Fisheries Wildlife (drainage of wetlands) Aesthetic and amenity considerations

Value of investment ensured only through regular maintainance

Why public involvement?

Arterial drainage is a public good Free-rider problem Transactions costs Role of externalities Income redistribution (using tax revenues to

enhance the productivity of land belonging to more deprived members of the community)

Rationale for involvement in field drainage Less clear cut? Higher discount rates of

elderly farmers

Analytical framework

To assess the net impacts of drainage, the prospective impacts if the land were not drained should be deducted from the prospective impacts with drainage (with-without approach)

Analysis of alternatives should proceed marginally Analysis should examine alternative means of

achieving the same objective (expanding net farm income)

Test validity of assumptions using ex post analysis Adjust flows of funds over time using appropriate

rate of interest

Select project with highest NPV(for mutually exclusive alternatives)

Year A B C

0 -10 -40 -30

10 40 100 60

Results

Present net worth 14.6 21.4 6.8

Internal rate of return (%)

14.9 9.6 7.2

Benefit/cost ratio 2.46 1.53 1.23Choosing the alternative which maximises either the IRR or the B/C ratio can mean foregoing increments at the margin which return more than they cost

Analytical framework

Shadow pricing Estimating the net additional income

accruing to the landholders Cost of labour (if otherwise unemployed)?

Valuation of environmental effects Assigning market values not practical Marginal analysis can help identify threshold

effects Secondary impacts

Multiplier effects – not valid

1945 Arterial Drainage Act

Based on recommendations of a Commission set up to study how the government might best intervene on arterial drainage Should focus on draining entire catchments Govt should bear all the cost of construction Beneficiaries should pay 70% of the assessed

improved annual value of the lands affected towards the cost of maintenance

Significant government subsidy justified on grounds of spin-off effects of greater agricultural prosperity, the positive impacts in making field drainage possible, avoiding remedial costs of deterioriating drainage conditions, indirect benefits to transport, public health, urban areas, sewage outfalls

Arterial draining is an “essential service” and a matter of “national pride”

The brave Mr Hanna (DoF, minority report) Did not accept that the (unestimated) benefits were

sufficient to justify a programme whose cost per acre was almost twice the prevailing price of land

Better agricultural returns could be had by improving existing dry land

The programme offered a “palliative of very limited efficacy for unemployment”

Dismissed the argument of indirect benefits, pointing out that investments in housing and primary education would be far more valuable

Concluded that adoption of the programme would involve “a dissipation rather than a creation of national wealth”.

Catchments should only be drained where returns commensurate with the cost of construction and maintenance could be obtained, and state should never contribute more than 60% of construction cost.

Government decision

Government adopted an arterial drainage programme covering twice the area recommended by the Commission (1.2 million acres rather than 600,000 acres)

28 major catchments, ranked by severity of flooding

Thirty years later, Department of Finance Appraisal Team (1968) noted that, often, cost of drainage exceeded full post-drainage value of affected lands and recommended halt to programme until full cost-benefit appraisal of future schemes

Hence introduction of CB analysis for arterial drainage in 1970

The OPW methodology

Primary benefit: increase in landholder income Secondary benefits: spin-off employment and

incidental benefits to non-farm community Direct costs of arterial drainage Cost of follow-up field drainage and investment in

livestock and buildings associated with the output targets

Environmental impacts not costed but reported in EIA

Fifty year time horizon and discount rate of 3.5% used

Difficulties with OPW methodology

Alternative measures of gains to landholder income Improved vs unimproved value of land NPV of additional income Requires assumption (90%) re proportion of

farmers that will adopt the follow up field drainage

Either estimate very sensitive to base year given volatility in prices/incomes

Treatment of secondary benefits Implications of cost overruns

Bruton and Convery recommendations Found evidence of diminishing returns – the

more cost-effective schemes already undertaken

Undertake more marginal analysis of schemes Introduce some cost-sharing to reduce demand

for what is perceived as a free public good Drop estimation of secondary benefits

(assuming zero shadow price of labour) Do more ex-post analysis Equalise rate of grant among competing

Department schemes

EXAMPLEEVALUATION OF AGRICULTURAL RESEARCH EXPENDITURE

Benefits from public sector R&D

Investment in agricultural research => changes in technology =>increases in agricultural productivity

Research as a public good Measuring the benefits from research

Estimating consumers and producer benefits using a supply and demand model of a commodity market

Using regression analysis to estimate parameters of an aggregate production function model

Both ex-ante and ex-post applications

Issues in measuring R&D benefits

Assessing the relationship between the size of investment in R&D and output or productivity

Identifying the relationships between increased productivity flows and economic benefits

Accounting for the timing of streams of benefits and costs, given lengthy lag times

The basic supply-and-demand model of research benefits

S0

S1

D

P

Q

P0

P1

A B

C

F

GH

Issues

Applies equally to demand-enhancing as to supply-increasing technological change

Clearly identifies distributional implications Measurement issues (slopes of curves…) Taking account of market distortions (price

support, environmental externalities) Potential for understating costs (cost of public

funds…) Potential for overstating benefits (ignoring other

sources of productivity gains, only evaluating ‘winners’,…)

EXAMPLEINDECON MID-TERM EVALUATION OF IRISH RURAL DEVELOPMENT PROGRAMME 2010

Pillar 2 Rural development

As set out in the Rural Development National Strategy, the programme is designed to contribute to the following overarching objectives:

Improving the competitiveness of agriculture by supporting restructuring, development and innovation (Axis 1);

Improving the environment and the countryside by supporting land management (Axis 2);

Improving the quality of life in rural areas and encouraging diversification of economic activity (Axis 3).

LEADER Action Groups (Axis 4)

Terms of reference

Assess the degree of utilisation of resources, the effectiveness and efficiency of the programming of the EAFRD, its socio-economic impact and its impact on the Community priorities;

Answer the Common Evaluation Questions (including the horizontal evaluation questions) required as part of the CMEF guidelines set out by the European Commission;

Assess measures with respect to their balance within the programme;

Judge on the degree to which the Rural Development Programme Ireland contributes to achieving the objectives set out in the national and Community strategy;

Identify the factors that contribute to the success or failure of the programme’s implementation, including as regards sustainability, and the identification of best practice;

Propose measures to improve the quality of the programme and its implementation;

Review the goals of the programme and aim to draw lessons concerning rural development policy;

Economic context required changes in RDP At EU level, the CAP reforms (including the CAP Health Check and the

European Economic Recovery Plan (EERP)) have combined with national budgetary decisions to require significant adaptation of Ireland’s RDP.

Closure of the Farm Improvement Scheme (Measure 121) to new applicants from October 2007;

Suspension of the Young Farmers Installation Scheme (Measure 112) and the related Early Retirement Scheme (Measure 113) to new applicants from October 2008;

Reduction in the maximum hectarage limit supported under the Less Favoured Area scheme (Measure 212) in October 2008; and

Reduction in the rate of the Rural Environment Protection Scheme (REPS) (Measure 214) in 2009 and the subsequent closure of the scheme to new entrants from July 2009.

In 2009, additional funds of €146.3 million were made available under the CAP Health Check modulation agreement and under the EERP. New monies allocated to a number of new schemes including a new investment scheme (the Targeted Agricultural Modernisation Scheme (TAMS)), a new Agri-Environment Options Scheme (AEOS), a new Natura 2000 scheme and a Rural Broadband scheme.

These modifications to the programme were finally agreed in March 2010 while adhering to the requirement to maintain a 10:80:10 ratio of EAFRD funding between Axes 1, 2 and 3-4.

Primary survey research

Survey of farmers supported under the Young Farmers Installation Scheme and under the Early Retirement Scheme – beneficiaries under these two schemes were surveyed together owing to the relationship between the two schemes;

Survey of farmers supported under the Farm Improvement Scheme;

Survey of farmers supported under the Less Favoured Area scheme, the Rural Environment Protection Scheme (REPS), Natura 2000 payments and Vocational Training for REPS;

Survey of Leader Local Action Groups (LAGs) – 36 groups in total.

Overview of expenditure progression

Axis 1 of the RDP aims to improve the competitiveness of agriculture in Ireland through support for farm restructuring, development and innovation

Measures Achieving the transfer of land to young trained

farmers better able to meet the new challenges facing Irish agriculture;

Providing training to beneficiaries under the NATURA 2000 and Agri-environment measures under Axis 2 to optimise delivery of these measures;

Encouraging the transfer of holdings from older farmers to young farmers (setting up or already established);

Supporting capital improvements in farm structures to ensure that primary agriculture becomes competitive and market-oriented.

Axis 1 monitoring indicators

Measure 111 indicators

Measure 111 – Views of participants

Measure 112 – Young Farmers’ Installation Scheme

The objectives of this measure are to: Increase the amount of land transferred to

young, trained farmers better able to meet the new challenges facing Irish agriculture,

Off-set the costs faced by young people when setting up their farm, and

Provide assistance for the investments required on such holdings.

Indicators

Evaluation

While 92.1% indicated that the Scheme was very significant or significant in its contribution to them setting up as a young farmer, only 30% of respondents indicated that they would not have embarked on a career in farming in the absence of YFIS financial supports.

DAFF VFM review of scheme Cost of scheme amounted to €1 for every €3

distributed to young farmers Limited impact in encouraging agricultural

qualifications (other schemes more important) Scheme was discontinued

Measure 113 Early retirement scheme

The objectives of this measure are to complement the YFIS and encourage land mobility. Older farmers are provided with an early pension to encourage the transfer of their land to younger trained farmers.

Indicators

Evaluation

Survey results suggest low deadweight

Axis 2

Axis 2 of the Rural Development Programme 2007-2013 is designed to ensure continued agricultural land use, promotion of environmentally friendly farming practices and preservation of the farmed landscape.

80% of expenditure under the RDP comes under Axis 2.

Measure 212 – Payments to Farmers in Areas with Handicaps: Less Favoured Areas This scheme aims to avoid land

abandonment and the disintegration of farming communities in less advantaged areas by ensuring the continued use of the agricultural land in an environmentally sustainable manner.

Environmental impacts

Although little data exists to prove the level of contribution of this scheme to maintaining the countryside and improving the environment, there are clear benefits to the environment from a farmer retaining their holding, farming in compliance with environmental protection requirements consistent with their statutory obligations and complying with a minimum stocking level.

Due to this measure 3.28 million hectares are now deemed to be successfully contributing to: Biodiversity Water Quality Climate change Improvement in soil quality Avoidance of marginalisation and land abandonment.

Evaluation of LFA scheme

Land abandonment - what is happening to land prices?

Measure 214 – Agri-environmental Payments: REPS scheme

The REPS 4 scheme was designed to promote the following: Techniques for using agricultural land which protect and

improve the environment, biodiversity, the landscape and its features, climate change, natural resources, water quality and the soil;

Environmentally friendly farming processes; The conservation of farmed environments which are of

high nature value; The upkeep of historical features on agricultural land; The use of environmental planning in farming practice; The continued use of agricultural lands (even in

disadvantaged areas); The continued vitality and viability of rural

communities; The conversion to organic production standards.

Evidence of REPS success?

Difference-in-difference approach

Hynes et al (2007) examined six variables over a decade (1995-2005).

This study assumed that input use on REPS farms would develop in a similar pattern to input use on extensive Non-REPS farms in the absence of REPS.

An extensive Non-REPS farm index was developed for the reference periods for the six variables. The extensive Non-REPS farm index was then applied to the base year figures for REPS farms to establish a benchmark for activities in a subsequent year.

The proportionate difference between the benchmark figure in the chosen year and the actual figure was then attributed to the presence of the REPS programme.

Hynes et al (2007) results

REPS and biodiversity

The REPS scheme also aims to improve and promote farming practices that protect and increase biodiversity (wildlife habitats and endangered species of flora and fauna).

Copland and O’ Halloran study aimed to determine whether REPS farms had greater numbers and more diversity of birds than Non-REPS farms. Bird and habitat data were collected from 61 REPS farms and 61 Non-REPS farms from three regions in Ireland over three bird breeding seasons.

The study found that overall there was no increased diversity among REPS farms compared to Non-REPS farms and concluded that REPS had little or no impact on bird diversity and numbers.

“The lack of any significant differences in species diversity between REPS and non-REPS farms is perhaps not too surprising given the “shallow” nature of the scheme. The basic eleven measures within REPS are likely to have limited impacts on bird diversity, with many of the measures requiring habitats to be “retained” or “maintained”, rather than managed, increased, created or improved.”

Further references

DAFF Value for Money Reviews Forestry

Bacon Report 2004, A Review and Appraisal of Ireland’s Forestry Development Strategy

Clinch, P. 1999, Economics of Irish forestry: evaluating the returns to economy and society, COFORD