results of the midterm evaluation exercise on the leader + programme for portugal special focus on...

Results of the midterm Results of the midterm evaluation exercise on the evaluation exercise on the Leader + programme for Leader + programme for PortugalPortugal

Special focus on evaluating innovationSpecial focus on evaluating innovation

Pedro Afonso FernandesPedro Afonso Fernandes(CIDEC – Lisbon - Portugal)(CIDEC – Lisbon - Portugal)

Expert meeting on Guidance for Common Monitoring and Expert meeting on Guidance for Common Monitoring and Evaluation FrameworkEvaluation Framework

Seminar on monitoring and evaluation of the LEADER approachSeminar on monitoring and evaluation of the LEADER approach

DG AGRI, Brussels – Room LOI 130BDG AGRI, Brussels – Room LOI 130B25 September 200625 September 2006

Pedro Afonso Fernandes ([email protected]) 2

Innovation on evaluating innovationInnovation on evaluating innovation

• The innovative starting point

• The 3 × 3 evaluation matrix

• Specific evaluation tools

• Deal with lack of self-evaluation

• Recommendations on evaluation procedures and tools


The innovative starting point (I)The innovative starting point (I)

• Evaluating a innovative programme such as LEADER+ requires a innovative starting point at … two levels:

• Community level: Guidelines produced by STAR committee (Doc. VI/43503/02-Rev.1, January 2002)

Detailed Common Evaluation Questions and Criteria that cover a wide range of evaluation topics:

Implementation of the LEADER+ method

Implementation of the 3 actions (integrated pilot strategies, co-operation and networking)

Impact – Overall objectives of the Structural Funds

Impact – Specific objectives of LEADER+ (Value Added)

Financing, management and evaluation


The innovative starting point (II)The innovative starting point (II)

• National level: the midterm evaluator (CIDEC) developed specific evaluation questions in order to cover the main concerns of the national authority (IDRHa – Institute of Rural Development and Hydraulics), namely:

Evaluation of the LAG’s functioning and practices

Evaluation of local strategies’ appropriateness and sustainability

Evaluation of the monitoring and information systems

• Note: the specific (national) evaluation questions complement the DG AGRI’s Common Questions

Those additional questions reinforce the evaluation of LEADER’s value added (e.g. development of competences at local level, strategies sustainability, diversification effects)


The 3 The 3 × 3 evaluation matrix× 3 evaluation matrix

• Evaluating LEADER is a challenge that results from the cross of 3 levels of intervention with 3 types of action

Action 1: local strategies

Action 2: co-operation

Action 3: networking

National level (IDRHa and A.3 partners)

Local level (LAG)

Project level (promoters)

NO TARGETS


Specific evaluation tools (I)Specific evaluation tools (I)

• Multiple methodological tools were mobilized in order to deal with the 3 × 3 matrix

Action 1: local strategies

Action 2: co-operation

Action 3: networking

National level (IDRHa and A.3 partners)

Interviews Interviews Interviews

Local level (LAG)

Inquiry

Case Studies

Inquiry

Case StudiesInquiry

Project level (promoters)

Inquiry

Case Studies

Inquiry

Case Studies-


Specific evaluation tools (II)Specific evaluation tools (II)

• Case studies:

26 LAG covered (one half of the total)

Interview with LAG’s coordinator

Contact with 2 projects from Action 1 and interview

Analysis of the administrative processes (dossiers) of 10 projects (9 from Action 1 and one from Action 2)

Telephonic contact with some local partners

Documental analysis (e.g. local strategies, criteria to select projects)

Quantitative treatment of qualitative information

Detailed report


Specific evaluation tools (III)Specific evaluation tools (III)

• Promoters Inquiry:

Main objective: find values for a set of indicators specially developed by the midterm evaluator with the support of the national authority (IDRHa)

The inquiry was applied with 9 different questionnaires according to the type of project Good response rate (51%) and reliability of the data

Quality control procedures: statistical tests that compared the data from the inquiry with the data from the case studies

Impact and results indicators were favoured in order to quantify the LEADER’s value added


Specific evaluation tools (IV)Specific evaluation tools (IV)

• Examples of impacts and results measured by the Promoters Inquiry for Portugal:

63% of the projects incorporated technologies (43% ITC)

40% of the projects are innovative (product or process)

10% of the projects involved universities or polytechnics

60% of the “productive” projects mobilized local raw materials and/or services

70% of jobs created were for women

LEADER+ stimulates the development of competences on 40% of the promoters inquired and …

… the development of new different activities (no financed) on 12% of the projects – Diversification effects


Deal with lack of self-evaluationDeal with lack of self-evaluation

• Only 5 on 26 LAG studied by the midterm evaluator have permanent self-evaluation practices

• Even if the LAG has this kind of practices, an independent evaluator are not mobilized typically

• The midterm evaluator deals with those problems by involving LAG’s representatives and (selected) promoters and partners in the case studies’ works

In the future, national (or even European) authorities should develop a vademecum or guide to favour the self-evaluation by the Local Action Groups


Recommendations on evaluation Recommendations on evaluation procedures and tools (I)procedures and tools (I)

• At European level, it is important to develop the Common Evaluation Questions and Criteria, that is, “The Starting Point” by:

Incorporating new questions/criteria, namely, suggested by midterm evaluators

Simplifying some topics (e.g. the CEQ 4.1 “To what extent has LEADER+ contributed to promote and disseminate new integrated approaches to rural development (…)” was already covered by the action-specific CEQ)

Developing a Common Set of Indicators that would facilitate the impact and results evaluation at European and National levels (the Portuguese set already developed could be a “starting point” for that huge task)


Recommendations on evaluation Recommendations on evaluation procedures and tools (II)procedures and tools (II)

• At National level, it is important:

To establish targets (quantified goals) for local strategies, co-operation and networking

To develop Complementary Evaluation Questions, Criteria and Indicators in order to reinforce the evaluation of LEADER’s value added and incorporate national concerns

To develop a mix of evaluation tools that could solve the puzzle of the 3 × 3 Matrix

Case studies should be privileged because of the innovative local-based nature of LEADER’s projects

Project inquiries are essential to collect quantitative data, namely impact and results indicators

results of the midterm evaluation exercise on the leader + programme for portugal special focus on...

Documents