1

FUTURREG Evaluation

Objectives and methodology

3nd Steering Committee Meeting

Malta, 28/6/2006

2

Evaluation concepts Objectives of the Futurreg On-

going Evaluation Theory Background Evaluation Methodology Evaluation activities

3

The evaluation is a systematic and objective process which values the relevance, efficiency and effectiveness of the policy, programmes and projects in relation with the established objectives (Fahernkrog et al 2002).

The evaluation has to carry out five levels of assessment:

Relevance and Coherence

Cost and Benefit, Efficiency

Effectiveness

Utility

Sustainability.

4

Internal Coherence

Relevance Efficiency

Effectiveness

Utility / Sustainability

Actual/Future Needs

ObjectivesGeneral/Specific

Impact

Results /Benefit

Input/Cost

Activity Outputs

Relation of the evaluation process

5

Why to Evaluate? Control and Accountability – what have

been done with all the money?

Knowledge about results – what are the effects of the activities and measures? Did we really achieve what we wanted to achieve?

Learning approach – what went well, which things went wrong, why did we (not) achieve the objectives? How can we improve working in the future?

Building link between involved actors

6

ISSUES FOR FURTHER DEBATEISSUES FOR FURTHER DEBATE

•Institutional Learning in project and programme management is necessary to exploit fully the potential of evaluations.

•Training and awareness-raising for both, project and programme managers and evaluators.

•Integrate Evaluation into the project-cycle, internalise the monitoring and evaluation function.

•Free evaluation from the stigma of being an instrument for control and punishment.

7

Learning approach

Taking desitions & policy makers

building linksRegional foresight capabilities

Triangle of Futurreg evaluation functions

Suggestion for

improvementImplementation

and impact

Goal control and follow-up

8

to provide information and recommendations in order to improve the Futurreg project and to contribute to the development of a regional foresight culture.

Evaluation in conformity with the provision under Interreg IIIC normative

To evaluate the relevance and coherence of Futurreg project.

To evaluate the utility and sustainability of Futurreg as a project.

To derive practical lessons and realistic recommendations which are important for the dinamization of foresight capabilities at regional level.

9

The Foresight can be defined as a systematic, participatory, future intelligence gathering and medium-to-long-term vision-building process aimed at present day decisions and mobilising joint actions.

(FOREN, EU, 2002)

10

Complex Subject and Project, wide variety of stakeholders.

Since the project is still running, it is often not possible to identify or examine final results or impacts.

Limited visibility and sometimes indirect effects of Futurreg actions on final beneficiaries (SMEs, enterprises), leds to difficulties integrating their perspectives in the analysis to the same extent as the points of view of intermediaries.

11

Main Objectives Basic sciences Key Technologies Innovation

 Social 

     

 Industrial 

     

 Military defense 

     

Caracostas y Muldur (1998)

1945 1970 2000 2020

Evolution of Innovation policy

Futures&

Foresight

12

Policy makers’

environment

Valovirta ja Hjelt (2005)

Evolution and foresight: interrelations

- Impact assesment- technology assesment

13

Policy makers’

environment

Valovirta ja Hjelt (2005)

Evolution and foresight: interrelations

14

• Development of Evaluation Methodology• Evaluation criteria for

applications/implementation projects• Development of indicators to evaluate

each component of the project• Development of a questionnaire for the

partners• On-going evaluation• Questionnaire of satisfaction for the actual

users of the application• Interviews with the regional actors benefit

of the project• Ex-post evaluation report

15

• Methodological tools for evaluation:

– Desk research (background literature, work programmes, project implementation reports, former foresight projects evaluations, studies, statistics, etc.)

– Work research (Interviews with partnership and select stakeholders at regional level, On-line Questionnaire).

– Permanent relationship with partner (meeting, e-mail, phone). Contrast dynamics and consent.

16

• Evaluation Framework of the project:

 

Evaluation Topics

Evaluation Questions

Indicators Data Sources

Data gathering (Methods)

Main Additional

Relevance          

Effectiveness          

Efficiency          

Effects and Impacts

         

  

17

CONCLUSIONCONCLUSION

•Evaluation will give new insights on how to improve.

• Evaluation will give you information on which measures work and which don’t.

•Evaluation gives the public institutions a proof of what you have done and achieved, and legitimise what you do at regional level.

18

The contribution of Futurreg Evaluation:

Ensures the RELEVANCE, FEASIBILITY and SUSTAINABILITY of a project

Facilitates a dialogue / OWNERSHIP Ensures that fundamental questions are

asked and weaknesses are analysed Defines the key elements & the settings of a

project Identifies measurements/ indicators of the

projects achievements Systematic common sense (helping to adapt

expectations to planned expenditure to time frames)

19

Structure of the Report

• Executive Summary

• Introduction

• Evaluation Methodology

• Analysis of:– Futurreg deliveries– Futures toolkit– Regional foresight capabilities impact

• Conclusions and Recommendations on the Overall Futurreg