regional policy how are evaluations used in the eu? how to make them more usable? stockholm, 8...

Regional PolicyRegional Policy

How are evaluations used in the EU? How to make them more usable?

Stockholm, 8 October 2015

Kai Stryczynski, DG Regional and Urban Policy

Unit Evaluation and European Semester

Cohesion Policy

1


Evaluations in a system of shared management - responsibilities

• Ex ante – Member States (MS)

• During the programming period – MS (and Commission)

• Ex post – Commission with MS

2

3

Focus of the Regional and

Cohesion Fund

evaluations, 2007-13

programming period


Lessons from 2007-2013: Evaluation

• Great diversity in practice and quality among 800 evaluations of Regional and Cohesion Fund supported interventions carried out between 2007-2012 and 721 evaluations of the ESF up to December 2013

• Limited but growing numbers of evaluations dealing with effects and impact

Cohesion Policy

4


Types of Use of Evaluation

• Instrumental (implementing recommendations)• Conceptual (helping policy makers to

understand)• Enlightenment (accumulating knowledge)• Process use (organizational changes)• Persuasive or Symbolic (accountability or

legitimization)

Saunders 2012

5


Maximising the Impact of Evaluation

• Research uncertain – no one way to maximise impact

• Use is diverse, unpredictable and can be political

• But, we can expect maximum influence when it is relevant, timely and of high quality

6


We are becoming better – but still have a quality problem

• Insufficient attempt to examine logic of interventions.• Deficiencies in the design and use of methods.• Terms of Reference often poorly specified, Sometimes

bias in the Terms of Reference, with indications given in the Terms of Reference of the conclusions to be drawn.

• A lack of public discussion of findings• (Gaffey, Riché, 2014)

•

7


Objectives in OPs – what do you want to change?

• Each priority axis shall articulate what it wants to change. What motivates the policy? Specific Objective

• Observe whether the change is taking place. How to know if policy succeeds? Result Indicator

In other words: the subject of future evaluations


9

..because too often it was not clear

"Lack of clarity in policy documents …on the precise objectives which programmes …were intended to achieve and how their achievement was to be verified"

Ex post evaluation 2007-2013, synthesis report, p157

One of the triggers of result orientation 2014 - 2020

Cohesion Policy


10

Link to policy monitoring

• Result indicators capture the development of the problem to be fixed – in a region,…

• …and should be monitored

• Interventions are set to contribute to solution – need for evaluation to estimate contribution

• Final step for evaluations – look at contribution and overall picture

Note: Values for beneficiaries are also needed!

Regional PolicyCohesion Policy


Impact evaluations

What is required by Regulation?- Evaluate contribution of OP to objectives of each

Priority Axis

What does this mean?- Each Specific Objective should be covered- To differentiate across specific objectives the rigour –

be demanding for new - important objectives - Theory-based and / or counterfactual evaluations, CBA,

…

Cohesion Policy

11


Evaluation plans

• Legal requirement – decided by monitoring committee

• Recommendation on content by Commission

• What we hope for from advanced regions:• Innovations• Counterfactual evaluations• Build evaluability into programmes


Evaluation plans (2) think about Usability

• Built into the design of the evaluation• Requires reflection on

Reasons and purposes of evaluation Potential users Evaluation objectives The nature of data and evidence Audience for the output Timing of release of evaluation output Agency undertaking the evaluation

Saunders 2012

13


14

Steps towards meta-evaluations (?)

•Legal requirement on COM to summarise evaluations from MS

•How to do it?

•Quality assurance?

•Use for future thematic work – ex post evaluation

Cohesion Policy

15

Further EC Support

REGIO/EMPL have put in place a Helpdesk:

•To review evaluation plans

•To provide targeted methodological support for selected evaluations/Member States who request support

•To organise trainings for managing authorities

Cohesion Policy

Further EU support (2)

CRIE: Center for Research on Impact Evaluation

-Focus on counterfactual impact evaluation-Capacity building of MS: training and practical advice-Regional Workshops on CIE Initiative -Accompanying pilot projects in different MS-https://crie.jrc.ec.europa.eu

16

Ex ante evaluations – some question marks

• Are ex ante evaluations still equally justified given the emphasis of impact evaluations during the programming period?

• Did ex ante evaluations deliver value for money? Was result orientation understood / explained well enough? Were evaluators critical enough? Were findings used by programmers?

Governance (responsibility) optimal?

17

2007-2013 ex post evaluations

Cohesion Policy

18

• Ambitious excercise- series of thematic evaluations and synthesis until spring 2016

• Aim of ex post evaluations- analyse the extent to which resources were used, effectiveness,

efficiency and impacts of funding

• Expected use- for accountability purposes to the Court of Auditors, European

Parliament, etc. - for learning purposes for '14-'20 and beyond.

Open data – transparency

• Open Data pilot project https://cohesiondata.ec.europa.eu/

• Data for researchhttp://ec.europa.eu/regional_policy/en/policy/

evaluations/data-for-research/

19

20

Evaluation Conference 2016EU Cohesion Policy at work: a results-oriented policy

Zagreb, 9-10 June 2016

Cohesion Policy

21

Competition for best evaluations and evaluation proposals

Completed evaluations by any individual or company of the effectiveness of EU Cohesion Policy interventions supported by the European Regional Development Fund, the Cohesion Fund or the European Social Fund.

Use at least one of 3 methods:Ex post cost benefit analysisTheory based evaluationCounterfactual evaluation

Deadline for the entries: 31 December 2015

For regular updates on the competition and the 2016 conference, visit the page: http://ec.europa.eu/regional_policy/impact/evaluation/index_en.cfm

Or contact:

[email protected]

Proposal for an impact evaluation of Cohesion Policy

Evaluation not started

Preferably using data in a new way – data on the DG REGIO or EMPL websites or data from a national or regional authority

Open to all individuals and participation of young researchers and PhD students is strongly encouraged

regional policy how are evaluations used in the eu? how to make them more usable? stockholm, 8...

Documents