Monitoring and evaluation of Cohesion Policy
Austrian experience and
perspectives
Bratislava, November 30th 2010
R. Hummelbrunner, GrazAustria
Slovak Society for EvaluationSlovak Society for Evaluation
Key features of regional policy in Austria Regional policy is competence of “Länder” (regions - NUTS II) National level: coordination, orientation (Federal Chancellery) Mechanism for aligning regional policies at national level (ÖROK)
Cohesion Policy – Structural Funds in Austria 8 OPs for Objective 2 at NUTS II level, 1 OP for Objective 1 – Phasing
Out (Burgenland), 7 OPs for Objective 3 (ETC) - CBC strand Programmes at national level for Employment and Rural Development No other thematic / sectoral OPs at national level Central Monitoring System for all ERDF Programmes Austrian Conference for Spatial Planning (ÖROK) functions as network
of MAs for ERDF Programmes
Regional and Cohesion Regional and Cohesion Policy in Austria Policy in Austria
Financial Resources 2007-13Financial Resources 2007-13
ObjectiveSF in Mio €
Current prices
Convergence / Phasing Out of which ERDF of which ESF
177,1125,0
52,1
Regional Competitiveness and Employment of which ERDF of which ESF
1.027,3555,0472,3
Territorial Co-operation 256,6
of which cross-border Co-operation 223,9
of which trans-national Co-operation 32,7
Total: 1.461,1
Source: ÖROK
Co-ordination / exchange of Co-ordination / exchange of evaluations at national level evaluations at national level
Operational Programmes
Evaluation Team
Platform Evaluation (ÖROK)
Contracts
Cross-programme exchange, horizontal themes
Evaluation Team
Evaluation Team
Evaluation Team
Evaluation Team
Key innovation during 2000 – 2006: Key innovation during 2000 – 2006: On-going evaluation On-going evaluation
Voluntary activity of various ERDF ProgrammesObjective 1 Programme Burgenland, five Objective 2 Programmes, four
INTERREG IIIA Programmes, URBAN II GrazPeriod between MTE and MTE Up-date (2003 – 2005)
Implementation of on-going evaluations Integral element of evaluation contracts by MAsSpecification and steering mostly by Working Group at programme level Focus: Result oriented (impact assessment) and/or process oriented
(understanding of implementation and framework conditions)Documentation: Internal papers, minutes of workshops/meetings,
contributions for Annual ReportsExchange / reflection on process and results of on-going evaluations at
national level (ÖROK)
Evaluation components in AustriaEvaluation components in AustriaProgramme period 2000-2006Programme period 2000-2006
MTE On-going
Purpose External needs Internal needs
Requirements Reg.1260; EC-WP Managing Authority
Timing Set deadlines continuous
Focus primarily result-oriented Result & Processoriented
2000 - 2001 - 2002 - 2003 - 2004 - 2005 - 2006
On-goingMTE Up-Date.
Mid-Term Evaluation & Up-date embedded in on-going evaluation process
Mid-Term Evaluation and Up-date External requirement, timing and topics pre-definedRelevant for: MAs (e.g. use of resources, n+2, achievement of targets) Monitoring Committee (e.g. Programme changes, reallocations)
On-going Evaluation (between MTE and Up-date) Utilization-focused: Information needs/questions of programme actors Quick results for programme modifications, quality improvementRelevant for: Implementing Bodies at Measure level (e.g. funding conditions, project
selection and development, publicity and visibility of measures) MAs (e.g. implementing mechanisms, results and impacts)
Relevance of evaluation for Relevance of evaluation for steering (actors, functions)steering (actors, functions)
Experience with On-going Experience with On-going Evaluations 2000 – 2006 Evaluations 2000 – 2006
Value-added of On-going EvaluationExternal view on programme implementation, „External mirror“ for MAsNew role of evaluators: „Coaching“ of programme actors in view of
professionalising the entire programme system In-depth treatment of selected topics / areasAccumulation of insights into programme realities Processing of information in small portions, easier to handle Quick and flexible reaction on questions / problems of MAsRapid implementation of findings / recommendationsTimely reaction and adjustments at programme or measure level
Governance process for NSRF (strat.at) and nine Objective 2 OPs, to Promote learning and exchange of experience on OPs Provide ideas and practical knowledge for various stakeholders Generate contributions for future regional policy (EU, national )
Based on established networks and structures Network of Managing Authorities (within ÖROK) Steering Group (on-going definition of thematic priorities, work plan)
Main elements of governance process Work commissioned by ÖROK (evidence, expertise) Workshops to discuss/validate expert inputs Events for wider public (dissemination, information)
Innovations 2007-13: Innovations 2007-13: Strategic governance processStrategic governance process
Outline of strategic governance Outline of strategic governance process in Austriaprocess in Austria
External influence
Outputs(Projekts)
Results
Community Strategic Guidelines
Community Strategic Guidelines
NSRFSTRAT.AT
NSRFSTRAT.AT
Objectives and Strategy(Sub)objectives per Priority
Strategy fields
GOVERNANCE PROCESS NSRF(= M&E, interactive exchange, Reflection >
institutional learning)
Territorial framework conditions (physical, socio-economical, institutional)
Analysis
Use of outputs
Contributions to objectives
Regional Impacts
PROGRAMME GOVERNANCE(= Achievement of objectives, regional impacts)
OperationalProgrammes
- Priorities- Actions- Mechanisms
Inputs(Resources, Rules)
Analysis of common themes / challenges Example: Assessment of Structural Funds implementation system in
Austria („Governance Check“) Joint reflection and learning
Example: 15 Years of INTERREG / ETC programmes in Austria - ‘Looking back - looking ahead’
Comparative implementation analysis Example: Cross-programme evaluation of national funding scheme
for environment / climate change Participation in larger scale cross-programme evaluations
Example: Cross-programme evaluation of ETC programmes in Central and Southeast Europe, Operational and Thematic aspects
Innovations 2007-13: Innovations 2007-13: Cross-programme activitiesCross-programme activities
Innovations 2007-13: Innovations 2007-13: Process Monitoring of ImpactsProcess Monitoring of Impacts What is Process Monitoring of Impacts?
Tool for impact-led monitoring of an intervention Focus on processes, which should lead to impacts Use of Logic Models (impact diagrammes)
What is it for? Illustrates causal relations between programme outputs and effects
in a plausible manner Enables a concise, systemic overview of impact patterns of
complex interventions, e.g. programmes Provides early information for programme management (MAs, IBs)
required for effective steering
Monitoring of change processesMonitoring of change processes
Impact chains as basic scheme
Inputs are used to produce outputs
(= projects)
Outputs are used (by someone, in a specific manner) to reach results
Results will lead to impacts (intended /
unexpected) in a plausible manner
Inputs Outputs Results Impacts
Financial Table Typology of projects (output) to be supported
Objectives at the level of Areas of intevention / Activities
Objectives at thelevel of Priority or Program
Result-indicators
Output-Indicators
Commitments
Disbursements
Impact diagramme AoI A 4.1 – Impact diagramme AoI A 4.1 – Softmeasures to strengthen Softmeasures to strengthen
innovations in entreprisesinnovations in entreprises
Enterprises (especially SME) carry out product finding processes
Enterprises (especially SME) conceive innovation-/ investment projects
New / improved products, services and production processes
Enterprises (especially SME) introduce new technologies
Enterprises (especially SME) collaborate in networks (also with large enterprises)
Increased linkages between enterprises at regional level / scale
Enterprises (especially SME)gain new markets
Adaptation to international competition
Increase in employment / new jobs
Services to sensitize for innovations AF6
External expertise, process consulting
AF6
Advise for co-operation
AF6
Enterprises (especially SME carry out reorganization processes Sustainable stabilization of
enterprises
New contacts with clients/ new orders
Pilot applications with programmes Pilot applications with programmes Reg. Competitiveness 2007 - 2013Reg. Competitiveness 2007 - 2013
Programme Styria Based on impact diagrams established during programme preparation Application with 5 Areas of Intervention (of totally 11) Project sample (used for up-dating of impact diagrams): 70 Projects approved (used for weighting of impact chains): 840 Duration of pilot application: July - November 2009
Programme Carinthia Internal assessement of projects approved so far (ca. 40) based on
impact diagrams (dating from ex-ante evaluation) Assessment accompanied externally Finalization (Report) October 2009
Application with programme Application with programme Styria 2007 - 2013Styria 2007 - 2013
Procedure / steps 1. Up-dating impact diagrams from the programming process (based on
current version of internal „Program Complement“ )2. Attributing all (840) projects in line with their primary impact chain
(basis: short descriptions contained in ERDF Monitoring system, information provided by IBs)
3. Weighting of impact paths according to number of projects and financial volume, calculation and graphic representation
4. Integrating indicators (from ERDF Monitoring database) to provide information on final achievements of processes
5. Validating and complementing impact hypothesis through interviews with (selected) project owners
6. Discussing findings / conclusions with IBs (Funding Authorities)
Impact diagramme AoI 4.1 – Impact diagramme AoI 4.1 – Softmeasures to strengthen Softmeasures to strengthen
innovations in entreprisesinnovations in entreprises
Enterprises (especially SME) carry out product finding processes
Enterprises (especially SME) conceive innovation-/ investment projects
New / improved products, services and production processes
Enterprises (especially SME) introduce new technologies
Enterprises (especially SME) collaborate in networks (also with large enterprises)
Increased linkages between enterprises at regional level / scale
Enterprises (especially SME)gain new markets
Adaptation to international competition
Increase in employment / new jobs
Services to sensitize for innovations AF6
External expertise, process consulting
AF6
Advise for co-operation
AF6
Enterprises (especially SME carry out reorganization processes Sustainable stabilization of
enterprises
New contacts with clients/ new orders
2
2
143 40
65
9
40
25
Impact diagramme AoI A 4.1 – Impact diagramme AoI A 4.1 – Softmeasures to strengthen Softmeasures to strengthen
innovations in entreprisesinnovations in entreprisesWeighting according to financial volume (public funds)
Enterprises (especially SME) carry out product finding processes
Enterprises (especially SME) conceive innovation-/ investment projects
New / improved products, services and production processes
Enterprises (especially SME) introduce new technologies
Enterprises (especially SME) collaborate in networks (also with large enterprises)
Increased linkages between enterprises at regional level / scale
Enterprises (especially SME)gain new markets
Adaptation to international competition
Increase in employment / new jobs
Services to sensitize for innovations
AF6
External expertise, process consulting
AF6
Advise for co-operation
AF6
Enterprises (especially SME carry out reorganization processes
Sustainable stabilization of enterprises
New contacts with clients/ new orders
127.900,00
439.728,68
3.620.240,85
1.652.266,92
789.269,49
1.084.671,36
352.997,53
436.771,66
Pilot applications with programmes Pilot applications with programmes Territorial Co-operation Territorial Co-operation
INTERREG IIIA programmes (2000-2006) On-going evaluation Austria – Slovenia Ex-post evaluation of Swiss participation in INTERREG III
ETC Cross-border Co-operation Programmes (ex-ante evaluation) Austria – Slovenia Austria – Bavaria Lake Constance (AT / DE / CH / LIE)
ETC Trans-national Co-operation Programmes (ex-ante evaluation) Central European Space South East European Space
Studies / Work for INTERACT downloadable at www.interact-eu.net/mt
Results and experience Results and experience gained so fargained so far Main results of method:
Sound and quantifiable connection between supported projects and effects at programme level (expected results, impacts)
Systematic capture (respectively clarification) of project types and categorisation of their use dimensions
Capture and representation of linkages (within and between Areas of Intervention, with other programmes / interventions)
Focus on supported project types and their use, thus early indication of likely achievement of impacts
Limits of the method: Provides assumptions on impact paths, but does not capture the actual
achievement of results / impacts However, they can be integrated (once achieved), e.g. via data input from
ERDF monitoring system, surveys, project reports
Evaluation of OPs Evaluation of OPs 2007 - 2013 2007 - 2013 National Evaluation Plan 2007 – 2013 (gradually specified)
Preparation by ÖROK, imbedded in Governance process of NSRF
Combination of considerations of MAs and national-level actors
Considerations by MAs for on-going evaluations
Utility for MAs in implementing their tasks (e.g. Annual Reports, justifications for programme changes)
Strategic assessments (e.g. achievement of objectives/results, mid-term reviews)
Operative assessments based on data from Central ERDF Monitoring System (often done internally)
Work commissioned ad-hoc, programme specific approach
Richard Hummelbrunner
ÖAR - RegionalberatungAlberstrasse 10,
8010 Graz, AUSTRIAT: +43/316/31 88 48
Thank you for your attention
