1

FINAL REPORT

EN

GL

ISH

VE

RSI

ON

20

13

IN I

TIN

ER

E E

VA

LU

AT

ION

E

VA

LU

AT

ION

RE

PO

RT

20

12

P a r c o u r i r L ’ E u r o p e S a n d r a U n g e m a c h - B e n s a i d ,

J e a n n e G e o f f r o y , D a n i e l P o u l e n a r d

9 r u e d e l a L a q u e

3 1 3 0 0 T o u l o u s e

w w w . r e s e a u - p a r c o u r i r . e u

2

CONTENTS:

I-EVALUATION PROCESS, METHOD RECAP........................................................................................... 3

An interactive, iterative, evolutionary evaluation ........................................................................... 3

Meta-indicators ................................................................................................................................ 6

II- EVALUATION PROJECT ANALYSIS ........................................................................................................ 10

Traditional Projects ........................................................................................................................ 10

Strategic projects ........................................................................................................................... 16

TARGET PROJECTS AND capitalisation ........................................................................................... 18

III - EVALUATION: PROGRAMME ANALYSIS .................................................................................................. 20

RECAP OF THE SITUATION AT THE mid-term review ..................................................................... 20

Programme adaptability ................................................................................................................ 23

ACTIVE TRANSITION TO A NEW PROGRAMME .............................................................................. 29

IV- SUMMARY OF EVALUATIONS ....................................................................................................... 31

V -RECOMMANDATIONS........................................................................................................................... 34

FOLLOW-UP of RECOMMENDATIONS MADE during THE mid-term EVALUATION (2011) ............ 34

Closing OF THE CURRENT PROGRAMME ....................................................................................... 38

DEVELOPING THE FUTURE PROGRAMME 2014-2020.................................................................... 41

Implementation tools..................................................................................................................... 42

Programme governance, communication ..................................................................................... 49

VI ANNEXES ........................................................................................................................................... 52

List of notes 2010-2011-2012: ....................................................................................................... 52

Interview guide for finalised projects. ........................................................................................... 55

List of interviews ............................................................................................................................ 58

Definition of selection critera of traditional projects .................................................................... 59

3

I-EVALUATION PROCESS, METHOD RECAP

AN INTERACTIVE, ITERATIVE, EVOLUTIONARY EVALUATION

Diagram of general changes to the MED programme 2009-2012

Evaluation work allowed a genuinely interactive, iterative and adaptable approach for

the following reasons:

Three full years of work allows several series of interviews and project / programme

analysis at different stages of their life. Work began at the same time as selection for

the second call for traditional projects in 2009 and ends after the projects from the

last call for projects have been selected (capitalisation projects, beginning of 2013).

This time-scale allowed measuring the extent to which recommendations had been

followed from the end of the first stage in mid-2011.

The programme changed significantly over these three years (see diagram below).

On the project side, calls for strategic projects were developed and implemented. As

well as 'targeted' projects and a final call for (capitalisation) projects. On the

programme side, capitalisation was launched. It should be noted that only some of

these changes were part of the programme's initial agenda.

On the governance side of the programme, new tools were developed and an

important number of meetings with Member States (in particular about strategic

Project approach Programme approach

In itin

ere

evalu

ation

Cap

italisation

wo

rk

Wo

rks Task

Force

MED PROGRAMME: Member States + MA + JTS

Develo

pm

ent o

f new

missio

ns at th

e JTS

Call 1 for traditional projects 2009

Call 2 for traditional projects 2010

Calls for Projects 1 & 2

Strategic projects

2010 & 2011

Call for targeted projects

2011/2012

Call for capitalisation projects

2012

Report 1

March 2011

Recommandations

June 2011

Report 2

May 2012

Final report March 2013

4

projects) took place. Since the beginning of 2012, Task Force meetings to prepare

the 2014-2020 programme were also important forums of discussion. Significant

changes also occurred in the composition of JTS/MA teams and Member State

representatives.

Throughout the evaluation process, the programme partners, JTS/MA and Member

States, showed themselves to be open and formed a constructive partnership with

the evaluation team. This close partnership in all areas of work allowed for a highly

interactive approach. We are very grateful to them.

All of these transformations meant evaluation had to undergo substantial adaptations.

The main stages of the evaluation and deliverables followed the terms of reference

set out at the application stage.

A system of working documents (16 documents - see report annex) was introduced.

This has mainly responded to three types of needs:

o Requests from the Monitoring Committee: For example to focus on 'transport'

projects in order to provide information for a possible change in the original

financing plan.

o To anticipate the analysis, conclusions or recommendations of the annual

reports 2010, 2011 and 2012.

o To share the evaluators' point of view on new programme issues: for example,

concerning the drafting of the call for Capitalisation projects or the work of the

‘Future MED' Task force.

Meetings: In addition to the work and documentary analysis, the evaluation team attended

many meetings:

Types Numbers

The annual events of the programme in this period in Thessaloniki, Barcelona, Nicosia.

3

Selection Committees and preparatory meetings with applicants 6

Programme Monitoring Committees (Greece, Spain, Cyprus, Portugal)

4

Task Force and Preparatory Working Group meetings

6

Technical meetings with the MA and JTS

12

Meetings with Member State representatives

8

Evaluation meetings: 38 meetings were conducted to evaluate 'traditional' and 'strategic'

projects in 2010, 2011 and 2012.

For each evaluation report (2010, 2011, 2012) the following was taken into account:

5

Work planned under the terms of reference of the evaluation.

Specific evaluation adapted to the scheduled development of the programme and

projects.

For this 2012 report the specific work was:

1. Drawing up the meta-indicator grid (part I)

2. Evaluating all projects finalised before the end of 2012 Evaluation interviews were

organised with the support of Member States in Marseilles, Thessaloniki, Florence

and Madrid. In particular this evaluation covered project results, outputs and

deliverables and sustainability of partnerships (part II).

3. Continuing the evaluation of strategic projects (part II)

4. Evaluating programme governance and changes observed over the period (part III)

5. Evaluating monitoring tools and methods put in place by JTS to monitor projects (part

III)

6. Evaluating the extent to which recommendations were followed over the three years

(part IV)

7. Recommendations for the end of the programme and the next programme (part IV)

6

META-INDICATORS

As suggested in the initial evaluation process, we developed a range of indicators. This process of developing and using indicators continued throughout the evaluation process. Some of these indicators were used to select the first projects evaluated and to draft evaluation documents. We decided to develop a 'meta-indicator' grid alongside existing programme indicators and the strategic evaluation methodology proposed by the evaluation team. These indicators come from observations and analysis that have been confirmed by the work done over these three years. The purpose of this process was to leave behind the dissatisfaction experienced by some of the programme managers in using the simplistic indicators that are in place today. In most cases they only give quantitative information about realisation/results and very little information about whether the right choices were made by the programme. The objectives of these indicators are:

To facilitate any future analysis of the impact of projects and the existing programme.

To provide information for developing a logical framework for the future programme and to improve instruments for measuring the programme's impact.

To be incorporated in future application and management evaluation tools for the next programme.

Process for developing meta-indicators

Selection of output / result indicators of

MED projects MED programme impact indicators

Additional indicators

Development of impact meta-

indicator, fed by base indicators +

additional documentary research &

expert opinion + feedback

projects’ analysis

Weighting + marking of meta-

indicators with specific criteria

Measurement of programme

impact

7

The selected meta-indicators: General principle: A meta-indicator is mainly the result of cross-referencing traditional indicators. For each meta-indicator we opted for: (1) A definition, (2) An objective, (3) How and why should it be used?

A. Relevance to partners/project activities and objectives: 1. This indicator is one of the most important as it represents the first level of coherence

of a given project. The same activity will produce very different results when worked on by a partner with the skills and capacity (legal, financial, technical) than by a partner without these.

2. This indicator is key to measuring the impact of the programme in a given cooperation area and meeting the OP’s objectives. Project and programme results depend on the 'quality' (type) of actors involved.

3. It can be used when selecting an application, while following-up the project implementation and for the capitalisation phase.

B. Form of results: 1. The first definition must be to place limits on what is considered to be a result. This

means listing the expected results given the partners involved and the programme resources (budget, duration and so on).

2. The objective is to move from demanding 'results' alone to a typology of results. 3. This indicator must be used before launching calls for projects (communication) and

after, to categorise and measure these results.

C. Taking into account the 'territorial' local/translational (territorial added value). 1. This indicator requires information from the territories (regions) and a macro-regional

analysis of each theme. It is the primary programme indicator and the main one in terms of differentiating between European Territorial Cooperation and the Commission's Thematic Programmes. How will the means used and the expected results of a project have an impact and on what scale?

2. The objective would be to make territorial added value a performance requirement. Each partnership must have a process, at local, regional or Med level to implement this added value (mutualisation results). There is significant room for improvement in this area as a substantial proportion of current partnerships only have a geographical 'coverage' rationale.

3. Territorial added value must be sought at each stage of project and programme implementation (programme communication, calls for projects, capitalisation and so on).

D. Relationship between target groups/results. 1. It is a key indicator for the consistency between results and the groups who can use

these results. Displaying these two factors separately in existing projects leads to many difficulties where the relevant project results have not reached the right people. We have to leave behind the idea of partnership by 'default'.

2. The objective is to incorporate the 'final' beneficiary in the project life cycle (concept of user committee).

3. This indicator must be used before launching calls for projects (communication) and at selection.

8

E. Overall cost-effectiveness by project. 1. It is about having data to compare MED projects’ results with sources of funding and

resources other than those provided by the programme. What type of results with which means?

2. The objective of this indicator is to measure the commitment of partners. 3. It is mainly used to select applications and evaluate results.

F. Programme Performance. 1. It must be possible to measure programme performance both by traditional means

(total resources used in relation to a result). It can also be evaluated by its capacity to measure the programme's achievements through monitoring projects (using the indicators cited) and derive the information necessary for a feedback.

2. The objective is to apply a quality assurance process to the programme.

G. Effectiveness of monitoring. 1. Information from projects must be part of a continuous process, for example using a

scorecard for each project, developed at the application stage. Initially, the only information that was compulsory was either financial in nature or related to the activity implementation. Other information relating to results, outputs, and target groups was given on a declarative mode. Today, thanks to the support from project officers, communication and the capitalisation process, this situation has been partially transformed.

2. The objective is to develop this process by selecting the data that allows key information to be obtained without adding to the project monitoring burden. This information forms the basis of the development of meta-indicators.

3. Such a process must be part of an overall approach to the programme to allow the information gained from monitoring the projects to become indicators for continuously measuring whether actions undertaken by the programme have been successful.

Following page: Table illustrating the information to be taken into account to evaluate a

project with a meta-indicator. Obviously the cross-referencing of several meta-indicators can

constitute a grid for evaluating projects.

A single meta-indicator cannot address project quality.

9

Example of a meta-indicator, “territorial added value”: Method

It is mainly the added value for all or part of the cooperation territory, not just a locally recognised

added value.

Information to be

taken into account

In analysing the

activities and results

of a project or the

application

Example of questions**

(all suggested examples have been encountered in the MED

projects evaluated, to ensure anonymity, they are not named)

Potential

territorial

added value

Partners Project partners are representative of a single economic sector This

sector is strongly represented in the area of cooperation.

Partners (e.g. towns) are directly responsible for managing a given

issue (waste, urban transport?).

High

Partners do not have the right skills to tackle the issue presented.

E.g. universities in a subject involving forest or river management?

Average

Partners do not have the right skills and tackle a Europe-wide issue. Low

Activities The activities are part of a described and specific strategy with

realistic intermediary objectives, and a coherent budget

(experimentation for a common standardised guide)

High

Activities differ from one partner to another depending on the usual

work and skills or partners.

Low

Expected results,

outputs

Project outputs and deliverables mean progress in a specific area

(for example, a database on the scale of the cooperation area or the

area affected by the project)

High

Project outputs vary due to the differing circumstances of partners. Average

Target groups Several target groups are identified according to the type of results.

Economic stakeholders, local representatives, researchers, residents

of a given geographical area

High

Specific work (study) is planned as part of the project to identify

target groups.

Average

Target groups are not identified and/or accessible for the type of

results suggested.

Low

Sustainability Sustainability relates to the project results and outputs. It is part of a

recognised formal framework (e.g. a partnership agreement of the

relevant sponsors and operators to manage a database)

High

The suggested sustainability involves maintaining or extending the

partnership to apply for a new programme (e.g. go from INTERREG

to Life) for an environmental topic.

Average

Statement of intent for after the project Low

** suggested questions are not exhaustive. It is about illustrating whether a meta-indicator is present in a

project

10

II- EVALUATION - PROJECT ANALYSIS

This part is based on projects’ feedback and opinion as well as JTS project monitoring. He purpose here is to highlight a bottom-up approach. A decision was made to prioritise traditional projects (first and second call for projects), as these are the only one that allow the whole project life cycle to be analysed, from the application to closing. In this report, although all phases of the project cycle are addressed, the analysis will focus on results, sustainability and capitalisation. The application phase will only give a recap of previous reports. The monitoring/implementation phase will only give a recap of changes in perception of project beneficiaries. Analysis from the evaluation of traditional projects has led to conclusive observations (see part IV Summary Table), which are a precursor to the recommendations set out in the last part of this report. The analysis from the evaluation of strategic projects (although they do not cover the whole of the project cycle) also feed into these conclusive observations. Projects from calls for targeted and capitalisation projects will only be addressed from the point of view of programme tools and governance. Indeed, we were unable to conduct analysis based on interviews, due to the selection calendar for these calls for projects. Nevertheless, an analysis of documents allowed us to draw initial conclusions about these specific projects.

TRADITIONAL PROJECTS

Methodological reminder

Principles:

Projects were selected as part of the calls for traditional projects.

Similarly, as the projects were at different stages, we limited the project

selection criteria to the design and selection of projects.

The range of projects is derived from a multi criteria analysis defined by table

(see annexes)

Closed Projects. • The projects that have been evaluated at this stage are those that have

closed. • Evaluations are based on the analysis of closing reports (for projects having

finalised their report and sent it to the JTS), a questionnaire sent to the project leader, interviews with the project leaders and/or project partners. Moreover, interviews were conducted with JTS project officers in order to cross-reference information.

• In addition to the quantitative and qualitative information that is specific to the project itself, the main objective is the analysis of interactions between the form of projects, their end results and impact in the short and medium term, the scalability of monitoring/management feedback, the global strategies of project partners (positioning of the project within an organisational strategy) and the territorial added value.

• 38 projects have been subject to various stages of evaluation (see annexes).

11

The questionnaire and interviews This questionnaire and interview guide (see annex) are broken down into several parts, each of them allowing the following points to be analysed: Questions about results, outputs and target groups, impact: the purpose of this part is to assess the projects against their short/long term objectives; to categorise their outputs and observe any discrepancy with the results forecast in the application; to analyse the type of target group (group involved with delivering project activities and group at whom the results are aimed); analyse the type of short term impact and planned impact at the end of the project as well as in the medium and long term. Questions on 'sustainability': the purpose of this part is to categorise the type of actions undertaken or planned to ensure the sustainability of results. Questions on the monitoring and management of projects by programme authorities: this part allows the point of view of project sponsors with regards the monitoring and changes they may have experienced throughout the programme to be analysed. Comments arising from project interviews and interviews with project monitoring managers at the JTS. The closing reports, questionnaires and analytical documents produced by the programme also contributed to building on these observations.

Results drawn from the evaluation of traditional projects

The previous report (report 2011) set out the initial results drawn from the evaluation of a few projects that were finalised in 2011. Interviews conducted in 2012 allowed these results to be clarified, the initial conclusions to be confirmed or corrected for all phases of the project cycle.

Source: Alpin Space programme

12

The Application As a reminder, the following observations were made after the first round of interviews:

'The application and selection process have on the whole been called into question.

Although, according to some partners, the two-stage selection process allows the activities

and objectives to be defined in a satisfactory way (project maturing between partners thanks

to these two phases), the application format is unsatisfactory according to most partners. It

does not allow the practical nature of the project to be reflected and does not allow partners

to develop the logical framework of their activities. This is true due to the nature of

information requested but also due to the strict word limit. For a large proportion of partners,

the application process does not adapt to the project, on the contrary, it is the project that

has to adapt to the application process. The use of the tool PRESAGE to post information on

line is seen by some to be an additional technical constraint and waste of time.

In addition to the application process, most partners find the selection process problematic. It

is too long and seen as non-transparent, particularly due to the lack of information given to

applicants in this phase. The evaluation of projects is felt to be formal with the majority of

partners expressing regret that there is no real technical evaluation of their application. '

It should be noted that the application form has not changed with the different calls for

traditional projects. Nevertheless, the projects from the second call for projects were less

critical of the selection process (and in particular considering the deadline).

Implementation. Reminder of analysis from the previous report 'At the end of their project, most partners still consider the project administration to be too onerous, the activities involved in this administration often take too much time to the detriment of technical achievements. This is in part attributed to the on-line administration using the Presage tool: although the application improved between the beginning and end of the project, the partners asked are still not convinced by the cost/time/efficiency benefit of this form of management. The majority of projects thus say that a very significant amount of time, sometimes 50% of the time of the people directly responsible for the project, is spent on administration. This situation is deemed to be detrimental to the development of the technical activities of the project. Another example of red tape raised by most people: a certification system that differs from one Member State to another, in addition to the length of time taken to repay expenses, can lead to cash-flow problems that are a real handicap for partners. However, we can offset this threat with the fact that none of the projects asked said they had abandoned a planned activity or output for reasons of cash flow. ' The last interviews conducted for the purposes of this report slightly corrected the observations made by the projects at the end of 2011. The vast majority of projects has highlighted positive changes in their project implementation. Positive change firstly in terms of procedures which are less time-consuming (in particular in terms of entering expenses into Presage). After quite a long period of adaptation, the project partners did take ownership of the tool, which they see as a useful way of archiving their expenses in particular. However, this change is offset by the on-line modification procedures which are generally seen as very

13

onerous. So, despite a slight improvement, administration still takes up too much time compared with technical activities. This positive vision also relates to monitoring by JTS project officers, who are seen as being more effective (in terms of the quality of answers for example). Observations about the red tape in the certification system and the refund process remain a constant of closed projects. Results, closing

The table below sets out the types of outputs (deliverables) and results observed in the 2011

and 2012 surveys.

Outputs (considered by the projects to be the

major outputs)

Form of results

- Methodological or technical guide - Benchmarking or diagnosis studies - Strategic document setting out the

components of a Mediterranean strategy - Method document (European charter) - Road map (Methodological tools) - Tool box - Database - On-line platforms

- Creation of a territorial cluster of public/private companies and stakeholders connected with the theme.

- Obtaining a formal legal status - Implementation of a shared database of tools and

knowledge - Building and coordinating a network to implement

common policies on key themes (transport, culture, environment, and agriculture) related to the development and cohesion of a Mediterranean Euroregion.

- Introduction of local pilot schemes which are all related to forms of innovation, whether of a technical, organisational or even territorial nature, through the transfer of know-how from one region to another.

- Increased stimulus for territories through public-private partnerships

- Public policy governance tools (action plans, strategic plans and implementation plans)

- Transferable model of governance

- The results of projects are perceived very differently by project partners and JTS

project officers. Although the projects are unanimously satisfied with their results and

consider that they have met their objectives, project officers are, on the whole,

disappointed. This discrepancy is based on cross-cutting observation: the qualitative

evaluation of projects is mainly based on the closing report, the existence of

deliverables and how these meet the objectives in the application. The technical

quality of outputs is not subject to investigation (internally or externally). Some

projects are critical of this as they see the absence of technical evaluation as a failure

to value their results. But this situation also allow projects that are used to

cooperation programmes (remaining mainly administrative management

programmes), to make up for the quality of their deliverables.

14

Project case study: Case of projects whose final output is an on-line resource:

Aside from the qualitative aspect it should be noted that most database or on-line resource

type outputs are faced with the following issues:

o No longer updated after the end of the project

o On-line resources are not exhaustive

o In some platforms, the on-line resource does not work

- Another observation: the responses of project partners in interviews and questionnaires do not reveal noteworthy trends allowing the specific added value of project results for a partnership or target group to be determined. Indeed this relationship is never spontaneously raised by project sponsors in interviews. Indeed, although they are aware of belonging to a certain profile of projects, it is not the case when there is a need to determine with them the specific partnership typology for their project.

- Transnational nature of results or intensity of cooperation Although the lead partners asked, mostly agree that partners were involved in project implementation in a satisfactory way (delivering activities in accordance with the work plan provided in the application), they have grater difficulty providing evidence of the transnational value of outputs. This difficulty thus reflects the overall weakness observed in applications in terms of defining the transnational added value of projects.

Case of projects where one of the final outputs is a diagnostic study: Most of these types of output are a compilation of local studies without any transnational analysis. Case of projects where one of the final outputs is a technical resource (guide, on-line resource) that can be used by all partners: Although in its very function, the output is transnational (as it can be used by all partners and is relevant for their territory), in most cases, this output is produced at the instigation of a single partner and cannot be considered to be a transnational output.) Capitalisation/diffusion The attitude of projects towards capitalization and dissemination of results is mixed. all the projects consider that they have capitalised and disseminated their results through project activities (closing seminars) and internal tools (web sites and regional or transnational distribution channels). However, these statements should be counterbalanced with the range of dissemination activities: in most cases the final seminars are local seminars without a desire to share the results beyond the project territories or with other cooperation projects with identical/similar themes. Only a third of projects asked recognised that they had taken part in the capitalisation process (activities and call for projects) organised by the programme, although this process evolved subsequently. Beyond the several events dealing with capitalization, the last call for projects (apart from the call for capitalization projects) highlighted the involvement of projects’ partners, through the amount of activities dedicated to capitalize their results. This evolution was made possible especially because of a better taking into account by the of the capitalisation strategy described in the application form during the selection process. In parallel to this relatively low participation rate, comments from projects reveal a poor understanding of the objective of capitalisation, particularly at the start of the process. Clustering, for example, did not immediately reveal benefits for the projects. Indeed, a great

15

amount of projects reported in interview that they did not feel to fit into the different clusters. But the announcement of the call for capitalisation projects radically changed this tendency. Indeed, the projects saw the clustering process as an opportunity to create partnerships, allowing them to participate in this call. Nevertheless, the modalities of this call for capitalisation projects created confusion. Indeed, the limited number of partners for a given project has not always enabled those partnerships which were created to present a project in full. The process was all the more disconcerting as the call for projects could allow projects that joined forces to respond to this call for projects:

- To disseminate their results more widely and highlight their achievements. - To develop or consolidate networks on the scale of the cooperation area and/or more

widely on specific themes (similar to what happened in the case of calls for strategic projects).

- So create synergies of tools and methods so that a broader public can benefit from them.

Sustainability

Ensuring sustainability is also a phase of the project life cycle which is perceived differently

by projects and JTS officers.

Whilst beneficiaries think they have on the whole carried out the necessary actions to ensure

the sustainability of results, the project officers see these actions as unsufficient. This can be

explained by the fact that they sometimes have difficulty obtaining the deliverables they want

to store in the programme's future library in a form that is sufficiently clear. Indeed, this

request is seen as an additional constraint on the projects as it is one of the changes to the

programme that has sometimes been poorly communicated.

However, drawing on the results of interviews and questionnaires, a typology of sustainability

activities can be established:

Typology of sustainability activities

Presentation of a

project aimed at the

sustainability and

exploitation of results

as part of a call for

targeted or

capitalisation projects.

Fund-raising

within other

programmes

Ongoing work through the network created by partners: however, although this desire is there in all cases, it should be see in perspective as few 'networks' wish to come together with a formal association status (or any other legal status)

Dissemination of results after the end of the project through the participation in previously identified seminars, scientific journals or other activities of dissemination

16

STRATEGIC PROJECTS

Recap of creation and process of identification and selection Recap of report 2011: the evaluation process and initial observations Selected project evaluation method Selected projects are evaluated based on face-to-face and telephone interviews. In the 2011 evaluation, these interviews were with thelead partner. Subsequently, these interviews were added to with information obtained from the interviews with finalised traditional projects by partners also participating in strategic projects.

Recap: key points from the first activities

To recap, the following points were observed following the interviews conducted at the end of

2011

Key points Strengths Weaknesses/possible improvements

First activities

Structure of the local Task Force eg. Regional Intervention Group bringing together all institutional levels in the case of the MARIE project (The RIG Generalitat Catalunya brings together 8 ministries).

Formation of local public/private partnerships, beneficiaries targeted by the pilot projects. For the MAIRIE project, four regions put together clusters of buildings, 75% of which are under private ownership. They help to put together the global town plan.

Clustering of local companies that are stakeholders in the implementation of pilot projects and transnational linking between clusters (début 2012 à Turin pour le project MARIE).

Very early in the life of the projects activities were organised between Elih-Med and MARIE and other traditional projects (IRH-NET, MED-CAT) to share initial methods and results.

Launch of first diagnostic studies ('Integrated Regional Benchmarking Analysis' for MARIE) under the Action Plan: identification of key priority zones for renovation, diagnosis of existing financial and legal instruments, creation of energy renovation models for districts (Barcelona) or even cities (for example for Montenegro's partner as part of MARIE)

Sometimes difficult to find/bring in local companies that are competitive and specialised in the sector. For the MAIRIE project, the companies of Northern Europe specialised in the energy efficiency of buildings have a more appropriate range than local companies.

Administrative and financial implementation

Lack of visibility of management activities at the start of the project. The difficulty in structuring the transnational partnership is a risk for the financial and administrative implementation of the programme.

17

Key points Strengths Weaknesses/possible improvements

Communication

Need to put in place a communication and dissemination plan

Need to target end groups/beneficiaries of the project (from the local to European level) and construct the message before the tools

Implementation of lasting project tools from the beginning: platform for MARIE, elearning for Elih-MED

The partners think more in terms of communication tools that of messages and target audiences. Support would be useful (training ...)

Sustainability

Need to put in place, from the outset, the financial and technical conditions to promote the sustainability of outputs/results (roadmap, action plan, best practice platform)

Lack of visibility with regards the final outputs to be sustained

These initial trends have been confirmed by the information gathered for this report. On financial and administrative implementation: just one year after the start of strategic projects, it would seem that their budget consumption capacity is lower than that of traditional projects. End of 2012, this observation was confirmed and all strategic projects were behind in terms of their planned consumption. This is partially explained by the fact that these projects have more wide-ranging partnerships which are more difficult to mange for the partner responsible for the work package 1 “management”. Moreover, the nature of the partners does not seem to favour effective financial implementation. During calls for strategic projects and selection, emphasis was placed on the size of partners (in comparison to during calls for traditional projects): the bigger budget than for traditional projects, the nature of activities developed and the objectives to be met, the profile (type) of partners and the partnership governance demonstrated in the application (work package 1) should prevent the difficulties encountered by certain traditional projects (in implementing activities and the budget). Yet, it would seem that most partners in strategic projects come from traditional project partnerships. Moreover, after over two years of implementation, it would also seem that the consumption capacity of strategic projects is no better than that of traditional projects. Comparing identical partners, it is the structure of partnership governance which might be called into question at this stage. Furthermore, the partners responsible for coordination do not have a higher budget than the partners involved in traditional projects, whilst the related administration costs are higher (due to the size of the partnership to be coordinated). Implementation of activities and outputs: at this stage, the strategic projects are following their work plan. At the local level, partnerships have not been created at the same speed but at this stage they are structured and are able to implement the planned pilots. These local partnerships add to the project's added value on the territorial and local level, compared with traditional projects. The first outputs (in most cases diagnosis, as is the case with MARIE) have also been produced, although it is still too early to analyse their transnational added value and, more broadly, their contribution to a Mediterranean strategy on the project theme. Conversely, the nature of transnational activities does not differ from that of traditional project. The majority of noteworthy transnational activities are, for the time being, those related to dissemination/capitalisation. Capitalisation: from the launch of projects, it would seem that significant effort has been made to implement joint activities by strategic projects or between a strategic project and other projects (traditional MED or other programmes).In fact, strategic projects are partly

18

developed on the basis of capitalised experiences, with the partnerships coming out of traditional existing MED projects (MARIE) or other non-MED projects (Elih-Med which itself came out of the SEE and ENPI projects) However, although the nature of the partnership facilitates this process of capitalisation, efforts are maintained by the projects: for example, at the end of 2012 the projects MARIE and Elih-Med came together in a joint event at Brussels to share their initial results and call on the European Commission to take into account measurements related to the energy efficiency of buildings in future operational programmes.

TARGETED PROJECTS AND CAPITALISATION

Due to the programme calendar (the first targeted projects were selected in December 2012 and the capitalisation projects are in the process of being prepared), conclusions about these specific projects could only be drawn on the basis of documentary analysis. Targeted projects Documentary analysis is based on figures from the documents produced by the programme. At this stage the following information allows an initial analysis: Calls for projects The procedure for calls for projects: the specifications are analysed in the table of the subsection entitled “Changes in procedures: making calls for projects gradually more specific and flexible” of part III Programme Evaluation Analysis Partnerships The composition of partnerships of targeted projects is one of the major developments compared to traditional and strategic projects: indeed, it would seem that there has been a turnover in partners:

o Of the 51 eligible projects: 366 different partners responded, 250 of which were new partners.

o The 19 projects above 300 points, (12 selected projects and 7 from the waiting list) incorporate161 partners, of which 92 new partners and 69 old partners.

But this change should be counterbalanced with the fact that 10 of these 69 of these old partners belong to the list of partners that are the most represented in the programme. Moreover, of the 12 selected projects, at the end of the Selection Committee meeting of 13 December 2012, 7 projects made up of at least 1 of the partners from the list of multiple partners (up to 4 in the case of Smartinmed). Furthermore, among the new partners, there are types that are largely represented in the existing partnerships (all calls for projects combined): regional development agencies, research centre, chambers of commerce. However, some (not for profit) private and international structures are also represented (rare type in pre-existing partnerships); it would therefore seem that the intended specialisation of themes under the calls for targeted projects may involve new stakeholders who are more operational (at least in terms of their nature)

At first sight, the mobilisation of new partners would appear to be a success for the programme but, on the other side of the coin: can projects of a 24 months duration, that are 2 years from the end of the programme, and whose financial and technical implementation must be efficient, take the risk of possible delay due to the lack of experience of new partners? Indeed, several partners (both from standard and strategic projects) reported in

19

interview some difficulties related to administrative management at the beginning of their project. Learning procedures linked to certifications and management tools require time and commitment. This commitment must be even more important when the project is short, as project extension is not anymore possible at the end of the programming period. Financial and technical efficiency must thus be present right from the start. These new partners could need enhanced follow-up by JTS members, as well as a dedicated project officer to monitor closely the implementation of operations.

Evaluation of applications The same grid is used for traditional projects (with a few minor changes in the way the different categories are organised), but there seems to be an improvement in the quality of evaluation: each category is subject to well-argued analysis whereas the evaluation of traditional projects was restricted to standard sentences.

Capitalisation projects Documentary analysis is partly based on the elements from the documents produced by Cespi; a first study of the application forms adds to this analysis: Call for projects The procedure for calls for projects: the specifications are analysed in the table of the subsection entitled “Changes in procedures: making calls for projects gradually more specific and flexible” of part III Programme Evaluation Analysis. Partnership The composition of partnerships, regarding the “nature” of the partners, is not fundamentally different from the traditional projects’ partnerships. This first observation seems quite logical, as capitalization projects aim to gather partnerships from traditonnal/strategic projects that have similar themes, in order for them to capitalize their results. Nevertheless it is worth noting that capitalization projects seem to have a better capacity of pulling partners from different territories than other types of call for projects: IAP, non MED areas from EU countries (eg Germany), non MED areas from non EU countries (South Mediterranean countries). This capacity is maybe due to the fact that beyond gathering partnerships from different MED projects, some application forms are the association of MED projects with non MED ones (under cooperation programmes of other areas, such as SUDOE, Interreg IVC, ENPI MED, but also “thematic” programme of the EU) Application form Even if they have more constraints, the application form of the call for capitalisation projects show the same difficulties than the ones presented under the call for traditional projects, for instance when they aim to present the results: the definition of the results is often quite weak, sometimes there are still confusion between results and deliverables. The description of the impacts and territorial added values resulted from the project is also quite weak. The application forms often show another fact: few partnerships defined the good practices they will capitalize within the application form. For most of them, this definition is one of the first activities, which means a risk in sharing, transferring and integrating theses good practices, regarding the short duration of these projects.

20

III - EVALUATION: PROGRAMME ANALYSIS

This part of the analysis complements the 'project analysis' part with a focus on:

- feedback from programme stakeholders (mainly MA, JTS and Member State representatives) on the process of governance / communication, definition / implementation of the capitalisation programme (sources: mainly evaluation interviews and formal/informal interviews from the programme)

- analysis of the documents produced by these stakeholders at the different stages of the life of the programme, i.e. the projects’ selection/programming, programme management and organisation tools, the closing of the existing programme and the preparation of the new programme (sources: Working, analytical and statistical documents produced as handouts for formal and informal programme meetings – committees, working groups and Task Force and day-to-day project monitoring process).

- Comparative analysis with other programmes of cooperation or thematic Community programmes

- Notes and evaluation reports produced on these different points

RECAP OF THE SITUATION AT THE MID-TERM REVIEW

The first evaluation report covering the year 2010 set out the logical framework of the programme in order to evaluate the achievements and failings observed between the objectives and means initially set in the operational programme and those observed when implementing the programme for four levels of programme positioning (general orientation, specific priorities, expected results and activities, as set out in the table below:

Presentation by

Programme goals Indicators (key questions

and target values) Verification

sources/resources

External performance assumptions (risks,

conditions etc.)

Level 1: General orientation Context/impact indicators (to

define through the OP

evaluation process) missing

data

From the SWOT analysis

and statistical data if

possible then ex-post

analysis of the OP or of

medium/long-term statistical

change

From the SWOT analysis,

diagnosis, benchmarking

and ex-post evaluations of

previous OPs

Level 2: Priority courses of

action (specific OP

priorities)

Programme indicators (EC

key indicators)

Outcomes of Programme

financial data and EC

recommendations

From OP strategy,

evaluations of previous OPs

ex-post and of OP ex-ante

Level 3: goals

(results/expected effects)

Operations targeting

Results indicators From the OP and projects

evaluation design process

(submissions and results)

Outcomes of methods

provided for implementation

of the OP and projects

Level 4: Activities and

performance (projects, tools,

reference documents etc.)

Performance indicators From projects and OP

documents and monitoring

tools (PRESAGE etc.).

From OP performance

(projects and outputs)

Source: 2010 evaluation report

This approach allowed us to identify a series of questions, successively addressed in reports and evaluation notes that we can break down into two main groups:

1) Programme governance and operations: positioning, expectations and feedback from programme stakeholders, decision-making process within the programme and communication between its different bodies, process of selection and programme

21

planning, operations of programme management and organisation, internal monitoring and evaluation system.

2) Specific 'work streams' launched or planned under the programme: capitalisation, communication, evaluation, preparation of the future programme (Task Force in particular)

We will recap the progress on these points at the mid-term review, i.e. the data in our first report and its annex, the set of recommendations, to enable developments to date to then be measured. This part will then conclude with a reminder of the raw programme data (programme planning, consumption) to put any developments observed and results obtained into perspective in relation to the simple programme indicators.

Programme governance and operations Interviews conducted with Member States representatives, the management authority and technical secretariat on the occasion of the first report revealed shared concerns and particularly:

- unsatisfactory quality of projects (hence the interest of stakeholders in launching calls for strategic projects), in particular due to the programme not being targeted properly and inadequate indicators, lack of turnover of programme participants and selection procedures that do not allow improvement in the standard of applications

- Lack of relevant expertise in the selection and monitoring of projects, operations conducted almost exclusively internally without external expertise and using an I.T tool (PRESAGE CTE) that is not flexible enough to adapt to changes in requests made by programme stakeholders

- Consensus-based decision-making process developed for optimal responsiveness (written procedures, informal meetings) but still perceived as a factor of slowing-down in terms of the programme's need to react quickly and adapt. Participation of all stakeholders in all formal and informal programme activities can also undermine this responsiveness (logistical problems), although it also provides an opportunity for fruitful discussion as was observed over the period of launching of calls for strategic projects.

In terms of performance, the functioning of the programme and, in particular, the monitoring of projects, the initial report highlights in particular the following:

One main point stands out with regard to the Programme's performance, particularly in terms of cost/efficiency. It seems that weaknesses in this field arise from the absence of monitoring procedures enabling better account to be taken of the stream of efficiency losses and better responsiveness to the unknown factors that always arise during a programming period (in this case: PRESAGE malfunctioning, overload of applications etc.). This also applies to factors that cause staff unavailability and, so, a requirement for flexibility, whether by ad hoc increase of staffing, or by reorganisation of procedures, or by recourse to outsourcing. This last point, however, needs to be handled carefully, as it could prove counter-productive if non-availability of steering bodies does not allow monitoring and supervision of outsourced assignments.

However, evaluation also shows strong committment of structures responsible for supporting project beneficiaries with a view to resolving problems which partially makes up for this weakness. Evaluation recommendations are to clearly identify the roles of assistance to beneficiaries in terms of managing coordination and communication to better define and monitor the allocation of resources and the optimisation of means.

22

Specific 'work strands' The requirements in terms of optimising communication (operational decisions connected to the PACA Region's communication system, the procedures of which are not always compatible with programme requirements) are clearly expressed, as are significant expectations in terms of capitalisation. In this area, a process was initiated in 2009 pending the recruitment of an external consultant for project capitalisation. The initial evaluation report feeds into this thinking by suggesting recommendations in terms of capitalisation, specified by three additional objectives:

(1) The goal of “feedback” Programme managers and member states need good “feedback” on the overall implementation of the programme and projects in order to improve the current programme and more importantly to design a new cooperation programme. The evaluation in-itinere is a key contributor to this feedback as are programme monitoring tools.

(2) The goal of “mutualisation” To help achieve the OP objectives, the programme needs a process to “consolidate” results, i.e. instruments to pool the various productions from projects (transnational platform?). This need exists for traditional projects and is obviously a strong requirement for strategic projects. For example, only by compiling the results from all projects related to a given theme will we be able to determine whether progress has been made in areas such as maritime safety, pollution control or renewable energy. New projects in a given area should be required to build on these achievements. Similarly, this mutualisation could provide the basis for a technical evaluation grid for future projects and answer the question “Where and how does a new project help develop a domain or consolidate it?” (See preceding paragraph regarding “relevant” projects). Today, the elements exist to make mutualisation possible, but they are scattered about. Because an approach to mutualisation is lacking, projects are duplicated in the same theme, in the same places and with the same players from one programme to the next. This penalises the macro-regional approach. It could help to define objectives.

(3) The goal of “capitalisation” Capitalisation26 in such a programme directly aims to retain a “trace” of the major activities of an approach to transnational work and if possible to diffuse it. The capitalisation process helps diffuse knowledge in more or less bigger circles by using different communication methods and tools such as brochures, seminars, etc. As it stands at the MED Programme, actions are undertaken to promote capitalisation (service market) and discussions are in progress for future programmes, similar to a feedback process. The entire system however needs to be developed extensively.

The diagram below (2010) sets out the main proposals in terms of organising the programme bodies.

23

PROGRAMME ADAPTABILITY

In the face of these initial observations and recommendations, significant changes have taken place, in particular in terms of monitoring and management, communication and capitalisation. Changes in procedures: making calls for projects gradually more specific and flexible: This has already been developed in the part about project analysis. It should however be noted that following two initial major calls for projects (almost 1,000 applications with only 100 projects selected) the quality of which was deemed to be inadequate, and following the launch of calls for strategic projects, then targeted calls for projects, the definition and targeting of programme expectations in terms of projects are becoming clearer. The observations made internally following the process of selection of targeted projects in particular show the participation of a significant proportion of new stakeholders in the targeted calls for projects, which meets the stakeholders' requirement to renew the programme from 2010. The table below illustrates changes in the respective specifications of various types of successive calls for projects. It should be noted here that this comparison cannot be extended to the programme content or results as only the so-called 'traditional' projects from the first two calls for projects are at a sufficiently advanced stage to make this comparison relevant. It is therefore here primarily the changes to procedures of definition then selectionand programming that will be of interest to us.

24

Item AP1/2: Traditional

Projects AP3/4: Strategic

projects AP5/6: targeted

projects AP7: Capitalisation

Projects

Procedure stipulated in the initial OP

Yes Yes Yes No

Approach Bottom-up Top-down Mixed Mixed

Level of specificity Low High High High

Selection rate 1/10 1/5 1/5 -

Eligibility rate AP1 (1er

tour) : 52% AP2 (1

er tour) : 74%

AP3 : 16% AP4 : 27%

69% 83%

Proceedings 2 stages Draft project then 1 stage

1 stage 1 stage

Partnership Untargetted Low constraint

Structured High constraint

Structured Constraint

Structured

Strategic positioning

Lisbon/ Göteborg Mainstream Coherence

EU 2020 Basin coherence

EU 2020 Interreg Continuity

EU 2020 New OP coherence

Theme All axis open – Themes very open

Targeted theme Specified axis

Targeted theme Specified axis

Open Targeting method

Project typology Unspecified Very regulated Predefined Predefined

Expected results Free choice Specification Specification form of results

Specification form of results

Evaluation Free Specified indicators Compulsory indicators

Compulsory indicators

Work plan Compulsory work plans Free content

Compulsory work plans Strong constraints

Compulsory work plans Strong constraints

Compulsory work plans Strong constraints

Eligibility criteria Strong Strong then flexible Flexible Flexible

Selection criteria Formal Formal but adaptable

Formal Formal

An expertise-based approach that has been tested but not incorporated in the programme planning and implementation processes. Although the use of external expertise in the project selection process, one of the evaluation's significant recommendations, has not yet been formally pursued on a long-term basis, programme stakeholders express a need for it. Indeed, to date recourse to experts remains on an ad-hoc basis: drawing up the specifications of strategic and targeted calls for tender, participation in thematic discussion sessions, (non-binding) opinions about applications for calls for strategic projects. This partially explains why primarily formal selection criteria have been maintained across all calls for projects, contrary to the other points of comparison in the table above. The notion of « external expertise » as used in this report refers to an organisation or individual with broad experience in a considered field of expertise. In this respect, the notion of “external view” is also relevant, in the sense that it associates to every step of a programming process, various relevant institutional stakeholders that give their opinion about any matter. The European Commission recalls in its guidelines related to the design of future structural programmes (cf European Code of Conduct on Partnership) how important is the close association of local authorities, representatives of academic and economic organisations and civil society in the process. This issue must therefore be addressed, but in the more general context of future programme’s governance bodies. As a matter of fact, asking stakeholders such as universities, chambers of commerce or local authorities to act as experts eg in a call for

25

projects’ selection process could be difficult to organise, even only for logistical reasons (Asking who? When? How?). Furthermore, and by definition, asking stakeholders to participate in such a process could lead to possible conflicts of interest or objections from other stakeholders. Eventually, this expertise induces staff availability, not always compatible with management standards of the programme stakeholders. Management and monitoring process: towards the emergence of coordination secretariat (still to be structured) However, this problem goes further than the issue of the selection process, the tools used in selecting applications remain too inflexible and difficult to adapt to a general targeted approach (PRESAGE CTE in particular). Indeed, this tool, although gradually adopted and recognised by its users - and in particular project partners - as a factor in simplifying the financial administration of projects, is not yet producing measurement tools that are sufficiently exploitable in terms of programme monitoring. The MED technical secretariat has gradually set up administration tools and tables, usually in Excel format, to make up for these shortcomings and a series of requests to improve the tool in time for the new programme is being listed. Moreover, in addition to simply managing projects from an administrative point of view, the JTS has also developed tools to monitor the content and outputs of projects, increased its participation in project activities (steering committees and final seminars), gained experience leading to greater efficiency and was able to take more strategic position, in particular after the period of excessive work at selection and at introduction of monitoring of the two initial project rounds. The move towards calls for specific and better defined projects generating fewer applications, contributed to this change, as well as the gradual closing down of traditional projects from the first then second call for projects, freeing up resources to provide projects with more support focused on their content. Similarly, the stabilisation then increase in staff numbers at the secretariat contributed to helping to develop staff skills. However, inadequacies remain:

- Tools created in formats that carry a risk of error (manual entry in spreadsheets, by several different people, updated inconsistently) and generally created as a reaction to an urgent need or requirement expressed in the committee and not as a result of a pre-planned strategy. The loss of energy and coherence therefore persists

- As the initial requirements for (traditional) projects were not specified from selection, it is difficult to make changes to them along the way without going through a decision-making process with bogged down in red tape. It will therefore be an uphill battle for these improvements to make an impact on the quality of projects in this programming period.

- The organisation of the roles of the secretariat is not yet fully structured. It is based on the individual experiences of people working on the programme, but cannot to date be quantitatively measured or accurately specified, formalised and shared, despite significant efforts being made on traceability (documents, methods - see list in annex). It is also subject to significant variations in terms of emergencies and 'service requirements' and as such it is difficult to evaluate certain aspects at this stage.

- Likewise, the functions occupied by MED liaison offices have moved in different directions: technical assistance similar to JTS approach in Thessaloniki (IPA/MED partner search and match-making, concrete information on applications and projects…), strategic definition in Valencia, closer to the traditional expected from a managing authorities. The specific context of each partner programme (IPA in the East, ENPI in the South) explains mainly this situation, and must be taken into

26

account in the future, in order to specify different job profiles according to each function to be performed.

A Communication role that is still too small Improvements have been observed in terms of the tools, and first and foremost the programme's web site, but the programme cannot access the procedures that would allow it to modify or acquire tools without the approval of PACA Region services, which acts as a significant break on outputs and planned tools, such as the library of project deliverables, the keystone of the programme mutualisation process described in the recommendations of the first evaluation report. While the increase of JTS human resources dedicated to communication – one full-time and part-time allocation of other employees – allowed to create conditions of a better external communication, but did not free up sufficient resources to implement a real communication strategy within a reasonable time-scale with identified and quantified target groups, approved messages, suitable tools and service and adequate evaluation procedures. The observations from the first evaluation reports therefore remain, for the most part, valid at the end of the process. From a more general perspective, the question here is not only whether the programme can autonomously deliver its communication strategy, but also at what level it is positioned in decision-making terms. Today the communication role remains mostly restricted to implementation. The evidence of this is that it does not cover internal communication, a role that is traditionally part of the management of an organisation. The fact that the programme's communication plan has not been revised is another sign, confirmed by the relative levels of interest of the decision-making bodies with regards this question. Indeed, despite communication having been an agenda item at each meeting of programme partnership and a coordination meeting organised at the Monitoring Committee of Ayia Napa in June 2012, internal procedures have not improved and the involvement of programme management bodies has remained constant. The benefits of the capitalisation process For a long time the process of capitalisation remained difficult to define even in the minds of the people behind it (the programme management bodies) as shown by the interviews conducted in the second evaluation report. The recruitment of a specialist consultant and the launch of a call for 'capitalisation' projects were, however, very positive for at least one of the capitalisation strands, the project strand (see part II). The 'mutualisation' and 'feedback' strands which are the subject of specific recommendations from the evaluation progressed to varying extents.

27

Projects’ deliverables database – JTS MED

Mutualisation, at least in terms of the projects, produced an interesting set of tools developed by JTS in connection with the outsourced mission. In particular, there is a database of project deliverables which must be connected to a specification table of the form and results of projects, which prefigures the MED library, in the process of being built as part of the programme's communication role. This mutualisation tool will be very useful not only to categorise and therefore evaluate the results of the MED projects, but also to capitalise on the MED project in a shared database accessible to future projects and therefore offering significant added value in the knowledge of the MED area. Moreover, this type of tool may support the preparation of future applications by making a full thematic overview available to applicants. This tool may not be functional before the end of the existing programme but remains very useful for the future.

Projects general database – JTS MED

Feedback from the programme bodies is however, not currently organised in such away as to be able to capitalise on procedures that have been experimented as part of the existing

Acronym Deliverable Component Phase Delivered Foreseen

number

Completed

number

Foreseen

print/production

run

Actual print run

2Bparks Protected

area list and

10

2Bparks description of

experineces to

be capitalized Yes 5 2

2Bparks network

1

2Bparks project

planning forms4

Project

identificationDeliverable identification and quantification

Acronyme SMG

Sectori

el KEEP db

Champs

d'application

général

04-Activité

économique Approche Moyen But

Type

d'innovati

on 1

Livrable

final Sujet

Bénéficiai

res Politiques

2Bparks Greener NON Gestion intégrée

des zones

côtières 04.17-Administration publique

Participation

publique

Plans

d'action

locaux

Guides

opératio

nnels

Environnement,

Aires protégées

2InS Clusters Smarter NON

Clustering

and

economic

cooperatio All

04.17-Administration publiqueDvpt et

soutien des

clusters

Établisse

ment de

réseau de

coopératio

Définitio

n de

cadre

commun

Gestion

et

promotio

n de

PME,

Clusters

Industries

créatives,

Innovation

AGRISLES

Greener

OUI

Agriculture

and

fisheries Agroalimentaire

04.01-Agriculture, chasse, sylviculture

Transferts

entre

partenaires

Établisse

ment de

réseau de

Définitio

n de

cadre

Lignes

directrices

,

Collectivité

s

territoriales

Agriculture,

agroalimentaire,

développement

AGROCHEPA

CK

Greener OUI Gestion des

déchets04.01-Agriculture, chasse, sylviculture

Dvpt de

critères

communs

Expérienc

es pilote

Modèle

de

gestion

des

Environement,

Agriculture,

Gestion des

déchets

AGRO-

ENVIRONME

Greener

OUI

Agriculture

and Agroalimentaire04.03-Industries alimentaires

Application

des

Établisse

ment de

Outils

d'appui

APICE Greener OUI Ports04.11-Transports

Analyses

régionales

Échanges Définitio

n de

Potential

voluntary

Environnement,

transports,

BACKGROUN

DS

Closer NON Ports

04.17-Administration publique

Révision de

données et

bonnes

Plans

d'action

locaux

Guides

opératio

nnels

Transport,

transport

maritime,

BIOLMED

Greener

OUI

Agriculture

and

fisheries Agroalimentaire

04.03-Industries alimentaires

Transferts

entre

partenaires

Établisse

ment de

réseau de

Définitio

n de

cadre Processus

Lignes

directrices

Gestion

agronom

ique, PME

Agriculture

biologique

C.U.L.T.U.R.E

Smarter

NON Patrimoine

Culturel04.17-Administration publique

Révision de

données et

Échanges Établiss

ement

Développement

urbain, Culture,

28

programme, which have been wide-ranging and contain learning points for the future programme, in particular the following:

- Changes to specifications of successive types of calls for projects - Toolbox for project monitoring and coordination - Involvement of Member States in defining programme developments (informal

meetings, working groups, 'Future MED' task force and so on. - Liaison offices whose role is to be redefined as part of an overarching communication

strategy for the programme. Feedback workshops were suggested as part of the recommendations of the initial evaluation report but no decision was made about their implementation. This 'feedback' could be made in another way, but it is, in any event, necessary for programme capitalisation to be complete. Indeed, capitalisation mainly focussed on projects (outsourced mission / 'project clustering', calls for capitalisation projects (subject to a specific evaluation note - note n° 11). In parallel, project results began to be pooled, which leaves unsolved the feedback work strand and the issue of training operational bodies to ensure achievements are carried over from one programme to another.

29

ACTIVE TRANSITION TO A NEW PROGRAMME

The shortcomings observed in terms of the in itinere evaluation are, therefore, far from being a reflection of inertia on the part of programme management bodies, whose reflection, progress and achievements are qualitatively very interesting from the perspective of new programming period. They simply reflect the size of the task and the fact that the organisational and structural gaps, in part dating from before the implementation of the existing programme, could not be corrected in full in the time available under the programme. This simple observation reveals the necessity to be able to use feedback to analyse the choices made throughout the programming period to derive good practice to roll out and mistakes not to be repeated. The results of ongoing changes in the way the programme is implemented However, the developments observed since the launch of the evaluation process have born fruit in terms of the requirements of the new programming process.

- Ongoing optimisation of processes - Thematic and strategic focus in calls for projects - Simplification of approaches of project beneficiaries (advice and administrative

support: problem solving) - Shared desire to implement a capitalisation process early in the existing

programme and to successfully evaluate in a continuous way rather than at mid-term - Looking ahead to the preparation of the future programme (proposed creation of a

Task Force from 2011 and launch of outsourced assistance missions to define and draw up programmes from autumn 2012).

If this capital of experience is used in a structured way and adapted to the context of a new programming process, itself constantly changing, these changes will bring about success in 2014-2020. An acceleration of processes All processes and work under way in terms of the governance and functioning of the existing programme have, since the creation of the 'Future Med' Task Force, converged to be put into perspective systematically with the new programme. This coherence is strengthened by the external missions of drafting diagnosis and programme for 2014-2020, conducted by the Task Force. Yet as the individuals responsible for monitoring and finalising the existing programme are the same ones preparing the future, this acceleration means a risk of confusion or superimposition with the outstanding tasks from the existing programme and the lack of organisation highlighted above adding to this. It is more important than ever when the trend towards the concurrent existence of two programmes is to develop and accelerate in the coming months, to ensure roles are clearly defined (with outputs organised) within the various bodies for each programme. Postponed implementation of evaluation recommendations

De facto, some recommendations, in particular in terms of the need to define a project and results typology whilst seeking out the most suitable partnerships for achieving these results, were incorporated in the thinking of the programme management bodies. They also strengthened the internal analysis arising from project practice, particularly through the programme's JTS. Nevertheless, in some cases, the time needed to appropriate these recommendations (external expertise to help select and monitor projects, stable structuring

30

of a project typology, defining target groups, feedback, managing the resources of management bodies...) prevents their implementation within the time-frame of the existing programme. The rapidly changing nature of programming procedures and decisions did not allow existing resources to work on several work strands of this size. However, thinking is under way on how to implement these recommendations using strategic options in the future programme. 2013 also sees the opening of several operational work strands within JTS which should allow:

- An operational feedback process to be structured - Recommendations to be incorporated before the programme is implemented - Management and coordination tools defined under the existing programme to be

optimised and sustained. - Thinking to be carried out on the positioning of the communication role within the

governance of the future programme

31

IV- SUMMARY OF EVALUATIONS

TABLE: SYNTHESIS BETWEEN PROJECTS AND PROGRAMME ANALYLSIS

Process of reference

Conclusive Observations Evaluation of projects

Conclusive observation Programme analysis

Synthesis Projects /

Programme

Selection of application forms/ Selection process – programming

The two-stage selection process is essential for the maturation of the project between the partners. For traditional projects, no changes in the application between the different calls for projects, the form of which is not satisfactory in terms of demonstrating the tangible nature of the project. Despite shortening selection deadlines, this phase is still too long for project sponsors The lack of technical or scientific expertise throughout the selection process is a missed opportunity in terms of ensuring the quality of selected projects.

Progressive specification of the call for projects, still to encourage regarding the tangible application of the programme’s strategy and expectations in term of imapcts

Effectiveness could be improved by targeting the calls and bringing on expertise on the applications’ contents

Monitoring/Implementation

Implementation (administration) makes up too

large a proportion of project activities to the

detriment of technical implementation

Proven improvement in administration tools but

not yet enough.

Positive opinion of monitoring by JTS members

The varying certification procedures across Member States is a break on the refund process

An emergence of a function and organization tools, still to outline. A financial management which are now professionalized, but yet

To improve performance: harmonizing 1st level control and structuring coordination

Results/finalisation process

A finalisation process in which projects very rarely meet the deadlines Qualitative technical improvements and final outputs and results non-existent No proven link between the profile of partnerships and the nature of outputs/results

Some difficulties to improve the efficiency of process regarding fundings consumption and finaisation of procedures.

To strenghten consistency of messages: creating obligations of results from the application stage

Communication / Governance

Communications mainly traditional for all projects without any significant investment from partners A need for more 'directive' tools as the programme demands are not always properly understood by the projects

A communication function still mainly under estimated, and still not enough associated with the programme strategy

Coherence between strategy and execution: adapt each task to the proper level of governance

Capitalisation / Sustainability

An initial process poorly understood by projects in terms of its purpose. The process of clustering deemed to be out of step with the processes of the calls for capitalisation projects.

An on going capitalisation process, and still iirregular regarding the themes: mutualisation still to finalize, experience feedback wtill not structured.

A finally understood process, to better structure for a better effectiveness

32

Process of reference

Conclusive Observations Evaluation of projects

Conclusive observation Programme analysis

Synthesis Projects /

Programme

Forms of sustainability very wide-ranging but never formalised: sustainability 'along the way' and not planned.

Definition of the expected impact still inadequate to express a clear vision of the programme expectation.

Emerging process to consolidate through feedback

PROJECTS /PROGRAMME REPORTS.

Recap of observations from the 2010 evaluation report The basic datas specified in the 2010 report were the following:

The primary hypothesis of this in itinere evaluation is to identify interdependences between

the MED projects and programme. This means that projects are not solely evaluated in terms

of how they meet programme requirements, but also as a breakdown of programme

strengths and weaknesses.

The first three observations after the first project evaluation were: The phase encompassing applications, calls for projects and selection can be defined as the demonstration to commit resources by which projects are bound.

The monitoring process is characterised by the demonstration of compliance. All

resources developed to monitor projects tend to help the projects to comply with the rules,

any controls and sometimes the procedure described in the application.

The expected results of the project should be demonstrated in the application but are not

subject to specific monitoring or controls. Due to the absence of monitoring of results and

structured capitalisation, improvisation prevails.

At the end of this evaluation at the beginning of 2013, important changes in the administration of the programme and the monitoring of projects allowed very significant progress to be made. Today, the gap between the actual content of projects and the available information has been vastly reduced. However there remains an area of low readability (see diagram below). This difficult readability is, according to our analysis, due to two main factors: The complexity of situations/projects: The in-depth interviews and analysis of projects results (particularly deliverables) show that situations are more complex than those we set out in the initial evaluation scenarios. The types of situations/projects are very diverse.

At the conclusion of their work, some 'traditional' projects appear to be projects that could be categorised as 'strategic'.

Others appear to be a combination of local projects, where the transnational element has generally been achieved but with very limited ambition.

High quality projects and results which have not benefited from the right partnership to produce results and which are very unlikely to result in any sustainability.

Etc.…

33

All 'traditional' projects have been selected using the same application which only left limited space for the information now required, for example, in terms of the results and deliverables. It is therefore difficult to get information from projects today that was not required initially. It should be noted that the capitalisation process and on-line deliverables library on the programme scale go some way to reducing this difficulty. Diagram: existing relationship between projects and programmes and how it has evolved Issues 2014-2020? Capitalisation Results/outputs Monitoring, support Implementation of project activities Evaluating applications Putting together applications (Selection) Programme administration project life cycle

Zone of

limited

readability

34

V -RECOMMANDATIONS

In June 2011, a list of recommendations (32 pages) was produced by the evaluation team: 'deliverable n°3' of the terms of reference of evaluation. These recommendations were the subject of several meetings with the JTS and AUG teams. In accordance with the terms of reference, in the first paragraph of this chapter we will set out the extent to which these recommendations have been followed by the end of the in itinere evaluation process. Throughout the evaluation process, several notes addressed the issues of recommendations (see annexes). At the beginning of 2012 a specific task was realised for the Task Force, a benchmarking analysis (Deliverable n°6). The other paragraphs will cover new or updated recommendations of 2011. To make it easier to read we have broken them down into several sections:

1. Closing the programme 2. Developing the future programme: mainly works on diagnosis and strategy 3. Implementation tools 4. Governance

FOLLOW-UP OF RECOMMENDATIONS MADE DURING THE MID-TERM EVALUATION (2011)

Table showing how recommendations have been followed by the programme half-way point (see table of recommendations in the annex of the first evaluation report). The purpose of this table is to show the extent to which recommendations have been followed as they were formulated following the evaluation report covering 2010. Of course, the daily experience gained by the programme stakeholders and the development of procedures during its implementation have changed things and the current situaiton no longer allows these recommendations to be made on the same basis as in 2010. For this reason, in the following paragraphs we will attempt to reorganise the key elements of these recommendations:

- based on these changes in context, - and according to a new calendar, mostly related to the finalisation of the current

programme and the launch of the following one If a general conclusion is required as to how these initial recommendations have been followed, it would be that their principle and their potential to improve the programme's performance have been incorporated, but that certain suggested actions have not yet been implemented in a structured and formal way according to internal and validated procedures, and then implemented. Some are at an experimental stage which, nevertheless, represents a very significant step in a three year period. This is why it remains vital first and foremost, before the programme is finalised, to organise programme activities and bodies so as to:

35

- Finalise the feedback process which will put in place or optimise information management and traceability between the stakeholders (governance and internal communication).

- Consolidate the tools created and approve the procedures for updating and using them (By whom? For whom? How? At what pace? For which results?) avoiding, in particular, multiplying parallel actions that cloud the readability of the current and future programme (mutualisation/pooling)

36

N°: Proposals

Implementation: planned

Implementation: actual

Comments on implementation Current

OP Future

OP D* ND* PD* P*

Developing the future programme:

1

Organising capitalisation, mutualisation and feedback

2011-2013

X

Process of implementation for capitalisation, less successful in mutualisation and not formalised for feedback

2 Mutualisation tools 2012-2013

X X X Stages to prepare an on-line library completed. Effective implementation as part of the new programme

3 Organising 'feedback' seminars

2011-2013

X

The issue was not addressed in the suggested form. It would be important for it to be completed in this form or in another with a view to the implementation of the future programme (the questionnaire sent to the members of the Task Force at the end of 2012 can be the beginnings of an answer)

4 Scientific or 'expert' committee

X X Current thinking on arrangements for bringing in experts and decision-making authority to be given to them

5

Defining from the OP the themes that are to be the subject of strategic projects

X X

The recommendation was followed and thinking is already under way in preparing the future OP

Calls for projects and evaluating applications

6 Thematic call for innovation-type projects

X X The gradual specification of types of partnership, activities and results expected for the strategic then targeted APs shows this recommendation has been followed. Without prejudging the nature of typologies which will ultimately be chosen for the future OP, it seems a given that there will be some and that they will have an impact on the procedures for future calls for projects.

7 Thematic call for network projects

X X

8 Thematic call for transversal projects

X X

9 Reorganisation of application evaluation. Option: bringing in

2012-2013

X

X Thinking is under way on the future OP. For the current period, tests have been conducted (strategic projects) to call on expert opinion of the applications. The procedure remained on a consultative basis. Whatever the process chosen for the next OP, the question of how applications are analysed

High level of importance

Medium level

37

N°: Proposals

Implementation: planned

Implementation: actual

Comments on implementation Current

OP Future

OP D* ND* PD* P*

outside experts should not be ignored and must be answered in terms of expertise (external or internal) describing the skills development process planned for the selection of high quality projects.

10

Reorganisation of application evaluation. Option: developing a reference tool

2012-2013

X

X

No implementation observed in the current OP: the AP specifications have changed significantly, unlike the format of application forms (even if there is much more content) and selection tools such as partner skills mapping, or thematic overview do not exist. As they depend on the nature of Member State involvement in the selection process (information resources on partners), the existence of mutualisation tools and the existence of expertise in the programme, these questions should be addressed together as part of the development of the future OP.

Improved relevance of projects and added value of results

11 Method for monitoring project results Option: Changing the application

X X

The recommendation was followed from the calls for strategic projects and systematically from this time in the following calls for projects. Tools to specify results were produced as part of the capitalisation and project monitoring process. These tools and related indicators remain to be structured in order to make them fully operational for the implementation of the future OP.

12

Scientific committee Option: incorporating internal expertise in the projects

2012-2013

X X

This recommendation appears intermittently according to the calls for projects (specification of the function of partners in the strategic CPs, requirements in terms of evaluation to be carried out by the projects...). It has not been formalised as such and could be incorporated in a more structured way when implementing the next OP.

Organisation of the monitoring of project coordination

13 Secretariat for administration and financial management

2012-2013

X X X Lack of feedback on the process of organising functions and tasks which makes it difficult to assess the actual contributions both in terms of coordination and administration. The main difficulty to be corrected remains, therefore, administrative in nature. In terms of project monitoring, qualitative and quantitative tools have been produced (PRESAGE is still not always usable) but with high risks of error due precisely to the fact that organisational processes and information transmission have not being formalised.

14 Coordination secretariat 2012-2013

X X X

15 Communication function X X X

The weaknesses observed at the beginning of the programme (and partly coming from the former programme) were too substantial to be compensated without starting from scratch which can only happen when the future OP is planned. Efforts made over the current period have not yet led to an overall restructuring of this function.

* D: Done ND: not done/ PD: Partially done / P: Postponed until the future programme

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 38

CLOSING OF THE CURRENT PROGRAMME

The dual process of mutualisation and feedback are connected to the current programming period and add to the 'bottom up' capitalisation (of projects) approach conducted in partnership with the external consultant appointed for this purpose and made tangible by the clustering process and the call for capitalisation projects. Recommendations produced in 2011 are a reminder that to be complete, the process of capitalisation must also include:

- a combined top-down and bottom up approach, mutualisation, work currently under way through the 'MED library' which has a dual purpose:

o Knowledge of the programme: gathering information on the projects developed under the current programme which means first categorising these products by theme, type, target group, territorial coverage: all elements that make it easier to pin down project results on the one hand and understanding of their contributions towards the programme on the other hand. In sum, a useful contribution for monitoring the programme

o Introduce the concept of quality commitment in terms of project outputs and raise awareness on the part of projects about making their outputs available to the public and, conversely, the option of preparing their applications and capitalisation using the existing programme knowledge. This means developing the reliability of available resources.

- A 'programme' capitalisation and feedback approach that has yet to be implemented in a structured way and is particularly aimed at optimising the management and monitoring process as well as, more generally, governance and implementation based on the knowledge of past practice (capitalisation of good practice, elimination of practices that do not bear fruit). At the dawn of a new programme which is to set a performance framework broken down into milestones, this operation makes complete sense.

These two functions must be finalised in order to fully exploit the achievements of this period for the benefit of the future programme and to thus initiate a quality control process which will be continued throughout the 2014-2020 period.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 39

(1) Mutualisation tools

Proposed solution

Developing and exploiting in a sustainable way a shared tool for promoting knowledge from the programme: 'MED Library'

Action points

- approve output categories by cross-referencing, in particular with the themes addressed, the methodologies used, the territories covered and the target groups - add to the database drawing on all MED projects (whilst managing issues of intellectual property, if possible before the end of projects: protocols…) - create conditionality for the new projects to make reference to the existing knowledge base - communicate widely on the existence of this database and thus support administration and knowledge-sharing procedures within the MED area.

Additional action points

- In the future: for some themes related to emerging areas or techniques, ensure before results are produced (i.e. from the project application) that a scientific or professional validation of outputs is made prior to dissemination, in particular through the creation of a duty to evaluate the project outputs technically.

Expected results

- continuously expanding knowledge base allowing changes to the programme and knowledge of themes to be traced and the selection of projects in accordance with this database to be facilitated. - increased quality of project outputs. - contribution to the evaluation of project results by specifying and feeding programme-specific results indicators.

(2) Feedback process

Proposed solution

Mapping the programme implementation process to facilitate closing operations, define implementation procedures for the new programme and optimise the quality of programme management processes and its effectiveness

Option 1 Organising 'feedback' workshops

Action points

Creation of cross-cutting thematic working groups that are not restricted to a single body (JTS or

MA)

Pros Cons

- Calling in experts and/or witnesses + benchmarking other INTERREGs - organising a cycle of workshops in two or three stages (audit, selection of good practices, optimising management tools) - conclusions and reference document

- co-building of the approach in common - opening on external practices (benchmarking) - shared approval on recorded practices

- delay for setting-up and organising exchanges - sincere communication when exressing oneself in public or in a hierarchical administrative context

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 40

Proposed solution

Mapping the programme implementation process to facilitate closing operations, define implementation procedures for the new programme and optimise the quality of programme management processes and its

effectiveness

Option 2 Conduct a survey of programme management stakeholders to identify,

reproduce and optimise the practices recorded.

Action points

Reproduce the processes using

methodologies that have been proven to work

Pros Cons

- Specifying items to be included in the survey - Recompiling formal and informal processes - Options for conserving the different levels of formalism recorded - Categorising then optimising practices - Producing a reference document

- speed of implementation - methodological expertise and proposal for turnkey solutions - sparing time of stakeholders

- without internal monitoring, risk of not achieving tangible results - weaker appropriation of the process - time and budget for outsourcing

Additional action point

It will be necessary to put the results into the perspective of the regulatory constraints of new programmes to continue feedback using an approach of continuous optimisation (quality approach and/or standard) Training new staff in the traceability of processes with a view to ensuring the sustainability of feedback and the quality of management. Sharing results between the programme bodies

Expected results

- Improved readability of practices and more efficient processes - Improved traceability and transferability of information - Increasing cost-effectiveness with regards frame of performance (new programme)

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 41

DEVELOPING THE FUTURE PROGRAMME 2014-2020

(1) The programme diagnosis and strategy The primary objective is to bring the diagnosis closer to the programme strategy and expected results. The Operational Programmes of the 2007-2013 generation were drafted on too ‘contained’ basis within the frameworks set out by European and national guidelines in this period (Lisbon, Goteborg). In practical terms, the programmes reproduced the same content from one cooperation area to another whilst, by definition, it is about territorial cooperation. The fact that these programmes did not reflect certain regional realities often manifested itself in repetition of the same projects and partnerships from one programme to another. These diagnoses, when examined as part of mid-term and ex-post evaluations, are not very useful in evaluating the added value of projects. Analysis is similar in case of how cooperation reflects regional policies (regional OPs), something that is difficult to measure. For the MED programme, an additional difficulty was in not having access to the achievements of previous programme. A MED programme developed for this period 2014-2020 will have access to all the achievements of work carried out during this programme-planning period thanks to the impetus of the programme management bodies. Three types of data are available

Data from capitalisation, evaluation and projects/ programme feedback to

categorise guidelines that have worked effectively and

those that have not

Data from studies and statistics about the MED zone which highlight the needs of territories (unemployment,

equal opportunities, insecurity, impact of the crisis

and so on).

A table including the programme offering in the

MED zone. Benchmarking analysis carried out for the Task

Force in 2012.

Diagnosis

Regulatory 'filter'

The following stage will be the strategic choices, the programme priorities and analysis of feasibility of the proposed choices in terms of available means. Given the results of evaluation work, we suggest three recommendations for these two final types prior to developing evaluation tools for 2014-2020,

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 42

Proposal 1 Propose a positioning for the MED Programme in relation to the general offer of European programmes.

Action points In terms of priorities that overlap with those of other programmes, strong territorial added value should be sought and project results should be focussed on broad dissemination and organised mutualisation.

Proposal 2 Seek convergence with Mainstream programmes.

Action points

The choice of priorities should draw on an analysis of the priorities that appear the most often in regional operational programmes to ensure impact across projects. The choice could be to draw on important regional policies or, conversely, sectors that have been underdeveloped by these policies.

Proposal 3 Analyse the feasibility of major plans to develop a Mediterranean exchange area; emergence of a macro-regional approach.

Action points On a few key factors emerging from the diagnosis, the proposal involves structuring a project approach for the whole cooperation area (e.g. marine pollution or green economy, urban issues, migration flows and so on).

IMPLEMENTATION TOOLS

(2) Types of calls for projects: From 2010 to 2012 the programme experimented with several types of calls for projects. This experimentation is a significant achievement for a programme of the MED's size. The implementation calendar for the projects selected in the last calls for projects (calls for targeted projects and capitalisation call) does not allow a full comparative analysis with the programme’s first two calls for projects (2008 and 2009). In the INTERREG cooperation programmes we can observe a significant number of this type of calls for projects. There seem to be several key factors contributing to their success and the quality of results.

The type of calls for projects and themes addressed (strategic, micro-projects, targeted and so on).

Procedures for calls for projects and the selection method (continuous, annual calls and so on).

Following this evaluation we think that these choices should be made on the basis of:

Strategic choices made when developing the programme.

The means in place to support the programme and its governance.

For this reason we suggest here open options which will remain adaptable according to choices made beforehand. These choices are obviously based on the following reference frameworks: Strategy EU2020, ETC rules, choice of priorities, performance framework and so on.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 43

Proposed solution

Calls for projects in keeping with the programme strategy

Option 1 Programme strategy focussed on meeting the needs of the cooperation area.

Action points

Calls for 'innovative' projects aimed at supporting and giving impetus to the development of growth sectors (jobs and competitiveness) for the MED territories. A typology of projects based on the 'needs' of the territory, population and crisis context.

Calls for ‘structural’ projects to allow MED area stakeholders to adquire and share knowledge. Building a Mediterranean strategic framework

Focus funding on new sectors and emerging 'activities'.

Favouring a 'start-up' rational at the Mediterranean level

Cooperation should accelerate the process

Focus funding on mutualisation information , methods on a Mediterranean scale

Favouring several geographic scales: Mediterranean as a whole, pluriregional, urban networks and so on

Example of possible themes: 1. Urban intelligence 2. E-services 3. Urban-Rural 4. Green economy 5. Social innovation 6. Agro-nutritional quality 7. Sustainable tourism

8. Climate change 9. Natural risks 10. Maritime risks 11. Air and water resources 12. Pollution 13. Protection of sensitive areas

Proposed solution

Calls for projects in keeping with the programme strategy

Option 2

Target the programme on a few main themes relating to the MED area whilst opening up the programme to new groups (these themes and these groups to be identified by current work of development of the new programme). This option would be in keeping with current MED experimentation.

Additional action point

Call for targeted projects

Call for strategic projects Call for micro-

projects

The themes would be those defined by the 4/11 chosen priorities As for the current programme's calls for projects, the objective is to seek more operational and transferable results.

These calls could address ‘cross-cutting’ themes (climate change, risks, etc.) and 1 or 2 themes coming from targeted calls and which require a horinzontal partnership (such as current strategic projects). These projects should have a longer duration

The idea is to allow projects open on civil society to favour transnational partnership building (equal opportunities, fight against poverty, etc.)

Expected results for each project

Each project should produce distinct outputs at the level of the partnership and cooperation zone.

Thes projects’ results should compulsorily be disseminated or accessible (database) for all MED area

These projects’ works should produce outputs as well as proposal to structure a network

Proposed solution

Calls for projects in keeping with the programme strategy

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 44

Option 2

Target the programme on a few main themes relating to the MED area whilst opening up the programme to new groups (these themes and these groups to be identified by current work of development of the new programme). This option would be in keeping with current MED experimentation.

Options for procedures to manage calls for projects

Calls for projects with a fixed calendar from the programme launch

Pros Cons

Readability for applicants and long time for preparing applications

Risk of an important number of applications

Annual calls for projects Time for preparation favourable to applicants and opportunities to re-

apply acording to marking

Risk of an important number of applications

'Continuous' calls for projects This formula often comes with ongoing application support Joint application process.

Favouring the development of quality of projects

Continuous monitoring and onerous follow-up

The phasing of projects: this phasing can be organized whatever the options are. The purpose would be to take into account the diversity of projects and partnerships. Indeed, the unified application process proposing the same activity schedule type and the same result obligations for all applicants showed their limits. This phasing could be organised with 3 main objectives:

Allowing important projects to schedule their activities more efficiently. For instance, allowing strategic projects to continue their cooperation using previous results. Midterm orientation of the projects, by implementing a validation process of phase 1, before entering phase 2. Phasing may also adjust activity schedules regarding the form of the projects. It may be for instance shorter project duration for micro-projects, and extended duration for strategic or structuring projects.

Micro-projects Strategic or structuring

projects Targeted projects

18 to 24 months Maximum of 48 months,

planned in several phases Maximum of 36 mois planned

in several phases

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 45

(3) Targeting groups. Analysis of target groups and mainly partners shows a few recurrent specifics: In answer to the question why? And how the partners find each other in MED? Or why they choose MED, we have made the following observations. The different types of calls for projects have helped to 'bring' new local stakeholders into the programme. Nevertheless, the block of usual MED partners are still very present. As these partners are experienced and will adapt to new situations resulting from the calls for projects. This is not negative in itself but lead us to wonder about the reasons for this situation. Of the problematic situations we have observed a specific type of partners or positions.

'Screen' partners who are often project leaders, who in some cases have management skills made available to others. Or partners occupying a 'position' of leadership by delegation because, for example, the programme does not allow the involvement of private stakeholders.

Partners who use influence strategies. They are 'leaders' on a given subject and want to organise a partnership centred on themselves. It is a positive approach if it means disseminating good practice, and negative if the intention is to 'control' a situation.

The partnership of 'means' is a partnership of stakeholders who alone are unable to achieve the critical mass to make an idea happen. This situation does not guarantee results.

From this point of view, applications are an exercise with a dual purpose, to:

1. Allow applicants to express their project.

2. Allow the evaluator to understand and prepare the selection.

Proposed solution

Incorporate thinking about the type of partners expected and set out the rules for multiple applications in calls for projects (as was the case for strategic projects)

Action points

Allow projects to clearly show in their application the function of each partner and ask projects to specify who the actual beneficiaries are. For example, a structure managing a cluster acting on behalf of X companies or X associations. Define a status of associate partnership or local partnership. Where legally possible, favour local partnership agreements (see procedures for Regional Framework Operations under the ex Interreg IIIC Programme). Restrict to one 'Project Leader' participation and 1 'Partner' participation per call for projects for the partners who do not make up multiple entities (e.g. Regions). This rule must be adjusted according to the themes of the calls for project.

Additional action point

This new approach must be one of the messages to communicate to the beneficiaries in all programme communication tools and at the calls for project launch events.

Expected results

Facilitate the selection of applications based on relevance criteria. Open the programme up to a broader group and bring the programme closer to the reality of how projects are actually run. Move on from the classic situation of INTERREG programmes which limits the visibility of beneficiaries to the project signatories alone.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 46

(4) Application format: It is impossible to represent the diversity of projects with the current PRESAGE format. Progress has been made in the calls for strategic projects and calls for targeted projects.

Proposed solution

In addition to improvements in PRESAGE, also give the applicants the option of adding 'annexes'.

Action points

This format is used by several programmes. If the structure of the application form is not changed, annexes relating to the following points could be created or strengthened:

'Logical framework'

'Justification of costs' (e.g. SUDOE programme) Excel format as an annex to the application form

'Quantified action plan' along the lines of the calls for strategic MED projects.

'Type of outputs'

'Type of target groups'

'Communication plan with quantified resources'

Additional action point

At the calls for projects launch events information must be given to explain how to fill out the annexes (idem for all supporting documents)

Expected results

Allow applicants to set out their project and to give details about the content.

To allow the evaluator to understand and prepare the selection. This project definition phase should also facilitate project monitoring and allow annexes to be used to produce monitoring forms.

(5) Evaluating applications:

This was one of the 2011 recommendations (n°10) Observations made in 2011 related only to evaluating the applications in calls for projects 1 and 2. Since then the conditions of this evaluation have substantially changed and improved thanks to feedback from theses calls for projects and new practices that have been experimented in the calls for projects (strategic and targeted). Part of this project 'memory' exists and is not yet incorporated in a real management tool. This recommendation also contributes to the overall readability of projects (see part IV). The very 'open' calls 1&2 for traditional projects gave rise to situations that were too wide-ranging. For example, partnerships were created on themes for which they were unable to produce tangible results. For a given subject we can have partners directly responsible for the subjects tackled; or partners who only raise these subjects at conferences.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 47

Proposed solution

Organise selection procedures tailored to the different types of calls for projects. Provide the evaluators with the necessary tools to improve the effectiveness of the work to be done.

Action point

For some calls for project and to produce resources, we have proposed bringing in thematic experts.

A database describing Member State by Member State the organisation of skills (existing documents in certain cooperation programmes), a skills map to improve the feasibility of the proposed activities. It should be noted that this debate took place within the MED selection committee during the calls for strategic projects.

Creating an evaluation grid with a thematic overview of what is expected of an Interreg project: This grid should allow an evaluation of the actual capacity to produce the results claimed in the application.

Database of previous MED programmes (at least 2007-2013) to identify partners. Possible separate partner database in which all applicant organisation should register prior to application

Simplified benchmarking grid in order to skim read the 'offer' in the cooperation area on a theme by theme basis.

Grid specific to the MED programme 2014-2020 and particularly relating to territorial added value.

Additional action point

This information would be the subject of a joint resource. A database on a simple software

Expected results

Increase the overall effectiveness of the programme by optimising the budgets invested in projects. Increase the reliability of selection criteria. Reduce the gap between the actual projects and the operational objectives of the programme.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 48

(6) Towards quality assurance and administrative simplification : a centralised

certification The current certification process increases the administrative burden, both at project and programme level. It is also perceived as an unequal treatment of partners within the same project. Here lies a key-factor able to accelerate implementation of activities and compliance with schedules, while the current scheme considerably increases the burden of projects’ follow-up by the JTS. Implementing centralised certification would be a major achievement considering the future 2014-2020 programming period, as part of the global quality assurance process to which the programme must adhere:

At the call for projects level

At the project monitoring level

At the project coordination level

At the capitalisation/pooling level

Proposed solution

Organise a unified certification scheme for the whole

programme

Expected improvements in terms of programme

management

Better image of the programme for target groups

Action point

Create a tripartite working group (Task force type): certification body, JTS/MA, Member States representatives. Publication of a framework document “joint guidelines on financial management”

Improvement of communication for all programme managers

Additional action point

Information and training for programme managers and beneficiaries.

Better readability in terms of financial management and reduction of decommitment risk. Economy of means for the JTS

Funding options

2 options: Direct funding by projects

Funding by MA and/or Member States

Expected results for projects

Better conditions for a compliance with activity time frame. Improved equality between project partners

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 49

This is the case in particular of the 'communication' function, which today mainly comes under MED JTS for most aspects: - strategy (communication plan) - implementation: events, publications, web,

networks... - inter-programme relations (IPA and ENPI liaison

offices) - Assessment of communication actions

It should be added that the 'internal communication' function has not been officially taken on, indeed, as is the case for many other programmes. In a classic organisational structure, the communication plan is drawn up and approved at a strategic level (in particular in terms of its key-messages and targets) then delivered by a dedicated operational function. In the case of MED, these functions are confused, which on the one hand takes away from the legitimacy and therefore the effectiveness of stakeholders, and on the other hand overloads the implementation function and slows down the completion of planned tasks. Finally, internal communication is not addressed and takes away from the effectiveness and readabilityof the processes.

PROGRAMME GOVERNANCE, COMMUNICATION

(7) Possible organisational options Main initial observations These options are first and foremost related to two points covered in our previous recommendations and reports, as well as the benchmarking conducted under deliverable 6 (for comparative analysis between MED and thematic programmes of the European Commission) and the initial evaluation report:

1) Positioning the communication function within the programme 2) The distribution and readability of functions between 'strategic governance' and

'executive’ or ‘operational governance' of the programme. The strategic governance, in quite a classic way, covers all decision-making functions to:

- Give the programme direction and position it in its territorial and political environment (existing policies in the territory where it is established)

- Approving and checking that programme implementation is in keeping with its strategic direction and position.

As for operational governance, it includes implementation on the ground (in terms of monitoring projects, procedures, assistance) deciding on strategy and reporting to the strategic governance bodies. Cooperation programmes have a duty to put together bodies with the powers to carry out these different functions (committees, authorities, secretariat and so on), which to a large extent gives the programme authorities the option of organising resources, procedures and relationships between bodies in the way that best meets their performance objectives.

For the MED programme, the study of certain functions, in some cases showed that some of them were either overlapping or inverted at the different levels of governance, Purpose of the recommendation Our reasoning here is essentially functional and its purpose is not to take a position on the statutory or legal arrangements for MED governance as the key point is to send a clear message about the possible distinctions to be made within the various functions responsible for designing and implementing the programme, between strategic and operational governance. The chart below illustrates a few examples of how functions are

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 50

broken down between bodies responsible for strategic governance and those responsible for operational governance of MED.

Options to implement the recommendation

1. Setting out the framework for developing programme functions between the various levels of governance at the design phase of the new programme, in order to improve readability. For example today, when Member States are involved in operational tasks to support the emergence of strategic projects, how should this function be differentiated from that of JTS which has a similar function? Also, how can the support function (NCP) be separated from the decision-making function of the selection committee (MS)?

2. Add as an annex to the communication plan, an internal communication plan under the strategic governance of the programme to clarify the decision-making levels for each implementation function. For example, as capitalisation largely comes from the implementation of the programme on the ground, management of it can perhaps be fully delegated to the JT, which is itself the main body for this implementation in its monitoring, management and coordination functions. Conversely, the evaluation function, although its implementation can be partially delegated or outsourced, is still connected to the initial strategic choices of the programme.

3. Bring in external expertise according to the requirements of each function. To take external expertise, subject of several recommendations in the mid-term evaluation report, the 'strategic committee' function which, in particular, has been experimented as part of the process of brainstorming calls for strategic projects, this still comes

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 51

under the governance function, whereas the concept of operational support could be illustrated by JTS bringing in thematic experts to support the project selection process (relevance of content) and monitoring their results (relevance of outputs in their sector).

4. Communication with other cooperation or thematic programmes must be able to be organised and over the long term. A position may be dedicated to this function based on the specifications in coordination with the Member States. This option takes over the field activities already set up by the Liaison office ENPI/MED.

In conclusion, the options for organising governance are all still based on a prerequisite: ensuring traceability of existing procedures through feedback in order to make strategic decisions in the future that incorporate practices with obligations under the future legal framework.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 52

VI ANNEXES

Evaluation resulted in six deliverables and fifteen notes being drafted. Deliverables were produced in accordance with the term of references of the mission (with the exception of deliverable 6 Benchmarking which was the subject of an amendment in 2011). The notes were drafted as a result of specific requests to support the work by authorities responsible for managing the programme.

LIST OF NOTES 2010-2011-2012:

1) Initial proposals in

the face of multiple

partnerships

developed by some

beneficiaries?

Summary following

In itinere

MA/evaluator

meeting on 12 April

2010.

BACKGROUND: MED project partners are involved with a significant number (6 to 15) of projects that have been selected or are in the the process of being selected; first and second call. Mostly their role is that of a simple partner. This situation carries underlying risks for the programme.

They have limited effectiveness in implementing projects and are often small structures. This problem could lead to a slow increase in expenditure and ultimately risks of disengagement.

They produce mediocre results due to the fact they are often on topics that are very similar or even identical and do not always bring in the local partners needed in order to be operational.

This situation must be offset against the fact that it is widespread across many cooperation programmes and a simple 'statistical' analysis can conceal a wide range of situations.

For an initial analysis we reread the information about them in the available documents (MED evaluation + other programmes + organisations and project

web sites).

2) Proposals for

guidance on

Strategic Project

pre-applications.

Following a meeting

of the selection

committee and

informal SANTORIN

meeting on 28 April

2010.

BACKGROUND: Following the Santorin discussions on any guidance to offer (or not) to applicants in the call for Strategic Projects process and reading the documents available at this stage as well as the summary produced for the Santorin meeting,

There appeared to be a consensus that the approach remains vague. This observation should be put into perspective due to the fact that the applicants only answered at this stage give their position or set a date.

The list of candidates and topics is, in many cases, similar to that of so-called traditional projects.

The desired combination of bottom up and top down is not always present in the terms of reference.

At this stage there are a significant number of pre-applications.

The suggested approach in the PS call for projects documents is 'support' in the project application process (clear document at all stages of the suggested process).

3) Proposals for a capitalisation process and questions for priority 3 measures 1 and 2.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 53

4) MA/Evaluation team

working document:

'Evaluation work

update'

BACKGROUND: This note follows on from the first and second stage of in

itinere evaluation work between January and October 2010. The first stage

covered the evaluation method. The evaluation team attended one selection

and monitoring committee meeting in May and June and the annual event in

2010 and conducted interviews with national authorities. The second stage

focussed on surveys with partners from a range of projects.

5) Options for the last phase of the programme: Evaluation of possible scenarios for using

available funds in the last programme planning phase

6) Extracts from the recommendations summary of 2010 evaluation report

BACKGROUND: This report includes excerpts from the 2010 evaluation

report that will be available in early March 2011. It was drafted so that the

results could be presented for the first time to the monitoring committee in

Valencia on February 22. The evaluation work started in May 2010 and the

following tasks were completed:

• An evaluation scoreboard was developed

• Programme documents and two calls for proposals from 2008 and

2009 were reviewed and benchmarked.

• Interview guidelines for programme authorities were drawn up

• Interviews with authorities in charge of the programme

• Formalisation of project panel to evaluate and an interview guide

drafted

• Conducted interviews (only traditional projects)

• JTS interviews conducted

• Evaluation report drafted

At the same time, the evaluation team participated in various transnational meetings throughout the period, including those relating to strategic projects. In 2011, strategic projects will start being evaluated.

7) Preparation of the deliverable 'List of Recommendations' Meeting of 19 April 2011 in

Marseilles.

8) Preparation of the

Work Plan of the New Programme Working Group or the Experience Feedback Workshop 13 July 2011.

BACKGROUND: The objective of this note is to prepare the work plan of the group comprising the JTS/MA and representatives of Member States to contribute to the preparation of the future programme. The idea for this working group was suggested during an informal meeting of Member States in Marseilles on 6 July 2011 on the preparation of the future programme and was referred to in the report of the meeting. The list of recommendations drafted by the in itinere evaluation team included the creation of an 'Experience Feedback' working group to analyse the currently observable benefits acquired and shortcomings incurred in the course of the projects and the programme in order to formulate proposals for the future. One of the objectives of this work would also be to enlist the participation of decision makers and the associated teams, thematic experts, the capitalisation team and the evaluation team with a view to gradually building up a real programme culture. One of the comments made in the first in itinere evaluation report noted the failure of previous programmes to capitalise the benefits.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 54

9) Analysis of 'capitalisation' documents from the meeting on 20 September 2011

10) Comparative analysis between thematic European programmes and INTERREG MED 11-11-2011

BACKGROUND: In July 2011, a 'Future Med' working group was established by representatives of the national delegations and the Managing Authority and Joint Technical Secretariat (MA/JTS) of the programme. The objective is to apply an experience feedback method to draw on the lessons learned from the current programme and the contributions from other programmes such as INTERACT and ESPON or from EUROSTAT. One of the shortcomings found with regard to establishing well-defined project objectives and calls for projects is the lack of measurement of the added value of MED programme cooperation projects and of those under thematic European Commission programmes. A contribution to the work of the "Future Med" working group will be a comparative analysis between project objectives and project results.

11) Recommendations for the call for capitalisation projects. June 2012

12) Analysis of priority 3 projects: Accessibility of the territory INTERREG MED 30 July 2012

BACKGROUND: Following a request of the programme monitoring committee, the in itinere evaluation team proceeded on the basis of the data provided by the JTS and on the basis of specific surveys, to analyse projects selected for priority 3 calls 1 and 2 (2009 and 2010) and on the basis of Strategic Projects applications. The specific objective of this evaluation was to analyse, based on a few criteria presented in this note, the strengths and weaknesses of such projects and to assess the added value for the programme. This request was made as a result of low consumption on this priority. Meanwhile a specific call on priority 3 was agreed in late 2011.

13) Benchmarking analysis Preliminary phase 12 September 2012

BACKGROUND: The evaluation of In Itinere in 2011 and 2012 focussed on shortfalls relating to the analysis of the added value of projects falling within the scope of the Programme. In this context, the 'Future Med' taskforce requested additional analysis in order to clearly define the objectives of the future Programme and Calls for Projects. This consists of measuring and comparing the added value of Cooperation Projects falling within the scope of the MED programme in relation to those which are part of the European Commission's thematic programmes. This work will contribute to the process of strategically positioning the future MED Programme in relation to Community thematic programmes

14) Benchmarking analysis Phase 1. December 2012

15) Task Force reference framework for the 11 priorities for 2014-2020: Task Force March 2013

BACKGROUND: Participants in the task force meeting of 12 December 2012 called for a reference framework to enable them to link each of the 11 priorities of the future European Technical Cooperation (ETC) regulation to findings from the evaluation of strengths and weaknesses of the projects and of the current programme.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 55

INTERVIEW GUIDE FOR FINALISED PROJECTS.

Guide d’entretien pour les projets MED clôturés (classique) Interview guide for MED finalized projects (traditional)

Résultats, productions & publics cibles/Outcomes, outputs groups

A) Résultats / Results

1) Au vu des réalisations de votre

projet, comment le classez-vous par rapport à sa finalité principale :

Construction ou renforcement d’un réseau d’acteurs? Thématique? Territorial?

Réalisation d’une expérimentation ? d’un transfert de méthodes ou d’innovation ?

Autre?

2) Finalité et (forme des) résultat(s) à court, moyen et long terme (incl. effets induits non prévus

1) Regarding the results and products of your project, how would you consider it with regards to its main finality:

Network building or strengthening? Thematic? Territorial?

Realization of experimentation? Of a methods’ or innovation’s transfer?

Other?

2) Purpose and (form of) result(s) at short, medium and long term (including induced unplanned effects ...)

…/…

B) Productions / Productions

1) En une phrase, décrivez votre

principale production/livrable :

2) Comment classer les principales productions du projet:

• Document méthode? • Base de données? • Feuille de route? • Outil partagé? • Brevet? • Etude, document de recherche,

état de l’art ? • Document de synthèse des

activités et résultats du projet ? • Autres?

3) Quelle est la (les) production(s) qui

fait (font) selon vous que le projet peut être considéré comme un succès ?

4) Est-ce une production prévue

initialement dans la candidature ? Sinon, décrivez le processus qui a fait évoluer le projet et ses résultats ?

1) In one sentence, please describe

your main production/deliverable :

2) How would you consider the main productions of the project

• Method document? • Data basis? • Action plan? • Conjoint tool? • Patent? • Study, research document, state

of the art? • Synthesis document of the

activities and results of the project?

• Others?

3) According to you, which production(s) would make our projects considered as successful?

4) Is it a production initially planned in the application form? If not, please describe the process which changed

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 56

5) A quel moment cette production a vu

le jour, pendant ou après le déroulement du projet ?

the project and its results

5) When the production has been released, during or after the project?

…/…

C) Publics cibles / Target groups

Au niveau des publics cibles :

• Quels sont les publics qui ont participé aux activités du projet ? S'agit-il de ceux prévus dans la candidature ? d’autres ?

• Quelle était la nature de leur participation ? Comment s’est-elle concrétisée ? pérennisée ?

• Quelles étaient les catégories les plus représentées : des élus, des techniciens ou experts, des entreprises ou des représentants professionnels, des scientifiques, des associations ? d’autres publics ?

• A qui ont été diffusé les livrables, les productions, les documents de communications du projet ?

Regarding the target groups :

Which groups did participate in the activities of the project? Were they planned in the application form? Others?

What is the nature of their participation? How did it become a reality? Made durable?

Which were the most represented categories: representatives, technicians or experts, enterprises or professional representatives, scientists, associations? Other types of groups?

For which groups the deliverables, productions, communication documents of the projects have been distributed?

…/…

Pérennisation et impacts/Sustainability and impact

A) Pérennisation / Sustainability

1) Quelles sont les actions de

pérennisation déjà réalisées :

Création d’une association, d’un groupement avec un statut juridique défini ? Constitution d’un « réseau » sans formalisation juridique ?

Dépôt d’un brevet ? Publication professionnelle ou scientifique ? Autre type de reconnaissance « expert » ?

Réunions ou autres activités réalisées prévues après la fin du projet ? au niveau du partenariat ? au niveau local et régional? Transnational ?

2) Autres actions de pérennisation et

suite entreprises à la fin du projet :

Dans le cadre de MED : participation à un projet stratégique ? Autre appel

1) Which are the actions making the

project durable already realized :

Creation of an association, with a defined juridical status? Creation of a “network” without juridical legalization?

Patent’s introduction? Professional or scientific publication? Other kind of experts’ recognition?

Meetings or other activities planned aften the end of the project? With the partnership only? At the local and regional level? At the transnational level?

2) Other actions making the project

durable and pursues undertaken at the end of the project:

In the framework of MED: participation in a strategic project?

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 57

projet classique ? Appel capitalisation ?

Autres programmes?

Autres sources de financement ? Autres acteurs ?

3) Quelle est la valeur ajoutée principale

du projet ?

4) Autres résultats que vous considérez comme importants pour votre projet ?

Other call for traditional project? Call for capitalization project?

Other programmes?

Other fundings? Other actors?

3) Which is the main added value of the

project?

4) Other results that you consider as important for your project?

……./……..

B) Impacts

Quels impacts le projet a eu ou pourrait avoir dans votre région :

Modifications dans les politiques régionales ? comment ?

Diffusion et multiplication des expérimentations ? de la mise en œuvre de la méthode ?

Meilleures connaissances par la diffusion d’informations ?

Autres impacts ? sur quel type de public ? élus, techniciens, experts, entreprises ? Associations, grand public ?

Which are the impacts your projects have or could have in your region :

Modifications in the regional policies? How?

Diffusion and multiplication of experimentations? Of the implementation of the method?

Better knowledges thanks to the diffusion of informations?

Other impacts? On which kind of groups? Representatives, technicians, experts, enterprises, associations, other?

…/…

Gestion & suivi par le programme / Management by the programme authorities

Au terme de votre projet, comment jugez-vous vos relations avec les instances de gestion du programme ?

Avez-vous rencontré une évolution dans le suivi de votre projet par les instances du programme ?

Avez-vous procédé à des changements dans votre projet (budget, partenariat, durée...) ?

Pouvez-vous indiquer selon vous les points forts/faibles de la gestion des projets par les instances du programme ?

Avez-vous des suggestions qui pourraient aider les projets à mieux fonctionner ?

avez-vous des éléments de comparaison avec d’autres programmes ?

At the end of your project, how do you consider the relationship you have with the programme authorities?

Do you think there have been some changes in the management of your project by the programme authorities?

Did you do some modifications in your project (budget, partnership, duration…)?

Could you please indicate us which are according to you the weaknesses/strengths of the programme authorities regarding projects’ management?

Do you have any suggestions that could help the projects working better?

Do you have any comparison’s elements with other programmes?

……./……..

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 58

LIST OF INTERVIEWS

ACRONYME Priority Objective

AGRO-ENVIRONMED BACKGROUNDS BIOLMED CAT-Med CHORD COASTANCE CREPUDMED DEVELOP-MED ETHIC FLORMED FORET MODELE FREE-MED IC MED INNOVATE-MED INSMED MACC BAM MED EMPORION Medgovernance MED-IPPC-NET MEDISS MedLab MEDOSSIC MEMO NOVAGRIMED PAYS.MED.URBAN Philoxenia Planet Design QUBIC RIMED Rururbal SECUR MED PLUS SHIFT SMILIES SOSTENUTO Teenergy schools TEXMEDIN TRANSit WASMAN

1 3 1 2 4 2 4 3 1 1 4 2 1 1 1 1 4 4 2 1 1 1 2 4 4 4 1 1 1 4 2 2 1 1 2 1 3 4

1 1 1 4 2 4 1 1 2 1 1 1 2 2 1 2 2 1 1 1 1 2 3 1 1 1 1 2 2 1 3 1 2 1 2 1 1 1

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 59

DEFINITION OF SELECTION CRITERA OF TRADITIONAL PROJECTS

Grading criteria Grading and weighting Sources and methods of verification

This section sets out the points which were taken into account in the project analysis

The grading and weighting rules are specified in this column.

Reference documents which were used to verify data are specified for each criteria

List of criteria: 1) Project typology - 2) Representativeness of the partnership - 3) Thematic representativeness - 4) Budgetary dimension - 5) Project selection process

As the table above illustrates, each criteria is oriented towards a specific qualitative theme corresponding to the evaluation assumptions and taking into account the key points to be analysed in order to extract information likely to represent added value for the programme analysis.

The first criterion naturally repeats the project classification as established earlier in the document. The weighting indicated for all other criteria also takes into account differentiation between the types of projects.

The five tables below represent, for each criteria, the detail, grading system, and sources used.

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 60

Theme covered by the criteria

Grading criteria Grading and weighting Sources and methods of verification

Thematic representati

veness

1) Correspondence with the MED OP axes / objectives: a. the application form’s compliance with the OP axis/objective eligibility criteria b. account taken of particularities/criteria linked to the thematic direction of the OP axes/objectives

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient: 2

- pre-application and application OP and implementation document

2) compliance with the mainstream OP in the area and related regional/national policies (including their evolution over time)

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient: 2

- pre-application and application - OP and implementation document - national/regional policy guidelines

3) Relevance of the thematic position to transnational MED challenges (emerging from the programme's SWOT diagnosis and analysis

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient: 2

- pre-application and application OP and implementation document

Representativeness of

the partnership

4) Legitimacy of the partners: a. Financial and operational capacity of the project leader (technology and management, recognition by other partners) b. Experience and recognition of partners: in terms of co-operation and expertise with funds, in terms of skills and mission in relation to the project theme c. Regional representativeness of the partners d. Complementarity between the partners, including their shared knowledge, their history of working together, their relevance in terms of the project WP ...

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient: 3

- pre-application and application - supporting and statutory documentation from partner organisations - participation in other projects, in the MED programme, in other programmes and/or earlier programmes - OP and implementation document

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 61

Theme covered by the criteria

Grading criteria Grading and weighting Sources and methods of verification

Project typology

1) Project maturity ("network"

projects): * a. origins and history of the project b. building upon earlier projects at the local or transnational level c. financial model (diverse funding sources, project corresponding to a wider operational field ...) d. form of the results expected from the project

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient:

- pre-application and application - supporting and statutory documentation from partner organisations - participation in other projects, in the MED programme, in other programmes and/or earlier programmes

1A) Innovativeness of the project

("innovative" projects): * a) origins and history of the project b. degree of novelty or innovation in relation to an earlier practice, product or procedure (in terms of the region under consideration and/or sectorial practices / equipment under consideration ...) c. financial model (varied funding sources, project corresponding to a wider operational field ...) d. form of the results expected from the project

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient:

- pre-application and application - supporting and statutory documents from partner organisations - reference documents or expertise (sector, innovation)

1B) Projects with a specific thematic approach ("atypical" projects):* a. relevant of the originality of the thematic position b. what this atypical position brings to the programme c. impact of the position on the financial model and what form the expected project results will take

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient:

- pre-application and application - supporting and statutory documents from partner organisations - reference documents or expertise (sector, theme)

Budgetary dimension

1) Means and resources are sufficient to achieve the project objectives a. in terms of partners' financial/technical capacities b. in terms of thematic and/or sectorial standards for conducting the project c. coherence with the methods for conducting activities and production of project results

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient:

- pre-application and application -supporting and statutory documents from partner organisations -project and programme indicators

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 62

Theme covered by the criteria

Grading criteria Grading and weighting Sources and methods of verification

2) Coherence with how often the programme measure is used in terms of the number and scale of projects as mentioned in the OP indicators/criteria ...

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient:

- pre-application and application + annexes - project and programme indicators

3) Cost-effectiveness of the budget in terms of the form the project results take a. Clarity of leveraging expected from the results of the project (in terms of financial, operational, technical added value ...) b. traceability of budgetary allocation through the results

Grading from 0 to 3 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria Weighting co-efficient: 2

- pre-application and application - project and programme indicators

Project selection process

1) Importance of changes made between phases 1 and 2 of the selection process: a. partnership modifications: retaining the characteristics of the new partnership/goals of the project b. budgetary modifications are coherent with the retention of objectives c. changes to the activity schedule/retention of project objectives

Grading from 0 to 3 0: radical changes risking altering the project 1: significant changes 2: minor changes 3: no changes - the project conforms to expectations

- pre-application and application (including the modified version) - background reports

2) Project grading in phase 1 a. correspondence with the OP evaluation chart/selection criteria b. correspondence with the specificities of the axis/objective requested

Grading from 0 to 3: 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria

- pre-application and application - background report

3) Project grading in phase 2 a. correspondence with the OP evaluation chart/selection criteria b. correspondence with the specificities of the axis/objective requested… with the OP

Grading from 0 to 3: 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria

- pre-application and application - instruction report - notification and convention

Evaluation Report 2012 delivrable 5- Evaluation in-itinere MED, février 2013 63

Theme covered by the criteria

Grading criteria Grading and weighting Sources and methods of verification

4) Re-presentation of an application after an initial rejection** a. Quantitative and qualitative improvements in the application b. Account taken of the remarks initially given for rejection

Grading from 0 to 3: 0: no correspondence with the criteria or criteria not supplied 1: weak correspondence with the criteria 2: good correspondence with the criteria 3: excellent correspondence with the criteria

- pre-application and successive application - successive background reports - successive notifications and convention

* Choice of criteria in terms of the project typology

** Taken into account only where projects fall within the scope of the criteria