monitoring and indicators in 2014-2020 workshop 30 april 2013 ines hartwig impact assessment &...

Monitoring and Indicators in 2014-2020

WORKSHOP

30 APRIL 2013

Ines HartwigImpact Assessment & Evaluation Unit

DG EMPL1

General observations on status quo in EU 27

• Number of indicators/priority varies considerably• Some OPs include impact indicators others focus

on financial or output indicators• Not all indicator sets link output with result• Some OP indicators cover large parts of a priority,

others only niche type of activities/target groups• Monitoring systems often distinguish between

Annex XXIII and indicators• In general data on performance seem to be of

lesser importance than financial data. 2

Moving towards result orientation in 2014-202

• Common indicators for outputs and results• Legal obligation to collect personal data on

participants• Collect and store individual participants' data

(micro-data)• AIR only admissible if all data is submitted• Suspension possible if there is a serious

deficiency in the quality and reliability of the monitoring system or of the indicator data

• MIS will be subject to EC audits3

Common ESF output indicatorsPeople Entities

1 unemployed, incl. LTU*2 LTU* 3 inactive*4 inactive, not in education/training* 5 employed, incl. self-employed*

14 number of projects fully or partially implemented by social partners or non-governmental organisations

6 below 25 years*7 above 54 years*

15 number of projects targeting publ. administrations or publ. services

8 with ISCED 1 or ISCED 2*9 with ISCED 3 or ISCED 4*10 with ISCED 5 to 8*

16 number of micro, small and medium-sized enterprises supported

11 migrants, people with a foreign background, minorities (incl. marginalised communities such as Roma)**12 disabled**13 other disadvantaged**

4

Common ESF result indicators

5

Only for a sample of participants.Only reported in 2019 and 2023

6

Indicator

Participant

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

1 1 1 0 0 

0 1 

0 0 

0 

 1 0   1 0  0  0  0  0  1  1 0  0  0  0 

2 0 0 

0 

0 

1 

1 

0 

1 

0 

0  1  0  0  0  0  0   0 0  1  0  0  0  1 

…                                              

Micro data for every participatory recordExample: common indicators

Labour market

age

education

disadvantage

entities

Immediate results

Longer-term results

Total

(AIR)

1 1 

0 

0 

1 

2 

1 

1 

0 

1  1  1  0  0  0  0   0 1  2   0  0  0 1 

How to set targets for common result indicators?

7

Example: Participants gaining a qualification upon leavingResult target is often expressed in %

The % requires a reference value, i.e. the relevant output data.

Example: Target 1: 50% gaining a qualification upon leaving50% of whom? Of all participants in the IP? No, 50% of the low skilled participants (ISCED 1 and 2) gaining a qualification upon leavingTarget 2: 40% of the unemployed participants gaining a qualification upon leaving 8

Programme-specific indicators

• Developed in addition to common indicators• Set at the level of investment priority or specific

objective

9

SROP indicators: Priority 2

1. Number of working age people (15-64 years) who participate in training financed in the framework of the priority

2. Number of people who participate in training serving the improvement of key competences supported in the framework of the priority

3. Enterprises affected by the trainings supported in the framework of the priority

4. Number of working age people (15-64 years) who successfully finish a training financed in the framework of the priority (pass final exam, acquire certificate)

5. Number of people who successfully finish a training which serves the improvement of key competences supported in the framework of the priority

10

11

Comparing monitoring data, Priority 2

Annex XXIII indicator

Annex XXIII data

Indicator data

OP indicator

Total number of participants

275 434 110 805 Number of working age people (15-64 years) who participate in training financed in the framework of the priority

169 170 Total number of reported people (adding up all indicators)

Young people 15-24

165 733

SROP indicators: Priority 31. Number of teachers accomplished a competence-based training in

education methodology supported in the frame of the 3rd priority (head) 2. Number of teachers accomplished a competence-based training in

education methodology supported in the frame of the 3rd priority (head)3. Number of participants in non- formal learning events organized in

cultural institutions and organizations supported in the frame of the 3rd priority (head)

4. Number of accumulatively handicapped students provided for an integrated education (head) (30,8 in 2010)

5. Ratio of 8th class students accomplishing not more than level 3 on the reading and text interpretation

6. Number of primary school task assignment locations training at least 80% handicapped students as a result of the activities implemented in the frame of the 3rd priority (pc)

12

13

Comparing monitoring data, Priority 3

Annex XXIII indicator

Annex XXIII data

Indicator data

OP indicator

Total number of participants

502 318 360 973 Total number of reported participants (adding up all indicators)

Principles for sound indicators

• Integrated set of indicators consisting of CI and programme-specific indicators• With a limited set of targets. Not all indicators need

a target. But the most important ones do.

• Indicators should cover the main scope of a priority (i.e. target group, type of activity etc.)• Targets should also cover large parts of a priority

• Indicators should be simple, data to be collected in monitoring system

• Indicators should have a clear definition14

Principles (2)• Output and result indicators – no impact indicators• Result indicators should be linked to participants/

supported entities, no macro-economic indicators• Result indicators need to be logically linked to

output indicators.• Indicators should be expressed in absolute

numbers (targets could be %)

15

NO: Employment ratio (%) of working ageYES: unemployed participants in employment upon leaving/6 months/12 months after leaving

Steps in designing programme-specific indicators

1. 'Deconstruct' the programme with view to the following categories:1. Target groups (people & entities)2. Type of planned activities 3. Themes4. Projects5. Type of expected results

2. Prioritise within each category3. Develop indicators e.g. by combining characteristics/

features from the various categories4. Review intervention logic to ensure that the most important

aspects are covered by indicators

16

Programme-specific indicators

- EXAMPLES -

17

18

• By combining different common indicators

Advantage: Data have to be collected anyway. Data only needs to be crossed.

Less administrative burden linked to data collection

Programme Output Indicator: Young low skilled inactive participants

19

With primary or lower

secondary education

Below 25 years

Inactive, not in education or training

Result indicator: young low skilled inactive participants gaining a qualification

20

With primary or lower

secondary education

Below 25 years

Inactive, not in education or training

Participants gaining a

qualification upon leaving

Output indicator: low skilled older workers

21

With primary or lower

secondary education

Above 54 years

Employed, including

self-employed

Result indicator: low skilled older workers gaining a qualification

22

With primary or lower

secondary education

Above 54 years

Employed, including

self-employed

Participants gaining a

qualification upon leaving

Longer term result indicator: low skilled older workers gaining a qualification and improving their labour market situation

23

With primary or lower

secondary education

Above 54 years

Employed, including

self-employed

Participants with an improved

labour market situation 6

months after leaving

Participants gaining a

qualification upon leaving

Output indicator: high skilled unemployed

24

With tertiary education

Unemployed, including long-term

unemployed

Result indicator: high skilled unemployed in employment

25

With tertiary education

Unemployed, including long-term

unemployed

Participants in

employment upon leaving

• By combining common indicators with programme specific characteristics/ features

Advantages: indicators can take into account specific aspects of the programme.

26

Output indicator: NEETS

27

Participants between 16 and 25 years

Unemployed, including long-term

unemployed

Inactive, not in education or training

Result indicator: NEETs in education or training

28

Participants between 15 and 24 years

Participants in education/

training upon leaving

Inactive, not in education or training

Unemployed, including long-term

unemployed

Output indicator: start-ups of high skilled young people

supported

29

Start-ups

Below 25 years

With tertiary education

Result indicator: start-ups of high skilled young people

sustained 6 months after leaving

30

Start-ups

Below 25 years

In self-employment six months

after leaving

With tertiary education

Output indicatorDisadvantaged pupils

31

Migrant and minorities

Pupils below 16 years

Disabled

Other disadvantage

d

Result indicatorDisadvantaged pupils in education or

training upon leaving

32

Migrant and minorities

Pupils below 16 years

Disabled

Other disadvantage

d

In education or training

upon leaving

Output indicator : Counselling youngsters (up to 18 years) with migrant background

33

Counselling

Below 19 years

Migrants and minorities

Result indicator : Counselling youngsters with migrant

background who finished an internship upon leaving

34

Counselling

Below 19 years

Migrants and minorities

Internship

• Thank you • for your attention

• Questions?• Comments?

• Contact: Ines Hartwig• ines.hartwig@ec.europa.eu

35