monitoring and indicators in 2014-2020 workshop 30 april 2013 ines hartwig impact assessment &...
TRANSCRIPT
Monitoring and Indicators in 2014-2020
WORKSHOP
30 APRIL 2013
Ines HartwigImpact Assessment & Evaluation Unit
DG EMPL1
General observations on status quo in EU 27
• Number of indicators/priority varies considerably• Some OPs include impact indicators others focus
on financial or output indicators• Not all indicator sets link output with result• Some OP indicators cover large parts of a priority,
others only niche type of activities/target groups• Monitoring systems often distinguish between
Annex XXIII and indicators• In general data on performance seem to be of
lesser importance than financial data. 2
Moving towards result orientation in 2014-202
• Common indicators for outputs and results• Legal obligation to collect personal data on
participants• Collect and store individual participants' data
(micro-data)• AIR only admissible if all data is submitted• Suspension possible if there is a serious
deficiency in the quality and reliability of the monitoring system or of the indicator data
• MIS will be subject to EC audits3
Common ESF output indicatorsPeople Entities
1 unemployed, incl. LTU*2 LTU* 3 inactive*4 inactive, not in education/training* 5 employed, incl. self-employed*
14 number of projects fully or partially implemented by social partners or non-governmental organisations
6 below 25 years*7 above 54 years*
15 number of projects targeting publ. administrations or publ. services
8 with ISCED 1 or ISCED 2*9 with ISCED 3 or ISCED 4*10 with ISCED 5 to 8*
16 number of micro, small and medium-sized enterprises supported
11 migrants, people with a foreign background, minorities (incl. marginalised communities such as Roma)**12 disabled**13 other disadvantaged**
4
Common ESF result indicators
5
Only for a sample of participants.Only reported in 2019 and 2023
6
Indicator
Participant
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23
1 1 1 0 0
0 1
0 0
0
1 0 1 0 0 0 0 0 1 1 0 0 0 0
2 0 0
0
0
1
1
0
1
0
0 1 0 0 0 0 0 0 0 1 0 0 0 1
…
Micro data for every participatory recordExample: common indicators
Labour market
age
education
disadvantage
entities
Immediate results
Longer-term results
Total
(AIR)
1 1
0
0
1
2
1
1
0
1 1 1 0 0 0 0 0 1 2 0 0 0 1
How to set targets for common result indicators?
7
Example: Participants gaining a qualification upon leavingResult target is often expressed in %
The % requires a reference value, i.e. the relevant output data.
Example: Target 1: 50% gaining a qualification upon leaving50% of whom? Of all participants in the IP? No, 50% of the low skilled participants (ISCED 1 and 2) gaining a qualification upon leavingTarget 2: 40% of the unemployed participants gaining a qualification upon leaving 8
Programme-specific indicators
• Developed in addition to common indicators• Set at the level of investment priority or specific
objective
9
SROP indicators: Priority 2
1. Number of working age people (15-64 years) who participate in training financed in the framework of the priority
2. Number of people who participate in training serving the improvement of key competences supported in the framework of the priority
3. Enterprises affected by the trainings supported in the framework of the priority
4. Number of working age people (15-64 years) who successfully finish a training financed in the framework of the priority (pass final exam, acquire certificate)
5. Number of people who successfully finish a training which serves the improvement of key competences supported in the framework of the priority
10
11
Comparing monitoring data, Priority 2
Annex XXIII indicator
Annex XXIII data
Indicator data
OP indicator
Total number of participants
275 434 110 805 Number of working age people (15-64 years) who participate in training financed in the framework of the priority
169 170 Total number of reported people (adding up all indicators)
Young people 15-24
165 733
SROP indicators: Priority 31. Number of teachers accomplished a competence-based training in
education methodology supported in the frame of the 3rd priority (head) 2. Number of teachers accomplished a competence-based training in
education methodology supported in the frame of the 3rd priority (head)3. Number of participants in non- formal learning events organized in
cultural institutions and organizations supported in the frame of the 3rd priority (head)
4. Number of accumulatively handicapped students provided for an integrated education (head) (30,8 in 2010)
5. Ratio of 8th class students accomplishing not more than level 3 on the reading and text interpretation
6. Number of primary school task assignment locations training at least 80% handicapped students as a result of the activities implemented in the frame of the 3rd priority (pc)
12
13
Comparing monitoring data, Priority 3
Annex XXIII indicator
Annex XXIII data
Indicator data
OP indicator
Total number of participants
502 318 360 973 Total number of reported participants (adding up all indicators)
Principles for sound indicators
• Integrated set of indicators consisting of CI and programme-specific indicators• With a limited set of targets. Not all indicators need
a target. But the most important ones do.
• Indicators should cover the main scope of a priority (i.e. target group, type of activity etc.)• Targets should also cover large parts of a priority
• Indicators should be simple, data to be collected in monitoring system
• Indicators should have a clear definition14
Principles (2)• Output and result indicators – no impact indicators• Result indicators should be linked to participants/
supported entities, no macro-economic indicators• Result indicators need to be logically linked to
output indicators.• Indicators should be expressed in absolute
numbers (targets could be %)
15
NO: Employment ratio (%) of working ageYES: unemployed participants in employment upon leaving/6 months/12 months after leaving
Steps in designing programme-specific indicators
1. 'Deconstruct' the programme with view to the following categories:1. Target groups (people & entities)2. Type of planned activities 3. Themes4. Projects5. Type of expected results
2. Prioritise within each category3. Develop indicators e.g. by combining characteristics/
features from the various categories4. Review intervention logic to ensure that the most important
aspects are covered by indicators
16
Programme-specific indicators
- EXAMPLES -
17
18
• By combining different common indicators
Advantage: Data have to be collected anyway. Data only needs to be crossed.
Less administrative burden linked to data collection
Programme Output Indicator: Young low skilled inactive participants
19
With primary or lower
secondary education
Below 25 years
Inactive, not in education or training
Result indicator: young low skilled inactive participants gaining a qualification
20
With primary or lower
secondary education
Below 25 years
Inactive, not in education or training
Participants gaining a
qualification upon leaving
Output indicator: low skilled older workers
21
With primary or lower
secondary education
Above 54 years
Employed, including
self-employed
Result indicator: low skilled older workers gaining a qualification
22
With primary or lower
secondary education
Above 54 years
Employed, including
self-employed
Participants gaining a
qualification upon leaving
Longer term result indicator: low skilled older workers gaining a qualification and improving their labour market situation
23
With primary or lower
secondary education
Above 54 years
Employed, including
self-employed
Participants with an improved
labour market situation 6
months after leaving
Participants gaining a
qualification upon leaving
Output indicator: high skilled unemployed
24
With tertiary education
Unemployed, including long-term
unemployed
Result indicator: high skilled unemployed in employment
25
With tertiary education
Unemployed, including long-term
unemployed
Participants in
employment upon leaving
• By combining common indicators with programme specific characteristics/ features
Advantages: indicators can take into account specific aspects of the programme.
26
Output indicator: NEETS
27
Participants between 16 and 25 years
Unemployed, including long-term
unemployed
Inactive, not in education or training
Result indicator: NEETs in education or training
28
Participants between 15 and 24 years
Participants in education/
training upon leaving
Inactive, not in education or training
Unemployed, including long-term
unemployed
Output indicator: start-ups of high skilled young people
supported
29
Start-ups
Below 25 years
With tertiary education
Result indicator: start-ups of high skilled young people
sustained 6 months after leaving
30
Start-ups
Below 25 years
In self-employment six months
after leaving
With tertiary education
Output indicatorDisadvantaged pupils
31
Migrant and minorities
Pupils below 16 years
Disabled
Other disadvantage
d
Result indicatorDisadvantaged pupils in education or
training upon leaving
32
Migrant and minorities
Pupils below 16 years
Disabled
Other disadvantage
d
In education or training
upon leaving
Output indicator : Counselling youngsters (up to 18 years) with migrant background
33
Counselling
Below 19 years
Migrants and minorities
Result indicator : Counselling youngsters with migrant
background who finished an internship upon leaving
34
Counselling
Below 19 years
Migrants and minorities
Internship
• Thank you • for your attention
• Questions?• Comments?
• Contact: Ines Hartwig• [email protected]
35