monitoring and effects measurement

Vi investerer i din fremtid

DEN EUROPÆISKE UNIONDen Europæiske Fond

for Regionaludvikling

DEN EUROPÆISKE UNIONDen Europæiske Socialfond

Monitoring and effects measurement

Structural FundS programme in the region oF Southern denmark 2007-2011

• Design and Creative Industries

• Energy Efficiency

• Offshore

• Health Care and Welfare Innovation

• Tourism

• Broad Measures

Analyse_effektmåling_2015_engelsk.indd 1 24-05-2016 11:22:04

w

Regional Udvikling / Strategi & AnalyseDecember 2015

Foto: Lasse Hyldager Fotografi

BackgroundIn order to establish a solid foundation for monitoring and evaluation of the business policy measures, the Danish Business Authority, Statistics Denmark, the five Danish regions and the Regional Municipality of Bornholm have developed a tool that enables the monitoring of those companies that participate in business development pro-jects.

The tool can furthermore be used to assess if the projects have had effects for the participating companies.

In the present report, the tool is utilised for monitoring and measuring the effects of Structural Funds initiatives wi-thin the strategic business development policy areas in the Region of Southern Denmark. All projects within the same area are treated as one in the analysis.

In the English version of the report, a selection of illustra-tive results is included, primarily those concerned with measuring effects.

The report has been produced in collaboration between Statistics Denmark, the Danish Business Authority and the Region of Southern Denmark.


3REGION SYDDANMARK -- regional vækst- og udviklingsstrategi -- /

w

Monitoring and effects MeasureMent

Summary of results

From 2007 to 2011 69 EU-supported projects were initiated in the

Region of Southern Denmark, after recommendation from the Growth

Forum of Southern Denmark. The projects fall under the key priority

areas of Design and Creative Industries, Energy Efficiency, Offshore,

Health Care and Welfare Innovation, Tourism, and Broad Measures. The

Structural Funds projects have reported a total of 5,900 participating

companies. The participating companies are primarily from the 22

municipalities of the Region of Southern Denmark, but there have also

been participating companies from the other danish regions.

employment and turnover growth

A simple monitoring shows that the total employment in the partici-

pating companies increased in all areas. Under the oldest projects, the

highest increase was within Health Care and Welfare Innovation, while

the highest increase among participants in the projects initiated in

2010 and 2011 was within Offshore.

Turnover growth is recorded within five of six areas in both project

periods. Under Energy Efficiency turnover more than doubled from

2009 to 2014 for the group of companies in projects initiated from

2007 to 2009. For projects initiated in 2010 and 2011, turnover for the

participating companies under Offshore increased by 37 per cent from

2011 to 2014.

employment effects

Measuring effects requires more than just monitoring. Effects are

measuring by comparing the partic-ipant group with a control group

to assess if there is a statistically significant difference between the

developments of the two groups.

For the projects initiated in 2007 to 2009, significant employment

effects after five years are found within the areas of Energy Efficiency,

Off-shore, Tourism, and Broad Measures. Similarly, significant employ-

ment effects after three years are found for the projects initiated in

2010 and 2011 within Design and Creative Industries, Energy Efficien-

cy, Offshore, Health Care and Welfare Innovation, and Broad Measures.

The effects measurement indicates that the Southern Denmark Struc-

tural Funds projects initiated from 2007 to 2009 in the subsequent

period from the second half of 2009 until the second half of 2011 have

created employment of 400 full-time equivalents (FTE) within Energy

Efficiency, 800 within Offshore, 320 within Tourism, and 430 within

Broad Measures — in participant companies with employment of less

than 100 FTEs.

For projects initiated in 2010 and 2011 the effects measurement

indicates employment effects in the period from second half of 2011 to

second half of 2014 of 530 FTEs within Design and Creative Industries,

800 within Energy Efficiency, 400 within Offshore, 470 within Health

Care and Welfare Innovation, and 860 within Broad Measures — in

participant companies with employment of less than 100 FTEs.

The figures cannot be summed across key priority areas or project

periods.

turnover effects

The measurement of turnover effects shows that in the period 2009 to

2014 there were significant effects within Energy Efficiency and Off-

shore for projects initiated from 2007 to 2009.

In the period 2011-2014 there were significant turnover effects within

Design and Creative Industries, Health Care and Welfare Innovation and

Broad Measures for projects initiated in 2010 and 2011.

interpret with caution

The results indicate that the Structural Funds projects have had an ef-

fect. Nevertheless, the results should be interpreted with caution. The

effects measurement ensures similar characteristics in the participant

and control group as far as company size, activity, geographical location

as well as growth path prior to the period for which effects are measu-

red. Such similarities cannot be guaran-teed on other characteristics,

such as the motivation and commitment of the companies, their focus

on growth, development needs, etc.

Furthermore, the measurement only covers companies with less than

100 FTEs. The larger companies are excluded in order to avoid their

potentially dominating the measurement. Had they been included,

the effects would possibly be larger, but this would also increase the

uncertainty as to whether the effects could be attributed to project

participation or not.

Structural Funds programme in the Region of Southern Denmark


-- monitorering og effektmåling 2015 -- /4

methodology

The data used is from Statistics Den-mark’s registers on the employment and turnover of Danish companies. Statistics Denmark and the Danish Business Aut-hority have carried out the data analyses, while interpretations and conclusions were made by the Region of Southern Denmark and the Danish Business Aut-hority.

The effects measurement is done by com-paring the development of participating companies with that of a control group of companies, which resemble the partici-pating companies but did not participate in the project themselves. If the group of participants performs better than the con-trol group, e.g. measures by job creation, this indicates that the project has had positive effects.

key priority areasThe present effects measurement focuses on those key priority areas around which the business de-velopment programme in the Region of Southern Denmark is centred:

• HealthCareandWelfareInnovation• EnergyEfficiency• Off-shore• DesignandCreativeIndustries• Tourism• BroadMeasures

All projects within the same area are tre-ated as one in the analysis. The projects that are included in the monitoring and effects measuring have different start and end dates. The earliest were initiated in 2007, while the most recent projects in this analysis started in 2011.

To treat all projects within the same priority area as one, i.e. as one project, provides a useful over-view, and it may furthermore reduce the statistical uncer-tainty in the measurement of effects. At the same time, however, other uncer-tainty elements are introduced, implying that one should be more careful when interpreting the estimated effects.The projects are divided into two groups in the analysis: projects initiated in 2007 to 2009 and in 2010 to 2011, respectively.

Structural Funds projectsThe analysis exclusively concerns Struc-tural Funds projects, of which a total of 69 were initiated in the period 2007-2011 in 6 areas

Following recommendation of the Growth Forum of Southern Denmark, the projects are supported with 563 mn. DKK in EU funds. The EU funds provide co-financing of up to 50 per cent of the total costs of a structural fund project. This implies that the total budget for the 69 projects is at least 1.126 mn. DKK.

private sector participant companiesAll Structural Fund projects report every six months to the Danish Business Authority which specific companies have participated in a project. The 69 projects have reported a total of nearly 5,900 partic-ipating companies.

The monitoring and measurement of ef-fects focuses on the private sector partici-pant companies. Public sector participants will typically also benefit from partici-pating in the projects, but the effects aredifficulttomeasureonthebasisofavailable register data. Thus, public sector entities among the 5,900 participants are excluded from the analysis.

In addition, double records – e.g. a company that has participated in more than one project – are excluded.

A company can only be included under the same key priority area once in each of the two project periods.

A company may, however, participate under more than one key priority area in both project periods. This means that one cannot sum the figures and effects across areas and project periods.

The 69 projects have different start and end dates. The earliest were initiated in 2007, the most recent ones in 2011. More-over, even within the same project, there may be differences in when companies start and finish their participation. The analysis does not account for this.

All projects and participating companies within each key priority area are treated as one for each of the project periods, regardless of the concrete start and end date of the individual project and regardless of when individual companies participated in the particular project.

Monitoring and effects measurement for projects initiated in 2007-2009 and 2010-2011, respectively, within six key priority areas:

• DesignandCreativeIndustries

• EnergyEfficiency

• Offshore

• HealthCareandWelfareInnovation

• Tourism

• BroadMeasures

Focus on private sector company participants:

• The companies’ p-numbers (local kind of activityunit numbers) are matched with Statistics Denmark’s registers

• The development in employment and turnover of participant companies is monitored

• Measurement of employment and turnover effects within the six key priority areas

project start 2007 to 2009:

• 44 projects have reported 2,037 par-ticipating companies. Double records, public sector entities, and terminated or inactive companies are taken out.

• 767 companies monitored in the peri-od from second half of 2009 to sec-ond half of 2014.

project start 2010 and 2011:

• 25 projects have reported 3,848 par-ticipating companies. Double records, public sector entities, and terminated or inactive companies are taken out.

• 1,550 companies are monitored in the period from second half of 2011 to second half of 2014




0

200

400

600

800

1000

1200

1400

2007 2008 2009 2010 2011

# o

f p

arti

cip

atin

g co

mp

anie

s

Design & Creative Industries Energy Efficiency Offshore Health Care & Welfare Innovation Tourism Broad Measures

44 projects started 2007 - 2009have reported 2.037 participating companies

25 projects started 2010 or 2011 have reported3.848 participating companies

69 projects with start in 2007 to 2011 have reported 5.885 participating companiesNo. of participants after starting year and key priority area

Job creation (Fte) turnover effects, mn.dkk.

project start 2007 — 2009 2010 og 2011 2007 — 2009 2010 og 2011

Design&CreativeIndustries — 532 — —

EnergyEfficiency 415 808 2.184 —

Offshore 808 409 569 —

HealthCare&WelfareInnovation —* 473 —* 526

Tourism 320 — — —

Broad Measures 434 864 — 2.098

Indicated total employment and turnover effectsEstimation in 2 nd half of 2014 — 3 and 5 years after project start

*To few participating companies



• The share of participating companies with a positive growth is compared to the correspond-ing share for the control group, i.e. non-participating companies with characteristics similar to those of the participating companies.

• The control group if constructed so as to control for company geographical location, activity, size and histori-cal growth pattern.

• • Effectsaremeasuredfor2009-2014forthosecompa-nies that participated in Structural Funds projects initi-ated 2007 to 2009, and for 2011-2014 for participants in projects initiated in 2010 and 2011.

• • Themeasurementofeffectsisdoneforeachofthesixkey priority areas and produces esti-mates of the abso-lute employment and turnover effects.

how to read the reSultS?

• The tables on the following pages show the “success rate” for the participant and control groups, respective-ly. Successes are companies with a positive employ-ment or turnover growth in the period from second half of 2009 to second half of 2014 or from second half of 2011 to second half of 2014, respectively.

• For example, 44 per cent of the participants in the De-sign and Creative Industries projects initiated in 2007-2009 experienced a positive employment growth from the second half of 2009 to the second half of 2010. The corresponding share for the control group is 33 per cent.

• In principle, the difference between the participant and control groups can be coincidental. This can be tested statistically, however. If the p-value is close to zero

(cf. tables), one can rule out that the difference bet-ween the two groups is due to coincidence.

• The p-value from the Design and Creative Industries example shows that with 11 per cent probability the difference after one year between the participant and control group is due to coincidence.

• Normally, it is assumed that the p-value should be 0.05 (5 per cent significance) or lower in order to be able to rule out coincidence. In the Design and Creative Industries example the p-value is higher, indicating that it cannot be ruled out, that the share of companies with positive growth is the same in the participant and control groups. In other words, there is not a significant difference between the share of successes among the two groups.

how are the eFFectS meaSured?

calculating effects estimates

Significant effects

As the tables demonstrate, significant employment effects after five years

are found for the projects initiated in 2007-2009 under the areas Energy

Efficiency,Off-shore,Tourism,andBroadMeasures.Similarly,significant

employment effects after three years are found for the projects initiated

in2010and2011withinDesignandCreativeIndustries,EnergyEfficiency,

Off-shore,HealthCareandWel-fareInnovation,andBroadMeasures.

The measurement of turnover effects shows that in the period 2009 to

2014thereweresignificanteffectswithinEnergyEfficiencyandOff-shore

for projects initiated from 2007 to 2009.

In the period 2011-2014 there were significant turnover effects within

DesignandCreativeIndustries,HealthCareandWelfareInnovationand

Broad Measures for projects initiated in 2010 and 2011.

calculating employment and turnover effects

Estimates of the employment and turnover effects are calculated for the

areas and project periods where significant effects are found.

For the projects initiated in 2007-2009, estimates are calculated for year 1,

2, 3, 4, and 5. For the projects initiated in 2010-2011, estimates are calcu-

lated for year 1, 2, and 3.

Visualisation of estimated effects

In the figures with employment and turnover effects estimates, the darker

curves show the actual total employment and turnover development for

the participating companies in the periods, respectively, from second half

of 2009 to second half of 2014 and from second half of 2011 to second half

of 2014.

The yellow and light blue curves show the counterfactual development,

i.e. what the effect measure-ment estimates the development would have

been, had the companies not participated in the Structural Funds projects.

The numbers cannot be summed across priority areas or project periods.



design & creative industries

employment turnover

The share of companies with growth The share of companies with growth

project start 2007-2009 Participants Controlgroup P-value Participants Controlgroup P-value

2nd half 2009 — 2nd half 2010 44% 33% 0,11 62% 57% 0,44

2nd half 2009 — 2nd half 2011 48% 35% 0,07 67% 57% 0,13

2nd half 2009 — 2nd half 2012 43% 31% 0,10 72% 53% 0,00*

2nd half 2009 — 2nd half 2013 39% 32% 0,27 62% 52% 0,12

2nd half 2009 — 2nd half 2014 38% 31% 0,33 64% 48% 0,02*

project start 2010 & 2011 Participants Controlgroup P-value Participants Controlgroup P-value

2nd half 2011 — 2nd half 2012 43% 38% 0,07 56% 46% 0,00*

2nd half 2011 — 2nd half 2013 49% 36% 0,00* 60% 47% 0,00*

2nd half 2011 — 2nd half 2014 50% 37% 0,00* 61% 46% 0,00*

energy effekticiencyemployment turnover



2nd half 2009 — 2nd half 2010 48% 41% 0,13 56% 51% 0,25

2nd half 2009 — 2nd half 2011 51% 41% 0,03* 60% 57% 0,48

2nd half 2009 — 2nd half 2012 59% 38% 0,00* 65% 56% 0,03*

2nd half 2009 — 2nd half 2013 56% 38% 0,00* 62% 55% 0,08

2nd half 2009 — 2nd half 2014 60% 38% 0,00* 65% 54% 0,01*


2nd half 2011 — 2nd half 2012 59% 41% 0,00* 58% 48% 0,01*

2nd half 2011 — 2nd half 2013 57% 39% 0,00* 55% 50% 0,16

2nd half 2011 — 2nd half 2014 54% 41% 0,00* 53% 50% 0,46

Design og creative industries, Energy Efficiency and Offshore The share of participating companies with a positive growth

offshoreemployment turnover



2nd half 2009 — 2nd half 2010 65% 36% 0,00* 67% 55% 0,07

2nd half 2009 — 2nd half 2011 62% 39% 0,00* 80% 57% 0,00*

2nd half 2009 — 2nd half 2012 63% 36% 0,00* 80% 52% 0,00*

2nd half 2009 — 2nd half 2013 62% 34% 0,00* 75% 51% 0,00*

2nd half 2009 — 2nd half 2014 73% 35% 0,00* 77% 49% 0,00*


2nd half 2011 — 2nd half 2012 81% 40% 0,00* 63% 49% 0,04*

2nd half 2011 — 2nd half 2013 75% 38% 0,00* 72% 47% 0,00*

2nd half 2011 — 2nd half 2014 79% 40% 0,00* 63% 51% 0,09

7region Syddanmark -- rEgIOnal vækst- Og uDvIklIngsstratEgI -- /

* The share of companies with positive growth in employment or turnover are significantly higher among

the parcipating companies than in the control group (5 per cent significance)



health care and welfare innovation

employment turnover




2nd half 2011 — 2nd half 2012 55% 42% 0,00* 55% 51% 0,43

2nd half 2011 — 2nd half 2013 58% 40% 0,00* 62% 51% 0,01 *

2nd half 2011 — 2nd half 2014 58% 41% 0,00* 64% 49% 0,00*

tourismemployment turnover



2nd half 2009 — 2nd half 2010 48% 40% 0,01* 52% 52% 0,97

2nd half 2009 — 2nd half 2011 48% 42% 0,08 51% 56% 0,18

2nd half 2009 — 2nd half 2012 47% 38% 0,01* 55% 55% 0,99

2nd half 2009 — 2nd half 2013 50% 37% 0,00* 58% 54% 0,19

2nd half 2009 — 2nd half 2014 50% 35% 0,00* 55% 52% 0,30


2nd half 2011 — 2nd half 2012 53% 47% 0,39 51% 53% 0,76

2nd half 2011 — 2nd half 2013 55% 41% 0,03* 59% 50% 0,16

2nd half 2011 — 2nd half 2014 55% 46% 0,15 59% 54% 0,39

Health Care and Welfare Innovation, tourism and Broad MeasuresThe share of participating companies with a positive growth

* The share of companies with positive growth in employment or turnover are significantly higher among

the parcipating companies than in the control group (5 per cent significance)

Broad measuresemployment turnover



2nd half 2009 — 2nd half 2010 59% 39% 0,00* 55% 55% 0,93

2nd half 2009 — 2nd half 2011 59% 40% 0,00* 59% 57% 0,81

2nd half 2009 — 2nd half 2012 58% 35% 0,00* 62% 54% 0,14

2nd half 2009 — 2nd half 2013 52% 34% 0,00* 62% 53% 0,10

2nd half 2009 — 2nd half 2014 54% 34% 0,00* 55% 51% 0,55


2nd half 2011 — 2nd half 2012 55% 40% 0,00* 55% 46% 0,00*

2nd half 2011 — 2nd half 2013 55% 40% 0,00* 60% 49% 0,00*

2nd half 2011 — 2nd half 2014 58% 41% 0,00* 63% 48% 0,00*

to few parcipating companies




2.100

2.200

2.300

2.400

2.500

2.600

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts

Actual development Counterfactual development

64 job*415 job*389 job*324 job*

Indicated employment effects within Energy efficiencyEmployment change in participating companies and estimated job creation

proJect Start 2007 —2009

* The share of companies with positive growth are significantly higher among the parcipating companies (5 per cent significance)

1.000

1.200

1.400

1.600

1.800

2.000

2.200

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


808 job*690 job*

465 job*227 job* 268 job*

Indicated employment effects within OffshoreEmployment change in participating companies and estimated job creation





2.400

2.600

2.800

3.000

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


320 job*284 job*132 job*

92 job*

Indicated employment effects within tourismeEmployment change in participating companies and estimated job creation



1.500

1.650

1.800

1.950

2.100

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


434 job*

419 job*

435 job*190 job* 301 job*

Indicated employment effects within Broad measuresEmployment change in participating companies and estimated job creation





5.000

5.200

5.400

5.600

5.800

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


532 job*575 job*

Indicated employment effects within Design and Creative IndustriesEmployment change in participating companies and estimated job creation


proJect Start 2010 — 2011

4.200

4.400

4.600

4.800

5.000

5.200

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


808 job*575 job*604 job*

Indicated employment effects within Energy efficiencyEmployment change in participating companies and estimated job creation





700

850

1.000

1.150

1.300

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


409 job*

238 job*

150 job*

Indicated employment effects within OffshoreEmployment change in participating companies and estimated job creation



2.000

2.200

2.400

2.600

2.800

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


473 job*268 job*

106 job*

Indicated employment effects within Health Care and Welfare innovationEmployment change in participating companies and estimated job creation





5.200

5.400

5.600

5.800

6.000

6.200

6.400

6.600

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

no.

of f

ull-

tim

e eq

uiv

alen

ts


864 job*633 job*387 job*

Indicated employment effects within Broad measuresEmployment change in participating companies and estimated job creation



1.900

2.400

2.900

3.400

3.900

4.400

4.900

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

mill

ion

DK

K.


2.184 mio.kr.*2.528 mio.kr.*

Indicated turnover effects within Energy efficiencyTurnover change in participating companies and estimated turnover effects





1.200

1.400

1.600

1.800

2.000

2nd half of2008

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

mill

ion

DK

K.


569 mio.kr.*

495 mio.kr.*507 mio.kr.*346 mio.kr.*

Indicated turnover effects within OffshoreTurnover change in participating companies and estimated turnover effects



4.900

5.100

5.300

5.500

5.700

5.900

6.100

6.300

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

mill

ion

DK

K.


Indicated turnover effects within Design and creative industriesTurnover change in participating companies and estimated turnover effects





1.900

2.100

2.300

2.500

2.700

2.900

3.100

3.300

3.500

3.700

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

mill

ion

DK

K.


526 mio.kr.*

1.016 mio.kr.*

Indicated turnover effects within Health Care and Welfare InnovationTurnover change in participating companies and estimated turnover effects



7.700

8.700

9.700

10.700

11.700

12.700

13.700

14.700

2nd half of2009

2nd half of2010

2nd half of2011

2nd half of2012

2nd half of2013

2nd half of2014

mill

ion

DK

K.


222 mio.kr.*

2.227 mio.kr.*2.098 mio.kr.*

Indicated turnover effects within Broad MeasuresTurnover change in participating companies and estimated turnover effects




Regional Udvikling / Strategi & Analyse Damhaven 12 DK - 7100 Vejle

In order to establish a solid foundation for monitoring and evaluation of the business policy measures, the Danish Business Authority, Statistics Denmark, the five Danish regions and the Regional Municipality of Bornholm have developed a tool that enables the monitoring of those companies that participate in business development pro-jects.

The tool can furthermore be used to assess if the projects have had effects for the participating companies.

In the present report, the tool is utilised for monitoring and measuring the effects of Structural Funds initiatives wi-thin the strategic business development policy areas in the Region of Southern Denmark. All projects within the same area are treated as one in the analysis.

In the English version of the report, a selection of illustra-tive results is included, primarily those concerned with measuring effects.

The report has been produced in collaboration between Statistics Denmark, the Danish Business Authority and the Region of Southern Denmark.

detgodeliv.regionsyddanmark.dk

attraktivaktiv

produktiv

VÆKST OG UDVIKLINGSSTRATEGI


monitoring and effects measurement

Documents