2011 evaluation of the innovation voucher programme in poland: results, methods and research...
TRANSCRIPT
2011
Evaluation of the Innovation Voucher Programme in Poland: Results, Methods and Research Opportunities
Polish Agency for Enterprise Development
Brussels, April 14th 2011
Jacek PokorskiAleksandra Jaskólska
Presentation
1. Innovation Voucher Programme
2. About the evaluation
3. Beneficiaries’ characteristics
4. Methodology
5. Key findings
6. The most important recommendations
7. Lesson learned - research opportunities
About the Programme
Pilot project 2008-2010, will be continued in 2011
Financed from domestic funds
The main objective: initiation of contacts between micro and small
entrepreneurs and science institutions
For micro and small enterprises
For research & development services provided by scientific institutions
Voucher’s worth about 3750 euro (15 000zł)
Innovation Voucher Programme
SCIENTIFIC INSTITUTION
(service provider)
Programme’s scheme
PAED
BENEFICIARY
FUNDSAPPLICATION
REPORT ON THE SERVICE
AGREEMENT
CHOICE
Innovation Voucher Programme
SERVICE
REPORT ON THE SERVICE
The evaluation focused on the first two editions of the Programme (2008, 2009) Research duration: september – november 2010 The primary evaluation goal was to assess the effectiveness, utility and sustainability of the programme resultsWe wanted to know more about:Characteristics of applicants, beneficiaries and projects carried out under the 1st and 2nd edition,Programme’s influence on starting business cooperation between enterprises and scientific institutions,Identification of the effects,Looking for proposals for the Programme’s modifications
About the evaluation
Methodology
DESK RESEARCH
PAED REPRESENTATIVES
INDIVIDUAL IN -DEPTH
INTERVIEWS
BENEFICIARIES INDIVIDUAL IN-
DEPTH INTERVIEWS
APPLICANTS WHO WERE GRANTED THE
SUPPORT BUT EVENTUALLY REFUSED
ITINDIVIDUAL IN-DEPTH
TELEPHONE INTERVIEWS
UNSUCCESSFUL APPLICANTS
INDIVIDUAL IN-DEPTH
TELEPHONE INTERVIEWS
SCIENTIFIC INSTITUTIONS
INDIVIDUAL IN-DEPTH
TELEPHONE INTERVIEWS
BENEFICIARIESCOMPUTER
ASSISTED TELEPHONE INTERVIEW
EXPERT REVIEW INDIVIDUAL IN -DEPTH
INTERVIEWS
Inception report
Final report
N=256N=10 N=10 N=13
N=3N=2
N=2
The avarage enterprise age was 9.6 years 66% enterprises were functioning longer than 5 years 55.1% enterprises’ income was less than 250.000 euro, 16.8 % - from 250.000 to 1 mln euro Only 6% Beneficiaries decided to invest their own funds in their projects (projects above 4 575 euro [18 300 zł])
N=256
Beneficiaries’ characteristics
INDUSTRY29,2%
PROFESSIONAL, SCIENTIFIC AND
TECHNICAL ACTIVITY22,9%TRADE AND
REPAIRS15,9%
INFORMATION, COMMUNICATION
12,5%OTHER19,8%
BENEFICIARIES’S SECTORSBENEFICIARIES’S SECTORS
Project’s objective
Beneficiaries were implementing projects which objectives were…
N=256
Key findings
%
9
The use of projects’ results
79.9% Beneficiaries declared that they use project’s results (N = 167)
75,9% enterprises currently produce products introduced or developed
within the Programme
78,5% use technology introduced or developed within the Programme
94,7% sell product certified within the Programme
90% exploit analysis developed within the Programme
83,3% apply solutions supporting enterprises’ management implemented
within the Programme
Key findings
10
Effects of the Programme
Almost 58% of Beneficiaries declared positive influence arising out of their participation in the Programme and c.a. 35% underlined that positive results should occur in the future. According to the Beneficiaries’ opinions, the value of the Programme is related with:improved quality of the offered products (almost 68% of opinions about the very high or high impact of such factor),positive impact of participation in the Programme on the Beneficiary’s image (approx. 61%), competitiveness and market position (57.2%),better management (55.5%). The Programme had lower influence on enterprises’ income, nevertheless such influence was regarded as considerable (32.5% of respondents declared high or very high importance of the supported project’s role in this aspect).
Key findings
11N=256
96.1% Beneficiaries were satisfied or very satisfied with cooperation with scientific institutions (services’ providers), 91.4% declared that no problems appeared during cooperation.A major group of the Beneficiaries (approx. 41%) continued cooperation with scientific institutions upon completion of the service financed under the Innovation Voucher Programme. 46% declared various plans concerning cooperation established with scientific institutions in the future in relation to new research & development projects. Among them almost half (c.a. 54%) intends to launch such cooperation within the next 6 months.
Key findings
Cooperation with the scientific institutions
We are going to start cooperation…
Up to 3 months
Within 3-6 months
Within 6-12
monthsDK
28.0% 26.3% 37.3% 8.5%
N=118
Recommendation 1The period for implementing the project should be at least 4-6 months, which shall facilitate smooth performance and limit the number of beneficiaries withdrawing from the Programme.
Recommendation 2Works should be commenced to launch a sub-programme offering grants of a higher value – approx. 7500 – 12500 euro (PLN 30000 – 50000 ) . If such sub-programme is launched, it shall be absolutely crucial to allow for a longer period of project implementation, approx. 6 months. In case of such sub-programme:A requirement should be introduced for the entrepreneur’s own contribution in the project This shall help minimize the risk of projects of questionable usefulness. The beneficiary’s own contribution cannot be excessive and should range from 10 to 20%.
Recommendation 3As the higher amount of grant shall stir greater interest in the sub-programme, consideration should be given to qualitative evaluation criteria included in the evaluation of applications so as to avoid the necessity of closing the enrollment process too quickly. Unfortunately, the introduction of such criteria may extend the evaluation process; hence, such criteria must be as simple as possible.
Important
MediumMedium
MediumMedium
Key findings and the most important recommendations
The evaluation’s general conclusion is that Innovation Voucher has positive influence on direct contacts and cooperation between micro and small enterprises and scientific institutions.
WHY? Programme has many underlying assummptions, which are not explicitly shown in the Programme’s documentation. There is a risk, that the methods we used didn't revealed all the Programme's causal chain failures. WHAT? Programme's assumptions (not explicitly made)HOW?Mapping out the programme’s theory (reconstruction), In-depth desk researchIn-depth qualitative methodsAn expert panel (involving experts which were implementing similar projects in other MSs)
Better reconstruction of the Programme's theory Focusing on projects’ quality and their sustainability Estimating impact in a more rigorous way
What could be done to improve measurement and conclusions ?
Better reconstruction of the programme's theory
Lesson learned
Research opportunities
Focusing on projects’ quality and their sustainability
WHY? We know that projects are useful for beneficiaries but we know little about their quality. The Programme’s requirements for service providers (scientific institutions) were not very strict, that is why projects’ quality verification process was limited. We also have problems with assessing how works the long-term sustainability of the Programme.WHAT? Projects’s quality and their long–term influence on enterprisesHOW?Case studies (selected projects)Focus Group InterviewsBenchmarking (involving similar programmes implemented in other MSs)
Research opportunities
WHY? We do not know to what extent the Programme itself stimulates cooperation between micro and small enterprises and scientific units. We rely on information about the Programme’s effects declared by the beneficiaries, that is why we need more rigorous measurement relating reliable counterfactual situation.WHAT? programme’s effectsHOW?Control group (problems: limited capability – for example those applicants, who were granted the support but eventually refused it or other scientific institutions’ clients)
Estimating impact in a more rigorous way
Other current PAED’s evaluations
of pro-innovative programmes
“The Innovation Barometer” – on going evaluation of 13 Measures of The Innovative Economy OP, 2007-2013, addressed to polish companies and the Business Support Org.
a systematic approach to tracking the situation of beneficiaries after receiving the support (from programme’s theory, outcomes indicators, tools for research to multi-evaluation scheme implementation - with net effect measurement in certain Measures IE OP)
Long-term impact evaluation of pro-innovative investment grant scheme, implemented during the previous financial perspective 2004-2006
the first results are available on www.parp.gov.pl (a year after The SOP Improvement of the competitiveness of enterprises completion)
Evaluation of the quality of pro-innovative services for enterprises, delivered by the Business Support Org. under The Innovation Economy OP, 2007-2013.
the wide range of the mystery shopping techniques have been applied (evidence-based feedback from potential beneficiaries)
Thank you for your attentionThank you for your attention
Polish Agency for Enterprise DevelopmentPolish Agency for Enterprise Development
81/83 Pańska Street81/83 Pańska Street
000-8340-834 Warsaw Warsaw, Poland, Poland
www.parp.gov.pl
For more information on the evaluation in the PAED, For more information on the evaluation in the PAED,
please contact us:please contact us:jj