![Page 1: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/1.jpg)
Towards a portfolio of additionality indicators:current STI evaluation practices and future challenges
Dirk Czarnitzki and Koenraad DebackereKU Leuven, Belgium
![Page 2: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/2.jpg)
Introduction• During the last decade, mircoeconometric econometric
„counterfactual impact evaluations“ have become an important tool in the area of public (enterprise) support policies.
• It became standard to use econometric methods, such aso Matching estimatorso (Conditional) Difference-in-Difference regressionso Instrumental variable regressionso More recently: Regression Discontinuity Designs
• Not standard (yet): o randomized control trials o „natural experiments“
![Page 3: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/3.jpg)
IntroductionWhy are these methods important?• Firms select themselves into subsidy programs• Goverments pick „winners“• Result:
o treated firms“ cannot be compared with non-treated firms without further adjustment for deriving effects of policies
o „Treatment“ is an endogenous variable in (OLS) regression models, and results would be biased if endogeneity is not addressed properly
![Page 4: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/4.jpg)
Introduction• Therefore all econometric evaluation methods seek to
establish a „correct“ control group approach to derive e.g. the treatment effect on the treated, i.e. o „How many jobs would a treated firm have created
if it had not been treated?“o „How much would have a firm invested in innovation activities
if it had not been subsidized?“o „Which sales with new products would a firm have achieved
if it had not gotten a start-up grant?“
![Page 5: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/5.jpg)
What is the current evidence?• Recent general review* of the available empirical evidence by
the “What Works Centre for Local Economic Growth”o Led by the London School of Economics and Political Scienceo www.whatworksgrowth.org
• 1,700 works (academic journals and policy reports) reviewed• Classied according to the strength of the empirical evidence
o Using a variant of the Scientic Maryland Scale
• Results: only view achieve highest methodological standardso However, limited data availability critically determines the researchers‘
options of applications.* not limited to enterprise support within STI ´policies; review covers all fields of economic evaluations.
![Page 6: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/6.jpg)
Modified Scientific Maryland Scale
5 Randomized control trials, ‘natural experiments‘, no selective sample attrition
4Instrumental variable techniques or RDD, proper balancing (OLS, matching), attrition discussed but not addressed
3Difference-in-Differences, balancing (OLS, matching), but uncontrolled differences likely remain
2‘Before and after‘ comparisons, or a comparison group but without balancing of covariates
1Correlation analysis, no control group, no attempt at establishing a counterfactual
![Page 7: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/7.jpg)
Methods are important, but…• it is equally important HOW we use the methods!• What does this mean? Example: • Evaluation of Eureka‘s Eurostars programme* (total budget
500 million EUR - co-financed by EC = 100 million EUR)• Dirk estimated:
o treatment effect of Eurostars with respect to job creation amounts to a 3.1% higher average annual employment growth-rate when compared to the counterfactual situation of no Eurostars grant
o Extrapolation from regression sample to total programme impact yields about 7,800 jobs created
* http://ec.europa.eu/research/sme-techweb/pdf/ejp_final_report_2014.pdf
![Page 8: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/8.jpg)
What is useful information?• Is this information useful for the policy maker? • Partially yes programme has an estimated positive
impact• HOWEVER, programme might continue to exist anyways.• More useful information would be:
o How can we make the programme better, i.e. more effective and/or more efficient?
• Search for heterogeneous treatment effectso See e.g. Czarnitzki/Lopes-Bento (2013)
![Page 9: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/9.jpg)
Heterogeneous treatment effects within
a policy scheme
![Page 10: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/10.jpg)
Heterogeneous treatment effectso Monetary value of granto Subsidy rateo Firm sizeo Heterogeneity of consortia
• start-ups, large firms, universitieso Proposal quality (Peer-review score)o Multiple grants
• Hünermund/Czarnitzki (2016) find that treatment effect varies with the peer-review score. Better proposals also yield higher treatment effects (but effect is not linear)o Note: LATE in RDD vs. ATT obtained with other estimators.
![Page 11: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/11.jpg)
-20
24
68
400 450 500Score
Heterogeneous treatment effects in Eurostars according to peer-review score (proposal quality)
![Page 12: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/12.jpg)
Heterogeneous treatments and their effects
acrosspolicy instruments
![Page 13: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/13.jpg)
Heterogeneous treatments• Instead of exploring heterogeneous treatment effects within a policy scheme,
• it is also possible to search for heterogenous effects across schemeso Problem: very „data hungry“o E.g. Czarnitzki et al. (2016): enterprise support in
German Cohesion Policy schemes• Also: „dynamic“ treatment effects
o Treatment effect could evolve over time rather than occuring in a single period
![Page 14: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/14.jpg)
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
General
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Consulting
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Innovation/R&D
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Training
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Networking
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Marketing
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Investment
-.2
0.2
.4.6
-3 -2 -1 0 1 2 3 4 5
Labor support
Treatment effect over time for ln(employment) by grant type
95% confidence interval
![Page 15: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/15.jpg)
Indirect effects• Policy scheme may have indirect effects • Example Eurostars: even rejected applications may have effects
o Beware: „contaminated control group“
![Page 16: Czarnitzki - Towards a portfolio of additionaliyu indicators](https://reader036.vdocuments.mx/reader036/viewer/2022070516/5873c2321a28abbc788b6a63/html5/thumbnails/16.jpg)
Conclusions• The use of appropriate econometrics methods increased
significantly in the last decades. • Next steps: • There is still room for improvement with respect to
„identification“o Exploit discontinuities, instruments, experiments
• apply methods in a more useful way for policy making(i.e. beyond homogenous „treatment effects on the treated“ of a single programme)o Design of policy schemeso Selection of policy schemes