@EuritoH2020 #EURITO

Web: eurito.eu

This project has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No. 770420 – EURITO. Disclaimer: This Project has been produced with the assistance of the European Union. The contents of this publication are the sole responsibility of the Consortium and can in no way be taken to reflect the views of the European Union.

Current Challenges for Measuring Innovation, their Implications for Evidence-based Innovation

Policy and the Opportunities of Big DataProfessor Dr. Knut Blind, Fraunhofer FOKUS & TU Berlin

Impact of Research and Innovation Policy at the Crossroads of Policy Design, Implementation and Evaluation

5 November 2018, Vienna

Agenda

• Background: Evidence-based Innovation Policy• Use of Indicators in the Policy Life Cycle• Requirements for Innovation Indicators• Traditional Innovation Indicators• Paradigm Shift in Innovation Indicators• Challenges and Opportunities

Background

• Policymaking in general is complex and dynamic where outcomes are constantly influenced by external factors

• Having the correct information and relevant evidence regarding a policy choice is crucial

• Evidence-informed policymaking: evidence can help explain the policy environment and options, the various effects of different policies and what path to take in order to reach a certain objective or goal

• Evidence comes in many forms and can be: indicators, historical facts, statistics, and results of experiments, texts, quotes from secondary sources, real experiences or histories and expert opinions

• New sources of evidence have an impact on evidence-based policy making

• Innovation policy in particular is highly complex and dynamic driven by new insights in science and technology, but also changes in markets and companies

• Timeliness of information more crucial for innovation policy, but heavily challenged by dynamics

• Multidimensional phenomena of innovation requires multiple indicators, which are challenging policymaking

• New innovation indicators: opportunities, but also challenging policymaking

Conceptual Model for Evidence-Informed Policy

Source: based on Strehlenert et al. 2015

Agenda setting

Policy implementation

Policy formulation

Using the evidenceSourcing the evidence

AwarenessKnowledge

of policy

AdoptionUptake by

target audience

ImplementationIntegration in real

setting

MaintenancePolicy as normal

operations

Policy outcomes

MonitoringEvaluation

Context

Policy Formulation Tasks in Evidence-Informed Policy

Source: own model based on Strehlenert et al. 2015 and Lehtonen 2017

Agenda setting

Policy implementation

Policy formulation

Using the evidence:Problem structuring

Specification of objectivesAssessment of policy options

Identification and design of policy options

Sourcing theevidence

Policy outcomes

MonitoringEvaluation

Context

Contribution of Indicators to Different Policy Formulation TasksPolicy formulation task Contribution of indicators

Problem structuring● Conceptualization of the problem bypolicymakers

● Baseline information (state of the environment indicators, sectoral indicators etc.)● Participatory elaboration of indicators● Indicators as input to participatory policymaking● Indicators as a tool for framing policy problems● Indicators defined according to dominant framings

Specification of objectives ● Forward-looking indicators as feedstock to scenarios● Quantification and simplification● Translation of broad policy aims into specific goals● Indicators as ‘vehicles’ carrying specific visions and worldviews

Assessment of policy options● Comparison of potential impacts ofdifferent options ● Assessment of past and future trends

● Indicators as input to formal assessment methods

Identification and design of policy options● Policy recommendations

● Indicators as input to formal assessment methods

Source: based on Lehtonen 2017

Requirements for Innovation Indicators

• Quality should be credible and analytically sound, i.e. carefully evaluated for their conceptual soundness and minimising measurement error

• Measurability and robustness, i.e. stable and obtainable information with wider coverage of countries as well as time periods

• Transparency of indicators incl. collection methods • Policy neutral, impartial to political motivations• Timeliness of data crucial to be used in the policy making process• Comparability critical for benchmarking, monitoring and evaluation purposes• By reducing information into a concise form, indicators can contribute to the communicability of a

public agenda to the general public.• Accessibility of indicators to users, e.g. in a user friendly format and to affordable costs • Relevance to policy goals as the most critical attribute for indicators

Source: Iizuka and Hollanders 2017

General Problems with Innovation Indicators

• Misinterpretation of innovation indicators, e.g. the more, the better • Inappropriate use of innovation indicators, e.g. ignoring the context• Misuse of innovation indicators for policy purposes, e.g. following policy targets without considering

the context, applying indicators to policy formulation without understanding conceptual design or data collection procedure, relying only on composite indicators

• Overlooked issues of innovation indicators in policy domain, e.g. omitting important source of innovation (non R&D based innovation) or ignoring dynamics in science and technology and their implication on indicators

• Mismatch between users and producers of innovation indicators, e.g. ignoring the delayed results of innovation surveys for policy formulation or the importance of comparability of indicators over time (changing questions in surveys)

Source: Iizuka and Hollanders 2016

Framework of Innovation Indicators

Source: Still et al. 2012

Paradigm Shift in Innovation Indicators

Source: Still et al. 2012

Analog Digital

Innovation CompaniesR&D, closed innovationFew innovation actorsNew technology

TangiblesWaterfall-model of innovationPatents, scientific publications, number of new products

Network of companies, (eco)systemsOpen innovation, co-creationMany innovation actors, including usersNew technology, new services, new processes, new products

IntangiblesAgile innovation, lean start-upsTime-to-market, scalability

Data Surveys, company reportingLack of dataStructured dataStatistically representative samples

Digital footprints of innovation actorsInformation overloadUnstructured, unorganized, incomplete dataBiased data

Indicators Lagging behindManual processes

Table format, some graphs

Possibilities for real-timeEfficient computer-powered processes, though challenging

Interactive, data-driven visualizations, network visualizations, timelines, geospatial representations, (eco)systemic level

New data opportunities

Source: based on Nesta 2017

• World-wide web

• LinkedIn

• Kickstarter

• GitHub

• Meetup

• Crunchbase

• Twitter

• YouTube Etc.

EIS 2017

Source: EIS 2017

EIS 2017

Source: EIS 2017

OECD OSLO Manual (2018)

• “Many of these existing and potential future sources may have “big data” attributes, namely they are too large or complex to be handled by conventional tools and techniques.

• Although useful for different purposes, these data sources all have limitations. Many do not provide representative coverage of innovation at either the industry or national level because the data are based on self-selection: only firms that choose to make a product announcement, apply for R&D funding, or license knowledge from universities are included. Information from business registers and social, entrepreneurship, and R&D surveys is often incomplete, covering only one facet of innovation. Corporate annual reports and websites are inconsistent in their coverage of innovation activities, although web-scraping techniques can automate searches for innovation activities on documentsposted on line and may be an increasingly valuable source of innovation data in the future. Two additional limitations are that none of these sources provide consistent, comparable data on the full range of innovation strategies and activities undertaken by all firms, …., and many of these sources cannot be accurately linked to other sources. Currently, the only source for a complete set of consistent and linkable data is a dedicated innovation survey based on a business register.”

Challenges

• Move from market to system failure and mission oriented innovation policy requires new innovation indicators

• Use of more demand-oriented innovation policy instruments requires new innovation indicators

• Many opportunities of failures from indicator development to indicator use and impact in policy life cycle

• Shift in the paradigm of innovation indicators from analog to digital still not accomplished due to technical and non-technical reasons

• Many opportunities of failures from indicator development to indicator use and impact in policy life cycle

• Old, but also new innovation indicators have to fulfill several requirements, but still several threats of misuse

• Too many data may lead to their ignorance!

Opportunities

• Intrinsic incentives in research community to develop new innovation indicators

• More competition between indicators will increase their quality

• Using behavioral economics and political economy approaches to explain use of traditional indicators and steer change

• Make use of complementary strengths of analog and digital indicators (incl. reciprocal validation by triangulation)

• Combine analog and digital indicators into a common system of indicators

• Make use of visualization of data to improve the understanding

• Strengthen absorptive capacity in indicator knowledge among policy makers by closer interaction with researchers

The R&I Policy Cycle, Existing Data Limitations and New Data Opportunities

Source: EURITO 2017

Moving forward - relevant, inclusive, trusted, timely, open R&I indicators

● Emergent technology ecosystems (AI) ● Nowcasting business R&D ● Technological change indicators ● Standards for the innovation diffusion indicators ● Evidence base for mission-driven R&I ● Advanced R&I funding analytics● Inclusive Innovation ● Linkages and Knowledge exchange indicators (health tech)

http://www.eurito.eu/

EuritoH2020@EuritoH2020 #EuritoEurito

Thank you.

Please send us any feedback or comments at: info@eurito.eu

The EURITO Consortium

This project has received funding from the European Union's Horizon 2020 research and innovation programme under Grant Agreement No. 770420 – EURITO.

Disclaimer: This Project has been produced with the assistance of the European Union. The contents of this publication are the sole responsibility of the Consortium and can in no way be taken to reflect the views of the European Union.