koch taftie-measuring the effects of research
DESCRIPTION
Presentation for TAFTIE on the use of R&D and innovation statistics in policy development.TRANSCRIPT
How to consider the effects of research and innovation investments
Per M. KochHead of the Science Project
2 Norwegian Ministry of Education and Research
Why do policy makers look for numbers?
• To understand the world and how research functions in it
• To develop a story that can be used to convince others to invest in R&D
• We are number fetishists– The prestige of the natural
sciences– The power of macro
economics and the ministries of finance
3 Norwegian Ministry of Education and Research
I am going to talk about:
• The potential and limitations of evaluating the effects of– Innovation and research policy measures
on the aggregate national level– Individual policy instruments
• The fact that the complexity of social systems makes these very challenging tasks
4 Norwegian Ministry of Education and Research
Data sources
5 Norwegian Ministry of Education and Research
The Frascati Manual
• June 1963: OECD NESTI group (National Experts on Science and Technology Indicators) met at Villa Falconieri in Frascati. Italy
• The document sets forth the fundamental definitions (type of activity: basic research, applied research, experimental development; research personnel: researchers, technicians, auxiliary personnel). It primarily deals with measuring the resources devoted to R&D – expenditure and personnel – in the performing sectors: higher education, government, business enterprise, private non-profit organizations.
6 Norwegian Ministry of Education and Research
The Oslo Manual• OECD Oslo 1992:
Guidelines for collecting and interpreting technological innovation data.
• Basis for the Eurostat Community Innovation Survey (CIS)
• Revision 2005 3rd edition:– Marketing– Organizational
innovation– Innovation linkages
7 Norwegian Ministry of Education and Research
Canberra Manual • NESTI Canberra, 1994: The Manual on the Measurement of Human Resources devoted to S&T (the "Canberra Manual")
• The OECD and Eurostat, the European Commission
• Definitions of human resources devoted to science and technology in terms of qualification (levels and fields of study) and occupation
• The International Standard Classification of Education (ISCED)
• The International Standard Classification of Occupation (ISCO)
8 Norwegian Ministry of Education and Research
Other data • Tax authorities• Social security agencies• Higher education
institutions• Accounting data
• Register-based statistical systems using standardized identifiers.– Problems due to
privacy considerations
9 Norwegian Ministry of Education and Research
Beyond evidence based policy development
10 Norwegian Ministry of Education and Research
Evidence Based Medicine
• Evidence-based medicine (EBM) aims to apply the best available evidence gained from the scientific method to medical decision making. It seeks to assess the quality of evidence of the risks and benefits of treatments (including lack of treatment).(Wikipedia)
• Using techniques from:– Science– Engineering– Statistics– meta-analysis of medical literature– risk-benefit analysis– randomized controlled trials
11 Norwegian Ministry of Education and Research
Evidence-based policy
• “…helps people make well informed decisions about policies, programmes and projects by putting the best available evidence from research at the heart of policy development and implementation” (P Davies, 1999)
• Vs. “opinion-based policy, which relies heavily on either the selective use of evidence … or on the untested views of individuals or groups, often inspired by ideological standpoints, prejudices, or speculative conjecture.” (P Davies 2004)
12 Norwegian Ministry of Education and Research
The myth of evidence based policy-development• “…policy making must be soundly based on
evidence of what works” Professional Policy Making for the Twenty-First Century (UK Cabinet Office, 1999)
• But how to you prove what works?– Any social and economic phenomena are a
result of interaction between a limitless number of factors
– Society is constantly changing, a moving target
• Any interpretation of the reality underlying policy-development will be only that: an interpretation
• Researchers are as fallible as other experts, anchored in their own belief systems and prejudices
13 Norwegian Ministry of Education and Research
There are many beliefs and narratives
• Different understanding of:– What society is and how it
works– How research interacts with
society– Common terms (e.g.
“innovation”, “research”)– What the role of science is– What the best theoretical
and methodological foundation for policy development is
• Different cultures have different narratives they use to make sense of their surroundings.
14 Norwegian Ministry of Education and Research
Linear vs. systemic views of innovation
15 Norwegian Ministry of Education and Research
The linear model works perfectly in a policy setting Research and
technological development in universities, RTOs and companies gives birth to an idea and relevant new knowledge
Companies make use of these ideas in the development of new products and processes
The company brings the new product to the market
16 Norwegian Ministry of Education and Research
Why?• It is a good story:
– The unselfish scientist striving for the common good
– It is simple: Give us the money and we will give you the results
– By black boxing the whole process it communicates easily. • You don’t need to know
how the computer works to make use of it
• It works well with many of the traditional macroeconomists
17 Norwegian Ministry of Education and Research
But this model leads to one specific interpretation of data
• A pipe-line model: 1. Research > findings > products > profits2. Science > findings > publications > scientific progress
• Main focus on input indicators – (money, doctorates)
• To the extent there is output data:– Publications and citations– Reported number of innovations in companies– Patents and licenses
18 Norwegian Ministry of Education and Research
Policy consequences of the linear model
• Input indicators as policy objectives:– 3% R&D of GDP (Barcelona)– Number of PhDs
• Focus on technology push rather than market pull
• Oversimplified idea of best practice as regards innovation (“More research!”)
19 Norwegian Ministry of Education and Research
The correlation problem
R&D-investments
Economic growth
It is possible to study the correlation between R&D and growth…
20 Norwegian Ministry of Education and Research
The correlation problem
R&D-investments Research
Economic growth
…but are we really studying the effects of research alone?
21 Norwegian Ministry of Education and Research
The correlation problem
R&D-investments
Research Science Development Learning Innovation Mobility Machinery Technology Education Literature Patents Licenses Design Marketing Branding Culture Regulations Globalization Experience Creativity
Economic growth
There is nothing wrong in having increased investments in R&D as a policy objective, but it cannot be used as a proxy for innovation
“Total factor productivity?”
22 Norwegian Ministry of Education and Research
We are missing a large part of the picture!
• Major problem: – there is no direct
correlation between national R&D investments, productivity and economic growth.
• The Norwegian paradox: seemingly low R&D investments, but record high productivity and economic growth
• The Swedish paradox: very high R&D investments, lower productivity and economic growth
23 Norwegian Ministry of Education and Research
Rogers
Klein
Havelock/Przybylinski
The linear model masks the complexity of the role of research in social systems
24 Norwegian Ministry of Education and Research
In-house learning based innovation
Following the OECD systemic approach: Research is given a new and broader role as a learning tool
market pull
Knowledge about customers and competitors
The company brings new or altered products, processes or services on to the market
Market competencesTacit knowledge
Acquired technology
Literature
Conferences and fairs
Recruits
Acquired R&D
In-house R&D
25 Norwegian Ministry of Education and Research
Company or public institution
Learning NetworksInnovation
Customers and users
Suppliers
Policy-institutions
FinancialInstitutions
R&DInstitutions
Consultants
Public policy Cultural framework
International framework Industrial structure
Understanding competence flows in the innovation system
26 Norwegian Ministry of Education and Research
Challenges beyond the pipe-line
27 Norwegian Ministry of Education and Research
Indirect use of research
• Research is embedded in technology and machinery. We need to measure the effect of research in the use of advanced technology in low tech industries.
• Research is embedded in students and employees. We need to measure the effect research based education has on the learning capabilities of men and women.
28 Norwegian Ministry of Education and Research
The role of competence flows
• Education• User-driven innovation• Customer/supplier
relationships• “Open innovation” and
industry collaboration• KIBS• Global companies as
gateways to the world
29 Norwegian Ministry of Education and Research
We need to get a better understanding of the role of services
• The largest part of the economy
• Interacts with all of industry
• A very heterogeneous sector
• Often left out of statistics
Until now: Industrial policies have been focused on manufacturing, even if 70 percent are employed in services
30 Norwegian Ministry of Education and Research
Understanding the effects of public sector innovation
• Innovation in the private sector is understood as an investment, in the public sector as an expense
• Innovation in the public sector and the effect on industry
• The effect of social welfare on risk taking and company behavior
• We have no output indicators!
31 Norwegian Ministry of Education and Research
The Copenhagen Manual• New Nordic project aimed at developing a
statistics manual for public sector innovation– Danish Agency for Science, Technology and
Innovation, Denmark– Danish Centre for Studies in Research and
Research Policy, CFA, – NIFU STEP, Norway – RANNIS, Iceland – Innovation Norway, Norway– The Research Council of Norway– DAMVAD, Denmark – Statistics Finland – Statistics Norway – Statistics Denmark – Statistics Sweden
In cooperation with: OECD NESTI NESTA UK
32 Norwegian Ministry of Education and Research
Understanding socio-cultural framework conditions
• Stable macro-economic framework conditions– Disciplined fiscal policy– Competition policy encouraging
innovation– Low taxes– An open economy
• Socio-cultural framework conditions– Egalitarian culture with high social
mobility– High wages for blue collar work gives
impetus towards innovation (robots, internet banking)
– High educational levels brings flexibility and labor mobility
– An efficient public sector helps industry– A trustworthy welfare system reduces
risk– Political and social stability gives trust
33 Norwegian Ministry of Education and Research
-1,00000 0,00000 1,00000
Market learning
10,00
20,00
30,00
40,00
Denmark
Germany Finland
Sw edenUK
Norw ay
Slovenia
Spain
Hungary
France
Austria
Be lgium
Czech R
Sw itzerlan
Turkey
GreeceIta ly
Netherland
Poland
Portugal
Slovak R
Ire land
Europe: market learningGDP/ capita
Source:Fourth European Working Conditions SurveyOECD
Understanding socio-cultural framework conditions
• European Working Conditions Survey (Translearn)The correlation between GDP and employer autonomy
Åge Mariussen
34 Norwegian Ministry of Education and Research
We must understand the heterogeneity of innovation systems
• Industrial structure– Large vs. small companies– Low tech vs. high tech– Traditional industries vs. services
• The knowledge system– What kind of universities?
• Research? Polytechnics? Humboldtian?– Other types of research institutions
• Institutes, government labs, academies• Globalization and integration in the world economy• Political stability and social welfare
35 Norwegian Ministry of Education and Research
European innovation scoreboard
• A praiseworthy effort to enlarge the scope of innovation indicators
• But the selection of indicators is slanted towards R&D intensive activities
• The same for the weighing of the indicators underpinning the composite indicator
Norway
36 Norwegian Ministry of Education and Research
EIS 2008
This figure reflects a specific view of innovation as much as it does the facts on the ground
37 Norwegian Ministry of Education and Research
Numbers must be accompanied by stories
• Data is of no value unless they are given in a meaningful context.
• Still: The EU keeps the Innovation Scoreboard but plans scrapping the Trend Chart on Innovation, the tool that interprets the statistics in a national context.
38 Norwegian Ministry of Education and Research
Measuring policy measures
39 Norwegian Ministry of Education and Research
From needs to ex-post impact assessments
• Needs: Problems to be solved• Objective: Outcomes intended to be
achieved• Inputs: Resources mobilized• Processes: Procedures and activities
employed• Outputs: Products and accomplishments• Outcomes: Changes caused by intervention
– Results: Immediate changes for addressees
– Impacts: Longer term socio-economic consequences
OECD DSTI/STP/TIP(2009)18 Henri Delanghe et al
Old focus
New focus
40 Norwegian Ministry of Education and Research
The use of evaluations
• Traditional qualitative evaluations are important– The generate insight into
• Research quality• The organization of
research• Collaboration with other
research institutions– Say little about the effects on
society– Say little about how the
research agenda is shaped through interaction between research and society
41 Norwegian Ministry of Education and Research
Using quantitative measurements for industry oriented policy measures1. The direct effects on the companies participating
1. Economic additionality (Company profits over time)
2. Behavioral additionality (learning, absorptive capacity, ability to change)
2. The direct effects on other companies and organizations1. Spill-over effects on suppliers
(innovation and profits)2. Spill-over effects on customers
(innovation, productivity and profits)
3. Indirect societal effects1. Profitable companies will expand and buy more goods and
services from others2. Their products and services increase productivity in other
parts of society3. Their competences are used by others
(through for instance mobility of personnel)
Incre
ased
com
ple
xity
an
d d
ifficu
lty
42 Norwegian Ministry of Education and Research
Intended and unintended consequences
• We develop policy instruments for a lot of reasons:– To solve a problem– To gain political visibility and influence– Because the Finns are doing it– To make the minister look good
• Policy instruments have a dynamic of their own– Based on the competences, interests
and initiatives of the people involved(They may use the instrument for a new purpose)
– They may have unexpected positive and negative effects elsewhere in the system
• Policy measures outside the realm of research and innovation may have a positive impact on innovation– Social security
Formal objective
Informal objective
Unintended effect
If you want to ascertain the success of a measure, you have to take all of these into consideration
Policy instrument
43 Norwegian Ministry of Education and Research
The role of risk
• Market failure: “When private rates of return, due to high risk, uncertainty or ‘externalities’, may be sufficiently low to deter private investment in areas of crucial long-term national interest.”
• Public measures must allow for higher risk.
• Still: Policy makers ask for high rate of returns!
44 Norwegian Ministry of Education and Research
RCN’s user-oriented R&D programs
• To measure the effects of the program participations in the
• Research done by Møreforskning in collaboration with the Research Council of Norway since the middle of the 1990s.
• Led by professor Arild Hervik.• Based on data from the
databases of the RCN and interviews with the companies
45 Norwegian Ministry of Education and Research
To make sure that the right projects are supported
• Good projects that would have been implemented anyway should not get support
• Low quality projects with no chance of success should be kept out
• You should support good projects where RCN support is vital for its implementation (Input additionality)
46 Norwegian Ministry of Education and Research
Project selection aimed at high social effects (The Jaffe Model)
47 Norwegian Ministry of Education and Research
Understanding the social dynamics of policy instrument development
• An evaluation of the success of a policy instruments must also include an analysis of its social and political context
• I.e.: we can not only mechanistically measure input and outcomes. We must look into the box of policy development.
• Interim evaluations
48 Norwegian Ministry of Education and Research
Traditional view of evidence based policy learning
Independent researchers or experts
Civil servants
Politicians
The wall of disinterested objectivity
49 Norwegian Ministry of Education and Research
An interactive learning arena
Independent researchers
(experts)
Civil servants(experts)
Politicians(experts)
Stake-holders
(experts)
They are all experts in their field, some in technology, others in policy
50 Norwegian Ministry of Education and Research
The ultimate challenge
Analysis of the national innovation system
+ policy instrument evaluations
policy mix reviews
– Green papers– White papers– OECD and EU
innovation policy country reports
VINNOVA’s Portfolio Approach
In order to evaluate public initiatives aimed at promoting innovation, it is necessary to focus on the combined impacts of public efforts.
51 Norwegian Ministry of Education and Research
Conclusions
• There is no simple, mathematical model that can be used to measure all effects of research and innovation.
• We need to combine methodologies, both quantitative and qualitative.
• We are all experts here, and this can not be left to the economists or statisticians
• We have to build common learning arenas, for measuring and for developing an understanding the context of innovation