essays on impact - university of exeter€¦ · university of exeter 7 essays on impact 5 agenda...

7 Essays on Impact

Authors:

Professor David Cope, Life Member, Clare Hall, University of Cambridge; Director,

Parliamentary Office of Science and Technology, 1998-2012

Dr Ian M Carter, Director of Research and Enterprise, University of Sussex, and Chair of the

Association of Research Managers and Administrators

Dr Molly Morgan Jones and Dr Jonathan Grant, RAND Europe

Professor Kaye Husbands Fealing, Center for Science, Technology, and Environmental Policy,

University of Minnesota

Dr Averil Horton, Visiting Fellow, Brunel University Business School

Dr Simon Waddington, King’s College London, UK

Anke Reinhardt, Director of the Information Management Group, German Research Foundation

Edited by:

Dr Andrew Dean, Dr Michael Wykes & Hilary Stevens, University of Exeter

Reference as: Dean et al (Eds) (2013) 7 Essays on Impact.

DESCRIBE Project Report for Jisc. University of Exeter.

University of Exeter 7 Essays on Impact

2

Introduction

Through the Jisc-funded DESCRIBE Project we have sought to undertake a rigorous assessment

of current standards relating to the evidence of impacts arising from Higher Education research.

This document contains seven valuable essays each exploring the topic of Impact. Each essay is

distinct and we have sought to enable selected thought-leaders and Impact experts to both

review the status quo, and to look to the future, making suggestions and recommendations for

the development of Impact in the sector. DESCRIBE has been managed by the University of

Exeter’s Research and Knowledge Transfer team in partnership with the Marchmont

Observatory. We have sought to combine the latest thinking on research Impact with examples

and recommendations which are practical and rooted in the art of the possible.

In the first of our essays, Professor David Cope, until very recently the Director of the

Parliamentary Office of Science and Technology, takes a very personal look at the Strategic

Case for Impact. Whilst in our second essay, Dr Ian Carter, Director of Research and Enterprise

at the University of Sussex, and Chair of the Association of Research Managers and

Administrators (ARMA) examines just how the current Impact agenda is impacting upon

universities as we know and understand them.

Dr Simon Waddington from Kings College London tackles the complex and somewhat

controversial topic of Impact Systems. In his essay Simon considers recent developments in the

implementation of Current Research Information Systems across the UK Higher Education sector

and the ways in which this is influencing the collection and reuse of research information,

particularly focusing on research impact.

Dr Averil Horton, from the Brunel University Business School, explores the use of two key

approaches, Impact as a Journey and Audience to enable valid impact identification and

reporting. Together with accompanying work by Averil on Pragmatic Impact Metrics, these two

key approaches currently form the basis for Brunel’s internal Impact Academy initiative.

Impact methods and methodologies is a large subject area and we have therefore engaged with

two distinct viewpoints. Dr Jonathan Grant and Dr Molly Morgan Jones at Rand Europe jointly

explore whether existing methodologies are up to the task of evaluating impact across differing

sectors and at differing rationales for assessing research impact. Complementing this Professor

Kaye Husbands Fealing from the University of Minnesota looks at the state of the art in some

high profile methodologies.

The international dimension of Impact is critical and Anke Reinhardt, Director of the Information

Management Group at the German Research Foundation tackles this topic for us. Anke

describes a number of national approaches and finds that the impact agenda is likely to continue

influencing research policies for years to come.

Accompanying this publication, and freely available from the University of Exeter’s DESCRIBE

website, is our Final Report, in which we have sought to make specific actionable and tangible

recommendations for the future as we seek to achieve a more nuanced understanding of Impact

and its associated evidence base. It is our hope that these insightful and personal essays will

help to inform the development and direction of the Impact agenda in the UK and further afield.

Dr Andrew Dean, Hilary Stevens & Dr Michael Wykes


3

CONTENTS Page

The Impact School… of Driving, that is... (or The Strategic Importance of Impact) 4

David Cope

The impact of Impact on Universities: Skills, Resources and Organisational Structures 14

Ian Carter

Making the Grade: Methodologies for assessing and evidencing research impact 25

Molly Morgan Jones and Jonathan Grant

Assessing Impacts of Higher Education Institutions 44

Kaye Husbands Fealing

Impact as a Journey - with Audience 62

Averil Horton

Impact Information Management Systems 76

Simon Waddington

Different pathways to impact? “Impact” and research fund allocation in selected European

countries 88

Anke Reinhardt

Acknowledgements 102


4

The Impact School… of Driving, that is... (or The Strategic Importance of Impact)

Professor David Cope, Life Member, Clare Hall, University of Cambridge; Director,

Parliamentary Office of Science and Technology, 1998-2012

Introduction

The word “impact” is indelibly associated in my mind with a romantic interlude in the early 1980s

which frequently found me around Chiswick in west London, the haunt of Idusia, my paramour.

There I would see regularly, gear-crunching around, learner drivers in cars of the “Impact Driving

School”! “Don’t those people know just how infelicitous that name is?”... I always chuckled to

myself. I am afraid it has become a visual ‘earworm’ for me ever since, although Idusia herself

was transient. However, the name does not seem to have done the company any harm at all – it

still exists and has even branched out into coach travel.

I have been asked to look in this thought-piece at the “Strategic Case for Impact” – to cast my

attention beyond the immediacies of the Research Excellence Framework (REF), “Pathways to

Impact” – and similar. That is a tough call – it reminds me of when a prestigious organisation,

celebrating a notable anniversary with a conference, invited me to give a concluding

presentation. “David, we’d like you to talk about the next 100 years”, they said, “Oh, and you’ve

got 10 minutes.” In this piece I will not discuss the ‘academic impact’ element of research impact

– matters such as the development of methods and techniques, the value of training highly

skilled researchers (in their role as researchers) and so on, but rather concentrate on what is

usually referred to as “economic and societal impact”, or “external impact”. The former term of

course encapsulates the ‘rub’ of the entire controversy about impact, because of the risk that “the

expression ‘impact’ … imperceptibly elide(s) with ‘economic impact’”, as Nicola Dandridge

concisely put it1. Her route to mitigation of this risk was a forceful assertion that impact includes

“economic, social, public policy, cultural and quality of life” elements.

The Origins of the Current Focus on Impact

From where has all the attention to impact originated? Of course, there was always discussion of

ideas of ‘value for money’, ‘pay-back’ and the like, ever since the state first became seriously

involved in funding of research at the turn of the 19th to 20th century. This has grown alongside

the inexorable expansion of that funding mechanism, to the point where no-one seriously

assumes that Terry Kealey’s call for the re-emergence of private philanthropy as the main

financial underpinning of research could ever become a reality2. But, I think, the current impact

1 Speech to the Royal Society, October 2009

2 Kealey, T, The Economic Laws of Scientific Research, 1996


5

agenda has emerged, through a dialectical process, in response to the very favourable treatment

given to the research budget by the post-1997 Labour government. This led to a euphoric ‘pay-

dirt’ period in UK research. Cautionary voices, including my own, that said, “enjoy while you can,

because before long, you will be asked to show where the beef is”, were largely ignored. So, we

have seen the ineluctable rise of ‘impact’. I am sure a word occurrence frequency analysis of say

the output of BIS, the research councils and the columns of the THES would show this quite

clearly. It has now reached the point, particularly with the research councils, where it seems that

every publication has to have the word impact in its title. Emblazoned across the Research

Councils UK website is the overall banner “Excellence with Impact”. How long, I wonder, before

someone writes a tease piece with the title “Impact – Schtimpact!”

Pure or Applied, “Curiosity-driven” or Instrumental

I should point out that in this paper there is frequent use of the term “science and technology”, or

the word “technology” alone. Of course, there is research in fields beyond science and

technology – research which is the province of the Arts and Humanities Research Council

(AHRC) and the Economic and Social Research Council (ESRC). The former disburses about

3% of the annual total UK research council spend, the latter around 6%3. The latter research

council was also infamously the subject of a 1981 controversy over its previous title of Social

Science Research Council, with the first two words being seen in some quarters as oxymoronic.

The first three words of the Council’s new title codified a division between economics and ‘social’

research in other areas that few, I think, would see as valid. I have occasionally wondered why

a debate on reverting to the former name has not emerged more recently. Perhaps there is a

feeling that it is better to let sleeping dogs lie.

From some perspectives, the arts and humanities, and possibly also parts of the social sciences,

are more challenged in the impact arena than science and technology. Am I misjudging it when I

say that, for example, reading AHRC literature sometimes suggests that it is really desperate to

demonstrate impact? Thus, there seems to be a strong emphasis on the “creative industries”

and performing arts – and on its relationships with its “Independent Research Organisations” – of

which the British Museum is one of 12. It is not difficult to see why these elements of its work are

singled out to the extent they are.

However, the major divisions that the impact dimension cleaves is not between the humanities,

the social or physical sciences but that between ‘pure’ research (or ‘curiosity-driven’, which

seems the more favoured current term) and ‘applied’ research. While most (but not all)

humanities research could be said to be curiosity-driven; virtually all medical research and the

lion’s share of research funded by the other research councils is ‘applied’.

It is very interesting that often an element in the ‘justification’, if that is the right word, of curiosity-

driven scientific research is what might be called the “serendipitous discovery” argument4. This

might be caricatured as “please give me the money to research oscillatory irregularities in

Cepheid variables – you never know, I might come up with the next thing to smart phones! And

3 It is my impression that, putting aside the exclusively medical research charities, the arts, humanities and social

sciences do relatively well in terms of the proportion of funding that the charitable research foundation sector directs

towards them. I am not aware of any analysis of this however. 4 The ‘serendipity card’ is less frequently played in arguing the case for research funding for curiosity-driven research

in the arts, humanities and social sciences, for self-evident reasons.


6

that’s even if my hypothesis on the causes of irregularities isn’t validated!” There’s usually much

mention of Petri dishes left lying around and uranium salts wrapped in photographic paper. This

resort to serendipity always seems to me a remarkably weak argument. The potential research

funder understandably tends to think, “but how likely is it that you will you come up with some

goodies? Can you give me at least a hint of what are the odds?” – and of course, it is impossible

to answer those questions – because the process is – serendipitous.

Technology (Impact) Assessment

That nagging driving school association I mention above has constantly resurfaced throughout

my career, which has been closely linked to ‘impact’ mainly through its involvement with

“technology assessment” (TA). The term “technology impact assessment” is rarely used, but this

is because the ‘impact’ element is essentially immanent – any ‘assessment’ exercise

necessitates evaluating impact – in fact, it could be argued that TA is nothing more than impact

(or more strictly, potential impact) assessment. Its final stage of identification of the policy

options5 that may be precipitated by the diffusion of a technology is the apotheosis of impact.

Therefore, in the rest of this thought-piece, where I use the term ‘technology assessment’ (TA), it

is used as shorthand for ‘technology impact assessment’.

“Now, hang on,” I suspect I hear at least some of my readers already thinking, “I can see where

this chap is going – he’s trying to argue that there are parallels between examining the impact of

a technology and examining the impact of research! But that’s a false analogy! There’s no way

that academic research is a ‘technology’!” Looking at these two manifestations of human

ingenuity in toto, I would agree with that last retort, even though I also agree with Alec Broers’

2005 Reith Lectures assertion that technology is human-kind’s greatest intellectual achievement6.

I am simply suggesting that there might be some valuable lessons to be learned from testing out

the hypothesis of an analogy in examining impacts. If I am wrong, then the sole impact of my

making the comparison will be to have pointed down a cul-de-sac, thereby wasting people’s time,

and most particularly, my own.

To develop my case, I will have to spend some time looking in a little more detail at TA – and it

turns out, at least in terms of anniversaries, to be a very auspicious time to do this. There is

considerable discussion among specialists about what was the ‘first’ TA7 but I think that there

would be universal agreement that a milestone in the emergence of ‘modern’ TA was the

opening, forty years ago this January, of the US Congressional Office of Technology Assessment

(OTA). This office, in its 23 year life, produced over 700 TAs. Beyond this, it was, to a greater or

lesser extent, the inspiration for the creation of similar functions serving all the parliaments of

major European countries, including the UK, even though in 1995, in a fit of cost-cutting zeal, the

5 The conditionality of the word ‘options’ is specifically identified in the titles of the official TA bodies at the French and

German parliaments and at the European Parliament. 6 2005 BBC Reith Lectures, Lord Broers, The Triumph of Technology

http://www.bbc.co.uk/radio4/reith2005/lecturer.shtml 7 I think that some of the UK’s Royal Commission investigations at the turn of the 19

th to 20

th century on matters such

as sewage and London locomotion qualify. Across the Atlantic, the US National Resources Committee was as early as

1937 looking at Technological trends and national policy: including the social implications of new inventions.


7

Congress voted to suspend the OTA’s budget. In 2003, the US Congress decided it wanted to

re-establish a TA function, which is now located within its Government Accountability Office8.

A Subroutine through Parliament

I should perhaps explain the parliamentary locus of these international TA activities. There are

two main reasons why parliaments particularly have championed TA. The first was out of self-

interest. A feeling arose, over at least the past half-century, that parliaments were losing out in

their critical ability to scrutinise government activity because more and more of that activity

involved issues with a grounding in science and technology - and that parliaments needed an

impeccably impartial source of expertise to bolster their scrutiny of such activity, more so than in

any other area that their scrutinising eye might fall upon. Associated with, but separate from, that

competence-boosting role was a sense that anyway the optimum place for locating a TA function

of any form was within, or controlled by, a parliament, to guarantee the independence and

impartiality of its outputs.

Before I go any further, I had better say that I am not in any way suggesting a formal and

continual role for the UK parliament in examining research impact procedures and outcomes.

There are several reasons why I would reject that. The current economic stringency essentially

rules out any major initiative with cost implications – the parliamentary institution best placed to

conduct a review – the House of Lords Science and Technology Committee9 has regrettably

recently been downsized. This committee did spend some time towards the end of the last

parliament looking at impact and in a 2010 report noted “reservations about the use of ‘impact’ as

a criterion in prospective assessments of individual applications for funding to research

councils.”10 This report was very influential in leading to the speedy decision by the new coalition

government later in 2010 to postpone the introduction of the REF by a year, specifically to re-

examine ‘impact’.

Therefore, I suggest that it would be highly desirable for this committee to revisit the subject at

some point in the next few years. The committee has a very good track record of follow-up

studies and would want to inform itself of how well HEFCE, BIS, the research councils and others

have addressed the committee’s reservations noted above – and about the subsequent

experience of researchers in the entire impact process. It could be a good idea to be planting the

idea of such a review even now. One specific matter which I would be interested to see teased

out by a future inquiry arises from an exchange in one of the committee’s original inquiry

evidence sessions between Sir Adrian Smith11 and Lord May of Oxford12. Without breaching

8 To explain, the title of the Office refers to the accountability of the government to the Congress. It is an office of the

Congress, not of the Executive, whose spending and other activities it scrutinises. 9 I had better flag up that this committee, which operates under the initially broad remit “to consider science and

technology” has always taken a very embracing view that sees all research, in whatever subjects, as falling under its

purview. 10

House of Lords Science and Technology Committee, Third Report, session 2009-10 Setting priorities for publicly

funded research, para. 64 11

In his role as Director General of Science and Research at the then Department for Innovation, Universities and

Skills 12

I quote from the Minutes of Evidence:

Professor Smith, [answering Q 552]: … All things being equal at certain points in the process, particularly if you are

running directed or managed programmes, a better case for impact might shade it [i.e. result in the awarding of a grant,

DRC]. I will be absolutely adamant: the research councils are applying the test of excellence when they are awarding

grants, and it is mischievous of people to say they are not.


8

confidentiality, it would be very valuable to know if (how often), when and how, ‘impact’ has been

used in such a way.

Before returning from this brief detour through parliament, let me touch on one other

parliamentary dimension that DESCRIBE partners have mentioned to me. This is the actual use

of academic research by parliament in its own activities – especially in the inquiries conducted by

parliamentary committees. A ‘rough and ready’ indicator of where the current interests of UK

parliamentarians13 lie is the listing of so-called “All-Party Groups” (APGs) that the UK parliament

publishes14. These are informal ‘clubs’ of like-minded parliamentarians which meet with varying

degrees of regularity to discuss their subject. There are many concerned with scientific and

technological matters15; there is a Social Science and Policy APG but not one specifically

concerned with the arts and humanities overall, although there are APGs for subjects such as

arts and heritage, archives and history and archaeology. The meetings of such groups can be a

good channel for academic researchers to interact with parliamentarians on an informal basis.

More important however are the formal inquiries conducted by parliamentary committees. In the

Commons, each government department has a standing mirror committee to scrutinise its

activities, and there are also a few non-departmental committees, such as the Public Accounts

and Public Administration committees. A comparatively recent development is ad-hoc

committees created specifically to examine Bills, or the operation of an Act some time after it has

come into force.16

An integral part of a committee inquiry is the issuing of a “Call for Evidence” at the time the

inquiry is announced. Parliamentary committee support staff do go to considerable lengths to

make certain that these calls are disseminated as widely as possible, and internet-based contact

groups can usually be relied on to alert their subscribers to any relevant calls issued. Every now

and then, however, some input that would have been useful slips between the cracks. From the

written evidence submitted, committees usually select a few sources whom they invite to be oral

witnesses at a committee meeting. Submissions to committees have to be original work, usually

restricted to 3000 words, and not previously published or circulated elsewhere17 – submitting a

brief note saying, “read my paper in the June 2012 Journal of Irreproducible Results”, will get you

nowhere. A tip which I will share with budding submitters seeking an impact route is, if you can

genuinely do it, to give information on analogous circumstances in other countries, especially

Europe and North America. There is nothing that parliamentarians love more than being able to

explore how things are done elsewhere, so that they can recommend either to emulate, or to

avoid, lessons from overseas.

Q553 Lord May of Oxford: It would not be unreasonable, as anecdotes suggest that on occasion it is used as a tie-

breaker [?]

Professor Smith: Absolutely. That is the context very explicitly where it may well be, but it is not the starting point. No

crap grant application with high impact will get funded, full stop.

House of Lords Science and Technology Committee - Third Report, session 2009-10

Setting priorities for publicly funded research, vol. II, Minutes of Evidence

13 by which is meant members of the Commons and Lords

14 www.publications.parliament.uk/pa/cm/cmallparty/memi01.htm

15 In fact, the ‘overarching’ Parliamentary and Scientific APG was the very first to be created, having its origins as far

back as 1936. 16

The committee system of the House of Lords is less structured – details can be found at

www.parliament.uk/business/committees/committees-a-z/lords-select/ 17

It is acceptable to give references to previous work within the submission itself.

http://www.publications.parliament.uk/pa/cm/cmallparty/memi01.htm


9

Most committees conducting an inquiry will also appoint a “specialist adviser” – an outside expert

to guide the inquiry and to help to evaluate the evidence. It is my impression that the great

majority of such advisers come from academia. This can be a very demanding task, which

receives comparatively parsimonious pecuniary reward but is a key manifestation of impact.

Finally, I should note that committee staff are always interested to receive recommendations on

future inquiries that a committee might be encouraged to undertake.

I will end this parliamentary excursion with a brief examination of the use of academic research

by the Parliamentary Office of Science and Technology (POST) during the 14 years that I was

Director. I should begin by noting that I have huge empathy with academic researchers facing

the challenge of demonstrating impact, as this was a call that POST itself faced on several

occasions! Another thing to note is that the number of academics spontaneously approaching

the office to draw attention to their research activities definitely increased in the recent past –

something I think that is directly attributable to expectations on them to demonstrate impact.

As essentially a ‘horizon-scanning’ institution, POST is continually interacting with the academic

community, not just to be informed on research results but also on research intentions. That

meshes well with a point made by several other commentators that research impact can come

(and perhaps desirably should come) before the completion of a final research report – that it

should be an iterative process from the very beginning of a research project, or even at its

conceptualisation stage. Another way in which the academic community impacts on POST’s

work is by acting as peer reviewers for briefings before they are finally published. The office has

always been extremely grateful to reviewers for this pro bono publico service, which very few of

those approached to assist decline to do.

If any readers are thinking in the backs of their minds that again, this is all very well for research

related to scientific or technological matters, I should point out that POST has always tried to

embrace the humanities as well. This is perhaps best illustrated by a briefing it issued in 2009,

partly in response to the peak of the discussion on “evidence-based policy”. Entitled Lessons

from History, it examined the application of historical analogies to inform current policy debates18.

In returning from this detour, I would admit that there will be some who might argue that

submitting evidence to a parliamentary inquiry is a rather limited form of impact. They might

argue that only if the committee specifically picks up on the research outcomes and favourably

mentions them in its report can impact really be said to have occurred – a fortiori if the

government, in responding to a committee report19, gives further acknowledgement. Those

arguments may be true – but without an initial submission, those two latter augmentations cannot

be a possibility.

Back to TA Generally

Let me return to the practice of TA rather than its locus. It would be quite misleading of me to

suggest that there is some defined corpus of TA research that could be taken, modified a bit, and

then applied directly to the evaluation of research endeavours TAMI ref. Instead, there is a

18 Lessons from History, POST note 323, January 2009

19 By custom, government departments (or the Cabinet Office) issue a response to parliamentary committee reports,

although this is not a formal procedural requirement.


10

somewhat loosely defined assemblage of approaches and techniques20. Expert analysis has

been the dominant approach but for 20 or so years this has been complemented by experiments

in ‘public engagement’ of various forms. Scenarios of various future states of economies,

societies and the technology under consideration are invariably envisaged – scenarios play a

crucial role – with the intention of trying to tease out the multiple contexts of a technology – both

its positive dimensions, and any regulatory guidance that might prove desirable. These

exploratory processes seem to me to have strong relevance to the research impact issue that

Nicola Dandridge has identified where “impact is not yet apparent, or has potential that has yet to

be fully realised”.

This could be a component of my perhaps self-evident conclusion arising from my

juxtapositioning of TA and research impact assessment – that I think it would be very useful for a

study to look at potential analogies in concepts and practices between the two (and I am well

aware of the irony that I am falling back on the old “more research!” chestnut). I would love to

write the impact assessment for that research proposal!

The Ineffable Appeal of Research

Who could be against research – the accumulation of knowledge and understanding and possibly

also wisdom? I suppose that there are a few people, perhaps harking back to the biblical story of

the fall, who would reject entirely the goal of seeking further knowledge21. More significantly,

some research is ruled out by social regulation, such as experiments on humans, animals, or the

use of embryonic stem cells in some cases. Beyond that, there is, of course, a whole machinery

of ethical review, to try to eliminate possible adverse consequences from the carrying out of

research.

I once heard an eminent scientist say that he hoped research on improving the spatial resolution

of climatic change impact modelling would not occur (although resignedly acknowledging that it

would). This was because he feared that this would lead to some countries seeing themselves

as ‘winners’ from climatic change, causing them to soft-pedal on emission reductions. I thought

that an outrageous thing to say.

Perhaps because research is widely seen as an absolute good, the overall demand for research

could be said to be limitless – and certainly far greater than current resources, public and private

– can sustain. There is a sense that the cut-off point, that is the marginal project that is NOT

funded, lies some way away from a Pareto optimal position, consequently, some form of

allocation mechanism is required.

Monetisation

I now want to dare to raise something that I am almost certain will start the brickbats flying in my

direction.

20 These were examined in a collaborative study some years ago between most of the European TA institutes - see

Decker, M. and Ladikas, M (eds.) Bridges between Science, Society and Policy, technology assessment - methods

and impacts, 2004. A follow-up study, called the PACITA project, is currently under way and will report in 2015. 21

Helga Nowotny, in Insatiable Curiosity: Innovation in a Fragile Future, (2010) has advanced a very subtle argument

that the current accumulation of knowledge per se could be seen as dysfunctional but that it is, or should be, made

functional by a complex interaction with innovation. She is not arguing against the conducting of academic research.


11

The relevant, recognised and desirable impacts of research, we are told repeatedly, go way

beyond the purely economic. There are some things in life which cannot be measured in

economic terms – and this includes many research impacts. “Where have I heard arguments like

that before?” I have often thought, “Ah, yes, environmental policy.” Let us, for the sake of brevity,

call this the “you cannot put a value on a sunset” (or a Siberian tiger, or whatever) type argument.

However, over the past 30 years or so, some environmental economists have mused “well,

perhaps you can, at least to some extent, and it is worth having a try because only by monetising

the value of environmental assets is it likely that they will be given the appropriate recognition in

decision-making”22. The basic approaches, fragile though they may be, are probably quite well

known, drawing on things like house price differentials related to proximity to an environmental

‘good’ (e.g. a national park) or amounts people will pay in, for example, travel costs, to access

such an environmental good. There is also a major reliance on direct survey techniques, asking

people questions such as “how much would you be willing to pay to ensure that Siberian tigers

do not become extinct?” Another element to environmental monetisation, one which is probably

the most controversial of all, is attempting to derive the net present benefit of environmental

quality delivered in the future. Could some analogous thinking lead to ingenious ways to address

the matter of delayed manifestation of beneficial research impacts? This harks back to the

‘delayed impacts’ on which I touched in the previous section. However, most of the focus of

environmental monetisation is very much on issues of the here and now.

I have been wondering whether monetisation offers a way, in particular, that purely curiosity-

driven research, whether in the humanities or the sciences, might be able to demonstrate its

legitimacy. Again, let me say straight away that I am not arguing that monetisation could capture

all the value of curiosity-driven research, by any means. I am sure there are some who would

riposte that again I am making a false analogy, this time between the monetisation of

environmental assets and of curiosity-driven research. They would say that while ‘conventional’

economics may have failed to attribute a value to the externalities embodied in a national park,

for example, the park undoubtedly exists as a concrete entity and is not ‘merely’ cerebral, unlike

the outcome of curiosity-driven research. At least some environmental economics research has,

however, attempted to put values on aesthetic and even purely existential dimensions of the

environment.

The AHRC maintains that its research “helps us to interpret our experiences, probe our identities,

interrogate our cultural assumptions and understand our historical, social, economic and political

context. It adds to the economic success of the UK, through its contributions to the knowledge

economy and innovation agenda. The research … can lead to improvements in social and

intellectual capital, community identity, learning skills, technological evolution and the quality of

life of the nation”23. To attempt to corroborate some of these claims, the AHRC has been

exploring some aspects of monetisation related to its supported research since 200424. It asserts

that “research in the arts and humanities has a diverse range of economic impacts, including

direct financial impacts from income generated by the attraction of exhibitions, through to

providing guidance on international relations and policy”. This interpretation leads it almost

inexorably towards an emphasis on its research in the fields of heritage, archaeology, museums,

22 The pioneering UK proponent of this approach was the late Prof. David Pearce, of UCL.

23 Extracted from a general description of the role and work of the AHRC that prefaces many of its recent publications

– see AHRC web site 24

See AHRC, The Impact of AHRC Research 2011/12 and Economic Impact Reporting Framework, 2009


12

etc., but I think it is an interesting exploratory path. However, supposing one were researching

“Reward and Betrayal in the Völsunga saga” – how might one attribute a monetary value to

greater knowledge about ninth to thirteenth century Norse society? Perhaps one might go down

the route of pointing out that this saga underpins the plot of Wagner’s Der Ring des Nibelungen

and teasing out some monetary value from that. This might seem a road too far! The AHRC’s

mention of “guidance on international relations and policy” is also interesting. Needless to say, a

priority area for AHRC international research is “China”. Could greater understanding of Chinese

history and culture contribute to the strengthening of the UK’s contemporary relationships? And

how could one monetise that?

My short answer to the rhetorical questions posed above is that I do not know – and again I fall

back on suggesting some modest research exploration. However, I would also caution that, were

such a study to be conducted, the researchers should constantly bear in mind Wilde’s Picture of

Dorian Gray and its famous epithet about knowing the price of everything and the value of

nothing.

Conclusions – Reflections on Reflections

Sir Adrian Smith25 has said that the aim of impact is to encourage researchers to “reflect” on the

unfolding of their research. Reflection is very much a higher order cerebral activity – with more

than a hint of introspection to it. One does not ‘reflect’ on whether to have boiled or scrambled

eggs for breakfast, but rather on ‘the meaning of life’, ‘human frailty’, ‘what might have been’, and

so on. Again resorting to analogy, perhaps ‘reflection’ is a good way to approach the strategic

implications of impact assessment. Certainly, if asked to reflect on 49 years of involvement in,

and observation of, UK academia, I would immediately respond that it has become more ’worldly’

– a word that I use in a totally positive sense. That is a process that began well before the

current emphasis on impact, but is one which I feel will be considerably advanced by it.

Of course, UK academia is facing some considerable challenges currently, and it is an intriguing

question whether the unfolding of the ‘impact agenda’ can contribute to overcoming at least some

of these challenges. It seems to me that reflection on the aims and values of academia,

examined through the lens of impact, would be extremely apposite at the moment. This should

be not just in terms of individual research endeavours but at departmental, faculty and institution

levels as well. In fact, efforts should go beyond individual academic institutions to the sector as a

whole.

I need not elaborate on the challenge of attracting first-rate international students but I cannot

help wondering whether a contribution to success here might come from setting out clearly, with

convincing substantiation, that advanced study at a UK institution would be set within a

framework of nurturing and encouraging a focus on impact – and that UK academia has an edge

on this compared with rival suitors. “Come to the UK to study – it will benefit not just yourself but

the world!” might be the claim that will motivate much thinking.

25 In the House of Lords Science and Technology committee inquiry referenced above.


13

A Personal Note

In my discussion of curiosity-driven research above, I gave totally made-up examples of research

proposals from the fields of astronomy26 and Old Norse studies. Just to be absolutely clear, in so

doing, I was not in any way being implicitly disparaging about those fields of study – quite the

reverse – they happen to be two areas of intense interest to me – and I consider them highly

deserving of substantial research support!

26 I have absolutely no idea whether there are irregularities in the oscillations of Cepheid variables!


14

The impact of Impact on Universities: Skills, Resources and Organisational Structures

Ian M Carter27 Director of Research and Enterprise, University of Sussex, and Chair of the

Association of Research Managers and Administrators (ARMA).

Summary

Universities and other research organisations are organising themselves to meet the

requirements of the impact agenda. This agenda is not new: a 1993 government white paper

changed the Royal Charters of the Research Councils; medical and other charities have long

been concerned about putting research into effect; and Bacon was speaking about it in the early

17th Century. It is about enabling, understanding, and describing the beneficial outcomes of

research.

Deriving benefits from research is not a linear process; nor is in unidirectional. It is multi-iterative,

parallel, multi-dimensional, and absorptive, and involves both push and pull. Dissemination,

translation, development, transfer and use need to be embedded in the research process, to

greater or lesser extent, not bolted on. Research and innovation are blurred in practice; and

innovation relates to policies, processes, attitudes and cultures, as well as to technology. Our

mechanisms and support therefore need to be able to cope with all of these aspects. There are

burdens associated with capturing relevant information, and writing for and discussing with

multiple audiences. Translation into practical use can generate conflicts between commercial

and public benefits that need to be managed and be reflected in an institution’s risk appetite.

A broad range of skills are needed, typically embodied in a team of individuals. Universities have

been investing in knowledge exchange posts for some time, on the back of earmarked funding,

but there has been a recent increase in activity dedicated to documentation of research impact

as a consequence of the Research Excellence Framework (REF). Such activity is supported by

both employed posts and contracted services. Skills development frameworks are now available,

from ARMA, AURIL, and Vitae, to support the individual and the institution in assessing and

gaining the necessary capabilities. Areas of knowledge and skill that are particularly required are

in communication to ensure access and understanding, commercial interactions to ensure value

across a portfolio, and in operational processes to ensure implementation and delivery.

Making non-academic impact a part of research applies as much to the support structures as to

the researchers and what they do. Time needs to be allowed for it, as it is essentially a people-

based process. Universities, being broadly-based, have multiple sectors and markets to

understand, which is a substantial challenge that is often not recognised by ourselves or others.

The policies and funding mechanisms of government and other research funders are strongly

encouraging institutions and individual researchers to take impact seriously, and most are now

doing so. Successful institutions will have flexible, adaptable approaches, with staff with

appropriate knowledge, skills and behaviours, in a resource environment that provides the ability

to respond.

27 These views are expressed in a personal capacity, and do not represent those of either organisation.


15

Introduction

The impact agenda has been with us for some time, but it is perhaps only in the last couple of

years that many of us have noticed. The 1993 white paper, Realising our Potential28, included

aims relating to wealth creation and the quality of life. Subsequent spending reviews, white

papers, and policies (e.g. the ten-year Science and Innovation Framework of 200429) have all

strengthened the relationship (actual, assumed or desired) between research, innovation,

enterprise, and societal benefit.

The higher education sector has often been directly involved in such developments, partly as a

means to protect and extend its funding allocations from government. We should not be

surprised that government (in particular, but not alone) wishes to see some tangible effects from

its investments. The challenge to the sector has always been to demonstrate the range of those

effects such that it justifies the funding and retains its freedoms to operate across the range of

disciplines, methodologies, modes, and mechanisms.

The 1993 white paper also created the current form of the Research Councils, in which their

Royal Charters include reference to “contributing to the economic competitiveness of Our United

Kingdom and the quality of life”30. Later modifications to the standard wording (EPSRC in 200331

and AHRC32 in its founding Charter in 2005) also include “the promotion and support of the

exploitation of research outcomes”.

It is not only governments that wish to see beneficial effects of research: much medical research

is funded through charitable bodies who wish to see progress in diagnosis, prevention and

treatment of illness and other conditions. Such desires link both to their own charitable objects,

and also to the means by which many raise their funding: by appealing to individuals and

organisations to donate, based on descriptions of the suffering as a consequence of the condition

and demonstration of the benefits arising from research.

The impact agenda has thus been part of mainstream governmental funding for twenty years,

rather than just the last, say, five years, as perceived by some parts of the community. But it

goes back somewhat further than this, as illustrated by Bacon:

‘…the real and legitimate goal of the sciences is the endowment of human life

with new

inventions and riches’ (Francis Bacon, Novum Organum, 1620)

Enabling, understanding, and describing the beneficial outcomes of research transcend funders

and centuries, and are a fundamental part of research culture. It is as a consequence of some

elements of currently required procedure that some of us feel uncomfortable, whether through

reaction to perceived interference in research direction, or possibly through diffidence in the

“usefulness” (in the broadest sense) of one’s own research.

Universities and other research organisations, and the individuals therein, have been engaging

with impact for many years, consciously and subconsciously. Many have taken explicit

28 http://www.official-documents.gov.uk/document/cm22/2250/2250.pdf

29 http://webarchive.nationalarchives.gov.uk/+/http:/www.hm-treasury.gov.uk/spending_sr04_science.htm

30 http://www.epsrc.ac.uk/about/history/Pages/royalcharter1994.aspx

31 http://www.epsrc.ac.uk/about/history/Pages/royalcharter2003.aspx

32 http://www.ahrc.ac.uk/About-Us/Policies,-standards,-and-forms/Royal-Charter/Pages/Royal-Charter.aspx

http://www.official-documents.gov.uk/document/cm22/2250/2250.pdf

http://webarchive.nationalarchives.gov.uk/+/http:/www.hm-treasury.gov.uk/spending_sr04_science.htm

http://www.epsrc.ac.uk/about/history/Pages/royalcharter1994.aspx

http://www.epsrc.ac.uk/about/history/Pages/royalcharter2003.aspx

http://www.ahrc.ac.uk/About-Us/Policies,-standards,-and-forms/Royal-Charter/Pages/Royal-Charter.aspx


16

measures, willingly or through various forms of coercion, whilst other developments have been

incremental or implicit. This paper explores some of the necessary features and elements of the

approaches taken.

Approaches to Impact

The embedding of engagement with non-academic participants and audiences into the research

process, and the support of the translation of research findings are becoming more common in

higher education. The boundaries between research and innovation are being blurred. The

effects of this can be positive on the research, in terms of the hypotheses posed, the information

collected, and the understanding produced. This is to be welcomed, but it does need conscious

effort to embed it in the majority of research activities, rather than to try to bolt it on at the end.

Indeed, there is a long-existing false perception that research to innovation and impact is a linear

model. Rather, the processes involved are multi-iterative, parallel, absorptive, multi-dimensional,

and multi-party, all of which makes them difficult to model, manage, and enable.

Some of the challenges for institutions and researchers are about cultural acceptance, whilst

others are about process and infrastructural support. The ability to capture, package,

communicate and translate research findings requires one or more mechanisms, and quite a

range of skills. Some will be the responsibility of the researcher, but some will involve

professional service staff within the institution or external professionals. Examples are:

The systematic capture of relevant information, without creating significant burden. Impact-

related information is likely to be both qualitative and quantitative, sometimes of a fuzzy nature,

spread over a considerable time period and from many sources. The information will need to be

analysed and constructed into meaningful packages. This is not something for which standard

administrative transactional systems have been designed.

The ability to write for multiple audiences, both highly specialist and more broadly based. The

targets for material are not the general press, but do include the specialist and trade press, as

well as trade and sector bodies, research funders, policy makers, and the wide range of current

and potential end users.

The interaction with organisations and individuals outside academia, to promote and explain the

research and what it means, and to negotiate its further development, translation, exchange,

transfer, and use. Acting in both commercial and wider public benefit modes at the same time

can be somewhat contradictory.

It is worth noting at this point that there is a subtle distinction between the capture and provision

of information about research in order to enable its use, and the capture of information about the

application and impact of that research. It is arguable that the former is more important than the

latter (as it is enabling), but that the latter currently has a greater focus because of the need to

report on it in the context of the Research Excellence Framework (REF). Thus, institutional

mechanisms need to address both elements, with the organisation ensuring that it is comfortable

with the balance between them.

The second example above highlights the need to make access to and understanding of

research findings easy. This is not simply about making research publications free at the point of

access (the Open Access agenda), although that is an important element. It is also about

enabling understanding and hence use.


17

The third example also introduces the matter of risk taking or acceptance. Universities and

publicly-funded organisations are often concerned about risk, especially reputational risk, but

sometimes struggle with commercial risk and opportunity risks. The speed of deliberation and

decision, along with constrained processes can also conflict with enabling use of research and

institutional knowledge capacity. Potentially, some operational aspects intended to manage and

reduce risk can actually have the effect of increasing risk at a strategic level. The impact agenda

is one of the current areas testing our systems and policies in this respect.

To support these activities requires us to have appropriate skills, resources and structures in

place. The following sections discuss each in a little more detail.

Skills

The range of skills required to enable, support, and undertake the development and translation of

research results is quite broad. Some are very closely related to those required to undertake

research, whilst others come from different fields. Some are technical, whilst others are soft and

people-related. All such skills could be embodied in a single individual, the researcher, who

would need to be something of a polymath. In most cases, however, one achieves this through a

number of people, acting together as a team in order to encompass the set of skills and

expertise, and to allow sufficient time and effort for both the research and the translation.

Similarly, there are questions and debates about how to organise the pools of skills, which will be

addressed more fully in the next section.

Institutions are taking various approaches to obtaining or accessing the necessary skills. These

include amending existing roles to incorporate relevant features and responsibilities, creating

new roles (e.g. there have been quite a number of adverts for impact-related staff over the last

12-24 months), buying in expertise and services, modifying resource allocation and reward

mechanisms, and adjusting recruitment and promotion criteria.

The provision by the Funding Bodies of specific funding to support knowledge exchange

capability and capacity (e.g. the Higher Education Innovation Fund, HEIF, in England, the

Knowledge Transfer Grant, KTG, in Scotland) encouraged institutions to think about how to

engage and deliver in this space. This has typically led to the employment of staff with specific,

relevant skills, often to help bridge the gap. However, such arrangements have not always been

adequately integrated with the research activities (in the broadest sense), and hence perhaps

have not been as effective in all cases as they might. It is interesting to see, currently, questions

about how knowledge exchange and technology transfer staff and units can become more

involved in the impact agenda, when one might have expected them already to be at the centre

of it.

As noted, there are opportunities for obtaining the necessary skills from outwith the institution.

There seems to be an explosion of providers, with at least one email a week offering services.

These are typically tuned towards the specific (perceived) needs of the REF, with some being,

frankly, cynical attempts to jump on a bandwagon and to exploit the natural worries of academic

units. This is not a sustainable approach. Institutions should be looking to embed expertise into

their routine processes and activities, so that they are fit for purpose, for delivering high quality

research and enabling its appropriate translation into practical benefit, not just responding to a

particular (albeit very important) assessment process.


18

There are also a number of partnerships between universities and commercial bodies in relation

to the exploitation of intellectual property. These can be very effective, bringing commercial

knowledge and access to investment, but they tend to be concentrated on technology

exploitation, thus not addressing the vast range of available know-how in all subject areas.

Having spoken about accessing skills, it is worth exploring a couple of specific types of skills,

given their centrality to enabling impact.

Communication Skills

Communication skills are essential to research. To persuade one’s peers and one’s funders that

particular questions are worth answering. To encourage participation in the research activity

itself. To formulate and disseminate the results to multiple audiences. To explain the value of the

subsequent answers. To allow others to make use of the results of research.

All of these involve interaction with individuals or communities outside the field of the research

and outside academia, which typically also requires the use of variations in language. How good

are we at doing this? How consciously do we form our messages for those different purposes?

A 2011 report commissioned by the Open Access Implementation Group (OAIG) on the Benefits

to the Private Sector of Open Access to Higher Education and Scholarly Research33 noted that

academic outputs are not always understandable, and hence usable, by non-academic

audiences. (The same might be true about their use by other parts of academia.) One of the

arguments in favour of open access is about the use of results for socio-economic benefit. If the

content of research outputs is not understandable, they will lose their value in that context. Thus

we, individuals, institutions, the scholarly communications system, need to ensure that we enable

understanding, such as through lay abstracts and summaries, clarity of messages, and so on.

A caveat to this observation is that the communication needs to retain its richness of information,

and we should not be tempted to create spin or to cater only to particular interest groups or

mechanisms. Institutional communication mechanisms need to aim at the well-informed, rather

than the lowest common denominator, and to target relevant types of outlet and media. The

purpose of this communication includes initiating and informing debate, leading to other insights

and effects. Structures and the skills to enable discourse must be supported.

This therefore leads to the conclusion that one should be developing the communications skills in

all relevant staff and also defining the research requirements of one’s institutional

communications strategy and processes. This would aim to provide some clarity (and

performance measures) about the forms, frequency and volume of communications activity, and

the end points that one has in mind.

Commercial Knowledge and Skills

Assisting impact from research is not solely about economic impacts. However, some is, and

hence a knowledge of commercial mechanisms and drivers is necessary. Such knowledge is

important in order to undertake research in a contractual environment, not just in the commercial

exploitation of research results.

33 http://open-access.org.uk/wp-content/uploads/2011/10/OAIG_Benefits_OA_PrivateSector.pdf

http://open-access.org.uk/wp-content/uploads/2011/10/OAIG_Benefits_OA_PrivateSector.pdf


19

Commercial skills and knowledge include understanding one’s market and customer, valuing (or

determining the value in) information, knowledge and facilities, and negotiating terms.

Negotiation can be as much about getting the overall purpose and structure of an arrangement

right as it is about agreeing the detail. Indeed, the latter should follow the former, which is not

always the case.

There is also a range of related technical knowledge, in particular about intellectual property (IP)

and legal matters. These typically require specialist professionals, both inside and outside the

institution, but the researcher and research manager also need to have a well-developed

understanding of the area, in order to be able to make the relevant decisions.

It is worth noting that commercial skills are as important in dealing with governmental and non-

governmental, non-commercial organisations as they are essential when dealing with commercial

organisations. This is because such skills are about recognising and agreeing value, and the

sharing of its benefits.

Very much related to commercial acumen are policy understanding and political nuance. Being

able to attune oneself to the needs of both academic researcher and non-academic user of

research helps to enable the relationship to develop and prosper. Recognising when to drive a

hard commercial bargain and when to encourage and allow the diffusion of knowledge without

direct gain is essential. Such skills can also be useful if and when tensions crop up.

Process and Systems Skills

Enabling impact involves a range of interactions, with a variety of people and organisations.

Being able to market and sell expertise and facilities, navigate the procedures, such as the

delights of structural funding, as well as negotiate terms, and meet all the internal process

requirements, can be challenging and frustrating. The out-going marketer is not necessarily the

best person to complete the internal costing, pricing and authorisation form, or to negotiate the

detailed contract terms.

Supporting processes, internal and external, is a necessary part of this, and it is dangerous to

undervalue this area in comparison to other areas of business development. Ultimately, this

provides the implementation of the ideas and agreements, manages the risk, provides the

governance assurance and due diligence, and captures the evidence of outcomes.

Institutional research management systems are in a state of evolution. A range of development

is still required, to enable adequate capture, analysis and use of whole-life information, to take us

out of the transactional and into the aggregated portfolio.

Codifying and Accessing Areas of Skill

The potential range of roles, skills and knowledge is illustrated by the list below, first used in

2004, which contributed to the creation of the ARMA Professional Development Framework

(PDF)34, launched in November 2011.

Diplomat, politician, people manager and motivator

Organiser, operational manager, project manager

HR, finance, estates

34 https://www.arma.ac.uk/professional-development-framework

https://www.arma.ac.uk/professional-development-framework


20

Legal (contract, IP, company, employment)

Systems, information, e-business

Marketing, PR, communicator, spokesperson

Technical subject knowledge, market needs

Policy maker and interpreter

The PDF is not the only relevant codification: AURIL have a Knowledge Transfer CPD

Framework35, and Vitae have created the Researcher Development Framework36. Whilst each

has a specific focus, there are overlaps and similarities, reinforcing the need for the range of

skills to be identified and supported.

These frameworks provide a means for an individual to identify areas in which they might seek to

develop their skills, and for an organisation or line manager to construct job descriptions and

training plans. They also help training providers to identify opportunities.

The task for the institution is to decide how much of this skill base should be in academic

researchers, and how much in professional service staff. Also, how much comes from within the

organisation and how much is outsourced to specialists. We certainly want and need our

researchers to have an appreciation of these areas of non-academic knowledge, so that they are

better able to frame and undertake their research, but they do not need to be the institutional

experts.

A related question is how one sources professional services staff in these areas. Should they

come with a research background, the better to understand their researcher colleagues? Or from

a business or governmental background, in order to speak the customer’s or funder’s language

and know how to navigate their structures? For staff from the professions such as law, how

much research contractual experience is necessary? Of course, there is no simple answer to

this. One needs rounded individuals who can understand and interact with both researchers and

external customers. Where specific skills or understanding are missing or under-developed, one

needs to commit to training and development. That is in the interests of both the individual and

the institution, and both have responsibilities in doing so. Two of the most important

responsibilities of any service director or manager are to obtain and retain the necessary level of

resource to perform the function, and to ensure one’s staff are adequately trained and supported.

Questions to Consider in Relation to Skills

The following are a number of illustrative questions for consideration in this area.

Do you know what skills you need to have in place, and which ones exist (and to what

level) in your staff?

What mechanisms do you have in place to develop the communication and commercial

skills of your researchers and your professional support staff?

What budget do you set aside for training and development and how do you determine

training plans, for individuals and for groups?

How do you define the research requirements of your communications strategy?

What communication mechanisms do you have in place, and how is their use prioritised?

35 http://www.auril.org.uk/CPD/KTCPDFramework/tabid/1160/Default.aspx

36 http://www.vitae.ac.uk/researchers/428241/Researcher-Development-Framework.html

http://www.auril.org.uk/CPD/KTCPDFramework/tabid/1160/Default.aspx

http://www.vitae.ac.uk/researchers/428241/Researcher-Development-Framework.html


21

How do you create and maintain adequate knowledge and records of relevant external

parties with whom to communicate, and on which topics?

How do you prioritise the acquisition and provision of skills, where resources are limited?

Resources and Structures

Things tend only to happen if we have resources available. This includes, crucially, time, along

with specific funds, facilities and the right people. As already noted, the sector has benefitted

from the provision of specific funds for knowledge exchange. These were generally introduced

as an intervention, to ensure that institutions built capacity and were thinking about knowledge

exchange. The success of this intervention is reported through the annual HE-Business and

Community Interaction (HEBCI) Survey37, but the extent to which impact is actually being built

into research activity is not necessarily obvious from the figures.

Activities to support involvement, translation, knowledge exchange, and ultimately impact need to

be built into research projects (noting that this is not a statement of support for a linear model). It

is not sufficient to collect the results of research and then to try to do something with them,

whether through availability, promotion or direct exploitation. One might have a level of success,

but could more be achieved if mechanisms for development were intrinsic to the research itself?

There are, of course, a number of funding mechanisms that support these later stage,

translational activities. However, they often require tests of the potential outcome, thus possibly

removing the opportunity from some. Wouldn’t it be better to build such activities into every

project from the outset, so that all are in a better position to enable impact of whatever sort?

An example of the challenge is for research where the effects come about through engagement,

dialogue and debate with amorphous communities, rather than through a neat set of translational

steps. This is particularly true of arts and humanities research, but also applies to much social

science, and also to areas of the hard sciences. The wide diffusion of new knowledge is

essential, but needs to happen in a supportive, flexible structure. The creation of a web page is

not enough, as its content will be lost; one has to find ways to bring material to others’ attention,

and to enable engagement and evolution.

Time is the most precious commodity. And use of research results takes time and effort to

achieve. We therefore need to allow sufficient time (elapsed as well as resource time) if we want

to achieve impact. Time for preparation, communication, engagement, exchange, and so on has

to be built into our programmes. Successful translation and use of research findings usually

require the involvement of the person or people who generated that knowledge. Research

results, including codified knowledge as expressed in a patent, are typically only valuable when

set in the context of the surrounding know-how.

Commercialisation has on occasions been seen as a means of funding a research institution, and

also as a way of assisting the national industrial base (which two ambitions can be in tension).

However, the economics of research commercialisation are not usually so neat and clean.

Nationally and internationally, a typical ratio of income from intellectual property (IP) to that for

research projects is between 1% and 4%38. Although a significant proportion of IP income is

unencumbered, most universities also share a significant proportion (e.g. a third) with their staff

and student inventors. Few universities will generate enough income to do much more than

37 http://www.hefce.ac.uk/whatwedo/kes/measureke/hebci/

38 The Russell Group average for 10/11 was 1.5%, and that of the 94 Group was 0.9%. Source: HESA data.

http://www.hefce.ac.uk/whatwedo/kes/measureke/hebci/


22

enable an on-going investment in the IP-related processes, and most will not achieve that.

Governmental targets for company creation, for example, are not helpful, as they may create

activity, but they do not necessarily create value.

The benefits from IP are much broader than direct income. IP, in all its forms, can be part of a

relationship between an institution and one or more other organisations. There may be greater

value in the relationship, due to research income, data or facilities, student experiences,

recruitment, donations, than from the IP itself. The wider value therefore needs to be recognised,

but without being blind to the costs involved. The sustainability of these activities rests on an

understanding and valuation of the portfolio, which is not a mature subject with a recognised

approach. Over-emphasis on single transactions, because they have explicit terms, can create a

dangerous narrowing of focus.

Gaining market understanding is important. One can do this through recruitment of individuals

with relevant experience, contracting of consultants, and purchase of market data and reports.

For any broadly-based university (of whatever size), this can be costly, because of the potential

range of topics, sectors, and geographical regions for which information might be required. For

larger institutions, there will be a challenge of volume; for smaller institutions, there will be the

problem of lack of familiarity because of the infrequency of any given topic or situation. One

therefore tends to have to be very selective, and to compromise (in comparison to a business’s

approach, where it is naturally much more focused). Such compromise can include using

specialist partners, and recognising the value that they bring.

The provision of HEIF, KTG and the like has helped to build capacity as well as encourage

institutional cultural change. Direct intervention is changing into core support, funded through the

range of activities embedded in core research and educational provision. Academic units are

now as likely to have targets for knowledge exchange as they are for research, with all aspects

incorporated into their resource models. It is only by doing so that we will maximise impact, and

achieve sustainability for its support.

The same is true of the structures required. These include internal organisational structures, but

they also include operational processes, such as objectives and workload allocation, promotion

and reward, and internal and external reporting.

Dedicated knowledge exchange funds have been important, but by earmarking such funds, they

have effectively required separation of such activities, in particular because of the reporting

requirements. The positive outcome was staff dedicated to enabling knowledge exchange, but a

negative effect was, in some cases, that these staff were disjointed from the core business of the

institution, not being adequately connected to the research process. Indeed, there have been a

number of cases of significant tension within an institution between those supporting research

and those supporting knowledge exchange.

The impact agenda, for all of the negativity that it has generated in some quarters, can help to

draw together the range of expertise, and provide value to academic units and their researchers

of the dedicated staff and specialist support groups. The drivers provided by policy and funding

mechanisms will have an effect.

One area that still needs attention is that of sector reporting. The differentiation between

research and “other services” in institutional accounts (driven by the external reporting

requirements) is not necessarily helpful. It forces a categorisation of an activity, which might also


23

have an effect on the motivations to undertake it. At one point, income relating to the Teaching

Company Scheme was deemed to be a service. But it could be included in the Research

Assessment Exercise as research income. Now income for its successor, Knowledge Transfer

Partnerships, is categorised as research income.

Research projects can include knowledge exchange activities, and all of the funding is classified

as research, whereas if the activity is separately funded, it may have to be classified as a service.

There can sometimes be considerable discussion about how to classify a project, which might be

time and effort better spent in undertaking the activity or in communicating about it. Is it time we

had another look at how we are required to categorise our income for reporting purposes, to

ensure that it is not having adverse motivational effects? Institutions certainly need to be aware

of any differences, subtle or otherwise, that this may be having. For example, is knowledge

exchange income classified as services valued equally alongside research income in reward and

promotion processes?

Knowledge exchange should be fully integrated into the normal academic business. If it is still

being treated as “special” or “different”, then the intervention has not been successful. That

applies to sector level arrangements as well as to institutional structures, policies and processes.

Questions to Consider in Relation to Resources and Structures

The following are a number of illustrative questions for consideration in this area.

Are the relationships between and responsibilities of the different parts of the academic

and professional services units clear?

Have you minimised the potential for single point failure in your processes?

Are all your policies (i.e. research, knowledge exchange, HR, finance, student

regulations) aligned with each other, and with the institution’s strategic goals?

Do institutional resource allocation mechanisms reflect support for knowledge exchange

activities in support of enabling impact from research?

Do institutional reward and promotion criteria and processes adequately value enterprise

and innovation activities?

Do the institutional grading processes adequately recognise specialist expertise, and thus

enable recruitment and retention of experts in professional support roles?

Do the institutional reports support the business need, or are they driven mostly by the

external reporting requirements?

Are research projects constructed so as to consider translation and plural dissemination

wherever possible?

Conclusions

The impact agenda has been with us for some time, but is now more visible and having a greater

effect on institutional strategy and operations, as well as on individual researchers. The

Research Councils’ Pathways to Impact and the Funding Bodies’ REF impact requirements have

generated an industry. Companies and individuals offering services to universities have become

more common (with a reasonable proportion being former employees of those or related

government bodies). Universities have individuals and groups responsible for supporting the

communication and translation of research results. The infrastructure has begun to develop, but

is not yet mature.


24

Core funding of capacity for knowledge exchange has enabled universities to develop their

capabilities, but more needs to be done to fully integrate that capacity. Pathways to Impact have

had some effect, but one might argue that they represent speculation about future possibilities,

and hence the value is in promoting such thoughts (rather than actually building relevant

elements into the projects, where appropriate). Impact in the REF, on the other hand, represents

20% of the resulting profile, and hence drives 20% of the annual funding, or about £360M p.a.

Put in those terms, it’s not surprising that institutions and individuals are starting to take notice

and behaviours are changing and adapting.

We will continue to see evolution. Progress has been made, but there will always be room for

further development. The pace and nature of change will also vary between organisations and

between subject or topic areas. This latter will be a challenge in any university, given the spread

of subject areas, and hence the potential differences between them. Successful institutions will

have flexible, adaptable approaches, which require staff with appropriate knowledge, skills and

behaviours, in a resource environment that provides the ability to respond.


25

Making the Grade: Methodologies for assessing and evidencing research impact 39

Dr Molly Morgan Jones and Dr Jonathan Grant,40 RAND Europe

Introduction

The UK invested £27.4 billion in research in 2011; £7.1 billion from public sources and £17.4

billion from private sources, with the remainder of expenditure coming from abroad41. This money

funds a broad spectrum of ‘basic’ and ‘applied’ research, from improving our fundamental

understanding of the cosmos to testing the effectiveness of new drugs on patient populations.

But assessing the impact of any research remains challenging. Does it improve the wealth and

wellbeing of societies? If so, what is the nature or size of that impact?

This paper explores whether existing methodologies are up to the task of evaluating impact

across different sectors, and different criteria for assessing research impact. We look at recent

methodological developments that attempt to account for the complex interactions occurring

within and between research disciplines and society. We analyse traditional and new

methodologies and offer some thoughts for ways in which current standards of evidencing the

impact of research can be strengthened and expanded.

The paper is structured around four key messages:

The choice of methodological approach must be informed by the objectives of the

research impact assessment.

There is a set of common methodological challenges that vary in importance depending

on the objectives of the research impact assessment.

Each method has its own strengths and weaknesses, and these vary in importance

depending on the objectives of the research impact assessment.

New methods are likely to face the same set of trade-offs.

We conclude by reflecting on challenges for the future of research impact assessment.

Understanding the objectives of research impact assessment

The assessment of the non-academic impact of research is not new42, but there is a growing

interest internationally in methodological approaches to measuring research impacts43. Here, we

39 We would like to thank Professor Joanna Chataway, Sue Smart, and Kate Kirk for comments and inputs to earlier

drafts. We would also like to acknowledge our colleagues at RAND Europe for many formative discussions on

research impact, REF and associated topics – a number of which have been captured in this paper and would not

have evolved without their inputs. Needless to say we are responsible for any errors, misrepresentations or

inaccuracies. 40

Molly Morgan Jones ([email protected]) and Jonathan Grant ([email protected]) are researchers at RAND Europe,

a not for profit public policy research institute, and can be contacted at Westbrook Centre, Milton Road, Cambridge,

CB4 1YG; Tel: + 44 1223 353329. 41

Office for National Statistics. (2013) UK Gross Domestic Expenditure on Research and Development, 2011. ONS:

London. [http://www.ons.gov.uk/ons/rel/rdit1/gross-domestic-expenditure-on-research-and-development/2011/stb-gerd-

2011.html#tab-Key-figures, accessed 22 April 2013] 42

Marjanovic, S., Hanney, S. and Wooding, S. (2009) A Historical Reflection on Research Evaluation Studies, Their

Recurrent Themes and Challenges. Cambridge, UK: RAND Europe. 43

OECD (2010) OECD-Norway Workshop on Performance-based Funding for Public Research in Tertiary Education

Institutions; Academy of Medical Sciences (2006) UK Evaluation Forum Medical Research: Assessing the benefits to

society, London: Academy of Medical Sciences.

mailto:[email protected]



26

take research impacts to be a demonstrable contribution to something outside the academic

system. The increased interest in measuring research impact is due to a number of different, but

not mutually exclusive, drivers that can be organised into the four “As” of advocacy,

accountability, analysis and allocation. Each driver has a slightly different rationale for it, with

corresponding implications for how impact might be evidenced.

Advocacy

Set against a background of fiscal austerity and a drive for efficiencies in many nations, research

funders and providers are having to compete with other public services and advocate the need

for, and continued funding of, research. Leaders within the sector need to have compelling

arguments to ‘make the case’ for research. These arguments are often best made at the

macroeconomic level, supported with compelling narratives or case studies. For example, Salter

and Martin44 reviewed the literature on the economic benefits of publicly funded research (across

all disciplines) and found that benefits from research take a variety of forms, and in particular that

spillover and localisation effects from research can be substantial. Buxton et al45 reviewed the

literature on economic gains from the biomedical and health sciences and found that different

methodologies provide different ways of considering economic benefits. These studies have

since been successfully used in advocacy. The Funding First initiative of the Mary Woodard

Lasker Charitable Trust supported the doubling of the US National Institutes of Health’s budget

between 1998 and 200346, and in the UK the Medical Research: What’s it worth?47 study

estimated that public sector research in the cardiovascular field had a 39% rate of return. This

finding played an important role in the discussions leading up to the 2010 Comprehensive

Spending review48. Organisations like the Union of Concerned Scientists in the US49 regularly

highlight scientific findings to try and advocate for continued funding of science in key policy

areas, and the inclusion of scientific evidence in policy making. In terms of narratives, the UK

Research Councils each publish an annual Impact Report which describes the ways in which

they are maximising the impacts of their investments and includes illustrations of how their

research and training has made a contribution to the economy and society50. The European

Commission recently produced a similar document highlighting the wider impacts of the 6th

Framework Programme51.

Accountability

Related to advocacy is the need for the research community to be accountable to those who fund

its activities, be they tax payers or donors. Good governance dictates that the recipients of public

44 Salter, A.J. and Martin, B.R. (2001) ‘The economic benefits of publicly funded basic research: a critical review’,

Research Policy, 30(3), pp. 509–32. 45

Buxton, M., Hanney, S. and Jones, T. (2004) ‘Estimating the Economic Value to Societies of the Impact of Health

Research: A Critical Review’, Bulletin of the World Health Organization, Vol. 82, pp. 733–739. 46

Office of Legislative Policy and Analysis (OLPA). (2013). Legislative Updates. National Institutes of Health:

Bethesda. [http://olpa.od.nih.gov/legislation/107/pendinglegislation/doubledec.asp, accessed on 22 April 2013] 47

Buxton et al (2008). Medical Research: What's It Worth?: Estimating the Economic Benefits from Medical Research

in the UK. Wellcome Trust/AMS/MRC 48

Brumfiel, G. (2010). ‘UK science saved from deepest cuts.’ Nature, 20 October 2010. doi:10.1038/news.2010.550. 49

See for example, http://www.ucsusa.org/global_warming/ 50

Research Councils UK. (2012) Impact Report 2012. Research Councils UK (RCUK): Swindon, UK.

[http://www.rcuk.ac.uk/Documents/publications/Impactreport2012.pdf, accessed 22 April 2013] 51

European Commission, DG Research. (2011). Secrets in your pocket: EU Funded Innovation – The road to success.

Luxembourg: European Union Publications.


27

funding should be able to provide an account of their decision making. In the context of research

funding, this means that funding decisions must be made in a transparent, merit based way, and

take into consideration the potential for a public benefit or social impact beyond academia. Such

accountability can occur through the collection and reporting of appropriate metrics. All Research

Councils in the UK have a system of collecting and reporting this information in their annual

Impact Reports, but the methods of collecting the information vary. For example, the UK Medical

Research Council uses the E-Val system and Researchfish52, which is based on a tool developed

by RAND Europe53, to highlight the activities, outputs, outcomes and impacts of the research.

Another approach is that adopted by the UK Economic and Social Research Council, which has

a strategic programme of research evaluation and provides tools to help researchers evaluate

their impacts54. The Excellence in Innovation for Australia Trial, a recent impact assessment

exercise, attempted to capture a range of impacts across different research universities and

disciplines as a means of accounting for the different ways that publicly funded research

contributes to society55.

Analysis

A third reason for assessing research impact is to provide a dependent variable in the analysis of

research policy or, to put it another way, to understand what works in research funding. The

‘science of science’, as it is has become known, is predicated on the ability to measure research,

with the aim of improving the effectiveness and value for money of research funding56. John

Marburger, the former Science Advisor to President George W. Bush, in an editorial in Science in

2005 explained that: “A new ‘science of science policy’ is emerging … But [this] demands the

attention of a specialist scholarly community. As more economists and social scientists turn to

these issues, the effectiveness of science policy will grow, and of science advocacy too” 57.

Knowing ‘what works’ and why will inform decisions about which areas of science to invest in,

determining how and who should invest, and identifying the returns58. Various groups have built

case studies and used cross-case analysis to identify the factors that lead to success, including

the Payback Framework59. The Science of Science and Innovation Policy Group at the US

52 Medical Research Council (2010) Outputs, outcomes and impact of MRC Research: Analysis of MRC e-Val Data.

Available at: http://www.mrc.ac.uk/consumption/groups/public/documents/content/mrc008191.pdf [Last Accessed 27th

February 2013]. 53

Wooding S, Nason E, Starkey T, Hanney H, Grant J (2009). Mapping the Impact: Exploring the Payback of Arthritis Research. RAND Europe, Cambridge (MG-862-ARC); Ismail S, Tiessen J and Wooding S (2010). Strengthening Research Portfolio Evaluation at the Medical Research Council. Developing a survey for the collection of information about research outputs. RAND Europe, Cambridge (TR-743-MRC).

54 See, for example, http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-evaluation/evaluation-

strategy.aspx [accessed 22 April 2013] 55

Australian Technology Network of Universities and Group of Eight (2012) Excellence in Innovation Research

Impacting Our Nation’s Future – Assessing the Benefits, Australian Technology Network of Universities. Available at:

http://www.atn.edu.au/eia/Docs/ATN-Go8-Report-web.pdf [last accessed 12 March 2013]. 56

Grant, J. and Wooding, S. (2010). ‘In search of the holy grail: Understanding research success’. Cambridge: RAND

Europe. Document number: RAND_OP295 57

Marburger III, J.H. (2005) ‘Wanted: better benchmarks’, editorial, Science 308(5725), pp. 1087. 58

Grant, J. and Wooding, S. (2010) In Search of the Holy Grail: Understanding Research Success, RAND_OP295,

Cambridge: RAND Europe. 59

See, for example, Wooding, S, Hanney, S, Buxton, MJ and, Grant, J (2005). Payback arising from research funding:

Evaluation of the Arthritis Research Campaign. Rheumatology, 44, 1145-1156.; Wooding S, Nason E, Starkey T,

Hanney H, Grant J (2009). Mapping the Impact: Exploring the Payback of Arthritis Research. RAND Europe,

Cambridge (MG-862-ARC); Canadian Academy of Health Sciences (2009) Making an Impact: A Preferred Framework

and Indicators to Measure Returns on Investment in Health Research. Report of the Panel on the Return on

http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-evaluation/evaluation-strategy.aspx

http://www.esrc.ac.uk/funding-and-guidance/tools-and-resources/impact-evaluation/evaluation-strategy.aspx

http://www.atn.edu.au/eia/Docs/ATN-Go8-Report-web.pdf


28

National Science Foundation funds research that improves, develops and expands techniques

which can be applied to and further inform the scientific decision making process, including the

evaluation of returns from science and furthering the understanding of how structures and

processes can “facilitate the development of usable knowledge” 60. As will be explored later in this

paper, this driver is one that we would argue should be more closely looked at in relation to

taking the impact agenda forward, particularly in the wake of the UK REF.

Allocation

The allocation of research funding based on non-academic impact is relatively new, with the UK

REF being the first example of its application across a research system. The Australians ran a

pilot exercise in 2006 during their development of the national Research Quality Framework

(RQF) which would have introduced impact assessment into their national research assessment

exercise61, but it was dropped when a new Labour government was elected in 2007. Interestingly,

the Australians are again considering adopting an impact assessment exercise, this time based

on the UK REF62,63. In November 2013, UK universities will make their REF submissions. The

REF will assess universities on the basis of the quality of research outputs, the wider impact of

research and the vitality of the research environment. Following a pilot exercise64, the UK funding

councils concluded that the peer review of research impact case studies was a workable

approach. The assessment of impact will be based on the expert review of case studies

submitted by Higher Education Institutes (HEIs) in the UK. The weighting for the impact

assessment part of the REF will be 20% of the total assessment in 2014, and this is likely to rise

to 25% in the future. Even at 20%, this equates to an expected £200m+ per year of allocated

funding, and so constitutes a significant amount of funding for research.

Common methodological challenges in assessing the impact of research

A number of challenges must be acknowledged and, where possible, addressed in measuring

research impact. These challenges are not new65 and differ in importance depending on the

Investments in Health Research. Available at: http://www.cahs-acss.ca/making-an-impact-a-preferred-framework-and-

indicators-to-measure-returns-on-investment-in-health-research-8/ [last accessed 27 February 2013].; and Wooding, S,

Hanney, S, Pollitt, A, Buxton, M, Grant, J, and on behalf of the Retrosight Project Team, (2011). Project Retrosight -

Understanding the returns from cardiovascular and stroke research: The policy report, Cambridge, UK: RAND Europe,

http://www.rand.org/pubs/monographs/MG1079.html 60

For a description of the Science of Science and Innovation Policy programme, see

http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=501084 [accessed 22 April 2013] 61

See for example Roberts, G., E. Arthur, M. Barber, R. Batterham, P. Callaghan, I. Chubb, P. Clark, A. Edwards, P.

Hoj, R. Milbourne, A. Pettigrew, P. Sheehan and I. Smith. 2005. Research Quality Framework: Assessing the quality

and impact of research in Australia, The Preferred Model. Expert Advisory Group for the RQF and Peacock, J., L.

Harvey, M. Barber, P. McFadden, J. Marceau, I. Davey, P. Clark, C. Fell, W. King, T. Enright, D. Schreuder and I.

Smith. 2006b. Research Quality Framework. Assessing the quality and impact of research in Australia: Research

Impact. Development Advisory Group for the RQF. 62

Australian Technology Network of Universities and Group of Eight (2012a) ‘Excellence in Innovation Research

Impacting Our Nation’s Future – Assessing the Benefits’. Australian Technology Network of Universities 63

Morgan Jones et al (2013) Making the grade: An analysis of how the Australian Technology Network of Universities

identified and demonstrated their impacts. RAND Europe: Cambridge. (RR-278-ATN)

64 Technopolis (2010) REF Research Impact Pilot Exercise Lessons-Learned Project: Feedback on Pilot Submissions:

Final report. Technopolis, November 2010. Available at: http://www.hefce.ac.uk/research/ref/pubs/other/re02_10/[Last

accessed 27th February 2013.

65 See for example, Academy of Medical Sciences (2006) UK Evaluation Forum Medical Research: Assessing the

benefits to society, London: Academy of Medical Sciences. Available at:

http://www.cahs-acss.ca/making-an-impact-a-preferred-framework-and-indicators-to-measure-returns-on-investment-in-health-research-8/

http://www.cahs-acss.ca/making-an-impact-a-preferred-framework-and-indicators-to-measure-returns-on-investment-in-health-research-8/

http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=501084


29

primary purpose of the assessment. Below, we identify five key challenges, all of which are

interlinked and interdependent in important ways. The discussion draws and builds upon

previous work conducted by the authors and colleagues, in particular our experience working

with a range of UK universities in preparation for the Research Excellence Framework (REF

2014) in the UK.

Time lags

The time it takes for research to translate from academia into wider societal benefits is largely

unknown but, at least in the biomedical and health sciences, has been estimated by a number of

different studies to be 17 years, on average66. Anecdotally, we know that advances in other

disciplines, like physics, may take 50 years or longer to materialise (e.g. the radio telescope).

This means that any assessment of contemporary impact may have to look at research that

occurred two or more decades ago. This raises a number of issues. First, is it possible to recall

such research with sufficient accuracy to enable a robust understanding of how the research may

have led to the impacts? The further we are from the research, the more difficult attribution and

contribution are to disentangle (see below). Second, is the context within which the research

occurred relevant today? If not, does this impact on our understanding of the wider research

system? There are implications for our ‘4 As’ in that our ability to analyse factors associated with

successful translation of research and, potentially, to allocate research funds based on such

historical performance may be hampered. Finally, given the mobility of university-based

researchers, is it possible to attribute research activity to one institution, given that a researcher

and research team may work in many different universities across their careers? This final

question leads us to the next challenge, that of attribution and contribution.

Attribution and contribution

The linkage between input, activity, output and outcome is crucial to our ability to understand how

research translates into public benefit. However, in research impact assessment this can prove

difficult to determine and, importantly, evidenced. The challenge of any system that evaluates

research impact is to ensure that we have an understanding of the ‘contribution’ and ‘attribution’

relative to the outputs and outcomes that result from the research input and activity. Here, we

take attribution to refer to the fractionated, or proportional, effort made by a research team to the

creation of the outputs, whereas contribution is reflective of the ability to claim that outcomes or

impacts have resulted from the research outputs, regardless of the relative amount of that

contribution. It is an important distinction to make and often the two terms are used

interchangeably. Moreover, the way contribution and attribution are, or should be, highlighted will

vary in importance depending on the purpose of the assessment. If the purpose is for advocacy

or accountability, then judgements about contribution will be more important than attribution67.

However, if the purpose is analysis or allocation, the attribution of the research both need to be

http://www.acmedsci.ac.uk/index.php?pid=99&puid=64 [last accessed 27 February 2013] and European Science

Foundation (2012b) The Challenges of Impact Assessment: Working Group 2: Impact Assessment ESF Member

Organisation Forum on Evaluation of Publicly Funded Research. Available at:

http://www.esf.org/index.php?eID=tx_nawsecuredl&u=0&file=fileadmin/be_user/CEO_Unit/MO_FORA/MOFORUM_Ev

al_PFR__II_/Publications/WG2_new.pdf&t=1362155294&hash=9ca06c34938cdc14b2966c170d3e0583c2c31752 [last

accessed 27 February 2013].

66 Slote Morris, Z., Wooding, S. and Grant, J. (2011) ‘The answer is 17 years, what is the question: understanding

time lags in medical research’, Journal of the Royal Society of Medicine, 104(12), pp. 510–20. 67 Canadian Academy of Health Sciences (2009) Making an Impact: A Preferred Framework and Indicators to Measure

Returns on Investment in Health Research. Report of the Panel on the Return on Investments in Health Research.

http://www.acmedsci.ac.uk/index.php?pid=99&puid=64

http://www.esf.org/index.php?eID=tx_nawsecuredl&u=0&file=fileadmin/be_user/CEO_Unit/MO_FORA/MOFORUM_Eval_PFR__II_/Publications/WG2_new.pdf&t=1362155294&hash=9ca06c34938cdc14b2966c170d3e0583c2c31752



30

taken into account in order to be objective, robust and to account for any double counting. In

practice, however, research is often incremental, collaborative and iterative, so isolating the

contribution and attribution of a particular piece of research to a given set of outputs, outcomes

and impacts is challenging and will inevitably rely on some form of judgement, including how to

make assessments at the margins.

Assessing the marginal differences

A third challenge of impact assessment is that any system must be able to differentiate and scale

different research impacts. Bibliometric analyses and relative citation counts provide an easy but

not uncontroversial way for scaling the impact of different papers68. Though traditionally used to

assess the academic impacts of research, bibliometrics is now being advocated for a broader

role in the assessment of impact. As discussed later in this paper, the techniques it employs in

counting papers, citations, collaborations and so on, could also be used in relation to tracking

movement of researchers within and across fields, countries and institutions, as well as tracing

the diffusion of knowledge not only through academia (through innovative citation analyses), but

also through society, through web crawling and other methods of searching the so-called ‘grey

literature’. One of the reasons that bibliometrics is advocated for use in research impact

assessment is that it provides a common unit of assessment across all disciplines, but this is not

always appropriate, or feasible, across all research disciplines. This has led to the introduction of

other, more qualitative approaches to assessing impact being introduced, such as peer review

and case studies. However, what is gained in nuance could be lost in the introduction of greater

requirements for judgement on the part of the reviewers. All the challenges of expert-based

systems, such as who sits on the panels, what views are represented and whose knowledge

‘counts’, come to the fore.69 Thus, the task becomes increasingly challenging when using these

more judgement-based approaches. Is it possible, for example, for experts to differentiate

between high and low research impact? What is it that marks the boundary between high and

low impact, and can it be defined objectively? There is a precedent for this in the academic and

research system. Academics are well practised at differentiating between Second and First class

honours degrees for undergraduate students, and researchers often sit on review panels that

allocate research grants based on peer review. However, is there a similarly shared

understanding of impact to the extent that definitions and distinctions will be shared across the

academic community, across disciplines, and across research user groups?

Transaction costs of assessing research impact

All forms of research assessment are costly and the relative benefits should be considered

before any impact system is introduced. The Higher Education Funding Council for England

spent £5.6m running the 2001 Research Assessment Exercise in the UK (less than 1% of the

total funding budget) and the total administrative cost of the funds provided by UK Research

Councils to universities was historically estimated to be around 10% of the funds awarded, but

68 Ismail, S., Nason ,E.,Marjanovic, S., Grant, JC.(2009).Biblio- metrics as a tool for supporting prospective R&D decision- making in

the health sciences strengths,weaknessesand options forfuturedevelopment.TR-6852009.Availablefrom:

/http://www.rand.org/content/dam/rand/pubs/technical_re ports/2009/RAND_TR685.pdf.

69 See for example, Jasanoff, S. (1990) The Fifth Branch: Science Advisors as Policy Makers. Cambridge: Harvard

University Press and Pielke, R. (2007). The Honest Broker: Making Sense of Science in Policy and Politics.

Cambridge: Cambridge University Press.


31

which may have declined in recent years70,71. In both cases, assessment systems to date have

focused on research quality and, given their longevity, seem to have been accepted by policy

and academic communities, though not without some dissent72. This suggests that the benefits of

assuring high quality research continue to be rewarded (through funding), and outweigh the costs

of producing that research. However the introduction of impact assessment alongside the

traditional assessment of research quality will inevitably add not only a new dimension to what is

meant by ‘quality’, but also to the transaction costs associated with demonstrating it. The

question for policy makers is whether such additional costs are justifiable in terms of the

anticipated benefits which will result73. This raises fundamental questions about what will result

from the production of approximately 7,000 ‘impact case studies’ in the UK, and what benefits will

be returned to the universities, research users, and the public.

Unit of assessment

A final challenge is determining an appropriate organisational unit of assessment for evaluating

impacts from research. The level of (dis)aggregation for assessment and the starting point for

analysis are the key issues. Does impact get evaluated in relation to its inputs (the research) or in

relation to its outputs (the types of impacts)? The decision made here can have consequences

not only for the outcome of the assessment, but its validity in the eyes of the community being

assessed74. One way to think about this is to ask whether research is more likely to be multi-

disciplinary in nature, or multi-impactful. The figure below represents the ‘extremes’ of the

different ways an impact pathway might unfold. The blue triangle represents the case where

research starts from multiple disciplinary origins but only has a narrow set of eventual impacts.

The red triangle represents research that originates from one type of discipline, but goes on to

have many different kinds of impacts.

Figure 1: The theoretical research-to-impact continuum

70 Sastry, T. and Bekhradnia, B. ‘Using metrics to allocate research funds: A short evaluation of alternatives to the

Research Assessment Exercise’. Higher Education Policy Institute. Available at:

http://www.hepi.ac.uk/files/23RAEandmetricsfullreport.pdf [Last accessed 15th February 2013] 71 RCUK (2006). Report of the Research Councils UK Efficiency and Effectiveness of Peer Review Project. Research

Councils UK, Swindon. [http://www.rcuk.ac.uk/documents/documents/rcukprreport.pdf; accessed 22/02/2013] 72 Roberts, G. (2003). Review of Research Assessment: Report by Sir Gareth Roberts to the UK Funding Councils.

HEFCE: London. [http://www.ra-review.ac.uk/reports/, accessed 22/04/2013] 73 As recommended by a 2003 UK National Audit Office report on using research in policy making. See National Audit

Office (NAO). (2003). Getting the Evidence: Using Research in Policy Making. NAO: London. 74

Saraweticz, D. 2011. ‘The dubious benefits of broader impact’. Nature 475: p. 141.


32

Obviously neither situation represents reality, but different assessment systems use different

approaches, highlighting the need to think carefully about the objectives of any given exercise

and match the methods to it. The Australian EIA analysed impacts by the most relevant sectors

(identified through socio-economic objective codes), whereas the REF will analyse impact arising

from research disciplines. STAR METRICS75, which has been introduced in the US to measure

the effects of research on innovation, competitiveness and science, uses individual researchers

as the unit of assessment, while the Productive Interactions project76 assesses the mechanisms

through which impacts from research occur, and the starting point is at the level of research

groups or institutions, as this is where shared aims and goals are met. Yet another approach is

that of the Research Councils in the UK, which all impact is assessed at a programme or initiative

level77.

We would argue that as a general principle, the organisational unit of analysis should be based

on the unit that is least likely to be multi-faceted. In our experience, it is more likely that research

will have multiple impacts, which supports the argument for assessing research impact from a

research perspective as a starting point, rather than trying to highlight particular areas of impact

and understanding what kinds of research, research groups, or researchers contributed to them.

However, as the different examples above highlight, the argument extends beyond this to

whether assessment is done at the individual, group, institution, or country level78.

The resolution of any of these points is inevitably linked to the purpose and objectives of the

assessment exercise, for instance an exercise done for allocation will need to ensure there is an

equivalent starting point across the system, whereas an assessment for advocacy purposes may

be less concerned with this point.

Different methodological approaches to research impact assessment

In this section we present four commonly used approaches to assessing research impact:

bibliometrics, case studies, peer review and economic analysis. In doing this, we distinguish

between ways of collecting data, such as surveys and site visits, and the approaches which

typically synthesise a number of data sources, such as case studies or economic analysis. For

example, bibliographic and patent information provides the data that is synthesised into

bibliometric analyses; likewise, interviews, surveys and document review can all provide data that

is synthesised into a case study. In

Box 1 we provide a list of the common data collection techniques that underpin these

approaches.

75 Lane, J. I. (2010) ‘Let’s Make Science Metrics More Scientific’, Nature, Vol. 464, pp. 488–489. And Lane, J. I. and Bertuzzi, S.

(2011) ‘Measuring the Results of Science Investments’, Science, Vol. 331, pp. 678–680. 76

Spaapen, J. and Van Drooge, L. (2011) ‘Introducing “Productive Interactions” in Social Impact Assessment’, Research Evaluation, Vol. 20, No. 3, pp. 211–218. 77

See, for example, the recent review of the impact of the Digital Economy programme, http://www.rcuk.ac.uk/research/xrcprogrammes/Digital/review/Pages/default.aspx [Accessed 22 April 2013] 78

Moed, H. F. (2005) ‘Citation Analysis’. Research Evaluation, Dordrecht: Springer.

http://www.rcuk.ac.uk/research/xrcprogrammes/Digital/review/Pages/default.aspx


33

Box 1 – Common data collection techniques for research impact assessment79

Bibliographic databases refers to a collection of references to published literature, including

journal articles, books, book chapters, conference proceedings, reports, patents, etc.

Bibliographic databases usually contain a standardised set of information about the references,

such as author, title, publication type, year of publication, etc, which can be analysed using

bibliometric methodologies.

Document reviews encompass a review of existing documentation and reports on a topic which

gives a broad overview of an issue and identifies ‘what is known’. Should be tightly focused to

avoid time consuming ‘trawling for evidence’ and is often a first step before other tools are used.

Adds to the existing pool of knowledge through a final synthesis process, despite collecting no

new primary data.

Interviews aim to obtain information and to access personal perspectives on a topic, or more

detailed contextual information. They are flexible and can cover a wide range of factors.

However, interviews can lead to bias from social factors, are time consuming, and may not

provide information that can be generalised.

Patent analysis is the systematic analysis of bibliographic patent data in order to identify trends

and other patterns in research outputs. While it provides one measure of assessing whether

research is translated into new ideas and technologies, it has limitations. These include time-

lags, barriers to accessing proprietary information and sectoral differences in tendency to patent,

all of which prevent this from giving a complete picture of research outputs across sectors.

Site visits involve a visit by an evaluating committee to a department and/or institution.

Generally consist of a series of meetings over one or more days with a range of relevant

stakeholders. Give access to a range of people, and opportunity for two-way feedback, as well as

being transparent to participants. Are time consuming, not highly scalable, not suitable for

frequent use, and not transparent to external parties. Often involves data validation rather than

data gathering.

Surveys provide a broad overview of the current status of a particular programme or body of

research and are widely used in research evaluation to provide comparable data across a range

or researchers and/or grants. They tend to provide broad rather than deep information and are

not always adaptable to individual circumstances, though this improves generalizability and

reduces bias. The methodology depends on purpose, but will consist of the following steps:

develop the survey (approach and question set); identify sample for survey; pilot the survey; train

surveyors if necessary; conduct the survey; analyse findings.

Our analysis proceeds in two stages. In Table 1, we briefly explain each assessment approach,

how it should be used, and its respective strengths and weakness; we also relate each approach

to the assessment challenges discussed above. What is evident from this analysis is that

different approaches have different strengths and weaknesses, highlighting that one of the key

challenges for any impact assessment is to select an appropriate approach that fits the objective

of the assessment and is not overly susceptible to the associated challenges. The second step

79 Guthrie, et al. (2012) Measuring research: A guide to research evaluation frameworks and tools. RAND Europe:

Cambridge. (MG-1217-AAMC).


34

in our analysis thus attempts to take this line of thinking one step further by cross tabulating the

assessment approach with the four different types of objectives for research impact assessment:

Advocacy, Accountability, Analysis and Allocation (see Error! Reference source not found.).

This subsequent step reinforces the notion that the approach taken must be informed by the

objectives of the research impact assessment. However, it does not address the question as to

whether existing (or future) approaches are up to the task of evaluating impact across differing

disciplines. This question – which was set as part of the brief to this paper – is informed by the

desire “to develop a single overarching framework for funding and assessment within which a

differentiated approach is possible for groups of disciplines”, as described by HEFCE in setting

out a future framework for research assessment and funding in a letter to all Principals and Vice

Chancellors in June 200780.

The inconvenient truth is that searching for a universal framework is, perhaps, unhelpful. The

reality is that disciplines require different approaches to the assessment of research impact. For

example, in an analysis of the ‘best’ RAE 2001 publications, around 84% of science papers were

indexed on either the Web of Science or Scopus, but only 25% of social sciences and humanities

papers81. Outputs in the social sciences and humanities are necessarily much broader and

diverse than the typical academic papers in the natural sciences, and might include books, book

chapters, artwork, opera scores, plays and grey literature. The challenges of drawing these

together extend beyond the simple diversity of outputs and also include the need to consider

outputs in multiple languages (a much bigger problem in this field than for the natural sciences

and medicine), the highly variable quality of existing bibliographic databases and information, and

the lack of a standardised database structure.

But this should not stand in the way of using this information to assess research quality and

impacts, it just means we have to think differently about how to go about doing it and the tools we

can and should draw upon. Not only are these research outputs more diverse, one can see how

they point to a wide range of potential impacts outside the academic sphere. The boundaries,

then, between research and impact are not as clear, making the use of bibliometric-based

techniques both intriguing and problematic at the same time. A comprehensive database of

research outputs from the social sciences and humanities that can be easily analysed in the

same way as Web of Science or Scopus is challenging, but could potentially be a rich starting

point for an analysis of impacts on society82. Similarly economic analysis is going to be easier

when there is a monetised end point (be that a quality adjusted life year in clinical medicine or a

new product in engineering). At a macro level (and for advocacy and accountability reasons) it is

80 HEFCE (2007) Circular letter 06/2007 from David Eastwood to Heads of HEFCE-funder HEIs and Heads of

Universities in Northern Ireland, 6 March, Higher Education Funding Council for England.

http://www.hefce.ac.uk/pubs/circlets/2007/cl06_07/ [accessed 22 April 2013] 81

HEFCE (2009a) Report on the Pilot Exercise to Develop Bibliometric Indicators for the Research Excellence

Framework, Higher Education Funding Council for England. As of 2 August 2012:

http://www.hefce.ac.uk/pubs/hefce/2009/09_39/. 82

European Science Foundation (2012b) The Challenges of Impact Assessment: Working Group 2: Impact

Assessment ESF Member Organisation Forum on Evaluation of Publicly Funded Research. Available at:

http://www.esf.org/index.php?eID=tx_nawsecuredl&u=0&file=fileadmin/be_user/CEO_Unit/MO_FORA/MOFORUM_Ev

al_PFR__II_/Publications/WG2_new.pdf&t=1362155294&hash=9ca06c34938cdc14b2966c170d3e0583c2c31752 [last

accessed 27 February 2013].

http://www.hefce.ac.uk/pubs/circlets/2007/cl06_07/




35

possible to use contingent valuation techniques to estimate the ‘value’ of different types of

research product and this should be applicable irrespective of discipline83.

83 Miller, F.A., Mentzakis, E., Axler, R., Lehouz, P., French, M., Tarride, J.E., Wodchis, W.P., Wilson, B.J., Longo, C.,

Bytautas, J.P. and Barbara, S. (2012) Adjudicating the returns on investment from basic biomedical research: a choice

experiment of Canadian researchers and lay publics.Academic Medicine, 1-24. (In Press).


36

Table 1. – The basics of each research impact assessment approach84

Tool What is it? How is it used? Strengths Weaknesses

Bib

lio

me

tric

s

Bibliometrics refers to a

number of techniques for

assessing the quantity,

dissemination, and content

of publications and patents.

It relies on bibliographic and

patent data, which are

conditioned and cleaned for

bibliometric purposes. For

example, with publications, it

uses quantitative analysis to

measure patterns of

scientific publication and

citation, typically focusing on

journal papers.

Bibliometrics are used for measuring the

production and dissemination of scientific

knowledge. It can be a useful way to quickly

identify the main research outputs and

associated impacts within a field. Though

traditionally used for assessing academic

impacts, it can also be used to analyse wider,

non-academic impacts. Measures used in

bibliometrics can include:

- Activity and productivity measures, including

volume of outputs and translation of outputs

into patents;

- Knowledge transfer between academics and

those outside academia, using citations as a

proxy to represent communication;

-Links and collaboration activities between

individuals, research fields, and sectors, which

can be mapped using citation analyses; and

- Guideline citation analysis to gauge

translation of research knowledge into policy

and practice

Quantitative – measures volume of

output and citation impact

Useful to see global trends

Repeatable analysis possible

Relatively low transaction costs; e.g. can

reduce cost and burden of assessment

Short time lags between publication and

indexation on bibliographic databases;

time lag to accrue citations typically up

to five years

Relatively straight forward to

differentiate between groups of papers

based on normalised citation indicators.

Can measure and map collaborations

and inter-disciplinarity

Estimates of quality as measured by citations

may not be reliable

Need to normalise citation practices to

compare across disciplines, countries,

institutions etc.

Challenge of attribution exists because the

contribution of different authors to the

research paper is not always clear, nor which

institutions authors are affiliated with

Skewed by biases in the available data

because some disciplines have low coverage,

and journal coverage is not even across

different bibliometric databases

The challenge of time lags can skew the

analysis

Citation behaviour between fields can vary

and cannot be relied upon as a consistent

measure

84 Guthrie, et al. (2012) Measuring research: A guide to research evaluation frameworks and tools. RAND Europe: Cambridge. (MG-1217-AAMC); The Wellcome Trust (2001).

Putting NHS research on the map. An analysis of scientific publications, 1990-97. The Wellcome Trust, London


37


Cas

e s

tud

ies

A case study is a descriptive

and explanatory analysis of a

phenomenon (or set of

phenomena) within its real-

life context. It is useful for

exploring the ‘how’ and ‘why’

of the phenomena and is

empirically based. Multiple

methods can be used in a

case study to explore a given

research question and the

findings arising from such

independent methods can

usefully be triangulated

Can be used in a variety of ways depending

on the aim of the exercise and/or research

project. Often used to provide the contextual

information alongside a complementary

method or approach which provides

generalisable information. Key considerations

in using case studies in the assessment of

non-academic impact are the unit of analysis

of the case study, and the sample selection

for multiple case studies. Generally speaking,

the structure and approach should reflect the

purpose of the exercise and should guide the

decisions made.

Provides in-depth and nuanced

understanding of the impact

Can identify all known and unknown

types of research impact

UK REF pilot and Australian EIA Trial

demonstrate that it is possible to

distinguish between different scales of

impact.

Flexible enough to capture a variety of

impacts, including the unexpected, and

can provide context around a piece of

research, researcher, or impact.

Can be difficult to assert and prove the

contribution a research group has made in an

impact

Transaction costs can be relatively high when

compared to other approaches (but may

decline once approach is embedded)

Single study may not be representative

Time lag between underpinning research and

impact can be long (e.g. 20 years)

Primary limitation is the generalisability of

findings - best used where examples are

needed rather than full coverage.

Ec

on

om

ic a

naly

sis

Comparative analysis of

costs (inputs) and

consequences (outputs).

Purpose is to assess

whether benefits outweigh

opportunity costs and

whether efficiency is

achieved. 3 types: Cost-

benefit analysis (CBA) Cost-

effectiveness analyses

(CEA) and cost-utility

analysis (CUA).

Economic analyses provide a systematic way

of producing comparisons, but the techniques

should not be used if assigning monetary

value is inappropriate and/or impossible. CBA

expresses impacts in monetary units, often

most useful but also most difficult. CEA

compares outcomes between alternatives and

estimates expected costs and outcomes in a

single dimension measure. CUA compares

costs and benefits of alternatives to help

determine their worth relative to an external

standard. Neither CEA nor CUA should be

used if data on alternatives are not

comparable, if one wants to understand value

of one unit of research independent of a

comparator, or if one wants to understand

externalities.

Quantitative method which can estimate

the economic benefits of research

Can explicitly address the

contribution/attribution issue by linking

research to specific economic impacts

CBA provides wider understanding of

outcomes.

Focuses on financial rather than social benefit

Time lags between underpinning research

and economic impact can be long (e.g. 20

years)

Requires many assumptions which may be

controversial and/or unreliable

Most approaches do not look at the marginal

economic benefits i.e. what is the value of this

extra piece of research

Complicated and expensive and so is more

suited to ‘one off’ exercises.


38


Pe

er

rev

iew

Provides a method for review

by peers, typically other

academics in the same or a

similar field, of different

elements of research. It is

traditionally used for

assessing research outputs,

for example for journal

publications, but there is

increasing use of peer

review for the purposes of

assessing non-academic

impact. Just as the rationale

for academic peer review is

that subject experts are

uniquely qualified to assess

the quality of the work of

others, with non-academic

peer review, it is the

research users who are

uniquely qualified to

comment on the nature and

types of impacts which could

and did result from research.

Peer review is used to assess the quality of

research proposed, performed and produced

by others in their field or discipline. Materials

are reviewed by peers individually or as a

group and scored or given a ranking.

Qualitative feedback is also often provided. It

is worth noting that peer review may be less

valid when assessing wider outputs and

impacts of research and this may require

bringing research users into the assessment

process.

Well understood and accepted

Well practised in making judgements at

the margin (as evidenced through peer

review funding panels)

Can take place at various stages and

thus time lags are less of an issue

(subject to the judgement of the panel)

Provides qualitative informed evaluation

Relies on expert judgement to assess

the contribution of the underpinning

research to the impact

Has credibility with the academic

community

Time limited and time consuming for experts

Concerns regarding objectivity and variability

of results

Costs can be relatively high

Relies on expert judgement to assess the

contribution of the underpinning research to

the impact

Criticised for being conservative, slow and

cumbersome, lacking transparency, and

disadvantageous to early career researchers.

New and emergent methods and challenges

In the previous section we presented an analysis of how different research impact

assessment approaches have different strengths and weaknesses, and how they can be

used in different ways and with different emphases depending on the purpose of the impact

assessment exercise. The nature of the challenges and opportunities vary accordingly. No

single approach emerges as favourable across the 4 As, and the five methodological

challenges described earlier – time lags, attribution and contribution, assessing the margin of

difference, transaction costs, and unit of assessment – still hold across many of them.

As impact has risen up national and global agendas and the need to demonstrate it has

intensified, the shortcomings of existing approaches have become increasingly evident,

hence a proliferation of new ideas in recent years have begun to take hold. These

approaches attempt to amalgamate various methodologies and techniques, as well as

introducing new ones. At the risk of over-simplification, two general types of approaches

stand out: data mining which relies on various ‘crawling’ technologies to comb through

existing databases of web-based information and can also be combined with data

visualisation techniques; and interaction based approaches, which focus on researcher and

stakeholder interactions as the means through which impact occurs. We give a brief

overview of each in turn, before concluding the paper with some thoughts for the future of

research impact assessment.

First, data mining approaches allows us to access and understand existing data by using

algorithms to find correlations and patterns within large datasets and present them in a

meaningful format, thereby reducing complexity without losing information. Different kinds of

data mining tools exist, including web crawling. Web crawling uses a distributed architecture

and often builds on previous searches to build a core set of search terms and archetypes. It

allows for the application of filtering algorithms to reduce the large number of websites and

also the large amount of content found on the websites. Generally, data mining approaches

have the potential to reduce the eventual burden of data collection on researchers by making

use of information already being collected. However, this strength is also a weakness in that

there is a reliance on availability and the quality of existing data. It can require a significant

investment to develop effective data mining processes which can be complex and time

consuming. Moreover, as we discuss below, the approach alone cannot provide the

assessments of non-academic impacts. There is still a need to develop more intelligent and

robust ways of making sense of the ‘mined’ data for the purposes of understanding the

nature and extent of the research impact.

While data mining is a catch-all approach, several ‘branded’ types of data mining exist, each

with different aims. STAR METRICS (Science and Technology for America’s Reinvestment:

Measuring the EffecTs of Research on Innovation, Competitiveness and Science) is a

metrics-based approach which aims to make the most use of existing datasets to create a

comparable and reproducible database of performance from individual researchers. STAR

METRICS measures performance according to two levels: 1) economic support through jobs

created and 2) wider impacts such as economic growth, workforce outcomes, scientific

knowledge and social outcomes. In the STAR METRICS approach, analysis is conducted

using individual researchers as the unit of assessment, on the basis that ‘science is done by


40

scientists’85. The exact metrics used to measure impact in these different areas are still

being developed, but it has been indicated that these will include economic, social and

health impacts as well as knowledge creation. This approach has a low participant burden

once set up and since there is no single indicator of comparative performance, it can provide

harmonisation between funders in one country, or internationally. It focuses on people as the

creators of wider impact, and introduces ways of tracing this as they, and their research,

move in and out of academic spheres. However, there is a high initial burden to ensure the

data mining algorithms are correct and the existing data is in place, which also means the

outputs are dependent upon the quality of the inputs. In addition, the fact it is a summative

method means it may not be amenable to formative analysis.

Altmetrics, another data mining approach, refers to the creation and study of new metrics

based on the Social Web for analysing and informing scholarship86. Altmetrics integrates

many different types of applications (apps) to trace research in multiple ways, including, for

example: ImpactStory, a way to track the impact of wide range of research ‘artifacts’;

ReaderMeter, a way to visualise statistics about authors and articles in relation to readership

populations; and Crowdometer, a way to display tweets about a given article. Altmetrics

provides an alternative way of understanding the wider impacts of research outside

traditional academic domains. It picks up and makes use of what are essentially bibliometric

approaches, but extends them in new ways. Its strengths lie in the idea that it can harness

the increasingly ‘e-oriented’ environment for publication, thereby allowing for more effective

tracking of the broader, non-traditional ways that research is having an impact. It provides a

way of keeping current with the vastly expanding set of knowledge and information87.

However, these strengths arguably lead to its weaknesses. Are measures of online activity

outside traditional academic circles going to be accepted as measures of impact, academic

or otherwise? What does it signal about wider impact to have your research ‘tweeted’? Does

‘re-posting’, ‘likes’, or a certain amount of blog comments indicate high impact? If so, what is

the threshold?

Finally, F1000 Prime provides an example of drawing together the wisdom of peer review

with the technologies of data mining and crowd-sourcing88. It provides an in-depth directory

of top articles in biology and medicine which are recommended by a ‘faculty’ of over 5000

expert scientists and clinical researchers, effectively establishing a system of peer review for

top articles. From the numerical ratings given to the articles, a unique system of for

quantifying the importance of articles is created. The advantage of this approach is that it

introduces the subjective element of peer review back into the assessment of quality,

arguably allowing us to overcome some of the challenges of Altmetrics as posed above.

What it doesn’t yet allow for, though, is a means of extending this outside the academic

domain.

Data visualisation is another approach to summarising large amounts of data in a visual

format for human comprehension and interpretation. It has the advantage of allowing data to

be explored intuitively without mathematical processing and can be a highly innovative way

85 Bertuzzi, S. (2010) ‘STAR METRICS’, presentation at First VIVO Annual Conference, 13 August.

86 Altmetrics. (2013) ‘Tools’. http://altmetrics.org/tools/ [accessed on 22 April 2013]

87 Kaur J, Hoang DT, Sun X, Possamai L, JafariAsbagh M, et al. (2012) Scholarometer: A Social Framework for

Analyzing Impact across Disciplines. PLoS ONE 7(9): e43235. doi:10.1371/journal.pone.0043235 88

F1000 Prime (2013). ‘What is F1000 Prime?’. http://f1000.com/prime/about/whatis [Accessed on 22 April 2013]

http://altmetrics.org/tools/

http://f1000.com/prime/about/whatis


41

of presenting non-academic impacts to society, or allowing them to see how academic

impacts can translate into non-academic ones. It is particularly valuable when data is highly

heterogeneous and noisy.

Finally, approaches which start from the basis that it is the interactions between researchers

and the people, places and things their research comes into contact with take a much more

qualitative stance. One such example is the ‘Productive Interactions’ approach, which uses

such interactions as a proxy for research impact. There are three main types of productive

interactions which have been identified in this framework: direct personal contacts via direct

communications, such as conversations or research collaborations; indirect interactions

which occur by sharing a publication or by interacting through a website, prototype or other

design; and financial interactions where there is an economic exchange89. Interactions are

deemed to be ‘productive’ when they lead to efforts by the stakeholders to apply their

research findings in some way, hence changing their behaviour. The methods used to

evaluate the interactions vary, but all research is assessed against the goals of the

institution, not broadly defined national or social goals. It is argued that this approach

eliminates time lags, making it easier to measure and can lead to deeper learning about

what works in producing impact. However, it is not easily comparable as there are not

standard indicators and can be challenging to implement.

Though it is too early to take this analysis to the next step and map each approach against

the 4 As, we would like to highlight a few main points. First, new approaches, particularly

those that rely on data mining such as Altmetrics, Total Impact and F1000 Prime, do seem

comprehensive as they allow one to select from a range of different data sources to compile

an overall set of metrics which work best for the purpose of the assessment. This not only

allows for a tailor-made analysis, but also permits a more customised approach to filtering

and understanding the vast array of quantitative, and qualitative, information that is available

in today’s internet-based era; as the Altmetrics manifesto states, “No one can read

everything”90.

However, what the Altmetrics manifesto goes on to state is indicative of its main shortcoming

for the impact agenda: “Altmetrics expand our view of what impact looks like, but also of

what’s making the impact. This matters because expressions of scholarship are becoming

more diverse.” The whole point of the impact agenda is to move away from scholarship and

into the space where scholarship and society interact. This is where approaches like the

productive interactions method come to the fore, but the problem here is that comparability is

difficult. This will be the case in any largely case study-based approach, and we see a

manifestation of this in the four separate sets of guidance across the UK REF panels. Main

Panels A and B explicitly state in their guidance that quantitative indicators of impact are

encouraged, and a standard list of areas like improved environmental outcomes, improved

health outcomes, economic growth from new products are given. Main Panel C, though,

instead provides a long list of potential examples which might be submitted. This seems to

us illustrative of the fact that just like bibliometric methods run into real problems when

applied to the social sciences and humanities, so too do impact methodologies. Until this

89 Spaapen, J. and Van Drooge, L. (2011) ‘Introducing “Productive Interactions” in Social Impact Assessment’,

Research Evaluation, Vol. 20, No. 3, pp. 211–218. 90

Priem, J, Taraborelli, D, Groth, P, and Neylon, C. (2011) altmetrics – a manifesto. Webpage:

http://altmetrics.org/manifesto/ [Accessed 22 April 2013]

http://altmetrics.org/manifesto/


42

issue is resolved, it seems unlikely the field will be able to move forward in an equal way

across all sectors.

Concluding thoughts

We conclude that the real challenge for assessing and evidencing research impact is in

understanding what kinds of impact categories and indicators will be most appropriate, and

in what contexts. This is both an opportunity and a real danger. Defining impact indicators up

front can unnecessarily ‘close down’ the exercise. Such concerns were raised during the

development of the RQF in Australia91. In the RQF, impact was meant to be concerned with

social, economic, environmental and cultural effects, which would, it was thought, result in a

‘quadruple bottom line’ for the measurement of impact which, according to Donovan, was

unique in the world92. However, the RQF never was able to reconcile the push and pull of

academic interests in wanting to be as inclusive of all disciplines as possible, and those of

government, who aimed to highlight end-user interactions and drive “behaviours that would

make Australia’s science base more efficient”93. This resulted in the exercise being dropped,

but also led to subsequent analyses concluding that the real challenge was in how impact

was captured and quantified94.

Taking this conceptual claim one step further, a recent paper which analysed the submission

to the REF pilot of the University of Oxford’s clinical medicine department seems to confirm

these previous assessments95. Here, they systematically analyse many of the indicators

suggested in the REF pilot and explore their strengths and weaknesses. Measures which

appear to be relatively straightforward ways of measuring wider impact such as patents or

the creation of new businesses are shown to have potentially serious limitations. For

example, simply counting numbers of patents says nothing about their quality or whether

they went on to generate any income. Even looking at returns on investment from intellectual

property can be misleading. Oxford’s innovation spin-out group reported £9.8 million in

returns from 2004/05 to 2008/09, 50% of which could roughly be attributed to clinical

medicine. However, when compared against the inputs to the department of around

£612.9m over the same period, the actual return was only 0.8%. The point is that any single

indicator highlighted in isolation risks distorting the bigger picture, and the examples here

don’t even touch on the issues of feasibility and attribution, which themselves are not

insignificant.

In this sense, there are perhaps parallels here in the way we think about innovation

indicators. NESTA’s reports on innovation suggest that up to 75% of innovation in the UK

91 Donovan, C. (2008) ‘The Australian Research Quality Framework: A live experiment in capturing the social,

economic, environmental, and cultural returns of publicly funded research’. In C.L.S. Coryn and M. Scriven

(Eds.), Reforming the evaluation of research. New Directions for Evaluation, 118, 47-60. 92

Ibid. 93

Ibid, p. 56. 94

Grant J, Brutscher Philipp-Bastian, Kirk Susan Ella, Butler Linda, and Wooding Steven (2010). Capturing

Research Impacts: A review of international practice. A report prepared for the Higher Education Funding Council

for England. RAND Europe. 95

Ovseiko, P, Oancea, A and Buchan, AM. (2012) ‘Assessing research impact in academic clinical medicine: a

study using Research Excellence Framework pilot impact indicators’. Health Services Research 12: 478.


43

comes from ‘hidden innovation’, from non-traditional areas not captured by GDP figures96.

This, and other literature, suggests that measures of GDP are not appropriate for measuring

the innovative capacity of a country’s research base. Equally, a more diverse set of

indicators may be required for understanding the types of impact which different disciplines

will generate, and ways to measure them. In this case, the need to explicitly capture this

becomes an imperative, and we must embrace more nuanced ways of understanding how

different kinds of impacts, and their associated indicators, interact.

If the real agenda in developing new approaches for assessing and evidencing research

impact is not just about the ideas, methods and techniques used, but the way we

conceptualise and develop actual indicators, qualitative and quantitative, then we are at the

beginning of a collective journey exploring the feasibility of developing impact indicators.

This raises as many questions as it answers, not the least of which is whether it is practical,

or even desirable, to go beyond just quantification to monetising impact. This would require a

detailed exploration of the ways value is placed on different kinds of impacts arising from

different disciplines and across society.

In this an important research agenda is set out, and one which links back to the four drivers

of impact assessment mentioned at the beginning. While at present there is an increasing

focus on the use of impact assessment for either allocation or advocacy, as seen in the REF

and EIA respectively, we see a real opportunity in the use of analysis to drive forward the

wider intellectual framework from which a robust evidence base for policy development in

this area can be built. Though REF is ambitious in what it is doing, future iterations, we

would argue, may need to allow for disciplinary differences to emerge. Trying to apply the

same approaches and indicators to each discipline may not work for the basic reason that

the type of information needed for each is likely to be different. Studies such as those

discussed throughout this paper, which have attempted to analyse the relationship between

research and the impacts it has had, can not only help to illuminate ‘what works’ in research

funding, but they can also help to drive forward this new evidence base for impact and

highlight good practice across the sector. Creation of a ‘culture of impact’ is not just

something for UK universities in response to the REF, but something our entire field can

grasp as an opportunity for future methodological and empirical development.

96 NESTA (2009). The Innovation Index: Measuring the UK’s investment in innovation and its effects. NESTA:

London.


44

Assessing Impacts of Higher Education Institutions Kaye Husbands Fealing97, Center for Science, Technology, and Environmental Policy, University of Minnesota

A given metric, once it becomes widely used, changes the behavior of the people and practices it attempts to measure. The worst thing a metric can do is not just to deliver a bad answer, it is to incentivize bad practice. Jevin West and Carl Bergstrom, Nature, June 2010

Introduction

DESCRIBE—Definitions, Evidence and Structures to Capture Research Impact and

Benefits—comprises a series of investigative activities, designed to codify current best

practices in assessing impacts of academic research on socio-economic well-being.

Focusing on evidence-building and actionable recommendations, DESCRIBE promises to

develop frameworks that address perennial questions on the impacts of Higher Education

Institutions (HEIs). Best practice should be informed by the goals of the system analyzed.

Therefore, mission-focused causal inference should be the primary focus on HEI

assessment activities.

The practice of assessment should be anchored in a theoretical framework that formally

represents the system under examination, and that offers clear direction on where the likely

outputs, outcomes and longer term impacts are that result from inputs and activities in the

system. This framework should also include elements from contextual environments that

influence and/or interact with various aspects of the system. However, research activities

within the university are by no means compartmentalized—it is not a closed system.

Interactions with government entities through policies and regulations, relationships with the

private sector (e.g, technology transfer), and even the inflow of foreign students, postdocs

and extramural research collaborations, mean that university administrators manage a multi-

sectoral and multi-cultural enterprise.

While models of the higher education system typically mirror those of private sector firms

(hence the terms inputs, outputs, and so on), it is critically important to note here that there

are stark differences in the mission and operations of HEIs that strain the metaphorical

reference. Although DESCRIBE focuses on research impacts, it is important to keep in mind

that research is co-produced with education (student learning) and community service.

Altbach, Reisberg and Rumbley (2009) indicate that: “Sophisticated, university-based

research is being conducted in an environment where there is pressure and need to

commercialize knowledge, but at the same time opposing pressure exists to treat knowledge

production and dissemination as a public good.” A recent National Research Council

(2012a) report echoes this sentiment, and presses further stating that evaluation of

outcomes is difficult because many university activities are not priced in the marketplace.

Intangible assets are the hallmark of university outputs.

A number of complexities characterize higher education production processes. These reflect the presence of (1) joint production—colleges and universities generate a number of outputs (such as educated and

97 The author thanks Julia Lane and Rosa Fernandez for insightful comments on an earlier version of this paper.


45

credentialed citizens, research findings, athletic events, hospital services), and the labor and other inputs involved cannot always be neatly allocated to them; (2) high variability in the quality and characteristics of inputs, such as teachers and students, and outputs, such as degrees; and (3) outputs (and inputs) of the production process that are nonmarket in nature. As is the case with other sectors of the economy, particularly services, productivity measurement for higher education is very much a work in progress in terms of its capacity to handle these complexities. Because no single metric can incorporate everything that is important, decision makers must appeal to a range of statistics or indicators when assessing policy options—but surely a well-conceived productivity measure is one of these. (NRC 2012a, p. 3)

Social science policy analysis frameworks develop means of understanding the spillovers in

the system, and techniques to obtain substrates of research outputs and impacts that are

generated by specific research funding streams. These methodologies go beyond mere

counts of research outputs. Such counts give the impression that outputs are unrelated

events, which is often an incorrect assumption. While counts of graduates with different

types of degrees are often looked to for assessment of the production of knowledge, it is

also critically important to understand where that knowledge goes and the influence it has in

the new locations. Mobility of students (and researchers) is important to track, for that is one

means of gauging the full impact of funding education and training. Some of the techniques

highlighted later in this paper allow for assessment of these important aspects of knowledge

production and distribution in HEIs. These are indeed the types of outputs and impact that

are not necessarily valued in monetary currency.

One principally important aspect of impact assessment is causal inference. Standard

definitions of inputs, activities, outputs, outcomes, and impacts are used in this paper. The

Government Accountability Office’s report Designing Evaluations (GAO, 2012) defines

impact evaluation as follows:

Impact evaluation is a form of outcome evaluation that assesses the net effect of a program (or its true effectiveness) by comparing the observed outcomes to an estimate of what would have happened in the absence of the program. While outcome measures can be incorporated into ongoing performance monitoring systems, evaluation studies are usually required to assess program net impacts. GAO p.16.

In the process of finding causal inference, “additionality” must be determined. This is best

observed by comparison of identified outcomes to the counterfactual. “...[T]he outcomes

observed typically reflect a combination of influences. To isolate the program’s unique

impacts, or contribution to those outcomes, an impact study must be carefully designed to

rule out plausible alternative explanations for the results.” GAO p. 39. HM Treasury’s report

The Magenta Book—Guidance for Evaluation (HMT 2011) also confirms that best practice in

impact evaluation is estimating the “what would have happened in the absence of the

policy...the counterfactual.” The report goes on to state that: “Establishing the

counterfactual is not easy, since by definition it cannot be observed – it is what would have

happened if the policy had not gone ahead. A strong evaluation is one which is successful in

isolating the effect of the policy from all other potential influences, thereby producing a good

estimate of the counterfactual.” (HMT p. 19)


46

Kelly, McLellan, and McNicoll (2009) define higher education impacts as:

1. The impact of a university or college as a business and the higher education sector as an industry;

2. Higher education increasing the skills base and ‘absorptive capacity’ through its students and graduates;

3. Research and innovation and the transfer of this knowledge to the host economy; and

4. Creation of Externalities: social, cultural and environmental.

While this paper pushes back a bit on the concept of a university as a business (bullet (1)), it is instructive to develop ways of assessing the productivity of multiple factors in the university system, and the spillover effects to local, national and even international economies.

Table 1. Framework for Assessing Impacts of HEI Research Activities

Actors Funders

Governments

Businesses

Private nonprofit organizations

Performers

Education and research institutions, including students, researchers, administrators (e.g., sponsored research officers; technology transfer officers)

Community (e.g, community-based research)

Activities

Research

Invention

Development

Engineering/design

Innovation

Diffusion

Education

Training

Service/outreach

Linkages

Grants

Contracts

Collaboration

Partnerships

Codevelopment

Copublication

Technology transfer

Outputs

Knowledge stocks

Intangibles

Products and services

Impacts

Productivity

Jobs

Job mobility

Other socio-economic impacts, well-being

Source: Adapted from National Research Council (2012b).

Table 1 shows the generally accepted system of scientific research and innovation, with

specific aspects related to HEIs listed in each of the columns—actors, activities, linkages,

outputs, and impacts. Although these factors are listed in sequence from left to right in the

table, that by no means implies linearity within the system. Indeed, this system has

recursive linkages, whereby the process of technology transfer or commercialization of a

given invention can lead to further discoveries that are often referred to as “basic science.”

In other words, the system is complex and dynamic, thereby increasing the importance for

developing a framework for impact assessment that mirrors and adapts to the characteristics

of the system. More will be said about this in the section below on improvements in the

practice of impact assessment.

In addition to guidance from the systems framework of structure and behavior or actors at

HEIs, it is also instructive to understand the demand side of this exercise: what do the


47

stakeholders want to understand about the system. Stakeholders include decision-makers

at HEIs, researchers (including faculty members and students), policy-makers who oversee

research and development and innovation budgets, intellectual property managers of

private-sector firms, and the general public. The ten (10) key questions that they want

answered are given in Table 2 below. These questions span several areas including policy

goals, the processes through which impacts occur, and frameworks, methodologies and

datasets that are needed to do high-quality impact assessments98.

Given this background to the issues contemplated by the DESCRIBE deep-dive, this paper

is a synthesis of methodological approaches that are used by decision-makers in HEIs to

address these questions. The size, portfolio, implementation, impact questions require

proper evaluative tools that oftentimes are overlooked in favor of readily available metrics.

This paper examines several existing frameworks, how they are used and where lessons

can be learned by decision-makers at HEIs. Section 2 presents contemporary tools for

impact assessment, narrowly focusing on elements that are important for research

generated at HEIs. While these are standard practices, Section 3 will illuminate key

limitations of these methods, while introducing recently developed techniques that provide

closer linkage of inputs to impacts. Section 4 gives reflections on this exercise.

98 Question (10) is a necessary element of impact analysis. Without comparison to the status quo alternative,

mere measurements of outputs and outcomes following implementation of an activity will most likely over

emphasize the impact that a policy has had. Therefore, focus on counting published articles and patents

following initiation of a funding stream, for example, ignores that outputs that would have been produced even

without new program funds.


48

Why is there a need for better impact assessment methods?

There are three compelling reasons for this DESCRIBE activity: (1) for the foreseeable

future, an environment of constrained operating budgets; (2) a general public that wants

transparent evidence on social impacts of their tax dollars that support university research

activities; and (3) a technological change, where improvements have been made in impact

assessment tools for tracing inputs to outcomes and impacts.

First, finance strategies at HEIs have changed, whereby universities are increasingly relying

on tuition and fees from students for operating expenses and capital investments. In the

United States, for example, the share of revenue from student tuition (including Federal

financial aid) at public four-year institutions of higher learning has increased from

approximately 20 percent to more than 40 percent between 1985 and 2010. During the

same time period, state and local funding of these institutions declined from 60 percent to

just under 40 percent (see Figure 1 below)99. With increasingly constrained budgets at all

99 U.S. Department of Treasury (2012), “New Report from Treasury, Education Departments; The Economic

Case for Higher Education,” http://www.treasury.gov/press-center/press-

releases/Documents/The%20Economics%20of%20Higher%20Education_REPORT%20CLEAN.pdf [Accessed

April 14, 2013]

Table 2. Key Questions that Research Impact Assessment Should Address

Policy Goals

1. What policies—incentives, institutions and governance—are necessary to bring latent innovations from bench to

market?

2. Where should investment be made for long-term growth: fields; technologies; regions; demographics; intramural

or extramural activities?

Processes through which Impact Occur

3. What are the causal effects of increased funding of academic research on knowledge generation and related

impacts, particularly as they pertain to the mission of the given institution?1

4. Who are the actors, and what are the activities, linkages and relationships (including grants, contracts,

collaboration, partnerships, co-development, and co-publication) that yield the highest positive impacts in higher

education institutions?

5. What are the institutional networks and other linkages that help transform inputs into outcomes (knowledge

transfer and not merely knowledge transmission)?

6. What are the opportunities and vulnerabilities in the system that influence impacts?

Frameworks, Methodologies, and Datasets for Impact Assessments

7. What are the best impact measures for specific scientific domains?

a. What are the key elements that should be measured beyond outputs (quantitatively and/or qualitatively)?

b. What are the best measures of innovation (including qualitative dimensions), productivity, economic

growth, employment, and other outcomes that are valued by society?

8. What methods should be used to estimate the length of time from initial research findings to realized impacts?

9. What method should be used to determine the optimal portfolio for funding research from natural, biological and

social sciences, including a mechanism for continual assessment that informs strategic change?

10. How do we assess additionality—that is, measurable changes attributable to a given policy or activity, beyond

what would have existed under status-quo conditions? What are the relevant counterfactual analyses that identify

specific impacts?

http://www.treasury.gov/press-center/press-releases/Documents/The%20Economics%20of%20Higher%20Education_REPORT%20CLEAN.pdf

http://www.treasury.gov/press-center/press-releases/Documents/The%20Economics%20of%20Higher%20Education_REPORT%20CLEAN.pdf


49

levels of government, there is concern that student tuitions will continue to increase (and

possibly escalate), thereby curtailing enrolments and hence the production of knowledge

assets. Moody’s 2013 outlook for higher education in the United States indicates that

operating revenues from state appropriations are expected to remain stagnant or even

decline in the future, suggesting that funding will be precariously dependent on federal

budget negotiations100. University administrators are, therefore, eager to develop another

source of funding—revenues from the intellectual property generated from research

activities.

Figure 1: Share of Revenue at Public Four-Year Institutions in the United States

Source: U.S. Department of Treasury (2012).

An Association of University Technology Managers (AUTM) survey shows 157 universities in

the United States reporting almost $1.8 billion in earnings from inventions in 2011.

Commercialization of research such as new breeds of wheat, new drug treatments for HIV,

and new products like Gatorade, were responsible for this uptick in revenues compared to

the year before. Respondents to AUTM’s survey also reported 5,398 licenses filed for

12,090 new patents, and 617 start-up companies created101. Northwestern University

headed the list of respondents, with more than $191 million in license income, while the

University of California System had the highest number of licenses and options executed

(292), highest number of startups (58), highest number of U.S. patents issued (343), and the

highest number of new patent applications (962). While these measures show outputs from

university activities, they do not convey how much revenue was generated by a specific

project or by a specific grant. These aggregate numbers do give evidence of activity, but

100 Moody’s Investors Service (2013), “Moody’s: 2013 outlook for entire US Higher Education sector changed to

negative,” 16 January, http://www.moodys.com/research/Moodys-2013-outlook-for-entire-US-Higher-Education-

sector-changed--PR_263866 [Accessed April 14, 2013]; Kevin Kiley (2013), “Moody’s report calls into question all

traditional university revenue sources,” 17 January, http://www.insidehighered.com/news/2013/01/17/moodys-

report-calls-question-all-traditional-university-revenue-sources [Accessed April 14, 2013]

101 Goldie Blumenstyk (2012), “Universities Report $1.8-billion in Earnings on Inventions in 2011,” The Chronicle

of Higher Education, 28 August. http://chronicle.com/article/University-Inventions-Earned/133972/ [Accessed

April 14, 2013]

http://www.moodys.com/research/Moodys-2013-outlook-for-entire-US-Higher-Education-sector-changed--PR_263866

http://www.moodys.com/research/Moodys-2013-outlook-for-entire-US-Higher-Education-sector-changed--PR_263866

http://www.insidehighered.com/news/2013/01/17/moodys-report-calls-question-all-traditional-university-revenue-sources

http://www.insidehighered.com/news/2013/01/17/moodys-report-calls-question-all-traditional-university-revenue-sources

http://chronicle.com/article/University-Inventions-Earned/133972/


50

they do not convey impacts—what would have occurred without a given funding source,

network of scientists, collaboration between university and industry scientists, and so on.

In Germany, “third party funds” are one quarter to a third of the financial base of some

universities. These funds are partly provided by private firms, which have research activities

within the universities or have collaborative relationships with university researchers. Third

party funds are an increasing part of the university’s funding portfolio for research.

Figure 2: Third-party funds: projects and revenue streams

Source: Ingo Rollwagen (2010), p. 9.

Although this third-party or “third stream”102 revenue for research at universities is to some

administrators a fruitful avenue to pursue, some administrators are concerned about making

commercial operations from research activities relevant to core university values—teaching,

discovery research and public service. In University in the Marketplace, Derek Bok

cautioned of the “growing danger of the corporate subsidization of continuing medical

education....teaching hospitals are surrendering their own professional responsibility for

education. In doing so, they risk losing the public’s trust.” The validity of university research

should not be compromised in the public’s view. It is important, therefore, to develop

institutions, incentives and standards that prevent capture of university research by private-

sector interest. Assessment of the HEI impacts should include qualitative information on

these characteristics.

102 Paul Hoskins (2011), “Why universities must optimize third stream revenue opportunities,” The Guardian, 4

October. http://www.guardian.co.uk/higher-education-network/blog/2011/oct/04/universities-optimise-third-

stream-revenues. [Accessed April 14, 2013]

http://www.guardian.co.uk/higher-education-network/blog/2011/oct/04/universities-optimise-third-stream-revenues

http://www.guardian.co.uk/higher-education-network/blog/2011/oct/04/universities-optimise-third-stream-revenues


51

A second area that the DESCRIBE activity addresses, therefore, is the assessments of the

public value of research at HEIs. It is important for university administrators to convey to tax

payers in plain terms what are the social implications of the funded research, and not merely

the pecuniary returns knowledge assets103. Clearly this is typically an exercise where the

incentive is to seek out positive spillovers, but if done correctly, this assessment would yield

a balanced view. Demonstration of the value of commercialized research is important for the

sustainability of its funding. Although also procyclical, it could allow for an endowment factor

that smooths expenditure on research activities. University administrators (especially at

public universities) find it essential to show evidence to policymakers and to the general

public that investments in education and training of students and postdoctoral fellows,

laboratories, and collaborative ventures in the private sector yield positive net benefits. As

indicated above, economic returns, such as financial earnings from patent licenses,

commercialized products, and spinoff companies, are one component of the public value of

research. Papers generated by researchers and number of students graduated are also

often counted as “returns” on financial investments in research activities at the universities—

e.g., yields from federally sponsored research and from private corporations.

Using economic multipliers, the Information Technology and Innovation Foundation (ITIF)

calculates that over the next nine years, the FY 2013 $9.5 billion reduction in federal

research and development funding would reduce gross domestic product by $154 billion.

These budget cuts would be responsible for an estimated job loss (or failure to create new

jobs) of 342,000 by 2016104. Murphy and Topel (1999) estimate that, for the period 1970-

1990, “improvements in life expectancy added approximately $2.8 trillion per year (in

constant 1992 dollars) to national wealth....the flow of uncounted additions to national wealth

due to rising longevity was more than half of measured GDP in a typical year.”105 Cutler,

Rosen and Vijan (2006) calculate the change in life expectancy for newborns that is

attributable to spending on biomedical research from 1960-2000 to have increased by 6.97

years, while “lifetime medical spending adjusted for inflation increased by approximately

$69,000, and the cost per year of life gained was $19,900.”

The use of multipliers conveys the existence of spillover effect from research activities.

However, these methods do not quantify impacts, whereby it is transparent how much of a

change in outcome is directly linked to a change in input or in the organization of activities.

The gross measures ignore the obvious necessary comparison—what is the additional

output from these investments beyond what would have occurred given the status quo.

Furthermore, these measures of outputs from research activities do not go far enough to

measure the social impacts of research. Assessing the public value of science and

technology, therefore, is a critically important activity, for without such assessments the

collective citizenry would not be able to grasp the return on their investments in the scientific

103 In section 2 of this paper, it will also become clear that mere counts of patents and other outputs from

research activities does not give policymakers the evidence that they need to make portfolio and organizational

decisions. 104

Don Troop (2013), “Federal Research Cuts Have a Multiplier Effect on U.S. Economy,” 10 April. [Accessed

April 11, 2013] 105

Interestingly, Murphy and Topel indicate that advances in medicine and in the public’s knowledge of what can

allow them to live longer and higher quality lives, can increase demand for medications and devices, and in turn

increase the prices that they pay. This has to be counted as costs against the benefits for improvements in items

generated by advances in medicine paid for by public investments.


52

enterprise. Such assessments require a multidimensional process of quantitative and

qualitative synergistic elements.

This raises the third reason for DESCRIBE activities—the search for recent advances in

frameworks and tools for assessing impacts of research activities in HEIs. Science of

Science and Innovation Policy (SciSIP) researchers are primarily in economics, sociology,

political science, psychology (social and organizational behavior), engineering, and

information systems fields. Evidence-based policy decisions require information platforms

rich in data, analytical frameworks and networked infrastructure. Resources such as money,

high-quality faculty and high-quality students (Astin and Antonio, 2012) are typically the

objects of analysis. However, if the evaluator just focuses on inputs-to-outputs and ignores

activities and linkages or networks inherent in the system, then the important impacts are

overlooked, especially those valued by major stakeholders.

Improvements in the practice of impact assessment

Standard practice for evaluating programs is summarized in Ruegg and Feller (2003). In the

Handbook on the Theory and Practice of Program Evaluation, Link and Vonortas (2012)

confirm that the list of methods in Table 3 (below), including surveys, case studies,

bilbliometrics, econometrics and statistical analyses, content analysis, and expert judgment,

are actively used in program evaluation. However, they note that these methods have

shortcomings when used to measure impacts. Specifically:

Timing: The effects of research are often manifested long after the research has been completed and the connections obscured.

Attribution: A given innovation may draw upon multiple research projects and a given research project may impact upon multiple innovations. In drawing pathways between them it is also the case that an innovation depends upon many inputs other than research before market and social effects are realized.

Appropriability: The beneficiaries of research may not be the same people or organizations who performed it; it may not be obvious where to look for effects.

Skewness of results: The distribution of impacts in a project portfolio is typically highly skewed. A small number of projects may account for the majority of effects, while a good number often just advance knowledge in a general way. This has implications for sampling strategies.

Committees of Visitors (COV) are panels of experts that can, for example, review portfolios

of research at a given institution and make a determination regarding the contribution of

outcomes to institutional goals. The COV can also compare stated plans with outcomes to

assess progress made. However, these panels are subjective in nature, and the resulting

report is swayed by the materials that the institution puts before the COV.

Bibliometric analysis is another common method of evaluation. The GAO defines it as a

means of “tracking the quantity of publications,” and “analysis of where, how often and by

whom the papers are cited can provide information about the perceived relevance, impact

and quality of the papers and can identify pathways of information flow.” When combined

with visual tools, this analysis can be useful to highlight networks and relationships that were

beforehand unknown or unobserved, particularly in a large institution or set of institutions.

However, there are many drawbacks to such analysis.


53

Lane (2012) describes major pitfalls of using bibliometric analysis. The main issue is

inability to detect causality. How much of an impact did a certain source of funding have on

research outputs? How much did research networks change because of a given project?

Bibliometric techniques can show that the publications/patents exist or that the networks

exist and may have evolved, but they cannot isolate impacts. The perceived relevance of

articles and patents is also often unobserved using this analysis. Basic bibliometric

measures lack hedonic sophistication that is necessary to compare research outputs. Also,

it suffers from the typical problem of any “nose-prints” analysis—it can be gamed. For

example, if someone knows what is being observed to be highly productive (and, in this case

an indicator of positive activity), then the incentive is to produce proxies that mimic those

outcomes without necessarily producing valuable outcomes.

Development of measures of research outputs that go beyond counts to a quality-adjusted

measure of output is still evolving. Bornmann and Marx (2013) recognize common practice

before giving strong cautions. They state that various measures based on publications and

citations are used by decision-makers at HEIs to determine promotions, and by policymakers

to make funding decisions and at times science policy. “Impact factors” (IF) are often used

as a proxy for the impact of a single publication, while the h-index (Hirsch index) allows for

differentiation among output for a given researcher based on number of times a given paper

is cited. However, Bornmann and Marx indicate that “expert” bibliometricians seek to

develop new measures that avoid the common pitfall of IF and h-index measures. They

seek measures that normalize citations against a standard or reference set. The authors

state that “...mere citation figures have little meaning without normalization for subject

category and publication year.”106 And they go on to stress that: “We need new citation

impact indicators that normalize for any factors other than quality that influence citation rates

and that take into account the skewed distributions of citations across papers.” Bornmann

and Marx describe recent techniques using percentiles as normalized indicators as an

advancement on the IF and h-index commonly used by bibliometricians. That said, it is

important to note here that even these normalized indicators, while able to establish a better

representation of output levels, will not be powerful enough to convey impacts—direct causal

outcomes.

106 Bornmann and Marx, p. 3.


54

Table 3. Overview of Evaluation Methods

Methods Brief description Example of use

Analytical/conceptual modeling of underlying theory

Investigating underlying concepts and developing models to advance understanding of some aspect of a program, project, or phenomenon.

To describe conceptually the paths through which spillover effects may occur.

Survey Asking multiple parties a uniform set of questions about activities, plans, relationships, accomplishments, value, or other topics, which can be statistically analyzed

To find out how many companies have licensed their newly developed technology to others.

Case study-descriptive Investigating in-depth a program or project, a technology, or a facility, describing and explaining how and why developments of interest have occurred

To recount how a particular joint venture was formed, how its participants shared research tasks, and why the collaboration was successful or unsuccessful.

Case study-economic estimation

Adding to descriptive case study quantification of economic effects, such as through benefit-cost analysis.

To estimate whether, and by how much, benefits of a project exceed its costs.

Econometric and statistical analysis

Using tools of statistics, mathematical economics, and econometrics to analyze functional relationships between economic and social phenomena and to forecast economic effects

To determine how public funding affects private funding of research.

Sociometric and social network analysis

Identifying and studying the structure of relationships by direct observation, survey, and statistical analysis of secondary databases to increase understanding of social organizational behavior and related economic outcomes.

To learn how projects can be structured to increase the diffusion of resulting knowledge.

Bibliometrics-Count Tracking the quantity of research outputs.

To find how many publications per research dollar a program generated.

Bibliometrics-Citations Assessing the frequency with which others cite publications or patents and noting who is doing the citing.

To learn the extent and pattern of dissemination of a project's publications and patents.

Bibliometrics-Content analysis

Extracting content information from text using techniques such as co-word analysis, database tomography, and textual data mining, supplemented by visualization techniques

To identify a project's contribution, and the timing of that contribution, to the evolution of a technology

Historical tracing Tracing forward from research to a future outcome or backward from an outcome to precursor contributing developments

To identify apparent linkages between a public research project and something of significance that happens later.

Expert judgment Using informed judgments to make assessments

To hypothesize the most likely first use of a new technology.

Source: Ruegg and Feller, 2003.


55

Historical tracing is another method to be considered. Horizons for measuring impacts are

critically important, especially for exploratory scientific research. Historical tracing relies on

context-specific analysis, expert evaluation and it is time intensive. This type of assessment

highlights the spillovers between projects, programs, labs, institutions, and so on. Take, for

example, the “Tire Tracks Model” published by the National Research Council in 2012.

Figure 3 (below) shows the impacts of several areas of basic research, some of which

originated at universities. Over decades, with inputs and collaborations with private industry,

and cross fertilization from multiple areas of research did some of the products that have

high commercial value come about. It is important to determine whether sufficient time has

passed to actually observe outcomes. If analysis is premature, a program might be deemed

ineffective, even though it is likely to yield significant results in the future, and it might be

critical for another program’s ongoing success. These types of externalities are extremely

difficult to measure, and oftentimes require detailed case studies to observe and appreciate

the potentially transformational yet veiled impacts.

Figure 3: Examples of the contributions of federally supported fundamental research

to the creation of IT sectors, firms, and products with large economic impacts.

Source: NRC (2012).


56

Practical Frameworks and Tools for Evaluating Impacts

The SciSIP literature is a rich resource for frameworks, tools and curated data collections

that provide an evidentiary platform for impact analysis. In this section, five examples show

the diversity of this research that informs measurement and policy.

Randomized experiments would (theoretically) yield superior results to the methods listed

in Table 3107. Azoulay (2012) suggests that randomized controlled experiments would, in

some cases, yield instructive results for policymakers at granting institutions, but

acknowledges that critics worry that this method could lead to grants being withheld from

worthy applicants. Interestingly, Azoulay has a powerful argument in his favor. He and his

colleagues at the Massachusetts Institute of Technology used randomized, controlled trials

to determine which method would be more effective at distributing mosquito nets in Sub-

Saharan Africa to prevent the spread of malaria. The project was successful and the

anticipated distortions did not occur. The use of carefully constructed comparison groups is

key to successful assessments using this method.

Exploratory case studies are recommended by Ruegg and Feller for assessments of

programs or reforms that are highly dependent on context, and when there is no clear

comparison case on which to base the analysis. In-depth interviews and observations over

a period of time add dimensionality of relationships and interactions to the quantitative

outcome measures. Eckel, Murdoch and Leonard (2012) used this method in conjunction

with an experimental design and survey to determine program impacts given different time

and risk preferences of the subjects in the study. While the process enabled the researchers

to target the impacts they wanted to observe, this type of study is time and human capital

intensive, and context specific. Eckel et al were able to determine which demographic

characteristics and risk profiles led to greater utilization of a government program. This

research shows a finer grain of analysis that is possible with careful experimental

procedures that were not intrusive nor did they breach unsavory ethical boundaries.

STAR METRICS108 is a data platform that allows the trace from knowledge generation

through impacts, with the use of additional data sources. The focus is on people who create

knowledge. Therefore, STAR METRICS allows assessment research impacts at HEIs by

following funding and knowledge flows through administrative payroll and grants records of

individual researchers, while maintaining confidentiality and privacy of information. The data

platform enables practitioners to do near-real time program and funding evaluations using

administrative records rather than surveys or questionnaires109. It also enables the

construction of comparison groups for counterfactual analysis. One important input and

output at an HEI are students: they are inputs in the production system (spillovers between

students are inputs in the production system at universities, and students are often teaching

assistants and research assistants); and they are outputs of the education system as well

107 HMT (p. 27) indicates that “The method offering the strongest measure of policy impact is randomisation,

often in a form known as a randomised controlled trial (RCT).”

108 STAR METRICS is the acronym for: Science and Technology for America's Reinvestment: Measuring the

Effect of Research on Innovation, Competitiveness and Science. See Largent and Lane (2012).

109 HMT (p. 69) cites the use of administrative records that were not gathered specifically for the purpose of

process or impact evaluations as valuable sources of information for assessing impacts.


57

(see Figure 4 below). A key part of technology transfer is the students who are trained

through research grants110.

Figure 4: STAR METRICS—A Data Platform that Automates Capture of Important Relationships in the Research System

Source: James Evans (2013).

The STAR METRICS data have been used to generate new data on who those students are;

details on the specific approach are provided in Lane and Schwarz (2012). The workforce

information includes data about the occupational distribution of the workforce, the number of

individuals supported and the scientific research areas in which they worked. It also includes

estimates of the institutional jobs generated at the research institutions, including financial,

information technology, and janitorial services. This is useful information because these

scholars represent the next generation of scientists and engineers. Undergraduate support

is especially important because there is evidence suggesting that early research

experiences contribute greatly to the propensity of students to enter scientific and technical

fields. The STAR METRICS system also permits the capture of much more detailed

information on time allocation than is possible from Federal agencies’ award data. Research

fields can be described by using topic modelling of the text of proposals to not only

understand who was being supported to do scientific research, but in what areas of

research. This helps identify the specific sectors in which the most research funding has

been provided, and which sectors are growing fastest. Finally, the STAR METRICS data

can be matched to patent data, for example, to identify the firms and collaborators,

particularly non-profit R&D firms, that utilize research produced at the university. Using the

STAR METRICS data along with other datasets, such as workforce or publications data, can

be a powerful tool to assess societal impacts of research at HEIs.

110 “The inputs to education are substantially similar to those of other productive sectors: labor, capital, and

purchased inputs. Higher education is distinct, however, in the nature of its outputs and their prices. The student

arrives at a university with some knowledge and capacities that are enhanced on the way to graduation. In this

instance, the consumer collaborates in producing the product.” NRC 2012a, p. 22.


58

In the recent Medical Research Council (2012) report on its consultation and workshop

regarding measurement of the economic impacts of research, Luke Georgehiou (Manchester

Business School, U.K.) was asked: “[D]oes the UK need a STAR METRICS programme or

other metrics on the scientific workforce?” Georgehiou responded in the affirmative, saying:

The matching of university administrative records with government databases to produce standardised reports on jobs, economic, scientific and social outcomes would be marked improvement over the disparate domestic reporting methods (eg HESA, RAE/REF), and undue resources deployed to inform allocation of quality-related funding and targeting of research investment. However, any retrospective methodology awarding funds on track record may not optimize research investment towards prospective development priorities. Tagging of data for sector and within sector priorities to derive institutional research intensity, development pathways etc, alongside BERD would provide data to analyse and project towards health and GDP benefits. It would also provide data to stimulate networking between academics and industry. Further investment or sunsetting of priorities could then be more timely, and improve estimates of contribution to GDP and employment. Such numbers should only be used in high aggregation - they are much less reliable at project level. (MRC 2011, p. 18)

Network analysis is a computational tool that organizes social and information networks

using logical algorithms. Jason Owen-Smith’s research uses this technique (using STAR

METRICS data), in combination with Census data to assess the economic value of research

and development. Specifically, Owen-Smith is in the process of trace inputs to impacts,

whereby he is attempting to answer the following questions: How much federal R&D

spending to universities “spills out” into the economy? Which industries and organizations

receive those payments? Where are the organizations located? What effects do the

payments have on economic health, job creation and retention, and profitability? How does

federal research performed at universities directly benefit sub-national and national areas?

Owen-Smith’s process goes beyond the basic bibliometrics algorithm, tracing outputs to

impacts. Indeed he is using data on individuals to understand results of specific activities—

e.g., positioning of researchers spatially in a building and the resulting collaborative

activities, training of graduate students, and research outcomes. Depending on the type of

organization examined, this analysis can show what the impacts are to various aspects of

social well-being and what determines the level of impact. This process is particularly

helpful for policymakers and administrators, who endeavor to make decisions that improve

the value of investments in research to the university and to the broader society.

Econometric analysis is also used to measure impacts, given appropriate discontinuity

designs. The list of illustrative studies is quite long in this arena. A recent study Stuen,

Mobarak and Maskus (2012) shows the impact of students who immigrate to the U.S. on

innovation. Teasing out these impacts is skilfully done using instrumental variables

procedure. The findings are compelling:

Our analysis demonstrates that both domestic and foreign students significantly increase the scientific productivity of US S&E departments, and their marginal effects are statistically comparable, which is consistent with the behaviour of an optimizing department. Further, the positive contribution of foreign students is muted when the students arrive as a result of macroeconomic shocks that differentially increase the proportion of paying


59

(non-scholarship) students. High-quality scholarship students are particularly valuable from the perspective of US innovation policy.

Arora and Gambardella (2005) use econometric analysis to assess the impact of National

Science Foundation grants on research output of individual researchers primarily at

universities. They use data on (quality-adjusted) publications for principal investigators five

years prior to getting an NSF grant and five years after the PIs received the NSF grant. The

unit of analysis is the grant, allowing the authors to determine directly the impact that getting

a grant has on their quality-adjusted measure of research output. Interestingly, Arora and

Gambardella find that “the majority of economists (junior PIs excepted) appear to derive little

productivity gains from funding so that research support is pure rent: The research being

funded would have been undertaken regardless of the outcome [of the grant decision].” The

junior PIs, however, did receive a productivity boost from NSF’s positive funding decisions.

Does this have implications for measuring impacts at HEIs? Certainly, in that the regression

discontinuity design used by Arora and Gambardella—along with the focus of analysis on

the grant or the PI—allows for the counterfactual question to be directly addressed.

Murray, Aghion, Dewartripont, Kolev, and Stern (2009) also use a before-and-after type of

econometric analysis to determine whether a change in policy by the National Institutes of

Health to allow greater sharing of intellectual property between researchers affected the

amount of experimentation done on basic science and applications leading to

commercialization of products and processes. They find that easing limitations on

intellectual property rights allowed for greater diversity in experimentation. Furman and

Stern (2006) observe the “special collections” of biological materials at biological resource

centers, to determine whether those institutions played a role in enhancing knowledge

creation. Again, there is a clear period in time when a change was made and the impact of

the change was observed at the micro level. The implication for HEIs here is that tracing

specific research activities through to outputs and observing natural experiments is one

means of isolating impacts and understanding causal effects. These two cases also show

that HEIs can develop policies that enhance knowledge generation, knowledge flows and

knowledge transfer outside of the institution, and that those policies can have real,

measurable impacts on socio-economic outcomes in addition to the generation of new

knowledge.

Summary remarks

Daniel Kahneman once said: “The first big breakthrough in our understanding of the

mechanism of association was an improvement in a method of measurement.” Impact

assessment requires measures—both quantitative and qualitative—that allow the

counterfactual to be taken away from the observed outcome. Policymakers and university

administrators have important questions that require this type of analysis. Otherwise,

projects are not appropriately funded, the public is uncertain about the benefits (and costs)

of their investments in research at HEIs, and there is reasonable doubt as to the efficacy of

policies, which could lead to misappropriation of scarce resources.

Impact assessment is undergoing a transformation in frameworks, tools and data platforms

used for assessment practices. There is a need for a deepening of research to improve

elements. In addition, it is critically important in impact analysis to work with data at the

project or researcher level. Aggregate proximate measures obfuscate causal effects.


60

References

Altbach, P, L. Reisberg and L. Rumbley (2009), Trends in Global Higher Education:

Tracking an Academic Revolution, A Report Prepared for the UNESCO 2009 World

Conference on Higher Education.

Arora, A. and Alfonso Gambardella (2005), “The Impact of NSF Support for Basic research

in Economics,” Annals of Economics and Statistics, No. 79/80, pp. 91-117.

Astin A. and A. Antonio (2012), Assessment for Excellence: The Philosophy and Practice of

Assessment and Evaluation in Higher Education, 2nd ed. preface.

Azlouley, P. (2012), “Turn the scientific method on ourselves,” Nature, Vol. 484, 5 April, p.

31.

Bornmann, L. and W. Marx (2013), “How Good is Research Really?” European Molecular

Biology Organization (EMBO) Reports, Vol. 14, pp. 226 – 230.

Blumenstyk, G. (2012), “Universities Report $1.8-billion in Earnings on Inventions in 2011,”

The Chronicle of Higher Education, 28 August. http://chronicle.com/article/University-

Inventions-Earned/133972/ [Accessed April 14, 2013]

Eckel, C., J. Murdoch, T. Leonard (2012), “Energy Policy for the Poor: An Assessment of

Subsidized Weatherization Programs to Reduce Residential Energy Usage,” presentation at

the Science of Science and Innovation Policy Principal Investigators Conference, National

Academy of Sciences, September.

Evans, J. (2013), “Scientific Text as Data,” presentation at the CIC-STAR METRICS

workshops, Chicago, IL, March 29.

Furman, J. and S. Stern (2006), “Climbing Atop the Shoulders of Giants: The Impact of

Institutions on Cumulative Research,” National Bureau of Economic Research, working

paper no. 12523.

Government Accountability Office (2012), Designing Evaluations, 2012 Revision.

Washington, D.C.

HM Treasury (2011), The Magenta Book: Guidance for Evaluation, The National Archives,

Kew, London.

Hoskins, P. (2011), “Why universities must optimize third stream revenue opportunities,” The

Guardian, 4 October. http://www.guardian.co.uk/higher-education-

network/blog/2011/oct/04/universities-optimise-third-stream-revenues. [Accessed April 14,

2013]

Kahneman, D. (2011), Thinking Fast and Slow, Farrar, Straus and Giroux, New York, p. 52.

Kelly, U. D. McLellan, and I. McNicoll (2009), Kelly, U., McLellan, D., & McNicoll I., The

impact of universities on the UK economy, Universities UK (2009)


61

Kiley, K. (2013), “Moody’s report calls into question all traditional university revenue

sources,” 17 January, http://www.insidehighered.com/news/2013/01/17/moodys-report-calls-

question-all-traditional-university-revenue-sources [Accessed April 14, 2013]

Lane, J. (2012), “The Uses, Benefits and Pitfalls of Bibliometrics,” presentation.

Largent, M.A. and J. Lane (2012), “STAR METRICS and the Science of Science Policy,”

Review of Policy Research, Vol. 29, No. 3, pp. 431-437.

Moody’s Investors Service (2013), “Moody’s: 2013 outlook for entire US Higher Education

sector changed to negative,” 16 January, http://www.moodys.com/research/Moodys-2013-

outlook-for-entire-US-Higher-Education-sector-changed--PR_263866 [Accessed April 14,

2013]

Medical Research council (2011), “Measuring the Link between Research and Economic

Impact,” Report of an MRC consultation and workshop.

Murray, F., P. Aghion, M. Dewartripont, J. Kolev, and S. Stern (2009), “Of Mice and

Academics: Examining the Effect of Openess on Innovation,” National Bureau of Economic

Research working paper no. 14819.

Murphy, K. and R. Topel (1999), “The Economic Value of Medical Research,” working paper.

http://www.laskerfoundation.org/media/pdf/economicvalue.pdf

National Research Council (2012a). Improving Measurement of Productivity in Higher

Education. Washington, DC: The National Academies Press.

National Research Council (2012b), Improving Measures of Science, Technology, and

Innovation: Interim Report, Panel on Developing Science, Technology, and Innovation

Indicators for the Future, Robert E. Litan, Andrew W. Wyckoff, and Kaye Husbands Fealing,

Editors, Washington, D.C.

Rollwagen, I. (2010), “Project economy approaches for higher education: diversifying the

revenue base of German universities”, Higher Education Management and Policy, Vol. 22/3.

http://dx.doi.org/10.1787/hemp-22-5km32wkjpjs4 [Accesses April 14, 2013]

Stuen, E., A. Mobarak, and K. Maskus (2012), “Skilled Immigration and Innovation:

Evidence from Enrollment Fluctuations in US Doctoral Programs,” The Economic Journal,

Vol. 122, Issue 565, pp. 1143-1176.

Troop, D. (2013), “Federal Research Cuts Have a Multiplier Effect on U.S. Economy,” 10

April. [Accessed April 11, 2013]

U.S. Department of Treasury (2012), “New Report from Treasury, Education Departments;

The Economic Case for Higher Education,” http://www.treasury.gov/press-center/press-

releases/Documents/The%20Economics%20of%20Higher%20Education_REPORT%20CLE

AN.pdf [Accessed April 14, 2013]

Vonortas, N. and A. Link, Handbook on the Theory and Practice of Program Evaluation,

Edward Elgar Publishing Limited.

http://www.laskerfoundation.org/media/pdf/economicvalue.pdf

http://dx.doi.org/10.1787/hemp-22-5km32wkjpjs4


62

Impact as a Journey - with Audience

Dr Averil Horton, Visiting Fellow, Brunel University Business School

Summary

Impact Journey: Understanding Impact as a Journey determines what can be reported,

irrespective of discipline or research type.

Audience: Understanding the Audience for impact determines how impact can be reported

for decision makers.

Combining the Impact Journey and Audience approaches mitigates most of the issues

associated with research impact and provides a universal and discipline-agnostic method for

exploring and reporting impact. The Translation stage of the impact journey could be a focus

valid for earlier impact reporting, in some disciplines and application environments, as long

as certain conditions are met.

Introduction: Impact Issues

Why has 'Impact' been such a highly charged political issue and why have many highly

regarded academics made clear their strong views on its value (or otherwise)? There are

several good reasons.

Firstly, impact is not a single thing, is not merely an endpoint, is not a single 'happening', and

does not occur at a single time; some genuine and valid issues arise as a result. The nature

of impact changes over time - thalidomide has gone from being hailed to despised and back

again. Impact can take a very long time to show - it took 80 years for Einstein's General

Relativity to make an impact in a Satnav. Over the course of time, the results of that

research become more diffuse, become part of other projects and even disciplines - just

keeping tabs on what happens becomes very difficult. And because impact is a process, a

journey, it depends on many other activities too, often not derived from, or even related to

the research - research alone is simply never sufficient.

Secondly, impact is not stand alone; research and impact are part of a whole system, from

which it can be difficult, if not impossible, to isolate the proximate benefits of a single

research project, as opposed to a research teams' work over several years, or even in the

expertise of a single researcher. Collaborative research makes this problem even more

difficult. Even trying to separate out what specifically arose from a particular piece of

research and rather from the previous expertise of those carrying out the research, is almost

impossible and probably undesirable. As a result it becomes very uncertain exactly what

research the impact should have a clear link to, or how specific this link needs to be.

Similarly, the research and impact system is part of a bigger system too - whether the

university, the funder, the country, or indeed the whole world. It is not at all uncommon for

outstanding research to make no impact at all, for exogenous reasons, such as changes in

government policy, merging institutions, key people moving on, or company restructuring. In

addition, the 'control experiment' is not possible - it can never be known what would have

happened if the research had not been carried out.

Thirdly, impact can, and should, mean different things to different people and organisations;

it has many different audiences, all of whom have different perspectives, driving forces,


63

values, and timescales. In particular, different audiences value different types of impact

differently, so that a funder may value being able to demonstrate the economic value of a

research call, a participating company may value the development of a new innovation that

ensured its survival, whereas for the employees of the company, the value of the impact may

be that they still have a job. Decisions not to follow a course of action, the absence of the

consequences of ‘the road not taken’, can also be valued. As a direct result, each party may

also be willing to accept a (different) cost of identifying the impact. Impacts can also be non-

positive for some audiences - a good, if controversial example, is the MMR vaccines and

autism research; the fact that the research was later discredited makes no difference. And

there are always the unintended consequences - social science research that benefits a

particular social group could may have an unwanted effect on the career of the researcher if

the findings contradict say, a key government policy; impact can be risky.

In addition to these issues there are also ideological issues, in particular that a developing

focus on impact is seen by some as a political intervention towards ‘marketising’ higher

education and distorting its freedom to pursue truth independent of political interference.

This is set too in the more general uncertainty over the role of academics and research and

potential lack of confidence and clarity on the objectives of research.

There are at least a dozen good technical reasons why 'impact' has been somewhat

controversial and distinctly uncomfortable for many in academia which can be summarised

as:

1. Change Over Time The nature of impact is not fixed, but changes over time

2. Time Lags Impact can take a long to show

3. Diffusion Research impact becomes more diffuse over time

4. Dependencies Other activities, outside of the research, are required to make an

impact

5. Attribution Connecting specific research to specific impacts is not easy

6. Additionality What might have happened anyway if the research did not happen

7. Disentanglement The impact of specific research as opposed to existing

expertise

8. Exogenous Factors Even the best research may make no impact for external

reasons.

9. Value Different impact types are valued differently by different groups

10. Opportunity Cost How can the cost of considering impact justify the benefit

11. Non-positive impacts Unwanted impacts can occur but are rarely

acknowledged

12. Unintended Consequences Impacts may not turn out as planned and can even

present risks

This paper explores the use of two key approaches, Impact as a Journey and Audience to

enable valid impact identification and reporting by mitigating these issues. It has emerged

from the need to enable Brunel University academics to understand, identify, report and

capture impact, for the REF, for Pathways to Impact statements and for their own career

development, all of which led the author to develop Brunel’s Impact Took-Kit, something

which the academics have found valuable. Together with other work by the author on

Pragmatic Impact Metrics, these two key approaches currently form the basis for Brunel’s

internal Impact Academy initiative.


64

The paper is a thought piece and deliberately takes a highly practical rather than academic

stance. It takes the author’s earlier work on applying foresight111as a starting point,

acknowledges the large and broad existing literature on impact112 (CIHE Review and this

project’s literature review), draws on some aspects of information theory113 (Shannon &

Weaver), communication theory114 (Richards & Ogden) and on the real basics of marketing

(communicating the value of a product or service to customers) but critically, moves on from

theory, through the author’s own on-the-ground experience, to develop a practical and

pragmatic view; the objective is to develop a better understanding of impact in a research

context and to provide genuine practical advice and recommendations.

The problems with impact will never disappear but this practical approach enables the

mitigation of many of the (valid) concerns with impact and provides a credible and plausible

way of navigating the issues.

Impact as a Journey

Central to this thought piece is the idea of an Impact Journey, of travelling from a research

idea to final benefits. This idea is certainly not new; it is present in various forms of

evaluation (although rarely described as such), in many systems or ideas relating to change

implementation, and in a much more sophisticated form, in the Brunel HERG health115

payback model. It is also similar to some ideas in project management.

The essential idea, as shown in the table below, is a progression from research idea, Input,

via the research, Activity, and dissemination, Output, and onto Translation, Usage and

eventual Impact. It is worth noting that the Impact Journey framework is not a statement of

how research and impact occur, nor a proposal about how it could or should occur.

Research and impact are never linear - they are both full of loops, revisions, dead ends and

iterations, but a linear, narrative, framework helps to identify, describe, illustrate, and even

measure impact (not dissimilar to the development of a coherent final publication from the

real, messy research). The stages of the Impact Journey are not hard and fast - they will

merge and overlap rather than being distinct and will to some extent depend on the nature of

the research and discipline. And like all journeys, the traveller can have a purpose or

destination in mind or simply travel hopefully.

The Impact Journey provides a framework in which to develop the narrative, the impact

story, to consider, demonstrate and report impact. It has been used very successfully at

Brunel University as the basis for its Pathways to Impact Toolkit, which helps academics to

write the Impact Summary and Pathways to Impact sections of funding proposals. This

thought piece concentrates on what happens after dissemination: there is already much

work available relating to the earlier stages covering peer review (inputs), research quality

(activity), and citations116 (output).

111 Horton A, A Simple Gide to Foresight, Foresight, Issue 1, Vol., 1, p 5, 1999

112 Hughes A and Martin B , ed. Docherty D, Enhancing Impact The Value of Public Sector R&D, CIHE, 2012

113 Weaver W and Shannon C , The Mathematical Theory of Communication, Univ. of Illinois Press, 1963

114 Richards, A and Ogden K, The Meaning of Meaning, Harvest, 1989

115 Donovan, C and Hanney, S, The 'Payback Framework' explained, Research Evaluation 20 (3) : 181- 183,

2011 116

LSE Public Policy Group, Maximising the Impacts of Your Research: A Handbook for Social Scientists, 2011

http://en.wikipedia.org/wiki/Communicating

http://en.wikipedia.org/wiki/Customers

http://books.google.com/?id=dk0n_eGcqsUC&pg=PP1&dq=inauthor:shannon+inauthor:weaver


65

There is a value in having an underlying model, however simple and obvious, within which to

consider impact. In particular it helps make clear that each stage in the Impact Journey is

concerned with different objectives, activities, outcomes, and timescales and therefore

provides for different disciplines and research types. The Impact Journey framework enables

the identification of these different parameters and the clarification of what is appropriate.

The key benefit of a simple underlying model is that it can be applied to all research types; it

offers a universal way in which to explore and report the impact of all HEI research. The

Impact Journey model is also discipline-agnostic and the can be used equally well to

describe the impact of social science, engineering, the arts or medical science as well as for

all the impact types as recognised by the Research Councils: knowledge, people/capacity,

social and economic.

The journey starts with Inputs - ideas, hypotheses, theories, problems to be solved, simple

curiosity; there is a change in ideas. At some point (usually when there is funding) Activities

commence - research, discovery, testing etc. Something is learnt; there is a change in

knowledge. Usually this new knowledge is then shared in Outputs - disseminated, published,

presented etc., and there is a change in the distribution of the knowledge. In a previous

world, this may have been sufficient, the end of the academic role.

However, for that research to make an impact (in any sphere), another party has to

Translate this knowledge into their own relevant context, whether this is academic, technical,

government, business, or social; there is a change in their understanding. But this is still not

sufficient for impact to happen - there also needs to be a change in behaviour - someone

has to do something different; the new understanding needs to be Used to make a difference

(often referred to as an outcome). In practice, there is usually a whole chain (or even parallel

sets of chains) of 'someone elses' doing 'something different' before the final impact arises.

Similarly impact is a change in condition. It is possible to consider both the General Impact -

a general change in, or contribution to a change in, condition and the Specific Impact - the

change in specific condition for a specific group. Reporting that, say, 'diabetic children can

now do something they could not previously', makes a better case than simply saying that'

diabetic health has improved'; specificity gives impact 'bite'.

The stages of the impact journey are necessary, but not necessarily sufficient, for impact to

occur; changes in capability and capacity (e.g. the absorptive capacity of an organisation,

the ability of people to change behaviour) are also necessary, something that may require

resource investment at earlier stages. It is also clear that user direction of research is not

necessary for impact to occur.

The earlier stages of the journey tend to be proximate, mainly about knowledge and

people/capacity impact types and can be reported with numbers - bean counting; the later

stages tend to be system-based, more reliant on external issues, moving towards the longer

term social and economic impact types and are reported as stories - case studies. Table 1

below summarises the Impact Journey stages including the narrative line and Table 2 gives

examples of the impact story at different stages.


66

Table 1. The Impact Journey

Inputs Activities Outputs Translation Usage General

Impact

Specific

Impact

Description Ideas, theories hypotheses

Discovery/

understanding

Engagement with others, especially users, communication

Translation/

awareness/brokerage

/mediation

/implication /influence

Utilization/implementation/

/execution/mobilisation

/agency/capacity

/application

More good things Fewer bad things New options

Specific benefits accruing to specific groups

The Storyline Our interest/the problem was...

and we had the expertise in…

So we researched ….

in order to …

Through the use of …

we ensured the right people know about our results

Through dialogue with....

the implications became clear in different contexts

Our research was used/adopted

/adapted

/applied/trialled

/tested by…

The general benefit was…

The specific benefits were

…. and accrued to….

Result Change in ideas

Change in knowledge

Change in knowledge distribution

Change in understanding

Change in behaviour Change in condition

Change specific in condition

Specificity Discipline Discipline Discipline Discipline & Application Discipline & Application Application Application

Reach Proximate Proximate Proximate Systemic Systemic Systemic Systemic

Most Common

Type

Knowledge Knowledge, People/capacity

Knowledge, People/capacity

Knowledge, People/capacity

People/capacity, social, economic

Social and economic

Social and economic

Development Impact plans Impact expectations

Impact intentions Impact opportunities Impact potential Impact emerging

Impact reality


67

Table 2. Examples of Impact by Journey Stage

Outputs Translation Usage General Impact Specific Impact

Change in distribution of knowledge

Change in understanding

Change in behaviour

Change in condition

Change specific in condition

1 The results of our cross-disciplinary study on cost

reduction in the XXX industry were published in the leading discipline journal, but also in

various professional publications. Seminars were

held for both trade associations and

professional societies, and contributions prepared for the radio series 'In Business' and

'Health Today'.

2 A set of one-to-one sessions were provided for the technical and commercial departments of several companies in the

XXX industry, and their professionals came to realise

that a change in their production process, and a

material substitution, could not only reduce production costs,

but thereby improve their competitive position

internationally. Following TV exposure, discussions and

workshops, health professionals uncovered the potential for significant health benefits from the removal of one particular material from

the production process.

3 Most of the companies who attended the technical and commercial seminars went

on to implement and scale up the detailed process change

information developed at bench scale in the research. Their sales teams, as well as

being able to go to customers with lower prices,

and hence compete with overseas competition, were

also able to encourage uptake of their products on a public health basis, thanks to

their PR teams' support in the preparation of targeted

materials.

4 By funding a £2m cross-disciplinary, multi-university project, we have ensured

that a market sector can now compete more effectively in the Far East. In addition, the project has led to significant health benefits (due to the

reduction in use of a particularly toxic material) for

suffers of ZZZ, which enabled the closure of a

special health centres for its treatment.

5 The £2m project for the XXX industry sector has resulted in a significantly

improved competitive position for several sector

companies, particularly relating in China. Overall their market share has increased. In addition, through toxic material

reduction, sufferers of ZZZ disease have seen a

dramatic reduction in their illnesses, to the extent where

they no longer require any specialist treatment and can

be treated at home.


68

Impact Audiences

The second key idea in this paper is the concept of Audience. Again this is not a new idea

and the concept is central to all forms of communication and marketing. Impact reports can,

and most likely will, be used by many different organisations for a wide range of possible

applications, for example to develop good practice, for staff development, for strategic

planning, and for PR, as well as for reporting to, and by, research councils and government.

The concept of Audience is similar to, but broader and more diverse than, that of

stakeholders; there are many Audiences that are not, and do not consider themselves to be,

stakeholders, and who have no knowledge or experience of research or its context. Different

types of reporting of the various aspects of impact will be required to properly connect the

information collected to specific users’ and others’ requirements and interests. Some types

of impact and evidence will be valued more by some Audiences than others and precision

and validity requirements will be different. Different Audiences will use impact reports at

different levels and will use different language.

At first sight, having a range of different Audiences for impact reports may appear to make

the subject much more complex and place a huge burden on researchers in terms of

information collection, collation and storage. However if the same underlying information and

evidence collected (including quantitative, qualitative, observational) is reported in a manner,

context, and timescale most suitable for each Audience then this complexity does not occur.

Although social science teaches that what you collect determines what is reported, and

although different users of impact reports will have different definitions, requirements &

objectives and will determine what is valid, practical and relevant for them, most cases

simply translate into different contexts and perspectives of the same underlying impact

information. Context and perspective is a critical part of considering Audience. Reporting

impact requires the appropriate background, language, and validity; in particular, considering

Audience allows for different disciplines to report their different types of impact appropriately

and allow for different audiences to use research differently. In addition the concept of

Audience ensures that decisions made by such Audiences (one of which could be the

researchers themselves) are based on suitably relevant information, appropriately

presented. Table 3 provides an example of the impact of a research project for several

different audiences (at the General Impact stage of the journey).

Table 3. Examples of Impact by Audience (at the General Impact journey stage)

Audience Interest Impact Report

Industry & Commerce

Satisfy shareholders and stock market

A By working with a local university we have improved our production process and measurement capability. As an indirect result we are also aware of new developments in the field that may allow us to launch two new products ahead of the Chinese competition. Overall the market sector is now able to compete more effectively with other overseas competition

UK Government, EU etc.

Justify research expenditure accountability, value for money

B UK university research has ensured that the UK is now better equipped to compete in the market sector that currently is worth £5bn to the UK economy. In addition it will help ensure that current legislation to ban a particularly toxic material will passed, so making the UK a safer place.


69

Research Councils, other funders

Secure funding, justify funding, accountability, audit

C By funding a £2m cross-disciplinary, multi-university project, we have ensured that a market sector can now compete more effectively in the Far East. In addition, the project has led to significant health benefits (due to the reduction in use of a particularly toxic material) for suffers of ZZZ, which enabled the closure of a special health centres for its treatment.

Academics & students

Learning, improvement, self-development, promotion

D Students involved with the research worked alongside the companies and together they tested a new production process. The work led to a significant reduction in the use of a toxic material in a production process, which has also reduced the incidence of a particular health problem. The market sector is now competes effectively with overseas competition.

Future staff, other institutions

Showing off, encouraging, PR

E The university has, through its cross-disciplinary work, developed a novel manufacturing process that, following patenting, is now enabling a key UK market sector to switch from using a highly toxic material to a much safer one. The industry sector is now fully viable and the incidence of a particular health problem has reduced dramatically. We intend to continue our process research to ensure that use of this toxic material can be eradicated entirely.

The Impact Journey - Audience Matrix

Matrix The two approaches of Impact Journey and Audience can be combined in a simple

matrix, Table 4, enabling the impact of any HEI research project, whatever discipline or

research type, to be explored, developed, and presented in terms appropriate to both Impact

Journey stage and Audience. The examples already given above, in tables 2 and 3, show

how this can be done (examples 1-5 for Journey Stage and A-E for Audience).

Table 4. The Impact Journey-Audience matrix Journey Stage/Audience Outputs Translation Usage General

Impact

Specific

Industry & Commerce A

UK Government, EU etc. B

Research Councils, other

funders

1 2 3 4 & C 5

Academics & students D

Future staff, other institutions E

1. Resources: Thus the Journey and Audience concepts provide the method to

appropriately report impact, but what are the practicalities and costs of collecting the

information, of developing and telling the impact story? In many cases it is certainly

not easy and in all cases it requires resources. Is this really possible - can this be

managed? It is possible, essentially through incorporating 'impact' into the research

process:

2. Having already thought of possible impact as part of the research proposal phase,

researchers can start to collect evidence from day one. Early in a project this is


70

relatively easy - activity measures dominate - hours, people, meetings, experiments,

debates, secondments, etc. As the journey progresses it gets harder, but it is

possible. Imagine a box, real or digital, into which is thrown, during the research

project, any bit of information that might contribute to the story in the future. For

example a conversation in the corridor that lead to a new idea, the emails that show

how a new potential user was engaged, a newspaper article that prompted a new

idea, or a chat at a conference that opened a new debate - all of which may serve, at

a later date, to tell the story across the various stages of the journey, whether it

turned out as expected or not.

3. Incorporate potential users in the research - before it even starts and on throughout

the project, so that simple regular contact generates the information; users can help

develop the story too from their perspective (another audience).

4. Keep the stories live over time and long after the research has concluded. Towards

the end of the project, turn the impact box into a draft story, add to it and refine as

time progresses, long after the project is complete, probably for years. If a researcher

or research team is still involved in the area, they should be aware of what is

happening. More generally it should become a key part of a research manager's job

to maintain, update, and keep alive the stories of what has happened to the team's

research over the years; individual researchers can keep their own stories live too,

perhaps on their CV alongside their publications lists.

5. Recognise that no-one can afford not to do it. The cost of reporting impact needs to

be a part of the research project in the same way as is ensuring research quality and

publishing. No publications? Then no track record and no future funding. Exactly the

same with impact.

Discipline Effects Whilst the Impact Journey-Audience matrix is discipline-agnostic,

discipline does have an effect. Discipline will determine how research is used and therefore

what different activities are appropriate for each journey stage. For example Usage in

astrophysics will be quite different to Usage in drama; similarly, research in drama is more

likely to have social impacts than research in physics. Discipline may also determine where

the Impact Journey 'starts'. Neither of these effects invalidate the Impact Journey-Audience

matrix but instead lead to a better understanding of 'benefit along the way'; the Impact

Journey enables descriptions and demonstrations of activities that lead up to, and are critical

too, the final impact; thus, these intermediate stages can be used to demonstrate value long

before the final impact might occur.

In terms of where the Impact Journey 'starts, there is potential for 'forward' or 'backward' shift

in all the stages of the impact journey relating to discipline. For example, basic physics

research may result in the understanding of a particular quantum effect relating to time

measurement in computer networks, so that the Translation might be influencing the design

of such networks, the Usage testing and application of the theory in a real or simulated

network, and the General Impact the development of quantum clocks for networks.

Computing research on the other hand may result in the incorporation of specialist quantum

clocks into computer networks, but in this case the Translation might be realising the benefit

of the clocks, perhaps after discussions with the physicists, Usage the incorporation of the

quantum clocks into aviation networks, General Impact coherent time keeping in aviation

networks, and Specific Impact a reduction in aircraft near misses. There is overlap here of

course and for some Audiences the stories could be combined.


71

Audience can also require the forward or backward shifts in the Journey. For example in

table 3, in economic terms relevant to the funder, the specific impact was an improvement

in the competitiveness of the sector, whilst for companies the specific impact was forward

shifted to the resulting improvement in the bottom line. Or consider a legal research

programme that has resulted in a change in government benefits policy. For the relevant

government department, the policy change may be the General Impact of the research

whilst the Specific Impact is reduced benefit costs; for the research funder there is a

backward shift in comparison, in that the policy change may only be the Usage of the

research, as the research concluded that a way to solve the problem it set out to examine

was to change behaviour by a change in government policy; the General Impact may be

that the cohort of interest now has new opportunities whilst the Specific Impact might be

their change in social status; for a new researcher in the project there could be a further

backward shift in that Usage could be civil servants reading her report, General Impact that

the university's research was used to support a policy change, and Specific Impact that her

international standing was raised.

Overall, those disciplines that are more applied will tend to have forward shifted Impact

Journeys, and those that are less so backward shifted Impact Journeys. Generally speaking

some disciplines, for example healthcare, are easier to write impact stories for than say,

English; but all can be done as the Impact Journey and Audience concepts are valid and

applicable for all disciplines, research types and impact types.

Impact Focus Bearing in mind the long impact time scales, is it valid for the Translation

stage to be a focus for earlier impact reporting? Is activity along the journey ever acceptable

as impact and can it be (formally) converted to evidencable impact? Again in practice this

happens - in medical disciplines the role & importance of knowledge brokerage is

recognised, but can this be applied more broadly?

Discipline will certainly play a role here and Translation (brokerage, mediation, influence) as

a focus for earlier impact reporting will be most valid in disciplines where there is a specific

application with a long journey distance between researcher and user. Similarly Translation

will have most benefit, and therefore be most valid as a reporting focus, when there is a

distinct user 'pull' rather than researcher 'push', but where the user/application environment

is complex - where the routes to application, values, drivers, and decision makers are

multiple and parallel. The medical environment is just that but another example is probably

education. Both these areas also have significant moral, and therefore emotional, issues;

this might be another reason to focus on the Translation stage - where the relevance or

value of the Impact stage could be disputed.

Other requirements for a model of Translation-stage impact reporting are that there needs to

be Audience acceptance that there is a valid link between Translation and Impact and that

brokerage needs to be professional and formal. Translation is on the (fuzzy) border of

research and application, so brokers need credibility in both the research and application

spheres. And perhaps most importantly, Translation activity needs to be recognised (and

paid for) as part of the research process.

Issue Mitigation Combining the Impact Journey and Audience approaches provides a route

to mitigate most of the issues associated with impact identified above, see Table 5 below.


72

The issues will of course always exist - the very nature of research ensures it - but

understanding and mitigating them will ensure that impact reporting is as valid as possible.


73

Table 5. Mitigation of Impact Issues through use of the Impact Journey and Audience Approaches Impact Journey Audience

Translation Usage General Impact Specific

1. Change over time The nature of impact changes over time

Identify forward link to anticipated impact metric

Identify intended impact & timescale.

Keep reports live or develop several versions over the time of the Impact Journey reporting Usage/Impact of relevance to the Audience

2. Time lags Impact can take a long to show

Start with Translation... …then incorporate Usage

Keep reports live for up to 25 years

Live reports for 25 years

Keep case study ‘live’, and/or develop sequential stories

3. Diffusion Impact of research becomes diffuse over time

Include plans and/or intentions for Usage.

Add in plans /intentions for General Impact

Add in plans /intentions for Specific Impact

4. Dependencies Many other outcomes, not derived from research, required for research to make impact

specifically relate to informing, mobilising, mediation, brokerage

Reports specifically relate to utilisation, and implementation (by others) and agency

The essence of Pathways to Impact – identify what is required of others to make impact arises

5. Attribution What specific research result led to what specific impact

Include clear link back to research

Include clear link back to Translation & Usage

Describe original research and the link that is of specific relevance to the Audience

6. Additionality What would have happened anyway if this specific research did not happen

Consider what would be the situation if no research had been done

Consider what would be the situation if there were no Translation

Context should identify the problem, and what was the situation prior to the research.

7. Disentanglement Clarity on about the impact of research, rather than existing expertise

Include how the new knowledge is being applied within existing knowledge

Maintain link to Usage Maintain link to Usage

8. Exogenous factors There is no mitigation possible, exogenous factors apply to life in general, not just research and its potential impacts

9. Value Different impact types are valued differently groups

Impact type, and appropriate reported for specific Audience

10. Opportunity Cost What could be done with the resources instead

The cost of measuring impact is to be part of the research project as is ensuring research quality

Once impact begins to arise, comparisons with the opportunity cost can be made.

Recognise that this is a cost of doing research in the future.

11. Non-positive impacts Positive for some Audiences

12. Unintended Consequences A good story could be told for some?


74

Where Now?

This paper is a thought piece and this final section explores, briefly, the author’s personal

ideas for further developing a better understanding of impact and providing genuine practical

advice and recommendations; questions are posed and suggestions are made, but no

answers are offered - yet.

Impact Metrics Missing from this thought piece is any discussion on measuring impact -

Impact Metrics. Whilst the early stages of the Impact Journey are usually discipline-specific

and can generally be evidenced in 'bean counting' terms (such as number of laboratories,

trials, experiments, test analyses etc. for the sciences and perhaps number of policy papers,

relationships, committee meetings, service accesses, for the arts and social sciences), the

later stages are very much application-specific and much less easy to count; we usually

resort to 'stories' - case studies.

Whilst metrics are harder to develop for 'social' than 'economic' impact types, and certainly

more difficult for the arts than, say, healthcare, it is possible. The author is currently working

on applying the Impact Journey and Audience approaches, together with the concept of

Metricated Case Studies, to the development of Pragmatic Impact Metrics, valid and robust

for all types of impact (social, people/capacity, knowledge, and economic) and all disciplines.

This work should significantly benefit those responsible for establishing impact criteria and

targets.

Impact Flows The author is also considering if there might be a parallel to the Impact

Journey in terms of impact types - perhaps a 'Flow' of impact between the types of impact -

from knowledge (required to develop absorptive capacity) to people/capacity (required as a

conduit), to social and eventually to economic? Such a flow could make capacity a proxy for

social and social a proxy for economic. Knowledge could be a (very poor) proxy for

(potential) impact. The author thinks that there is a flow, and that the flow concept could be

valuable for understanding, reporting and measuring impact; but the flow not a simple or

linear.

knowledge → people/capacity → social → economic?

Impact Proxies Does progression (albeit not guaranteed) along the Impact Journey mean

that earlier stages are valid proxies for later stages? In practice this happens - many impact

case studies actually report translation and usage activities rather than impact, particularly

for disciplines with very long timescales and where the final impact tends to be diffuse.

Depending on the audience, this could be considered the forward and backward shifting of

the journey. However there are still several interesting questions to consider on impact

proxies.

Is there a valid and useable concept of proxy distance? The author thinks there is and that it

would be a useful idea, but would be discipline and application dependant.


75

As metrics are developed could a weighting, proxy-impact correlation, or proxy

‘intensification factor’ be derived relating to potential 'amplification potential' along the

journey? The author thinks this may be possible, but probably only after a certain journey

point has been reached, probably Translation. Such a weighting would certainly be discipline

dependant, would probably be application dependant, and would require full acceptance of

impact metrics. So a long way off.


76

Impact Information Management Systems

Dr Simon Waddington117, King’s College London, UK

Introduction

In this paper, we consider recent developments in the implementation of Current Research

Information Systems across the UK Higher Education (HE) sector and the ways in which this

is influencing the ways that research information is collected and reused, particularly

focusing on research impact.

Over many years, there has been a relatively low investment by institutions in systems and

infrastructures to support the management of research activities, in comparison, for instance,

to investments made in teaching. Within the last two years, there has been major activity in

implementing institutional systems for research information management. An important

enabler has been Common European Research Information Format (CERIF) [1]. CERIF has

emerged as the de-facto UK standard for representation of research information across the

HE sector, driven by a combination of policy, reporting requirements and support from Jisc.

CERIF facilitates the exchange and reuse of research information, as well as providing a

basis for commercial CRIS products.

Research information management is characterised by a wide array of processes and

information requirements reflecting the diversity of funding organisations across the sector.

Funders have set up their own IT systems to collect research information directly from

researchers. This enables them to collect information to meet their exact requirements, but

makes the task for institutions to understand and plan the research activities they are hosting

extremely difficult.

In parallel to the development of research information systems, there has been increasing

interest in understanding and representing research impact itself. Examples are Project

Snowball [40] as well as the Jisc-funded DESCRIBE [46] and MICE [22] projects. A

distinction is often made between ‘academic impact’, understood as the intellectual

contribution to one’s field of study within academia and ‘external socio-economic impact’

beyond academia [45]. “Impact” has become the term of choice in the UK for research

influence beyond academia.

For the purposes of this article, we regard impact information in the context of all research

information, which covers the full range of information about research projects, including

people, funding, equipment, finances and outputs. Impact of research beyond academia,

such as the development of a new drug, often relies on the work of many researchers, as

well as non-researchers, over a long timeframe. There is often a complex chain of influence,

which makes direct attribution extremely difficult. In some cases, the true value of

fundamental research might only be realised many years later, and many steps away from

its originators. Impact can only be effectively evaluated through an analysis of a

comprehensive and rich collection of research information spanning many years. Often, the

culture and skills within institutions are often not appropriate to tackle these issues.

117 Address correspondence to Simon Waddington, Centre for e-Research, King’s College London, 26-29 Drury Lane, London WC2B 5RL, UK E-mail: [email protected].


77

UKRISS feasibility study

The discussion in this paper draws on a study carried out in the first part of the Jisc-funded

UKRISS project [41], conducted between March and December 2012. The study aimed to

investigate the potential for harmonisation of the reporting of research information at a

national level within the UK Higher Education sector, based on adoption of the CERIF

standard. During the course of the study interviews were conducted with over forty

stakeholders across the sector. This included representatives, at different levels, within

research management staff of HEIs representing the five major interest groups (Russell,

1994, Alliance, GuildHE and Million Plus), as well as two independent research institutes.

Several representatives of funders from RCUK were interviewed as well as representatives

of large and small charities. National bodies HEFCE and HESA were included, as well as the

major commercial vendors of research information systems and the umbrella organisations

ARMA and UCISA.

The interviews followed a structured set of sixty questions that was tailored according

typology of the interviewee. Recorded interviews were analysed and requirements extracted

and de-duplicated. The full set of stakeholders, interview questions and requirements is

available on the UKRISS blog site [41].

The study examined the full range of research information collected by institutions, the

systems used to gather this information, both within institutions and funding bodies, as well

as the motivation for information gathering and reuse.

Current research information management system landscape

CERIF

CERIF [1] has been a key enabler in the field of research information management in the UK

Higher Education (HE) sector, both for the exchange of research information as well as for

the adoption of IT systems for managing research information.

CERIF was developed with the support of the European Commission (EC) in two major

phases: from 1987-1990 and 1997-1999. It is a standard as well as a recommendation by

the European Union to its member states [50]. Since 2000, care and custody of CERIF has

been handed by the European Commission to euroCRIS [2], a not-for-profit organisation

dedicated to the promotion of Current Research Information Systems (CRIS). CERIF defines

a model for representing the entities in research information (e.g. people, outputs,

equipment) and their relationships.

The EXRI-UK [3] study of 2009, commissioned by Jisc, conducted a review of available

standards for representation and exchange of research information. It recommended the

adoption and further development of CERIF (the latest version was CERIF 2008). The

recommendations included developing pilots to demonstrate the application of CERIF in

specific use cases. In the UK, Jisc has been responsible for commissioning a number of

projects (e.g. RMAS [49] , CERIF in Action [19], IRIOS 1 and 2 [17] [18], MICE [22], R4R

[21], BRUCE [20]) that have contributed to the validation and extension of the standard.


78

In an international context, CERIF is not the only standard for research information. In North

America, both VIVO and CASRAI have emerged as potentially overlapping standards

activities.

The EXRI-UK study was supported by a further study commissioned by Jisc in 2010 to

examine the business case for CERIF adoption [5]. It concluded that the overall cost of

either deploying a CERIF-compliant institutional and funder systems, or writing CERIF

wrappers around non-CERIF compliant systems was low in relation to the benefits that could

be realised in terms of reduced complexity of information exchange, compared to exchanges

in multiple ad hoc formats.

Institutional CRIS systems

A CRIS is a database or other information system storing data on current research by

organisations and people, usually through some kind of project activity, financed by a

funding programme [47].

Since the EXRI report in 2009, the number of UK institutions implementing an institutional

CRIS has risen dramatically. A Jisc-funded report [4] released in January 2012 examined the

adoption of CERIF-compliant systems within UK HE institutions. At that point, around 30% of

UK Higher Education Institutions (HEIs) had either introduced a CRIS or had firm plans to do

so, with a rapidly rising adoption rate. All but one of these CRIS’s was supplied by a

commercial vendor, with the single exception being an in-house development.

Currently there are three major commercial CRIS systems in the UK market: Pure from Atira

[6], Converis from Avedas [8], and Elements from Symplectic [7]. Additionally, the ePrints

repository [27] has a plug-in, which provides additional features to a CRIS.

Commercial CRIS offerings typically provide support for the pre and post-award research

lifecycle, including reporting, repository deposit, Research Excellent Framework (REF)

submissions and management of research entities (e.g. projects, grants, publications,

awards, patents). Integration with external sources such as bibliometric services (e.g. Web

of Science, Scopus, PubMed) is possible, as well as integration with corporate systems (HR,

finance). Analysis and visualisation tools are provided at varying levels of complexity to

provide business intelligence and management information.

Implementation of an institutional CRIS represents a major investment for HEIs.

Procurement costs typically involve not only setting up and installation of relevant servers

and software, but also major work to integrate data across multiple sources within the

institution. At 2010 prices, Bolton [5] estimated the annualised cost over ten years of

purchasing and maintaining a CRIS system at between £10k and £20k per annum.

Additional system and data integration costs may make this figure considerably higher, as

well as costs for staff training. The move to CRIS systems has been led by larger research-

led institutions, where considerable efficiency savings can be demonstrated. For smaller

institutions, the high fixed costs of CRIS implementation are still a major barrier. There is a

gap in the market for solutions that can be provided as software-as-a-service, which would

remove the need for maintaining and running IT infrastructure internally.


79

Funder CRIS systems

In parallel to the uptake of CRIS systems within institutions, funders have implemented

systems for capture research outputs and impact information. Five of the RCUK members

(AHRC, BBSRC, EPSRC, ESRC and NERC) have adopted the Research Outputs System

(ROS) [9]. The two remaining RCUK councils (MRC and STFC) have developed a system

originally called e-Val, but now referred to by the name of the commercial provider Research

Fish [10]. Research Fish also hosts outputs systems for Wellcome Trust and a number of

smaller charity funders.

ROS enables both input of research outputs by individual researchers as well as bulk upload

by research offices. With the increasing adoption of institutional CRIS’s, RCUK is extending

the capabilities of system-to-system submission, with around 50% of submissions already

using this method [51]. This will enable institutions to collect and aggregate information in-

house prior to uploading the data to ROS. Although ROS is not natively CERIF compliant,

work has been carried out to implement CERIF wrappers to enable data exchange in CERIF

format. Although all research councils have adopted Je-S [25] as the common portal for

grant submissions, the councils using ROS each have an independent database. This has

led to often minor discrepancies in the ways output and impact information is collected and

represented, limiting the reuse of the information for cross-funder analysis.

The primary mode of data collection by Research Fish is direct entry by researchers. Unlike

ROS, where research outputs and impact information are collected on an on-going basis

throughout the lifetime of the grant and beyond, Research Fish information is collected in a

fixed time window each year. Users of Research Fish can be penalised for failing to enter

the required information by potential withdrawal of funding. Typically there is a much greater

degree of direct interaction with research council staff and researchers to ensure quality of

the data. Research Fish provides a standard core question set that is common to all funders

using the system. Over the five years that Researchfish has been in operation, a large

corpus of high quality and comparable data has been collected.

Due to the increasing number of co-funded research grants, cross-referencing of information

in ROS and Research Fish is now possible. Researchfish, however, does not currently

support CERIF as an information exchange format.

National CRIS system

Each of the RCUK funders currently maintains a separate public portal for providing

information about funded research, including outputs and impact information known as

Grants on the Web. Gateway to Research (GtR) [11] is an initiative funded by the

Department for Business, Innovation and Skills (BIS) to build a single national portal to

facilitate public access to the UK’s research outputs. In particular, a key aim is to improve

access to publicly funded research by entrepreneurs and SMEs to promote economic

activity. Successful implementation and uptake of such a system is likely in itself to increase

the impact of academic research on business, health and society.

GtR is a major driver to harmonise research information that is collected by the research

councils, since information harvested from ROS and Research Fish needs to be normalised

to a single format. Currently only a relatively small proportion of the data in the funder

systems is available through GtR. The Jisc funded G4HE project [42] is aiming to close the


80

loop to institutions by providing interfaces for institutions and other users to harvest

information stored in GtR for benchmarking and other purposes.

A number of other countries have implemented national CRIS systems, some of which have

been in existence for ten years or more, and provide an indicator for future directions in the

UK. Existing national or regional CRIS systems include CRISTin in Norway [28], FRIS in

Flanders [29], HUNCRIS in Hungary [30], SICRIS in Slovenia [31], Star Metrics in USA [32],

and NARCIS in the Netherlands [33] as well as METIS [34]

The Czech national CRIS Error! Reference source not found. has been in existence since

1994. It contributes to providing trust in national research and innovation activities, by

providing open access to a large volume of research information funded by national

programmes, which is widely used by government offices, funders, researchers and the

general public. Research impact information is reported by researchers to their institutions,

which then perform bulk upload to the national CRIS. At each stage, the information is de-

duplicated and undergoes quality assurance reviews, resulting in a high quality final output.

This is reflected in the usage statistics collected for the service. Within a 2 year period, the

system is thought to have been used by around 39,000 academic users. For comparison,

there are 43,000 researchers and 78,000 R&D personnel in the country. The system is used

by researchers and the general public to access information about funding, people,

organisations and research results. Government bodies and agencies are able to perform

benchmarking and analysis. The system is also used to support project proposal evaluation.

Impact information

Impact information capture

Gathering research impact information is a challenging activity due to the length of time with

which data needs to be captured and the difficulty of representing the information in a

systematic way that lends itself to analysis and benchmarking. In addition, impact varies

significantly between disciplines and according to the position in the research and

development lifecycle. Much entry of research information is still done manually by

researchers. For instance, many institutions still rely on manual upload publications, a

process which can be largely automated through the use of bibliometric services and open

access repositories. Many institutions are collecting a much wider range of information than

publications, which include “grey” outputs such as unpublished reports, presentations etc.,

which have a bearing on the overall impact but are not regarded as traditional outputs.

Repositories and other systems can now collect information about downloads of research

papers. In itself a large number of downloads (from different locations) indicates a higher

level of interest in research outputs. Indeed social networking tools for researchers may

provide a further mechanism for assessing impact. Academics are increasingly moving much

of their research activity to the web, including not only publications, but also self-publishing

(e.g. blogs), software and datasets. Altmetrics [52] is a form of crowd-sourced peer review,

where the impact of an article might be assessed by thousands of conversations and

bookmarks in a week. This should be contrasted to the many months required for peer

review, or several years required to count citations. Altmetrics can be harvested and

processed using big data algorithms and used to provide impact measures as well as search

filters and recommendations.


81

Impact may occur a long time after a piece of fundamental research has been carried out.

The researcher concerned may have left the institution. Hence there is a need for

administrators or other researchers to be able to enter information, or for there to be

exchange of impact information either between institutions or funders. In most cases, the PI

of a project is best able to understand and report on the impact of their research, and indeed

outcomes systems such as Research Fish use researchers as their main point of contact.

Motivation for gathering impact

The UKRISS study looked in depth at the motivations for institutions and funders in investing

in CRIS systems for the management of information on research activities, outputs and

impact.

Both institutions and funders are motivated by an increasing desire to improve business

intelligence in order to manage their research portfolios and inform strategy and planning.

Business intelligence also involves being able to capture and validate the whole range of

activities, outputs and impacts of an individual researcher to enable informed decision-

making and performance management. Senior leaders and research office staff within

institutions want to collect information on research impact to benchmark their institution

against other institutions, both within the UK, but also internationally.

Many stakeholders, particularly those responsible for overseeing research in HE institutions,

indicated that improving the quality and impact of their institution’s research was their key

driver. The UKRISS feasibility study [41] indicated there is a wide consensus between

government, funding bodies and institutional senior management that better quality research

information was essential to enabling this improvement. This research information

underpinned their business intelligence, and its quality, presence or absence had a

significant effect on their ability to plan and manage their research portfolio. Good business

intelligence also allowed stakeholders to demonstrate value, exploit strategic gaps and

opportunities, and remain competitive.

There is an increasing burden on research managers at institutions and funders to respond

rapidly to multiple requests for information on research impact, both internal and external.

Indeed this emerged as a motivation for implementing a CRIS system in the UKRISS

feasibility study [41]. Institutions need to respond effectively to statutory reporting such as

REF, HEBCIS and HESA returns, as well as reporting on research projects from a wide

range of funding organisations with differing reporting requirements.

RCUK funders themselves need to be able to respond rapidly to requests for information

from government departments to justify their existing and future resourcing. There was a

clear distinction between the motivations of RCUK and charity funders. Whereas RCUK

funders place more emphasis on quantitative impact information in order to provide evidence

of value for money to government, charity funders were often more interested in qualitative

information on impact. To support their fundraising activities, charities are particularly keen

to collect narrative information from researchers on a particular impact that has been

achieved as a result of their funded research, such as a new medical treatment, which can

be used in marketing materials.

Cost savings are an important factor in deploying CRIS systems. The shared goal of

reducing the reporting burden on researchers and research administration by greater


82

automation of information management was recognised as important but not described by

stakeholders in the context of cost savings. However, on-going costs for the sustainability of

IT solutions were a consideration.

Motivations for researchers to upload impact information were mostly compliance-based.

That is, it was mandated by the funder or host institution. An emerging driver in this regard

was observed where an institutional CRIS was the unique source of information for

performance reviews, promotion panels, REF submissions and other reporting that had a

direct influence on career progression. An incentive for researchers included auto-generation

of CVs and web profiles that were configurable, and the more general ability for researchers

to extract and re-use the information they had submitted. Researcher adoption was closely

linked to the ease of use of the system interface.

Government departments have a major interest economic and societal impact. There is an

on-going need to justify funding given to HE sector. Academic funding is regarded as a

catalyst for SMEs and new business development, development of new products. The

Higher Education sector itself is a major contributor to the UK economy through attracting

students and skilled researchers to the UK, for which good performance in international

research benchmarks is a critical factor.

Challenges to impact evaluation

Research reporting landscape

Institutions are faced with a bewildering array of reporting requirements on both teaching

and research activities. In the research domain, these include responses to funders,

including state-funded, charity and commercial organisations. The HESA Information

Landscape project Error! Reference source not found. conducted a wide-ranging study of the

information requests made to institutions by external bodies. The survey compiled a

catalogue of over five hundred and fifty distinct requests covering teaching and research.

The reporting requirements and processes for research vary significantly between research

councils using ROS and those using Research Fish. Even amongst councils and other

funders using respectively the ROS and Research Fish systems, there are significant

differences in the inputs required.

Research impact has played a key role in the 2014 REF. HESA returns are mandatory for

institutions. Although primarily focused on teaching, there is significant overlap, particularly

in the area of research students. The annual HE-BCI survey [37], collected by HESA on

behalf of HEFCE collects information directly relevant to wider impact such as patents and

spin-offs.

This results in an additional workload for institutions in collecting the same data multiple

times or reformatting information for different purposes. A subset of this information collected

concerns research impact. Thus although much research information is already collected, it

may be specified in different ways.

Harmonisation and interoperability

The diversity of information requests made to institutions raises two issues. On the one

hand, there is a need for harmonisation so where common information fields are required by


83

multiple external organisations, they should as far as possible be aligned, to avoid duplicate

information collection. On the other hand, the same information fields are collected, the

semantics of the information request should be the same, so that information can be

exchanged and reused.

As a follow-on to the HESA Information Landscape Study, the Regulatory Partnership Group

[48] has been established to develop a proposal for a governance structure to promote the

coordination of information requests made to HE institutions by external organisations. This

activity will aim to identify opportunities to align information gathering requirements and to

define common dictionaries where the requirements are the same. The different

understanding of impact information and its reuse mean that this is an area for which

consensus will be difficult to achieve in the short term.

The widespread adoption of CERIF in the UK HE sector has the potential to greatly facilitate

the flow of research information across the sector. CERIF provides a model for representing

the relationships between research entities. CERIF offers a great deal of flexibility in

representation of the relationships, but the precise semantics are not defined in the

standard. The issue of semantic dictionaries is being addressed by CASRAI and euroCRIS.

The definition of identifiers is also crucial for the work, in particularly ORCID [24] for research

staff, FundRef [38] for grants and CrossRef [39] for publications.

Data quality

Data quality is a major issue within research information systems. The RMAS project [49]

found that over 30% of research information stored at partner institutions was either missing

or incorrect. There is an on-going need for manual entry of information by researchers. This

includes information on “grey” outputs such as unpublished reports, and subjective

information on impact that cannot easily be harvested from other sources. Hence there need

to be clear incentives to enter information, and the systems should be easy to use.

Researchers and research administrators submitting data need to have a clear indication

about the purpose for which data is being collected and is to be used, and the potential

impact this may have on them.

The process adopted by the Czech National System Error! Reference source not found. of

validating research information at institutional, funder and national levels indicates that such

an approach yields high quality information. Research Fish has succeeded in collecting high

quality data through rigorous checking by research council staff and strict enforcement

policies for data collection. This high level of manual intervention has a cost implication for

the funders. Linking funder outputs systems to other information sources such as

repositories and publisher systems should enable greater automated validation of

information.

Traceability

Determining impact of fundamental research requires a consistent set of research

information collected over a number of years to trace back from an impact through the

research projects that contributed to that advance. This is valuable not only to attribute the

appropriate credit to researchers who made advances to this impact. It can also be used by

funders to determine the costs of the research versus the benefits, and the effectiveness of

the research programmes. Institutions can make strategic decisions based on their own


84

resource allocations and commitments. It should be kept in mind that research may be built

on many years of activity, requiring collaboration across multiple institutions, some of which

may be outside the UK. There is a requirement for CRIS systems to enable the capture of

high quality information, but also the need for stability and harmonisation in the way the

information is captured.

Socio-economic impact

Increasing or ring-fencing the research budget has been seen by governments as a route to

stimulating economic activity following the 2009 banking crisis. In the USA, the American

Recovery and Reinvestment Act (ARRA) provided an additional $21 billion of research

funding, and created a political need to measure the return on this investment. This led to

the creation of Star Metrics [54], a five year programme to create a set of metrics to measure

research impact. Although the initial focus was on job creation, the aim is to measure or

predict economic, scientific, and societal benefits, and provides indicators for possible

approaches for the UK.

The view of successive UK governments in the past thirty years has been based on an

assumption that research should be justified in large part by its economic impact. However,

as Margaret Thatcher recognised in 1998 [53]: “First, although basic science can have

colossal economic rewards, they are totally unpredictable. And therefore the rewards cannot

be judged by immediate results.” Thus there is still a requirement for basic research driven

by intellectual curiosity.

A major challenge in this area is to provide robust methods that can produce indicators of

wider socio-economic impact without increasing the burden on researchers and research

organisations. The US Committee National Statistics 2012 interim report [55] recommends

performing large scale analysis on the growing amount of data available online and in

publicly accessible databases to extract impact related information. This relates in particular

to analysing information from outside the HE sector such as activities of high-growth

companies and labour market data collected by government departments. This suggests a

much wider role of CRIS systems within the national research landscape and a requirement

for information harmonisation and sharing across the public sector.

Future developments and trends

A number of clear trends are emerging in the area of research impact management

processes and systems.

1. An increasingly wide deployment of CERIF-compliant CRIS systems within both

institutions and funders across the UK HE sector. We expect this trend to continue,

particularly within larger research-oriented institutions. Existing CRIS systems are

expensive and complex to deploy. Interviews with staff at smaller institutions conducted

during the UKRISS study [41] indicate there remains a significant gap in the market for

scalable solutions for such institutions.

2. Increasing harmonisation in the information requests made to institutions from external

bodies. This includes requests made by HEFCE and research funders around the

research. This is both a political as well as a technical issue. Although progress here is

likely to be slow, unification around a core set of information fields seems to be a realistic

goal in the short to medium term.


85

3. Increasing progress to standardise semantic dictionaries and to provide global identifiers.

ORCID is on the verge of adoption, which would already be a major advance, and

enable researchers to be unambiguously identified. euroCRIS and CASRAI are working

towards a full interoperability of CERIF through agreed data dictionaries and identifiers.

Gateway to Research is also proving to be a major driver within the UK research

councils. There still remain major gaps such as agreeing common definitions of

institutional structures.

Given further progress on points 1-3, it is reasonable to expect an increase in the volume of

research information that can be exchanged and reused. This will lead to an increase in

tools to exploit such information. There is a large appetite within the sector of business

intelligence and management tools, including benchmarking. There is a potential for more

sophisticated research impact measures that can mine information gathered across

traditional research boundaries, as well as over historical data.

Conclusions

In this paper, we have described the current and future trends for implementing CRIS

systems for the management of research information and the impact this is having on

institutions. The landscape is changing at a number of levels:

1. At an institutional level, there is an increasing interest in strategic management of

research and consequent investment in IT infrastructure. The implementation of a single

CRIS system within an institution, which collects and integrates data about research,

which was previously distributed across multiple systems (finance, HR, repository). This

system and data integration task in itself requires a substantial commitment in resources,

and there are substantial issues to be overcome, such as data quality. The access to

integrated information about research across multiple disciplines provides an opportunity

for strategic research management, including the potential for analysis and deriving

impact information.

2. At a funder level, many funders are implementing systems to collect reporting, outputs

and impact information systematically across their research portfolios.

3. At a national level, there is a strong interest in improving access for industry, and

particularly SMEs, to the outputs of the UK HE sector to support economic growth and

development. This has manifested itself in activities such as the Gateway to Research

(GtR) project, which aims to create a national portal based on research council funded

research.

The final link in this chain will be to provide institutions with high quality aggregated research

information, based on data collected in a national portal, which will facilitate cross-sector

analysis of research impact.

Acknowledgements

I would like to acknowledge the contribution of my colleagues on the Jisc-funded UK

Research Information Shared Service (UKRISS) project in the preparation of the UKRISS

Feasibility Study, on which this article is based. In particular thanks go to Richard Gartner

(King’s College London), Keith Jeffery (euroCRIS), Richard Jones (Cottage Labs), Lorna


86

Mitchell, Rosa Scoble (Brunel University), Allan Sudlow, Karen Walshe (British Library), and

Stephen Trowell (University of Exeter).

References

[1] Common European Research Information Format (CERIF):

http://www.eurocris.org/Index.php?page=CERIFreleases&t=1#.

[2] euroCRIS: http://www.eurocris.org.

[3] EXRI-UK Report (2009): http://repository.jisc.ac.uk/448.

[4] Jisc Adoption of CERIF in UK HEIs report (2012):

http://isc.ukoln.ac.uk/2012/03/15/adoption-of-cerif-in-uk-heis-report-just-published/.

[5] The Business Case for the Adoption of a UK Standard for Research Information

Interchange: Report to Jisc (2010):

http://www.jisc.ac.uk/publications/reports/2010/businesscasefinalreport.aspx.

[6] Atira Pure: http://atira.dk/en/pure/.

[7] Symplectic Elements: http://www.symplectic.co.uk/.

[8] Avedas Converis: http://www.avedas.com/.

[9] Research Outcomes System:

http://www.rcuk.ac.uk/research/Pages/ResearchOutcomesProject.aspx.

[10] Research Fish: http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e-

Val/index.htm.

[11] Gateway to Research project: http://www.rcuk.ac.uk/research/Pages/gtr.aspx.

[12] HESA Information Landscape Study: http://landscape.hesa.ac.uk/wp-

content/uploads/2012/01/IRPG_PrjB_Final_Report.pdf.

[13] Jisc Readiness for REF (R4R): http://r4r.cerch.kcl.ac.uk/.

[14] VIVO: http://vivoweb.org/.

[15] CASRAI: http://casrai.org/.

[16] Research Management and Administration System (RMAS) project:

http://www.rmas.ac.uk/.

[17] IRIOS project: http://www.irios.sunderland.ac.uk.

[18] IRIOS2 project: http://irios2.wordpress.com/.

[19] CERIF in Action (CiA) project: http://cerifinaction.wordpress.com/.

[20] BRUCE project: http://bruceatbrunel.wordpress.com/.

[21] Readiness for REF (R4R) project: http://r4r.cerch.kcl.ac.uk.

[22] Measuring Impact under CERIF (MICE) project: http://mice.cerch.kcl.ac.uk.

[23] RiO Extension project:

http://www.jisc.ac.uk/whatwedo/programmes/di_researchmanagement/repositories/rioext

ension.aspx.

[24] ORCID: https://orcid.org/.

[25] RCUK Je-S System: http://je-s.rcuk.ac.uk/.

[26] RCUK Research Outcomes System (ROS):

http://www.rcuk.ac.uk/research/researchoutcomes/pages/ROS.aspx.

[27] ePrints repository system: http://www.eprints.org/.

[28] CRISTin system: http://www.cristin.no/english/.

[29] FRIS system: http://www.researchportal.be/en/.

[30] HUNCRIS system: http://www.icsti.su/uploaded/200906/praha2009/hung_prs.pdf.

[31] SICRIS system: http://sicris.izum.si/.

http://www.eurocris.org/Index.php?page=CERIFreleases&t=1

http://www.eurocris.org/

http://repository.jisc.ac.uk/448

http://isc.ukoln.ac.uk/2012/03/15/adoption-of-cerif-in-uk-heis-report-just-published/

http://www.jisc.ac.uk/publications/reports/2010/businesscasefinalreport.aspx

http://atira.dk/en/pure

http://www.symplectic.co.uk/

http://www.avedas.com/

http://www.rcuk.ac.uk/research/Pages/ResearchOutcomesProject.aspx

http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e-Val/index.htm

http://www.mrc.ac.uk/Achievementsimpact/Outputsoutcomes/e-Val/index.htm

http://www.rcuk.ac.uk/research/Pages/gtr.aspx

http://landscape.hesa.ac.uk/wp-content/uploads/2012/01/IRPG_PrjB_Final_Report.pdf

http://landscape.hesa.ac.uk/wp-content/uploads/2012/01/IRPG_PrjB_Final_Report.pdf

http://r4r.cerch.kcl.ac.uk/

http://vivoweb.org/

http://casrai.org/

http://www.rmas.ac.uk/

http://www.irios.sunderland.ac.uk/

http://irios2.wordpress.com/

http://cerifinaction.wordpress.com/

http://bruceatbrunel.wordpress.com/

http://r4r.cerch.kcl.ac.uk/

http://mice.cerch.kcl.ac.uk/

http://www.jisc.ac.uk/whatwedo/programmes/di_researchmanagement/repositories/rioextension.aspx

http://www.jisc.ac.uk/whatwedo/programmes/di_researchmanagement/repositories/rioextension.aspx

https://orcid.org/

http://je-s.rcuk.ac.uk/

http://www.rcuk.ac.uk/research/researchoutcomes/pages/ROS.aspx

http://www.eprints.org/

http://www.cristin.no/english/

http://www.researchportal.be/en/

http://www.icsti.su/uploaded/200906/praha2009/hung_prs.pdf

http://sicris.izum.si/


87

[32] Star Metrics: https://www.starmetrics.nih.gov/.

[33] NARCIS: http://www.narcis.nl/.

[34] METIS research information system:

http://www.surf.nl/en/themas/openonderzoek/infrastructuur/Pages/Researchinformations

ystems.aspx.

[35] Thomson Reuters Web of Knowledge: http://wokinfo.com/.

[36] Scopus (Elsevier): http://www.info.sciverse.com/scopus.

[37] HE-BCI survey: http://www.hefce.ac.uk/whatwedo/kes/measureke/hebci/.

[38] FundRef: http://www.crossref.org/fundref/index.html.

[39] CrossRef: http://www.crossref.org.

[40] Project Snowball: http://www.snowballmetrics.com/.

[41] UKRISS project: http://ukriss.cerch.kcl.ac.uk.

[42] G4HE project: http://g4he.wordpress.com/.

[43] Czech National System. euroCRIS conference paper.

[44] T. Chudlarský, J. Dvořák, A National CRIS Infrastructure as the Cornerstone of

Transparency in the Research Domain, In: Jeffery, Keith G; Dvořák, Jan (eds.): E-

Infrastructures for Research and Innovation: Linking Information Systems to Improve

Scientific Knowledge Production: Proceedings of the 11th International Conference on

Current Research Information Systems (June 6-9, 2012, Prague, Czech Republic). Pp.

9-17. ISBN 978-80-86742-33-5.

[45] T. Penfield, M. Baker, R. Scoble and M. Wykes Assessments, Evaluations and

Definitions of Research Impact: A Review, Preprint. University of Exeter.

[46] DESCRIBE project: http://www.exeter.ac.uk/research/excellence/describe.

[47] Wikipedia, CRIS system:

http://en.wikipedia.org/wiki/Current_Research_Information_System.

[48] Regulatory Partnership Group (RPG): http://www.hefce.ac.uk/about/intro/wip/rpg/.

[49] RMAS project: http://www.rmas.ac.uk/.

[50] CERIF – 1.3 Semantics: Research Vocabulary:

http://www.eurocris.org/Uploads/Web%20pages/CERIF-

1.3/Specifications/CERIF1.3_Semantics.pdf.

[51] J. Toon, Survey Results and extended use cases (CIA and ROS), Preentation at

CERIF in Action Workshop, London, 19th October 2012:

http://cerifinaction.wordpress.com/cia-workshop-19th-october-2012.

[52] Altmetrics manifesto: http://altmetrics.org/manifesto/.

[53] M. Thatcher, Lecture to the Royal Society, 27th Sept., 1998,

http://www.margaretthatcher.org/document/107346.

[54] Star Metrics: https://www.starmetrics.nih.gov/.

[55] Improving Measures of Science, Technology, and Innovation: Interim Report (2012),

Committee on National Statistics (CNSTAT),

http://www.nap.edu/openbook.php?record_id=13358.

https://www.starmetrics.nih.gov/

http://www.narcis.nl/

http://www.surf.nl/en/themas/openonderzoek/infrastructuur/Pages/Researchinformationsystems.aspx

http://www.surf.nl/en/themas/openonderzoek/infrastructuur/Pages/Researchinformationsystems.aspx

http://wokinfo.com/

http://www.info.sciverse.com/scopus

http://www.hefce.ac.uk/whatwedo/kes/measureke/hebci/

http://www.crossref.org/fundref/index.html

http://www.crossref.org/

http://www.snowballmetrics.com/

http://ukriss.cerch.kcl.ac.uk/

http://g4he.wordpress.com/

http://www.exeter.ac.uk/research/excellence/describe

http://en.wikipedia.org/wiki/Current_Research_Information_System

http://www.hefce.ac.uk/about/intro/wip/rpg/

http://www.rmas.ac.uk/

http://www.eurocris.org/Uploads/Web%20pages/CERIF-1.3/Specifications/CERIF1.3_Semantics.pdf

http://www.eurocris.org/Uploads/Web%20pages/CERIF-1.3/Specifications/CERIF1.3_Semantics.pdf

http://cerifinaction.wordpress.com/cia-workshop-19th-october-2012

http://altmetrics.org/manifesto/

http://www.margaretthatcher.org/document/107346

https://www.starmetrics.nih.gov/

http://www.nap.edu/openbook.php?record_id=13358


88

Different pathways to impact? “Impact” and research fund allocation in selected

European countries

Anke Reinhardt, Director of Evaluation and Monitoring at the German Research

Foundation (Deutsche Forschungsgemeinschaft/ DFG)118

Introduction

“Impact” is one of the signs of the times – a concept that appeared in research and higher

education policies in OECD countries after the turn of the millennium. And as it seems, the

impact agenda is likely to continue influencing research policies in the years to come.

Yet, the popularity of the term should not conceal the fact that impact means different things

in different countries: it follows a different logic, influences research fund allocation

differently, and will have a different effect on the respective national research systems.

Among European countries, the United Kingdom is at the forefront of implementing a

comprehensive “impact” agenda that not only seeks to capture impact but also allocates a

share of the public research budget according to its definition of “impact”119. If Britain is at the

forefront, are other countries going to follow or are they choosing a different path?

I will argue here that while everything else being equal, virtually all governments and other

funders120 favour funding research that has impact over research that only spurs academic

debates or that has no impact at all. But as I will demonstrate, everything else is not equal.

Countries differ largely in the share of public funding of research, research systems differ in

respect to the balance between basic research and applied research, and the political and

economic systems of countries differ in the extent to which they are open to seek advice and

stimulus which is based on research121. This also influences the way countries conceive of

the term “impact” and the optimal way of capturing and assessing it.

Specifically, this article starts from the well-known (though simplified) distinction between

basic and applied research suggested by Vannevar Bush (1945). Accordingly, applied

research seeks an immediate impact, while basic research aims at understanding complex

phenomena and does not intend to develop immediate real impact.122 However, it would be

wrong to assume that basic research fails to influence the “real” world. For example, in the

first decades the wider “impact” of Einstein’s research on light quanta remained

insignificantly different from zero outside the academic sphere. In a longer perspective,

however, Einstein’s theory of light – amongst others - laid the foundations for laser

118 The article reflects the personal views of Anke Reinhardt.

119 In the following, I will use quotation marks to signal the use of the term “impact” in this article as a very specific

concept which encompasses not only academic impact of research but other effects (mainly socio-economic, but

also cultural, environmental etc impacts). 120

Such as the European Commission or private foundations such as the Wellcome Trust. 121

Bozeman (2000) calls this “demand environment”, where industry serves as an example for a “transfer

recipient”. In his “contingency effectiveness model of technology” supply and demand of research have to match

to result in effective technology transfer. 122

The OECD defines: “Basic research is experimental or theoretical work undertaken primarily to acquire new

knowledge of the underlying foundations of phenomena and observable facts, without any particular application

or use in view. Applied research is also original investigation undertaken in order to acquire new knowledge. It is,

however, directed primarily towards a specific practical aim or objective” (OECD 2002, p. 30).


89

technology – a multi-billion pound technology that affected virtually all areas of modern life

including medicine, entertainment, and the economy (Zinth, Laubereau and Kaiser 2011).

The remainder of this article proceeds as follows: I will first sketch the logic of research fund

allocation and the British concept of “impact”. Then I will outline the international situation,

followed by country case studies of Germany, France and the Netherlands. Finally, I will

discuss the findings.

The Politics of Research Fund Allocation

Funding allocation is governments’ main steering instrument to achieve science policy

objectives (Kjelstrup 2001). Decades ago many countries started to think about the optimal

distribution of research funding123.

While the distinction between basic and applied research proves useful to differentiate

between research projects, the distinction has much less appeal for the categorisation of

research agendas and research fund allocation policies. Most countries use a three-tier

funding approach. In practice, the portfolio of most research agendas will contain basic and

applied research projects. To use the same example: the modern research agenda on light

quanta and light-emitting devices includes both applied and basic research projects. To steer

research funds, governments can decide to direct funds to selected research agendas.

Stokes (1997) uses the label “strategic research” for this category, which comprises applied

and basic research projects.124

To support these different types of research, governments use two main funding streams:

First, research funds are directly allocated as institutional funding to research institutes that

concentrate on either type of research125. For example, Germany provides funds for the

Fraunhofer Institute (which concentrates on applied research with strong links to industry),

for the Max Planck Society (focusing on basic research) and for the Helmholtz centres

(performing basic and applied research in a specified fields, e.g. energy, and maintaining

large research facilities). As a steering or distribution mechanism, some countries use

research evaluation systems based on and intending to stimulate research excellence.

Second, funding agencies distribute research grants for individual projects and/or individual

scientists on a competitive basis for the above-mentioned types of research126.

These strategies allow governments and research agencies to use research funds in a

strategic and flexible way. Typically, they use different funding criteria for each respective

category.

The British “Impact” blueprint

The UK has a very strong research base, especially in the field of basic research. After the

US, China and Japan, it is the no. 4 producer of scientific publications (NSF 2012). The UK

123 The “Bush-Report” of 1945, commissioned by President Roosevelt, was one of the most prominent milestones

in that debate, followed by contributions by – inter alia – Weinberg 1964 and Toulmin 1964. 124

This third category is often called “strategic”, “thematic” or “mission-oriented basic research”. “Oriented basic

research is carried out with the expectation that it will produce a broad base of knowledge likely to form the basis

of the solution to recognized or expected, current of future problems or possibilities” (OECD 2002, p. 78). 125

This could be interpreted as an ex post-evaluation. 126

The grants are evaluated in an ex-ante fashion, mainly using peer review.


90

focuses its funding on universities but also supports a number of Government laboratories

and research facilities, plus institutes maintained by the Research Councils. A high

percentage of its scientists are working at HEIs (Cunningham, Sveinsdottir and Gok, p. 7).

Private investment in research is low in comparison to other nations.127 According to the

Strata- Etan Expert Group (2001, p. 58) the UK innovation system also demonstrates

weaknesses in technological innovation and absorptive capacity.128

The UK is described as a “dual support system” of HEFCE and the Research Councils

(Jongbloed 2009). HEFCE distributes institutional funding separately for teaching and

research. For the research part, it uses a strong evaluation component, the so-called “REF”,

formerly Research Assessment Exercise (Barker 2007). Discipline-oriented Research

Councils distribute competitive grants for all types of research (Jongbloed 2009, p. 35).

The “impact” agenda is a new phenomenon. For a long time, impact was solely understood

as scientific impact, the academic knowledge production which is mainly captured in

publications. Now, in the UK, like in other countries, “[p]ublic research funding is increasingly

understood as a strategic investment where state and economic and regulatory strategies

are oriented towards maximising returns.” (Kearnes and Wienroth 2011: p. 157). The term

was recoined, and “impact” defined more in terms of return-on-investment type has become

a frontpage catchword in the UK. The objective behind the “impact” agenda is to “[foster]

global economic performance, and specifically the economic competitiveness of the United

Kingdom, [increase] the effectiveness of public services and policy, (and) [enhance] quality

of life, health and creative output.”129 Non-scientific “impact” became a main component of

the Government’s research funding allocation strategy (BIS 2010) and is applied to every

funding agency (HEFCE and Research Councils), every type of research and every

discipline.

The British “impact” agenda is defined by the following characteristics:

First, it explicitly asks for evidence of impact and provides a timeline for this proof. In the

current version of the research assessment exercise, HEFCE asks for proof of impact which

has to link a specific 2-, 3- or 4-star publication from the past 20 years to observable “impact”

in case studies. RCUK requires the provision of evidence for impact via databases

(MRCeval/Researchfish, Research Outcomes Systems) for up to five years after a funded

research project is completed. Categories of data collected include for example products and

interventions, spin outs and influence on policy.

The underlying understanding is that impact occurs in a linear chain of events. Impact can

be inferred by backward induction from the research project, via dissemination events, to

decision-makers in politics, business and society. Also, researchers are required to know of

the chain of events that connects their own research to an invention or any other form of

impact130. This worries researchers who believe it implicitly assumes that impact can be

127 For some overview statistics, see the Annex.

128 To tackle this, HEFCE offers funding to encourage university-industry collaboration via HEIF, currently over

600 million Pound in the current 5-year-funding period (PACEC 2012). 129

http://www.rcuk.ac.uk/kei/impacts/Pages/meanbyimpact.aspx 130

The British Research Councils (RCUK) define research impact as “the demonstrable contribution that

excellent research makes to society and the economy.”


91

orchestrated by them, thereby making the researcher responsible whether the result has

been taken up or will be taken up131.

Second, past impact is used as a funding criterion. In the UK, unlike other countries,

research evaluation is an important distributive mechanism in institutional funding (Molas-

Gallart 2012). In the REF, the “impact” section determines 20 percent of the funding.

Research funding agencies do not define “impact” as a funding criterion or as a criterion for

the assessment of a completed project, but reserve the option to use it to inform peers

deciding on a current proposal about past achievements of the researcher.132

Third, the above-mentioned elements of the understanding and use of “impact” are not

limited to applied research, but are equally applied to basic research although

methodological problems remain unsolved133.

“Impact” in Comparative Perspective

The need to demonstrate accountability and transparency in how research money was spent

and the wish to make research funding more effective is shared among many European

countries (ESF 2009, ESF 2012b). The increasing number of impact assessment studies,

which usually have a macro perspective (system, research institutes or research

programmes) shows the interest on part of the commissioning party to learn about the

effects of and nurturing conditions for research (ESF 2012a). Also, facilitated by

technological developments and spurred by interest in evidence-based decision-making,

research funding agencies are increasingly tending to capture research outputs

electronically (ESF 2012c). It varies among countries whether their systems do or do not

include impacts apart from academic publications.

In what follows, I study the research funding policy in general and the role of impact in three

European countries. Those countries are all strong scientific nations and strong innovators

(UNESCO 2010), but represent different profiles of science nations and funding regimes.

According to key innovation indicators (see Table 1 in the Annex), Germany has a

competitive scientific community, but has also a very strong industrial base and a strong

industry component in the funding of research. France, being of comparable population size

to the UK, has a research system with a strong non-university research sector. The

Netherlands represent – like the UK – a country with a very successful base in basic

research regarding the impact of publications and a comparable R&D intensity, which is

focused on public funding of HEI. I discuss each case in turn.

131 For a discussion of the debate, see Penfield, Baker, Scoble and Wykes (2013), p. 3.

132 “Data from ROS will not be used to judge the performance of individual researchers, organisations or

departments. However, it may be used to provide information on track record for the peer review of subsequent

research proposals, it is used to populate publically accessible systems and it is subject to Freedom of

Information legislation.” http://www.epsrc.ac.uk/funding/guidance/managing/ros/Pages/ros.aspx 133

When assessing the impact of basic research, several methodological problems are commonly referred to

(ESF 2012a, for an overview see Penfield, Baker, Scoble and Wykes 2013, p. 8-11): amongst those are timing ,

i.e. the point in time when an impact occurs is unforeseeable and does not necessarily coincide with the point in

time when the impact is evaluated (Morris, Wooding and Grant 2011), attribution, i.e. the proof of a substantial

responsibility for the observed change, which is now agreed to be viewed as “contribution” to change (Levitt et al.

2010, p. 31f., ESF 2012a), and the counterfactual argument (Ramberg and Knell 2012, pp. 9-11).

http://www.epsrc.ac.uk/funding/guidance/managing/ros/Pages/ros.aspx


92

“Impact” and research fund allocation in Germany

With a 5.7 percent share of global publication output, Germany ranks no. 5 in the world (NSF

2012). The country’s particular strength lies in the Natural and Engineering sciences (OST

2010). Investment in research has increased, even during the financial crisis (Aschhoff and

Rammer 2013, p. 14). In Germany, about one third of all investments in science come from

public, about two thirds from private sources, and as an innovation system, it builds on a

strong research system which correlates with good technological performance (Strata-Etan

Expert Group 2002, p. 58).

The German research system is characterised by a division of responsibility (and especially

finances) for research institutions between the federal level (Bund) and the regional level

(16 Länder). The Länder are responsible for funding Higher Education Institutions, which

comprises of app. 100 universities with a double role of teaching and conducting research,

and roughly 180 universities of applied science focusing on teaching (Jongbloed 2009,

p.8f.). They are funded on a lump-sum basis with only about 10 percent (depending on the

“Land”) formula-based funding, dependent on input variables like number of students, and to

a small extent, output variables like publications and awarded third-party funding(Jongbloed,

2009, p. 9). The expectation towards the impact of HEI is mainly to carry out high-quality

research, to educate future employees and also to play a role in the “democratic” society.

However, this demand is not operationalised and no funding criterion.

The Bund has a stronger role in financing non-university research, especially via the major

research organisations including the Max Planck society (MPG), the Helmholtz society

(HGF), the Fraunhofer Institutes (FhG), and the Leibniz Gemeinschaft (WGL) which provides

basic research and service institutes, which are – to different degrees - jointly funded with

the Länder. Since 2005, the “pact for research and innovation” strongly supports non-

university research organisations with an annual growth rate of 5 percent and relaxation of

some regulations, e.g. liberalizing the pay scale for researchers. In return, the government

asks the organisations to report on different policy goals attached to the pact, amongst

others interdisciplinary, gender equality, internationalisation of science and cooperation with

industry. There is no direct link to further funding.

While research evaluations are carried out on a regular basis, they are not used as a

distribution mechanism. Universities and research organisations each have their own

evaluation systems, which are based on the mission of the institute. For example the Max-

Planck-Society, focusing on basic research, is regarded as the stronghold of research

excellence in Germany, so the ex-post evaluation emphasises publication output and global

visibility (Max Planck-Gesellschaft 2010)134. Scientific merit is paramount, non-scientific

“impact” is only a minor criterion in the evaluations and funding decisions are only very

loosely based on them.

Targeted funding for “research excellence” was only recently introduced in Germany. The

“Excellence Initiative” started in 2006 with a total budget of 4.6 billion Euro (2006-2017). It is

a deliberate attempt to introduce a funding mechanism that spurs world-class research at

universities (1994 Group 2011, p. 30), with all the associated benefits like increasing

international visibility and attracting internationally renowned scientists. It also pursues

134 http://www.mpg.de/199400/evaluation2010.pdf

http://www.mpg.de/199400/evaluation2010.pdf


93

structural objectives, for example to diminish the polarisation of the German research system

between university and non-university research, but it has no “impact” requirement.

Competitive funding plays an increasingly important role in German research funding. While

the institutional budgets for HEI have remained stable, the absolute amount of third-party

grants rose constantly. Third-party grants now contribute to more than 25 percent of a

university’s budget (DFG 2012, p. 30). The main funders of competitive grants in Germany

are the Ministry of Research and Education which focuses on applied research and

programme-oriented research and the German Research Foundation (DFG), the main

funder of basic research.

The “High-Tech Strategy 2020” of the Federal Ministry of Education and Research135 starting

in 2006 comprises many measures, for example it relaxed many bureaucratic rules and

provides incentives of starting research-based companies. It targets ten specific economic

and societal challenges by providing programme-oriented funding. As these programmes

focus on tackling societal or economic challenges, the expected socio-economic impact has

to be described and naturally is a prime funding criterion.136

In the application process of the DFG, intended non-scientific impact can but does not need

to be stated. In the guidelines for reviewers, the potential “wider impact” is mentioned as a

“sub-criterion” for quality but not assessed separately137. Past “impact” other than

publications is not used in funding decisions.

The lack of requirement for researchers to follow the outcome of their research beyond the

mere publication does not mean that there is not an increased focus on offering researchers

the opportunity to bring their research closer to application. For example, with a “knowledge

transfer initiative” starting in 2009, the DFG advertised the possibility to apply for funding in

projects that bring together basic research and application partners on equal terms138.

In sum then, “impact” is not omnipresent in the German discourse on science policy. I have

argued that this is due to “division of labour” – the strategic pre-proposal research fund

allocation between basic research, applied research and strategic funding. Germany aims to

tackle its perceived “grand challenges” by distributing funds to mission-oriented research via

program calls or institutional funding, or funds applied research directly. “Impact” does not

need special consideration because it can almost certainly be expected in these cases. In

contrast, basic research receives a “reserved budget” where short-term impact is not

expected.

“Impact” and research fund allocation in France

Being ranked no. 6 in global publication output (NSF 2012), France has a comparatively

strong governmental and non-university research sector (see Table 1 in the Annex). The

biggest of those research organisations are CNRS (the National Centre for Scientific

Research), the National Institute for Health and Medical Research (INSERM), the National

Institute for Agronomic Research (INRA), the National Institute for Computer Science and

135 http://www.bmbf.de/en/14397.php

136 See for example: http://www.bmbf.de/foerderungen/12230.php

137 http://www.dfg.de/formulare/10_20/10_20e.pdf

138 http://www.dfg.de/foerderung/grundlagen_dfg_foerderung/erkenntnistransfer/index.html

http://www.dfg.de/formulare/10_20/10_20e.pdf


94

Automation (INRIA), and the French Alternative Energies and Atomic Energy Commission

(CEA). The French government has prioritised R&D spending and has increased its public

funding. Major reforms in the research system have occurred from 2006 on, with the

National Research and Innovation Strategy in 2009 reinforcing them. It pronounced the

importance of supporting the national economy by research and innovation. It also aims to

better take into account the possibilities of the commercialisation of research outcomes and

knowledge transfer from public research to business (Eparvier, Mallet and Rivoire 2013, p.

3).

Universities in France (about 80 in total) receive a lump sum for higher education by the

national government with some revenue additionally raised by student fees and industry

contributions. A second stream of HEI funding goes to research. Research is carried out in

universities, but mainly in laboratories of non-university research organisations who also

administer the major share of funding for research (“unite mixte de recherché”) (Maassen

2000).

The evaluation culture or reliance on evidence-based decision-making for research funding

was traditionally weak in France, but it has gained much more importance in recent years.

Since 2006, the constitutional bylaw on budget acts (LOLF) required that the research

programmes set up defined a strategy, objectives and indicators and were subsequently

evaluated139. Research organisations and universities are now evaluated on a regular basis

by committees of peers, coordinated by the Agency for the Evaluation of Research and

Higher Education (AERES), which was created as a result of the “Pact for Research”140 in

2006.

In a self-evaluation report which is part of the evaluation procedure, the unit of analysis

(department or institute) has to report on excellence (ex-post), on partners (ex-post) and on

strategy (ex-ante). In recent years, “impact” has been emphasized as an independent

section in the self-evaluation report in the “strategy” chapter with the purpose of

communicating benefits of research to society. The assessment of the self-evaluation

reports is the duty of a committee invited by AERES. The results of the evaluation feed into

future funding decisions insofar as the aggregated evaluation reports are presented to the

Ministry responsible for funding, which uses them (and other deliberations) to discuss the

budget for the next funding period. However, there is no direct link from an evaluation to

funding of the research organisations or institutes.

Next to HEI and research organisations, institutional funding is also directed to clusters and

networks. For example, in 2006, the government provided extensive funding to the newly

created “Carnot Institutes Network”, comprising 33 institutes. They are obliged to develop

research partnerships with companies and to spur technology transfer141.

Aiming to foster scientific excellence especially in basic research, the “excellence initiative”

(IDEX), part of the “Investments for the Future” programme, supports selected universities

139 http://www.performance-

publique.budget.gouv.fr/fileadmin/medias/documents/ressources/guides/guide_pratique_LOLF_oct2008_anglais.

pdf 140

http://www.enseignementsup-recherche.gouv.fr/cid20235/le-pacte-pour-la-recherche.html 141

http://www.instituts-carnot.eu/en

http://www.performance-publique.budget.gouv.fr/fileadmin/medias/documents/ressources/guides/guide_pratique_LOLF_oct2008_anglais.pdf



http://www.enseignementsup-recherche.gouv.fr/cid20235/le-pacte-pour-la-recherche.html

http://www.instituts-carnot.eu/en


95

and research clusters with 7.7 billion Euro to increase their quality and visibility (Eparvier,

Mallet and Rivoire 2013, p. 15).

Research funding on a competitive basis is a very new phenomenon in France and was

introduced as late as 2006 with the creation of the Agence National de la Recherche

(ANR).142 ANR offers programmatic funding to strengthen specific fields that are deemed

strategically important. For example, in the “Investments for the Future Programme” three

thematic priorities (Health, Agriculture and Biotechnology, Environment, Energy and Traffic

and IT and Nanotechnology) were identified that are supported with an extra 7.9 billion Euro

(Eparvier, Mallet and Rivoire 2013, p. 3). Additionally, researchers can apply for non-directed

funding for basic research in the “programme blanc”.

ANR main funding criterion is scientific excellence. In total, five evaluation criteria are

applied, of which one is “global impact”. Only in 2012, ANR sharpened this criterion and

included “promotion of scientific culture and research communication” and “actions to

disseminate research results in higher education”.143

ANR funds one-off projects, and is not gathering evidence of impact of a grant. However,

after completion of a project, a grant-holder must submit a scientific and a financial report.144

In sum, France has recently changed its research system on a grand scale. While still a

small share in financial terms, it now includes an element of competition on the basis of

research grants. Also, new funding opportunities are supposed to foster research

excellence. To achieve non-scientific impact, programme-oriented research plays a big role.

But also in funding basic research, socio-economic and other forms of impact are

increasingly asked for in institute evaluations and research grant proposals. Until now, this

demand is restricted to ex-ante assessments (in grant funding) and the strategy of institutes

(block grants) and funding is not directly linked to it.

“Impact” and research fund allocation in the Netherlands

The Netherlands have, compared to its size, a very successful research system with a high

scientific impact (Deuten 2013, p. 2). Research is mainly concentrated in the Higher

Education Sector, with comparably low contributions (funding and performing) by the

business sector (see Table 1 in Annex) which is attributed to the small number of R&D

intensive firms (Deuten 2013, p. 11). In terms of publication output, the Netherlands ranks

no. 13 in the world (NSF 2012).

In the Netherlands, there are 47 universities of applied sciences and 14 research universities

(Jongbloed 2009, p. 16). The research grant is mainly distributed as a lump sum, with a

small percentage based on quantitative criteria such as amount of contract research

(Jongbloed 2009, p. 17). In addition to HEIs, the Netherlands have some non-university

research institutes. Both are evaluated periodically according to the “Standard Evaluation

Protocol” administered by VSNU (Association of the Universities in the Netherlands), KNAW

142 http://www.agence-nationale-recherche.fr/en/project-based-funding-to-advance-french-research/

143 http://www.agence-nationale-recherche.fr/fileadmin/user_upload/documents/aap/2012/aap-blanc-2012.pdf

144 http://www.agence-nationale-recherche.fr/fileadmin/user_upload/documents/uploaded/2007/reglement-

modalites-attribution-aide.pdf

http://www.agence-nationale-recherche.fr/en/project-based-funding-to-advance-french-research/

http://www.agence-nationale-recherche.fr/fileadmin/user_upload/documents/aap/2012/aap-blanc-2012.pdf

http://www.agence-nationale-recherche.fr/fileadmin/user_upload/documents/uploaded/2007/reglement-modalites-attribution-aide.pdf

http://www.agence-nationale-recherche.fr/fileadmin/user_upload/documents/uploaded/2007/reglement-modalites-attribution-aide.pdf


96

(Royal Academy) and the Research Council (NWO) by international peer review

committees145.

A very strong discourse on impact of science has developed in the Netherlands over the

past 10 years. Research projects like “Evaluating Research in Context” (ERIC)146 or the

collaborative SIAMPI project147 have found their way into science policy and – on practical

terms – in the Standard Evaluation Protocol. While “impact” is part of the self-evaluation

report, it is explicitly only used to learn about effects of research, not to judge it (Spaapen

and van Drooge 2011). The results of the evaluations are used for improvement and

accountability, but do not influence funding directly.

On top of their basic funding, universities can – and are supposed to – apply for additional

research grants from research councils or the ministry (Maassen 2000). Competitive

Funding is distributed by the NWO and represents about 10 percent of the funding streams

of universities (Jongbloed 2009, p. 16).

The NWO funds applied as well as blue-sky research and offers different programmes

accordingly. In 2011, the Dutch government identified nine “top sectors” (thematic fields)

which are deemed especially worthy of funding (Deuten 2013, p. 14). The top sector

programme especially, but not exclusively, targets university-industry cooperation and has

established clusters to achieve this end. 50 percent of NWO funding has to go to the “top

sector” fields, and all projects, funded under targeted programmes, programmes for applied

or for basic research, which can be assigned to any of the areas, count towards that goal.

The highest funding criterion is research excellence, but the expected non-scientific impact

is considered in the forward-looking evaluation of project proposals, with varying degrees of

importance according to the character of the programme (applied or basic). In many cases,

the requirement that applicants elaborate on “knowledge transfer” is meant to raise

awareness, but knowledge transfer is not meant as a demand in itself. Similarly, to

strengthen the awareness towards wider dissemination, knowledge utilisation has to be

reported in all after-grant final reports. 148

To conclude, the Netherlands focus on socio-economic impact of science for about a decade

already. This might be to counteract the “natural” weakness of the innovation system in

terms of research performed or funded by the business sector. Relying heavily on logistics,

the economy of the Netherlands is not very prone to rely on research. Additionally, at the

turn of the century, large multinational companies (Philips, Shell, Unilever) relocated their

research divisions to other countries and the research investment of the Dutch business

sector dropped from 1.07 percent of GDP in 2000 to 0.89 percent in 2010 (OECD 2012). The

large-scale “top sector” programme is an intense attempt by the government to bring

industry and HEIs (back) together, focusing on strategically important fields. Other than that,

the link between the assessment of “impact” and funding is indirect.

145 http://www.knaw.nl/smartsite.dws?id=26104&pub=20091052

146 http://www.rathenau.nl/en/themes/theme/project/eric-evaluating-research-in-context.html

147 http://www.siampi.eu/Pages/SIA/12/642.bGFuZz1FTkc.html

148 “The funding instruments cover the entire spectrum of fundamental and applied research. Knowledge

utilisation (societal and scientific applicability of the results) is increasingly a criterion in the assessment of

funding instruments”. http://www.nwo.nl/en/funding/research+funding

http://www.knaw.nl/smartsite.dws?id=26104&pub=20091052

http://www.rathenau.nl/en/themes/theme/project/eric-evaluating-research-in-context.html

http://www.siampi.eu/Pages/SIA/12/642.bGFuZz1FTkc.html

http://www.nwo.nl/en/funding/research+funding


97

Discussion and conclusion

The Inuit are said to distinguish 50 words for snow. In contrast, European research agencies

talk about impact, impact and impact, but they all mean different concepts, attach different

importance to it and implement it in different ways.

Both the Netherlands and France use “impact” as a reference for research and all countries

studied here clearly intend to improve the degree to which funded research stimulates

economic development and technological change. Virtually all research funding agencies

invest time and money to stimulate cooperation between researchers and private companies

and other “users” of research output. All these countries also require the discussion of

potential impact in research proposals.

The United Kingdom goes further, however, in asking all research fund application writers to

discuss the potential impact of their research, in requiring that all finished research projects

report on research impact in a central data base (which may or may not inform funding

decisions in the future) and in imposing a short-term perspective on impact. Most

importantly, by incorporating “impact” into the research assessment and therefore linking

large parts of funding – with no discrimination between types of research – to the evidence

of impact, UK research policy sets an incentive to carrying out research that is likely to have

a demonstrable impact as applied research does (or to inflate supposed effects).

From a government’s perspective this is very rational: old and mature research ideas diffuse,

thereby driving a wedge between the location in which the basic research idea was

developed and the location in which the marketable product is invented. Indeed, the laser

technology was not invented in Switzerland, the country in which Einstein formulated his

theory of light quanta. Thus, investment in basic research, while potentially having an

important impact on global welfare, is unlikely to improve the relative economic position of

the country that funds it (Stokes 2000). In contrast, by funding applied research the

government and funding agencies can directly invite researchers to tackle apparent social,

economic and political problems, which must seem much more appealing to politicians.149

If the approaches to “impact” are seen on a continuum, the UK places itself on one far side

of the scale. It remains to be seen whether other countries follow suit. The assumption that

they will would be supported by the fact that “impact” does play an increasing role in other

European countries’ research policy discourse and has been included in funding allocation

procedures, though on a small scale. Also, the systematic collection of research outputs as

well as evaluation and impact assessment studies are on the rise.

On the other side, some reasons make it unlikely that other countries are going to take a

similar approach. While other countries do indeed allocate earmarked funding to applied

research or to areas which are perceived to be socially or economically important, often on a

larger scale than the UK, they reserve a share of the total research fund to basic research

149 For example, the EPSRC specifies: “The purpose of National Importance is to encourage applicants to

articulate why it’s important for their research to be supported by the UK taxpayer so that the UK remains

internationally competitive” (Highlights by EPSRC).

http://www.epsrc.ac.uk/funding/guidance/preparing/Pages/economicimpact.aspx

http://www.epsrc.ac.uk/funding/guidance/preparing/Pages/economicimpact.aspx


98

with no “impact requirement”. If used at all in competitive funding, “impact” is almost

exclusively used in the ex-ante assessment or proposals. Most importantly, due to a different

funding allocation system which mainly relies on block grants, institutional funding is not

directly linked to evidence of past “impact” in other countries. In evaluation processes,

“impact” is mainly used for strategy or learning purposes with a much vaguer definition of

“impact” and a longer (and mostly not specified) time horizon.

One of the reasons why governments handle “impact” differently is that they pursue different

policy agendas. For example, Germany and France aim to attract more international

researchers and improve their countries’ research excellence and visibility – something

which the UK is arguably not in need of. On the other side, the UK lacks a strong industrial

research sector and an industrial base which provides a “natural” absorption capacity of

research results. It remains to be seen whether the “impact” agenda, which effectively shifts

all – including basic research – to application, can resolve this situation or whether it

endangers the one sector in which the UK performs particularly well – higher education and

basic research.

Different countries use different pathways to realise wider benefits of research. The

continued existence of diverse research policy provides a de facto experiment on the impact

of the “impact” agenda. Future will tell whether the impact agenda affected basic research in

the way and to the extent worried by many researchers or whether the UK’s research policy

will re-vitalise the British society and economy as hoped for by the government.

References

Aschhoff, Birgit and Christian Rammer (2013): Erawatch Country Reports 2011: Germany.

JRC Scientific and Policy Reports. Luxembourg.

http://erawatch.jrc.ec.europa.eu/erawatch/export/sites/default/galleries/generic_files/file_033

3.pdf

Barker, Katharine (2007): The UK Research Assessment Exercise: the evolution of a

national research evaluation system. In: Research Evaluation, 16,1, pp. 3-12.

BIS Department for Business Innovation & Skills (2010): The allocation of science and

research funding 2011/12 to 2014/15. London.

Bozeman, Barry (2000): Technology transfer and public policy: a review of research and

theory. In: Research Policy 29, pp. 627–655.

Bush, Vannevar (1945): Science. The Endless Frontier. A Report to the President by

Vannevar Bush, Director of the Office of Scientific Research and Development, United

States Government Printing Office, Washington.

Cunningham, Paul, Thordis Sveinsdottir and Abdullah Gok (2013): Erawatch Country

Reports 2011: United Kingdom. JRC Scientific and Policy Reports. Luxembourg.


8.pdf

Deuten, Jasper (2013): Erawatch Country Reports 2011: The Netherlands. JRC Scientific

and Policy Reports. Luxembourg.

http://erawatch.jrc.ec.europa.eu/erawatch/export/sites/default/galleries/generic_files/file_0333.pdf





99


9.pdf

DFG – Deutsche Forschungsgemeinschaft (2012): Förderatlas. Kennzahlen zur öffentlich

geförderten Forschung in Deutschland. Bonn.

EFI – Expertenkommission Forschung und Innovation (2013): Gutachten zu Forschung,

Innovation und Technologischer Leistungsfähigkeit Deutschlands. Berlin.

Eparvier, Patrick, Olivier Mallet and Léonor Rivoire (2013): Erawatch Country Reports 2011:

France. JRC Scientific and Policy Reports. Luxembourg.


6.pdf

European Science Foundation (2009): Evaluation in National Research Funding Agencies:

approaches, experiences and case studies – A report of the ESF Member Organisation

Forum on Ex-Post Evaluation of Funding Schemes and Research Programmes. Strasbourg.

European Science Foundation (2012a): The Challenges of Impact Assessment – A report of

the ESF Member Organisation Forum on Evaluation of Publicly Funded Research.

Strasbourg.

European Science Foundation (2012b): Evaluation in Research and Research Funding

Organisations: European Practices. A report of the ESF Member Organisation Forum on

Evaluation of Publicly Funded Research. Strasbourg.

European Science Foundation (2012c): Research Funders and research output collection. A

report of the ESF Member Organisation Forum on Evaluation of Publicly Funded Research.

Strasbourg.

Jongbloed, Ben (2009): Higher Education Funding Systems: An overview covering five

European Jurisdictions and the Canadian province of Ontario. Report prepared for the Hong

Kong University Grants Committee.

Kearnes, Matthew and Matthias Wienroth (2011): Tools of the Trade: UK Research

Intermediaries and the Politics of Impacts. In: Minerva. 49, 2, pp. 153-174.

Kjelstrup, Signe (2001): „Basic and applied research in the university – have they changed?

OECD Workshop on basic research: policy relevant definitions and measurement. Oslo, Oct.

29-30, 2001. http://www.oecd.org/science/sci-tech/2674543.pdf

Levitt, Ruth, Claire Celia, Stephanie Diepeveen, Siobhán Ní Chonaill, Lila Rabinovich and

Jan Tiessen (2010): Assessing the impact of arts and humanities research at the University

of Cambridge. RAND Europe.

Maassen, Peter (2000): Models of Financing Higher Education in Europe. Background

Report to the Nordic Council of Ministers.

Max Planck Gesellschaft (2010): Evaluation. The procedures of the Max-Planck-Society.

München.





http://link.springer.com/journal/11024

http://www.oecd.org/science/sci-tech/2674543.pdf


100

Molas-Gallart, Jordi (2012): Research Governance and the Role of Evaluation: A

Comparative Study. In: American Journal of Evaluation, 33,4 , pp. 583-598.

Morris, Zoe Slote, Wooding, Steven and Jonathan Grant (2011): The Answer Is 17 Years,

What Is the Question Understanding Time Lags in Translational Research. In: Journal of the

Royal Society of Medicine, 104,12, pp. 510-520.

NSF - National Science Foundation (2012): Science and Engineering Indicators 2012.

http://www.nsf.gov/statistics/seind12/c5/c5s4.htm#s1

OECD (2002): Frascati Manual, Sixth edition. Paris. http://www.oecd-ilibrary.org/science-

and-technology/frascati-manual-2002_9789264199040-en.

OECD (2012): Main Science and Technology Indicators: Volume 2011/2012. Paris

OST – Observatoire des Sciences et de Techniques (2010): Indicateurs de sciences et de

technologies. Paris.

PACEC – Public and Corporate Economic Consultants (2012) : Strengthening the

Contribution of English Higher Education Institutions to the Innovation System : Knowledge

Exchange and HEIF funding. A report for HEFCE. Cambridge.

https://secure.pacec.co.uk/documents/HEIF11-15-FullReport.pdf

Penfield, Teresa, Matt Baker, Rosa Scoble and Michael Wykes (2013): Assessment,

Evaluations and Definitions of Impact: A Review. Unpublished Working Paper. Exeter.

Ramberg, Inge and Mark Knell (2012): Challenges measuring effects of research and

innovation policy interventions in ex-post impact evaluations. A synthesis report. Oslo.

Spaapen, Jack and Leonie van Drooge (2011): Introducing ‘productive interactions’ in social

impact assessment. In: Research Evaluation 20, 3, pp. 211-218.

Stokes, Donald (2000): Completing the Bush Model: Pasteur’s Quadrant. Conference

Proceedings: Science the Endless Frontier 1945-1995. Learning from the Past, Designing

for the Future http://www.cspo.org/products/conferences/bush/Stokes.pdf

STRATA-ETAN Expert Group (2002): Benchmarking national research policies: The Impact

of RTD on Competitiveness and Employment (IRCE), European Commission, Brussels.

Toulmin, Stephen (1964): “The Complexity of Scientific Choice: A Stocktaking,” Minerva 2,

pp. 343-359.

UNESCO (2010): UNESCO Science Report 2010. Paris.

Weinberg, Alvin M. (1964): “Criteria for Scientific Choice II: The Two Cultures,” Minerva 3,

pp. 3-14.

Zinth, Wolfgang, Alfred Laubereau and Wolfgang Kaiser (2011): The long journey to the

laser and its rapid development after 1960. In: European Physical Journal H. 36,2 , pp. 153-

181.

http://www.nsf.gov/statistics/seind12/c5/c5s4.htm#s1

http://www.oecd-ilibrary.org/science-and-technology/frascati-manual-2002_9789264199040-en

http://www.oecd-ilibrary.org/science-and-technology/frascati-manual-2002_9789264199040-en

http://rev.oxfordjournals.org/search?author1=Jack+Spaapen&sortspec=date&submit=Submit

http://rev.oxfordjournals.org/search?author1=Leonie+van+Drooge&sortspec=date&submit=Submit

http://www.cspo.org/products/conferences/bush/Stokes.pdf


101

1994 Group (2011): Mapping Research Excellence: exploring the links between research

excellence and research funding policy.

http://info.scival.com/UserFiles/mapping_research_excellence.pdf

Annex

Table 1. Key science indicators

UK German

y

France Netherlan

ds

Gross domestic expenditure on R&D

(GERD) as a percentage of GDP 2010 (1)

1.77% 2.82% 2.26% 1.83%

Gross domestic expenditure on R&D

financed by industry 2010 (1)

45.1% 66.1% 51.0% 45.1%

Increase of Gross domestic expenditure on

R&D between 2005-2009 (1, own

calculations)

18.2% 28.6% 22.2% 12.5%

Higher Education Researchers as

percentage of national total 2009

(1)

61.7% 26.5% 30.1%

(2008)

41.9%

Percentage of Gross domestic expenditure

performed by the business enterprise

sector 2010 (1)

60.9% 67.3% 61.2% 47.3%


performed by the higher education sector

2010 (1)

27.2% 18.0% 21.3% 40.8%


performed by the government sector 2010

(1)

9.4% 14.7% 16.4% 11.9%

World share in scientific publications 2008

(2)

5.7% 5.7% 4.2% 1.7%

Scientific Publications: H-index after 2

years (2008) compared to World average

(1.00) (2)

1.25 1.20 1.01 1.33

Sources:

(1) OECD Main Science and Technology Indicators: Volume 2011/2012

(2) OST 2010

http://info.scival.com/UserFiles/mapping_research_excellence.pdf


102

Acknowledgements

We are grateful to the help of a number of colleagues and subject experts for reviewing and

commenting on this document. They are: David Baker, Rosa Fernandes, Philip Graham,

Kimberley Hackett, Neil Jacobs, Keith Jeffrey, Julia Lane, Paul Manners, Geoff

Rodgers, Rosa Scoble, Sue Smart and Philip Ternouth. Thanks are also due to Karin James

for managing the production of this report so efficiently.

Contact

Dr Michael Wykes

Policy, Impact, and Performance Manager

Research and Knowledge Transfer

University of Exeter

Innovation Centre Phase 2

Rennes Drive

Exeter, EX4 4RN

UK

[email protected]

+44 (0)1392 722351
