methodologies for measuring research impact for a new interdisciplinary institute
DESCRIPTION
Methodologies for measuring research impact for a new interdisciplinary institute. Janet Baker and Dr Nina Prasolova Queensland University of Technology Library. Complicated subject and so best told as a story. The request The task – impact report for a “virtual” institute The challenge - PowerPoint PPT PresentationTRANSCRIPT
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
L I B R A R Y S E R V I C E S
www.library.qut.edu.au
D I V I S I O N O F T E C H N O L O G Y , I N F O R M A T I O N A N D L E A R N I N G S U P P O R TD I V I S I O N O F T E C H N O L O G Y , I N F O R M A T I O N A N D L E A R N I N G S U P P O R T
Methodologies for measuring research impact for a new interdisciplinary institute
Janet Baker and Dr Nina PrasolovaQueensland University of Technology Library
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Complicated subject and so best told as a story
The request The task – impact report for a “virtual” institute The challenge The negotiations The methodologies The workflow The lessons we learned
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Characteristics of Institutes or Research Groups
Established institutes
Clear purpose and identity Set agenda, coherent workflow Membership list Physical space with known teams High quality resources already available to measure
research impact Benchmarking progressive and peer group known
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Institute for Future Environments
“Virtual” institute
Based around broad concepts, a Big Idea or evolving research themes with loosely affiliated individual researchers
“Fluid” relationships Disparate output types and citing cultures to interpret
www.library.qut.edu.au
L I B R A R Y S E R V I C E SInterdisciplinarity: a field of study that crosses traditional boundaries between academic disciplines, which integrates these different approaches or methods (Center for the Study of Interdisciplinarity).
Boston Interdisciplinary Instituteshttp://www.bu.edu/cgs/citl/interdisciplinary-institutes/
Literature review reveals increasing number of reports and interest in a growing body of literature around issues, policies, challenges regarding IDR
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Challenges
No clear boundaries or disciplinary structure Loosely affiliated clusters Defining Interdisciplinarity - subject categories or
research codes Multiple variables – type of publication, where indexed,
differing culture of citation, collaborative arrangements Expectations – make clear what is possible
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Expectations/ Negotiations: What is the goal?To evaluate performance – factors to consider Look at the traditional metrics – what are you trying to illustrate, what
can you use - Which indicators are consistent for group What tools will give greatest coverage/accuracy Is it possible to analyse cited references - interdisciplinarity Journal subject categories? – WoS and Scopus Percentiles and Average Citation Metrics Percentage share of total publications
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Expectations - What is the goal?To predict and for strategic planning Co-authorship graphs – analysis of effective collaborations Co-citation analysis (SciVal) identifying emerging trends Identify Networks outside the discipline boundaries
To improve communications and reporting efficiency Find other research support staff also collecting and
evaluating impact? Tacit knowledge (people evaluating are not the researchers) Knitting together support around the researcher
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
What is the reality?
Each subgroup is different – refine the theme and vocabulary
The type of research output dictates the assessment tools
Which metrics are available across the theme Benchmarking strategy is specific for the
subgroup/cluster
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Identifying clusters – establish borders
Field of Research Codes MeSH or other Thesauri Journal CategoriesKeyword analysis
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Street computing : towards an integrated open data Application Programming Interface (API) for cities
http://eprints.qut.edu.au/59160/
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Analysis of productivity and impact
Percentage of the outputs in the theme Distribution of citations (average and percentile) Collaborations Interdisciplinarity Downloads Attention Metrics Media releases
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Metrics of Research Productivity and Impact
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
ESI Table
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Search Scopus – using identified research themes then work out citation percentiles
Map QUT output to the total output and citation percentiles
Export and analyse patterns
Assess Collaborations
Assess interdisciplinarity
Workflow
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Scopus benchmarking (share and percentiles of ‘cited by’)“Tropical Crop*” (All fields) (limit to Country: Australia)
QUT share in the total pubs: 33 (36%) out of 91
3 QUT papers in 10th percentile and9 QUT papers in 25th percentile
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Author collaborations
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Analysing collaborations in SciVal
SciVal (Elsevier) January, 2014
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Summary: what we have learnt
For “virtual” institutes it is important to identify clusters Baselines – sometimes need to change, cultures differ Google Scholar Citations Profile – appears most useful for
some clusters, but data needs to be checked Possible to manually granulate by retrieving those
publications that are in the theme or cluster and then benchmark your institute
Final report needs detailed explanation of metrics and tools chosen and their applicability and replicability
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
What we learned
There is a place for librarians in this exercise
For emerging new institutes librarians could be instrumental providing input into shaping the research clusters and showing future opportunities in interdisciplinarity and collaborations
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
References
Huutoniemi, K., Klein, J. T., Bruun, H., & Hukkinen, J. (2010). Analyzing interdisciplinarity: Typology and indicators. Research Policy, 39(1), 79-88. doi: http://dx.doi.org/10.1016/j.respol.2009.09.011
Roessner, D., Porter, A. L., Nersessian, N. J., & Carley, S. (2013). Validating indicators of interdisciplinarity: linking bibliometric measures to studies of engineering research labs. Scientometrics, 94(2), 439-468.
Wagner, C. S., Roessner, J. D., Bobb, K., Klein, J. T., Boyack, K. W., Keyton, J., . . . Börner, K. (2011). Approaches to understanding and measuring interdisciplinary scientific research (IDR): A review of the literature. Journal of Informetrics, 5(1), 14-26.
Rymer, L. (2011). Go8 Backgrounder 23 | Measuring the impact of research – The context for metric development. Turner, ACT: The Group of Eight. Retrieved from
Center for the Study of Interdisciplinarity. (2013). Retrieved from http://csid.unt.edu/research/ria_overview.html
LSE Public Policy Group. (2011). Maximizing the impacts of your research: A handbook for social scientists (Consultation Draft 3). Retrieved from http://www.lse.ac.uk/government/research/resgroups/LSEPublicPolicy/Docs/LSE_Impact_Handbook_April_2011.pdf
www.library.qut.edu.au
L I B R A R Y S E R V I C E S
Creative Solutions
Valerie Ucumari(CC BY-NC-ND 2.0)