Download - Understanding impact through alternative metrics: developing library-based assessment services
Kristi Holmes, PhD 2AM - Amsterdam 7 October 2015
Understanding impact through alternative metrics Library-based assessment services
Northwestern University Clinical and Translational Sciences (NUCATS) Institute
Mission: Speeding transformative research discoveries to patients
and the community
http://nucats.northwestern.edu/
What is Translational Science?
Translation form “Bench to Bedside”
• Circa 2000: 18 years from discovery to dissemination... - for successful therapies
• Circa 2015: 7 years... - and we are smarter about what should go
through the pipeline • And we understand that translation has to go beyond the bedside
CTSAs – Mission and Structure
10/7/15 5
Emphasis on local/institutional support for
Preclinical developmental therapeutics Enhanced clinical trial infrastructure Community-engaged research
Incorporated aspects of training and career development of translational scientists (K/T awards)
http://www.academicdepartments.musc.edu_sebin_b_o_Translational_Science.jpg
Translational Medicine
Northwestern University Clinical and Translational Sciences (NUCATS) Institute Evaluation and Continuous Improvement Program
Mission: Speeding transformative research discoveries to patients and the community
http://nucats.northwestern.edu/
Some sample metrics…
OUTPUT Metrics
Time to Publication or other output
Number of technology transfer products
ROI of pilot awards
ROI of TL1, T32, KL2 scholars
Time from publication to research synthesis
IMPACT Metrics
Influence of a research output
Researcher and institutional collaborations
Career development and path/trajectory
PROCESS Metrics
Time from IRB submission to approval
Time from grant award to start of study
Volume of investigators who use services, take training, and other activities
Principles to guide ECI
• Open and collaborative processes • Good data (quality and quantity) • Look for opportunities for data reuse • Limited human interaction with data whenever possible
• Leverage common workflows, look for opportunities
Desperately seeking data
What do we want? a better perspective of both productivity and impact
• Multiple sources covering multiple subjects and disciplines; authoritative sources
• Disambiguated • De-duplicated
1. quality metadata 2. a perspective of attention 3. a way to tie these outputs to institutional efforts, systems,
and workflows in a meaningful way grants and to our IR
9
Papers published by NU-affiliated authors 1950-present (as indexed in Scopus)
10
Lots of missing data here: • Citations to these works • Valuable bibliographic metadata (collaborators, journal
titles, affiliations, keywords and indexing, etc.) • Non-NU affiliated papers, but written by NU people (e.g.,
before they arrived) • Reliable links to funding • Open access status? Compliance status? • Attention metrics • Only traditional outputs are accounted for…so several
other outputs are missed! Low hanging fruit – already addressing this with the Galter Metric and Impact Core and other key initiatives.
What is research impact?
A Definition of Research Impact Research impact is the demonstrable contribution that research makes to the economy, society, culture, national security, public policy or services, health, the environment, or quality of life, beyond contributions to academia.
Academic impact The demonstrable contribution that excellent research makes to academic advances, across and within disciplines, including significant advances in understanding, methods, theory and application.
Economic and societal impacts The demonstrable contribution that excellent research makes to society and the economy. Economic and societal impacts embrace all the extremely diverse ways in which research-related knowledge and skills benefit individuals, organizations and nations
Research Impact Principles and Framework, Australian Research Council http://www.arc.gov.au/general/impact.htm
Pathways to Impacts, Research Councils UK http://www.rcuk.ac.uk/ke/impacts/
Context is everything!
• Measurement instruments • Con.nuing educa.on materials • Quality measure guidelines • Cost-‐effec.ve interven.on • Consensus development conferences • American Medical Associa.on Current
Procedural Terminology (CPT) codes • Change in delivery of healthcare services • Gray literature
Going beyond the counts to find evidence of meaningful impact
• New experimental methods, data models, databases, or soFware tools
• New diagnos.c criteria • New standards of care • Biological materials or animal models • Consent documents • Clinical/prac.ce guidelines • Quality measure guidelines
https://becker.wustl.edu/impact-assessment http://nucats.northwestern.edu/
Diverse outputs Diverse impacts
Diverse roles
Each a critical component of the research process
Building an ecosystem
Library as Partner
Opportunity!
Why a library--based support model?
• Are a trusted, neutral space • Have a tradition of service and support • Strive to serve all missions of the institution • Are technology centers and have IT and data expertise
• Have skills—information organization, instruction, usability, subject expertise, ontologies and controlled vocabularies
• Have close relationships with their clients (buy in) • Understand user needs • Understand the importance of collaboration and know
how to bring people together • Have knowledge of institution, research, education,
clinical landscape
Library Staff:
Libraries:
Galter Library’s Metrics and Impact Core
The Metrics and Impact Core (MIC), housed in Galter Library, has expertise in bibliometrics, data visualization, continuous improvement, information systems and alternative metrics. The core provides extensive advisory services for researchers, groups or departments on topics such as:
• developing successful publishing strategies • managing or tracking publications • maintaining an impactful online identity • measuring or assessing research impact by discipline • communicating research impact to audiences
MIC uses a wide collection of resources, including Scopus, Web of Science, Essential Science Indicators, Google Scholar, NU Scholars, Journal Citation Reports, and more, to provide services and reports for: - Researchers or clinicians to demonstrate impact of published works to promotion or tenure committees, or
the impact of research studies to funding agencies when applying for funding - Research groups/institutions/departments to discover how research findings are being used to promote
science, or an overall view of research publications and outputs by a specific group
So why did you name it …CORE?
NUCATS Evaluation Metrics and Approach
19
Workflows, dashboards, system integrations, feedback loops
20
7 steps for success
1. Identify stakeholders, audience, and champions 2. Pay attention to local and national/global issues 3. What’s missing? What’s needed? 4. What can you do today? Next week? In a year? 5. What do you have in hand? What do you need? 6. Communicate – develop a communication plan to
encourage engagement during development and launch
7. Don’t wait until you (fill in the blank)! You can do something terrific NOW!
…and CONNECT!
• [email protected] - Librarians and information scientists doing assessment and
visualizations - Very practical group – sharing ideas, strategies, etc.
• Mosaic: Be Part of the Big Picture joint meeting – to be held May 13–18, 2016, in Toronto, ON. - Medical Library Association (MLA) - Canadian Health Libraries Association/Association des
bibliothéques de la santé du Canada (CHLA/ABSC)
- International Clinical Librarian Conference (ICLC)
Thank you! Kristi Holmes @kristiholmes [email protected] orcid.org/0000-0001-8420-5254