evaluating the big deal: usage statistics for decision making
DESCRIPTION
Presentation delivered at the UKSG Usage Statistics for Decision Making workshop. Held at the Institute of Materials, Minerals and Mining, London. 2nd Febrary 2012.TRANSCRIPT
![Page 1: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/1.jpg)
Evaluating the Big Deal: What metrics matter?
Usage Statistics for Decision MakingLondon, 2 February 2012
Selena KillickLibrary Quality Officer
![Page 2: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/2.jpg)
Introduction
• Institutional, financial and strategic context
• Evaluating the Big Deals
• Quantitative Reporting
• Qualitative Measures
• Using the Results
• Conclusions
![Page 3: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/3.jpg)
Cranfield University
• The UK's only wholly postgraduate university focused
on science, technology, engineering and
management
• One of the UK's top five research intensive
universities
• Annual turnover £150m
• 40% of our students study whilst in employment
• We deliver the UK Ministry of Defence's largest
educational contract
![Page 4: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/4.jpg)
Key Drivers
• Financial realities
• Demonstrating value for money
• Strategic alignment
• Research Excellence Framework (REF)
• Income
• ReputationMissioncritical
![Page 5: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/5.jpg)
Expenditure on Journals
2006-07 2007-08 2008-09 2009-10
Journal Spend
![Page 6: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/6.jpg)
Information Expenditure by Format 2010-11
Books4%
eBooks4%
Journals68%
Databases24%
![Page 7: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/7.jpg)
Information Expenditure by Format 2010-11
Books4%
eBooks4%
Journals31%
Big Deals37%
Databases24%
![Page 8: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/8.jpg)
Evaluating the‘Big Deals’
![Page 9: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/9.jpg)
Previous Techniques Used:
Annual journals review using the follow data
• Circulation figures – issues and renewals
• “Sweep survey” to capture in-house use
• Journal contents page requests
• Download figures
• Journal prices v the cost of ILL requests
More recent focus on “cost per download”
![Page 10: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/10.jpg)
New Approach
Quantitative:• Size
• Usage
• Coverage
• Value for Money
Qualitative:• Academic Liaison
• Reading Lists Review
• REF Preferred
![Page 11: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/11.jpg)
Requirements
• Systematic
• Sustainable
• Internal benchmarking
• Elevator pitch
• So what?
• Enable informed decision making
• Demonstrate smart procurement
![Page 12: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/12.jpg)
Quantitative Reporting
![Page 13: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/13.jpg)
Brought to you by the letters…
&
![Page 14: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/14.jpg)
Our Approach
• What has everyone else done?
• Analysing Publisher Deals Project
• Storage centre
• Excel training
• Template design
![Page 15: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/15.jpg)
Basic Metrics
• Number of titles within a package
• Total annual full-text downloads
• Cost:
• Core titles
• e-Access Fee
• Total costs
![Page 16: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/16.jpg)
Value Metrics
• Average number of requests per title
• Average cost per title
• Total cost as % of information provision expenditure
• Cost per full-text download
• Average download per FTE student/staff/total
• Average cost per FTE student/staff/total
![Page 17: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/17.jpg)
The Long Tail
Do
wn
loa
ds
Titles Titles Titles
Long Tail Short Tail No Tail
![Page 18: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/18.jpg)
Subscribed Titles
• Reviewing performance of core collection
• REF Preferred?
• Popular?
• Three year trends in cost / downloads / CPD
• Cost / Downloads / CPD categorised:• Zero
• Low
• Medium
• High
• Cancel?
![Page 19: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/19.jpg)
Popular Titles
• Which titles are the most popular?
• Top 30 titles in the package
• Three year trends in downloads
• REF Preferred?
• Subscribed title?
![Page 20: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/20.jpg)
Considerations
• When to measure from/to?• calendar, financial/academic, or contract year?
• Which titles make up our core collection?
• Do we have access to all of the „zero use‟ titles?
• What constitutes Low/Medium/High?
• What about the aggregator usage statistics?
• Do we trust the usage statistics?
• What is the size of the target population?
![Page 21: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/21.jpg)
Capturing the data
![Page 22: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/22.jpg)
Downloading Statistics
• Get organised
• Gather your usernames and passwords
• Create local files to save and store usage reports
• Software now on the market to manage this for you
• Joint Usage Statistics Portal
![Page 23: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/23.jpg)
![Page 24: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/24.jpg)
Introductory
Workshop
April 18
Birmingham
![Page 25: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/25.jpg)
Reporting Period
• Calendar, financial/academic, or contract year?
• COUNTER Reports = Calendar year
• Converted using Vlookup on ISSNs
• Manual
• Problematic
• Automatically converted using the JUSP
![Page 26: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/26.jpg)
Aggregator Usage Statistics
• Combining usage from publishers and aggregators at a
title level
• Combined using Pivot Tables
• Manual
• Problematic
• Where possible combined using JUSP
![Page 27: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/27.jpg)
Analysing the data
![Page 28: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/28.jpg)
Excel Template
• Two main data sources:
• COUNTER JR1
• Subscription agent financial report
• Automated as much as possible
• Match formulas working with ISSNs to link title price to
usage/holdings
• All calculations are completed automatically when the
data sources are added
![Page 29: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/29.jpg)
Quantitative Reporting
• Systematic
• Sustainable
• Internal benchmarking
• Elevator pitch
• So what?
• Enable informed decision making
• Demonstrate smart procurement
![Page 30: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/30.jpg)
Qualitative Measures
![Page 31: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/31.jpg)
Academic Liaison
• Who‟s using it?
• Why?
• How?
• What will be the impact if we cancel?
• Teaching?
• Research?
• How valuable is it?
![Page 32: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/32.jpg)
Quantitative on the Qualitative:
Analysis on the five REF Preferred Recommended
Journals Lists:
• Overlapping titles
• Unsubscribed titles
• Financial shortfall
• Current recommended subscribed titles
• Usage data
![Page 33: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/33.jpg)
Reading List Review
Qualitative analysis on course reading lists:
• What are our academic recommending?
• Where is it published?
• How often is it recommended?
• Are there alternatives?
![Page 34: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/34.jpg)
Quantitative & Qualitative Reporting
• Systematic
• Sustainable
• Internal benchmarking
• Elevator pitch
• So what?
• Enable informed decision making
• Demonstrate smart procurement
![Page 35: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/35.jpg)
Using the results
![Page 36: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/36.jpg)
What they can do:
• Both qualitative and quantitative measures tell the
story of the resource
• Aid decision making
• Justify procurement
• Safeguard budgets…?
![Page 37: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/37.jpg)
What they can’t do:
![Page 38: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/38.jpg)
Conclusions
![Page 39: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/39.jpg)
Closing thoughts
• Is it worth investing in this?
• Qualitative & Quantitative
• Danger of relying on cost-per-download
![Page 40: Evaluating the Big Deal: Usage Statistics for Decision Making](https://reader034.vdocuments.mx/reader034/viewer/2022051817/547d4561b37959492b8b51fc/html5/thumbnails/40.jpg)
Looking Ahead
• Review of all budgets
• All Resources
• Systems
• Staff
• Services
• Demonstrating Value and Impact
• Resources
• Services