levine-clark, michael, jane burke, and henning schönenberger, “assessing the value and impact of...

ASSESSING THE VALUE AND IMPACT OF DISCOVERY SYSTEMS

Library PerspectiveMichael Levine-ClarkUniversity of Denver

Why do libraries implement a discovery system?

Discovery System Goals• To improve the user experience• To provide a Google-like experience• One-stop shopping - too many resources, too hard to keep them straight

o for undergrads or beginners / for everyoneo for particular disciplines / for every discipline / for cross-discipline needso primarily for articles / for articles and books / for all library resources

• One starting point (when you don’t know where to search)• To replace the catalog• To reduce the number of individual A&I databases (to reduce costs?)• To increase the number of users starting with the library / Compete w/

Google

Components of ValueUsage - (searches or full text downloads)

• Perfect example of variation in stakeholder perspectives

• Publishers want their usage to go up• Librarians want users to find relevant content

efficientlyo could lead to decreased usageo in tension with usage-based resource evaluation (i.e. ⬆

CPU)• Discovery vendors need to respond to both of those

needsJSTOR as example

Components of Value

• Effectiveness – accuracy• Efficiency – speed• Comprehensiveness• Integration with other library tools• Reaching users wherever they might be• Vary across stakeholders • Others?

Components of ImpactUsage

• Goes up or down overall• Proportion coming through library (vs Google)• Proportion coming thru discovery system vs other

library sources• Publisher-hosted vs Aggregator-hosted content • Proportion in different library resource formats

o Print books vs ebooks vs articleso Increased usage of previously siloed content?

e.g. Digital library, special collections, or IR results?

Components of ImpactOther

• Bringing users (back) to library resourceso Directly or Indirectly

• Reduction in number of individual databases?• Fewer questions about basic searching from users

o Users don’t have to wonder where/how to search - it’s obvious

• Reference/Instruction librarians freed up to focus on higher-level consultations/instruction

• Others?

Impact of Discovery Systems on Journal UsageOur study compared usage before and after 2011 implementations at the journal level, which is only one way of measuring impact.

Levine-Clark, McDonald, & Price (2014) “Discovery or Displacement? A Large-Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage,” UKSG Insights 27(3): 249-256.

Study Results1.All Discovery systems (DS) increased usage relative to

control, and the degree of increase varied from one system to another

2.There is as much variation within DS as across them

3.Discovery systems may sometimes shift usage toward the versions of the content on their aggregated platform

4.DS impact is statistically significant, but explains only a small portion of the variance

1. All Discovery Systems increased usage vs control group

Control DS1 DS 2 DS3 DS4

Study Results1.All Discovery systems (DS) increased usage relative to control,

and the degree of increase varied from one system to another




2. There is as much variation within DS as across them

Control DS1 DS 2 DS3 DS4

DS may sometimes shift usage toward their aggregated platform (unpublished data)

● 2 DS Vendors that also have aggregated journal collections● Numbers indicate the number of journals that were hosted and used on both

the publisher & aggregator platform during 2013-14● Different publishers for each DS, but same pattern● Unlikely to be intentional; can be configured by library





4.DS impact is statistically significant, but explains only a small portion of the variance (η2 < 1% -- .0086)

Where do we go from here? Ideal Questions -- Participation by multiple stakeholders with an answer everyone wants to know

Two categories of questions

Broader Synthetic Questions

• How much of an impact do discovery services have?

Narrower Analytic Questions

• [maybe only meaningful if DS impact is compelling?]

• Address the why behind some aspects of impact

Do library discovery systems really make a difference?

Single publisher site referring URL data

But what about referrals that lead to Full Text use – aren’t these more important?

• Are users finding/accessing more relevant content?

• Are users who’d previously given up on the library convinced to come

back and use a Discovery System?

U of Denver

Synthetic - Is Google Scholar a viable alternative to a library discovery system?

Analytic - To what extent does library e-resource management and linking configuration limit DS effectiveness?

Why to study

• Wide variation across libraries with same system in our study. This could be one reason why

• If discovery implementation is only as effective as these other decisions, important to know

How to study

• Compare libraries with same discovery system and different configuration options

• Compare same library before/after changing configuration

Analytic - Does discovery system configuration affect the user experience?

Why to study

• Wide variation in our study

• Many libraries appear not to think too carefully about this. An afterthought or trial and error

How to study

• Compare libraries with different configurations

• Compare before/after in library after changing configurations

• Compare before/after across discovery system when systemwide change made

Difficulty: Libraries often reconfigure gradually. Hard to link specific configuration choice to impact.

Is there a future (in search & content) for libraries?

The even broader question that we rarely ask

• A variation on this question is, “what happens if we cede discovery to Google Scholar and its ilk?”

• The promise (and threat) of Open Access

• Influence/Change User Behavior

o retain or grow dependence on library resources

o awareness of their use of library resources

22

Discovery and Metadata from a Publisher Perspective | Henning Schoenenberger | Gulf SLA | April 20, 2016

Discovery and Metadata from a

Publisher PerspectiveHenning Schoenenberger

Springer Nature | April 20, 2016

23


• Henning Schoenenberger• Director Product Data and Metadata at Springer Nature

Personal Introduction

24


• Springer Nature is one of the world’s leading global research, educational and professional publishers.

• Springer Nature is the world’s largest academic book publisher, publisher of the world’s most influential journals and a pioneer in the field of open research.

• 3,000+ journals and more than 210k eBooks available on link.springer.com.• The company was formed in 2015 through the merger of Nature Publishing

Group, Palgrave Macmillan, Macmillan Education and Springer Science+Business Media, with offices in over 50 countries.

Introduction Springer Nature

25


Data Source: 2014 Springer Global Academic and Government Customer Survey

Does your library use a web-scale discovery service,e. g. Summon, EDS, Primo, or OCLC WorldCat/Local?

66%

34%

26


Research/ Manuscript

Creation

Manuscript Submission

Peer Review/ Proposal

Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Publishing Life Cycle

Researcher

27



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


28



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


29



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


30



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


31



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


32



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


33



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


34



Creation



Stage

Planning

Production

Publication

Distribution/ Sales

Discovery

Researcher


35


Research Life Cycle

36


Research Life Cycle

37


• Discovery is key for our customers, researcher, students, graduates, professionals who want to find and want to be found.

• Discovery is not as simple as Google – it is a complex, constantly changing environment.

• Google and Google Scholar drive much traffic to our content platforms, but they also drive many denials.

• Library-engineered tools drive more traffic AND downloads.• Of course this requires attention and coordination between content provider,

discovery service provider and library.

The Importance of Discoverability

Source: Research conducted in 2015 by Bob Boissy, Director of Institutional Marketing and Account Development – Americas

38


The Information Food Chain

Content Provider

Discovery Service

Provider

Library Catalog

Researcher / End User

Discoverability Usage

39


SpringerLink User Survey

SpringerLink, including Springer for R&D as well as Springer for Health & Hospitals, is one of the leading scientific websites in the world. In 2013, the website recorded more than 220 million visits, and is, according to the web traffic measurement companies Alexa.com and Compete.com, among the 2,500 most popular global websites.

However, so far Springer had only little knowledge about who the actual users are and for what purposes they use SpringerLink. The goal of this survey was to obtain more information on these two questions.

3,370 users responded to an online survey.

Background and Objectives

Source: Research conducted in 2014 by Harald Wirsching, Director, Market Intelligence and Webanalytics

40


Users come from a wide range of organizations and positions

Roughly three quarters of all users are from academic institutions, with strong usage from all groups within academia: Professors, lecturers, researchers, Ph.D. candidates and students

Further main user groups are researchers in non-academic research organizations, physicians & clinicians as well as researchers and R&D professionals in the industry

The ‘other’ category includes librarians, research and business consultants as well as technicians, engineers and IT professionals

Academic re-searcher/ Pro-

fessor/ Lecturer29%

Ph.D. candi-date/ Doctoral

student21%

Students28%

Researcher governm./ non-

profit7%

Physician/ clin-ician5%

Researcher/ R&D prof. industry

4%Other

6%

What is your current primary position?

N=3,370

41


Use of main search tools by user group, age and medium

N=3,369

Two-thirds of users started their search on Google web search or Google Scholar

Other important referrers include library websites, A&I databases, CrossRef links, recommendations from peers, and TOC alerts

Results are very much in line with data from the Springer Link web analytics tool

Google Scholar; 33%

Google web search; 30%

Library website; 9%

Abstract & Index-ing; 8%

Reference link-ing; 6%

Direct visit; 5%

Recommen-dation, 3% eMail alert; 3% Other; 4%

Where did you start this visit to the SpringerLink website, or who directed you to SpringerLink?

42


How could publishers help you to better achieve your goals(to get your job done)?

Better e-mail alerts

Better abstracts

Lower prices

Provide related content

Better search functionalities

Free / open access to content

0% 10% 20% 30% 40%

3%

3%

6%

10%

14%

30%

N=1,749; open-ended question

• Two main topics were brought up. Firstly, the topic about having access to all desired content, and related to this, the perceived high prices of subscriptions and PPV.

• Secondly, the whole search & discovery process, the ability for an user to quickly identify and evaluate the most relevant content to his research question. Here the search functionalities on SpringerLink were mentioned, but also a wish to identify ‘related’ content to a piece of content that already has been found to be useful. Other topics were better abstracts that help users to more quickly understand the main points of a document as well as better, keyword-based alerting services.

43


• Springer Nature has agreements with discovery services to ensure that full text data and metadata are being fully ingested.

• Springer Nature sends full text XML of our archival and frontlist content.• Discovery services use this data for indexing and loading chapter/article level

bibliographic information.• Springer Nature works closely with vendors to ensure completeness and

discoverability of our content.

Discovery Services – What Springer Nature Sends

44


• Springer Nature provides downloadable title lists in KBART format for all of our sellable packages.

• Additionally, Springer Nature provides consortia and regional package title lists.• Springer Nature works closely with link resolver vendors to keep collection

information up to date and to create clear targets.

Link Resolvers – What Springer Nature Provides

45


SpringerLink KBART Title Portal at http://link.springer.com/lists• shows all sellable packages based on the KBART Phase II Recommendations

Downloader Tool at www.springer.com/marc• Provides our MARC records and eBooks title lists for download• Includes Palgrave content

Link Resolvers – How do they receive our data?

http://link.springer.com/lists

http://www.springer.com/marc

46


• Springer Nature provides free MARC records for Protocols and eBooks including the Springer Book Archives and Springer References (downloader tool at www.springer.com/marc) as well as for journals and journal articles. Contact [email protected].

• Springer Nature MARC records are also available for free through OCLC.

MARC Records

http://www.springer.com/marc

mailto:[email protected]

47


• Query Springer Nature’s metadata (books, journals) or full-text content of Open Access journal articles

• Different output formats such as XML, JSON, PAM/PRISM• Access to the portal: https://dev.springer.com

API Portal

https://dev.springer.com/

https://dev.springer.com/

48


• Springer Nature works with just about every A&I service, over 400 in total, through a dedicated A&I department.

• Notable A&I services include:• Google Scholar• ISI Databases• PubMed• PubMed Central• Medline• SCOPUS

Abstracting & Indexing

49


• Our Discovery Services Implementation Guide outlines how Springer works with third party vendors to increase the discoverability of our content.

• PDF document available on www.springer.com/discovery.

• Springer Nature Discovery Guide coming soon!

Discovery Services Implementation Guide

http://www.springer.com/discovery

50


• Perform “discovery reviews” to assess how Springer Nature interacts with library’s online environment

• Guide customers on how Springer Nature works with discovery services and advise on best practices for discovery

• Next generation systems combine the various layers (discovery, link resolver and MARC records etc.) into one suite:

• Examples: OCLC’s WorldShare and ExLibris’ Alma• Stay up-to-date on latest developments in our industry, e. g. research data,

BIBFRAME and RDA

Ongoing Activities

51


• One of the pain points coming out of the SpringerLink User Survey seems to be an inefficient search process.

• It seems that users are overwhelmed with too many search results and would appreciate every effort that would help them to identify the most ‘relevant’ content quickly.

• Quite a few users are unsatisfied with the way that our content is indexed. Here semantic indexing with controlled vocabulary could add value, with the possibility for users to filter content more precisely (incl. synonyms, additional tags etc.).

• Examples where Springer Nature collaborates with partners active in the field of semantic technologies …

Semantic Search

52


Semantic Search

53


Semantic Search

54


Semantic Search

55


• With http://www.nature.com/ontologies and http://lod.springer.com Springer Nature is already very active in the field of LOD.

• Project ongoing to combine both sides.• The more your data is linked, the more it gets used: Discoverability and visibility.

Linked Open Data

http://www.nature.com/ontologies



http://lod.springer.com/

56


Linked Open Data

57


Linked Open Data

58


Linked Open Data

59


Linked Open Data

60


Linked Open Data

61


Linked Open Data

62


Linked Open Data

63


Linked Open Data

64


Linked Open Data

65


Linked Open Data

66


Linked Open Data

67


Enriching metadata and linking it to other available data

Linked Open Data (LOD) Cloud

68


• ORCID is a digital identifier for researchers and their work

ORCID: Open Researcher and Contributor ID

logo is clickable

PDF metadata

http://link.springer.com/article/10.1007/s10750-015-2261-x

69


• Persistent identifier for institutions• Disambiguation: alternative institution names, acronyms, language variants• Mapped to other standard identifiers (e. g. ISNI, OrgRef, Wikidata, FundRef)

GRID: Global Research Identifier Database

https://grid.ac/

70


• Example: Kuwait University (grid.411196.a): https://grid.ac/institutes/grid.411196.a

GRID: Global Research Identifier Database

https://grid.ac/institutes/grid.411196.a



71


• CrossRef’s Open Funder Registry provides a standard taxonomy for funding organizations: standardized name + unique identifier

• Springer Nature participates in this initiative

• Funding information is given both in the PDF content (acknowledgements section) and the metadata:

Funding Data: FundRef

http://link.springer.com/article/10.1007/s10549-015-3300-z

72


• Global subject classification scheme for books hosted by EDItEUR• Springer Nature among the participating organizations• Has been recently added to our metadata formats

Thema Classification

SCY12005

Clinical Psychology

Thema code: MKM

Springer Nature Subject

Classification Code

Thema term: Clinical

Psychology

73


• Since mid 2014 new Springer journal articles are marked with the CrossMark logo.• Example: http://link.springer.com/article/10.1186/s40638-014-0018-z

CrossMark



74


CrossMark

75


• Listening to what discovery services need from us as their scholarly publishing partner

• Participating in professional conferences• Abiding by industry standards co-developed by the various players• Introducing new technologies to make online resources more visible, discoverable

and usable• Staying engaged

Collaboration

76


Contact UsFor more information visit: springer.com/discovery E-mail [email protected] springer.com/salescontacts to find your local Springer Nature representativeLike us on Facebook: SpringerLibraryZoneFollow us on Twitter: @Library_Zone

Henning Schoenenberger:[email protected]

http://www.springer.com/discovery


http://www.springer.com/salescontacts

http://www.springer.com/salescontacts

https://www.facebook.com/SpringerLibraryZone/

https://twitter.com/library_zone

https://twitter.com/library_zone


Assessing the Value and Impact of

Discovery Systems:Discovery Provider

Viewpoint Jane BurkeEx Libris, a ProQuest CompanyApril 20, 2016

The reality is user expectations have evolved and library systems have not kept up.

Meetings Users’ Expectations

• Self sufficient and do NOT ask librarians for help

• Expect everything to be online and immediately accessible

• Use multiple devices and are increasingly mobile• Want to share and collaborate

Understanding today’s users

Presented by Alison Head. ACRL conference, April 11, 2013

• The importance of the role of the library as a gateway for locating information has fallen over time

• The library is increasingly disintermediated from the actual research process

Faculty Survey 2009: Key Strategic Insights for Libraries, Publishers, and SocietiesApril 7, 2010Authors:Roger C. Schonfeld (Manager of Research) & Ross Housewright (Analyst)

Discovery Services Drive Growth

Discovery services drive growth in usage

Increased usage of publisher-hosted journal content Discovery or Displacement?: A Large Scale Longitudinal Study of the Effect of Discovery Systems on Online Journal Usage Michael Levine-Clark (University of Denver), John McDonald (University of Southern California), Jason Price (SCELC Consortium)

Increase usage of JSTOR content for mutual US Higher Education clients analyzed.Plato’s Cave RevisitedBruce Heterick (JSTOR)

Charleston Conference 2013

http://www.against-the-grain.com/2013/11/discovery-or-displacement/

http://www.against-the-grain.com/2013/11/discovery-or-displacement/

http://www.against-the-grain.com/2013/11/revisiting-platos-cave/

Delivering Proven Value

Discovery delivers proven value

Academic Search Premier

APA PsycArticles

Business Source Complete

CINAHL Plus with Full Text

Education Full Text

Education Research Complete

Emerald Journals

General OneFile

Health Reference Center Academic

Journals@Ovid Ovid Full Text

JSTOR Biological Sciences Archive Collection

LexisNexis Academic

Literature Online (LION)

Oxford Journals

Print at GVSU Libraries

SAGE Complete A-Z List

ScienceDirect Journals

Wiley-Blackwell Journals (Frontfile Content)

0% 200% 400% 600% 800% 1000% 1200% 1400%

Click Through Growth by Database from 2008 - 2011

486%

306%475%

320%429%

410%

1028%359%

879%722%

430%

991%399%

1322%

840%263%

288%

160%

Understanding user search behaviors

1 10 100Number of terms per querySummon query analysis –

• High frequency of misspellings

• Natural language

queries are common

• Users often edit search terms rather than seek new strategies

• 85% of users never go past first page of results

Erin Dorris Cassidy , Glenda Jones , Lynn McMain , Lisa Shen & Scott Vieira (2014) Student Searching with EBSCO Discovery: A Usability Study, Journal of Electronic Resources Librarianship, 26:1, 17-35, DOI: 10.1080/1941126X.2014.877331

45% of Searches 3 words or

less

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/1941126X.2014.877331

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/1941126X.2014.877331

Image & Content Spotlighting

Reference Content &Related topics

Contextual Recommendations

Data-driven Related Search Suggestion

Data-driven Autocomplete

The Single Search Box is not enough

Over 50,000 Topics

7 languages and more comingAutomated Query Expansion™

Connects Users to Librarians

Librarian controlled recommendations

Data-driven search suggestions

Embedded reference chat

Newspaper Article Spotlighting

Image Spotlighting

Linking and access are critical to Discovery

• Users want immediate access to full text

• Users expect the same reliability of links that Google provides

• Failed (dead-end) links have real consequences, they:

• Frustrate users and lead to dissatisfaction with the library

• Negatively impact usage of library resources

• Linking provides a bridge from Google to Library resources

Linking is a top complaint of library end users…

“Bad links equal less users”Steven Bell. “Resolving the Link Resolver Problem”

From the Bell Tower, August 7, 2014. http://bit.ly/1ly5cci

http://bit.ly/1ly5cci



OpenURL Linking is unpredictable

“5-30% of OpenURL Links commonly fail”

Jason S. Price and Cindi Trainor. "Digging into the Data: Exposing the

Causes of Resolver Failure."

Library Technology Reports 46.7 (October 2010): 15-26.

Link resolver menus are confusing

“25% of users failed to click the appropriate link to get them to the desired content.”

Bonnie Imler and Michelle Eichenberger. “Do they ‘Get it?’

Student Usage of SFX Citation Linking Software.” College and

Research Libraries. (September 2011) 454-463.

DiscoveryIEDLLinks

OpenURL-based Linkers

KnowledgebaseRights Management

Transforming the Link ResolverWhat we’ve done:

1. Integrated Index-Enhanced Direct Linking (IEDL) technology into link resolver. • Provides the most direct and reliable links to

content and moves libraries beyond sole reliance on OpenURL

2. Introduced a new Sidebar Helper frame giving libraries a persistent presence in the research workflow• Offers improved user experience + opportunity

for libraries to be relevant and promote additional services

3. Launching new, modern user experience• Eliminates most common confusion/failure

points• Offers library’s flexible configuration and

customization options

360 Link Sidebar Helper frame

New Sidebar Helper frame

Linking embedded in Google Scholar

Link Resolver Embedded in Google

US Faculty Survey 2015

http://www.sr.ithaka.org/publications/research-data-management

Areas Of Focus

102

Outreach Instruction Research Data

“Interest in supporting students and their competencies and learning outcomes shows signs of surging… increase in the perceived importance of the role of the library in helping

undergraduate students develop research, critical analysis, and information literacy skills”

And …

• Discovery starting points remain in flux. “After faculty members expressed strongly

preferring starting their research with a specific electronic research resource/database… they are now reporting being equally as likely to begin with a general purpose search engine as they are with a specific electronic research resource/database.

Furthermore, the online library website/catalog has become increasingly important for conducting research since the previous cycle of the survey.”

Increasing Number of Data Sources

104

Administration Tools & Normalization Rules

ILS, URMAlma, Aleph, Voyager, Millennium, Symphony, Unicorn, Talis, …

Library ResourcesCourse Reserves, LibGuides, Websites, …

Digital RepositoriesRosetta, DSpace, CONTENTdm, Fedora Commons, …

Online DatabasesSubscription and open access databases, newspapers articles, …

Supplementary EntitiesResearchers, Librarians, …

Research MaterialsDatasets, outputs, raw data, faculty pages, …

Reach Out – Search Engines

105

Analytics Driven Design

106

Focus On Outreach

107

Collection exposure and publishing tools

Support Reading List Life Cycle

108

Reading List solutions such as Leganto and SIPX

Instruction

109

Education and information literacy tools:• E.g. Reading lists

More focus on outcomes, such as student success:• Demonstrate correlation between

students use of library resources & services and learning outcomes

Research Data Management

110http://www.sr.ithaka.org/publications/research-data-management

Research Data Management

“The library is well situated to manage activitiessuch as outreach, data deposit, metadata creation, and preservation; some university libraries are directed to do so, while other proactively offer their services.”

Library as a Research Data Service Provider

112

Education Through Services’ Demonstration

Low hanging fruit: discoverability of research data

Library as a Dissemination Service

113

Library Controlled Recommendations

Goal: Librarians embedded in research workflow

Integrated Reference Chat

Recommended Librarians & Research Guides