NISO Webinar: Keyword Search =
"Improve Discovery Systems"
November 12, 2014Speakers:
John McDonald, Associate Dean, Collections, University of Southern California LibrariesJason Price, Program Manager, Statewide California Electronic Library Consortium
(SCELC)
Steve Guttman, Senior Director of Product Management, ProQuest
Marshall Breeding, Library Consultant, librarytechnology.org
http://www.niso.org/news/events/2014/webinars/improve_discovery/
Differential Discovery: The Effect of Discovery
Services on Journal Usage
NISO WebinarNovember 12, 2014
Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California
Jason Price, SCELC Consortium
…our customers insist that usage of our content decreased after implementation of discovery service “X”.
A publisher told us . . .
https://www.flickr.com/photos/kongping/7192138660/in/gallery-flickr-72157645846953449/
Librarians speculate . . .…of course discovery vendors direct their users to their own aggregated content.
http://www.flickr.com/photos/hypotekyfidler/15012731920
Goals of our research
• Determine whether discovery services impact usage
• Help librarians and publishers understand how their choices impact use
• Help librarians, publishers, and vendors improve the discovery experience for end users
What did we measure?
• Whether there is an effect
• NOT why that effect exists
Caveat 1: Publisher-hosted journals are only part of the picture
eBooks, pBooks, newspaper articles, aggregator journal content, etc.
publisher journal content
The six publishers in this study
Not to scale!
Caveat 2: More usage may not be better!
• Decreased usage might be a sign of greater efficiency– Relevant articles found faster = fewer articles to examine
OR– Fewer articles examined because other relevant content
types found
Does implementation of a discovery service impact usage of publisher-
hosted journal content?
Did Journal Usage Change? (and if so, to what extent?)
• 4 discovery services– 6 libraries in each group
• A control group– 9 libraries that did not implement a discovery
service in this time period• 6 major journal publishers
– 9,206 Journals in the study– 163,545 Observations (Library + Journal)
Participating libraries• 157 asked for permission, 155 granted permission
– 124 from the US, 33 from other English-speaking countries• Has your library used a different discovery service in
the past?– Only libraries answering “No” were selected
• Is your discovery service featured on your homepage?– All participants answered, “Yes, with a search box”
• To what extent did your library market the discovery service at its release?– 4 said “None”, one from each vendor– 12 said “A limited extent”, 2 WCL, 2 EDS, 4 Summon, 4 Primo– 8 said “A significant extent, 4 WCL, 2 EDS, 1 Summon, 1
Primo
Methodology
Compared COUNTER JR1 total full text article views for the
12 months before vs 12 months after implementation date
June
201
0St
art
Impl
emen
tatio
nM
ay 2
011
May
201
2En
d
Year 1 Year 2
Included implementation month in Year 1 to ensure that both periods included an entire academic year
Tota
l Stu
dent
FT
E
Journals by Library & Service
EDS Primo Summon WorldCat Control
Tota
l # o
f Jo
urna
ls b
y In
stitu
tion
Average Journal Usage by Library
Testable Effects
• Discovery Service– Implemented by multiple libraries– Used to find content from all publishers
• Publisher– Accessible in all discovery services– Accessible across all libraries
• Library– Uses content from multiple publishers– Uses only one discovery service
Nested ANOVA Model Results
How does usage change differ across discovery services?
ABB
C
D
Letters indicate statistically significant differences (Tukey multiple comparisons, p < .05)
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Does usage change differ across libraries within discovery services?
How does usage change differ across publishers?
Summary of Results• Discovery Service
– Every service increased usage compared to control – Some services increased usage more than others
• Library– The degree of usage change differed among libraries
using the same discovery service• Publisher
– Usage change differed across publishers:• 1 of 6 publishers saw a significant decrease• 2 of 6 publishers saw significant increases• 3 of 6 publishers saw no detectable change
Conclusions to be avoided…
Our research does not indicate that:a) one service increases usage more than another for
every library or publisher effects vary across libraries and publishers
b) one service is better than another libraries or their users may benefit from increased usage of
other content instead higher usage may indicate lower efficiency
Next Steps• Design & test for effects of:
–Aggregator full text availability–Linking configuration options in discovery services
• Expand pool of libraries• Explore the why?• Other possibilities
–Journal Subject?–Journal age (archive vs current)?–eBook usage?
1. What is the best service: Summon, Primo, EDS or Worldcat? • Our research can’t prove one is better than another, and
usage is only one reason to install a discovery system
2. Why was there so much variation in the effect of implementation across publishers at my institution? Why did some increase and some decrease?
Potential Librarian Concerns (1)
3. What about configuration differences?• This undoubtedly contributes to some variance, so be
careful to review and maintain your configurations and preferences.
4. What about other resource types (ebooks, print books, etc)? • They may be affected, we haven’t studied these.
5. Was it missing metadata that caused the differences?• Maybe, we can’t know for sure, so the publishers and
vendors need to come up with best practices for metadata exchange to ensure everyone is on an equal playing field (encourage participation in NISO ODI)
Potential Librarian Concerns (2)
Potential Publisher Issues (1)1. Are users being directed to Aggregated full text
before publisher hosted full text? • Maybe, but it might be the Library’s configuration that is to
blame.• This could be counterproductive for aggregators.
2. Does implementation of abstract-based discovery reduce the ranking of publisher content that is not indexed (or available full text) in a library’s aggregator databases?• Maybe, but it its up to the Library to decide if that is a
desired effect or not.
Potential Publisher Issues (2)3. Why was there so much variation in the effect of
implementation on change in usage of our content across institutions using the same discovery service? • Because usage can affect the publisher bottom line, this is a
key question. Some publishers are finding it of value to invest in research and engagement of customers at the extremities of change after implementation.
4. How can publishers know what is being done with the metadata they send? • Discovery vendors need to be more proactive in proving
the positive benefits of providing more robust metadata
1. How do we prove we’re content neutral?• Develop best practices that indicate to libraries their
configuration and linking choices.• Allow for independent studies• Make the case for the risks related to bias?
2. Our systems make other resource types (ebooks, print books, etc) more discoverable, does anyone value that? • More research needs to be done to assess effectiveness for
other content types
3. Does increased usage necessarily mean a discovery system is better? • Definitely not! Depends on what a library values.
Potential Discovery Vendor Concerns
Differential Discovery: The Effect of Discovery
Services on Journal Usage
NISO WebinarNovember 12, 2014
Michael Levine-Clark, University of DenverJohn McDonald, University of Southern California
Jason Price, SCELC Consortium
A SINGLE SEARCH BOX IS NOT (nearly) ENOUGH
Transforming Discovery with ProQuest
Steve GuttmanSr. Dir Product Mgmt, ProQuest
Steve GuttmanSr. Director of Product Management,Research & DiscoveryProQuest
Former:MarkLogicMicrosoftAdobe
It’s “Discovery” not “Search”
Christian Reusch, Flickr creative commons
Discovery is more than a Single Search Box
What does it mean to “go beyond” a single search box?
04/13/2023 35
What does it mean to “go beyond” a single search box?
04/13/2023 36
context
What does it mean to “go beyond” a single search box?
04/13/2023 37
context
suggestions
What does it mean to “go beyond” a single search box?
04/13/2023 38
context
suggestions
direction
What does it mean to “go beyond” a single search box?
04/13/2023 39
context
suggestions
direction
ease of use
Discovery needs to “channel” the Library
04/13/2023 40Nico Kaiser, Flickr creative commons
3 Pillars of Discovery
1.Help users “discover”
2.Institutional enrichment
3.Bring patron insights to libraries
Help Users “Discover”
Chris Harrison, Flickr creative commons
Discovery encourages exploration
• Auto-suggest
• Topic Explorer pane
• Related search suggestions
• Automated query expansion
• Fluid previews
• Dynamic content spotlighting
• Live Help with integrated chat
• Faceted search
• Responsive Design
Discovery encourages exploration
• Auto-suggest
• Topic Explorer pane
• Related topic suggestions
• Automated query expansion
• Fluid previews
• Dynamic content spotlighting
• Live Help with integrated chat
• Faceted search
• Responsive Design
Discovery encourages exploration
• Auto-suggest
• Topic Explorer pane
• Related topic suggestions
• Automated query expansion
• Fluid previews
• Dynamic content spotlighting
• Live Help with integrated chat
• Faceted search
• Responsive Design
Discovery encourages exploration
• Auto-suggest
• Topic Explorer pane
• Related topic suggestions
• Automated query expansion
• Fluid previews
• Dynamic content spotlighting
• Live Help with integrated chat
• Faceted search
• Responsive Design
Discovery encourages exploration
• Auto-suggest
• Topic Explorer pane
• Related topic suggestions
• Automated query expansion
• Fluid previews
• Dynamic content spotlighting
• Live Help with integrated chat
• Faceted search
• Responsive Design
User Experience Matters!
User Experience Matters!
Summon design tenet:Never take the user away from their results
No modal dialogs!
Reveal info when needed
Institutional Enrichment = Making the library part of Discovery
Rich Grundy, Flickr creative commons
What is Institutional Enrichment?
Suggested Librarian
Libguides, Research Guides
Database Recommender
Bring patron insights to libraries
James Royal-Lawson, Flickr creative commons
Bring patron insights to libraries
Features that make you more efficientFeatures that help libraries understand users and content better
3 Pillars of Discovery
1.Help users “discover”
2.Institutional enrichment
3.Bring patron insights to libraries
3 Pillars of Discovery
1.Help users “discover”
2.Institutional enrichment
3.Bring patron insights to libraries
4
4.Integration
DefinitionConceptualization, Funding
DiscoverySearch, Recommen-dations,
Networking
Organization
ConsumingReading, Annotating
Sharing, Collaboration
AuthoringSynthesis, Collaborative authoring
(Pre)pubSubmission, OA prepublication,
hosting
Researcher Workflow
DefinitionConceptualization, Funding
DiscoverySearch, Recommen-dations,
Networking
Organization
ConsumingReading, Annotating
Sharing, Collaboration
AuthoringSynthesis, Collaborative authoring
(Pre)pubSubmission, OA prepublication,
hosting
ResearchCompanion
Pivot
Summon360 Linkebrary
COS
Flow
Flowebrary
Flow
RefWorks,Flow
?
Researcher Workflow
3rd party ecosystem integration
MS WordGoogle Docs
(OA) publishersPeer review systems (Aries)
figshare
Social mediaDropbox
ORCIDVIVO
Transforming Discovery
ACCESS
DISCOVER
USE & COLLABORATE
RESE
ARCH
ERS
LIBR
ARIA
NS
Alan Levine, Flickr creative commons
04/13/2023 60
LIBRARY RESOURCE DISCOVERY:
Marshall BreedingIndependent Consultant,Founder and Publisher, Library Technology Guideshttp://www.librarytechnology.org/http://twitter.com/mbreeding
November 12, 2014NISO Webinar: Keyword Search="Improve Discovery Systems"
Next Steps
Talking points
Current state of the industry NISO Open Discovery Initiative Discovery White Paper for NISO
Emergence of Discovery
Online catalogs and specialized research databases
Metasearch Local discovery interfaces Index-based discovery …?
Issues with index-based discovery Discovery services populated through
private agreements Uneven participation Library uncertainty regarding
performance and capability Interest in increased transparency
Update on the NISOOpen Discovery Initiative
Balance of Constituents
Libraries
Publishers
Service Providers
66
Marshall Breeding, Vanderbilt UniversityJamene Brooks-Kieffer, Kansas State University Laura Morse, Harvard UniversityKen Varnum, University of Michigan
Sara Brownmiller, University of OregonLucy Harrison, College Center for Library Automation (D2D liaison/observer)Michele Newberry
Lettie Conrad, SAGE PublicationsRoger Schonfeld, ITHAKA/JSTOR/PorticoJeff Lang, Thomson Reuters
Linda Beebe, American Psychological AssocAaron Wood, Alexander Street Press
Jenny Walker, Ex Libris GroupJohn Law, Serials SolutionsMichael Gorrell, EBSCO Information Services
David Lindahl, University of Rochester (XC)Jeff Penka, OCLC (D2D liaison/observer)
ODI deliverables
Standard vocabulary NISO Recommended Practice:
Data format & transfer Communicating content rights Levels of indexing, content availability Linking to content Usage statistics Evaluate compliance
Inform and Promote Adoption
67
ODI Timeline
Milestone Target Date Status
Appointment of working group Dec 2011
Approval of charge and initial work plan Mar 2012
Completion of information gathering Jan 2013
Completion of initial draft Jun 2013
Completion of final draft Sep 2013
Public Review Period commences Sep 2013
NISO Publishes Recommended Practice June 2014
68
ODI Recommended Practices Metadata elements for content providers
to contribute to discovery service providers
Content providers disclose extent to which they participate with each discovery service
Discovery Service providers disclose what content is represented in index
Discovery services disclose any bias in search results or relevancy relative to business relationships
Discovery services provide use statistics
ODI Standing Committee
Fulfilling recommendation of the ODI that NISO charge an ongoing committee to promote ODI best practices and related issues.
Discussions may include but are not limited to: brainstorming on ways to publicize and educate
the community on ODI answering any support questions checking on status of vendor support liaising with other standards efforts as applicable determining when is an appropriate time to
consider updating ODI
ODI Standing Committee Roster
Laura Morse – Harvard University
Lettie Conrad – SAGE Aaron Wood – Ingram
Content Elise Sassone – Springer Jason Price – SCELC Jill O’Neill – NFAIS Julie Zhu – IEEE
Marshall Breeding – Independent Consultant
John McCullough – OCLC Michael McFarland –
Credo Rachel Kessler – Ex Libris Scott Bernier – EBSCO Steven Guttman –
ProQuest Ken Varnum – University of
Michigan Library
Current State of Resource Discovery Four commercial index-based discovery
services Summon, EDS, WorldCat Discover Service,
Primo Many commercial and open source
discovery interfaces Library Portal products: BiblioCMS, Arena,
Iguana, etc Increasing penetration of commercial
products in academic libraries
Web-scale Index-based Discovery
Search:
Digital Collections
Web Site Content
Institutional
Repositories
…E-Journals
Reference Sources
Search Results
Pre-built harvesting and indexing
Conso
lidate
d In
dex
ILS Data
Aggregated Content packages
Usage-generate
dData
Customer
Profile
Open Access
Public Library Information Portal
Search:
Digital Collections
Web Site ContentCommunit
yInformatio
n
…Customer-providedcontent
Reference Sources
Search Results
Pre-built harvesting and indexing
Conso
lidate
d In
dex
ILS Data
Aggregated Content packages
Archives
Usage-generate
dData
Customer
Profile
NISO Discover White Paper
Advise Discovery to Delivery Topic Committee on possible areas of future interest or activity
Overview of the current state of library resource discovery
Recommendations for next stages of ODI API ecosystem: extend and interoperate Discovery beyond the library Importance of Linked Data on future models of
discovery Extend keyword relevancy to leverage Linked
Data
White Paper Status
Draft submitted to D2D Topic Committee Comments expected through November Review and Finalize Informal concepts: not yet a NISO
product Still working to fill in gaps or missing
topics
Open Access / Open Source
Open source tools exist for discovery Interfaces: VuFind Blacklight
No open access discovery indexes High threshold of expense and difficulty to build
index Platform costs Software development Publisher relations Billions of content items to index and maintain
Opportunities to lower barriers to entry?
Issues and concerns
No open access discovery index Uneven participation despite ODI (too
early?) Inherent limitations of library-centered
interfaces Patrons ignore library-provided tools Interest in discoverability and delivery
via global internet infrastructure Exploration of enhanced discovery
through linked data mechanisms
Discovery White Paper Topics
Integration of Discovery with Resource Management
Varying degrees of association Tightly Coupled: high level of difficulty to
mix discovery and management platforms Alma / Primo Intota / Summon OCLC WorldShare / WorldCat
Loosely Coupled: Kuali OLE / Any discovery Any Management product/ EBSCO Discovery
Service
Integration expectations
Should there be better defined mechanisms for integration
Disclosure of integration support Recommended practices?
Linked data
Not yet a fully operational method for library-oriented content Increasing representation of bibliographic
resources BIBFRAME stands to make great impact
Universe of scholarly resources not well represented
Will current expectations for content providers to make metadata or full text available for discovery expand to exposure as open linked data?
OLD Projects Proliferate
VIAF: Virtual International Authority File Europeana (
http://pro.europeana.eu/linked-open-data ) Library of Congress: subjects and
authorities. (http://id.loc.gov/) National Library of Sweden: LIBRIS
Swedish Union Catalogue Zepheira: BIBFRAME, LibHub … Many others
Tools and Technologies for Linked Data
Easily implemented components and toolkits for keyword-based discovery SOLR Elastic Search
Identify or package similar tools for linked data
Hybrid models
Can index-based search tools be improved through Linked Data Browse to related resources Add additional hierarchies of structure to
search results
Gaps in Discovery Remain
Coverage remains uneven Weak coverage in non-roman languages Systematic coverage of open access
materials Difficulty with known-item and precision
search Some new systems lack traditional catalogs
Inconsistent linking mechanisms
Opportunities for Future Enhancements
Improved ecosystem of APIs Increased use of social and usage data to
improve relevancy and retrieval Non-textual search tools Expansion of discovery scope:
Research data Special collections / Archives
More sophisticated performance statistics Altmetrics
Possible new topics for ODI
Address topics marked out of scope by initial ODI workgroup More conducive to A&I resources Relevancy Data exchange protocols
Initial phase described rather than prescribed transfer mechanisms
High threshold of difficulty remains for new services Interoperability with library resource management
systems Interoperability with university learning
management systems
Discovery beyond Library Interfaces
Improved performance of library content through Google Scholar Same expectations for transparency?
Better exposure of library-oriented content Schema.org or other microdata formats
Better exposure of scholarly resources Open access & Proprietary
Embedded tools in other campus interfaces
The future of Resource Discovery More comprehensive discovery indexes Stronger technologies for search and
retrieval Discovery beyond library-provided
interfaces Linked Data to supplement discovery
indexes
Questions and discussion
NISO Webinar • November 12, 2014
Questions?All questions will be posted with presenter answers on the NISO website following the webinar:
http://www.niso.org/news/events/2014/webinars/improve_discovery/
NISO Webinar: Keyword Search = "Improve Discovery Systems"
Thank you for joining us today. Please take a moment to fill out the brief online survey.
We look forward to hearing from you!
THANK YOU