NISO’s IOTA InitiativeMeasuring the Quality of OpenURL Links

NASIG Annual ConferenceSt. Louis, MO

June 2 – 5, 2011 

Rafal Kasprowski, Rice University

Agenda

• In the Beginning: Full-text linking and Advent of OpenURL • IOTA: Created in response to OpenURL linking problems

 • IOTA’s analytical approach

 • Community-derived reports comparing quality of vendor

OpenURLs • Concept of the OpenURL Quality Index

• IOTA & KBART: relationship & joint initiative • Community involvement in IOTA: necessary for best outcomes

Before OpenURL: Proprietary Linking

 • Certain A&I database providers (e.g., CSA, PubMed)

offered full-text linking option for a select number of content providers.

 • Libraries manually activated full-text linking with

providers they had subscriptions with. • A&I --> Full Text

 

Proprietary Linking: Pros and Cons

• Linking had to be activated manually by libraries for each full-text provider.

 • A&I providers offering this option were few.

 • Selection of full-text providers was limited.

 But... •  Once set up, the static links to full texts were

accurate.

• Problem source pinpointed easily: A&I --> Full Text

Advent of OpenURL

• Objective: Deliver full texts unrestrained by proprietary silos. • Open standard generating dynamic links at time of request.

 • A-Z list (e.g., e-journal, e-books):

o Knowledge base (KB) with library's holdings.o Replaces librarian as intermediary in linking.o Indicates provider of "appropriate copy"

 • A&I ("Source") --> A-Z list ("KB") --> Full Text ("Target")

A, Bernand, et al. "A versatile nanotechnology to connect individual nano-objects for the fabrication of hybrid single-electron devices." Nanotechnology 21, no. 44 (November 5, 2010): 445201. Academic Search Complete, EBSCOhost (accessed October 24, 2010).

OpenURL: resolver, syntax, linking nodes

http://ps4ps6lm2r.search.serialssolutions.com/?issn=0957-4484&volume=21&issue=44&date=20101105&spage=445201&title=Nanotechnology&atitle=A+ versatile+nanotechnology+to+connect+individual+nano-objects+for+the+ fabrication+of+hybrid+single-electron+devices.&aulast=A++Bernand

Source Citation

Target OpenURL (Source OpenURL structured similarly)

Pros & Cons of OpenURL

Pros:• KB/Resolver vendors took over most of the linking setup:

Less work for libraries and providers.• Participation by A&I platforms and full-text providers

exceeded proprietary linking: OpenURL scales better. Cons:• Dynamic linking less predictable than static linking: more

difficult to pinpoint cause of link failures.• OpenURL linking not improved significantly last 10

years.• No systematic method exists to benchmark OpenURLs.

Identifying source of problem…"72% of respondents to the online survey either agreed or strongly agreed that a significant problem for link resolvers is the generation of incomplete or inaccurate OpenURLs by databases (for example, A&I products)."Culling, James. 2007. Link Resolvers and the Serials Supply Chain: Final Project Report for UKSG, p.33. http://www.uksg.org/sites/uksg.org/files/uksg_link_resolvers_final_report.pdf.

 Defining methodology for addressing problem… 

Recently, researchers have indicated the need for metadata quality metrics, including:• completeness;• accuracy;• conformance to expectations;• logical consistency and coherence.

Bruce, Thomas R. and Hillmann, Diane I. 2004. The Continuum of Metadata Quality: Defining, Expressing, Exploiting. In Metadata in Practice. Ed. Diane I. Hillmann and Elaine L. Westbrooks. Chicago: American Library Association, pp. 238-256.

Année philologique OpenURL Study

2008 Cornell study led by Adam Chandler*• Problem: Too often links sent from Aph did not

successfully resolve to requested resource.• Objective: Examine quality of OpenURLs offered to

users by Aph in order to improve the linking. Aph Study investigated:• Faulty citation metadata from source database.• Method to evaluate the OpenURLs.

*Chandler, Adam. 2009. Results of L’Année philologique online OpenURL Quality Investigation: Mellon Planning Grant Final Report.http://metadata.library.cornell.edu/oq/files/200902%20lannee-mellonreport-openurlquality-final.pdf.

Scoring System & Aph Study Outcomes

Concept of scoring in Aph study (based on B. Hughes study)*• establish a baseline for comparison; • results to be shared with data providers;• develop a best practice.

 Problem analysis in Aph study limited to: • source link• presence/absence of citation metadata elements

 Results:• OpenURL quality model: compares elements in Aph

OpenURLs to those of other providers.• No scoring was achieved for Aph, but model is first step

towards scoring system.

*Hughes, Baden. 2004. Metadata Quality Evaluation: Experience from the Open Language Archives Community. In Digital Libraries: International Collaboration and Cross-Fertilization. Ed. Zhaoneng Chen et al. Berlin: Springer-Verlag, 2004, pp. 320-329.

Creation of IOTA

NISO:• Accepts proposal to take Aph Study to wider community.• New OpenURL quality metrics initiative formed in Jan. 2011.• Branded as: Improving OpenURLs Through Analytics.

 Basic Assumptions:• Results are achieved through an analytical investigation of

how OpenURL links work.• Practical Goal: Not the OpenURL standard is addressed, but

the links (OpenURLs) generated by standard.• Selective changes to OpenURLs will lead to significant

improvement in linking success rate.o Motto: "small changes. big improvements"

IOTA Desired Outcomes

• Produce qualitative reports that will help OpenURL providers quickly compare their OpenURL quality to that of their peers.

• Develop community-recognized index for measuring the quality of OpenURL links generated by content providers.

  • Method:

o fair;o transparent;o scalable across all OpenURLs and their providers.

How is comparing OpenURLs useful?

• Content providers generating OpenURLs can:– compare their OpenURLs with other providers;– make improvements to their OpenURLs.

• Institutions can:– compare OpenURL providers;– make local adjustments to OpenURL setup.

• Link resolver vendors can:– compare OpenURL providers;– change their OpenURL provider settings:

• Link resolvers;• Web-scale discovery products.

Report types

• Source reports– Viewing how a particular (1) vendor or (2)

database• A. uses OpenURL elements (element frequency)• B. formats OpenURL elements (pattern frequency)

• Element / Pattern reports– Viewing how a particular (1) element or format

• A. is used across vendors• B. is used across databases

Running reports

Reports: Log file providers

Report: source (vendor or database)

Report: source = vendor

Report: element, source = vendor

Report: pattern, source = vendor

Report: source = database

Report: element, source = database

Report: pattern, source = database

Report: element and pattern frequency

Report: element & pattern frequency:Vendor option

Report: element/pattern frequency:Choosing Metric

Report: element/pattern by vendor

Report: element & pattern frequency:Database option

Report: element/pattern by database

Reporting System: improvements underway

• Consolidating variant instances of databases and vendors if the same;

• Separating article-like requests from book-like requests– Either/Or situation: most resources do not offer

both formats– Once separation is completed, users will be

given corresponding options to select OpenURL data by format: ARTICLE or BOOK

• These improvements will also benefit the accuracy of the OpenURL scoring system.

OpenURL Quality Index: initial version

1. Core elements:• Any element contained in IOTA's OpenURL reporting system;• 13M OpenURLs already obtained from libraries content

providers. 2. Scoring system based on assumption:• Correlation exists between

o # of core elements ("OpenURL completeness") & o ability of OpenURLs to link to specific content.

  3.  Weighting assigned to core elements:• Based on relative importance

o spage vs atitleo issn vs jtitleo doi/pmid vs date, etc.

OpenURL Quality Index: vendor rating

Work in Progress

• Element weighting still in progress:o E.g., importance of identifiers (doi, pmid) vs bibliographic data

(issn, volume, spage).

 • Currently, IOTA focuses on OpenURLs from citation

sources only. OpenURL quality is also influenced by:o knowledge base,o resolver,o full-text provider (target).

 • High "completeness" score of OpenURLs not always

indicative of "success" in linking to full textso Combination of multiple indexes along linking nodes may provide

more complete picture.

IOTA & KBART: NISO working groups

IOTA•Deals with issues specific to OpenURL linking;

•Seeks improvements in OpenURL elements used by:– OpenURL providers.

KBART•“Knowledge Bases And Related Tools”

•Deals with data issues at the KB level

•Seeks improvements in data exchange practices between:– content providers (e.g. OpenURL providers);

– product vendors (e.g. link resolver vendors);

– subscription agents.

IOTA & KBART: related through OpenURL

• IOTA node: – analyzing data sent from OpenURL source to link

resolver.

• KBART node:– creating formatting best practices for data sent from

content providers to knowledge base (and link resolver) vendors.

KBART-IOTA joint initiative

• KBART-IOTA node:– Exploring together the third source of failures:

• link-to (or target) syntax and behavior which couples link resolvers to content providers

• Collaboration begun in March 2011 is meant to address OpenURL quality in a broader context.

How can I get involved?

If you are an OpenURL provider:• Contribute data to IOTA• Review the IOTA data • This data is meant to help you make

improvements in your OpenURL linking If you are a librarian:• Contribute data to IOTA• Spread the word to vendors about IOTA

OpenURL data contact:• Adam Chandler, alc28@cornell.edu

IOTA Web Presence

Questions?

• http://openurlquality.niso.org• http://www.niso.org/workrooms/

openurlquality• @nisoiota on twitter

Rafal KasprowskiElectronic Resources LibrarianRice University, Fondren Library MS 44Houston, TX 77005rk11@rice.eduhttp://www.slideshare.net/rkaspro