www.arl.orgassociation of research libraries measures for electronic use: the arl e-metrics project...

www.arl.orgAssociation of Research Libraries

Measures for Electronic Use: The ARL E-Metrics Project

Statistics in Practice - Measuring & Managing

IFLA Satellite Conference • 13-25 August 2002 • Loughborough, UK

Julia C. Blixrud, Association of Research Libraries


Forces and Challenges

• Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution

• Increasing pressure to maximize use of resources through benchmarking resulting in:– Cost savings– Reallocation


ARL New Measures Initiative

• Collaboration among member leaders with strong interests

• Specific projects developed with different models for exploration

• Projects self-funded by interested members • Intent to make resulting tools and

methodologies available to full membership and wider community

• Continue to collect, but limit/freeze modifications to existing descriptive measures


ARL New Measures Projects

• Demonstration project for service effectiveness measures (LibQUAL+)

• Identification of measures that demonstrate a library’s contribution to student learning outcomes

• Investigation of role libraries play in support of the research process

• Development of tools to address cost effectiveness of library operations (staff allocation, ILL/DD study)

• Project to define usage measures for electronic information resources (E-metrics)


Electronic Serials Expenditures

Total Electronic Serials Expenditures, 1995-2001

-

20,000,000

40,000,000

60,000,000

80,000,000

100,000,000

120,000,000

140,000,000

1995 1996 1997 1998 1999 2000 2001

Year


Percentages of Acquisitions Dollars Devoted to Electronic Resources

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

16.00%

18.00%

1993 1994 1995 1996 1997 1998 1999 2000 2001


Average Yearly Increases in Electronic Resources and Total Library Materials Expenditures

4.32%

37.28%

8.48%

47.46%

4.18%

12.86%

9.18%

23.22%

5.53%

22.5%

6.35%

24.81%

3.4%

27.06%

9.12%

32.69%

0

5

10

15

20

25

30

35

40

45

50

1992-93 1993-94 1994-95 1995-96 1996-97 1997-98 1998-99 1999-2000 2000-01

Average Increase in Library Materials Expenditures Average Increase in Electronic Resources Expenditures


Need for Networked Data & Statistics

Financial Support

– To justify - make a case for continued current support for digital collections

– To make a case for additional support for technology & infrastructure

Funding



Better Internal Processes– To measure & track changes in

internal processes– To enable better decision-making in

allocating & prioritizing resources & needs

– To enable assessment of service quality in a networked environment

Infrastructure



Institutional Comparisons

– For benchmarking digital services

– To enable competition for resources with other departments on campus

For Comparisons



• Need for accurate reporting of network use

• Need for accurate estimates of per client use

• Ability to compare overlapping coverage• Need the ability to pressure vendors to

price according to the library’s real need

Vendor Negotiation


ARL E-Metrics Project

Three phases:– Initial Phase (May-October 2000): What do we

know? Inventory of current practices at ARL libraries as to statistics, measures, processes, and activities that pertain to networked resources and services.

– Second Phase (November 2000-June 2001): What can we collect? Identified and field tested an initial draft set of statistics and measures

– Final phase (July 2001-December 2001): What difference does this make? Build linkages to: educational outcomes/impact, research, technical infrastructure


Project Participants

Alberta Maryland USC

Arizona State U. Mass Texas A & M

Auburn Nebraska Virginia Tech

Chicago Notre Dame W. Ontario

Connecticut Pennsylvania Wisconsin

Cornell Penn State Yale

Illinois-Chicago Pittsburgh LC

Manitoba Purdue NYPL


Project Leaders

• Co-Chairs– Rush Miller, University of Pittsburgh– Sherrie Schmidt, Arizona State University

• Information Use Management and Policy Institute, Florida State University– Charles R. McClure– Wonsik “Jeff” Shim– John Carlo Bertot

• ARL– Duane Webster– Martha Kyrillidou


Working Definition of Networked Services

Those electronic information resources and/or services that users access electronically via a computer network:

1. From on-site in the library2. Remote to the library – but from a

campus facility3. Remote from the library & campus


Networked Information Resources

• Locally Licensed Databases

• Regional or Statewide Consortia Licensed Databases

• Aggregated Databases

• Publishers Databases

• Publicly Available (Web) Resources


Networked Information Services

• Access to text & numerical databases, electronic journals, electronic books, e-lists, e-mail

• Instruction, training & workshops• Reference & information services• Virtual reference• Interlibrary loan, document delivery• IT infrastructure• Institutional & personal portals


Vendor StatisticsWorking Group

12 major ARL vendors met with project team in Denver preceding 2000 ACRL Meeting

Academic Press/IDEAL *Elsevier/Science DirectLexis/NexisOvidBell & HowellGale Group ISI * †

netLibrarySilver Platter *EBSCOJSTOROCLC/First Search

* Unable to attend Denver Meeting† Nonparticipant in project.


Related E-Metrics Projects–Some Have Formal Collaboration with ARL

• ICOLC Guidelines• Equinox (European Union)• ISO Library Statistics standard under development • NISO – updating Z39.7 standard on statistics• NCLIS – working on standardized online database usage

statistics• IMLS – Sponsored research to develop statistics for public

libraries• CLIR – Consultants to review this area• DLF – Measuring digital library use• PALS (Publishers and Libraries Solutions Committee)

represents the scholarly publishers and academic libraries in the UK

• CENDI Web Metrics


Organizational Context

Technical Infrastructure

Information Content

Information Services

Support Management

•(D3) Cost of digital collection and management

• (R1) # of e-full- text journals• (R2) # of e- reference sources• (R3) # of e-books• (D1) Size of digital library collection• (U5) # virtual visits to the resources• (P2) % of virtual visits of all visits• (P3) % of e-books to all books

• (D2) Use of digital collection• (U1) # e-ref transactions• (U2) # of logins (sessions) to e-databases• (U3) # of queries in e-databases• (U4) Items requested in e-databases• (U5) # virtual visits• (P1) % of e-ref of total ref

• (D3) Cost of digital collection construction & management

• (C1) Cost of e-full-text journals• (C2) Cost of e-ref sources• (C3) cost of e-books• (C4) Library exp. for bibl-utilities, networks & consortia• (C5) External exp. for bibl-utilities, networks & consortia• (D3) Cost of digital collection construction..

Library Networked Services and Resources


Recommended Statistics & Measures

• Patron Accessible Electronic Resources (R1-3)• Use of Networked Resources & Services (U1-5)• Expenditures for Networked Resources & Related

Infrastructure (C1-3)• Library Digitization Activities (D1-3)• Performance Measures (P1-3)


Patron Accessible Electronic Resources

• R1 – Number of electronic full-text journals

• R2 – Number of electronic reference sources

• R3 – Number of electronic books


Use of Networked Resources & Related Infrastructure

• U1 – Number of electronic reference transactions

• U2 – Number of logins (sessions) to electronic databases

• U3 – Number of queries (searches) in electronic databases

• U4 – Items requested in electronic databases• U5 – Virtual visits to library’s website and

catalog


Expenditures for Networked Resources & Related Infrastructure

• C1 Cost of electronic full-text journals

• C2 Cost of electronic reference sources

• C3 Cost of electronic books

• C4 Library expenditures for bibliographic utilities, networks & consortia

• C5 External expenditures for bibliographic utilities, networks & consortia


Library Digitization Activities

• D1 – Size of library digital collection• D2 – Use of library digital collection• D3 – Cost of digital collection

construction & management

(Collecting these data requires staff familiar with the digital environment.)


Characteristics of Each Recommended Measure

• Definition• Rationale• Unit of Measure• Data source• Frequency• Process• Related Issues


R1 – Number of Electronic Full-text Journals

• Definition - Number of electronic full-text journal subscriptions – by individual institution or consortia licensing.

• Rationale – Documents degree of expansion of electronic subscriptions available – can be used to show good coverage & need for more funding.

• Unit of measure – the journal subscription• Data source – local or vendors.• Frequency – annual, monthly, etc.• Process – parse into database or spreadsheet,

update dynamically from local catalog or vendor record.


U1 – Number of Electronic Reference Transactions

• Definition - number of electronic reference transactions – via e-mail, WWW form, etc.

• Rationale – libraries are interested in tracking the development of new electronic services. Attempt to measure reference transactions through new electronic tools and services.

• Unit of Measure – request count, time it took.• Data Source – local server, manual tally, e-mail count.• Frequency – daily, monthly, annually, etc.• Process – clarify process, identify activity points, identify

collectors of data, consolidate data.• Related Issues –This measure may have to broken down into

additional data types – time, type of query, type of interaction, scheduling issues, measures of quality and reliability.


Performance Measures

• P1 – Percentage of electronic reference transactions of total reference

• P2 – Percentage of virtual visits of all library visits

• P3 – Percentage of electronic books to all monographs

• Percentage of electronic journals to serial subscriptions


E-Metrics Assessment Plan

• Have administrative clarity• Reflect institutional structure and staff

functions• Balance stakeholder needs with availability

of data• Provide for the input, structure, housing &

archiving of data• Propose a structure to disseminate data,

reports & information – dynamic web intranet (portal)


E-Metrics Implementation Process

1. Preparation• Immediate• Long term

2. Identification of Tasks, Data and Needs3. Data Collection4. Information Management5. Reporting, Dissemination & Feedback


E-Metrics Next Steps

Call for participation among members to test proposed measures for 2002/2003 (over 35 participants to date)– Examination of the deliverables from the first phases– Collect FY02 totals

• Compilation• Data analysis• Distribution for discussion

– Analysis of approaches– Best practices for work processes

• From E-metrics project• Locally developed


E-Metrics Next Steps

• Continued work with vendors through international COUNTER project

• Continued work with international standards activities

• Workshops and training to develop necessary data analysis skills


(Carla Stoffle, University of Arizona)

Developing measures and evaluation techniques for networked services will take time, effort, and on-going learning on everyone’s part – but we must begin now.

We not only need to measure things in new ways but we also need to measure new things. (Sherrie Schmidt,

Arizona State University)


Project Documents

• Measures for Electronic Resources (E-Metrics) Part 1: Project Background and Phase One Report

Part 2: Phase Two Report

Part 3: E-Metrics Instructional Module

Part 4: Data Collection Manual

Part 5: Library and Institutional Outcomes

• www.arl.org/stats/newmeas/emetrics/


Julia C. Blixrud

Director of Information Services

Association of Research Libraries

21 Dupont Circle, Ste 800

Washington, DC 20036

[email protected]

202-296-2296 ext. 133

202-872-0884 (fax)

202-251-4678 (cell)

www.arl.orgassociation of research libraries measures for electronic use: the arl e-metrics project...

Documents

arl libraries

electronic resources

use of resources

usage measures

descriptive measures

electronic use

current support

internal processes