through the bytes darkly through the bytes darkly, management information and the digital library...

Through the Bytes Darkly

Through the Bytes Darkly,

Management Information and the Digital Library

Joe ZuccaAssessment, Planning and Publications LibrarianUniversity of Pennsylvania Library

Information Technology Interest GroupACRL, New England ChapterMay 17, 2002


1. Environmental Audit: Key Factors That Influence Our Ability to Measure Digital Information Use

2. From Low Resolution to High Resolution Data: Mining the Server Logs

3. The Data Farm Experiment: Tools That Serve Access Can Also Serve Measurement

4. Why the Data Are Important

Four Sections of This Presentation:


Strategic Focus

Base planning, goal setting/assessment on empirical evidence. From 1996- an element of Penn’s Strategic Plan

Operational Imperatives

1) Make evaluation and measurement a component of each program and project

2) Construct relays that feed data to people who need quantitative information to strategize and manage

Experimental Attitude

Leverage the data you have; usually they’re “good enough” to validate organizational experience and knowledge

1. Organization and Culture

Measuring Electronic Use at Penn: Environmental Influences


3 9 86

1394

2438

4932

6875

0

1000

2000

3000

4000

5000

6000

7000

1991 1993 1996 1999 2000 2001 2002

Num

ber

of T

itles

2. Proliferation of Electronic Resources

Article indexes, e-journals and other full-text resources



2.1. Growth of Expenditures for Electronic Resources


1991 1993 1996 1999 2000 20013.7% 3.2% 5.5% 13.2% 13.9% 15.7%

E-Resources as a percent of acquisitions budget

Annual Growth of Expenditures for Electronic Information Based on 1991

0%

100%

200%

300%

400%

500%

600%

700%

800%

900%

1000%

1991 1993 1996 1999 2000 2001

PC

T I

ncr

ea

se in

Exp

en

ditu

re


Volatile metrics (“The new system doesn’t count that way!”)

Ever-changing data elements (“sets are out “searches” are in)

No common metrics (log-ins, sessions, searches, browses, page hits…)

No measurement standards (What’s a “search”?, What’s a Web “session”?)

Non existent or inaccessible data (the vendor problem)

Approximate & hard to obtain statistics (lots of data, no information)

Fleeting benchmarks

3. Technology’s Hostility to Measurement



From Low Resolution to High Resolution Data:

Mining the Server Logs for

Descriptive Statistics

dial-123-130.dial. indiana.edu - - [04/ Feb/2001 :00:18:02 -0500] "GET /special/ photos/ theater/504.html HTTP/1.0" 200 3247 "http://www.library.upenn. edu /special/photos/ theater /503.html" "Mozilla/4.7 C-CCK MCD {C-UDP; EBM-APPLE} (Macintosh; I; PPC)” dialin1085. upenn.edu--[04/Feb/ 2001:00:18: 04 -0500]"GET/facilities/count_ use.html?resource =China%20Economic%20 Review& method= ejs& url= http://www.sciencedirect.com/ science/journal/ 1043951XHT TP/1.0" 200 2027 "http:// www.library.upenn.edu/webbin 5/ resources/ejspubl ic5.cgi?homepage=http:// www. library.upenn.edu/lipp incott/&community= Business" "Mozilla/ 4.0 (compatible; MSIE 5.0; Windows 98; DigExt; SPIKE 5)” 203.197. 226.240 - - [04/Feb/2001:00:18:07 -0500] "GET /etext/sasia/aiis/ architecture/khajuraho/ 010a.jpg HTTP/1.0" 200 89117 "http://www.library.upenn.edu/etext/sasia/ aiis/arch itecture/khajuraho/010.html" "Mozilla/4.7 [en] (Win95; I)”


Records in locally-managed databases (including the OPAC)………………………26,332,138

Number of journal article indexes & full-text files (e.g. Academic Index)…….……………...267

Number of e-journals (from publishers such as Elsevier and free sources)…..…………..6,608

Number of digital books (locally created, aggregated and licensed)…….……………...110,000

Number of locally digitized and accessible images (e.g. fine art slides, ms facsimiles)..82,356

Number of records in the OPAC ……………………………….....……………………...2,879,696

Number of pages, forms and directories constituting the library web site……………….32,000

InputsLow Resolution


Low Resolution

Web Pages Served 1995-2001 from www.library.upenn.edu. 3-month moving average

The Load on Our Machines

0

200,000

400,000

600,000

800,000

1,000,000

1,200,000

1,400,000

1,600,000

1,800,000

2,000,000

Jul-9

5

Oct

-95

Jan-

96

Apr

-96

Jul-9

6

Oct

-96

Jan-

97

Apr

-97

Jul-9

7

Oct

-97

Jan-

98

Apr

-98

Jul-9

8

Oct

-98

Jan-

99

Apr

-99

Jul-9

9

Oct

-99

Jan-

00

Apr

-00

Jul-0

0

Oct

-00

Jan-

01

Apr

-01

Web

Pag

es R

eque

sted

per

Mon

th

Total pages requested: FY 96 2,481,146

FY 97 5,316,283 one-year increase................114.3%

FY 98 7,038,872 ...................32.4%

FY 99 11,807,289 ...................67.7%

FY00 12,540,531 .....................6.2%

FY01 14,461,712 ....................15.3%


0

5,000,000

10,000,000

15,000,000

20,000,000

25,000,000

1996 1997 1998 1999 2000 2001

OPAC

Web

Changing Machine Demand

Pages Served by the Main Library Web Server + OPAC Server

2002

BlackBoard

Low Resolution

Projected


0

500,000

1,000,000

1,500,000

2,000,000

2,500,000

3,000,000

3,500,000

fy97 fy99 fy00 fy01

Search Activity Over Time

sea

rch

es

Annual Searches in Licensed Databases (e.g., MEDLINE), FY97-01

Low Resolution


0

1000

2000

3000

4000

5000

6000

0 4000 8000 12000

sessions

tim

e

0

10000

20000

30000

40000

50000

60000

70000

80000

0 1000 2000 3000 4000 5000

tim e

sets

0

10000

20000

30000

40000

50000

60000

70000

80000

0 4000 8000 12000

sessions

sets

Correlation Matrix of Use Metrics Available for Ovid Files

Pearson r for Sessions, Connect Time, Sets, Documents Viewed

99 cases

Sessions Time Sets Docs.Viewed

Sessions 1.00

Time .980 1.00

Sets .905 .971 1.00

Documents Viewed .844 .932 .983 1.00


0

20

40

60

80

100

120

140

0 100 200 300 400 500

Sessions

Tim

e

0

200

400

600

800

1000

1200

1400

1600

1800

0 20 40 60 80 100 120 140

Tim e

Sea

rch

es

0

200400

600800

10001200

14001600

1800

0 100 200 300 400 500

Sessions

Sea

rch

es

Correlation Matrix of Use Metrics Available for SilverPlatter Files

Pearson r for Sessions, Connect Time, Searches, Documents Viewed

94 cases

Sessions Time Searches Abs. Viewed

Sessions 1.00

Time .975 1.00

Searches .899 .901 1.00

Abstracts Viewed .840 .870 .855 1.00


Are we choosing the right information sources for our audiences?

…optimizing the delivery of electronic information?

…making access as easy and seamless as possible?

…spending our dollars wisely?

…able to detect and respond to change in the patterns of resource use?

High Resolution Data + User Input + Good Program Liaison and Knowledge

Support Resource Management, and Inform Basic Questions, e.g.:


Using the Architecture of the Web to Increase Data Resolution

www.library.upenn.edu/facilities/count_use.html


dial-123-130.dial. indiana.edu - - [04/Feb/2001:00:17:38-0500] "GET/special/photos /theater/505.html HTTP/1.0" 200 3086 "http://www.library. upenn.edu/special/photos/theater/504.html" "Mozilla/4.7C-CCK-MCD {C-UDP; EBM-APPLE} (Macintosh; I; PPC)” recrawler 1.bos2.fastsearch.net - -[04/Feb/2001:00: 18:21- 0500] "GET /etext/ sasia/skt-mss/1549 /15a.html HTTP/1.0" 200 2736 "-" "FAST -WebCrawler/2.2-pre27 (crawler@ fast.no; http://www .fast.no/faq/ faqfastweb

search/faqfastwebcrawler.html)" 130.91.196.245.in-addr.arpa--[04/Feb/2001:00 :17:40 -0500] "GET /facilities/count_use.html?resource =ABI/Inform %20 %20Ovid &method= Ovid&url=http:// www.abi-ovid.library.upenn.edu/ovid web/ovidweb.cgi? T=JS& PAGE =main&MODE=ovid& D=infoz HTTP/1.1" 200 2039 "http://www.library.upenn.edu/webbin5/resources/ databases.cgi? business" "Mozilla/4.0 (compatible; MSIE 5.5; Windows NT 4.0)” 203.197.226.240 - - [04/Feb/2001:00:17:41 -0500] "GET /etext/sasia/aiis/architecture /khajuraho/010.html HTTP/1.0" 200 4427 "http://www. library.upenn.edu/etext/ sasia/ aiis/architecture/ khajur aho/" "Mozilla/4.7 [en] (Win95; I)” 203.197.226. 240- -[04/Feb/200 1:00:17:44 -0500] "GET /images/banner. gifHTTP/1.0" 404 2814 "http://www.library. upenn. edu/etext/sasi a/aiis/architecture /khajuraho/010.html" "Mozilla /4.7 [en] (Win95; I)"pub237.lib.upenn.edu - - [04/Feb/ 2001:00:17:48 -0500] "GET / HTTP/1.0" 200 8070 "-" "WebTrends Alert” dial-123-130.dial. indiana.edu - - [04/ Feb/2001 :00:18:02 -0500] "GET /special/ photos/ theater/504.html HTTP/1.0" 200 3247 "http://www.library.upenn. edu /special/photos/ theater /503.html" "Mozilla/4.7 C-CCK MCD {C-UDP; EBM-APPLE} (Macintosh; I; PPC)” dialin1085. upenn.edu--[04/Feb/ 2001:00:18: 04 -0500]"GET/facilities/count_use.html?resource=China%20Economic%20 Review& method= ejs& url= http://www.sciencedirect.com/ science/journal/ 1043951XHT TP/1.0" 200 2027 "http:// www.library.upenn.edu/webbin 5/ resources/ejspubl ic5.cgi?homepage=http:// www. library.upenn.edu/lipp incott/&community= Business" "Mozilla/ 4.0 (compatible; MSIE 5.0; Windows 98; DigExt; SPIKE 5)” 203.197. 226.240 - - [04/Feb/2001:00:18:07 -0500] "GET /etext/sasia/aiis/ architecture/khajuraho/ 010a.jpg HTTP/1.0" 200 89117 "http://www.library.upenn.edu/etext/sasia/ aiis/arch itecture/khajuraho/010.html" "Mozilla/4.7 [en] (Win95; I)”

Beginning with a stream of unprocessed log data...


Æ |http://www.uqtr.uquebec.ca/AE/index.html|World||||History of Art|F-T|No|07-16-1999 : 11:11|10-25-2000 : 11:30||ABA Bank Compliance|http://proquest.umi.com/pqdlink?Ver=1&Exp=07-01-2003&REQ=3&PUB=14954&Cert=0CEccdp7 aMS6kuCDmdhPNL%2bQ2tTOLTrDEHAz%2bYmHN172RUqZPCJ2SvATX%2bFGA7htIYkVlFVWSyawE0NvKlpBZ%2bO%2f%2bLEWBnchnwLT9%2b%2fdGGHSlx0PO3dxUQd3g2S9QP2FghKaQ2ncl5EdDKBum2vykhvxsyRQutjuMGKfxAKHOA4-|Penn|ABI/Inform|||Business,Finance|F-TPI| No|03-13-2001: 00:01|03-14-2001 : 11:31|mw|ABA Journal|http://proquest.umi.com/pqdlink?Ver=1&Exp=07-012003&REQ=3&PUB=27585&Cert=PfySiFXf1 0i6kuCDmdhPNL%2bQ2tTOLTrDEHAz%2bYmHN172RUqZPCJ2SvATX%2bFGA7ht1pGvDP%2bFxrGwE0NvKlpBZ%2bO%2f%2bLEWBnchnwLT9%2b%2fdGGHSlx0PO3dxUQd3g2S9QP2FghKaQ2ncl5EdDKBum2vykhvxsyRQutjuAyIsegc4Y7Y-|Penn|ABI/Inform|||Finance|F-TPI|No|03-13-2001: 00:01||mw|

ABI/Inform |http://www.umi.com/pqdauto|Penn||||Biomedical Research,Management,Business,Clinical Medicine,Clinical Medicine,Nursing, Econo mics, Health Care Policy & Management| F-TSDb|No|07-16-1999 :11:11|02-09-2001 12:14||

…and information culled from databases that generate our Web pages...


…to extracting, parsing, storing, and mining for significant content.

hrn1117.resnet-student.upenn.edu4-Feb-01 17:25:31 Music Index Online | | |

dhcp004.lib.upenn.edu 4-Feb-01 17:26:54 LEXIS/NEXIS Academic Universe | | |

node.uphs.upenn.edu 4-Feb-01 17:27:13 Internet Grateful Med | | |

biomed07.med.upenn.edu 4-Feb-01 17:27:44 MEDLINE | | |

janus.siast.sk.ca 4-Feb-01 17:30:18 Design Studies |ScienceDirect |05,24,40,68 |F-TPI

lrsm228pc2.lrsm.upenn.edu 4-Feb-01 17:30:29 Physical Review B | |10,55 |F-TPI

sub200141.colstate.edu 4-Feb-01 17:32:15 Representations |JSTOR |05,14,25,29,31,36,38,64|F-TPI

ref124.lib.upenn.edu 4-Feb-01 17:36:05 British Journal of Aesthetics |Oxford |54 |F-TPI

client-151-200-195-5.silvmul.com4-Feb-01 17:37:20 Dow Jones Interactive | |08,09,13,20,27,32,37,51,56,59,64,70|F-TSDb

janet.sas.upenn.edu 4-Feb-01 17:39:28 MEDLINE | | |

130.91.196.245 4-Feb-01 17:40:43 ABI/Inform |Ovid |08,44,09,13,20,27,32,37,51,56,59,64,70|F-TSDb

198.84.16.26 4-Feb-01 17:41:48 MEDLINE | | |

ip-20-12-201.philly-t.navipath.net4-Feb-01 17:44:52 Dow Jones Interactive | |08,09,13,20,27,32,37,51,56,59,64,70|F-TSDb

t13pc.seas.upenn.edu 4-Feb-01 17:48:10 MEDLINE | | |

dialin1085.upenn.edu 4-Feb-01 18:04:05 China Economic Review |ScienceDirect |09 |

dhcp066.lib.upenn.edu 4-Feb-01 18:07:12 Anthropological Literature | | |

hil-235-113.resnet.upenn.edu4-Feb-01 18:09:08 Journal of Sociolinguistics |ECO |02,43,64 |F-TPI

dhcp082.lib.upenn.edu 4-Feb-01 20:11:30 Art Index | | |

gvdf.med.upenn.edu 4-Feb-01 21:07:12 Biochemistry |ACS |10 |F-TPI

lsw-103-197.greeknet-student.upenn.edu4-Feb-01 22:07:12 LEXIS/NEXIS Academic Universe | | |

nic-21-95.resnet.upenn.edu 4-Feb-01 23:07:12 ERIC | | |


Database Log-ins Pct Total Cost Per Login

Use of Licensed Resources

What Databases Do Our Clients Use at What Cost?

15 Most Frequently Used Index/Abstract/Full-text Databases in FY 2001

MEDLINE 205,150 22.9% $ 0.10

LEXIS/NEXIS 63,817 7.1% $ 0.42

Academic Index 52,407 5.9% $ 0.58

Dow Jones 39,828 4.5% $ 0.68

ISI Citation Indexes 39,753 4.4% $ 2.75

ABI/Inform 36,190 4.0% $ 1.09

PsycINFO 27,636 3.1% $ 0.89 Investext 17,695 2.0% $ 0.68

Business & Industry 16,797 1.9% $ 0.55

CINAHL/Nursing 16,232 1.8% $ 0.36

PubMed 15,610 1.7% $ -

MLA International 13,359 1.5% $ 0.41

Multex 12,196 1.4% $ 0.10

ERIC 10,852 1.2% $ 0.54

EconLit 8,940 1.0% $ 0.80

Hoovers Online 8,905 1.0% $ 0.22

Inter Bibliog Soc Science 8,152 0.9% $ 0.38

Sociological Abstracts 7,703 0.9% $ 1.58

S&P Industry Surveys 7,346 0.8% $ 0.63

D&B Million $ Database 6,376 0.7% $ 1.74

All others 894,416 100.0%


Science 4,232 1.5% 3,114 1,057

Nature 4,081 1.4% 2,880 1,173

Journal of Biological Chemistry 2,408 0.8% 1,883 519

Journal of the American Chemical Society 2,405 0.8% 2,153 247

New England Journal of Medicine 1,994 0.7% 1,359 620

Angewandte Chemie (international edition) 1,836 0.6% 1,665 167

Journal of Organic Chemistry 1,660 0.6% 1,504 150

Proceedings of the National Academy of Sciences 1,608 0.6% 1,246 360

Tetrahedron Letters 1,361 0.5% 1,218 143

Organic Letters 1,308 0.5% 1,208 99

Proceedings of the National Academy of Sciences, U.S. 1,285 0.5% 1,017 266

Journal of Molecular Biology 1,060 0.4% 850 210

JAMA: The Journal of the American Medical Association 1,023 0.4% 650 352

Journal of Chemical Physics 992 0.3% 819 172

Journal of Finance 887 0.3% 423 378

Lancet 867 0.3% 637 227

American Journal of Sociology 860 0.3% 384 373

Medicine 849 0.3% 580 263

Applied Physics Letters 834 0.3% 751 83

Physical Review B 826 0.3% 727 98

What Are the High Use E-Journals, Data for FY2001

Title Log-ins Pct Total Log-ins Log-ins

On Campus Off Campus



ScienceDirect 139,727 27.1% $0.63

ECO 70,730 13.7% $0.09

JSTOR 48,668 9.4% $0.35

Wiley 38,255 7.4% $0.09

ACS 31,865 6.2% $0.12

Ideal 30,568 5.9% $5.51

Blackwell/Munksgaard 28,940 5.6% $0.27

Journals@Ovid 26,982 5.2% n/a

Oxford 14,819 2.9% $0.20

SpringerLINK 13,507 2.6% n/a

ABI/Inform 12,785 2.5% $3.08

Project Muse 11,438 2.2% $1.22

AIP 7,873 1.5% $5.01

Cambridge 7,835 1.5% n/a

Annual Reviews 7,215 1.4% $0.08

IEEE 7,132 1.4% $6.73

RSC 5,661 1.1% n/a

Others† 11,451 2.2%

Total 515,451 100%

† 11 publishers

Publisher Log-ins Pct of Total Cost Per Login

How Much Bang Do We Get on the Dollar For E-Journals?

E-Journal Subscription Costs Per Log-In, FY2002 (July-April)



0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

PCT of Titles

PC

T o

f Use


How Does Use Scatter Across Databases

Use Measured in Log-ins for FY 2001


0

5

10

15

20

25

30

35

40

45

50

55

ASCSSW

MED

NUR

WHRT

VET

ADMSEAS

SASGSE

GSFA

DENTALLA

W†

Log-

ins

Per

Cap

ita

†Does not include resources licensed by the Law Library for Law school affiliates


How Does Database Use Distribute By Communities?

School and Center Domains

Per Capita Use of Databases by Penn’s Schools and Centers, FY 2001

9.3%Dorms

0.5%Law

0.9%Dental

1.0%Fine Arts

1.3%Commnctn

1.8%Social Wrk

1.4%Education

2.5%Veterinary

3.5%Nursing

4.4%Enginrng

4.8%Admin

12.3%Wharton

12.8%In-Library

20.4%Arts & Sci

23.2%Medicine

School Pct of Log-ins

Database Use by Penn’s Schools & Centers


Human. Life Social Business Physical TotalScience Science Science

Administration 21.1% 36.5% 13.9% 07.0% 21.6% 100.0%Wharton 02.9% 74.3% 03.2% 19.2% 00.5% 100.0%Annenberg 15.2% 32.1% 42.3% 08.9% 01.5% 100.0%Medical 02.3% 86.0% 01.9% 01.0% 08.8% 100.0%Dental 01.8% 87.7% 08.9% 00.2% 01.4% 100.0%Veterinary 01.7% 96.0% 00.6% 00.4% 01.3% 100.0%Dialin 08.5% 63.2% 09.9% 15.4% 02.9% 100.0%Education 24.6% 13.1% 61.5% 00.8% 00.0% 100.0%Fine Arts 29.0% 18.5% 45.7% 5.6% 01.2% 100.0%Law 13.0% 26.6% 20.9% 37.0% 02.4% 100.0%Library 21.3% 54.8% 09.1% 08.5% 06.3% 100.0%Nursing 15.9% 73.1% 07.8% 03.2% 00.0% 100.0%Student Residences 18.9% 57.0% 12.6% 09.0% 02.5% 100.0%Arts and Sciences 08.2% 26.3% 5.7% 09.9% 49.9% 100.0%Engineering 0 1.5% 29.5% 2.3% 01.2% 65.6% 100.0%Social Work 20.6% 29.1% 41.6% 06.1% 02.7% 100.0%Unresolved 18.9% 44.7% 17.8% 10.0% 08.6% 100.0%

Total 14.7% 50.7% 11.9% 8.6% 14.1% 100.0%

Network Domain

Subject focus

Database & E-Journal Log-ins by Subject (based on log samples from FY2001)



On-Campus Depts50%

In-Library25%

Off-Campus15%

Campus Residences10%


Where Do Our Clients Access Information?

Database Log-ins by Domain, FY2001


Database Log-ins from Off Campus as a Percent of Total Log-ins, FY2001

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

90.0%

100.0%

SSWGSE

WHRT

NURSSAS

DENTL

GSFA

SEASASC

MED

LAW

ADM†

VET

On Campus

Off-Campus

School or Center


Where Do Communities of Clients Work?

Pct

. of L

og-in

s


When Are They Working?Use of Licensed Resources

Database Use by Time of Day, FY2001

0

5000

10000

15000

20000

25000

12

-1A

M

1-2

AM

2-3

AM

3-4

AM

4-5

AM

5-6

AM

6-7

AM

7-8

AM

8-9

AM

9-1

0 A

M

10

-11

AM

11

-12

AM

12

-1 P

M

1-2

PM

2-3

PM

3-4

PM

4-5

PM

5-6

PM

6-7

PM

7-8

PM

8-9

PM

9-1

0 P

M

10

-11

PM

11

-12

PM

Att

empt

ed L

ogon

s

In-Library Student Houses Schools Campus Modem Pool


0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

12-1

AM

1-2

AM

2-3

AM

3-4

AM

4-5

AM

5-6

AM

6-7

AM

7-8

AM

8-9

AM

9-10

AM

10-1

1 A

M

11-1

2 A

M

12-1

PM

1-2

PM

2-3

PM

3-4

PM

4-5

PM

5-6

PM

6-7

PM

7-8

PM

8-9

PM

9-10

PM

10-1

1 P

M

11-1

2 P

M

Per

cen

t o

f T

ota

l Lo

go

ns

for

Eac

h H

ou

r

In-L ibrary Student Houses Schools Campus Modem Pool All Other (non-Penn domains)

The points on this line total 100% of logons for the hour 5-6 AM . In this example, "All Other Domains" accounted for 46.6% of attempted log-ons during the given hour.

Network Domains:


How Does Audience Composition Change Through the Day?

Database Use by hour, FY2001


The Data Farm Experiment: Tools That Serve Information Access Can Also Serve Measurement


Schematic of the Data FarmAs of May 2002


To Demonstrate Accountability:

Is the library spending the Schools’ money effectively? (Pressures of Penn’s responsibility center budget environment)

To Understand and Describe the Transfer of Technology:

Is the academic information universe a digital universe (as some at Penn believe)?

Is the digital universe more cost efficient than the paper one (as some at Penn believe)?

To Guide the Improvement of Existing and the Development of New Services

To Ensure the Successful Fulfillment of Our Mission

Why Are the Data Important?

“If you don’t know where you’re going, you’ll probably end up somewhere else” - Casey Stengel


Through the Bytes Darkly,

Management Information and the Digital Library

Joe Zucca University of Pennsylvania Library [email protected]