Sue SamsoHea

ServiceThe Un

bsSebastian

FiServices C

The Un

bsebHolly EggBusiness

Bibliog

The Un

bholl

476 The Jou

Networked Resources, Assessment and CollectionDevelopment

by Sue Samson, Sebastian Derry, and Holly Eggleston

Available online 18 September 2004

This project provides a criticalevaluation of networked

resources as they relate to thelibrary’s collection

development policy, identifiesareas of the curriculum not well

represented, establishes areliable method of assessing

usage across all resources, anddevelops a framework of

quantitative data for collectiondevelopment decision making.

n is Associate Professor, andd, Information and Researchs Division, Mansfield Library,iversity of Montana-Missoula,MT 59812-9936, United Statesue.samson@umontana.eduN;Derry is Assistant Professor,

ne Arts Libararian, and Mediaoordinator, Mansfield Library,iversity of Montana-Missoula,MT 59812-9936, United Statesastian.derry@umontana.eduN;leston is Assistant Professor,Librarian, and Interim Head,

raphic Management Services,Mansfield Library,

iversity of Montana-Missoula,MT 59812-9936, United Statesy.eggleston@umontana.eduN.

rnal of Academic Librarianship, Volume

Academic library collections havechanged dramatically during thepast decade. Reference collec-

tions, the bastion of bound volumes ofindexes of peer-reviewed research liter-ature, have been transformed into com-puter banks that offer these sameresources in the form of networkeddatabases offered via the World WideWeb. The benefits are overwhelming:desktop access from any computer withInternet access; keyword searching; com-bined author and keyword searching;limiters that include date ranges andlanguage preferences; and, in someinstances, the inclusion of or linking tothe full-text of the articles retrieved. Thenetworked environment provides chal-lenges as well: networked resources andservices tend to be more expensive;measuring and evaluating these resourcesis problematic; and, in most libraries,networked resources operate within thecontext of traditional services.1 Theproject described in this article wasundertaken to provide a critical evalua-tion of networked resources as theyrelate to the library’s collection develop-ment policy, to identify areas of thecurriculum not well represented, toestablish a reliable method of assessingusage across all resources, and todevelop a framework of quantitative datafor collection development decision mak-ing. This project builds on and expandsthe methodology of assessing networkedresources described in the literature byother academic libraries.

METHODS AND RESULTS:

MEASURING THE NETWORKED

ENVIRONMENT AT THE

UNIVERSITY OF MONTANA

The University of Montana-Missoula isa coeducational, doctoral university.

30, Number 6, pages 476–481

There are 700 general faculty and astudent enrollment of 13,000. Similarto other academic institutions, net-worked resources at the MansfieldLibrary have expanded exponentiallyduring the past decade, growing innumbers and changing services rapidly(Fig. 1). Efforts to measure the usageof these resources was limited tovendor reports, in-house web logs,surveys, and word-of-mouth. Duringspring semester 2003, the CollectionDevelopment Team Chair initiated aproject to systematically develop a planby which data could be gathered toprovide a point of comparison amongall databases across all disciplines. Thegoal of this project was to provide anoverview of networked resources asthey relate to the collection develop-ment policy, identify areas of thecurriculum not well represented, estab-lish a reliable method to determineusage across all resources, and thedevelopment of quantitative data fordecision-making. The project wasundertaken in four stages: a compila-tion of networked resources by subjectdiscipline; an overview of costs byboth individual resource and discipline;an assessment of comparable usagestatistics for each resource; and areview of overlap in full-text journalcoverage among resources.

Curriculum Coverage

To establish an overview of theelectronic resources as they relate tothe collection development policy andto identify areas of the curriculum notwell represented, subject bibliographerswere asked to review the databasesand identify those that were consideredprimary tools within their subjects.This methodology built on the efforts

Figure 1The Number of Subscription Networked Resources Available from the

Mansfield Library Website, 1997 to 2004

Figure 2This Sample Shows the Number of Subscription Networked Resources

Considered Primary for a Random Selection of Curricular Areas DuringAcademic Year 2002–2003

of Croneis and Dahlbach2 who usedthe Classification of Instructional Pro-grams as a base for their categories.The framework for this selection wasbased on the joint efforts of theCollection Development Team in thedesign of the dynamic database thathosts the networked resources. TheWeb page that provides access to thedynamic database of networked resour-ces offers four points of access: abasic link to core multi-disciplinarysources with full-text; an alphabeticallink by database name; a keywordsearch that searches title, contentsnote, and keyword fields; and subjectcategories that list primary and relatedsources specific to a curricular area ofstudy. Within these subject categories,bibliographers identified the primarydatabases considered to be coreindexes for the curriculum. The num-bers of resources identified for eachcurricular area were then tabulated(Fig. 2).

Cost Comparisons

Annual costs for each networkedresource were also analyzed. This wasaccomplished in two ways. First, eachelectronic resource was tabulated with itsannual cost. This was interestingly com-plicated. Certain resources are purchasedin packages, others on a per-search basis,many depend on consortial arrange-ments, and others reflect collaborativepurchasing agreements within andamong departments. Identifying all ofthese unique elements in a spreadsheetand tabulating the total annual costs on a

bar chart provided another step in theanalysis.

The second analysis of the annual costscombined elements from the curricularanalysis described above. Costs for eachnetworked resource identified as primaryin a particular subject category weretotaled. This provided an overview ofthe distribution of acquisition dollarsacross the disciplines. Costs for multi-disciplinary databases considered primaryfor multiple subjects were divided by thenumber of disciplines and distributed.

These two analyses of costs providedanother aspect to the overview.

‘‘The most complicated aspectof this analysis was in the

compilation of usage statistics.’’

Usage Statistics

The most complicated aspect of thisanalysis was in the compilation of usagestatistics. For this analysis, usage wasdefined as a single access to a specificdatabase. The target was to obtain com-parable data that would be a generalindicator of the use of each resource.The System Administrator had this exactdata for a portion of the databases. Thesedata are obtained by using EZproxy and aclick-through method of counting accessto each resource. When a user clicks on aproxied link to one of the resources, aunique entry is recorded into the proxyserver log file. This entry records the timeof the request, the specific resourcerequested, and the requesting computer’sIP address along with some other non-relevant information. Statistics are gener-ated by running a small script against thelog files to format them into CommonLog Format (CLF). The CLF file is then

November 2004 477

Table 1Overlap of Full-Text Journal Coverage Between a Random Selection from

Approximately 65 Databases that Include Full-Text

ProviderNumber ofduplicates

Total numberof Titles

Percent oOverlap

Blackwell-Synergy 524 565 92.74

SIAM Journals Online 12 13 92.31

Figure 3Usage Statistics were Gathered from Both In-House and from Vendor Data for All Subscription Databases

During Academic Year 2002–2003. Usage was Defined as a Single Access to a Specific Database

This Figure Provides a Sample Comparison of Access to a Random Selection of Over 110 Databases.

processed using a program called TheWebalizer to produce usage analysisnumbers. Since the analysis project wasinitiated, these data are now gathered forall databases and will establish a consis-tent counter method for comparing data-base use and identifying trends andpatterns. This method is not perfectbecause it does not count those userswho bookmark a direct link to a specificdatabase. However, this method is con-sistent with models used in other aca-demic libraries and provides comparabledata across all resources.3,4

In order to gather this data for thedatabases that were not yet proxied,vendor usage reports were consulted. Itwas possible to refine these usage reportsand select data that represented the accesslevel targeted for this analysis. Thisincluded reports from two primary ven-dors who defined their data differentlyfrom one another but did include the dataneeded for their suite of databases. Theseresults were tabulated for each databaseand provide a significant element in theoverview that reflects comparable use andtrends (Fig. 3).

Ingenta Select 426 469 90.38

American Physical Society 7 8 87.50

Expanded Academic ASAP 1842 2136 86.24

Full-Text Overlap

The final step in the analysis was acomparison of journal-level full-text

478 The Journal of Academic Librarianship

overlap between databases. Data wasgathered to identify the percent of titleoverlap between the approximately sixty-five subscription databases providingfull-text access to journals (Table 1). Thisoverlap comparison was further refinedby removing those databases consideredto provide unstable or incomplete accessto full-text journals. Databases thatremained in the comparison met thefollowing criteria: dates of coverageinclude access to current issues; thesearch capability of the database providestitle level and table-of-contents searching;the complete contents of a journal isincluded; and the rate that titles are

dropped and added is relatively low.Serials Solutions holdings data wasimported into an Access database forcomprehensive querying. Those data-bases that were identified to have a50% or greater overlap in full-text cover-age within the collection of stable data-base were targeted for a more detailedanalysis of specific journal coverage(Table 2). The detailed analysis was alsobased on the specific criteria listed abovebut was expanded to include the value ofthe journal as a core title for specificcurricula, dates of coverage, and therelative costs to maintain the title inmultiple electronic versions as well as

f

Table 2Databases Identified with 50% or Greater Overlap in Full-Text

Coverage were Analyzed at the Journal-Title Level

Journal Title ISSNLiteratureOnline

ProjectMuse

American Imago 0065-860X Winter 2002- 1995-

American Jewish History 0164-0178 3/1/2002- 1996-

Asian Theatre Journal 0742-5457 Spring 1998- Fall 1999-

Biography (Honolulu) 0162-4962 Spring 1998- 2000-

Callaloo 0161-2492 Fall 2002- 1995-

Children’s Literature (Storrs, CN) 0092-8208 1998- 2003-

This table contains sample data from over 65 subscription databases that contain full-text coverage.

comparative overlap between individualselected databases.

DISCUSSION

With the advent of networked environ-ments, the range of resources andaccompanying services in academiclibraries has undergone significantchange during the past decade andimpacted collection development prior-ities and acquisition budgets. In tandemwith these changing services, the need toprovide quantitative data in the frame-work of learning outcomes and substan-tive assessment that transition fromtraditional library use measures hasbecome increasingly important as aca-demic institutions compete for studentsin new market models. As networkedresources were implemented, librariesbegan to review usage statistics providedby database vendors and were immedi-ately confronted with a vast array ofincompatible statistical data.5

Efforts to standardize vendor-supplieddata have been initiated. The InternationalCoalition of Library Consortia6 developedguidelines that included five data ele-ments with implications for collectionmanagement: variability of data overtime; ratios of queries per session; hourlyuse; use of e-journal collections; and theratio of uses per FTE in a discipline; andare further discussed by Blecic, et al.7 TheAssociation of Research Libraries (ARL)initiated an E-Metrics project to developstatistics and performance measures forvendors to provide. Phase I identifiedcurrent practices among ARL librariesbased on: survey questionnaires; sitevisits to selected libraries; sample vendorreports; sample library generated reports;and follow-up interviews.8 Phase II iden-tified selected key statistics and measures

and standardized procedures and defini-tions for data collection.9–11 Anotherinitiative is Project COUNTER (CountingOnline Usage of NeTworked ElectronicResources) that is developing an interna-tional Code of Practice to govern therecording and exchange of online usagedata.12

These efforts are aimed at collaborat-ing with vendors to provide libraries withappropriate and compatible usage data.Breeding13 and Van Epps14 provide prac-tical in-house applications for librariansto document usage of electronic contentand services in the current networkedenvironment. Breeding’s approach iscomprehensive and focuses on measuringthe use of electronic and digital servicesand content within the broader scope ofthe library’s services, giving the tradi-tional services equal voice. Van Eppstargets the specific collection of usagestatistics with a quick re-direct Web-logthat provide numbers that can be com-pared to each other to determine whichresources are being used and how much.This method of data collection does notinclude users who bookmark databasesnor does it address how users search oncethey connect. However, while identifyingthe limitations of their approaches,Breeding and Van Epps emphasize theadvantages of establishing a consistentusage measurement that reflects trendsand patterns and that provides consis-tent data for comparison and decisionmaking.

Another example of a locally-devel-oped approach to collect usage data isdescribed by Duy and Vaughan15 whocompared locally collected data withvendor-provided data. They offer thatthe local tracking of data provides thebenefits of being collected in a uniformmanner for all products, collected in a

library-controlled way, and compiled in acentral location.

These local models define data collec-tion in terms of counting the number ofusers that access an electronic resourceand utilize the statistics to identify trends,to compare database use across all resour-ces, and to broadly determine use pat-terns. How patrons use the electronicresources once that have access, theefficiencies of their search strategies, andother detailed information will have tocome from vendors. To address the broadscope of service assessment, Bertot16

provides an excellent overview of meas-uring service quality in the networkedenvironment by incorporating the findingsfrom research efforts into four models ofdetermining library network statistics andperformance measures.

PROJECT OUTCOMES

This project expanded on the modelsdescribed above by not only developingbasic usage statistics for networkedresources but by adding three morelayers of quantitative data that provideda comprehensive overview of the elec-tronic resources collection. This projectwas designed to provide an overview ofnetworked resources as they relate to thecollection development policy by: 1.quantifying networked resources interms of curriculum representation, 2.identifying costs in terms of bothindividual databases and curriculum, 3.establishing a reliable method to deter-mine usage across all resources, and 4.comparing full-text journal coveragebetween databases.

‘‘The administration has usedthis data to leverage

collaborative buying powerfrom individual Schools,

Colleges, and Research andDevelopment areas directly

needing access to theseresources.’’

Library administration was initiallyinterested in the databases used mostfrequently (Fig. 3). These data providejustification for their acquisition, establisha direct link to the curriculum andresearch efforts of students and faculty,

November 2004 479

and underscore the collection develop-ment policy that targets this primaryacademic audience. The administrationhas used this data to leverage collabora-tive buying power from individualSchools, Colleges, and Research and De-velopment areas directly needing accessto these resources. These campus-wideinitiatives have been a source of publicrelations for the library and haveincreased knowledge of the value ofinformation resources in the Academy.Subject bibliographers were able to iden-tify those curricular areas under-repre-sented by networked resources, comparecosts and numbers across the disciplines,identify usage patterns of each resource,and use all of this quantitative data in theirrenewal and selection decisions. As liai-sons to curricular departments, the bib-liographers were able to share specificdata in support of the library collectiondecision-making.

For the first time, the CollectionDevelopment Team was able to reviewthe networked resources collection froman overview of quantitative data andrelate it to the collection developmentpolicy. Were all curricular areas repre-sented? Do the distribution of costs reflectusage patterns, student and faculty num-bers, and the collection developmentpolicy? More specifically, how muchoverlap of full-text journal coverage existsbetween complementary databases andcan efficiencies be realized while main-taining a high level of service?

The data collected that compared jour-nal full-text overlap between databasesprovided another access point for deci-sion-making relative to the networkedenvironment (Tables 1 and 2). Based onthese data, over $40,000 of duplicate printsubscriptions were migrated to electronicaccess. The criteria used for this decision-making also supported the decision tomaintain subscriptions to multiple inter-disciplinary databases for another yearwhile more detailed journal level analysisis completed.

With these data, it is possible tocompare number of databases, usage,and overall costs across the disciplines.These data underscore the need forfurther analysis of the electronic resourcecollection to include data elements thatcould be developed into a formula fordistribution of allocations. Electronicresource distribution needs to reflectnumber of undergraduate and graduatestudents, faculty, and programs in addi-tion to other quantifiable data that is

480 The Journal of Academic Librarianship

relevant to the collection developmentpolicy for monographs.

During the next year, the CollectionDevelopment Team is committed to con-tinuing this analysis. These data whencompared with costs, curricular coverage,and usage patterns for each databaseestablishes the foundation for effectivedecision-making.

CONCLUSIONS

Three primary conclusions evolved fromthe process to evaluate networkedresources. First, each discipline wasidentified relative to its representationwithin the networked collection both innumber of primary databases and costsfor these databases. This provided animmediate overview that could be usedin conjunction with numbers of students,faculty, and degree programs to targetthose areas of the curriculum that werenot well represented.

Second, the usage statistics provided abasis for the comparison of the use of allresources. Although not a perfect methodof counting usage, these comparable datawere used to identify the most frequentlyused databases, to compare use across allresources, and to identify trends andpatterns that exist among and betweenthese networked resources. In addition, inthe process of reviewing usage statistics,the systems administrator refined thecollection of these data so that futureassessment will be even more consistentand more easily compiled.

Third, the detailed analysis of overlapin full-text journal coverage providedcritical data for a review of potential ef-ficiencies. When used in concert with thedata provided through an analysis ofcurriculum coverage, cost comparisons,and usage statistics, these data wereessential in decision-making that madeit possible for the Collection Develop-ment Team to target potential databasesfor review while maintaining a high levelof service.

‘‘. . .the design and developmentof the assessment provides a

mechanism for continued andregular analysis.’’

In summary, the design and develop-ment of the assessment provides a

mechanism for continued and regularanalysis. In addition, the framework forthe analysis is now in place. These datacan be easily updated in the existingspreadsheets and databases for the com-pilation of regular reviews that will showchanges and trends semester-to-semesterand year-to-year. Further, the sameframework for the analysis of networkedresources should also be put in place forthe analysis of the monograph, media,and serial collections. Although data forthese collections are available to someextent, they have not been compiled inthe same way. The Collection Develop-ment Team will initiate this analysis sothat all resources can be evenly com-pared whenever possible. These numberswill be included in tables alongsidenetworked resources to establish onemore level of quantitative analysis fordecision-making.

In addition, the literature reviewidentified other academic libraries striv-ing to assess their electronic resources atthe local library level. Their experiencescorroborated our own, were used tofurther refine our methodology, andprovided a base to expand our ownassessment beyond the single elementof usage statistics to include curricularcoverage, cost comparisons, and overlapof journal coverage between databases.At the same time, national-level initia-tives to work with vendors to stand-ardize vendor-supplied data provide thefuture opportunity to further expand theanalysis to include how users actuallyuse these resources once they areaccessed.

NOTES AND REFERENCES

1. Marshall Breeding, ‘‘Strategies for Meas-uring and Implementing E-use,’’ LibraryTechnology Reports 38 (2002, May/June):1–68.

2. Karen S. Croneis & Barbara J. Dahlbach,‘‘Evaluating Electronic Database Collec-tions: Using Classification of InstructionalPrograms (CIP),’’ Poster Session presentedat the Association of College and ResearchLibraries Conference, Charlotte, NorthCarolina: April 10–13, 2003.

3. Amy S. Van Epps, ‘‘When VendorStatistics Are Not Enough: DeterminingUse of Electronic Databases,’’ Science& Technology Libraries 21(1/2)( 2001):119–126.

4. Joanna Duy & Liwen Vaughan, ‘‘UsageData for Electronic Resources: A Compar-ison Between Locally Collected and Ven-dor-Provided Statistics,’’ Journal of

Academic Librarianship 29 (2003, Janu-ary): 16–22.

5. Denise T. Covey, Usage and UsabilityAssessment: Library Practices and Con-cerns (Washington, DC: Digital LibraryFederation, Council on Library and Infor-mation Resources, 2002).

6. The International Coalition of LibraryConsortia, ‘‘Guidelines for StatisticalMeasures of Usage of Web-based Indexed,Abstracted, and Full-text Resources,’’ 12November 1998. Online. Available: http://www.library.yale.edu/consortia/webstats.html (March 7, 2004).

7. Deborah D. Blecic, Joan B. Fiscella, &Stepehn E. Wiberley Jr., ‘‘The Measure-ment of Use of Web-based InformationResources: An Early Look at Vendor-supplied Data,’’ College and ResearchLibraries 62 (2001, September): 434–453.

8. Wonsik Shim, Charles R. McClure, & JohnC. Bertot, ARL E-Metrics Project: Devel-oping Statistics and Performance Meas-ures to Describe Electronic InformationServices and Resources for ARL Libraries:Phase One Report (Tallahassee, FL: Infor-mation Use Management and Policy Insti-tute, School of Information Studies, FloridaState University, 2000). Online. Available:http://www.arl.org/stats/newmeas/emetrics/phaseone.pdf (March 7, 2004); WonsikShim, ‘‘Measuring Services, Resources,Users and Use in the Networked Environ-ment,’’ Journal of Library Administration,35(4)(2001): 71-84.

9. Shim, ‘‘Measuring Services, Resources,Users and Use in the NetworkedEnvironment.’’

10. Shim, McClure, & Bertot, ARL E-MetricsProject.

11. Wonsik Shim & Char les R.McClure, ‘‘Improving Database Ven-dors? Usage Statistics Reportingthrough Collaboration Between Libra-ries and Vendors,’’ College andResearch Libraries 63 (2002, Novem-ber): 499–514.

12. Peter Shepherd, ‘‘Keeping Count,’’ LibraryJournal 128(2)(2003): 46–48.

13. Breeding, ‘‘Strategies for Measuring andImplementing E-use.’’

14. Van Epps, ‘‘When Vendor Statistics AreNot Enough.’’

15. Duy & Vaughan, ‘‘Usage Data for Elec-tronic Resources.’’

16. John C. Bertot, ‘‘Measuring ServiceQuality in the Networked Environment:Approaches and Considerations,’’Library Trends 49 (2001, Spring):758–775.

November 2004 481