assessing the value of e-resources to york university faculty using the mines for libraries...

This article was downloaded by: [Universite Laval]On: 12 July 2014, At: 03:12Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Web LibrarianshipPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/wjwl20

Assessing the Value of E-Resourcesto York University Faculty Using theMINES for Libraries Protocol: An EvolvingLandscapeAaron Lupton a & Catherine Davidson aa Scott Library , York University , Toronto , Ontario , CanadaPublished online: 04 Dec 2013.

To cite this article: Aaron Lupton & Catherine Davidson (2013) Assessing the Value of E-Resources toYork University Faculty Using the MINES for Libraries Protocol: An Evolving Landscape, Journal of WebLibrarianship, 7:4, 422-433, DOI: 10.1080/19322909.2013.839849

To link to this article: http://dx.doi.org/10.1080/19322909.2013.839849

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/loi/wjwl20

http://www.tandfonline.com/action/showCitFormats?doi=10.1080/19322909.2013.839849

http://dx.doi.org/10.1080/19322909.2013.839849

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

Journal of Web Librarianship, 7:422–433, 2013© 2013 Crown CopyrightISSN: 1932-2909 print / 1932-2917 onlineDOI: 10.1080/19322909.2013.839849

Assessing the Value of E-Resources to YorkUniversity Faculty Using the MINES

for Libraries Protocol: An Evolving Landscape

AARON LUPTON and CATHERINE DAVIDSONScott Library, York University, Toronto, Ontario, Canada

This article describes how York University analyzed the results fromits 2010–11 implementation of the Measuring the Impact of Net-worked Electronic Services for Libraries survey to measure value ofinvestments in e-resources. The results were assessed through twovariables: e-resources used by faculty when carrying out fundedand non-funded research, and e-resources ranked by faculty asbeing important in their specific field. The article also includes adescription of work currently underway to assess e-resource datathrough other lenses, such as effects on student success.

KEYWORDS usage study, database use, consortia purchases,MINES

INTRODUCTION

Measuring the Impact of Networked Electronic Services (MINES) for Librariesis an online, transaction-based, point-of-use intercept Web survey method-ology, in use since 2000. MINES collects data on the purpose of use ofelectronic resources and on the demographics of users.

In 2004–05, the Ontario Council for University Libraries (OCUL) usedthe MINES for Libraries protocol for surveying its members and found it tobe effective (Kyrillidou et al. 2005). The OCUL directors decided to conductMINES again in 2009–10, with similar objectives:

• To capture in-library and remote Web usage of the OCUL Libraries’e-resources;

Received 9 May 2013; accepted 9 June 2013.Address correspondence to Aaron Lupton, E-Resources Librarian, Scott Library, York

University Libraries, 105B Scott Library, 4700 Keele St., Toronto, Ontario, Canada, M3J 1P3.E-mail: [email protected]

422

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014

Assessing the Value of E-Resources with MINES 423

• To identify the demographic differences between in-house library users ascompared to remote users;

• To identify users’ purposes for accessing OCUL’s e-resources;• To develop an evaluation infrastructure to make studies of patron usage

of networked e-resources routine, robust, and integrated into the decision-making process.

York University participated in both the 2004–05 and 2009–10 MINES forLibraries surveys.

LITERATURE REVIEW

A wealth of resources on MINES for Libraries is available on the As-sociation of Research Libraries (ARL) Web site (http://www.arl.org/stats/initiatives/mines/index.shtml). For a general overview of MINES origins andhow it has been used in academic and medical libraries, see “SuccessfulWeb Survey Methodologies for Measuring the Impact of Networked Elec-tronic Services (MINES for Libraries)” by Brinley Franklin and Terry Plum(2006). Franklin and Plum noted that the results of the MINES survey imple-mented in U.S. academic health sciences libraries and academic main campuslibraries can be used to inform decision-making related to providing more ac-cess to electronic resources. For example, because grant-funded researchersin health sciences are accessing e-resources for sponsored projects primar-ily from the office or lab on campus rather than in the library, e-resourcesfor these specific disciplines should be promoted more from Web-basedresearch portals than from the physical library.

Consortial coverage of OCUL implementations of MINES is quite thor-ough: Martha Kyrillidou, Terry Plum, and Bruce Thompson (2010) examinedcurrent methodological considerations with MINES and its future develop-ment. The article emphasized the value of point-of-use Web surveys in as-sessing e-resources and learning about library users, and as a complementto typical vendor-supplied usage data. Dana Thomas and colleagues (2012)described the 2009–10 OCUL implementation. In this article, initial surveyresults data were used to examine how usage of e-resources acquired viaconsortia compared to those acquired locally, how users discovered differentformats, and more. We also analyzed the mechanics of the survey and exam-ined the implications of running the survey in mandatory versus voluntarymodes as well as the effects of launching the survey through an OpenURLresolver (in this case, SFX). No one has yet published research or analy-sis that specifically addresses how the MINES survey data can be used tomeasure value of e-resources to faculty.

The concept of measuring the value of the library is currently a popularone and is increasingly becoming one of the focuses of library assessment

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014

424 A. Lupton and C. Davidson

literature. In particular, more and more research is being done on howthe library and library resources affect student success measures. One well-known example of a project designed to measure library usage’s impacton student success was conducted by the University of Wollongong Library,which found a strong correlation between students’ grades and use of libraryinformation resources (Cox and Jantti 2012). Another popular example oflibrary value research is the Library Data and Student Success project, ajoint effort between the University of Minnesota Libraries and their Office ofInstitutional Research, which started in April 2012. The project’s goals are toconnect library use and student success using university-level metrics andto “provide evidence-based data that can inform service improvements andefficiencies, the development of new services, and the allocation of resourcesfor necessary impact” (2012, n.p.).

The ARL Assess listserv featured lengthy discussions in 2012 about theseprojects and their assumptions, challenges, and merit. Finally, Joe Zucca,the Director for Planning and Communication at University of PennsylvaniaLibraries, explained the Penn Library Data Farm project’s aims are to create asystem for decision making using library data from multiple sources (2008).

We were inspired by the movement of using library data to both illustratelibrary value and to support decision making. Therefore, the MINES resultsdata were used to illustrate the value of e-resource investments and to seewhether the data could inform acquisitions decisions.

METHODOLOGY

Library patrons encountered the MINES survey while en route to accessinga full-text e-resource. The 2010 iteration of the MINES survey used SFX,the OpenURL resolver that connects users to full-text digital content fromabstract and index databases, as the delivery mechanism. The patron waspresented with a five-question survey that asked the following:

(1) Patron status: faculty, staff, grad, undergrad;(2) Affiliation: department;(3) Location: in library, on campus but not in the library, off campus;(4) Purpose of use: funded/non-funded research, teaching, coursework; and(5) Why they chose the resource: important resource in field, recommended,

reference from another source, and so forth.

When the survey was launched in 2010, the content type most represented inthe SFX knowledge base was, by far, the electronic journal. Other resources,such as e-books, data sets, print journals, audio visual, and other non-textualresources were also included but were underrepresented.

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


FIGURE 1 Screenshot of LimeSurvey results of MINES data. (Color figure available online.)

The survey was launched every nth time a user attempted to use ene-resource from the SFX menu. At York, it was launched every 250th time ane-resource was accessed. Unfortunately, York’s ethics review board did notpermit the survey to operate in mandatory mode, even though technically itwould have been possible to do so. Instead, the survey was run in optionalmode. The survey ran for twelve months from February 2010 to January2011.

Preliminary MINES survey results were first made available to all partici-pating OCUL members in July 2010 via LimeSurvey, a free, open source Websurvey tool (Figure 1). The results could be cross-tabulated and presented(e.g., “Patron Status” could be combined with “Location”).

After the conclusion of the survey, ARL provided all OCUL institutionswith a report summarizing each school’s local results. These included

• Frequency by affiliation (the discipline of the survey respondent);• Frequency by user status (faculty, grad, undergrad, or staff);• Frequency by location (in the library, on campus but not in the library, off

campus);

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


• Frequency by purpose (for what type of work was the respondent usingthe resource); and

• Frequency by reason for use (why the respondent chose that particularresource).

The report also cross-tabulated some results such as user status with purposeof use and user status with reason for use.

Each institution also received the survey results data in SPSS format fromthe Evaluation and Assessment Librarian at Scholars Portal, which includeddifferent dimensions, most notably, the specific e-resource being used whenthe survey was triggered. Using SPSS, we performed additional analysis be-yond what was included in the OCUL report, cross-tabulating not only thestandard survey results, but also the new dimensions. Using a combinationof SPSS and Excel, the following dimensions were examined for faculty re-spondents:

• Analysis of the “Research” purpose of use: What specific resources arecited as being used for both funded and unfunded research?

• Analysis of the reason e-resources were chosen: What specific resourceswere identified as being particularly important in a researcher’s area ofstudy?

By cross-tabulating the specific e-resources used with responses choosing“Important resource in my field,” we created a ranking of how often specifice-resources were being used due to their importance from the faculty mem-ber’s point of view. Equally important, we combined the responses “non-funded research” with “funded research” and cross-tabulated them with thespecific resource used. This provided us with a ranking of which resourceswere being used most frequently in the research process. For York UniversityLibraries, it is critical to gain a better understanding of the extent to whichour licensed e-resources support faculty research.

The timing of this analysis coincided with larger institutional goals fo-cused on building research intensity at York. Knowing which e-resourceswere being used in research projects provided evidence of value of thoseresources. Additionally, it allowed us to note which of those resources de-termined to be important or valuable in the research process were acquiredvia consortia, thus providing evidence of value in consortia negotiations.

RESULTS

The York MINES survey resulted in 1,884 responses for a response rateof approximately 4 percent. The most frequent survey respondents were inhealth/nursing (23 percent) and social sciences (24 percent). Undergraduatesmade up the majority of responses (70 percent), which is logical given York’s

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


large undergraduate population. The majority of respondents were from offcampus (75 percent), which again is an expected response given that Yorkis largely a commuter campus and its large FTE (52,000) means that onlyso many students can make use of resources while physically in the library.Frequency by purpose revealed coursework/assignments as the number oneuse of e-resources, which makes sense given the large number of under-graduate responses. Finally, the reason of use most frequently selected was“important resource in my field” (ARL 2011, 7).

A total of 79 faculty members responded to the survey. ARL’s crosstabulation of user status and location revealed that a majority of faculty usee-resources from off campus (66 percent) as opposed to from their offices(27 percent) or the library (8 percent) (Figure 2).

Additionally, we looked at the type of resources that were used bysurvey respondents. As shown in Figure 3, the vast majority of resourcesused were e-journals (85 percent). Therefore, when discussing additionalanalysis of the MINES survey data and when drawing conclusions from thatdata, one should keep in mind that the results are really talking about usageof e-journals; measurement of usage of other e-resource types in the MINESsurvey results was negligible. This does not mean that other format typesare not being used; rather, it is directly attributable to that fact that in thisinstance of MINES, the survey was linked to the OpenURL resolver SFX, andthe SFX knowledge base at that time was primarily populated by e-journals.

FIGURE 2 York University MINES results: location of faculty using e-resources (n = 79).(Color figure available online.)

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


FIGURE 3 York University MINES results: resource types being measured (n = 1,872). (Colorfigure available online.)

Measuring Value: Primary Purpose of Use

One of the questions asked in the MINES survey was “primary purpose forusing this electronic resource.” As mentioned above, the bulk of responsesat York were related to coursework (72 percent), followed by non-fundedresearch (12 percent), then funded research (7 percent) (ARL 2011, 7). Theseresponses were likely a reflection of the respondents’ status, because a major-ity of them were undergraduates (70 percent) (ARL 2011, 6) who are typicallynot involved in in-depth research projects outside of regular coursework.

When the data were cross-tabulated to examine the purpose of use withuser status, it was revealed that of the faculty who responded to this question,37 percent indicated they were using e-resources for non-funded research,while 27 percent indicated they were using e-resources for funded research.This compares to 25 percent of faculty using e-resources for purposes relatedto teaching (ARL 2011, 14) (Figure 4). This response was perhaps a reflectionof York’s strategic focus on research intensification, and it prompted us tofurther examine the York data to learn more about what e-resources facultywere using for research and how they used them.

SPSS was used to cross-tabulate Primary Purpose of Use with the actualresource chosen when the survey was triggered; thus, we were able toview those e-resources that were chosen when faculty indicated that theywere performing funded and non-funded research. As shown in Figure 5,the top e-resources that were indicated as being used in funded and non-funded research were Elsevier (20 percent), Sage (12 percent), EBSCO (8

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


FIGURE 4 York University MINES results: faculty purposes for using e-resources (n = 79).(Color figure available online.)

FIGURE 5 York University MINES results: resources used by faculty for research (n = 49).(Color figure available online.)

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


percent), and InformaWorld (8 percent). These types of data were usefulbecause they provided additional contextual information (i.e., which specificresources were useful and for what purpose they were used). York UniversityLibraries’ Strategic Plan for 2012–15, New Horizons for the Digital Age, reflectsthe institution’s larger focus on research intensification through its theme of“Advancing York’s Research Culture and Reputation.” Thus, investments inspecific e-resources faculty use for research contribute to fulfilling that aspectof the strategic plan.

It is also worth noting that the e-resources being used for research areprimarily acquired through two consortia, Canadian Research KnowledgeNetwork (CRKN) and the OCUL. This additional analysis of the MINES dataprovides evidence of use of big-deal packages (e.g., Elsevier) that can beused to illustrate the value of negotiation efforts by CRKN and OCUL toobtain these packages.

Measuring Value: Reason Resource Was Chosen

Another question asked in the MINES survey was “Why did you choose thisresource?” As mentioned previously, the most common response was thatit was an important resource in the user’s field (59 percent), followed byreference/citation from another source (23 percent), and then recommendedby a professor/colleague (18 percent) (ARL 2011, 7).

In cross tabulating “Why did you choose this resource?” with user status,the pattern was true with faculty to an even greater extent (69 percentcompared to the 59 percent noted above) (Figure 6).

FIGURE 6 York University MINES results: the reason faculty chose an e-resource (n = 77).(Color figure available online.)

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


In cross tabulating “Why did you choose this resource?” with the actualresource chosen when the survey was triggered, we were able to see e-resources identified by faculty as an important resource in their field. Thetop e-resources indicated as being important were Elsevier (15 percent),Wiley Blackwell (11 percent), ProQuest (11 percent), and InformaWorld (10percent). While it would be incorrect to conclude that the percentage ofresponses indicated relative value of e-resources to each other (i.e., Elsevierwas more valuable than Informaworld), it does indicate that these resourceswere of high value to York’s faculty and contributed directly to the workthey performed.

This kind of analysis could be extended to a more granular level as timeand resources permit. For example, it would be possible to examine which e-resources were ranked highly within each discipline. Doing so would provideadded depth to the measure of value being prescribed and would also allowthe library to indicate which programs it is supporting the most, and in whichones it could invest more resources.

LIMITATIONS

With 1,884 total responses, the response rate of the MINES survey at York waslow (4 percent), which is something that must be taken into account wheninterpreting the results of the survey and making conclusions about whate-resources were being used most often and for what purpose. Similarly, thefaculty response rate (n = 79) represents only 5 percent of total faculty in2011. As a result, this specific set of results may not be valid for drawingconclusions about faculty’s view of resources. However, the fact that theresponses mirrored our expectations related to perceived value of resources(i.e., large packages of e-journals by reputable publishers as opposed tosmaller more niche products), leads to us to believe that the results arereflective of the York population, including faculty.

A second limitation is the MINES survey’s use of SFX as the deliverymechanism. Because e-journals make up the majority of e-resources in-cluded in the SFX knowledge base, the vast majority of responses camefrom users accessing an e-journal. Therefore, large packages of e-journals,such as Elsevier, have the potential to skew the data since they representa large proportion of e-journal articles held at York and therefore, by de-fault, rank high in terms of frequency of use. Additional analysis would berequired to determine whether a specific journal publisher is preferred overanother for research.

Finally, the optional nature of the survey may influence the type of per-son that responded. Self-selected survey participants are different from thegeneral population. For example, it is possible that optional survey respon-dents are by their nature more involved in research, which could explain the

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


high number of times faculty said they were using the resource for researchpurposes (as opposed to teaching).

CONCLUDING OBSERVATIONS: CURRENT PLANS

In digging deeper into the institutional MINES data, we found results thatwent beyond the qualitative and quantitative information provided in theinitial results from ARL and were able to elicit a more nuanced view of howa specific user group (in this case, faculty) used and valued the e-resources.The exercise provided evidence to confirm previously held assumptionsabout which e-resources were ranked highly by faculty. It also allowed usto assign a quantitative value to licensed e-resources; the analysis offersa ranking of e-resources that are most important to faculty. The fact thatwe could see which of those e-resources are licensed by consortia is alsoimportant, since it supports the efforts by organizations such as CRKN andOCUL to obtain those resources at an advantageous price.

York University Libraries’ Assessment Committee is working toward itsown Wollongong-style student impact project. The committee has consid-ered a variety of library measures, with analysis of EZproxy logs emergingas a logical starting point. Of course, if a user logs into an e-resource viaEZproxy that means they are off-campus, but as the data from MINES in-dicates, approximately 70 percent of e-resource usage is from off-campus,so this would still be a large sample. The committee held meetings withYork’s Office of Institutional Research Analysis, who plan to work with theLibraries’ computing department to link the EZproxy logs to specific user IDsand GPAs while maintaining user confidentiality in the process. The goal ofthe project will be to identify links between usage of library resources andstudent success, as defined by GPA.

The analysis performed using MINES supports the assertion that usagedata can be used to illustrate the value of the library to institutional success.By showing how faculty value e-resources and use them in research, thelibrary can quantify how e-resources contribute to the work faculty perform,and by extension, to the success of the university. The possibility of con-ducting similar analysis on other formats is also appealing. To date, MINESresults reflect primarily on e-journal usage. Being able to conduct similaranalysis for e-books and other digital resources by the same measures of pa-tron status, affiliation, location, purpose of use, and reason for their choicewould be the logical next step.

For the next iteration of MINES, other formats should be taken intoconsideration, but perhaps the survey can collect data on specifically whichtitles were being used when the survey was being triggered. For example, aprovider such as ProQuest has many specific packages and titles. Determin-ing which titles are being used and ranked as “important” by faculty would

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014


be a useful bit of knowledge indeed, not only for collections budgetingand negotiations, but also in demonstrating the value provided by librariesof acquiring the best and most useful resources. If the libraries are able todetermine what resources are most valuable in each discipline, the librarieswould be able to provide measures of how well we are meeting faculty ex-pectations relating to resources in specific disciplines as well as to highlightareas for growth in e-resource support.

REFERENCES

Association of Research Libraries. 2011. “Measuring the Impact of Network ElectronicServices and the Ontario Council of University Libraries’ Scholars Portal: FinalReport York University 2011.”

Cox, Brian, and Margie Jantti. 2012. “Discovering the Impact of Library Use and Stu-dent Performance.” http://www.educause.edu/ero/article/discovering-impact-library-use-and-student-performance.

Franklin, Brinley, and Terry Plum. 2006. “Successful Web Survey Methodologies forMeasuring the Impact of Networked Electronic Services (MINES for Libraries).”IFLA Journal 32 (1):28–40.

Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. 2005. “The Storybehind the Numbers: Measuring the Impact of Networked Electronic Services(MINES) and the Assessment of the Ontario Council of University Libraries’Scholars Portal.” Paper presented at the 6th Northumbria International Con-ference on Performance Measurement in Libraries and Information Services,Durham, England, August 23.

Kyrillidou, Martha, Terry Plum, and Bruce Thompson. 2010. “Evaluating Usage andImpact of Networked Electronic Resources through Point-Of-Use Surveys: AMINES for Libraries Study.” The Serials Librarian 59:159–83.

Thomas, Dana, Catherine Davidson, Martha Kyrillidou, and Terry Plum. 2012. “Mea-suring Use of Licensed Electronic Resources: A Second Iteration of the MINESfor Libraries Survey on Scholars Portal and Other Resources for the OntarioCouncil of University Libraries.” Library Management 33 (6/7):374–88.

University of Minnesota Libraries. 2012. “University of Minnesota Library Data andStudent Success Project.” http://blog.lib.umn.edu/ldss/.

Zucca, Joe. 2008. “Building Frameworks of Organizational Intelligence.” Presentedat ARL Assessment Conference, Seattle, WA, August 4.

Dow

nloa

ded

by [

Uni

vers

ite L

aval

] at

03:

12 1

2 Ju

ly 2

014

assessing the value of e-resources to york university faculty using the mines for libraries...

Documents