global developments in assessment: libqual+ ® and beyond… 6th international jisc/cni conference...
TRANSCRIPT
Global Developments in Assessment: LibQUAL+® and Beyond…
6th International JISC/CNI ConferenceYork, U.K.
July 6-7, 2006
Duane WebsterExecutive Director
Association of Research Libraries
Selena LockResearch and Development
Cranfield University
ARL www.arl.org
Duane Webster, Executive Director
What are some of the current developments with library assessments efforts?
ARL StatsQUAL™
E-Metrics
LibQUAL+®
DigiQUAL™
MINES for Libraries™
Where are the most critical needs and opportunities?
What are the lessons learned?
Library Assessment in an Electronic Era
www.arl.org
Duane Webster, Executive Director
Mission: Shaping the future of research libraries in the changing environment of public policy and scholarly communication.
Members: 123 major research libraries in North America.
Ratios: 4 percent of the higher education institutions providing 40 percent of the information resources.
Users: Three million students and faculty served.
Expenditures: $3.4 billion annually, $1.1 billion for acquisitions of which 31 percent is invested in access to electronic resources.
www.arl.org
Duane Webster, Executive Director
Thinking Strategically About Library Futures
• What is the central work of the library and how can we do more, differently, and at less cost?
• What important set of services does the library provide that others can’t? What new roles are needed?
• What advantages does the research library possess?
• What will be the most needed by our community of users in the next decade? How is user behavior changing?
• What should our libraries aspire to be ten years from now? What are the implications of technology driven change?
• What are the essential factors responsible for the success of the library?
www.arl.org
Duane Webster, Executive Director
Defining Success in a Digital Environment
• Crafting new measures of success.
• Moving from measuring inputs to outputs.
• Understanding impact of library roles and services.
• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.
• Reallocating and managing capabilities to focus on new definitions of success.
www.arl.org
Duane Webster, Executive Director
ARL Statistics and Measurement
www.arl.org/stats/
Duane Webster, Executive Director
.
StatsQUAL™
Duane Webster, Executive Director www.statsqual.org
Duane Webster, Executive Director
StatsHome
LibQUAL+DigiQUAL(/digiqual)
MINES(/mines)
SAILS(/sails)
E-Metrics(/emetrics)
ARL Statistics(/arlstats)
Interactive Statistics
(/interactive)?
Login
UserProfile
InstitutionProfile
www.libqual.org
SurveyManagement
StatsQUAL™
www.statsqual.org
Duane Webster, Executive Director
Updating the Traditional ARL Statistics
• E-Metrics = ARL Supplementary Statistics– On going efforts to update and refine core data.
– Exploring feasibility of collecting e-metrics.
• ARL Task Force on New Ways of Measuring Collections :– Growing concern with utility of membership index.
– Study ARL statistics to determine relevance.
– Develop Profile of Emerging Research Libraries.
www.statsqual.org
Duane Webster, Executive Director
E-Metrics: Recommended Statistics
Patron Accessible Electronic Resources (R1-3)
Use of Networked Resources and Services (U1-5)
Expenditures for Networked Resources and Related Infrastructure (C1-3)
Library Digitization Activities (D1-3)
www.arl.org/stats/newmeas/emetrics/index.html
Duane Webster, Executive Director
E-Metrics: Recommended Statistics
Patron Accessible Electronic Resources (R1-3)
• R1 – Number of electronic full-text journals
• R2 – Number of electronic reference sources
• R3 – Number of electronic books
www.arl.org/stats/newmeas/emetrics/index.html
Duane Webster, Executive Director
E-Metrics: Recommended Statistics
Use of Networked Resources and Services (U1-5)
• U1 – Number of electronic reference transactions
• U2 – Number of logins (sessions) to electronic databases
• U3 – Number of queries (searches) in electronic databases
• U4 – Items requested in electronic databases
• U5 – Virtual visits to library’s website and catalog
www.arl.org/stats/newmeas/emetrics/index.html
Duane Webster, Executive Director
Expenditures for Networked Resources and Related Infrastructure (C1-3)
• C1 Cost of electronic full-text journals
• C2 Cost of electronic reference sources
• C3 Cost of electronic books
• C4 Library expenditures for bibliographic utilities, networks and consortia
• C5 External expenditures for bibliographic utilities, networks and consortia
E-Metrics: Recommended Statistics
www.arl.org/stats/newmeas/emetrics/index.html
Duane Webster, Executive Director
Library Digitization Activities
Library Digitization Activities (D1-3)
• D1 – Size of library digital collection
• D2 – Use of library digital collection
• D3 – Cost of digital collection construction and management(Collecting these data requires staff
familiar with the digital environment.)
www.arl.org/stats/newmeas/emetrics/index.html
Duane Webster, Executive Director
A LibQUAL+® Update
• The LibQUAL+® premise, dimensions, and methodology
• LibQUAL+® results
• LibQUAL+ in action
www.libqual.org
http://www.libqual.org
LibQUAL+™
Presented by:
Selena LockSelena LockCranfield UniversityCranfield University
6th InternationalJISC/CNI Conference York, U.K. July 6-7, 2006
A A LibQUAL+LibQUAL+®® Update Update
http://www.libqual.org
A LibQUAL+® Update
The LibQUAL+® premise, dimensions, and methodology
LibQUAL+® resultsLibQUAL+® in action
http://www.libqual.org
LibQUAL+™
The LibQUAL+® premise, dimensions, and methodology
http://www.libqual.org
The need for LibQUAL+®
Underlying need to demonstrate our worthThe reallocation of resources from
traditional services and functionsRapid shifts in information-seeking
behavior• Need to keep abreast of customer demands
Increasing user demands• 37% of UK 16 – 18 year olds expect better
libraries in return for their top-up fees
http://www.libqual.org
PERCEPTIONS SERVICE
“….only customers judge quality;
all other judgments are essentially
irrelevant”
Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.
The LibQUAL+® Premise
http://www.libqual.org
Multiple Methodsof Listening to Customers
Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture*A SERVQUAL-type instrument is most suitable for these methods
Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000).
Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.
http://www.libqual.org
LibQUAL+® Development
An ARL/Texas A&M University joint developmental effort based on SERVQUAL
LibQUAL+® initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE)
Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology
Survey conducted at over 700 libraries resulting in a data base of over a million user responses
13 LibrariesEnglish LibQUAL+® Version
4000 Respondents
QUAL
QUAN
QUAL
QUAL
QUAN
QUAL
PURPOSE DATA ANALYSIS PRODUCT/RESULTDescribe library environment;build theory of library service quality from user perspective
Test LibQUAL+® instrument
Refine theoryof service quality
Refine LibQUAL+® instrument
Test LibQUAL+® instrument
Refine theory
Unstructured interviewsat 8 ARL institutions
Web-delivered survey
Unstructured interviews at Health Sciences and the Smithsonian libraries
E-mail to surveyadministrators
Web-delivered survey
Focus groups
Content analysis:(cards & Atlas TI)
Reliability/validityanalyses: CronbachsAlpha, factor analysis,SEM, descriptive statistics
Content analysis
Content analysis
Reliability/validity analyses including Cronbachs Alpha,factor analysis, SEM, descriptive statistics
Content analysis
VignetteRe-tooling
Iterative
Emergent2000
2005700 Libraries English, Dutch, Swedish,
German LibQUAL+® Versions160,000 anticipated respondents
LibQUAL+LibQUAL+® Project® Project
Case studies1
Valid LibQUAL+® protocol
Scalable process
Enhanced understanding of user-centered views of service quality in the library environment2
Cultural perspective3
Refined survey delivery process and theory of service quality4
Refined LibQUAL+® instrument5
Local contextual understanding of LibQUAL+® survey responses6
http://www.libqual.org
76 Interviews Conducted
York UniversityUniversity of ArizonaArizona StateUniversity of ConnecticutUniversity of HoustonUniversity of Kansas
University of MinnesotaUniversity of
PennsylvaniaUniversity of WashingtonSmithsonianNorthwestern Medical
LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred
http://www.libqual.org
Dimensions ofLibrary Service Quality
Information Control
LibraryServiceQuality
Model 3
Self-Reliance
Equipment
Timeliness
Ease of Navigation
Convenience
Scope of Content
Affect of Service
Library as Place
Reliability
Assurance
Responsiveness
Empathy
Refuge
Symbol
Utilitarian Space
http://www.libqual.org
LibQUAL+® Participants
12
43
164
316
207
255
0
50
100
150
200
250
300
350
2000 2001 2002 2003 2004 2005
http://www.libqual.org
Rapid Growth in Other Areas
Languages• American English• British English• French• Dutch• Swedish
In development• Chinese• Greek• Spanish• German
Consortia• Each may create 5 local
questions to add to their survey
Types of Institutions• Academic Health Sciences• Academic Law• Academic Military• College or University• Community College• European Business• Hospital• Public• State
Countries• U.S., U.K., Canada, the
Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia
Participating Libraries
World LibQUAL+® Survey 2005
http://www.libqual.org
LibQUAL+® Languages
American English
Dutch EnglishFrench Canadian DutchSwedish
Swedish(British English)
Afrikaans
DanishFinnishGerman Norwegian
British English
Continental French
Over 700 institutions 1,000,000 respondents
http://www.libqual.org
Survey Instrument – “22 items…
http://www.libqual.org
…and a Box”
Why the Box is so Important:• About 40% of participants provide open-
ended comments, and these are linked to demographics and quantitative data
• Users elaborate the details of their concerns• Users feel the need to be constructive in their
criticisms, and offer specific suggestions for action
http://www.libqual.org
LibQUAL+™
LibQUAL+® results
http://www.libqual.org
Understanding LibQUAL+®
Results
For the 22 items LibQUAL+ asks users’ to rate their:• Minimum service level• Desired service level• Perceived service performance
This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings
Perception ratings ideally fall within the Zone of Tolerance
http://www.libqual.org
Key to Bar Charts Key to Radar Charts
http://www.libqual.org
LibQUAL+® 2005 Summary Collegesor Universities (American English)
http://www.libqual.org
LibQUAL+® 2005 Summary Collegesor Universities (British English)
http://www.libqual.org
Longitudinal Analysis for Collegesor Universities (American English)
5.50
6.00
6.50
7.00
7.50
8.00
8.50
2003 2004 2005
Minimum Mean
Desired Mean
Perceived Mean
http://www.libqual.org
Longitudinal Analysis for Collegesor Universities (British English)
5.50
6.00
6.50
7.00
7.50
8.00
8.50
2003 2004 2005
Minimum Mean
Desired Mean
Perceived Mean
http://www.libqual.org
Comparisons by user groupCollege or University (British English)
5.50
6.00
6.50
7.00
7.50
8.00
8.50
2003 2004 2005
Minimum Mean
Desired Mean
Perceived Mean
http://www.libqual.org
General findings
Highly desired• Making electronic resources accessible from my
home or office• Print and/or electronic journals I require for my work• A haven for study, learning or research
Lowest • Library staff who instil confidence in users• Giving users individual attention• Space for group learning and group study
http://www.libqual.org
LibQUAL+™
LibQUAL+® in Action
http://www.libqual.org
UK University Case Study – Using LibQUAL+® Results
Strategic Service Developments• Data to support service development• Ability to identify where not meeting expectations• Measure if change has met need
Budget Discussions• Data to support bid for increased funding• Data to support case for change in emphasis (towards
e-provision)Marketing Position
• Status of the library within the University• Importance of national & international benchmarking
http://www.libqual.org
LibQUAL+® Outcomes
Funding received for:New Web Services AdministratorIncreased opening hours
• Now providing 222,578 seat hours per week
Library refurbishment programme reinstated at costs in excess of £8 million
http://www.libqual.org
Library Refurbishment
From: To:
http://www.libqual.org
Why use LibQUAL+®?Feedback from LibQUAL+® Users
“Why did you choose to use LibQUAL+®?”LibQUAL+® was recommended to us as offering
a well designed, thoroughly Library-focused set of survey tools
Opportunity to benchmarkCost-effectivenessAutomated processing & fast delivery of resultsRespectability and comparability (with others
and historically)
http://www.libqual.org
The benefits of LibQUAL+®
LibQUAL+® has enabled us to find out what a broad range of our users thought of the services we offer; what level of service-delivery quality we had achieved in their eyes, and to get a clear picture of what they actually wanted the Library to deliver (as opposed to what we thought they wanted).
UK HE Institution, 2006
http://www.libqual.org
In Closing LibQUAL+®
Focuses on success from the users’ point of view (outcomes)
Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense
Requires limited local survey expertise and resources Analysis available at local, national and inter-institutional
levels Offers opportunities for highlighting and improving your
status within the institution Can help in securing funding for the Library
http://www.libqual.org
LibQUAL+® Resources
LibQUAL+® Website:http://www.libqual.org
Publications: http://www.libqual.org/publications
Events and Training: http://www.libqual.org/events
LibQUAL+® Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib
LibQUAL+® Procedures Manual: http://www.libqual.org/Information/Manual/index.cfm
http://www.libqual.org
Acknowledgements
Fred Heath, Bruce Thompson, Colleen Cook, Martha Kyrillidou and the rest of the LibQUAL+® team
Stephen Town, Cranfield UniversityHelen Durndell and Jacqui Dowd, Glasgow
UniversityAll SCONUL LibQUAL+® Participants and the
SCONUL Working Group on Performance Improvement
http://www.libqual.org
LibQUAL+® Contact Information
Martha Kyrillidou• Director, ARL Statistics and Measurement Program
Selena Lock• Research and Development, Cranfield University
Developing the DigiQUAL™ Protocol forDigital Library Evaluation
www.digiqual.org
DigiQUAL™
• NSF Funding
• Building on the LibQUAL+® experience
• Secures feedback on user’s perceptions of library’s web site
• Five questions on services, functionality, and content
• Goal is to determine utility, reliability, and trustworthiness
Duane Webster, Executive Director
Background: ServQUAL LibQUAL+® DigiQUAL™
LibQUAL+® Dimensions of Service Quality:• Affect of Service• Information Control• Library as Place
Developing DigiQUAL™ Survey Items
DigiQUAL™12 themes of service quality:• Accessibility• Navigability• Interoperability• Collection building• Resource Use• Evaluating collections• DL as community for users• DL as community for developers• DL as community for reviewers• Copyright• Role of Federations• DL Sustainability
www.digiqual.org
Duane Webster, Executive Director
Pilot Testing Survey Items and Implementation
www.digiqual.org
Duane Webster, Executive Director
Building a Survey
• Review and select items
• Issue: aligning items to individual DL needs & users –
vocabulary and content
www.digiqual.org
Duane Webster, Executive Director
CustomizeSurvey
Issue: Flexibility vs. Standardization
Building a Survey, cont.
www.digiqual.org
Duane Webster, Executive Director
Implementing Survey – Notification Methods
Links on siteNewsletters
Next to resources
Issues: no pop-ups, no individual emailswww.digiqual.org
Duane Webster, Executive Director
Implementing Survey - Incentives
Issues: must be easily transferable, requires email address - clear IRB
www.digiqual.org
Duane Webster, Executive Director
(mis)Interpreting results from individual digital libraries in the context of other sites
Sites reluctant to sharedata and results
Analysis and Reporting - Issues
www.digiqual.org
Duane Webster, Executive Director
Outstanding Issues and Challenges
• Unique DLs: niche market, critical mass, both?
• Balance:– custom vs. generic content results– flexible vs. standard implementation scaling
• Mixed methods– Preserving user privacy– Collecting truly useful data
• Moving target: digital libraries as… it depends.
www.digiqual.org
Duane Webster, Executive Director
Assessing the Value of Networked Electronic Services:
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
www.arl.org/stats/newmeas/mines.html
The MINES survey
Duane Webster, Executive Director
• A research methodology consisting of a web-based survey form and a sampling plan.
• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.
• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.
• Different from other electronic resource usage measures that quantify total usage (e.g., COUNTER, EQUINOX, E-Metrics, ICOLC guidelines, ISO and NISO standards) or measure how well a library makes electronic resources available (LibQUAL+®, DigiQUAL™).
What is MINES?
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
• How extensively do sponsored researchers use the new digital information environment?
• Are researchers more likely to use networked electronic resources from inside or outside the library?
• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
• Are particular network configurations more conducive to studies of digital libraries patron use?
Questions Addressed
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
• A representative sampling plan, including sample size, is determined at the outset. Typically, there are 48 hours of surveying over 12 months at a medical library and 24 hours a year at a main library.
• Random moment/web-based surveys are employed at each site.
• Participation is usually mandatory, negating non-respondent bias, and is based on actual use in real-time.
• Libraries with database-to-web gateways or proxy re-writers offer the most comprehensive networking solution for surveying all networked services users during survey periods.
MINES Methodological Considerations
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Web Survey Design Guidelines
• Simple text for different browsers – no graphics.– Different browsers render web pages differently.
• Few questions per screen or simply few questions.• Easy to navigate.• Short and plain.• No scrolling.• Clear and encouraging error or warning messages.• Every question answered in a similar way - consistent.
– Radio buttons, drop downs.• ADA compliant.• Introduction page or paragraph.• Easy to read.
– Must see definitions of sponsored research. • Can present questions in response to answers.
Dillman, D.A. 2000 (December). Mail and Internet Surveys, The Tailored Design Method.
2nd Ed. New York: John Wiley & Sons. www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Library User Survey
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Data was collected at seven main campus libraries and seven academic health science libraries in the U.S., between January, 2003 and January, 2005.
Recent DataCollection Activities
Main University Libraries Academic Medical Libraries
University of Colorado University of Connecticut Health Center
University of Connecticut University of North Carolina
University of North Carolina University of Texas Medical Branch
Oregon State University University of Texas Southwestern
University of Utah University of Utah
University of Virginia University of Virginia
Washington University Washington University
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
• More than 45,000 networked electronic services uses were surveyed.
• At each library, the MINES survey was one component of a comprehensive cost analysis study that assigned all library costs to sponsored research, instruction/education/non-sponsored research, patient care, other sponsored activities and other activities.
• MINES is also being conducted by the Ontario Council of University Libraries (OCUL) in 2004-2005 in conjunction with ARL.
Recent DataCollection Activities
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Purpose of Use By Location Main Campus Libraries
2003 – 2005
Sponsored Research
Instruction
Other
Other SponsoredActivities
In the Libraryn = 7,618
On-Campus, not in the Libraryn = 6,641
All Usagen = 19,271
•72% of sponsored research usage of electronic resources occurred outside the library; 83% of this took place on campus.
Off-Campusn = 5,012
26%
3%
5%
66%
33%
2%
6%
59%
24%
2%
11%
63%
14%2%
21%
63%
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
13,930
31,451
Location of Users2003-2005
7,618
11,653
Medical Library Main Library
All Libraries
30%
40%
60%
70%
76%24%
6,312
19,798
In Library Users
Remote Users
76%24%
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Demographics by Location of User Main Libraries
On Campus, Not in the Libraryn = 6,391
Inside the Libraryn = 7,064
Off-Campusn = 4,953
Total Usersn = 18,408
20%8%
27%
45%
27%39%
12%
22%
40%3%
33%
24%
38%
62%
In the Library
Outside the Library
Graduate Students
Faculty, Staff, Research Fellows
Undergraduate Students
All Other Users
www.arl.org/stats/newmeas/mines.html
Duane Webster, Executive Director
Electronic Services Sponsored Research Use
Compared to Print Journal and Total Library Use Main Libraries
www.arl.org/stats/newmeas/mines.html
E-Resources
Print Journals
Total Use
Duane Webster, Executive Director
Where are the most critical assessment needs and opportunities?
• Complementing LibQUAL+® with additional measures.
• Developing impact studies on user success, economic value, and community return on investment.
• Moving target: what is a digital library?
• E-Resources: understanding usage.
• Gaining acceptance and use of standard measures for e-resources.
• Building a climate of assessment throughout library.
Duane Webster, Executive Director
What are the lessons learned?
• Understanding changes in users approach to information resources.
• Service quality improvement is a key factor.
• Understanding the impact of e-resources on library services - TRL.
• Learning how to compete with Google.
• Upfront investment in design and development.
• Making the assessment service affordable, practical, & effective.
• Assessment needs to be satisfying and fun.
Duane Webster, Executive Director
In Closing
• As higher education is challenged on accountability and effectiveness issues so will libraries.
• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.
• Basic questions of role, vision, and impact must be answered by library community.
Duane Webster, Executive Director
Selected References
• Kyrillidou, Martha and Sarah Giersch. “Developing the DigiQUAL Protocol for Digital Library Evaluation.” Paper Presented at JCDL - Joint Conference on Digital Libraries, Denver, CO, June 6-11, 2005. [Available at http://www.libqual.org/documents/admin/digiqual-jcdl05-v5.pdf]
• Kyrillidou, Martha, Toni Olshen, Brinley Franklin, and Terry Plum. “MINES for Libraries(tm): Measuring the Impact of Networked Electronic Services and the Ontario Council of University Libraries' Scholar Portal, Final Report.” Presented at the 6th Northumbria International Conference on Performance Measurement in Libraries and Information Services, Durham, England, Aug. 23, 2005. [Available at http://www.libqual.org/documents/admin/FINAL%20REPORT_Jan26mk.pdf]
• Franklin, Brinley and Terry Plum. "Library usage patterns in the electronic information environment" Information Research, 9(4) paper 187 (2004). [Available at http://InformationR.net/ir/9-4/paper187.html]
• Franklin, Brinley, and Terry Plum. "Documenting Usage Patterns of Networked Electronic Services." ARL: A Bimonthly Report on Research Library Issues and Actions from ARL, CNI, and SPARC, 230/231 (2003): 20-21. [Available at http://www.arl.org/newsltr/230/usage.html].
• Cook, Colleen, Fred Heath, Martha Kyrillidou, Yvonna Lincoln, Bruce Thompson, and Duane Webster. “Developing a National Science Digital Library (NSDL) LibQUAL+™ Protocol: An E-service for Assessing the Library of the 21st Century” Submitted for the Developing an Evaluation Strategy for the Educational Impact of the National Science Digital Library Workshop, Washington DC, October 2-3, 2003. [Available at http://www.libqual.org/documents/admin/NSDL_workshop_web1.pdf]
• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “Evaluating the NSF National Science Digital Library Collections.” Paper presented at the Multiple Educational Resources for Learning and Online Technologies (MERLOT) Conference, Costa Mesa, California, August 3-6, 2004. [Available at http://www.libqual.org/documents/admin/MERLOT%20Paper2_final.pdf]
• Lincoln, Yvonna, Colleen Cook and Martha Kyrillidou. “User Perspectives Into Designs for Both Physical and Digital Libraries: New Insights on Commonalities/Similarities and Differences from the NDSLDigital Libraries and LibQUAL+™ Data Bases.” 7th ISKO-Spain Conference, The human dimension of knowledge organization, Barcelona, Spain July, 6-8, 2005. [Available at http://www.libqual.org/documents/admin/ISKO.PDF]
Selected References