performance measurement in academic libraries martha kyrillidou, arl steve hiller, university of...
TRANSCRIPT
Performance Measurement Performance Measurement in Academic Librariesin Academic Libraries
Martha Kyrillidou, ARLSteve Hiller, University of Washington
Jim Self, University of Virginia
EBLIP 4 WorkshopNorth CarolinaMay 11, 2007
ARL Tools and R&DARL Tools and R&D
Martha KyrillidouDirector, ARL Statistics and Service Quality Programs
Association of Research Libraries
ARL www.arl.org
““Life is not measured by the breaths we Life is not measured by the breaths we take...but by the moments that take our take...but by the moments that take our
breath away.”breath away.”
ARL www.arl.org
. . . chart a course for every endeavor that we take the people's money for, see how well we are progressing, tell the public how we are doing, stop the things that don't work, and never stop improving the things that we think are worth investing in.
– President William J. Clinton, on signing the Government Performance and Results Act of 1993
ARL www.arl.org
University of HawaiiUniversity of Hawaii
• Goal 1: Educational Effectiveness and Student Success
• Goal 2: A Learning, Research, and Service Network
• Goal 3: A Model Local, Regional, and Global University
• Goal 4: Investment in Faculty, Staff, Students, and Their Environment
• Goal 5: Resources and Stewardship» Measuring Our Progress Report 2006,
http://www.hawaii.edu/ovppp/mop/mop06_webaccesible.html#satisfaction
ARL www.arl.org
University of Hawaii (cont’d)University of Hawaii (cont’d)
Student Engagement
How engaged are University of Hawaii students in their educational experience at upper division/four-year campuses?
Benchmark #1 Level of Academic ChallengeBenchmark #2 Active and Collaborative LearningBenchmark #3 Student-Faculty InteractionBenchmark #4 Enriching Educational ExperiencesBenchmark #5 Supportive Campus Environment
ARL www.arl.org
University of Hawaii (cont’d)University of Hawaii (cont’d)
Information and Technology Resources: LibraryHow does U H’s major library compare on a national basis?U H Manoa ranks 68th among the 113 ranked university
libraries that are members of the Association of Research Libraries (A R L). Source: 2003–2004 A R L Membership and Statistics
The indexed ranking is based on the number of volumes held, number of volumes added in the last fiscal year, number of current serials, number of permanent staff, and total operating expenditures.
The library aspires to regain its previous higher standing which was significantly impacted by budget cuts in the mid- to late 1990s, and from which the library has been slowly recovering.
ARL www.arl.org
Mission: Shaping the future of research libraries in the changing environment of public policy and
scholarly communication.
Members: 123 major research libraries in North America.
Ratios: 4 percent of the higher education institutions providing 40 percent of the information
resources.
Users: Three million students and faculty served.
Expenditures: 37 percent is invested in access to electronic resources.
ARL www.arl.org
Library Assessment in an Library Assessment in an Electronic EraElectronic Era
What are some of the current developments with library assessments efforts?
ARL StatsQUAL™
E-Metrics
LibQUAL+®
DigiQUAL™
MINES for Libraries™
Where are the most critical needs and opportunities?
What are the lessons learned?
ARL www.arl.org
Thinking Strategically About Thinking Strategically About Libraries’ FuturesLibraries’ Futures
• What is the central work of the library and how can we do more, differently, and at less cost?
• What important set of services does the library provide that others can’t? What new roles are needed?
• What advantages does the library possess?• What will be the most needed by our community of users in
the next decade? How is user behavior changing?• What should our libraries aspire to be ten years from now?
What are the implications of technology driven change?• What are the essential factors responsible for the success
of the library?
ARL www.arl.org
Defining Success in a Digital Defining Success in a Digital EnvironmentEnvironment
• Crafting new measures of success.
• Moving from measuring inputs to outputs and outcomes
• Understanding impact of library roles and services.
• Agreeing on qualitative measures of success: user perceptions, user success, creating value, advancing HE goals.
• Reallocating and managing capabilities to focus on new definitions of success.
ARL www.arl.org
Updating the Traditional ARL Updating the Traditional ARL StatisticsStatistics
• E-Metrics = ARL Supplementary Statistics– On going efforts to update and refine core data.
– Exploring feasibility of collecting e-metrics.
• ARL Task Force on New Ways of Measuring Collections :– Growing concern with utility of membership index.
– Study ARL statistics to determine relevance.
– Develop Profile of Emerging Research Libraries.
ARL www.arl.org
The The LibQUAL+LibQUAL+®® Update Update
• The LibQUAL+® premise, dimensions, and methodology
• LibQUAL+® Results
• LibQUAL+® in Action
The LibQUAL+® premise, dimensions, and
methodology
ARL www.arl.org
The Need for The Need for LibQUAL+LibQUAL+®®
• Underlying need to demonstrate our worth
• The reallocation of resources from traditional services and functions
• Rapid shifts in information-seeking behavior• Need to keep abreast of customer demands
• Increasing user demands
ARL www.arl.org
Why Use Why Use LibQUAL+LibQUAL+®®??Feedback from Feedback from LibQUAL+LibQUAL+® ® UsersUsers
“Why did you choose to use LibQUAL+®?”
• LibQUAL+® was recommended to us as offering a well designed, thoroughly Library-focused set of survey tools
• Opportunity to benchmark• Cost-effectiveness• Automated processing & fast delivery of results• Respectability and comparability (with others and
historically)
ARL www.arl.org
The The LibQUAL+LibQUAL+® ® PremisePremise
PERCEPTIONS SERVICE
“….only customers judge quality; all other judgments are essentially irrelevant”
Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.
ARL www.arl.org
Multiple Methods for Listening Multiple Methods for Listening to Customersto Customers
• Transactional surveys*• Mystery shopping• New, declining, and lost-customer surveys• Focus group interviews• Customer advisory panels• Service reviews• Customer complaint, comment, and inquiry capture• Total market surveys*• Employee field reporting• Employee surveys• Service operating data capture
*A SERVQUAL-type instrument is most suitable for these methods
ARL www.arl.org
World World LibQUAL+LibQUAL+®® Survey 2005 Survey 2005
ARL www.arl.org
LibQUAL+LibQUAL+® ® LanguagesLanguages
American English
Dutch EnglishFrench Canadian DutchSwedish
Swedish(British English)
Afrikaans
DanishFinnishGerman Norwegian
British English
Continental French
Over 1,000 institutions 1,000,000 respondents
ARL www.arl.org
Survey Instrument – “22 Survey Instrument – “22 items…items…
ARL www.arl.org
and A Box.”and A Box.”
• Why the Box is so Important:– About 40% of participants provide open-ended
comments, and these are linked to demographics and quantitative data
– Users elaborate the details of their concerns– Users feel the need to be constructive in their
criticisms, and offer specific suggestions for action
LibQUAL+® Results
ARL www.arl.org
Understanding Understanding LibQUAL+LibQUAL+®® ResultsResults
• For the 22 items LibQUAL+ asks users’ to rate their:• Minimum service level• Desired service level• Perceived service performance
• This gives us a ‘Zone of Tolerance’ for each question; the distance between minimally acceptable and desired service ratings
• Perception ratings ideally fall within the Zone of Tolerance
ARL www.arl.org
General FindingsGeneral Findings
• Highly desired• Making electronic resources accessible from my
home or office• Print and/or electronic journals I require for my
work• A haven for study, learning or research
• Lowest • Library staff who instil confidence in users• Giving users individual attention• Space for group learning and group study
LibQUAL+® in Action
ARL www.arl.org
Using Using LibQUAL+LibQUAL+®® Results Results
• Strategic Service Developments• Data to support service development• Ability to identify where not meeting expectations• Measure if change has met need
• Budget Discussions• Data to support bid for increased funding• Data to support case for change in emphasis (towards
e-provision)
• Marketing Position• Status of the library within the University• Importance of national & international benchmarking
ARL www.arl.org
In Closing, In Closing, LibQUAL+LibQUAL+®®::
• Focuses on the users’ point of view (outcomes)
• Requires limited local survey expertise and resources
• Analysis available at local, national and inter-institutional levels
• Offers opportunities for highlighting and improving your status within the institution
• Can help in securing funding for the Library
ARL www.arl.org
"Each organization must create and communicate performance measures that reflect its unique strategy."
Dr. Robert S. Kaplan, Harvard Business School
Developing the DigiQUALDeveloping the DigiQUAL™™ Protocol for Protocol forDigital Library EvaluationDigital Library Evaluation
• Building on the LibQUAL+® experience
• Secures feedback on user’s perceptions of library’s web site
• Five questions on services, functionality, and content
• Goal is to determine utility, reliability, and trustworthiness
ARL www.arl.org
DigiQUAL™ DimensionsDigiQUAL™ Dimensions
AccessibilityNavigability
InteroperabilityCollection building
Resource UseEvaluating collections
DL as community: users, developers, reviewersCopyright
Role of FederationsDL Sustainability
ARL www.arl.org
Outstanding Issues and Outstanding Issues and ChallengesChallenges
• Unique DLs: niche market, critical mass, both?
• Balance:– custom vs. generic content results– flexible vs. standard implementation scaling
• Mixed methods– Preserving user privacy–Collecting truly useful data
• Moving target: digital libraries as… it depends.
ARL www.arl.org
Assessing the Value of Assessing the Value of Networked Electronic ServicesNetworked Electronic Services
The MINES Survey
Measuring the Impact of Networked Electronic Services (MINES) - MINES for Libraries™
ARL www.arl.org
What is MINES for Libraries™?What is MINES for Libraries™?
• A research methodology consisting of a web-based survey form and a sampling plan.
• Measures who is using electronic resources, where users are located at the time of use, and their purpose of use.
• Adopted by the Association of Research Libraries (ARL) as a part of the “New Measures” toolkit May, 2003.
• Different from other electronic resource usage measures that quantify total usage or measure how well a library makes electronic resources available.
ARL www.arl.org
Questions AddressedQuestions Addressed
• How extensively do sponsored researchers use the new digital information environment?
• Are there differences in usage of electronic information based on the user’s location (e.g., in the library; on-campus, but not in the library; or off-campus)?
• What is a statistically valid methodology for capturing electronic services usage both in the library and remotely through web surveys?
• Are particular network configurations more conducive to studies of digital libraries patron use?
ARL www.arl.org
Library User SurveyLibrary User Survey
ARL www.arl.org
What are the most critical assessment What are the most critical assessment needs and opportunities?needs and opportunities?
• Complementing LibQUAL+® with additional measures.
• Developing impact studies on user success, economic value, and community return on investment.
• Moving target: what is a digital library?
• E-Resources: understanding usage.
• Gaining acceptance and use of standard measures for e-resources.
• Building a climate of assessment throughout library.
ARL www.arl.org
What is the lesson learned?What is the lesson learned?
• Building standardized assessment methods and tools are a key component of understanding users, performance measurement, and improvement of services.
ARL www.arl.org
In ClosingIn Closing
• As higher education is challenged on accountability and effectiveness issues so will libraries.
• A growing appreciation of need for fresh assessment measures, techniques, and processes - old arguments don’t work.
• Basic questions of role, vision, and impact must be answered by library community.
ARL www.arl.org
What’s in a word?What’s in a word?
LIBRARY
ARL www.arl.org
A word is not crystal, transparent and unchanged; it is the skin of a living thought, and may vary greatly in color and content according to the circumstances and time in which it is used.
--Justice Oliver Wendell Holmes
User Needs Assessment or Through the Looking Glass
Steve HillerDirector of Assessment and Planning
University of Washington Libraries
ARL www.arl.org
How have things changed as far as getting your information in the past 5 years?
“We never have to go to the library.”
(sounds of laughter and lots of paper ripping noise on audio tape)Faculty Focus Group 2000 (UW College of Education)
User Needs Assessment or Through the Looking Glass
ARL www.arl.org
Traditional Library Core BusinessTraditional Library Core Business
• Physical Collections – Print (primarily)– Microforms– Other (minor)
• Facilities – House collections– Customer service and work space– Staff work space
• Services– Reference– Instruction– Access
ARL www.arl.org
1306986
116786311555521166147
1018721 1040719 997831 1006856 10321551092525
534266
616683
7993251048437
1286490
2121895
1831768 1853964
1653298
1453632
500000
700000
900000
1100000
1300000
1500000
1700000
1900000
2100000
1995-96 96-97 97-98 98-99 99-00 00-01 2001-02 2002-03 2003-04 2004-05
CHECKOUT IN-LIBRARY USE
Use of UW Physical Collections 1995-96 To 2004-05 DOWN
ARL www.arl.org
UW Libraries Gate Counts UW Libraries Gate Counts DOWNDOWN
Gate Count Large Libraries
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
1,600,000
1,800,000
2,000,000
Health Sci Undergrad Main
Gate Count Branch Libaries
0
50,000
100,000
150,000
200,000
250,000
300,000
Arch Art EastAsia
Engrg Fish-Ocean
FosterBus
Math Music PhysicAstro
SocialWk
FY2002-03
FY2003-04
FY2004-05
ARL www.arl.org
UW Libraries In-Person Reference UW Libraries In-Person Reference Queries Queries DOWN DOWN
141,618
134,525 (-5%)
120,746 (-10%)
106,456 (-12%)
100,000
110,000
120,000
130,000
140,000
150,000
2002-03 2003-04 2004-05 2005-06
ARL www.arl.org
Time for a New Business Model?Time for a New Business Model?
Try the Customer-Centered Library• All services & activities viewed through the eyes of
customers• Customers determine quality • Library services and resources add value to the customerNot better libraries . . . Not better customers . . . but Assess/Measure the Value the Library Provides the
University Community“Documenting the libraries contributions to quality teaching,
student outcomes, and research productivity will become critical.”
(Yvonna Lincoln 2006)
ARL www.arl.org
The Value of User Needs AssessmentThe Value of User Needs Assessment
• Decisions based on data not assumptions -“assumicide”
Fundamental to User-Centered Library• Users determine quality, importance and success• Evaluation and assessment focus on user outcomes• Align collections, resources and services with user needs• Identify differences/similarities in needs and use by
academic areas/groups• Data that can help tell our storyEnsure libraries are responsive to their communities
ARL www.arl.org
What We Need to Know to Support What We Need to Know to Support Our CommunitiesOur Communities
• Who are our customers (and potential customers)?• What are their teaching, learning, clinical and research
interests? How do they work? What’s important for their work?
• What are their library and information needs?• How do they currently use library/information services? • How would they prefer to do so? • How do they differ from each other in library use/needs?• How does the library add value to their work?
ARL www.arl.org
Understand Differences in Your Understand Differences in Your CommunityCommunity
• Library and information needs and use may differ substantially by academic area, groups, and culture
• Identifying and understanding these differences enables libraries to target and market services that add the most value for each group or area
• Multiple assessment methods, including both quantitative and qualitative data, can identify differences and provide the most comprehensive picture of these communities
ARL www.arl.org
Multidimensional Library Assessment: Multidimensional Library Assessment: Beyond Counts and Satisfaction SurveysBeyond Counts and Satisfaction Surveys
• Data based decision making needs good data source• Use multiple assessment methods• Focus on user work and their information seeking and
using behavior• Increased reliance on qualitative data to identify issues
from the perspective of users• Learning from our users• Partnering with other campus programs• Repurposing existing data when possible
How Do We Measure Value?
ARL www.arl.org
UW Assessment PrioritiesUW Assessment PrioritiesCustomer Needs, Use and SuccessCustomer Needs, Use and Success
• Information seeking behavior and use• Patterns of library use• Value of library• User needs• Library contribution to customer success• User satisfaction with services, collections, overall• Data to make informed and wise decisions that lead to
resources and services that contribute to user success
ARL www.arl.org
University of Washington Libraries University of Washington Libraries Assessment Methods UsedAssessment Methods Used
• Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007 – All faculty – Samples of undergraduate and graduate students– Research scientists, Health Sciences fellow/residents 2004-
• In-library use surveys every 3 years beginning 1993• LibQUAL+™ from 2000-2003 • Focus groups/Interviews (annually since 1998)• Observation (guided and non-obtrusive)• Usability• Use statistics/data mining• Information about assessment program available at:
http://www.lib.washington.edu/assessment/
ARL www.arl.org
Our Latest Assessment MethodOur Latest Assessment Method
ARL www.arl.org
UW Triennial Library Use SurveyUW Triennial Library Use SurveyNumber of Respondents and Response Rates 1992-2004Number of Respondents and Response Rates 1992-2004Large number of respondents allows for analysis within groupsLarge number of respondents allows for analysis within groups
2004 2001 1998 1995 1992
Faculty 1560
40%
1345
36%
1503
40%
1359
31%
1108
28%
Grad
Student
627
40%
597
40%
457
46%
409
41%
560
56%
Undergrad 502
25%
497
25%
787
39%
463
23%
407
41%
ARL www.arl.org
Reasons for In-Person Library Visits 2001Reasons for In-Person Library Visits 2001Faculty and Undergrads Visiting Weekly or More OftenFaculty and Undergrads Visiting Weekly or More Often
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Use Collections Use space orservices
Only Collections Only space orservices
Faculty
Undergrads
ARL www.arl.org
Changes in UW Libraries Use Patterns Changes in UW Libraries Use Patterns 1998-2004 1998-2004 (% of each group who use library at least weekly)(% of each group who use library at least weekly)
Visit
1998
Visit
2001
Visit
2004
Remote
1998
Remote
2001
Remote 2004
FacultyChange
47% 40%
-15%
29%
-28%
73% 79%
+8%
91%
+15%
GradChange
78% 59%
-24%
52%
-12%
63% 75%
+19%
87%
+16%
UndergradChange“Visit library for”
67%
(78%)
61%
-9%(72%)
61%
(76%)
43% 54%
+26%
57%
+6%
ARL www.arl.org
UW Faculty Mode of Use by Academic Area 1998/2004 (w eekly+)
1% 2% 3%Visit Only 6%Visit Only
10%Visit Only10%
Remote & Visit 32%
Remote & Visit 45%
Remote & Visit 51%
Remote & Visit 27%
Remote & Visit 39%
Remote & Visit16%
Remote Only45%
Remote Only74%
Remote Only26%
Remote Only 61%
Remote Only 23%
Remote Only 41%
Non- Weekly 17%
Non- Weekly8%
Non- Weekly 25%
Non- Weekly 10%
Non- Weekly 15%
Non- Weekly,9%
Health Sci1998
Health Sci2004
Science-Engin1998
Science-Engin2004
Hum-Soc Sci1998
Hum-Soc Sci2004
ARL www.arl.org
E-Journals Drive Remote Use E-Journals Drive Remote Use Look for E-Journals at Least 2x week Faculty by AreaLook for E-Journals at Least 2x week Faculty by Area
Health Sciences
Humanities-Social Sciences
Science-Engineering
10%
20%
30%
40%
50%
60%
70%
2001 2004
10%
20%
30%
40%
50%
60%
70%
Journal Article Downloads Journal Article Downloads 4,761,704 (2004-05)4,761,704 (2004-05)
ARL www.arl.org
2004 Resource Type Importance 2004 Resource Type Importance Faculty By Selected CollegesFaculty By Selected Colleges
Books
Journals>1985
Journals<1985
Bib Databases
3.00
3.20
3.40
3.60
3.80
4.00
4.20
4.40
4.60
4.80
5.00
Humanities Social Sci Business Science Engineering Public Health
ARL www.arl.org
2004 Overall Collections Satisfaction 2004 Overall Collections Satisfaction By Group in Selected CollegesBy Group in Selected Colleges
FacultyFaculty
Grad
Grad
Undergrad
Undergrad
3.60
3.70
3.80
3.90
4.00
4.10
4.20
4.30
4.40
4.50
4.60
Humanities Social Sci Business Science Engineering Public Health
ARL www.arl.org
Overall Satisfaction by GroupOverall Satisfaction by Group1995-20041995-2004
4.33 4.33
Faculty 4.44
Faculty 4.25Undergrad 4.32
4.22
3.99Undergrad 3.97
Grad 4.184.11
4.26
Grad 4.34
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
1995 1998 2001 2004
3.8
3.9
4
4.1
4.2
4.3
4.4
4.5
4.6
ARL www.arl.org
2004 Top Priorities by UW Group2004 Top Priorities by UW Group
0% 20% 40% 60% 80% 100%
Increase hours
Extend wireless
E-Reserves
Quiet work areas
Print Coll Quality
Older journals online
More E-Journal titles
Undergrads Grads Faculty
ARL www.arl.org
UW Faculty Top Priorities by UW Faculty Top Priorities by Academic Area (2004) Academic Area (2004)
0% 20% 40% 60% 80% 100%
Increase hours
Preservation
Scholarly PubChanges
Print Collection
Older journals online
More E-Journal titles
Health Sciences
Science-Engineering
Humanities-Social Sciences
ARL www.arl.org
UW Libraries 2007 Triennial Survey – UW Libraries 2007 Triennial Survey – Outcomes Question for FacultyOutcomes Question for Faculty
9. What contribution does the UW Libraries make to: (Likert scale of 1 (minor) to 5 (major)
• Keeping current in your field• Finding information in related fields or new areas• Being a more productive researcher• Being a more effective instructor• Enriching student learning experiences• Helping you make more efficient use of your time• Recruiting colleagues and students to UW
ARL www.arl.org
Use of Physical LibraryUse of Physical Library2005 In-Library Use Survey (Triennial)2005 In-Library Use Survey (Triennial)
• One page survey distributed to people entering 15 campus libraries during 4-6 two hour time blocks
• 3861 surveys returned (minimum of 50 for any library)• Surveys asked:
– What did you do in this library today*– How often do you use this library– How important are these services to you*– How would you rate these library services and resources*– What can we do to make this library better for you (write in)
*Option for local library question
ARL www.arl.org
Respondents by User Group forRespondents by User Group forSelected LibrariesSelected Libraries
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Main Art Business Health Sci Math Music Undergrad
Undergrad Grad Faculty Other
ARL www.arl.org
What Do They Do in the Library?Activities by Group UW 2005 In-Library Use Survey
(Undergrads 70%, Grads 25%, Faculty/Staff 5%)
0%
10%
20%
30%
40%
50%
60%
70%
Ask for help Look formaterial
Copy Work alone Work in groups Use LibComputer
Use printer
Undergrads
Grad Students
Faculty/Staff
ARL www.arl.org
The Qualitative Provides the Key The Qualitative Provides the Key
• Increasing use of such qualitative methods as, comments interviews, focus groups, usability, observation
• Statistics often can’t tell us – Who, how, why
– Value, impact, outcomes
• Qualitative provides information directly from users– Their language
– Their issues
– Their work
• Qualitative provides understanding
ARL www.arl.org
Information Literacy Focus Groups: Information Literacy Focus Groups: FindingsFindings• The information environment is too complex• General search engines (e.g. Google) are preferred
over library licensed/provided interfaces• Undergrads have difficulty determining which library
sources to use• Faculty “dumbing down” library research
assignments• Ubiquity of library research – any place, any time
has changed research patterns• Availability online is more efficient way to research• The personal connection with a librarian is important
ARL www.arl.org
Guided Observation 2003Guided Observation 2003Bibliographic Database SearchingBibliographic Database Searching
• Faculty and graduate students search very differently than we think they should
• Common observations included:– Prefer to use single keyword search box
– Little use of Boolean commands
– Limits or format changes rarely employed
– Commands need to be on first page or lost
– Visible links to full-text critical
• Important features for librarians are not necessarily important to faculty and students
ARL www.arl.org
Understanding How Researchers Understanding How Researchers Work: Four Recent StudiesWork: Four Recent Studies
Focus first on work of faculty/grad students and then on connection to library
• University of Minnesota Humanities/Social Science research (2005 Mellon funded)
• New York University 21st Century Library Study (2006)
• Ithaka Faculty survey and interviews on scholarly communication (2006)
• University of Washington Biosciences Review (2006)
ARL www.arl.org
Initial ResultsInitial Results
• University of Minnesota– Extremely comfortable with electronic sources– Inadequate methods for organizing their research materials
• New York University– Researchers (all disciplines) no longer tied to physical library– Expectations for info shaped by Web and commercial sector
• Ithaka– Prefer publishing in widely circulated journals at no cost to them– Biologists frequently search for information outside their area
• University of Washington– Start info search outside library space (virtual and physical)– Could not come up with “new library services” unprompted
ARL www.arl.org
Biosciences Review Task Force Biosciences Review Task Force (2005-06): Reasons for Review(2005-06): Reasons for Review
• Better understand how bioscientists work
• Growing interdisciplinarity
• Significant change in use patterns
• Libraries organizational structure in subject-based silos
• Value of research enterprise to the University
• Strengthening library connection to research
ARL www.arl.org
Biosciences Review ProcessBiosciences Review Process
• Define scope• Mine existing data• Acquire new information
– Environmental scan Jan-May 2006
– Interviews (biosciences faculty) Feb 2006
– Focus groups (biosci faculty & students) Mar-Apr 2006
– Peer library surveys Apr 2006
• Synthesis and first draft May-Aug 2006
• Final report and recommendations Sep-Dec 2006
• Incorporate into Libraries plan 2007-
ARL www.arl.org
UW Students, Faculty and Doctorates UW Students, Faculty and Doctorates Awarded by Academic AreaAwarded by Academic Area
Undergraduate Majors
Phy Sci - Eng 22%
Health Sci
5%
Bioscience
21%
Other 52%
Grad/Professional Students
Bioscience, 12%
Health Sci 31%
Phy Sci - Eng 18%
Other 38%
FacultyBioscience
10%
Health Sci 47%
Phy Sci - Eng 18%
Other 20%
Doctorates Awarded
Other 31%
Phy Sci - Eng 30%
Bioscience 19%
Health Sci, 20%
ARL www.arl.org
FY 2005 External Funding By Source and UW Faculty Area
School of Medicine 470
Health and Human Services 536
Other Health Sciences 198
National Science Foundation 88
Sciences & Natural Resources 173
US Dept of Def 40
Engineering 67
US Dept of Educ 46
Hum/SocSci/Arts39
Other Federal 81
Other Programs 45
Industry-Found 79
State of Wash 21
Other Non-Fed 105
Awards by Source (in millions) Awards by Area (in millions)
ARL www.arl.org
External Funding Source by Academic Area UW Faculty 2004 Survey
Fed Only 8%
Fed Only 34%
Fed Only 28% Fed and Other 9%
Fed and Other42%Fed and Other
43%
Other Only 21%
Other Only 8%Other Only 11% None
62%
None16%
None 17%
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Health Sciences (728) Sciences-Engineering (389) Humanities/Soc Sci/Arts (365)
ARL www.arl.org
Physical Library Use by Group and Physical Library Use by Group and Academic AreaAcademic Area (2005 In-Library Use Survey)(2005 In-Library Use Survey)
Libraries Used Biology Undergrads (n=126)
Other
Chem9%
Other Sci
Health Sci
Main33%
OUGL 44%
Health Sci Library Use - Grad by Area (n=186)
0% 20% 40% 60% 80% 100%
Health Sci
Other
Unknown
ARL www.arl.org
2004 - Top 4 Faculty Priorities2004 - Top 4 Faculty Prioritiesby Academic Area by Academic Area
More ejournal titles
EJournal backfiles
Print collection quality
Scholarly Communications
20%
30%
40%
50%
60%
70%
80%
90%
Health Sci Faculty Bio Sci Faculty Hum-Soc Sci Faculty
ARL www.arl.org
Faculty Interview ThemesFaculty Interview Themes
• Library seen primarily as E-Journal provider • Physical library used only for items not available online• Start information search with Google and PubMed• Too busy for training, instruction, workshops• Faculty who teach undergrads use libraries differently• Could not come up with “new library services”
unprompted
ARL www.arl.org
Focus Group Themes Print Is Dead, Really Focus Group Themes Print Is Dead, Really Dead; Our Virtual Space Not YoursDead; Our Virtual Space Not Yours
• Google, PubMed, Web of Science starting points for all• Faculty identify library with E-journals• Want more online, including older materials; if not online
deliver digitally• Faculty/many grads go to physical library as last resort• Too many physical libraries• E-Science emerging as a new priority• Lack understanding of many library services, resources • Undergrads rarely use print unless assigned by faculty• Increasing overlap between “bio” research and other science
research
ARL www.arl.org
Task Force RecommendationsTask Force Recommendations
• Consolidate collections and service points– Reduce print holdings; focus on online resources
• Reorganize libraries around broad user communities• Integrate search/discovery tools into users workflow• Expand/improve information/service delivery options• Use an integrated approach to collection allocations• Increase integration of librarians with user workflow• Lead scholarly communication and e-science work• Partner with broader science/technology community• Provide more targeted communication and marketing
ARL www.arl.org
What We’ve Learned about the UW What We’ve Learned about the UW CommunityCommunity
• Libraries are still important source of information used for teaching, learning and research but lessening in value
• Library needs/use patterns vary by and within academic areas and groups
• Remote access is preferred method and has changed the way faculty and students work and use libraries
• Faculty and students find information and use libraries differently than librarians prefer them too
• Library/information environment is perceived as too complex; users find simpler ways (Google) to get info
• Customers cannot predict the Libraries future
From Data to Outcomes
Jim SelfDirector, Management Information Services
University of Virginia Library
ARL www.arl.org
The next topicsThe next topics
• Using data for improvement and reassurance– LibQUAL+ as a case study– Corroboration– Benchmarking
• The Balanced Scorecard
“…“…but to suppose that the facts, once but to suppose that the facts, once established in all their fullness, will ‘speak for established in all their fullness, will ‘speak for
themselves’ is an illusion.”themselves’ is an illusion.”
Carl BeckerAnnual Address of the President of the American Historical Association, 1931
LibQUAL+LibQUAL+®®
Finding the right numbersFinding the right numbers
ARL www.arl.org
Initial ExaminationInitial Examination
• Focus on the separate user categories– Faculty– Undergraduates– Graduate students
• Tally the number of responses– 100+ needed in each category
ARL www.arl.org
UVa 2006UVa 2006
• Responses per category– Faculty -- 219– Undergraduates -- 210– Graduate students -- 244
ARL www.arl.org
The 22 core questionsThe 22 core questions
• Examine the notebooks– Scan results for each user category– Scan the summary charts by dimension– Rearrange the dimension charts
• Create thermometer graphs– Question by question– 22 bars for each user category
• Identify the red zones
ARL www.arl.org
Faculty
Library Staff
Grad Students
Dimension Scores UVa 2006
Undergrads
ARL www.arl.org
LibQUAL+ 2006Ratings of Affect of Service
5.00
6.00
7.00
8.00
9.00
Faculty Affect Grad Affect Undergrad Affect Library Staff Affect
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Faculty
GradsUndergrads
Library Staff
ARL www.arl.org
LibQUAL+ 2006Ratings of Information Control
5.00
6.00
7.00
8.00
9.00
Faculty Info Control Grad Info Control Undergrad Info Control Library Staff InfoControl
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
FacultyGrads
Undergrads
Library Staff
ARL www.arl.org
LibQUAL+ 2006Ratings of Library as Place
5.00
6.00
7.00
8.00
9.00
Faculty Place Grad Place Undergrad Place Library Staff Place
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Faculty
Grads
Undergrads
Library Staff
ARL www.arl.org
LibQUAL+ 2006University of Virginia Faculty
5.00
6.00
7.00
8.00
9.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
ARL www.arl.org
LibQUAL+ 2006University of Virginia Graduate Students
5.00
6.00
7.00
8.00
9.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
ARL www.arl.org
LibQUAL+ 2006University of Virginia Undergraduate Students
5.00
6.00
7.00
8.00
9.00
AS-1 AS-2 AS-3 AS-4 AS-5 AS-6 AS-7 AS-8 AS-9 IC-1 IC-2 IC-3 IC-4 IC-5 IC-6 IC-7 IC-8 LP-1 LP-2 LP-3 LP-4 LP-5
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
ARL www.arl.org
Red Zones at UVaRed Zones at UVa
• Journal Collections– Faculty– Grad Students
• Website– Faculty
• Remote access– Grad students
ARL www.arl.org
LibQUAL+ 2006IC-1: Making electronic resources accessible from my home or office
5.00
6.00
7.00
8.00
9.00
Faculty IC-1 Grad IC-1 Undergrad IC-1 Lib Staff IC-1
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Faculty Grads
UndergradsLibrary Staff
ARL www.arl.org
LibQUAL+ 2006IC-8: Print and/or electronic journal collections I require for my work
5.00
6.00
7.00
8.00
9.00
Faculty IC-8 Grad IC-8 Undergrad IC-8 Lib Staff IC-8
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
Faculty Grads
Undergrads
Library Staff
ARL www.arl.org
LibQUAL+ 2006IC-2: A library Web site enabling me to locate information on my own
5.00
6.00
7.00
8.00
9.00
Faculty IC-2 Grad IC-2 Undergrad IC-2 Library Staff IC-2
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
FacultyGrads
Undergrads
Library Staff
ARL www.arl.org
Following up with JournalsFollowing up with Journals
• Who is unhappy?– Drill down by discipline and user category.
• Why are they unhappy?– Read the comments– Conduct targeted interviews
ARL www.arl.org
LibQUAL+ 2006UVA Faculty and Graduate Student Ratings of Journal Collections
5.00
6.00
7.00
8.00
9.00
ArchitectureFaculty
EducationFaculty
EngineeringFaculty
HumanitiesFaculty
Science/MathFaculty
SocialScienceFaculty
ArchitectureGrads
EducationGrads
EngineeringGrads
HumanitiesGrads
Science/MathGrads
SocialScienceGrads
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
ARL www.arl.org
Among the 22 core questions,Among the 22 core questions,the most desired are:the most desired are:
• Faculty– Journal collections 8.60
– Web site 8.49
• Grad Students• Journal collections 8.61• Remote access 8.53
• Undergrads– Modern equipment 8.35
– Comfortable and inviting location 8.29
– Space that inspires study and learning 8.29
ARL www.arl.org
Among the 22 core questions,Among the 22 core questions,the least desired are:the least desired are:
• Faculty– Community space for group learning 6.56
– Quiet space for individual activities 7.03
• Grad Students• Community space for group learning 6.86• Giving users individual attention 7.27
• Undergrads– Giving users individual attention 7.06
– Employees who instill confidence 7.30
ARL www.arl.org
Benchmarking with Peers:Benchmarking with Peers:General Satisfaction QuestionsGeneral Satisfaction Questions
• In general, I am satisfied with the way I am treated at the library.
• In general, I am satisfied with library support for my learning, research, and/or teaching.
• How would you rate the overall quality of the service provided by the library?
ARL www.arl.org
LibQUAL+ 2006Overall Quality of the Service Provided by the Library
38 ARL Libraries
7.87
7.637.63
6.51
6.61
5.87
7.487.52
7.87
5.5
Undergraduates
Graduates
Faculty
UVA UVA
UVA
ARL www.arl.org
LibQUAL+ 2006Faculty Ratings of Journal Collections
ARL Libraries
5.00
6.00
7.00
8.00
9.00
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37
Red Square = Perceived Service Performance
Top of Blue Bar = Desired Level of Service
Bottom of Blue Bar = Minimum Level of Service
UVA
ARL www.arl.org
……beyond beyond LibQUAL+LibQUAL+®®
• Corroboration
• Data mining
• Balanced Scorecard
ARL www.arl.org
CorroborationCorroboration
• Data are more credible if they are supported by other information
• John Le Carre’s two proofs
ARL www.arl.org
Analyzing U.Va. Survey ResultsAnalyzing U.Va. Survey Results
• Two Scores for Resources, Services, Facilities– Satisfaction = Mean Rating (1 to 5)– Visibility = Percentage Answering the Question
• Permits comparison over time and among groups
• Identifies areas that need more attention
ARL www.arl.org
Reference Activity and Visibility Reference Activity and Visibility in Student Surveysin Student Surveys
1,756
6,008
34%Visibility
39% Visibility
75% Visibililty
64% Visibility
1,000
7,000
1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Re
fere
nc
e Q
ue
sti
on
s
Re
co
rde
d p
er
We
ek
in
An
nu
al S
am
ple
10%
Re
fere
nc
e V
isib
ilit
y
am
on
g U
nd
erg
rad
ua
te
ARL www.arl.org
Investment and Customer ActivityInvestment and Customer ActivityUniversity of Virginia LibraryUniversity of Virginia Library1993-20061993-2006
Customer Activities
0
600,000
1,200,000
1,800,000
2,400,000
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
E-Journals
Circulation
Reference
Acquisitions Expenditures by Format
$0
$1,000,000
$2,000,000
$3,000,000
FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06
Electronic Resources
Print Monographs
Print Serials
ARL www.arl.org
BenchmarkingBenchmarkingComparisons with PeersComparisons with Peers
• Within the University
• Within ARL
ARL www.arl.org
Total Expenditures at UVA Total Expenditures at UVA 1989-20031989-2003
-50%
300%
Fiscal Year
% C
hang
e si
nce
1989 Other Academic Support
(+200%)
Research (+219%)
Total Academic Division (+140%)
Libraries (+81%)
Instruction (+80%)
ARL www.arl.org
Collections ExpendituresCollections ExpendituresUVA vs. ARLUVA vs. ARL
•
-10%
40%
90%
140%
190%
1991 2002
Fiscal Year
% C
han
ge S
ince 1
991
ARL PercentageGrow th Through 2003(+102%)
UVa PercentageGrow th(+48%)
2012
ARL Projection(+180%)
UVa Projection(+104%)
ARL www.arl.org
Median Faculty SalariesMedian Faculty SalariesUniversity of Virginia LibraryUniversity of Virginia LibraryCompared to ARL MedianCompared to ARL Median
$30,000
$35,000
$40,000
$45,000
$50,000
$55,000
1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
Fiscal Year
U.Va.
ARL
ARL www.arl.org
The Balanced ScorecardThe Balanced Scorecard
• A layered and categorized instrument that– Identifies the important statistics– Ensures a proper balance– Organizes multiple statistics into an intelligible
framework
ARL www.arl.org
The Balanced ScorecardThe Balanced Scorecard
• Reflects the organization’s vision
• Clarifies and communicates the vision
• Provides a quick, but comprehensive, picture of the organization’s health
ARL www.arl.org
The scorecard measures are The scorecard measures are “balanced” into four areas“balanced” into four areas
• The user perspective
• The finance perspective
• The internal process perspective
• The learning and growth perspective
ARL www.arl.org
MetricsMetrics
• Specific targets indicating full success, partial success, and failure
• At the end of the year we know if we have met our target for each metric
• The metric may be a complex measure encompassing several elements
ARL www.arl.org
The BSC at the U.Va. LibraryThe BSC at the U.Va. Library
• Implemented in 2001
• Reports for FY02 to FY06
• Completing metrics for FY08
• Will tally FY07 in July and August
• A work in progress
ARL www.arl.org
Core QuestionsCore Questions
• User Perspective – How well is the library meeting user needs?
• Internal Processes– Do the library’s processes function efficiently?
• Finance– How well are the library’s finances managed?
• Learning and Growth– Is the library well positioned for the future?
ARL www.arl.org
Metric U.1.A: Overall rating in Metric U.1.A: Overall rating in student and faculty surveysstudent and faculty surveys
• Target1: An average score of at least 4.00 (out of 5.00) from each of the major constituencies.
• Target2: A score of at least 3.90.
FY06 Result: Target1 – Graduate students 4.08 – Undergraduates 4.11
ARL www.arl.org
Metric U.3.A: Circulation of new Metric U.3.A: Circulation of new monographsmonographs
• Target1: 60% of newly cataloged monographs should circulate within two years.
• Target2: 50% of new monographs should circulate within two years.
• Result FY06: Target 1.– 61.2% circulated (15,213 out of 24,852)
ARL www.arl.org
Metric U.4.B: Turnaround time Metric U.4.B: Turnaround time for user requestsfor user requests
• Target1: 75% of user requests for new books should be filled within 7 days.
• Target2: 50% of user requests for new books should be filled within 7 days.
• Result FY06: Target1.– 79% filled within 7 days.
ARL www.arl.org
Metric F.1.B.: Library spending Metric F.1.B.: Library spending compared to University expenditures compared to University expenditures
• Target1: : The University Library will account for at least 2.50% of the University’s academic division expenditures.
• Target2: : The Library will account for at least 2.25% of expenditures.
• Result FY06: Target1. – 2.57% ($25.2M of $972M)
ARL www.arl.org
Metric F.2.A: Unit Cost of Metric F.2.A: Unit Cost of Electronic Serial UseElectronic Serial Use
• Target1: There should be no increase in unit cost each year.
• Target2: Less than 5% annual increase in unit cost.
• Result FY06: Target not met.– 8.8% increase ($2.10 vs. $1.93)
ARL www.arl.org
Metric I.1.A: Processing Time for Metric I.1.A: Processing Time for Routine AcquisitionsRoutine Acquisitions
• Target1: 90% of in-print books from North America should be processed within one month.
• Target2: 80% should be processed within one month.
• Result FY06: Target2.– 87.2% processed.
ARL www.arl.org
Metric I.2.A.Metric I.2.A.: : Internal Internal Communications Communications
• Target1: Positive scores (4 or 5) on internal communications statements from 80% of respondents in the biennial work-life survey.
• Target2: Positive scores from 60%. • Result FY06: Target not met.
– 48% gave positive scores.
ARL www.arl.org
Metric L.3.A.:Metric L.3.A.: Expenditures for Expenditures for digital materials digital materials
• Target1: Rank in the top 25% of ARL libraries in percentage of collections dollars spent on digital materials.
• Target2: Rank in top 33%.
• Result FY06: Target not met.
– Ranked 74 of 109.
ARL www.arl.org
Metric L.2.C.Metric L.2.C.: : Compare staff Compare staff salaries to peer groups. salaries to peer groups.
• Target1: Library faculty salaries should rank in the top 40% of salaries at ARL libraries.
• Target2: Rank in top 50%.
• Result FY06: Target1.
– Ranked 33 of 113. (Top 28%)
ARL www.arl.org
Reporting BSC ResultsReporting BSC Results
University of Virginia LibraryBalanced Scorecard
FY06 Results
Target1Target2Not Met