Effective, Sustainable & Practical

Assessment

Steve Hiller, Director, Assessment and Planning, UWMartha Kyrillidou, Statistics and Service Quality Programs, ARLJim Self, Director, Management and Information Services, UVA

Boston UniversitySeptember 08, 2008

Monday Morning Metrics With Steve

BaseballPresentation #535

8 September 2008Steve Hiller, UW Libraries

Case Study: Worst Trade Ever Made by the Seattle Mariners

• When• Who• Why was it the worst• Evaluating performance• Defining success and value

– Statistics

– “Intangibles”

Library AssessmentMore than Numbers

Library assessment is a structured process:

• To learn about our communities

• To respond to the needs of our users

• To improve our programs and services

• To support the goals of the communities

Why Assess?

• Accountability and justification• Improvement of services • Comparisons with others• Identification of changing patterns• Marketing and promotion • Opportunity to tell our own story

• Using data, not assumptions, to make decisions

– Assumicide!

The Challenge for LibrariesThe Challenge for Libraries

• Traditional statistics are no longer sufficient– Emphasize inputs – how big and how many

– Do not tell the library’s story

– May not align with organizational goals and plans

– Do not measure service quality

• Need measurements from the user’s perspective• Need the organizational culture and the skills to

answer a basic question:

What difference do we make to our communities?

“If you think you're too small to have an impact, try going to bed with a mosquito.”

»Anita Roddick

Wide Array of User Studies Now Available

Students start with

Google

Format Agnostic

Seek convenien

ce

Findings

Charting User Change

User BehaviorUser Behavior User ExpectationsUser Expectations

Customer Service (qualified & helpful staff)

Library as a place, symbol, refuge

Self-sufficiency & control of information

seeking process

Ready access to wide range of

content (e.g. complete runs of journals)

BornDigital

Showcase for recruitment

Re-conceptualizing Library Facilities

Re-configuring library facilities:Learning commonsCollaborativ

e

studySocial and intellectual center

Secondary Storage

Library as physical place, intellectual space, and community center

Changing nature of library usage

ARL Sponsored Assessment • Tools - StatsQUAL

– ARL Statistics™ - descriptive, longitudinal, comparative

– LibQUAL+®, ClimateQUAL™, DigiQUAL™

– MINES for Libraries®, E-metrics …. Google Analytics

• Building a Community of Practice– Library Assessment Conferences

– Service Quality Evaluation Academy (training events)

– Library Assessment blog

• Individual Library Consultation (Jim and Steve)– Making Library Assessment Work (24 libraries in 2005-06)

– Effective, Sustainable, Practical Library Assessment (14 in 2007-08)

ARL Tools for Library Assessment

As a result of the work of the New Measures and Assessment Initiative (1999)…

ARL Statistics™

Since 1907-08

LibQUAL+®

Since 2000

MINES for Libraries™

Since 2003

DigiQUAL®

Since 2003

ClimateQUAL™

Since 2007

Survey Structure Survey Structure (Detail View)(Detail View)

“22 Items and The Box….”

Why the Box is so Important– About 40% of participants provide open-

ended comments, and these are linked to demographics and quantitative data.

– Users elaborate the details of their concerns.

– Users feel the need to be constructive in their criticisms, and offer specific suggestions for action.

MINES for LibrariesTM

• MINES is a transaction-based research methodology consisting of a web-based survey form and a random moments sampling plan

• MINES typically measures who is using electronic resources, where users are located at the time of use, and their purpose of use in the least obtrusive way

• MINES was adopted by the Association of Research Libraries (ARL) as part of the “New Measures” toolkit in May, 2003.

• MINES is different from other electronic resource usage measures that quantify total usage (e.g., Project COUNTER, E-Metrics) or measure how well a library makes electronic resources accessible (LibQUAL+™).

ESP Insights

• Strong interest in using assessment to improve • Uncertainty on how to establish and sustain assessment• Lack of assessment knowledge among staff• More data collection than data utilization• Effectiveness not dependent on library size or budget• Each library has a unique culture and mission

Effective Assessment

• Focuses on the customer • Is aligned with library and university goals• Assesses what is important• Is outcomes oriented• Develops criteria for success• Uses appropriate and multiple assessment methods• Uses corroboration from other sources• Provides results that can be used

Sustainable Assessment Needs . . • Organizational leadership• Sufficient resources• Supportive organizational culture• Identifiable organizational responsibility • Connection to strategic planning and priorities• Iterative process of data collection, analysis, and use • Involvement of customers, staff and stakeholders

Practical Assessment • Keep it simple and focused – “less is more”• Know when enough is enough• Use assessment that adds value for customers• Present results that are understandable• Organize to act on results

The University of Virginia• 14,000 undergraduates

– 66% in-state, 34% out

– Most notable for liberal arts

– Highly ranked by U.S. News

• 6,000 graduate students– Prominent for humanities,

law, business

– Plans expansion in sciences

• Located in Charlottesville– Metro population of 160,000

Collecting the Data at the U.Va. Library

• Customer Surveys

• Staff Surveys

• Mining Existing Records

• Comparisons with peers

• Qualitative techniques

• Long involvement with ARL statistics

Management Information Services

• MIS committee formed in 1992

• Evolved into a department 1996-2000

• Currently three staff

• Coordinates collection of statistics

• Publishes annual statistical report

• Coordinates assessment

• Resource for management and staff

www.arl.org

“…but to suppose that the facts, once established in all their fullness, will

‘speak for themselves’ is an illusion.”

Carl BeckerAnnual Address of the President of the

American Historical Association, 1931

UVa Customer Surveys• Faculty

– 1993, 1996, 2000, 2004– Response rates 59% to 70%

• Students – 1994, 1998, 2001, 2005– Separate analysis for grads and undergrads– Response rates 43% to 63%

• LibQual+ 2006– Response rates 14% to 24%

• Annual Surveys 2008– Student samples– One third of faculty– Response rates 29% to 47%

www.arl.org

Corroboration

• Data are more credible if they are supported by other information

• John Le Carre’s two proofs

www.arl.org

Analyzing U.Va. Survey Results

• Two Scores for Resources, Services, Facilities– Satisfaction = Mean Rating (1 to 5)– Visibility = Percentage Answering the Question

• Permits comparison over time and among groups

• Identifies areas that need more attention

www.arl.org

U.Va. Reference Activity and Reference Visibility in Student Surveys

1,756

6,008

34%Visibility

39% Visibility

75% Visibililty

64% Visibility

1,000

7,000

1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Re

fere

nc

e Q

ue

sti

on

s

Re

co

rde

d p

er

We

ek

in

An

nu

al S

am

ple

10%

Re

fere

nc

e V

isib

ilit

y

am

on

g U

nd

erg

rad

ua

te

www.arl.org

Data Mining

• Acquisitions

• Circulation

• Finance

• University Records

Investment and Customer ActivityUniversity of Virginia Library

1993-2006

Customer Activities

0

600,000

1,200,000

1,800,000

2,400,000

FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06

E-Journals

Circulation

Reference

Acquisitions Expenditures by Format

$0

$1,000,000

$2,000,000

$3,000,000

FY93 FY94 FY95 FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 FY04 FY05 FY06

Electronic Resources

Print Monographs

Print Serials

The Balanced Scorecardat the U.Va. Library

• Implemented in 2001

• Results tallied FY02 through FY07

• Tallying results for FY08

• Completing metrics for FY09

• Builds upon a rich history of collecting data

• A work in progress

The Balanced ScorecardManaging and Assessing Data

• The Balanced Scorecard is a layered and categorized instrument that– Identifies the important statistics– Ensures a proper balance– Organizes multiple statistics into an intelligible

framework

Metrics

• Specific targets indicating full success, partial success, and failure

• At the end of the year we know if we have met our target for each metric

• The metric may be a complex measure encompassing several elements

What Do We Measure?

• Customer survey ratings

• Staff survey ratings

• Timeliness and cost of service

• Usability testing of web resources

• Success in fundraising

• Comparisons with peers

Metric L.2.B: Retention Rate of Employees

• Target1: Retain 95% of employees.

• Target2: Retain 90% of employees.

• Result FY08: Target1.– 95% of employees retained.

Metric U.4.B: Turnaround time for user requests

• Target1: 75% of user requests for new books should be filled within 7 days.

• Target2: 50% of user requests for new books should be filled within 7 days.

• Result FY07: Target1.– 77% filled within 7 days.

Metric U.3.A: Circulation of New Monographs

• Target1: 60% of newly cataloged monographs should circulate within two years.

• Target2: 50% of new monographs should circulate within two years.

• Result FY07: Target1.– 63% circulated.

Balanced ScorecardU.Va. Library FY2007

Target1

Target2

Not Met

Using Data for Results at UVa

• Additional resources for the science libraries (1994+)

• Redefinition of collection development (1996)

• Initiative to improve shelving (1999)

• Undergraduate library open 24 hours (2000)

• Additional resources for the Fine Arts Library (2000)

• Support for transition from print to e-journals (2004)

• New and Improved Study Space (2005-06)

• Increased appreciation of the role of journals (2007)

University of Washington

• Located in Seattle; metro population 3.2 million

• Comprehensive public research university – 27,000 undergraduate students– 12,000 graduate and professional

students (80 doctoral programs)– 4,000 research and teaching

faculty

• $800 million yearly in U.S. research funds (#2)

• Large research library system– $40 million annual budget– 150 librarians on 3 campuses

The Basic Question How Does the Library Contribute to the Success

of our Researchers and Students?

Our assessment priorities:• Information seeking behavior and use

• Patterns of library use

• User needs

• Library contribution to learning and research

• User satisfaction with services, collections, overall

What have we learned (short version):• Faculty perceive success through collections

• Grad students through timely access to resources and services

• Undergrads through library as place for work and community

University of Washington Libraries Assessment Methods Used

• Large scale user surveys every 3 years (“triennial survey”): 1992, 1995, 1998, 2001, 2004, 2007

• In-library use surveys every 3 years beginning 1993– 4000 surveys returned in 2008

• Focus groups/Interviews (annually since 1998)• Observation (guided and non-obtrusive)• Usability• Usage statistics/data mining

Information about assessment program available at:http://www.lib.washington.edu/assessment/

UW Triennial Library Survey Number of Respondents and Response Rate 1992-2007

2007 2004 2001 1998 1995 1992

Faculty 1455

36%

1560

40%

1345

36%

1503

40%

1359

31%

1108

28%

Grad

Student

580

33%

627

40%

597

40%

457

46%

409

41%

560

56%

Undergrad 467

20%

502

25%

497

25%

787

39%

463

23%

407

41%

Library as Place Change In Frequency of In-Person Visits 1998-2007 (weekly+)

Faculty

Faculty

Grad

Grad

Undergrad

Undergrad

20%

30%

40%

50%

60%

70%

80%

1998 2001 2004 2007

What Did You Do in the Library Today?(In-Library Use Surveys 2008/2005)

0%

10%

20%

30%

40%

50%

60%

70%

Ask for help Look formaterial

Work alone Work ingroups

Use LibComputer

Use OwnComputer

2008 2005

Activity in the Library by Group2008 Users: 73% UG, 22% Grad, 5% Faculty

0%

10%

20%

30%

40%

50%

60%

70%

Ask for help Look formaterial

Work alone Work ingroups

Use LibComputer

Use OwnComputer

Undergrad 2008 Grad 2008 Faculty/Staff 2008

Usefulness of New and/or Expanded Services for Undergrads: Library as Place (2007 Triennial Survey)

0% 10% 20% 30% 40% 50% 60%

Group/PresentationSpaces

Book Self-Check out

Integrate services intocampus Web sites

More library computers

Increase weekend hours

Quiet work/study areas

Library As Place: Using the Results • Libraries are student places

– 350 computer lab installed in Undergrad Library Autumn 1998

– Hours extended to 24/5.5 in Undergrad Library 2002

– Collection footprint reduced

– Diversified user spaces provided (group, quiet, presentation)

– Student advisory committee provides ongoing feedback

– Add other collaborative student support services into library

• Upgrade/renovate facilities to meet student needs– Furniture that encourages collaboration

– More electrical outlets

– Better lighting and noise control

• Plan for major renovation of Undergraduate Library

What Faculty/Grad Students Told Us Bioscience Interview/Focus Groups (2006)

• Content is primary link to the library– Identify library with ejournals; want more titles & backfiles

• Provide library-related services and resources in our space not yours– Discovery begins primarily outside of library space with

Google and Pub Med; Web of Science also important

– Library services/tools seen as overly complex and fragmented

• Print is dead, really dead– If not online want digital delivery/too many libraries

– Go to physical library only as last resort

• Uneven awareness of library resources and services

Sources Consulted for Information on Research Topics (Scale of 1 “Not at All” to 5 “Usually”)

Faculty

Grad

Undergrad

2.5 2.75 3 3.25 3.5 3.75 4 4.25 4.5

BibliographicDatabases

Library Catalog

Open Internet RefSource

Open Internet Search

Off-Campus Remote Use 1998-2007(Percentage using library services/collections at least 2x week)

Faculty

Undergrad

Grad

0%

10%

20%

30%

40%

50%

60%

70%

1998 2001 2004 2007

0%

10%

20%

30%

40%

50%

60%

70%

76% of faculty and 72% of graduate students use library services online at least 2x week

Primary Reasons for Faculty Use of Libraries Web Sites 2007 (at least 2x per week)

15%

25%

35%

45%

55%

65%

75%

Health Sci Science-Engin Hum-Soc Sci

Library Catalog Bib Database Online journal articlesThe ability to access full-text or pdf research articles online through the library subscriptions is my primary use of the library and is central to my research. Neurobiology Grad Student

E-Journal Usage at UW

Scholarly Stats 2007: 5 million article requests from 19 vendors/platforms - 7 accounted for nearly 75%

• High Wire Press 1,200,000 Biomedical

• Science Direct 975,000 Science-Medical

• JSTOR 750,000 Multidisciplinary

• Nature 300,000 Science-Medical

• Meta Press 250,000 Science-Medical

• Blackwell Synergy 225,000 Science-Medical

• Ovid 225,000 Biomedical

Importance of Books, Journals, DatabasesAcademic Area (2007, Faculty, Scale of 1 “not important” to 5 “very important)

Books

Journals<1985

Bib Databases

Journals>1985

3.2

3.4

3.6

3.8

4

4.2

4.4

4.6

4.8

Health Sciences Science-Engineering Hum-Soc Science

Books Journals<1985 Bib Databases Journals>1985

Bibliographic Database Use: Login Sessions Database 2005 2006 2007 Change

Art Abstracts 4120 2887 2330 -43%

Geobase 4504 4435 3933 -13%

MLA 23246 22189 20696 -11%

ERIC 17063 16078 15263 -11%

Expanded Academic

191095 176483 141887 -26%

Web of Science 158451 162600 154546 -2%

Librarian Liaison Satisfaction & VisibilityBy Selected School (2007 Triennial Survey; Satisfaction on 1 to 5 scale; visibility % who rated)

Fine Arts

Humanities

Science

Engineering

MedicinePublic Health Business

30%

40%

50%

60%

70%

80%

90%

100%

3.8 4 4.2 4.4 4.6 4.8

Satisfaction

Vis

ibili

ty

Usefulness of New/Expanded Services for Faculty & Grads: Integrate into my space

0% 10% 20% 30% 40% 50% 60% 70% 80%

Manage your info &data

Integrate services intocampus Web sites

Office delivery ofbooks

Digitize specializedcollections

Scan on Demand

Grad Faculty

Integrate Library Services & Resources into User Workflows: Follow-Up Actions

• WorldCat Local became primary catalog+ in 2007• “Scan on demand” pilot began 2008• Increase chat and remote services staffing• Increase ILL staffing (due to WorldCat Local)• Integrate course reserves & services into “My UW” portal• Redesign UW Libraries Homepage• Use qualitative methods to gain deeper understanding of

user work and behavior • Strengthen librarian liaison efforts to academic programs

How UW Libraries Has Used Assessment

• Extend hours in Undergraduate Library (24/5.5)• Create more diversified student learning spaces• Enhance usability of discovery tools and website• Provide standardized service training for all staff• Review and restructure librarian liaison program • Consolidate and merge branch libraries• Change/reallocate collections budget• Change/reallocate staffing• Support budget requests to University

Overall Satisfaction by Group 1995-2007

Faculty 4.25

4.44

4.33 4.33

Faculty 4.56

Undergrad 3.97 3.99

4.22

4.32

Undergrad 4.36

4.34

4.26

4.11Grad 4.18

Grad 4.36

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

1995 1998 2001 2004 2007

3.8

3.9

4

4.1

4.2

4.3

4.4

4.5

4.6

You guys and gals rock!!!!!! We need to invest in our library system to keep it the best system in America. The tops! My reputation is in large part due to you. Professor, Forest Resources

Four Useful Assessment Assumptions

• Your problem/issue is not as unique as you think• You have more data/information than you think• You need less data/information than you think• There are useful methods that are much simpler than

you think

Adapted from Douglas Hubbard, “How to Measure Anything” (2007)

in conclusion Assessment is not…

• Free and easy• A one-time effort• A complete diagnosis• A roadmap to the future

Assessment is…

• A way to improve • An opportunity to know our customers• A chance to tell our own story• A positive experience

Moving Forward

• Keep expectations reasonable and achievable• Strive for accuracy and honesty—not perfection• Assess what is important• Use the data to improve • Keep everyone involved and informed• Focus on the customer

For more information…

• Steve Hillerhiller@u.washington.eduwww.lib.washington.edu/assessment/

• Jim Self– self@virginia.edu– www.lib.virginia.edu/mis– www.lib.virginia.edu/bsc

• ARL Assessment Servicewww.arl.org/stats/initiatives/esp/