roswitha poll münster, germany ten years after: „measuring quality“ revised

Roswitha PollRoswitha PollMünster, GermanyMünster, Germany

Ten years after:„Measuring Quality“ revised

2/21

MeasuringQuality

1st edition 1996 English, German, French, Russian, Italian only academic libraries 17 indicators

3/21

MeasuringQuality

2nd revised edition 2007 academic and public libraries 40 indicators Authors: Peter te Boekhorst, Roswitha Poll

4/21

MeasuringQuality: Why?

So what are you doing at the moment?Writing a book about performance indicators for libraries

What’s that, performance indicators? Methods for seeing whether the library is good

And how do you know whether you are good? By asking users, by measuring whether we are doing things right

For example? That all the books are always in their right place

Yes, that’s important. And such things show you whether you are good?Well, not quite, but I can compare with other libraries

And if they are worse, you know that you are good?That’s about it

Okay, sounds easyBut why do you need a book for

that?

5/21

Changes in the library world

electronic services predominant the teaching library as new role cost-effectiveness crucial competition of the institutions library as physical place widens to meeting and communication

6/21

Trends in performance measurement

stakeholders‘ views quantitative and qualitative measures standardisation has proceeded - ISO 11620 will comprise e-measures user surveys find broad acceptance impact/outcome measures

7/21

More use of performance measures?

More data available

For electronic services - resources- expenditure- usage

In national statistics- reference transactions- data for library space- attendances at user training and events

But:• Visits still only physical visits• Digitisation missing

8/21


Joint benchmarking projects

BIX –Library IndexBIX –Library IndexGerman public and academic librariesGerman public and academic libraries

CASL CASL (Council of Australian State Libraries)(Council of Australian State Libraries)public librariespublic libraries

Swedish Quality HandbookSwedish Quality Handbookall types of librariesall types of libraries

HELMS HELMS (UK Higher Education(UK Higher Education Library Library Management Statistics)Management Statistics) academic librariesacademic libraries

Netherlands University LibrariesNetherlands University Libraries

9/21


Joint benchmarking projects

Projects: Projects: 55

Indicators: Indicators: 5555

Corresponding to ISO 11620: Corresponding to ISO 11620: 2525

Used in more than one project: Used in more than one project: 2020

Individuality obvious

10/21


Interactive national statistics allow combinations of data (e.g. per capita) Less reports from individual libraries But: service level agreements with performance indicators on websites Performance measures have been integrated

Time to re-shelve: 100% in 6 hours

11/21

The handbook:structure

Balanced Scorecard chapter on impact/outcome bibliography for each indicator examples for possible scores

12/21

The handbook:How to choose indicators

The set of indicators should cover the full range of library services

consider traditional and electronic services if possible „merged“ indicators include indicators for „potentials and development“ consider the view of different stakeholders

13/21

Library quality:Stakeholder views

UsersAccess to information worlwideDelivery of information to the desktopSpeed and accuracy of deliveryGood in-library working conditionsResponsiveness of staffReliability of services

14/21


Financing authoritiesCost-effectivenessClear planning, effective organisationPositive outcome on usersBenefits for the institution’s goalsEffective cooperation High reputation of the library

15/21


StaffGood working conditionsClear planning, straight processesSystematic staff developmentHigh reputation of the library

16/21

The handbook:How to choose indicators

Only one indicator for one question?Example: How intensively is the collection used?Indicators:

collection turnoverloans per capitapercentage of stock not used

Each indicator gives a different picture and might serve different purposes of the evaluating library

17/21

The indicatorsA. Resources, infrastructure: What does the library offer?

Library as place forlearning and research

1. User area per capita

2. Seats per capita

3. Opening hours compared to demand

Collections 4. Expenditure on information provision per capita

5. Availability of required titles

6. Percentage of rejected sessions

7. Ratio of requests received to requests sent out in interlibrary lending

8. Immediate availability

Staff 9. Staff per capita

Website 10. Direct access from the homepage

18/21

Opening hours compared to demand

DefinitionThe actual number and time of opening hours compared to opening hours as desired by users Method: SurveyPossible formula: A/B A = the number of present opening hours

B = the number of hours which the users state that they need (present hours + additional hours)

If a library opens 60 hours per week and in the survey users ask for 10 hours more, the score would be 60:70 = 0.86

19/21

Immediate availability

DefinitionThe percentage of immediate loans of total loans Method:Immediate loans = loans minus reservationsTotal loans = loans plus ILL loans received

Question: The probability that a user’s loan request will be fulfilled immediately

20/21

Direct access from the homepage

DefinitionThe availability of the most frequently used resources and services via the homepage of the library’s website, measured by the number of clicks necessary and the comprehensibility of the terms usedMethodCognitive walk-through: A small group of experts simulates user behaviourQuestionWhether the homepage leads directly or very quickly to the most frequently needed information

21/21


Academic libraries

AddressAddress

Way to the libraryWay to the library

Opening timesOpening times

Online catalogueOnline catalogue

Lending serviceLending service

User cardUser card

User accountUser account

ILL/document deliveryILL/document delivery

ReferenceReference

DatabasesDatabases

E-journalsE-journals

Subject accessSubject access

User trainingUser training

News, eventsNews, events

22/21


Public libraries

AddressAddress

Way to the libraryWay to the library

Opening timesOpening times

Online catalogueOnline catalogue

Lending serviceLending service

User cardUser card

User accountUser account

Electronic collectionElectronic collection

Link collectionLink collection

ReferenceReference

FeesFees

Services for childrenServices for children

Branch librariesBranch libraries

Community servicesCommunity services

News, eventsNews, events

Service Clicks Points

direct information on the homepage, e.g. times open, address, catalogue search

0 10

unmistakable term on the homepage

1 8


2 6


3 4


3 0

ambiguous term on the homepage 1 2

ambiguous term on the homepage 1 0

Access via the homepage

24/21

The indicatorsB. Use: How are the services accepted?

General 1. Market penetration

2. User satisfaction

3. Library visits per capita

Library as place

4. Seat occupancy rate

Collections 5. Number of content units downloaded per capita

6. Collection use (turnover)

7. Percentage of stock not used

8. Loans per capita

9. Percentage of loans to external users

Information services

10. Attendances at training lessons per capita

11. Reference questions per capita

Cultural activities

12. Attendances at events per capita

25/21

Library visits per capita

DefinitionThe total number of library visits per year, either physical or virtual, by members of the population, divided by the number of persons in the populationMethodPhysical visits: turnstileVirtual visits:

web browser visitsweb browser visits

IP visitsIP visits

homepage visitshomepage visits

26/21

Library visits per capita:A method for counting homepage visits

Counting the deliveries of a Counting the deliveries of a Transparent GIF Transparent GIF

placed on the home page for counting purposesplaced on the home page for counting purposes

The number of deliveries of the pixel is identical The number of deliveries of the pixel is identical

to the number of page deliveries. to the number of page deliveries.

The advantage of this method is that it easy to use. The advantage of this method is that it easy to use.

The disadvantage is that access to other pages The disadvantage is that access to other pages

is not counted.is not counted.

German Benchmarking Project BIX http://www.bix-bibliotheksindex.de/

http://www.bix-bibliotheksindex.de/

27/21

The indicatorsC. Efficiency: Are the services offered cost-effectively?

General 1. Cost per user

2. Cost per visit

3. Cost per use

4. Ratio of acquisitions costs to staff costs

Collection costs 5. Cost per database session

6. Cost per download

Processes - speed

7. Acquisition speed

8. Media processing speed

9. Employee productivity in media processing

10. Lending speed

11. Interlibrary loan speed

Processes - reliability

12. Correct answer fill rate

13. Shelving accuracy

28/21

Cost per use

DefinitionThe total operating or recurrent expenditure of the library during the reporting year divided by the number of loans + in-house use + downloads from the electronic collection Methodloans = lending systemin-house use = samplingdownloads = suppliers and library servers

Question The indicator assesses the library costs per case of collection use and therewith the cost-efficiency of library services

29/21

The indicatorsD. Potentials and development: Is the library fit for future?

Electronic services

1.Percentage of acquisitions expenditure spent on the electronic collection

2. Percentage of library staff providing end developing electronic services

Staff development

3. Attendances at training lessons per staff member

Budget 4. Percentage of library means received by special grants or income generation

5. Percentage of institutional means allocated to the library

30/21

Attendances at training lessons per staff member

DefinitionThe number of attendance hours of staff members at formal training lessons during one year divided by the total number of library staffMethod- number of attendants at each training- duration of a trainingA training of 3 hours with 12 attendants would be 36 attendance hours.

Question What priority does the library give to staff training and therewith to the library’s ability to cope with development?

31/21

Staff training: What has been counted?

Hours of training per staff memberDays of training per staff member (hours calculated in days)Percentage of staff members who received training during the yearNumber of training lessons per staff memberTraining hours as percentage of total staff working hoursExpenditure for staff training

32/21

Measuring Quality: What we tried to do

indicators for all

services

new indicators

practical examples for each indicator

„merged“ indicators

find the ultimate set of measures

??

bibliography for each indicator

It is difficult to catch a black cat in a dark room especially when it is not there

Chinese Proverb

roswitha poll münster, germany ten years after: „measuring quality“ revised

Documents