impact measures for libraries and information services roswitha poll münster bielefeld 2006...
TRANSCRIPT
Impact measuresfor libraries and
information services
Roswitha PollMünster
Bielefeld2006
Bielefeld2006
The importance of libraries
What libraries ensure (IFLA)
Democraticvalues
Economicprogress
The open information
society
Information equality
Reduction of poverty
Information literacy
Intellectual freedom
Culturaldiversity
Libraries make a differenceBut who will believe it?
Bielefeld2006
Bielefeld2006
inputinput
activitiesactivities
impactimpact
outcomeoutcome
valuevalue
outcome
a contribution of work, information, or materiala contribution of work, information, or material
activities change input into outputactivities change input into output
the effect or influence of one person, thing, orthe effect or influence of one person, thing, oraction, on anotheraction, on another
the consequence, visible or practical result orthe consequence, visible or practical result oreffect of an event or activityeffect of an event or activity
the importance or preciousness of something,the importance or preciousness of something,the perception of actual or potential benefitthe perception of actual or potential benefit
outputoutput the amount produced, the results suppliedthe amount produced, the results supplied
Bielefeld2006
Bielefeld2006
Don Revill
ACRL
Peter Brophy
Different outcomes
positive — negative
direct/immediate — long-term
actual — potential
intended — unexpected
Bielefeld2006
Bielefeld2006
Impact/outcome of cultural institutions
knowledge
information literacy
academic or professional success
social inclusion
individual well-being
changes in skills, competences,
attitudes, behaviour
Bielefeld2006
Bielefeld2006
Outcomes of universities
recruitment and retention of students and excellent academic staff
effective teaching - high graduation rates - high grades in examinations - high employment rates after examination
effective research- high renown and use of research results and publications - high amount of special grants - awards, honours
Evidence needed:The library supportsthese goals
Bielefeld2006
Bielefeld2006
Problems of assessing outcome
Value and outcome of a service can differ between user groups
Data may not be available because of data protection rules
Results of projects are often not comparable because of different data collection methods
Long-term effects cannot be assessed as persons are no more available
All tested methods are time-consuming
Influences on individuals are
manifold
Bielefeld2006
Bielefeld2006
Methods for assessing impact
Quantitative measures
tests assessing user skills before and after training on a service
performance monitoring / data mining
observation (unobtrusive)
analysis of documents and bibliographies compiled by users
correlation between library use and success
Bielefeld2006
Bielefeld2006
"Surrogate measures" have
to be used
Methods for assessing impact
Qualitative measures
Surveys (print, telephone, online)
Interviews
Focus groups, discussion groups
Self-assessment of users
Facts and stories must be organized to show patterns
The "anecdotal evidence"
supports data
Bielefeld2006
Bielefeld2006
ACRL
Jennifer Cram
User satisfaction as outcome measure?
Bielefeld2006
Bielefeld2006
A change of behaviour?
Or a basis forsuch change?
Correlation of library useand academic success
Frequency (loans, visits)Frequency (loans, visits)Range of services used (reference service, user training, ILL)Range of services used (reference service, user training, ILL)Range of collections used (undergraduate collection, E-journals, special Range of collections used (undergraduate collection, E-journals, special material) material) Attendance at training sessionsAttendance at training sessions
Success, expressed bySuccess, expressed byDuration of studiesDuration of studiesGrades in examsGrades in examsStudent retentionStudent retentionEmployment rate after examEmployment rate after examQuality of publications (impact factor,Quality of publications (impact factor,peer-review journals)peer-review journals)
Library use, expressed byLibrary use, expressed by
Assessing impact
Bielefeld2006
Bielefeld2006
The library's impact on information literacy
• surveys after instructionsurveys after instruction
• pretest / posttestpretest / posttest
• self-assessment of usersself-assessment of users
• behavioural observationbehavioural observation
• transaction logstransaction logs
• longitudinal studieslongitudinal studies
• assessing changes in students' bibliographiesassessing changes in students' bibliographies
Outcome of a specific training or series of Outcome of a specific training or series of trainingstrainings
Assessing impact
Bielefeld2006
Bielefeld2006
Importance of the local library for research
2 2 Estimated percentage of the information neededEstimated percentage of the information needed for a publication / paper / report found via the local libraryfor a publication / paper / report found via the local library
• questionnaire/interview after publicationquestionnaire/interview after publication
11 Percentage of material citedPercentage of material cited in academic publications in academic publications that was (could have been) retrieved via the local library.that was (could have been) retrieved via the local library.Citations inCitations in• dissertationsdissertations• student papersstudent papers• publications of a facultypublications of a faculty
Assessing impact
3
Bielefeld2006
Bielefeld2006
Impact on social inclusion and community life
Does the existence of a specific library / of libraries further• individual well-being• the quality of life in a society
- democracy - social inclusion - cultural life - local identity - life-long learning
Method: Assessing public opinion (users and / or non-users) as to the benefits of libraries and special library services
• to the population (indirect benefit)to the population (indirect benefit)• to the individual user (direct benefit)to the individual user (direct benefit)
Assessing impact
4
Bielefeld2006
Bielefeld2006
The financial value of libraries
Assessing impact
5
Assessing time costs (replacement value of a client's time)Assessing time costs (replacement value of a client's time)Users invest time and effort in order to use library services. The Users invest time and effort in order to use library services. The value that they - or their institution - place on that use must be at value that they - or their institution - place on that use must be at least as high as their „sacrifice“ of time.least as high as their „sacrifice“ of time.(Costs are calculated by time and average salary)(Costs are calculated by time and average salary)
Using the contingent valuation methodUsing the contingent valuation method
Willingness-to-pay What would users pay for the What would users pay for the maintenance of a service? maintenance of a service?
Willingness-to-accept Which sum would users accept as Which sum would users accept as equivalent for the deletion of a service?equivalent for the deletion of a service?
Bielefeld2006
Bielefeld2006
For every £1 of public funding the British Library receives each year,
£4.4 is generated for the economy
Using the results
accountability
decision making
resource management
improvement of services
promotion of the library's role
Libraries on the agenda
(Claudia Lux)
Bielefeld2006
Bielefeld2006
Outcome projects
eVALUEd: in "evidence base", University of Central England: Toolkit for evaluating electronic information serviceshttp://www.evalued.uce.uk/index.htm
IMLS (Institute of Museum and Library Services): outcome-based evaluation of projectshttp://www.imls.gov/index.htm
ARL New Measures Initiative: several projects - Learning outcomes - Higher education outcomes research review - MINES (Measuring the impact of networked electronic services)http://www.arl.org/stats/newmeas/index.html
Bielefeld2006
Bielefeld2006
Outcome projectsIBEC: Information School of the University of Washington and University of Michigan School of Information: Toolkit for assessing the impact of information in communitieshttp://ibec.ischool.washington.edu/default1024.aspx
SCONUL and LIRG (Library and Information Research Group): impact initiative and mailing listhttp://www.sconul.ac.uk/activities/performance/impact.html
IFLA Section Statistics and Evaluation: working group on outcome/impact; bibliography of literature and projects worldwide http://www.ulb.uni-muenster.de/outcome.html
Bielefeld2006
Bielefeld2006