a.m.tammarouniversity of tbilisi, 5-15 july 2010 evaluation of digital library anna maria tammaro...
TRANSCRIPT
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Evaluation of Digital library
Anna Maria TammaroUniversity of Parma
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Outline What is the digital library? Why evaluate? Evaluation cycle
What? How?
Good practices
A.M.Tammaro University of Tbilisi, 5-15 July 2010
What is a digital library? What is encompassed?
Visions of library
What elements to take?
What is critical?
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Why evaluate? Evaluation is a fact
finding,
Evidence based value measuring,
Integrated in the management process of digital libraries
Accountability: evidence of resources spent
Effectiveness: understanding basic phenomena (as information seeking)
Impact: as increased learning, research, dissemination
A.M.Tammaro University of Tbilisi, 5-15 July 2010
4 major questions for evaluation What actually
occurred? How can it be
improved? Did it accomplish the
objectives? What impact did it
have?
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Description What actually occurred? Documentation
Evaluation, MIS
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Improvement How can it be
improved? Formative evaluation
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Fit for purposes? Did it accomplish its
objectives? Effectiveness
evaluation
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Impact of digital library What impact did it have?
The ultimate question for evaluation is:
“How are digital libraries transforming research,education, learning and living?” (Saracevic 2002, p. 368)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
What evaluate? Content
Services/system
Users and uses
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Content evaluation Content quality (subject coverage,
relevance) Content scope (what is included? Online
journals, ebook) Content organisation (metadata,
bibliographic organisation,indexing) Effectiveness (management, user support) Efficiency (cost)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
System interface Interface (usability, design, accessibility) System performance
(interactivity,algorithms for searching, processing time)
System configuration (networks, security,authentication)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Outcomes The ways in which library users are
changed as a result of their contact with the library resources and programs (ARL 1998)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Outcomes based evaluation Have audiences been sufficiently identified? Are outcomes clearly written? Are outcomes sufficient to describe what you
hope will happen? Are data collection methods cost efficient?
Add: Do they provide the data you want and need?
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Users Who are they? (researchers, students, remote,
etc.? What is their context?) How do they access the digital library?
(infomation seeking behviour, usability) Why do they need the digital library? (activities,
expectations) What type of resources do they need? (subject,
etc.) What is the value of digital library? (impact,
outcomes, potential for community building)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
European Minerva Project Minerva Handbook on cultural web user interaction http://www.minervaeurope.org/
publications/handbookwebusers.htm
A.M.Tammaro University of Tbilisi, 5-15 July 2010
How to evaluate? Survey Focus group Interviews Transaction logs Observation Ethnographic
evaluation
Usability Combined methods Longitudinal studies Crosscultural
assessment Benchmarking
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Standard– COUNTER, SUSHI (NISO standard usage statistics
harvesting initiative) No benchmarking or longitudinal studies
(for the rate of change)
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Good practice DigiQual
– http:// www. digiqual. org/ PEAK
– http:// www. dlib. org/ dlib/ june99/ 06bonn. html
E- valued– http:// www. evalued. uce. ac. uk
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Bad news There is no single, easy to administer,
inexpensive, reliable, and valid approach to evaluating interactive learning from DLs.
A.M.Tammaro University of Tbilisi, 5-15 July 2010
Good newsThere are practical strategies fordocumenting the development anduse of interactive learning, improvingit, and building a case for its effectivenessand impact.
