6th Annual Hong Kong Innovative Users Group Meeting

9th December 2005

Lingnan University

WebPAC

vs

Selected Databases

A Comparative Study of Searching Efficiency and Effectiven

ess

by

Eugenie NG

(Acquisitions Librarian, LU Library)

&

Patrick LO

(Cataloguing Librarian, LU Library)

Background

Development of the library online catalogue1st stage-electronic card catalogue-access bibliographic information of the print collection-character based-command driven

2nd stage-more and better searching capabilities-access to reference works, full-text journal articles and graphics

Background

3rd stage

-gateway to the web

-portal to library resources, including Z39.50 database and digital collections

- One stop search

LU Library WebPAC

- Lingnan Library online catalogue is at the 3rd stage o

f development. The Library WebPAC has been converted into an Electronic Resource Portal to provide one stop search for users.

- LU users now have a choice to search online information through WebPAC or an alternative database or on the Web.

LU’s Resource Portal

Users can search WebPAC or alternative data

bases directly

Purpose of Study

To find out whether or not:

“Searching for defined information through WebPAC is more efficient and effective than searching directly in alternative databases.”

Purpose of Study

Definition

“Efficiency” is the time for locating the assigned titles.

“Effectiveness” is the number of titles / records retrieved within the time limit.

Purpose of Study The underlying assumptions:

- Users are more experienced in using WebPAC than using an alternative database.

- Users are able to access information directly from the library catalogue without repeating the search in individual databases.

- Interfaces of databases are different, users need time to figure out the functionalities.

- Users need time to find out and pick the right database.

Method of the Study

Three databases were selected for the study:

Naxos Music Library : Online Audio-music Database

Naxos Spoken Word Library : Online Audio-book Database

NetLibrary : E-Book Database

• Titles in these databases are catalogued and accessi

ble from the WebPAC.

WebPAC Display of NetLibrary Titles

Naxos Spoken Word Library Titles at WebPAC

Naxos Music Library Titles at WebPAC

Design of the Search Exercise

Exercise Min. no. of search

A (10 titles) WebPAC NetLibrary 20

B (10 titles) WebPAC NSWL 20

C (10 titles) WebPAC NML 20

•Total 30 titles. • 30 participants

– 10 Humanities; 10 BBA; 10 SocSci.– 10 year 1, 10 year 2, 10 year 3

Sample Citations for the Exercises

Ex. A : Titles from NetLibrary

• The origins of music [electronic resource] / edited by Nils Wallin, Björn Merker, and Steven Brown

• The literate executive [electronic resource] / Laurie Rozakis

• The Chinese language [electronic resource] : fact and fantasy / John Defrancis

Ex. B: Titles from NSW Library• Divine Comedy, Part 1 – The Inferno by Dante Alighieri (unabridged)

• The Sonnets by William Shakespeare

• Anna Karenina by Leo Tolstoy

Sample Citations

Ex. C : Titles from Naxos Music Library

• Mozart Piano Concerto (any recording)

• Italian opera Tosca, featuring singer Maria Callas (soprano)(complete opera recording)

• Yellow River Concerto (「黃河」鋼琴協奏曲 ) / Butterfly Lovers Violin Concerto (「梁祝」小提琴協奏曲 ), featuring Shanghai Conservatory Symphony Orchestra, featuring violinist, Nishizaki Takako (西崎崇子 )

Method of Study

• The time for retrieving each title was recorded.

• About 30 min. for each exercise.

• Participants also filled in a short questionnaire, which elicited background information, e.g. search experience etc. before and after the searching exercises.

Results (1a) : EfficiencyEfficiency : the time used for searching

Average Search Time Per Exercise

Ex. A Time (min.) Ex. B Time (min.) Ex. C Time (min.) A+B+C

WebPAC (a)

5.32 WebPAC 4 WebPAC 8.56 17.88

NetLibrary (b) 7.05 NSWL 13.22 NML 18.71 38.98

Difference (c)

1.73 Difference 9.22 Difference 10.15 21.1

% c/b 24.5% % 69.7% % 54.2% 54%

Results (1b) : EfficiencyAverage Search Time By Year

　 　 　 　

Year WebPAC Databases Difference %

Year 1 18.26 38.92 20.66 53%

Year 2 19.41 38.48 19.07 50%

Year 3 15.86 39.52 23.66 60%

By Programme 　 　 　 　

Programme WebPAC Database Difference %

Humanities 17.02 38.96 21.94 56%

BBA 19.02 38.84 19.82 51%

SocSci 17.48 39.13 21.65 55%

By Gender 　 　 　 　

Gender WebPAC Database Difference %

Female 17.28 38.61 21.33 55%

Male 18.8 39.61 20.81 53%

Results (1c)

Results confirmed that searching in WebPAC was more efficient as

– Less time was used for searching in WebPAC than alternative databases

– Less time for searching in WebPAC was also shown by analyzing data by Year, Programme and gender.

Results (2a) : Effectiveness Number of titles retrieved within time limit

A. Count by Title

EX. A no. of Success % no. of Failure %

WebPAC 296 99% 4 1%

NetLibrary 300 100% 0 0%

EX. B

WebPAC 297 99% 3 1%

NSWL 271 90% 29 10%

Ex. C　

WebPAC 275 92% 25 8%

NML 212 71% 88 29%

Results (2b): EffectivenessNumber of participants who succeeded in retrieving all titles within time limit

B. Count by Participant

Ex. A　 no. of success % no. of failure %

WebPAC 27 90% 3 10%

NetLibrary 30 100% 0 0%

Ex.B　

WebPAC 28 93% 2 7%

NSWL 16 53% 14 47%

Ex. C　WebPAC 12 40% 18 60%

NML 1 3% 29 97%

Effectiveness (Summary)Success by Title Success by Participant

Overall Success Rate

WebPAC (95%) > Databases (87%)

WebPAC (74%) > Databases (52%)

Rank Interfaces by Success Rate

NetLibrary>WebPAC>NSWL, NML

NetLibrary>WebPAC>NSWL, NML

Failure Rate for NML

29% 97%

Discussion: Why More Efficient/Effective with WebPAC?

• Information from the questionnaire (Pre-test) showed that

All participants were more familiar with the WebPAC than the selected databases.

Prior experience with Selected Databases

No of ParticipantsNetLibrary 1

NSWL & NML 1None of them 28

Training for WebPAC Search

No. of Participants

%

Yes 14 46.7%

No 16 53.3%

Frequency of WebPAC Use

Frequency Number of Participant

%

Daily 4 13%

Several times / week

17 57%

Several times / month

9 30%

Never 0 0%

Students’ Search Habits

Books / Serials AV Materials

No. of students

% No. of students

%

Search WebPAC First

15 50% 16 53.3%

Browsing the Shelf First

2 6.7% 7 23.3%

Both 13 43.3% 7 23.3%

Discussion: Why More Efficient / Effective with WebPAC?

B: Information from the questionnaire (post test): • To rank the interfaces by User-friendliness & Prefe

rence :• WebPAC - highest scores on a 5-point ranking sc

ale.

Ranking of Interfaces

Scores User-

friendlinessPreference

WebPAC 115 113

NetLibrary 86 88

NSWL 59 60

NML 40 39

Effective rate is high with Netlibrary as its interface is simpler and close to WebPAC

NSWL lnterface looks much different from WebPAC

NML interface is entirely different from WebPAC

Limitation

1. Time constraint for measuring“Effectiveness”: 3 min. is given for retrieving one title, over 3 min. will be treated as failure. -The time was set arbitrarily after a pilot study. It was fo

und that the time for the 3 searching exercises would drag for too long and participants became impatient if a time limit was not set. It would be difficult to find for participants.

2. Small sample size (10 titles from each database)

Conclusion/Recommendation • We can advise users to check WebPAC first if they have de

fined information such as the TITLE.

• As users know WebPAC better and use it more often, titles of databases not known by users will be retrieved. So cataloguing and linking titles of these databases will likely increase usage.

• Databases with interfaces entirely different from WebPAC may be better promoted by information classes.

.

Further Research

1. To compare with word/subject search

2. To compare with journal or multi-disciplinary databases of high usage

Are we going to have similar results?

Further Research

3. Failure rates were higher for searching music titles.

a . Interface? – what are the features needed for locating music

titles? – what does OPAC lack?

b. subject knowledge?

Students’ Music Training

Average Search Time (min.)

WebPAC Database Overall

Yes (A) 16 38 54

No (B) 19 39 58Difference 3 1 4% (B-A)/B 15.8% 2.6% 6.9%

Music training = Taking music lessons other than school lessons.

Thank You!

Feedback :

eugenien@ln.edu.hk or patrickl@ln.edu.hk