performance indicators for academic libraries in malaysia · ** pustakawan, perpustakaan sultanah...

Performance Indicators forAcademic Libraries in Malaysia

Rosna Taib**

ABSTRACTThe performance of an academic library is one of the criteria for accreditation of an academic

programme and ranking of a university, PERPUN, through its task force, is looking into the designof an instrument to measure performance indicators of academic libraries in Malaysia. After

examining similar efforts in other parts of the world, the task force adopted the ISO 11620"Information and documentation - Library performance indicators." The task force will do a pilot

run of the instrument before applying it to the member libraries.

THE ACADEMIC LIBRARY

An academic library is the seat of knowledgein a university or college. It welcomes thebirth of a new knowledge, nurtures theexisting one and preserves the old knowledge. It iswhere rationale and reasoning intertwine, opposingideologies co-exist and the old and new sciencesreinforce each other. It contributes to thedevelopment of the scholar through the strength ofits collection, its services, technology and its staff.

The library is one of the criteria for accreditation,evaluation and rating of academic programmes aswell as ranking of universities and colleges. Theacademic library supports directly and contributesto the success of an academic programme, more sothan other support services of a university or college.Besides being accountable to the parent organizationi.e the university or college, a library is expected toprove the value of its contribution to achieving theoverall goals of the parent constituency. The library'sservices are expected to support the educational lifeof students while its collections are expected to meetcurricular, research and service needs of students andfaculty with particular attention to special programmeor objectives of the parent organization.

The accreditation process takes into consideration notonly the size of a library's collection, but moreimportantly how well the library helps to prepare thestudents for their future careers. Managementgenerally favours the young professionals who couldwork independently and show certain level of abilityin conducting search for facts and in report writing.Employers do expect for at least knowledge level onefrom graduates of accredited schools or programmes.This is especially true in the present scenario ofinformation profusion where it is imperative oneknows how to evaluate the value of a certain piece ofinformation.

With the rapid development of information andcommunication technology (ICT), there is increasingdemand for some form of distance education. Thisdevelopment is the added impetus that has led to thevirtual library concept. The form and servicesprovided by this kind of library would certainly beof greater importance in the accreditation process.University ranking can boost the standing of auniversity or bring it down in the eyes of the publicand the university's stakeholders. This has drivenuniversity management to identify niche areas wherethey could excel.

** Pustakawan, Perpustakaan Sultanah Zanariah Universiti Teknologi [email protected],utm.my

* Based on paper presented al PPM/PSZ Conference on Knowledge Management in the Digital World: Harnessing Information Skills forCompetitive Edge. Johor Bahru, 22-24 October 2001.

Hak Cipta Terpelihara © 2002 – Perpustakaan Negara Malaysia


THE QUALITY FACTOR

A library has to align itself with the criteria by whichuniversities and academic programmes are judged.Without doubt this will lead to a discussion on theefficiency and effectiveness of the library. Efficiencyis the measure of the utilization of resources in orderto realize a library's objectives. Effectiveness is thedegree to which a library's objectives are realized.Both efficiency in its performance and effectivenessin satisfying stated and implied needs of its customersconstitute the quality by which a library is known.The term "measure" to some people implies thatsomething can be measured in a direct way."Indicator" is considered more relevant in the contextof performance and evaluation. PerformanceIndicator is therefore defined as the verbal ornumerical expression derived from library statisticsand other data used to characterize the performanceof a library.

There are several models of performancemeasurement found in the literature on this topic,among them stressing on goal attainment, systemresource, internal process and constituencysatisfaction. Generic measures include input, outputand service domain while derived measures refer tooperational performance, operational effectiveness,cost effectiveness and impact. Yet other modelsmeasure process, organization, interaction and access.It is interesting to note in one of the articles writtenin the nineties, the question was posed whether theISO 9000 could be adopted as an indicator of quality.To date at least three academic libraries in Malaysiahave been certified to the ISO i.e. PerpustakaanSultanah Zanariah of Universiti Teknologi Malaysia,Perpustakaan Sultanah Bahiyah of Universiti UtaraMalaysia and Universiti Sains Malaysia Library.

The need for evaluation of academic library serviceswas first discussed at an international forum duringthe International Federation of Library Associations(IFLA) Conference in 1988. The expressed aim wasto convince the funding authorities of the relevanceof the libraries' goals and to show their effectivenessto obtain the required financial support for theirservices. The first draft of guidelines and indicators

that have been tested in some libraries was presentedat IFLA 1993. This draft, plus comments and resultsof further testing were incorporated in an internationalstandard on performance indicators for librariesproduced by The International Organization forStandardization i.e. ISO 11620 Information anddocumentation - Library performance indicators in1998. The objective of these indicators is "to functionas tools to assess the quality and effectiveness ofservices provided by a library and of other activitiesundertaken by a library, and to assess the efficiencyof resources allocated by the library to such servicesand other activities." The standard claims to havebeen thoroughly tested by widespread use in librariesor by explicit testing by researchers. The standardshave been formulated, guided by the followingcriteria:

• Informative content• Reliability• Validity

AppropriatenessPracticality

• Comparability.

At least 29 indicators have been put together,covering major activities of a library such asproviding and retrieving of documents, lending anddelivery of documents, enquiry, reference andinformation searching, facilities and technicalservices. Each indicator has a unique, descriptivename, and an explicit objective, staled in terms ofthe service, activity or use of resources to beevaluated. The scope and definition of each indicatorare specified. The standard also cautions that theindicators should be used in the context of themission, goals and objectives of the library and ifcomparison is to be made, it should be between likeentities only. The indicators are only tools forplanning and decision-making, facilitating control inthe management process and as a basis for referenceand dialogue between library staff, funding bodiesand the user community.

In the United States of America, two prevailingstandards seem to be used. Between the Associationof College and Research Libraries (ACRL) and the



Regional Association of Schools and Colleges,twenty (20) areas of library management areaddressed as criteria for purposes of accreditation.

LIBRARY PERFORMANCE IN MALAYSIA

An earlier study of library performance indicatorswas done as a short-term research project by a teamcomprising librarians and academic staff of UniversitiTeknologi Malaysia (UTM) in 1995. The studyinvolved 17 academic libraries in Malaysia,Singapore, Thailand and Hong Kong. Respectingconfidentiality, requested by the respondents, the datawere used only in UTM for purposes of assessingthe standing of its library in different aspects oflibrary management. Although the exercise was notrepeated, it nevertheless has given a good indicatorwho UTM could benchmark its Library with.

In 1997 "Asiaweek" conducted a ranking ofuniversities in Asia, leaving mixed feelings in somequarters. That was the first time any ranking ofuniversities has been done in this region and for manyof the universities, it was also the first time to beranked against others. The magazine has since thensuccessfully done so every year1. In some ways it isan event anticipated by many in the middle of eachyear. About the same time too Lembaga AkreditasiNegara (LAN) or the National Accreditation Boardwas set up. Both Asiaweek and LAN consider thelibrary as one of the criteria in the ranking, with LANgiving more details on how the libraries were to beevaluated. LAN announced that amongst its firstassignments was to do a ranking of the universitiesin Malaysia. Although this exercise has not beendone yet, it would be proprietary to be prepared forit.

Persidangan Perpustakaan Universiti dan Negara(PERPUN) or Conference of University and NationalLibraries has taken the initiative to set up a task forceto come out with some set of indicators which wouldreflect how they felt academic libraries should be

evaluated. With increasing cost of library materialsand decreasing resources given to libraries, there is areal need to justify for budget allocation and to provehigh quality and performance of services. The TaskForce on Performance Indicators2 met several timesand discussed how best Malaysian academic librariescould be evaluated. They examined the indicatorsused in the United States as found in the ACRLStandards for College Libraries and the accreditationcriteria as stipulated by the Regional Association ofSchools and Colleges. They also examined thesituation in the United Kingdom and also took notethat a series of workshops were conducted in Europeon the implementation of performance indicators.Several projects were conducted in Europe under theEU Telematics for Libraries initiative as a result ofadopting the ISO indicators.

ISO 11620 seems to have incorporated most of thecriteria laid down by all these parties. Consideringit is an international standard and adopted in severalcountries, the Task Force has decided to use it asbase for coming out with an instrument that could beused by all participating libraries. The hope ofPERPUN is so that there would be an instrument thatcould be used for self-evaluation as well as forcomparison of similar kind of libraries. The TaskForce recognized that different libraries requiredifferent indicators to suit their needs and no one setof indicators will suit all libraries. Therefore it wasdecided that the application of this instrument willbe done as a pilot project between the public-fundedlibraries first since there are similarities in goals andobjectives as well as funding and control practicesamong them. They all are accountable in general tothe same Ministry and are governed by similar set ofrules and regulations. The instrument could beextended to the private universities and colleges,depending on the success of the pilot run.Modifications will have to be done to the instrumentto suit it to the environment of private institutions.

1 Asiaweek ceased publication at the end of 200J.2 Author is Chairperson of Task Force on Performance Indicators set up by PERPUN.



ADOPTION OF ISO 11620

The ISO 11620 contains 29 indicators residing onseveral data sets. The standard is divided into twosections, one for the qualitative measure and the otherfor quantitative measure. The qualitative aspectmeasures customer satisfaction under the headingUser Perception. The quantitative section attemptsto measure inputs of the library against the populationor other variables. Using the standard closely asguideline, the Task Force was able to produce a draftinstrument that could be used in all PERPUN memberlibraries. The Task Force decided to concentrate firston indicators for the performance of activities andservices. This is because most libraries regularlycollect statistical data of their resources and output.By doing so, usefulness of existing data could bemaximized, increasing understanding of trends andpatterns and hence contribute to the decision makingprocess. Draft questionnaires, which would try toelicit information on both the quantitative andqualitative aspects of library service and products,were prepared. To date only the draft on thequantitative indicators have been deliberated. Inprinciple. PERPUN, at a meeting in February 2002,has agreed to adopt the instrument presented by theTask Force.

The Task Force's guiding principles in selecting andinterpreting the ISO standard is that any indicatoradopted must be useful in the decision-makingprocess and the method to obtain data on theseindicators should not be too complicated norobtrusive. Library management is hard put as it iswith the demands from its clientele and all itsconstituents without having more weight added toits already heavy burden. Most important, anindicator should answer the question for whatpurpose do we collect data and formulate into anindicator? If it does not help the management inplanning its resources, its services and forecast thefuture, the indicator is not valid. Another premiseestablished by the Task Force was that theseindicators are for the use of the public-fundeduniversity libraries that are all automated and areheavy users of information technology. Since theseindicators are measurement of performance in a

certain year, all one-time expenditures for purposesof asset building, are excluded in the calculation of alibrary's expenditure

The Task Force started its assignment by establishinga common understanding of terminologies and theirdefinitions. This is required of the ISO standard,which is actually only a guide that could be adoptedby different kinds of libraries, at different levels ofdevelopment and operating in different kinds ofenvironment. Even in a group of homogeneouslibraries like those of public-funded universitylibraries, a discussion on terminologies and theirdefinitions could lead to long debates. The TaskForce also discussed and agreed on the samplingmethod to be used for certain indicators.

CRITIQUE OF ISO 11620

Understandably it must have been a colossal task tocome out with the standard, but it is for the very reasonthat it has several setbacks. In the measurement of theactivities, services and library management, ISO 11620is quite thorough. So it came as a surprise that it couldnot offer any measure on certain aspects of librarymanagement and activities that are important to alibrary especially today. This includes areas of UserEducation, Promotion of Services and the Availabilityand Use of Human Resources. Most libraries havesome clear ideas on the measure of performanceindicators in other more traditional library activitiesand services and have looked forward to a guide froman international body like the InternationalOrganization of Standards. In the absence of reliableindicators in ISO 11620, The Task Force hasformulated indicators for these three areas of concern.A first test of an indicator is that the method ofderiving the indicator should meet the objective ofthe indicator. The objective of the Median Time ofDocument Retrieval from Open Areas is "to assesswhether self-explanatory sign-posting and correctshelving allow prompt access to documents." Butthe method of collecting the data requires a test personto check a random listing of titles against thecatalogue and to retrieve them from the shelves.Distance covered, whether climbing of stairs is



involved and whether the materials are near thestaircase or in the distant corner of the floor wouldaffect the speed of retrieval. This kind of indicatoris better elicited through the use of questionnaire andtherefore belongs to the User Perception part of thestandard.

The use of proxy users or surrogates for certainindicators is a worrying proposition, It is certainlyan obtrusive method and it is not good managementpractice to expose the weakness of the library staffto "outsiders". This practice could also underminethe confidence of library patrons on the capability ofthe library staff. The Task Force is of the opinionthat a library manager who is concerned on thecompetencies of his/her staff could easily conductthis test internally.

The standard does not make any distinction betweenthe different kinds of enquiry and reference questionsa library gets from its customers. Ambiguity shouldbe avoided in a good standard. The Task Force hasidentified four types of enquiries that are normallyreceived by academic libraries i.e. directional,instructional, quick reference and information search.These different types of enquiries take differentamount of library staff time and resources, thereforeimpacting staff deployment and resource allocation.

Computers are widely used in academic libraries andincreasingly, more information is offered as non-printmaterials, especially in machine-readable forms. Inan environment where library users have beenexposed to the quick search on OPAC through theuse of keywords and free-text indexing, a test oninformation retrieval distinctly by "title" and"subject" is quite irrelevant. Similarly the indicatorAutomated System Availability does not reflect thestage of library development in Malaysia. It is thegeneral practice for a server to run twenty-four hoursnon-stop and therefore making the catalogue moreaccessible. After all this is the raison d'etre forautomation. This indicator is better replaced by aratio of computers to the number of users andexpenditure on electronic materials per user to reflectthe use of electronic resources and services inlibraries. Indeed the absence of indicators for the

application of information technology is verynoticeable, considering the heavy investment ininformation technology especially by academiclibraries in Malaysia. The same sentiment is felt inEurope where several projects are underway toaddress the need of all libraries to develop and utilizeperformance measures for the new networked,electronic environment, alongside traditionalmeasures.

CONCLUSION

The use of the standard ISO 11620 is a good start tomeasure the performance of academic libraries inMalaysia provided some modifications are done.Such exercise should measure and incorporate libraryvalue into processes that involve the universitiesconcerned such as academic accreditation,educational assessment and rating of certainprogrammes. Consequently it contributes to theorganizational analysis and decision-making processof the universities. This could well be the beginningof benchmarking among academic libraries inMalaysia.

A set of performance indicators should necessarilypay attention equally to the traditional analysis ofdata of services and products as well as to establishinguser satisfaction as a performance indicator. Afterthe initial round of applying the instrument asformulated by the Task Force, the next step foracademic libraries in Malaysia is to develop outcomemeasures that convey customer expectation andmeasure both user satisfaction and the impact oflibrary services on their customers.



REFERENCES

ACRL College Libraries Section. StandardsCommittee. Standards for college libraries: adraft. h t t p : / / w w w . a l a . o r g / a c r l / g u i d e s /standraft.html/. 15th January 2000.

Blagden, John and Jane Barton (1998). Can youcompare one university's library performancewith another? Proceedings of the 2nd NorthumbriaInternational Conference on PerformanceMeasurement in Libraries and InformationServices, Northumberland. England. 7-11 Sep.1997. Newcaslle-upon-Tyne: InformationNorth,1998.

Bowden, Lisa (2000). EQUINOX : Deliverable 2.1.2Annual Report.http://equinox.deu.ie/reports/annual2/d212.html. 27thth August 2001.

Brinkley, Monica (2001). The EQUINOX Project.Library performance measurement and qualitymanagement system.http://www.exploit-lib.org/issue3/equinox/. 27th August 2001.

Brophy, Peter et al. (2000). EQUINOX : performanceindicators for electronic library services, http://equinox.deu.ie/reports/pilist.html. 27th August2001.

Brophy, Peter (1989). Performance measurement inacademic libraries: a polytechnic perspective.British Journal of Academic Librarianship, vol.4 no.2 1989:99-110.

Calvert, Philip J. (Jul. 1998). A different time, adifferent country: an instrument for measuringservice quality in Singapore's polytechniclibraries. Journal of Academic Librarianship, vol.24 no.4 : 296-304.

Clarke, Zoe, Alan MacDougall, Shelagh Fisher(1998). An opportunity for software developmentin library quality management and performancemeasurement: the ECLIPSE experience.Proceedings of the 2nd Northumbria InternationalConference on Performance Measurement inLibraries and Information Services,Northumberland, England, 7 -11 Sep. 1997.Newcastle-upon-Tyne: Information North : pp.405412.

Coleman. Paul and Ada D. Jared (Nov. 1994).Regional Association criteria and the "Standardsfor college libraries": the informal role ofquantitative input measures for libraries inaccreditation. Journal of Academic Librarianship,pp. 273-277.

Ebbinghouse, Carol (1999). Library standards:evidence of library effectiveness andaccreditation. Searcher, vol. 7 no. 8: pp. 20-26.

Harnesk, Jakob (1998). The ISO standard on libraryperformance indicators (ISO 11620). Proceedingsof the 2'"' Northumbria International Conferenceon Performance Measurement in Libraries andInformation Services, Northumberland, England,7-11 Sep. 1997. Newcastle -upon-Tyne:Information North : pp. 61-70.

International Organization for standardization (1998).Information and documentation - Libraryperformance indicators. Geneve. (InternationalStandard, Reference number ISO 11620:1998(E)).

Jacobs, N.A. (1995) and R.C.Young (1995).Measuring book availability in an academiclibrary: a methodological comparison. Journalof Documentation, vol. 51 no.3: pp.281-290.

Kyrillidou, Martha and Julia Blixrud (2001). Currentcontext for performance indicators in highereducation: introduction, http://www.arl.org/stats/arlstat/context98.html.

Poll, Roswitha and Peter te Boekhorst (1996).Measuring quality: international guidelines forperformance measurement in academic libraries.Munchen: K.G.Saur. (IFLA Publication, 76).

Pritchard, Sarah M. (1996). Determining quality inacademic libraries. Library trends, vol. 44no.3: pp. 572-594.

Sabariah Din et al. (1996). Library usage study(academic libraries in Malaysia). (Unpublished).

Willemse, John (1998). Performance assessment inIFLA and United Kingdom academic libraries.South African Journal of Library & InformationScience, vol. 66 no. 4: pp. 161-166.



performance indicators for academic libraries in malaysia · ** pustakawan, perpustakaan sultanah...

Documents