excellence and evaluation in higher education: some...

HEP Contributions No. 11

Excellence and evaluation in higher education: s o m e international

perspectives

Bikas C . Sanyal

International Institute for Educational Planning

Included in the same series:

1. New strategies for financing diversified forms of education and training Sylvain Lourié

2. Educational planning: reflecting on the past and its prospects for the future1

Jacques Hallak

3. Education for all: high expectations or false hopes] Jacques Hallak

4. Educational planning looks towards the year 2000х

Françoise Caillods

5. Partnership in education: the role of universities in the Pacific Rim Bikas C . Sanyal

6. Educational policies in a comparative perspective: suggestions for a research agenda Jacques Hallak

7. Staff management in African universities Bikas C . Sanyal

8. Managing schools for educational quality and equity: finding the proper mix to make it work Jacques Hallak

9. Education in a period of change and adjustment: some international perspectives Bikas C . Sanyal

10. Capacity building for educational planning and administration: HEP's experience Gabriel Carrón

11. Excellence and evaluation in higher education: some international perspectives Bikas С Sanyal

1. Also published in French and Spanish.

Excellence and evaluation in higher education: some international perspectives

П Е Р Contributions N o . 11

Excellence and evaluation in higher education: some international

perspectives

by

Bikas C . Sanyal

This paper was presented at the Conference on Excellence in and Evaluation of Higher Education Systems in Comparative Perspectives held at the

Institute of Education, University of London, United Kingdom from 16 - 18 June 1992

International Institute for Educational Planning (established by U N E S C O )

The views and opinions expressed in this paper are those of the author and do not necessarily represent the views of U N E S C O or of the П Е Р . The designations employed and the presentation of material throughout this paper do not imply the expression of any opinions whatsoever on the part of U N E S C O or П Е Р concerning the legal status of any country, territory, city or area or its authorities, or concerning its frontiers or boundaries.

The publication costs of this document have been covered through a grant-in-aid offered by U N E S C O and by voluntary contributions m a d e by several M e m b e r States of U N E S C O , the list of which will be found at the end of the document.

This document has been typeset using I Œ P ' s computer facilities and has been printed in IffiP's printshop.

International Institute for Educational Planning 7-9 rue Eugène-Delacroix, 75116 Paris

© UNESCO October 1992

This lecture is being given on 16th June 1992 - the 90th birthday of Dr. C E . Beeby, w h o has devoted his life to improving the quality of education. Lecturer during the 1920s in philosophy and education at Canterbury University College, University of N e w Zealand, Beeb (as he is affectionately known around the world) became Director of the N e w Zealand Council of Educational Research in the 1930s and Director of Education in N e w Zealand from 1940 to 1960 (with a break in 1948-49 when he served as Assistant Director-General for Education at U N E S C O , Paris). H e was N e w Zealand's Ambassador to France in the early 1960s and Commonwealth Visiting Professor at the Institute of Education, University of London, in 1967-68.

This lecture is m y birthday present to Beeb.

Bikas C . Sanyal, 1 Ш Р , Paris

Contents

Page

1. Introduction 1

2. Different concepts of excellence in the international context 2

3. State of excellence in higher education: some examples in

the international context 9

4. Evaluation in higher education 13

4.1 The objectives of evaluation 13

4.2 History of evaluation of the higher education system 14 4.3 The approach of evaluation 15 4.4 Evaluation mechanism 16 4.5 The practice of evaluation in the developed countries 17 4.6 The prevailing state of evaluation in developing countries:

some examples 18 5. Problems of evaluation procedures 20

6. Impact of the evolution of the concept of excellence on evaluation procedure and vice versa 21

7. Conclusion 23

Excellence a n d evaluation in higher education: s o m e international perspectives

by Bikas C . Sanyal

1. Introduction

According to Webster's Third International Dictionary, excellence is defined as the state of possessing good qualities in an eminent degree, and quality is defined as the degree of conformance to a standard. Excellence in a higher education system would m e a n that the system possesses the characteristics of conforming to standards in an eminent degree i.e. to what extent the programme has achieved the desired outcomes. This leads to the identification of the desired outcomes or objectives of a higher education system. A s objectives change, the concept of excellence will also change and the objectives of higher education have changed with time as well as space.1

In ancient India the purpose of higher education of the universities of Taxila and Nalanda was to impart spiritual and mental skills to the students. Later in the Al-Azhar University of Egypt and the Universities of Fès and Rabat of Morocco, the same basic characteristic was used, where protection and preservation of Islamic culture and civilization had a more pronounced place, than instruction for material well being.

It is said that it was prohibited in the past to use the diploma of the University of Al-Azhar for any gainful profession. O n e m a y observe a similar situation in the mediaeval universities of Europe. Except for the University of Salerno, Italy, where medical science was being taught as early as the ninth century, all the other universities of Bologna, Paris, Oxford, etc. were emphazing spiritual and moral studies. In a keynote address, Charles Carter referred to the extent of indifference of the universities towards material well-being by saying that: "It is only two centuries since doctors were trained by apprenticeship (in the United Kingdom) and A d a m Smith was protesting at the folly of transferring their training to universities". But as time passed, the objectives of the universities and their functions and role in society was extended and modified to suit the particular societal context. The end of the eighteenth century saw the rise of polytechnics in Europe to serve socio-economic needs for skills. Elsewhere, two events marked a similar change during the nineteenth century, i.e the Land Grant College (Morrill) Act of 1862 in the U S A promoted the establishment of institutions of higher education for useful and mechanical arts and the Meiji Restoration in Japan promoted considerations of public utility and efficiency in its institutions of higher education. Objectives and functions are n o w changing rapidly, since it is perceived that the best w a y higher education can serve society is to prepare people capable of tackling the emerging problems that beset them. These problems are moral and ethical on the one hand and materialistic on the other, related to the basic necessities of life, namely food and nutrition, health and sanitation, housing and shelter. These basic needs can only be satisfied through economic activity, and the education system is required to provide the necessary skills in order that these basic needs m a y be supplied. In a static, subsistence economy, higher education m a y only need to bother about mind and character, in an information, industrial and

1. Charles F. Carter Keynote address, Third International Conference on Higher Education at the University of Lancaster, United Kingdom, 1-5 September 1975.

1


commercial economy, it has to cater for 'matter' as well. Clearly, higher education today cannot justify its existence by providing only academic learning for mental or spiritual development; it has to convey the necessary skills for economic development in addition to cultural and social development, in order that natural resources might be explored and exploited, products stored and distributed, services managed, and resources conserved for future generation. Such skills would not only enable solutions to be found to present problems but also prepare young people to work on the unpredictable, complex and dynamic problems of the future.

Together with the expectations and perceptions of the academic community, the functions of the higher education system have changed today. Today they are considered to be:

(1) Providing education and training within a structure integrating research and instruction.

(2) Providing professional training. (3) Carrying out research in a broad range of disciplines and training qualified

people for all fields of employment. (4) Playing a part in regional development and developing international contacts. (5) Fostering the intellectual and social development of society.

The importance of the above functions evidently varies from place to place and in time. So will the requirements for excellence change, so that it must be seen as intrinsically relative.

2 . Different concepts of excellence in the international context

About 25 years ago the International Institute for Educational Planning organized a symposium on 'the qualitative aspects of educational planning with particular reference to developing countries'. The symposium dealt with the concept of quality both intensively and extensively and most of its deliberations are valid even today. I shall therefore base m y presentation on them.

In dealing with the issue of excellence in the international context, it is important to m a k e a distinction between product and process. Measured by any test, the higher education system of a developed country will be able to produce, in most cases, graduates of higher excellence on average than a system of a developing country. Even within a country an elite institution of higher education m a y have an output of higher quality than other institutions, although its instructional processes m a y not be good. But if w e consider the graduates of an ordinary institution of higher education in a developing country and compare them with their background and educational levels when they entered the system, w e m a y find their achievements have been significant and the institution has in fact done an excellent job, i.e. the instructional process could be judged as excellent.

Another distinction in the concept of excellence is w h e n w e look at the system of higher education from within (internal excellence) and excellence viewed as the fitness of the system to the society it serves. This means that an institution m a y be judged to be internally excellent (i.e. attain the objectives it set out to achieve) yet be unfit in meeting the needs of

2

ПЕР Contribution No. 11

the society. For example, in a country where managers and engineers are already in short supply and the system produces humanistic and theological scholars, the system will have a low score on the scale of excellence, no matter h o w excellent its standard of scholarship. Similarly, a higher education system which produces only professional workers and no humanists will have the same problem. In a developing country, social demand for education is on the increase because of the instrumental value of higher education - its usefulness in getting suitable employment.

In a survey conducted by the International Institute for Educational Planning (IIEP), the importance of career prospects as perceived by the students and the graduates of selected countries is clearly demonstrated.2 (see Table 1)

Table 1. Reasons for pursuit of higher education (percentage of respondents)

Country

India ( W . Bengal)

Egypt (1)

Indonesia (2)

Philippines

Zambia

Tanzania

P D R Yemen (2)

Sudan (2)

Sudan (3)

Botswana (4)

Sri Lanka

Career prospects

63

2.1

72

86

85

95

69

70

[67]

95

56.5

Study for its o w n sake

21

2.0

8

-

9

4

9

23

[19]

5

-

Scholarship/ grants

11

1.4

0.4

-

1

1

-

-

[11]

-

-

Others

5

1.2

19.6

"-

5 -

22

7

[3]

-

-

Total

100

-

100

-

100

100

100

100

[100]

100

-

(1) T h e value is the rank order on a three-point scale. 1 denoting unimportant. 3 denoting very important. (2) Preference for urban life has been included. The figures for Indonesia, P D R Y e m e n , and Sudan were respectively

13.7, 17.6 and 3.3. (3) T h e figures in parantheses relate (4) Relate to graduate survey.

No t available.

to graduate survey of Sudan.

Educated members of the community are quick to recognize this. But less educated members are likely to cling to the old concept of excellence which is the intrinsic value of education i.e. its internal quality.

However the internal excellence of a higher education system m a y be brought more closely into line with the concept of excellence in the system's fitness to society's present and future needs3

B . C . Sanyal. 'Higher education and employment: an international comparative analysis', T h e Falmer Press, London and U N E S C O , Paris, 1987, p .38 .

C E . Beeby (ed). Qualitative aspects of educational planning, I I E P / U N E S C O , Paris, 1969.


I quote the following paragraphs from an article of D . Smith published in the World Yearbook of Education 1990, but originally written by Harold Benjamin4 to m a k e the above point:

'Once upon a time, in the paleolithic era, there was a tribe of cavemen. Their life was simple, but hard. Then as n o w , there were few lengths to which m e n would not go to avoid the pain and labour of thought. But a thinker arose. His n a m e was New-fist-hammer-maker (New-Fist, for short). H e thought out ways in which life might be m a d e better for himself, his family and his tribe. 'If I could only get children to do the things that will give more and better food, shelter and security', thought New-Fist, 'I would be helping the tribe to have a better life...'. Having set up an education goal, New-Fist proceeded to construct a curriculum. 'What things must w e tribesmen k n o w in order to live with full bellies, w a r m backs and minds free from fear?', he asked.

New-Fist identified three factors which were central to the life of the tribe:

(i) clubbing woolly horses for food; (ii) catching fish with bare hands; (iii) scaring sabre-tooth tigers with fire.

H e constructed a curriculum accordingly.

H e educated his o w n children according to his curriculum, and they thrived. S o m e of the other more intelligent members of the tribe followed New-Fist's example, and eventually the teaching of horse-clubbing, fish-grabbing and tiger-scaring came to be accepted as the core of all real education.

Gradually, being a statesman as well as an educational administrator and theorist, New-Fist overcame all opposition (practical, theoretical and theological) to his curriculum. Long after his death the tribe continued to prosper.

But times changed. A new ice-age came. Fish grew difficult to catch with bare hands. The woolly horses migrated away and were replaced by fast, agile antelopes. The sabre-tooths died out and their place was taken by enormous cave bears which were totally unimpressed by fire. The tribe was soon in danger of following the sabre-tooths into extinction.

Fortunately for the tribe, there were still m e n in it of the New-Fist breed, m e n w h o had the ability to do and the daring to think. Between them they solved the survival problem by inventing fishnet making, antelope snare construction and the theory, practice and operation of catching cave bears in pits. Once again the tribe prospered.

There were a thoughtful few w h o asked questions as they worked. S o m e of them even criticized the schools. They asked w h y these n e w subjects should not be taught. But the majority of the tribe had long ago learned that schools had nothing to do with real life, and anyway, the wise old m e n w h o controlled education had an answer: "That wouldn't be

4 . Benjamin, H . (1938, republished 1975). The Sabre Tooth Curriculum. In Colby M . , Greenwald, J. & West, R . (eds), Curriculum Design Milton Keynes, Open University Press. 7-14.

4


education; it would be mere training, and anyway, the curriculum is already too crowded. W h a t our people need is a more thorough grounding in the basics". The old subjects, it seemed, were taught not for themselves, but for the sake of generalized skills.

But the radicals persisted: "Times have changed. Perhaps these up-to-date activities have some educational value after all?" The wise old m e n were appalled: "The essence of true education is timelessness. Y o u must k n o w that there are some eternal verities, and the sabre-tooth curriculum is one of them!" The debate still goes on w h e n w e discuss curricular reform.

The other dimension of the concept of excellence deals with the three ideals of excellence: the productivist, democratic and humanistic. The productivist ideal of excellence is defined as one under which "the principal, if not the sole, aim is to produce as much as possible or to increase as much and as rapidly as possible the amount of goods available in the community" (Aron, 1969)5. This is different from the extrinsic value of education (the concept of fitness discussed above) in that it excludes non productive socially useful ideals of education and which apparently conflicts, at least in the short run, with the democratic ideal. The democratic ideal of excellence is that every capable individual should have the opportunity of higher education. It does not mean that every citizen must be given the same education. It means that the available educational opportunities should be distributed according to the student's ability to benefit from them rather than according to their socioeconomic status i.e. access to institutions of higher education should be based on merit. But given the educational development of the developing countries, merit itself is unequally distributed favouring the upper social groups.

In real life a compromise has to be derived, working at both ends of the scale and balancing the weights of the two factors. S o m e countries are concentrating on productivity in the short run and playing d o w n the 'softer principles' until they can afford them, lest both the academic excellence and the country's productivity suffer. It should be understood that the productivist ideal of excellence and the democratic ideal are interdependent and both should be given due weight which is not an easy task for a decision-maker. The democratic ideal of excellence will be out of place in a situation where most students try to crowd through the same path of higher education which is generally liberal types of university education while ignoring the n e w routes of higher education opening to them, especially the short cycle professional types.

The humanistic ideal of excellence is defined as the degree to which intelligences and personalities are trained in accordance with a more or less closely defined ideal. H u m a n i s m means different things to different societies. Here it is meant to cover all the aspects of the citizen's intellectual and cultural education that are not directly - or at least obviously -determined by economic or political ends6. The universities of the ancient and the mediaeval era noted in the introduction are concerned mostly with the humanistic ideal of excellence. Also as seen in the concepts of the intrinsic and extrinsic excellence of higher education above, by giving a broad structural base to professional and vocational education, one m a y

5. R . Aron, Sociological comments as concepts of quality and quantity in education in C E . Beeby (ed) ibid.

6. C E . Beeby, op. cit. p. 58.

5


reconcile the apparently conflicting productivist and humanistic ideals of excellence in higher education. Developing countries then m a y produce few philosophers but have more managers, engineers and entrepreneurs with some claim to be educated citizens.

A further dimension of the concept of excellence is the horizontal and vertical difference in the concept. A wide range of views exist on what is meant by excellence in any country at any point in time. The academic staff, the students, the parents, the employers, the researchers, planners and managers of the system - all w h o have vested interests in the higher education system tend to emphasize the importance of different components of it, leading to a broad spectrum of views. These are horizontal differences in the concept of excellence, w h e n the variations refer to the same place and time. The vertical difference relates to the stage of economic development of a country. T h e objectives and functions of higher education will change with the stage of economic development of a country in the international context. Changes in the objectives of higher education due to lower or higher economic status of each country, lead to vertical differences in the concept of excellence of higher education. It is not so obvious in higher education because of the belief in certain quarters that academic standards in the universities are universal and sacrosanct and once they are allowed to fall, it is impossible to recover. A s a result some universities in developing countries have tied their curricula and standards to those of their former ruling countries.

They have also refused to introduce short cycle non degree courses to safe-guard their image in the academic world as maintaining a vaguely defined 'world standard'.

Given the poor quality of most of the secondary schools which provide the input to the universities, the poor quality of equipment, facilities and libraries and the rigidity, trying to adopt a world standard, in m a n y developing countries, is indeed unrealistic. As long as a system sets for itself training goals which are relevant to its needs and to its potential over a limited planning period and achieves these goals, the system can be considered as excellent whether or not it conforms to 'world standards'. Once the type of graduates needed is decided, the system should insist on getting the highest standard for that type. ' A first rate technician m a y be m u c h more useful than a third rate engineer'7.

Insistence on high standards, readjustment of job nomenclature, overhaul of salary scales for middle-level managers and technicians and the provision of a ladder in the higher education system allowing able workers access to a higher education and career m a y help set up a concept of excellence in higher education more fitted to m a n y developing countries. The sharp edge of the choice between 'world standards' and 'local standards' m a y be blunted by the establishment of a few centres of excellence to provide the stimulus and seed bed for other institutions, a critical mass of high quality products of world standard. T h e international requirement of excellence cannot, however, be compromised in the case of some specific professions such as airline pilots or maintenance engineers.

7. C E . Beeby, op. cit. 1969, p. 63.

6


The next dimension of the concept of excellence cuts across the discussions above: this relates to the three categories of excellence: breadth, pyramid and success8. Breadth refers to the choice of different fields of study. Pyramid is concerned with proportions in the higher education system at different ages and with the different fields of studies at each age, and success refers to the extent to which the system actually achieves with the students what it set out to do. So far as breadth is a concept of excellence, relative weights given to different fields of study will depend upon the market needs for skills related to the field. This then becomes part of the concept of extrinsic excellence. It is, of course, observed that institutions of higher education today are better off by providing as m u c h general education as possible within each field of study leaving the specialisation to be taken care of on the job. This is because of quick obsolescence of skills in these days of knowledge explosion. There is also a case for formal specialised training. A s the relative numbers demanding higher education grow and as the financial conditions of the institutions deteriorate, principally because of the reduction in government subsidies, institutions of higher education take on more and more specialized training, mainly of the short cycle type, on a cost recovery basis (e.g. University of Salford, United Kingdom and University of Lagos, Nigeria).

The pyramid as a concept of excellence relates to the relative proportions of the relevant age group in higher education. In an industrialized country, there will be more students at the top of the pyramid than in a developing country because of the need for high level skills in the economy. This concept is again related to the concept of extrinsic excellence. F r o m the point of job opportunities, the angle of the pyramid will depend on the stage of economic development. A subsistence economy has less use and absorptive capacity for university products than a commercial economy, except for doctors. But the capacity increases as the subsistence sector diminishes. Thus attempts are m a d e , as far as possible, to relate the shape of the pyramid to the absorptive capacity of the market. In a developing country, where financial resources are becoming inceasingly scarce, m o n e y saved from a rational allocation at the higher level of education, m a y be used for the bottom of the pyramid i.e. at the primary and secondary levels so that the intake to the higher education level m a y be better in the long run along with other benefits which m a y be derived from reallocation, including the democratic ideal of excellence discussed above.

In a system of higher education, the post secondary vocational sub-system needs special attention in developing countries because of the shortage of middle-level technicians and managers. It is, however, observed that isolating this sub-system from the users i.e. the enterprises and the employers has done a lot of harm. Co-operation with employers, a better incentive system for the graduates based on market forces of demand and supply and maintaining the prestige of the institutions by bringing them within the university system m a y m a k e the pyramid more rationally shaped and contribute to higher excellence in this subsystem. The university sub-system m a y operate at four different levels given the country: the junior/community colleges, the post secondary technical institutions mentioned above, the undergraduate level in arts and science, and the highly specialized and professional level. The shape of the pyramid for these different levels is difficult to decide upon and countries have to tackle the problem differently. Although market factors should not be decisive on the shape, given the social pressure against them, some effort should be m a d e to maintain a balance. The developing countries confront pressure also from the lower levels to include all of them into the university because of the availability of managerial skills, prestige and

8. A . Lewis. Economic aspects of quality in education in C E . Beeby (ed), op. cit.

7


because it m a y be cheaper to centralise the administration. This however creates some difficulty for the university because of concerns about lowering standards, the enormity of administration and, increased cost of teaching. Controlling the shape of the pyramid is a difficult task. It is, however, necessary that professional students be in relatively smaller proportions with respect to their subordinates in developing countries than in the industrialised countries as has been noted earlier. The third aspect of any higher education system's excellence is success i.e. h o w effectively it achieves its goals. Given the content, the system's excellence depends on (1) sharing of a c o m m o n culture between h o m e and school (2) care in the choice of students (3) quantity of physical, h u m a n and financial resources available per student and (4) h o w efficiently these resources are utilized.

Finally as A . Lewis states:

"There is a pyramid of quality just as of quantity. A n y system has a few first class institutions and m a n y of lower quality. The question whether to increase the number or improve the quality of the lower institutions can be answered by the market, since one can calculate whether the earnings of graduates from different institutions correspond to differences in the cost of teaching. However , the market's answer is not decisive9".

The last consideration for excellence in m y presentation is related to cost. If the excellence of an institution affects that of another, then the excellence must be within range of others. In most developing countries especially in some Asian and African countries, where universities had been modelled on the basis of those in the United K i n g d o m , France, or the Netherlands in the n a m e of excellence, they have been found to be misfits for social needs. Again the higher education system mostly funded from government subsidies has to ensure proper utilisation of resources. Given the same level of excellence and all other characteristics more or less similar, the one functioning at a lower cost will have a higher rating in excellence.

T o add to the above concepts of excellence, w e m a y derive, based on Bergquist and Armstrong10, the following criteria for an excellent higher education system:

Attractive: It does something that brings people to it and appeals to students, parents, sponsoring community, potential funding sources and to the people w h o run the system.

Beneficial: It does something that is helpful to the individuals and the community involved in it including the preservation of the environment.

Congruent: It does what it says it will do. Distinctive: It is responsive to the unique characteristics of its people and thus is

unlike other systems. Effective: It achieves what it set out to achieve and demonstrates its

achievements.

9. A . Lewis, op. cit.

10. W . H . Bergquist; J.L. Armstrong. 'Planning effectively for educational quality', Jossey-Bass Publishers, 1986.

8


Functional: It provides learners with attributes needed to perform successfully in to-day's society.

Growth producing: It enhances growth in cognitive, affective, ethical, moral, social, physical and interpersonal dimensions of the learner's development.

International: It plays an important role in developing mutual trust and respect between nations and establishes mutual exchange in academic programmes among countries.

After this attempt at a description of the concept of excellence in higher education, w e m a y m o v e to the existing state of excellence in higher education in the international context.

3. State of excellence in higher education: s o m e examples in the international context

Higher education systems all over the world have undergone massive expansion. A system which was designed in the early years for the elite of society has n o w been opened to the masses (see Table 2 for the expansion of enrolment and teachers during 1975-89). It has created problems in maintaining excellence both in the developed and developing countries of the world. W e shall first consider the case of the developed countries. With the expansion in enrolment, the quality of the intake declined due to relaxation of entry requirements with regard to academic competence.

Table 2. Index numbers of total enrolment and teaching staff in higher education

World total Developed countries Developing countries Africa (excluding Arab states) Asia (excluding Arab states) Arab states Northern America Oceania Latin America and the Caribbean

1975

100 100 100

100

100 100 100

100

Enrolment

1980

119 109 140

197

134 163 108

117

1985

135 115 178

315

159 221 112

137

1989

155 128 210

400

189 262 126

166

1975

100 100 100

100

100 100 100

100

Teaching

1980

119 111 136

193

130 184 104

110

; staff

1985

136 116 176

287

161 245 104

117

1989

154 126 209

343

190 295 110

135

Source: U N E S C O . Statistical Yearbook, 1991, U N E S C O , Paris.

The expansion in enrolment was not matched with corresponding expansion in the number of qualified teachers, classrooms, laboratories, equipment and other physical facilities, and in some cases finance. This resulted in overcrowded classrooms, reduction in staff-student contacts, deteriorated working conditions of the students and teachers (including job insecurity). This coupled with the lower quality student intake, has affected the intrinsic excellence of the higher education system.

Universities in some developed countries have remained alienated from the users of the products of the higher education system e.g. the employers in the public and the private

9


sector. The teaching staff, busy with the expanded instructional responsibility has less time for external contacts in the area of training or research. This has resulted in lowering of extrinsic excellence of the higher education system. The situation has been described in perhaps an exaggerated w a y by a Norwegian bureaucrat as follows:

"There exists very little real knowledge about the usefulness of the training provided today, and even less about what qualifications will be needed in a future characterized by rapid changes in products, technology and forms of organisation"11.

T h e situation has led a French researcher to m a k e the following statement: "... B y a historical deviation too complex to be explained here, a dual system particular to France came into being: the 'Grandes écoles' (...) the quality', Universities: the quantity (over-crowded first cycles, badly managed, very often disconnected from research, which give a false impression of French universities in which training as from the second cycle is of a high quality and research excellence (Mr. Schwartz's report published in 1989 which created a scandal revealing that half of French universities had renounced all research, and were aiming particularly at the first cycle)."12.

In Japan, the Planning Director of the Ministry of Education, Science and Culture laments about the country's higher education system: various problems arose such as concentration of universities and junior colleges in large cities, imbalance in the structure of course specialisation, and unsatisfactory conditions of education and research at private higher education institutions because of the rapid quantitative expansion of higher education in the country13.

Similar questions about the excellence of higher education have been raised in the United K i n g d o m as well as in the U S A .

T h e situation in the former Eastern European countries has attracted m u c h attention these days. The criteria of excellence in higher education in these countries are being redefined because of the invalidity of the old criteria and the revelation that higher education under the communist regimes had neither instrinsic excellence nor extrinsic quality (Jabíes 3 and 4 demonstrate the situation for the former Soviet Union and Poland). The systems are undergoing thorough reform in almost all these countries especially in the late seventies with emphasis on the extrinsic concept of excellence.

11. Quoted from D . Kaville, 'Quality in university educational research' in Evaluation and Quality 1991/4, C R E , Geneva.

12. Translated from M . Cusin : Perception et promotion de la qualité dans les universités françaises. Evaluation et qualité 1991/4 C R E Geneva.

13. U N E S C O . The role of higher education in society: quality and pertinence, N e w Papers on Higher Education, N o . 1, 1991.

10


Table 3. Degree of statisfaction of engineers with their job in the former U S S R (%)

Degree of satisfaction

Very satisfied

Satisfied

Indifferent and no opinion

Dissatisfied

Total

Overall (all industries)

21.2

41.6

9.9

27.3

100.0

Construction manufacturing

18.7

39.0

9.7

32.6

100.0

Cheminai industries

18.7

40.0

9.2

32.0

100.0

Textile and others

18.8

43.6

9.6

27.6

100.0

Research institutes

19.2

42.8

10.2

24.0

100.0

Source: B . C . Sanyal, op. cit. 1987, p . 165.

Table 4. Assessment of training related to type of employing organization in Poland

Type of organization Adequate to job Knowledge acquired Required performed insufficiently used supplementing

Production 18.2 59.7 23.6

Applied research 24.0 38.0 38.0

Applied education and science 30.4 15.5 53.7

Education 37.4 38.3 24.7

Provincial and central administration 26.1 47.6 28.7 Others 23.1 44.0 30.5

Note: Difference from 100 per cent - question not answered Source: B . C . Sanyal, op. cit. 1987, p. 166.

This should not however lead us to think that all the systems of higher education in the developed world are experiencing difficulties as regards excellence. O n the contrary, the increased sensitivity of these systems towards the societal perceptions of excellence in recent years has m a d e them adopt n e w modes of functioning, n e w content, structure and methods of instructions, n e w roles in research and in public service. W e shall discuss the process of this change later.

In the developing countries the situation is worse because expansion has been more rapid and with less preparation. However the situation varies from continent to continent.

11


In Asia and the Pacific there is a complaint about the transplantation of western concepts and values in higher education systems neglecting the eastern philosophy and culture thus affecting the instrinsic excellence of the system. L o w quality of intake, teaching staff and physical facilities, and lack of quality control has significantly affected the standard of 'excellence' of higher education. Lack of correspondence of expansion in enrolment with economic needs for skills has resulted in severe unemployment a m o n g certain types of graduates and scarcity of certain other types of graduates (namely, managers and technologists). The increased role of the private sector in higher education has also affected excellence in standards because expansion in this sector has taken in areas where cost of instruction is low and where market d e m a n d for specialists is less (arts and social sciences). Maintenance of excellence and its improvement has been difficult in the Asian and Pacific continent also because of inadequate allocation of funds.

In Africa, the same problems of Asia and Pacific exist but with greater intensity: intrinsic excellence has deteriorated not only because of over-expansion in enrolment as compared to the teaching force, physical facilities and finance but also because of the drastic reduction in the purchasing power of teachers, their salaries in real terms having been severely reduced. Moreover, the financial crisis left libraries without books, laboratories with unrepaired equipment, classrooms delapidated and overcrowded. All these affected excellence in higher education in a devastating w a y . Universities had no research funds nor the teachers any incentive to carry on research. S o m e countries attempted to change their curriculum to suit national aspirations and needs (e.g. Tanzania) but the economic crisis and subsesquent adoption of structural adjustment programmes resulted in serious financial crisis for the countries' university systems affecting their excellence. T h e higher education systems of the majority of the countries produce too m a n y graduates of dubious quality and relevance, and generate too little knowledge and direct development support. The quality of the graduates has declined so m u c h that the effectiveness of the system is in doubt. In these countries, higher education is extremely costly and the pattern of financing is socially inequitable and economically insufficient.14 Thus excellence is affected in all its dimensions. In South Africa, the higher education system has been discriminatory in the extreme, having neglected most of its people. Although some of the universities in the past had been intrinsically excellent, their isolation from the rest of the world had been responsible for deterioration of their standards also.

In Latin America and the Caribbean, the situation is slightly better than in Africa. However , problems of irrelevance of the content of instruction and/or imbalance in distribution of graduates vis-à-vis labour market needs have contributed largely to a low standard of excellence. Lack of public resources coupled with expansion of enrolments in higher education has caused deterioration in the quality of the public universities15. According to one diagnosis the training provided by the universities is to a high degree out of step as regards the complex reality of the countries in the area and has m a d e little contribution to the search for valid and lasting solutions to tackle the serious problems

14. World Bank. Education in sub-saharan Africa : Policies for adjustment, revitalisation and expansion, Washington D . C . , 1988.

15. World Bank. Seminar on efficiency and quality of higher education in Latin America, Final Report, Brasilia, November 27-29, 1988.

12


affecting these societies: poor cultural integration, growing economic deterioration, increasing technological dependency, permanent social and political conflicts, to mention only a few of them16.

In the Arab states also the enrolment explosion has affected quality in higher education. The content, structures and method of instruction have not evolved with the changing needs of Arab Societies. Enrolment distribution is unbalanced - humanities and social sciences having the major share leading to unemployment of graduates and making education irrelevant for a large number of the products. Brain drain of academics from some countries has also contributed to the deterioration of quality17.

Given the state of 'excellence' of the systems of higher education discussed in the preceding paragraphs, stakeholders in higher education, are becoming more vocal and active in the evaluation of systems. The objectives of evaluation, its history, the approach, the actors, the mechanisms and the prevalent state-of-the-art are the subjects of analysis in the following paragraphs.

4 . Evaluation in higher education

Evaluations are frequently called for in situations of crisis, of serious dysfunction, of extreme dissatisfaction on the part of the users, but also as the result of the clear conclusions that the system is expensive and is becoming even more so and that governments are being incessantly requested to increase budget allocations for higher education18. The above statement about the functioning of the systems of higher education around the world sets out the basic reasons for introduction of more rigorous evaluation. This is gaining wide acceptance not only in the developed countries but also as w e shall see later in the developing countries.

4.1 The objectives of evaluation

Evaluation is undertaken to meet one or more of the following objectives:

(1) for internal use to improve upon excellence; (2) to face competition and attract clientele; (3) to delineate a relevant profile for institutions to achieve better results, and (4) to provide the basis for granting funds - governmental or otherwise.

16. C R E S A L C . "Reforms and innovations in higher education in Latin America and the Caribbean" in Higher Education - Bulletin of the Regional Centre for Higher Education in Latin America and the Caribbean, N o . 15, 1984.

17. Bubtana, Abdalla. Perspectives on trends and issues facing Arab Higher Education within the context of comprehensive development U N E D B A S , Beirut, 1991.

18. A . Staropoli. O ù va l'université : Rapport du Comité national d'évaluation, Paris, Gallimard, 1987.

13


The concept of 'value for m o n e y ' prevalent in the monitoring of the private sector has become a significant factor for the public sector, and higher education system. Evaluation of the performance of the system, it is believed, could produce meaningful and convincing arguments in the debate on value for money 1 9 . The m o v e in the direction of self regulation of the system of higher education is being accompanied by an opposite m o v e towards increasing accountability. The introduction of an evaluation system of the performance of institutions, it is believed, will put into force the necessary process of accountability. Allocation models of public resources, in the past were based on inputs, but are n o w changing to become output oriented, and thus the need for qualitative refinements in the measurement of outputs becomes necessary. This calls for adopting procedures by which the qualitative dimensions of the performance of institution can be identified. Evaluation by means of quality control procedures also becomes necessary as a means of restoring the legitimacy of ministerial bureaucracies as allocators of public resources w h e n questions are raised about wastage in the universities in respect of staff and student time, unrest a m o n g students and complaints of some academics about the neglect of merit in the n a m e of distributive justice.

A s a consequence of shrinking resources, university managers are n o w assuming responsibility for redistributing resources between departments and programmes. Evaluation of performance becomes a necessary management tool for them to take decisions. University managers such as deans or heads of departments also need evaluation or quality control to judge the merits and deficiencies of their colleagues and as a basis for decision-making. In some cases, good evaluation procedures constitute an important component of the internal culture of a system and create a favourable atmosphere for good management. It is also believed that with the gradual disappearance of 'tenure', decisions in respect of personnel m a y n o w be based on well defined and effectively based evaluation procedures resulting in improved staff management which could contribute to increasing standards of excellence in the system.

4.2 History of evaluation of the higher education system

Evaluation of the higher education system has been a unique feature in the U S A leading to the development of a wide variety of procedures, methods and approaches for the measurement, description and appraisal of the accomplishments of faculties, colleges and universities and have led to a large number of studies and analysis of the activities of higher education institutions20.

T h e academic origins of academic personnel of the higher education system available from W h o ' s W h o in America and American M e n of Science provided the basis for ranking institutions of higher education as early as 1925. The same method prevailed until the late fifties. 'Peer review' was initiated by Hayward Keniston in 1959 for evaluating 25 universities in 28 disciplines in respect of their intrinsic excellence. Since then, funding allocation, future careers of academic personnel, distribution of honours and awards are based

19. R . In't Veld. 'Threats and opportunities for evaluation in higher education in dimensions of evaluation in higher education'; U . Dahllöf et al (ed) Jessica Kingsley Publishers, London 1991.

20. K . Hiifner and E . Rau. 'Measuring performance in higher education' - Problems and perspectives in higher education in Europe, Vol. XII, n° 4 , 1987.

14


most frequently on the judgments of academic peers. With time, confidence in peer reviews has diminished and exploration of more objective methods led to the development of performance indicators which are appearing more and more in the evaluation of higher education systems (as w e shall see later, these are not without faults either).

4.3 The approach of evaluation

The evaluation approach varies according to its purpose. Evaluation m a y be undertaken to m a k e judgments about an already installed change in the system, or about a change to be introduced or to ascertain if a change is needed in the system to improve upon its excellence. For each of these cases, the m o d e of evaluation will be different. The nature of evaluation will be different for different objectives e.g. improvements in intrinsic excellence, to achieve the same degree of excellence with better utilisation of resources i.e. less cost, improve democratic ideals of excellence such as equity, and improvement of working conditions of teachers and students. The nature will also vary according to the sponsors of the evaluation: the state, the funding agencies, political leaders, client groups, or the staff w h o are the subject of evaluation.

K o g a n has outlined several important dichotomies in the approaches of evaluation. These are discussed below:21

(1) Scientific experimental (based on the establishment of a control group against which comparative observation can be m a d e over time: applicable to m a k e judgments about an installed change) versus qualitative or non controlled (where no testable hypothesis is rigorously established; evaluation involves collecting as m u c h information as possible to m a k e judgments upon impact perceived and derive patterns and trends of action). In education, especially higher education, it has been argued that the scientific experimental approach is not rigorously applicable because of the complexity of the system where 'subjective experience is both varied and important'.

(2) Formative (where an evaluator collects and reports data and judgments to assist the development of a system by redirecting its course, if necessary based on the earlier steps of evaluation) versus summative involving an independent evaluator giving judgment on a system based on accumulated evidence about h o w it compares with other systems in meeting the objectives.

(3) Evaluation of the product (based on the output and outcome of a system) versus the process (the functioning of the system so that the satisfaction enjoyed, the experiences received, and the style used, become more relevant). W e have noted the same dichotomies when discussing excellence.

(4) Instrumental evaluation (employed by professional evaluators to retain impartiality and universality by using measures for comparison and control), versus interactive evaluation (where the uniqueness and untidiness of the educational context is recognised especially because it deals with h u m a n beings, the process is based on negotiations and dialogue) versus individualistic

21. M . Kogan (ed.) 'Evaluating higher education', Jessica Kingsley Publishers, London, 1989.

15


evaluation (close to a narrative history based on a critical case study). In any single evaluation exercise, several different approaches can be combined.

4.4 Evaluation mechanism

T h e evaluation mechanism is related to the evaluation approach and will depend, more or less on the level of authority in the higher education system. Evaluation initiated by a central ministry m a y be highly authoritative in that it m a y influence flow of resources or the status of the institution or the academic community. In such a situation, evaluation is carried on by 'bodies' which do not directly involve those being evaluated and recently has a tendency to use 'performance indicators'. For example, professional bodies accrediting programmes carry authority. The next in order at the institutional level are non authoritative evaluations which have the power to affect normative judgments m a d e about academic work or institutional performance.

Peer review is of this order. This affects reputations, m a y not lead to but m a y affect authoritative decisions also. At the other extreme of authoritative evaluation at the basic unit level is self-evaluation or evaluation m a d e on a consultancy or colleague basis to enhance self improvement but does not overtly or deliberately, have any authority, but m a y be used for authoritative judgments. The relationship between the level of decision-making and the influence of evaluation is not always related in order.

A combination of several mechanisms is also possible given a particular situation. A higher level of authority m a y sponsor self-evaluation and attempt to act non authoritatively by providing advisory services. Self evaluation by a basic unit m a y lead to authoritative judgments elsewhere. Central authorities m a y continue to encourage formative, interactive, process oriented evaluation as against summative, quantitative and product related evaluation.

Techniques of evaluation will depend on the level at which the evaluation is undertaken, it m a y be the entire national system or an institution as a system. They will also depend on the sponsor of the evaluation and the purpose. U s e of performance indicators, peer review or self evaluation - whatever is the method, data for evaluation will have to cover, for purposes of quantitative analysis (i) input to the system in respect of resources (human, student and staff, physical and financial) (ii) process of functioning of the system (internal efficiency, teaching methods, recruitment procedures, management of the system; (iii) output of the system (graduates by disciplines, research publications, community/public services) and (iv) outcome (contribution to scientific and technological development, employability, etc.). For qualitative analysis, information on the extent that criteria of excellence are met e.g. attractiveness, benefit derived, distinctiveness, effectiveness, functionality, growth producing characteristics and internationality discussed above, is helpful. Such information can be collected by means of sample surveys of students, parents, graduates, employers, academic members and concerned social groups. This leads us to the discussion of the prevalent state-of-the-art of evaluation in the international context A s in the case of excellence, w e shall start with the situation in the developed countries.

16


4.5 The practice of evaluation in the developed countries

In the U S A , several agencies are involved in evaluating the higher education system which have their o w n objectives and standards especially in granting accreditation to institutions. A n institution m a y seek accreditation with one or more agencies relevant to the institution's objectives. S o m e agencies focus on subject matter excellence (intrinsic), others emphasize extrinsic excellence focusing on specific categories of professions. S o m e focus on teaching, others on research. The accrediting agencies use various mechanisms and procedures, as discussed above (performance indicators, self-evaluation, peer review, etc.).

In several O E C D countries, performance indicators are n o w being proposed for use in evaluating the higher education system. These indicators relate to (i) inputs to the system (ii) process of the system (iii) output and outcome of the system and (iv) the general assessment of institutions. A n exhaustive list of the indicators is available. Australia, Austria, Canada, D e n m a r k , Finland, France, Greece, Netherlands, Norway , Sweden and the United Kingdom are using on an experimental basis or have proposed to use performance indicators with various objectives ranging from assessing intrinsic and extrinsic excellence to funding decisions. However their use is subject to m u c h question, especially due to the inadequacy of management information systems to sustain performance indicators. There is some degree of satisfaction with indicators on research performance but a lot of doubt on the use of indicators to measure performance in teaching, public service and use of finance. The most serious concerns are expressed in the use of performance indicators for funding decisions and ranking of institutions because of their inadequacies22.

In addition to performance indicators other methods of evaluation are also being applied in different countries. W e have had the privilege of listening to some of them in detail at this conference. The following is a brief survey of some of the methods applied in selected countries. The Council for National Academic Awards ( C N A A ) established in 1964, had been using external peer review to assess excellence in British institutions of higher education. Recently C N A A and the Committee of Vice-Chancellors and Principals ( C V C P ) set up Academic Audit Units to monitor and comment on the mechanisms by which the institutions themselves assure the quality of the programmes they offer. The assessment is concerned with the capacity of an institution to identify its strengths and weaknesses so as to improve its quality23. Proposals have also been put forward to create three types of evaluation of teaching in all institutions of higher education: quality control (internal monitoring for maintaining and enhancing quality); quality audit (external scrutiny to check on quality control methods of the institutions and quality assessment ('external review of and judgments about the quality of teaching and learning in the institutions').

22. H . . R . Keils (ed.) "The development of performance indicators for higher education : a compendium of eleven countries', O E C D , Paris, 1990.

23. Frans van Vught, 'Higher education quality assessment in Europe: the next step', in Evaluation and Quality, ibid.

17


In France, the 'Comité National d'Evaluation' ( C N E ) attached to the Ministry of Education but reporting directly to the President of the Republic was set up in 1985 with the objectives of 'issuing authoritative value judgements on the quality of research and education and on the w a y the university fulfills its assignment' without however having any regulatory powers, which he with the Minister of Education. The Committee first consults the President of the university to obtain his and his offices' consent to the evaluation. It then appoints two or three experts to examine each component of the university on the spot and to report back to it. The experts must have access to the information possessed by the central government on the university, and the reports and opinions of major research institutions. They are provided with all available documents and information by the university and the heads of its components, which is necessary for a clear understanding of its functioning.

The evaluation covers all aspects of the university's life. The C N E is a permanent body with fifteen members each working half time for it and has its o w n secretariat to perform the tasks of research and processing of information. Over fifty institutions have been evaluated so far.

In the Netherlands, the Association of Co-operating Universities ( A C U P ) puts emphasis on self-evaluation complemented by visits of peers to each programme site of a specific discipline. The self studies have appeared to be important incentives for adaptation of existing programmes. The peer review process takes the form of a comparative analysis although this is not used for ranking the individual programme sites.

In Sweden, the National Board has n o w been changed from a planning body to a monitoring, assessing and supervisory agent with a great deal of independence from the government and the institutions. Its monitoring programme systematically collects information on the system's performance and analyses the changes and tendencies so as to suggest corrective measures. Its evaluation programme carries on in-depth studies of programmes of teaching and research with a view to improvement.

Studies are not of uniform patterns for the whole system but oriented towards 'quality promotion'. The National Board is also 'to take responsibility for national reviews based on self-assessments at local level and carried out by national or international peers.' The Board has to supply the government with an in-depth analysis of the whole education system. The Board is also involved in developing methods of quality control and in providing service to the institutions so as to improve their performance.24

In Spain, the Minister of Education and Science set up an expert committee in October, 1989 to explore ways and means to evaluate the education system, especially the excellence of the academic staff. A set of criteria has been developed for the purpose, covering teaching, research and public service functions.

4.6 The prevailing state of evaluation in developing countries: some examples

Under pressure from the national governments, as well as from international funding agencies, university systems of developing countries are beginning to adopt systematic

24. Frans van Vught, op. ciL

18


evaluation procedures. W e shall discuss here a few cases based on a study sponsored by U N E S C O and conducted by the International Association of Educational Assessment25.

In Colombia, the Instituto Colombiano Para Fomento de la Educación Superior (ICFES) is in charge of the academic evaluation of programmes in higher education institutions, for accreditation and for improvements in quality. The Academic Vice President of I C F E S is in charge of evaluation. His office establishes criteria, sets up each year a programme of evaluations for institutions, and carries them out with an external evaluator and a m e m b e r of his staff. The evaluators prepare a report based on their work on the spot and submit it to the Board of Directors of I C F E S which is chaired by the Minister of Education and consists of representatives of private and public universities and the Director of I C F E S . Before the visit, the evaluators receive exhaustive information about the institution, its objectives, programmes, structure, methods of instruction, resources and detailed managerial process. The programme of evaluation proceeds in five sequential stages:

(1) Stage of document analysis: at this stage results of previous evaluations are reexamined, and any change in objectives, programmes, methods of instruction is analysed. This is done in I C F E S .

(2) Study of current situation: at this stage the evolution of the programme is compared with the current situation and the purpose of evaluation is clarified, providing the necessary input to establish the profile of the external evaluator w h o is selected at this stage and briefed about the objectives and the method of evaluation.

(3) Stage of direct contact: the actual evaluation starts at this stage through collection of information from students, staff and community of the institution as well as from the documents already received.

(4) Stage of evaluation consolidation: using the report of the external evaluator and his o w n perception of the functioning of the programme the co-ordinator of the evaluation (ICFES staff) prepares the final report, which includes recommendations for improving the quality of the programme.

(5) Stage of decision-making: I C F E S , based on the report of both external evaluator and the I C F E S staff, takes the decision on accreditation of a programme - either for continuation or for initiation. The decision m a y also lead to corrective measures of the continuing programme. The report of the evaluation becomes a public document. I C F E S also assists individual institutions of higher education to conduct their o w n evaluation.

Evaluation in Colombia does not influence funding decisions for the institutions.

In Nigeria, a visitation panel consisting of a chairman with judicial background, a university person, an Education Ministry Official, a federal government representative, a legal

25. W . B . Dockrel (Co-ordinator). 'Evaluation procedures used to measure the efficiency of higher education systems and institutions', U N E S C O , Paris, 1990.

19


expert, an accounting expert and a secretary representing the National University Commission, is set up to evaluate the affairs of a university under the authority of the Head of the State in the case of a federal university or the State Governor, in the case of a state university. The panel is assisted by a secretariat where information is collected through all the documents, files, procedures of meetings as well as submissions from the community at large and oral interviews of relevant persons.

The evaluation is based on information related to finance, administration/service, research, teaching, personnel and students. The panel prepares a confidential report on each department/faculty/administrative unit for the visitor (Head of State or Governor). A second version of the report, called a 'blue paper' containing the findings and the recommendations is available for public use. A third version called the 'white paper' contains the final results of the evaluation and the government approved recommendations which become binding on the institution. The evaluation is used for allocation of resources (unlike Colombia), continuation/cancellation/moderation of a programme, discipline and review of activities.

In Nigeria, in addition to the external evaluation, universities also conduct internal evaluation of their academic programme, financial programmes, and personnel.

In Malaysia, evaluation of the higher education system does not receive as m u c h attention as in Colombia or Nigeria. However at the middle and the end of the fifth national development plan period (1986-1990), evaluation of the system took place to check on the programmes/courses and the extent to which the objectives set in the plan had been achieved. The evaluation takes place in the form of an external evaluators' report or follow up, or as reactions to press comments or public views.

In Jordan, the law of higher education was established in 1985 empowering the Ministry of Higher Education to carry out evaluation of institutions of higher education through its two offices: the office of the development of higher education and that of community colleges. Each of these offices has a 'studies and evaluation section' which evaluates programmes for accreditation and monitor programmes for quality assurance. At the institutional level, the method adapted is self-evaluation where department heads submit an annual summative evaluation report covering all aspects of the work (teaching, research and public services) to the Dean w h o in his turn follows the same exercise for the President of the University.

5 . Prob lems of evaluation procedures

The discussion above demonstrates that the adoption of evaluation procedures is being an increasingly c o m m o n phenomenon all over the world. The traditional, informal evaluation practices namely, self-evaluation and peer review focussing on intrinsic excellence are giving w a y to more comprehensive and formal evaluation focussing on extrinsic excellence. Universities have to be accountable. But one wonders if focus on extrinsic excellence does not diminish the importance of the internal quality in higher education without which 'external quality' cannot be sustained. The second problem is that higher education is a complex phenomenon making its evaluation very difficult. It concerns all segments of society, 'involves the best h u m a n mind has to offer as well as the most delicate, organisational

20


structures that have evolved over centuries'.26 The more complex a phenomenon the more its evaluation depends upon judgment and less on objective criteria and standards because of absence of a clear and widely agreed upon criteria and because, as w e have discussed in detail in the first half of the paper, quality is a multidimensional concept. The methods adopted have, therefore, a lot of deficiencies. The performance indicators discussed cannot capture all aspects of the input, process, output and outcome of higher education. The further w e m o v e from input towards outcome, the more difficult it becomes to measure, however careful and competent the evaluators are. Self-evaluation, on the other hand, also has its deficiencies (i) Its scope in assessing the extrinsic dimension of excellence is limited (ii) objectivity m a y not be maintained (iii) most important, the evaluation m a y not give credibility to external clients and the criteria of evaluation m a y not be based upon consensus of the academic community.

Peer review has, therefore, been more acceptable and has been described as providing a 'reliable index of quality in a world where infallibility is unattainable'. It is also not free from criticism. The primary criticism m a d e against it is social bias. It has been observed that peers not only judge the content of a process or a product but 'social inputs like the reputation of an institution or the personal characteristics of an academic also play an important role'. The second bias labelled 'intellectual bias' has to do with the relationship which sometimes appears to exist between the intellectual organisation of a discipline and judgment of the quality of products from that discipline found in reference processes where specific orientations or methodologies are preferred. A third criticism, that of 'random error' reflects the failure of experts to agree on acceptable levels of quality leading, often for example, to the acceptance of a contribution in a scholarly journal or award of a research grant depending on chance factors27. Evaluation of higher education runs the risk of 'reducing the capacity of the less successful rather than improving them' through its use in resource flows yielding exactly the opposite of what it is intended for.

6. I m p a c t of the evolution of the concept of excellence o n evaluation procedure a n d vice versa

W e have noted above that as the objectives of higher education evolved over time, the concept of excellence also evolved because of the definition of the term excellence. In the early days w h e n the objective of higher education was only to achieve mental and spiritual development and general intellectual capacity of individuals, the definition of excellence was limited to the state of developing qualities in an eminent degree to provide these characteristics, i.e. limited to the intrinsic dimension only. Evaluation procedures could be restricted to the analysis of academic excellence: to what extent higher education could provide a m o n g citizens moral and cultural values and subject matter competence for mental and spiritual development as well as the general intellectual competence. At that stage the system could be evaluated often by the self-evaluation and sometimes by peer review procedures. In fact, expertise for evaluation was available only within the system and in the peer group.

26. M . Kogan. 'The case of the United Kingdom in Evaluation and Quality', op. ciL

27. F. van Vught, op. cit.

21


A s the objectives of higher education were extended to cover skill formation for different vocations of life, the concept of excellence had to include in addition to its intrinsic dimension, extrinsic dimensions as well. Evaluation procedures had to involve the community higher education served i.e. enterprises, the parents, the students and the community at large became partners in the evaluation procedure, which m o v e d from self-evaluation to external evaluation with less emphasis on self-evaluation to more on peer review and the user of the products.

Under the pressure of financial restraints, the concept of excellence was extended further. Achievement of societal objectives was no longer enough, these objectives had to be achieved more efficiently in the economic sense. Inputs to the system had to be related to the process, the output and the outcome. Wherever possible objective, quantitative measures must be applied to obtain precision. The role of performance indicators became important, so important in some countries like the United Kingdom that funding of the system is n o w directly geared to the universities' ranking in research evaluation exercises, their capacity to recruit students and the funding councils' judgments on quality. The evaluation procedure widened from self evaluation and peer review to cover performance indicators. Evolution of evaluation procedures had been necessary to control and develop quality. Without them the higher education system might have lost its usefulness to society as m a y be the case in some countries where the system is going through overhaul.

However , the recent emphasis on indicators of performance has some dangers because of their incapability to measure performance adequately as discussed above. Use of these indicators for decision-making in the system m a y reduce quality as Kogan has warned28.

Under the circumstances, use of evaluation m a y be restricted to the purpose of accrediting in which a combination of self-evaluation, peer review and performance indicators will be used. Different agencies with different objectives and standards could grant accreditation to different institutions with specific missions29 and until one is sure of the correctness of evaluation, decision on resource flows should be postponed to avoid possible deterioration in the excellence of the system. Suggestions have been m a d e to improve management of excellence through newly developed procedures called 'Total Quality Management ' at the institutional level which has the possibility to cover the whole system of higher education30.

This emphasizes the process dimention of quality through 'Plan-do-Check-Act' stages with mission and customer forms, systematic approach to operations, vigorous development of resources and long-term thinking.

28. M . Kogan. The case of the United Kingdom in evaluation and quality, op. cit.

29. F . van Vught, op. cit.

30. L . A . Sherr and D J . Teeter (eds). Total quality management in higher education. N e w directions for institutional research, N o . 71, San Francisco, Jossey-Bass Inc., 1991.

22


7. Conclusion

Excellence in higher education is a multi-dimensional concept and the concept has evolved over time with the evolution of the aspirations and expectations of the society it serves. The rapid expansion of higher education without corresponding provision of infrastructure facilities and resources has affected excellence of the system in both developed and developing countries, although the nature and extent has varied in different systems. Evaluation procedures have helped to assess the level of excellence and maintain it in some systems. In others these procedures are being developed and adopted. With the evolution of the concept of excellence, the evaluation procedures have also evolved. But these procedures are not without faults and the results should be used with care. There is a shift to introduction of indicators of performance, but because of certain inadequacies the importance of self-evaluation and peer review should be maintained. A system of multiple accreditation with a number of agencies granting accreditation in the light of their specific interest, based on a combination of procedures could help assess, maintain and develop excellence in higher education systems in the international context as recommended by van Vught for Europe. The use of Total Quality Management ( T Q M ) as a technique might also be explored in different contexts for the same purpose. Given the theme of the Conference w e have emphasized higher education systems in this document. The same concepts and procedures will also apply at the institutional and programme level.

23

П Е Р publications and documents

M o r e than 650 titles on all aspects of educational planning have been published by the International Institute for Educational Planning. A comprehensive catalogue, giving details of their availability, includes research reports, case studies, seminar documents, training materials, occasional papers and reference books in the following subject categories:

Economics of education, costs and financing.

Manpower and employment.

Demographic studies.

The location of schools (school map) and sub-national planning.

Administration and management.

Curriculum development and evaluation.

Educational technology.

Primary, secondary and higher education.

Vocational and technical education.

Non-formal, out-of-school, adult and rural education.

Copies of the catalogue m a y be obtained from the П Е Р Publications Unit on request.

T h e International Institute for Educational Planning

The International Institute for Educational Planning (ПЕР) is an international centre for advanced training and research in the field of educational planning. It was established by U N E S C O in 1963 and is financed by U N E S C O and by voluntary contributions from M e m b e r States. In recent years the following M e m b e r States have provided voluntary contributions to the Institute: Belgium, Canada, Denmark, Finland, Iceland, India, Ireland, Norway, Sweden, Switzerland and Venezuela.

The Institute's aim is to contribute to the development of education throughout the world, by expanding both knowledge and the supply of competent professionals in the field of educational planning. In this endeavour the Institute co-operates with interested training and research organizations in M e m b e r States. The Governing Board of the H E P , which approves the Institute's programme and budget, consists of eight elected members and four members designated by the United Nations Organization and certain of its specialized agencies and institutes.

Chairman: Victor L. Urquidi, (Mexico) Research Professor Emeritus, El Colegio de México, Mexico.

Designated Members: Arturo Nunez del Prado, Director, Latin American and the Caribbean Institute for Economic

and Social Planning, Santiago. Cristian Ossa, Director, Development Policy and Analysis Division, Department of Economic

and Social Development, United Nations. Visvanathan Rajagopalan, Vice-President, Sector Policy and Research, Policy, Research and

External Affairs, The World Bank. Allan F. Salt, Director, Training Department, International Labour Office.

Elected Members: Isao Amagi (Japan), Special Advisor to the Minister of Education, Science and Culture,

Ministry of Education, Science and Culture, Tokyo. Henri Bartoli (France), Professor, Séminaire d'Economie du Travail, Centre Pierre Mendès-

France, Paris. Mohamed Dowidar (Egypt), Professor and President of the Department of Economics, L a w

Faculty, University of Alexandria, Alexandria. Kabiru Kinyanjui (Kenya), Senior Programme Officer, Social Sciences Division, International

Development Research Centre, Nairobi. Yolanda M. Rojas (Costa Rica), Academic Vice-Rector, Faculty of Education, University of

Costa Rica, San José. Lennart Wohlgemuth, (Sweden), Assistant Director-General, Swedish International

Development Authority, Stockholm.

Inquiries about the Institute should be addressed to: The Office of the Director, International Institute for Educational Planning, 7-9 rue Eugène-Delacroix, 75116 Paris, France.

П Е Р Contributions

A s part of D E P ' s programme of co-operation with other organisations, members of the Institute's staff are frequently invited to present papers at conferences, seminars and workshops organised by these institutions or to submit articles to specialised journals. The series of HEP Contributions, which includes a selection of these papers and articles, is designed to ensure that such writings - often basically of ephemeral nature - are disseminated widely and quickly.

IEEP Contributions N o . 11

The paper attempts to define excellence in higher education from different points of view and traces its evolution over time in different contexts. It then gives an overview of the prevailing state of excellence in different systems in the world and identifies the reasons for the differences. The role of evaluation is examined in assessing, maintaining and developing excellence of higher education systems in the international context. The different procedures are elaborated and compared. The impact of the evolution of the concept of excellence on evaluation procedures and vise versa is examined, and the role of evaluation in the maintenance and development of excellence in higher education is indicated as a conclusive factor.

T h e author

Dr. Bikas C . Sanyal is a Senior staff member of the International Institute for Educational Planning ( U N E S C O ) , Paris, and at present responsible for the 1 Ш Р research programme on Improving the effectiveness of higher education institutions: studies of the management of change. H e has previously directed the П Е Р research programmes on Higher education and employment, and technological development and educational planning. H e is the author/co-author of a large number of books, monographs and papers on educational policy and planning.

excellence and evaluation in higher education: some...

Documents