evaluation of instruction in higher education: some critical issues

17
141 Higher Education 8 (1979) 141-157 Elsevier Scientific Publishing Company, Amsterdam - Printed in the Netherlands EVALUATION OF INSTRUCTION IN HIGHER EDUCATION: SOME CRITICAL ISSUES ARYE PERLBERG Technion - Israel Institute of Technology, Haifa, Israel ABSTRACT Critical issues arising from the evaluation of instruction in higher education are reviewed. The purposes of evaluation (for reward and accountability, for improvement of instruction) and identity of the evaluations (students, reporters, peers, administrators, consultants) are discussed and a plea made for tying evaluation to systematic training schemes. Introduction Faculty development programs are not a recent development. For years most colleges and universities have had some sort of practices designed to aid the professional development of their staff such as sabbatical leaves, time release for the preparation of new courses, financial help to attend profes- sional meetings, etc. However, the main contribution of these activities is directed to the professional development of the faculty in the specialization of the subject matter, rather than to the development of instructional skills. According to Centra (1976) "it has been mainly in the 1970s that fac- ulty development has expanded to include a variety of practices and special programs", which have attempted to "help faculty members grow in teaching effectiveness by sharpening their teaching skills and knowledge." In support- ing this observation, Centra (1976) cites, among others, the writings of Alexander and Yelon (1972), Freedman (1973), Group for Human Develop- ment in Higher Education (1974), Erickson (1975), Bergquist and Phillips (1975), Gaff (1975) and O'Banion (1976). The above mentioned literature refers mainly to the situation in the U.S.A., but there are ample testimonies on the recent expansion of instruc- tional development agencies and faculty development programs the world over. Examples of these can be found in the proceedings of two international conferences on Improving University Teaching sponsored by the University of Maryland, University College, in cooperation with Heidelberg University (1975, 1976), in the agenda of the present international conference sponsored

Upload: arye-perlberg

Post on 06-Jul-2016

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Evaluation of instruction in higher education: Some critical issues

141

Higher Education 8 (1979) 141-157 �9 Elsevier Scientific Publishing Company, Amsterdam - Printed in the Netherlands

EVALUATION OF INSTRUCTION IN HIGHER EDUCATION: SOME CRITICAL ISSUES

ARYE PERLBERG

Technion - Israel Institute o f Technology, Haifa, Israel

ABSTRACT

Critical issues arising from the evaluation of instruction in higher education are reviewed. The purposes of evaluation (for reward and accountability, for improvement of instruction) and identity of the evaluations (students, reporters, peers, administrators, consultants) are discussed and a plea made for tying evaluation to systematic training schemes.

Introduction

Faculty development programs are not a recent development. For years most colleges and universities have had some sort of practices designed to aid the professional development of their staff such as sabbatical leaves, time release for the preparation of new courses, financial help to attend profes- sional meetings, etc. However, the main contribution of these activities is directed to the professional development of the faculty in the specialization of the subject matter, rather than to the development of instructional skills.

According to Centra (1976) "it has been mainly in the 1970s that fac- ulty development has expanded to include a variety of practices and special programs", which have attempted to "help faculty members grow in teaching effectiveness by sharpening their teaching skills and knowledge." In support- ing this observation, Centra (1976) cites, among others, the writings of Alexander and Yelon (1972), Freedman (1973), Group for Human Develop- ment in Higher Education (1974), Erickson (1975), Bergquist and Phillips (1975), Gaff (1975) and O'Banion (1976).

The above mentioned literature refers mainly to the situation in the U.S.A., but there are ample testimonies on the recent expansion of instruc- tional development agencies and faculty development programs the world over. Examples of these can be found in the proceedings of two international conferences on Improving University Teaching sponsored by the University of Maryland, University College, in cooperation with Heidelberg University (1975, 1976), in the agenda of the present international conference sponsored

Page 2: Evaluation of instruction in higher education: Some critical issues

142

by the University of Maryland in cooperation with the Newcastle-upon-Tyne Polytechnic, an international conference sponsored by UNESCO and IAU at the University of Amsterdam in 1970 (IAU, 1971), a similar conference, also sponsored by UNESCO together with the University of Massachusetts (1974) and the papers of the working party of the 3rd International Con- ference on Higher Education at the University of Lancaster in 1975.

From a survey of the literature cited above, it appears that programs referring to evaluation, or assessment of instruction or instructors, evaluation or assessment of teaching, and evaluation or assessment of teacher effective- ness or competence are very popular and not new. According to Aleamoni (1974) "many of the problems and questions concerning student evaluation of instruction can be found in a plethora of research that spans at least 50 years." However, many of these "problems and questions" have not yet received satisfactory answers. Moreover, the recent upsurge of programs intensified the severity of the problems and calls for special attention and effort to find some answers to them.

Within the limits of this paper it is not intended to review, touch upon or even list all the perplexing problems pertaining to the subject. We have chosen rather to discuss and focus upon a few of the critical issues in the area which, in our opinion, do not receive enough attention in the literature.

In the above discussion we mentioned several terms or names used syn- onymously in the literature to denominate the area under discussion. These terms and names have, however, different meanings because each of them represents a different philosphy and concept of evaluation. The respective differences in meaning and the concepts they represent will be discussed later on. For the purpose of this discussion we have chosen the term "eval- uation of instruction in higher education '~ which will be used throughout this paper.

Purposes of Evaluation of Instruction

EVALUATION OF INSTRUCTION FOR REWARD AND ACCOUNTABILITY

In his discussion on the purposes of evaluation of instruction Dressel (1960) states that

The first, and perhaps the most commonly recognized, purpose of evaluation is that of supplying recognition of and reward for good instruction. For reasons which have already been developed, some of the more satisfactory means of evaluation of instruction are usually ignored, and frequently the whole matter of evaluation is ignored until the situation arises in which an individual is to be considered for a promotion, for permanent tenure, or for release. At this point, department heads and other administrative officers display concern as to whether the individual is a good or a poor teacher.

Page 3: Evaluation of instruction in higher education: Some critical issues

143

�9 . . all too frequently the a t tempt to evaluate teaching in connection with promot ion and financial increments ends up by being nothing more than an at tempt to find out whether an individual's teaching is so bad that promotion or salary increase must be refused. Accordingly, there is generally no confidence in faculty members that really outstanding teaching will receive adequate recognition. Indeed, there is more than a suspicion that any review of teaching competencies will be sole- ly to determine those who are inadequate in this regard. A continuing program of evaluation not always obviously related to tangible reward, will help to mitigate this suspicion. To be effective such a program must be initiated and maintained by the faculty. Much of the failure of administrative officials to give appropriate atten- t ion to good teaching arises out of the failure of the faculty to provide evidence on this po in t

W h i l e Dres se l s p e a k s a b o u t e v a l u a t i o n f o r r e w a r d p u r p o s e s , a n e w p u r p o s e

e n t i t l e d a c c o u n t a b i l i t y was a d v a n c e d in r e c e n t y e a r s . In his d i s c u s s i o n o f t h e

r e a s o n s fo r t h e r e c e n t u p s u r g e in f a c u l t y d e v e l o p m e n t p r o g r a m s , in w h i c h

e v a l u a t i o n o f i n s t r u c t i o n is an i m p o r t a n t e l e m e n t , C e n t r a ( 1 9 7 6 ) s t a t e s t h a t

Another reason for the recent emphasis on faculty development and instructional improvement is the general disenchantment - expressed by students, parents, and legislators - with the quality of college instruction�9 Students seem less timid about expressing their dissatisfaction than they once were, and many parents are not at all sure that instruction is as effective as the high costs of a college education suggest it should be. Legislators have pressured public institutions to become more accountable and in some states have earmarked funds specifically for instructional improvement. At the national level, a 1972 report submitted to the President and Congress by the National Advisory Council on Education Professions Development singled out the need for more effective training of community college teachers�9

In a n o t h e r r e c e n t d i s c u s s i o n o n t h e p r o b l e m R o t e m a n d G l a s m a n ( 1 9 7 7 )

s t a t e t h a t

� 9 a predominant feature of the context of evaluation of instructors in American universities and colleges is that it includes a dual set of foci, the improvement focus and the accountabili ty focus�9 In the institutional subcontext within which the pro- fessor works, a reasonable balance between the two foci has been maintained, largely because the accountabili ty focus has not by itself created a significant discrepancy between the professor's scholarly autonomy and the trend for evaluating instructors for improvement�9 The balance in the societal context in which the university as a whole operates has recently become a difficult objective to attain, because the accountabili ty focus has intensified the above discrepancy.

I t is i n t e r e s t i n g to n o t e t h a t R o t e m a n d G l a s m a n s p e a k o f e v a l u a t i o n o f

i n s t r u c t o r s r a t h e r t h a n i n s t r u c t i o n , t h i s b e i n g a lso t h e t i t l e o f t h e i r p a p e r .

T h e i r c h o i c e is n o t a c c i d e n t a l a n d r e p r e s e n t s t h e a t t i t u d e s o f t h e a c c o u n t a b i l i t y m o v e m e n t .

In a n o t h e r p a p e r G l a s m a n ( 1 9 7 6 ) r e i t e r a t e s t h e a c c o u n t a b i l i t y p o i n t

Page 4: Evaluation of instruction in higher education: Some critical issues

144

o f view. R o t e m and Glasman admit tha t the balance be tween the reward pur- pose and the i m p r o v e m e n t purpose was upset . In an a t t e m p t to regain a balance be tween the two purposes these au thors suggest tha t

�9 . . the university administration should intensify its own demands for evaluation for instructional improvement, and be able to advance and defend the argument to the society that increased instructional improvement serves the accountability objective in the most effective manner�9 To the extent that the administrator can utilize societal demands for accountability as stimuli for heightening the priority of teaching and its improvement within the university, he will satisfy societal demands and simultaneously facilitate the achievement of professional standards of faculty.

La te r on in this discussion, we shall e labora te on the ques t ion w h e t h e r a func t iona l balance be tween these two purposes m ay be achieved at all and i f so, what are the ways to achieve this.

EVALUATION OF INSTRUCTION FOR IMPROVEMENT OF INSTRUCTION

The second purpose o f evaluat ion is i m p r o v e m e n t o f ins t ruc t ion , evalua- t ion as an e lement in the process o f training, change and deve lopment . Dressel (1960) views this t ype o f evaluat ion as an ac t ion research t y p e o f p rogram in which findings and exper iences , at any stage o f the evaluat ion, can bring abou t changes in the a t t i tudes and v iewpoints o f b o t h teacher and s tudents . Dressel goes on to s tate tha t

If an environment can be cultivated in which teachers feel free to try out new ideas without fear that failure will jeopardize their future, and with assurance that the extra time spent will be appreciated by colleagues and superiors, it can be expected that much more in the way of actual improvement of instruction will take place than when evaluation is administratively initiated, with the major purpose being that of making a decision about the continuation or promotion of the individual.

Dressel poin ts ou t tha t the mos t c o m m o n l y recognized purpose o f eval- ua t ion is tha t o f supplying recogni t ion , reward and accountabi l i ty . However , his pre judice is tha t evaluat ion o f ins t ruc t ion should be pr imari ly o r ien ted to i m p r o v e m e n t and deve lopment .

As was quo t ed above, even p r o p o n e n t s o f the accoun tab i l i ty purpose such as R o t e m and Glasman (1977) , agree tha t the pr ime purpose o f the accoun tab i l i t y object ive is the i m p r o v e m e n t o f ins t ruc t ion and that the admin is t ra t ion should util ize societal demands for accountab i l i ty as st imuli fo r he ightening the pr ior i ty o f teaching and its i m p r o v e m e n t wi thin the univers i ty , thus satisfying these demands while facil i tating the i m p r o v e m e n t o f ins t ruct ion .

Page 5: Evaluation of instruction in higher education: Some critical issues

145

Even Dressel, who was quoted as preferring the improvement purpose, recognizes this duality and states that " improvement and reward are really but opposite faces of the same coin, neither can proceed without the other". He then ends his discussion on this duality by stating

To develop the reputation of being a good teacher, to be so regarded on a campus by the students, and to have one's course sought for on these grounds, is a satisfying experience, but it is not enough if the monetary rewards and promotions go to others who are indifferent teachers. We must recognize and reward good teaching if we are to maintain and increase interest in it in our colleges and universities.

Who are the Evaluators of Instruction?

Until now we have focused on the purposes of evaluation so as to be able to decide whether we can achieve a balance between the purposes mentioned, or if we must give preference to one or another of them. In order to solve this problem we must take into account the various sources o f evaluation.

STUDENTS AS EVALUATORS OR REPORTERS OF INSTRUCTION

Menges (1973) quotes Steele, House and Kerins (1971) in their observa- tion that

The teacher and his students are the only direct sources of information on what occurs in the college classroom. Uncounted hours of exposure to teaching do not make students perfectly reliable reporters, but students are privy to events witnessed by no one else except the teacher, and teachers have been shown to be less accurate reporters than their students. [and then Menges adds]

If classroom events are one's focus for the evaluation of teaching, observations by stu- dents seem essential.

Menges agrees that students should be considered reporters rather than evaluators. He considers them interested observers rather than judges

Students can describe classroom events without having to judge their value. That is, they can report the amount of work done for a course without determining whether the amount was appropriate. They can describe the instructor's use of illustrations from other fields without judging the instructor's breadth of knowledge. They can report their perceptions of classroom atmosphere ("I felt free to express my opions even when they disagreed with those of the instructor") without having to assess the instructor's personality.

Dressel (1960) also states that extensive research on student ratings of instruction has demonstrated that, in most cases, these ratings are fairly con-

Page 6: Evaluation of instruction in higher education: Some critical issues

146

sistent, regardless o f grade or class level. However , he is aware o f the fact that , despi te the evidence o f the rel iabil i ty and the val idi ty o f s tuden t ratings, ins t ruc tors are general ly hes i tant in accept ing, and even resentful , when these ratings are forced u p o n them. Dressel t he re fo re suggests tha t

It may be more politic to ask the student to rate the course rather than the instruc- tor. In rating the instructor, the student is asked to act as a judge of the adequacies of the instructor, perhaps inclusive of personal characteristics, scholarship and his classroom performance�9 This approach may imply to the student that he is the ulti- mate authority as to the adequacy of teaching.

� 9 It may, therefore, be more appropriate to ask him to engage in a broader type of evaluation in reference to the course or in reference to the total learning experi- ence rather than solely to the instruction. An approach to student appraisal which asks the student to indicate what he feels are the objectives of the course, what relationship he found between the activities in which he engaged and these objec- tives, what he did in the way of studying for the course, and what changes in views, attitudes, or behavior generally have taken place as a result of the course causes him to reflect on his own behavior as well as on that of the instructor. Into the student responses to such questions the instructor can readily read something of his own adequacy but with somewhat less defensive reactions than when he feels criticisms are directed entirely at personal shortcomings.

PEERS AND ADMINISTRATORS AS EVALUATORS

Observat ion by peers or by adminis t ra t ive superiors, especially fo r the purpose o f recogni t ion , reward , t enure and accoun tab i l i ty have been regular ly prac t iced in some universi t ies and colleges. Many professors believe that in t rus ion in to the i r c lassroom by others , excep t when invited, violates the i r academic f reedom. However , very o f t en , the concep t o f academic f r eed o m becomes a license which permi ts the p rofessor to teach as badly as he wishes. Dressel (1960) suggests, tha t in spite o f the prevailing resistance, this pract ice should be established, especially wi th all new appointees . In these days, when pressure fo r accountab i l i ty , lesser mobi l i ty and shortage o f funds prevai l , it would be easier to spread the pract ice o f evaluat ion by adminis t ra tors , and make it no t on ly a "necessary evil" fo r the purpose o f reward and tenure , bu t a posit ive channel o f f eedback for improvemen t .

EVALUATION BY AN EDUCATIONAL CONSULTANT

With the es tab l i shment o f agencies for ins t ruct ional and facul ty develop- men t , there has been a growing t e n d e n c y among professors to invite educa- t ional consul tan ts f rom these agencies to visit the i r regular classrooms and provide diagnost ic f eedback for the purpose o f i m p r o v e m e n t and develop- ment . In some instances, discussions be tween the consu l tan t and the pro-

Page 7: Evaluation of instruction in higher education: Some critical issues

147

fessor are based both on this visit and on the feedback from the students. The advent o f portable video equipment has enabled the taping of regu-

lar classroom sessions and of lessons given in a teaching laboratory, which provide a basis for discussion. The purpose of all these practices is improve- ment of teaching through feedback and training (Perlberg, 1970, 1972a, 1972b, 1974, 1975, 1976; Melnick, 1972; Bergquist and Phillips, 1975).

Evaluation of Instruction for Improvement - Our Point of View

From the above, we conclude that evaluation of instruction by students, when sponsored by the university or college administration through one of its agencies, (such as an evaluation and measurement office, or an instructional development agency) should be voluntary and geared mainly to the purpose o f training development and improvement. It should be the professor's prerogative to submit the results of this evaluation to the administration for purposes of reward, promotion, tenure and accountability.

This point of view raises several questions to which I will refer in the following discussion. First, there is growing societal pressure for greater accountability. As a result, there is a growing tendency of administrators to view results of evaluation as one o f the factors to consider for reward, promo- tion and tenure. In view of these recent developments, how does one cope with professors who resist evaluation? How does one motivate those of the faculty who already have tenure?

It seems that the most important step that must be taken in this area, is to convince faculty that the administration really means it when it states that teaching is just as important as research. In their decisions about reward, promotion and tenure, administration must show real interest in reliable evidence about teaching effectiveness. Many faculty members feel that even though there is a growing recognition o f the importance of teaching and eval- uation o f instruction, in reality, it is a consideration only in a few extreme cases. Moreover, many faculty members do not trust the administration in dealing with this matter and think that it only pays lip service to public demands (Popham, 1974). In spite of the growing movement of evaluation of instruction, teaching is not yet honored on the campus the same way as research. Wilson's (1967) observation on this matter, although made a decade ago, still prevails

If teaching is honored on our campuses, it will be cultivated there and will finally be done well there. If it does not find honor, expressed in the respect and prestige granted the teacher by his colleagues and by the dollars paid him by the comp- troller, it is not likely to be cultivated nor to improve.

Page 8: Evaluation of instruction in higher education: Some critical issues

148

It would seem that if teaching is to be honored on the campus as being equal to research, then the faculty will have intrinsic motivation to devote the necessary time to improve their teaching. They will strive to fulfill and actualize (Maslow, 1954) themselves not only through research,, but also through teaching. At the universities teaching is a significant part of their work and in other institutions their main task. This will lead to group pres- sure to improve teaching. Societal pressure from the public at large and par- ticularly pressure of students, will be most effective. Younger faculty mem- bers who are still being considered for tenure and promotion will certainly be affected. It is to be hoped that full-tenured professors will also feel the need to do a better job of teaching which, after all, is an essential part of their lives.

Up until now, we have referred to student evaluation of instruction sponsored by the administration through an instructional development agency and designed mainly for the purpose of improvement. However, this is not the only type of student evaluation. In some campuses, student organ- izations, with the aid of qualified experts, have initiated their own evaluation of instruction and instructors. They functioned like a consumer organization to provide its members with information about the quality of the "products" available, in order to help students choose courses more wisely (Menges, 1973). This evaluation does not require the consent of the faculty member being evaluated and it is "forced" upon him by the consumer. As a result, the administration is able to receive information about him without taking the initiative. In some cases, when the administration of some universities and colleges started to use this information for reward, promotion and tenure, the faculty, particularly those who distrusted or did not like the results of the student-initiated evaluation, demanded formal institutional evaluation. The students were thus instrumental in indirectly pressuring the administra- tion to engage in evaluation programs of its staff.

In both of the examples described above, the students were the eval- uators. But, as already stated, there is need to expand sources of feedback and evaluation. Peers and superiors could also provide feedback in the case of faculty members who are considered for reward, promotion and tenure, who do not consent to student evaluation. Whenever there is such consent the evaluation of both students, and peers and superiors, should be used for improvement and reward.

Another problem arises when a faculty member consents to evaluation and the feedback obtained is not positive. His dilemma is whether to volun- tarily submit such incriminating feedback to the administration. In case of poor teaching, when even the educational consultant has despaired of improvement, it may be necessary to withhold reward, promotion and tenure. On the other hand, there are cases of innovative and creative teachers who do not always receive good ratings. As stated by Menges (1973)

Page 9: Evaluation of instruction in higher education: Some critical issues

149

I believe that the consequences of a responsible program of student evaluation of teaching are overwhelmingly positive. Of the possible negative consequences, I am most concerned about a leveling effect on creative teaching. If evaluations are built from a narrow empiricist tradition, they are likely to be severely inhibiting. The most "useful" empirical results are sometimes the most dehumanizing. Imposing uniform rating procedures on a diverse faculty can only raise the cry, "All teachers are exceptional. By virtue of being human, they are bound to be highly individual" (Kossoff, 1971).

Some of the most innovative teachers, probably the source of the most significant educational reforms, are highly individualistic teachers. Characteristics of these "new teachers" include styles and techniques which do not fit most rating forms. "His teaching style is a life style, and life styles are highly individualistic and depend on make-up and background and often vary from situation to situation . . . The New Teacher is learning, often the hard way, that not all students still see a need to intellectualize everything". (Flournoy, 1972). Teachers who are growing and experi- menting may not fare well under conventional student ratings. That possibility does not make them less accountable for providing some evidence of their impact as teachers, but it does suggest that institutions must take special care to ensure that a variety of evidence is acceptable.

O n e p r o m i s i n g s o l u t i o n t o this p r o b l e m is to a d o p t eva lua t i on p roce -

dures t a i lo red , as m u c h as poss ib le , t o the i n s t r u c t o r s ' goals . " T y p i c a l t each-

ing c i r c u m s t a n c e s , i nnova t ive t e c h n i q u e s , o r s t u d e n t s w i t h u n u s u a l charac-

ter is t ics m a y requi re a specia l ly des igned i n s t r u m e n t " (Menges , 1973) . T h e

I n s t r u c t i o n a l D e v e l o p m e n t and Ef fec t iveness Asse s smen t S y s t e m ( I D E A )

d e v e l o p e d at Kansas S ta te Un ive r s i ty b y H o y t and his co l leagues ( H o y t ,

1969 , 1973a , 1 9 7 3 b ; H o y t and Cashin , 1 9 7 7 ; Biles, 1976) , is one g o o d

e x a m p l e o f such so lu t ion .

T h e C e n t e r f o r F a c u l t y E v a l u a t i o n and D e v e l o p m e n t in Higher E d u c a -

t i o n at Kansas S ta te Un ive r s i ty descr ibes the I D E A c o n c e p t as fo l lows :

The IDEA system is based on two propositions: (1) effective teaching is best recognized by its impact on students; and (2) the amount of impact on students is a function of the instructor's classroom behavior.

The IDEA system defines instructional effectiveness as reports by students of their progress on those teaching objectives which the faculty member specified as important for that particular course.

If these two propositions are true, it should be possible to use a single ap- proach to evaluate instruction in courses which differ widely in objectives, levels, and methods; this is because the measure of effectiveness is progress on relevant objectives (i.e., objectives which are selected by the instructor). It should also be possible to determine whether or not student gains on a given objective are more likely to occur if the instructor employs certain teaching techniques.

Several assumptions were made in the process of developing the IDEA system:

1 A list of course objectives could be developed which would emcompass the objectives of nearly all instructors and courses.

Page 10: Evaluation of instruction in higher education: Some critical issues

150

2 A list of instructional behaviors could be developed which would encompass most of the important teaching options available to instructors of conventional classes (i.e., those courses which employ lecture, discussion, exams, etc.).

3 Students could reliably and validly rate their own progress on specific course objectives.

4 Students could reliably describe the classroom behavior of the instructor. 5 Relationships between the classroom behavior of instructors and student progress

would be contingent on the size of the class and on the instructional objectives.

In addition to substantiating these assumptions (see Hoyt and Cashin, 1977), research on the system has shown that:

1 There is no single model of effective instruction contrary to the assumption made in many student rating programs. The IDEA system provides for a multi- tude of different models which reflect substantial differences in emphasis (objec- tives).

2 It is essential that adjustments be made for differences in the motivation level of students as well as for differences in class size to accurately infer instructional effectiveness from student ratings.

3 It is possible for effective instruction to occur even though the instructor em- ploys techniques and procedures which are unsuccessful for the average faculty member. Therefore, effectiveness should be judged less by how the instructor behaves than by how students are affected.

This is one detailed example. There are several others, such as one developed by Starks et al. (1973).

Evaluation of Instruction and Training for Improvement

In reviewing the literature on the effect of student feedback on teaching, Menges (1973) states that

Unfortunately, evidence is mixed about the effects on teaching of providing student rating information to teachers. The balance of studies reviewed by Trent and Cohen (1973) favors change in teaching as a result of feedback from students. By contrast, in a carefully controlled study with teaching assistants, Miller (1971) found very little impact on teaching from feedback. In a larger study involving five colleges, the Student Instructional Report was administered at mid-term with some teachers subsequently receiving feedback. When the form was readministered at the end of the semester, no differences were found in changed ratings for those who received feedback compared to those who had not. Further analysis did reveal an effect of feedback for those teachers whose self-ratings were higher than their student ratings. "The greater the discrepancy - where the discrepancy reflected the extent to which students rated teachers less favorably than the teacher apparently expected - the greater the likelihood of change" (Centra, 1972). Thus feedback alone without additional training may produce change for cer- tain teachers under certain conditions. Knowledge of student ratings, especially if

Page 11: Evaluation of instruction in higher education: Some critical issues

151

the items are behaviorally stated, may improve the teacher's discrimination of "appropriate" and "inappropriate" behaviors. Given adequate motivation, such improved discrimination may be sufficient for change (Wagner, 1973). The likeli- hood of change is increased, of course, if the institution provides resources and training programs for its faculty.

Costin et al. (1971), in their review of the literature, emphasize the fact that there is not enough evidence that student rating improves teaching. They could not cite reliable studies proving that changes had occurred in faculty who had received feedback, saw discrepancies, experienced dissonance and developed needs. On the contrary, Costin et al. cite evidence that teachers who were rated "good" and teachers who were rated "bad" at the beginning of the year, were rated the same way by the same students at the end of the year, i.e. the status quo was apparently maintained.

Aleamoni (1974) also states that there is no clear-cut evidence to show that feedback does have an effect on instruction. He cites, however, his own recent study (1974) that student rating had effects on teaching as follows

The results revealed that there was a significant increase in the student ratings of these faculty on the two lowest rated CEQ subscales that were discussed in the meetings with the measurement and evaluation expert. On the other hand, the group of faculty who was not able to avail themselves of expert consultation but did receive the CEQ results, remained unchanged in their subscale ratings.

The effects o f feedback are more complex however and cannot be limited to their impact on teaching only. Many writers in the field agree that feedback from students or other sources is a process of self-confrontation. This process produces psychological dissonance (Festinger, 1957), creates discrepancies, tension and even anxiety (Perlberg, 1970, 1972a, 1972b, 1975; Centra, 1976;Glasman, 1976; Rotem and Glasman, 1977). It seems that even some of those who subscribe to the concept o f equating feedback in instruc- tion with self-confrontation tend to see only the positive potential o f the system and underestimate the psychological damage of providing feedback without adequate provisions to help those, who as a result o f it, are in need of help. Such att i tude is unethical, harmful and damages the whole concept (Perlberg, 1975). Smock and Crooks (1973) state that "any evaluation plan which does not include an adjunct service which can effectively assist faculty members in the improvement of their teaching performance is ethically ques- t ionable."

Centra (1976), in his recent survey found that systematic ratings by students are widely used, but perceived as only moderately effective. Respon- dents in his survey rated formal or informal assessment by colleagues as less effective than either consulting with faculty who had expertise or working with master teachers. The analysis o f in-class video taping was considered as

Page 12: Evaluation of instruction in higher education: Some critical issues

152

one of the more effective practices, although it was frequently used by only a small proportion of the faculty on campuses where it was available. Another practice rated effective, but rarely used, was the professional and personal development plan for individual faculty members.

It seems logical to conclude that evaluation of instruction must be tied to systematic training schemes which will bring about improvement. Is this the prevailing situation? Unfortunately not. In most cases, faculty develop- ment centers or agencies are small and have very limited funds. In many cases, they are staffed with on ly one coordinator. Actually, the initiation and administration of evaluation of instruction schemes are relatively easy and could be implemented by a technician under the guidance of an expert. This could easily be a small part of the coordinator's job. In many instances, facul- ty development agencies adopted a ready-made evaluation system along with its computer program and diagnostic printout, processing it on their own computer. In other cases, they found it more economical to distribute, col- lect the questionnaires and then send them for processing to a central agency. Such a service is offered by the Kansas State University Center for Faculty Evaluation and Development (1977). According to the unpublished status report, it will have serviced about a hundred institutions during 1976-1977 and processed 350,000 response cards representing an estimated 18,421 classes. These represent an estimate of 11,605 faculty members who have used the IDEA system in one or more of their classes. These numbers are certainly very impressive and could be processed very economically within the available technology. Even though the system provides a relativels~ ' elaborate diagnostic report on each participant, it must still be interpreted with expert help. Moreover, even expert interpretation of the feedback does not assure impro- vement. There is a need to chart a personal development plan which should include training activities in a teaching laboratory, course design sessions with an expert, and evaluation techniques, among others. Such a program cannot be implemented by understaffed and underbudgeted instructional develop- ment centers. It ~vould seem, though, that even those institutions which view faculty evaluation as a vehicle for development and improvement, are not equipped to perform these services. Unfortunately, the impressive statistics of many centers in the area of evaluation can only be regarded as satisfying their own ego and societal pressures for accountability. All this is a far dis- tance from change. It seems that the administrators do not yet accept the fact that growth in the number of faculty being evaluated must bring about growth in the number of consultants, experts, training and development activities. Centra's (1976) findings that consulting with faculty who had expertise, or working with master teachers was more effective than formal or informal assessment by colleagues, emphasizes the need to expand the number of personnel in faculty development centers.

Page 13: Evaluation of instruction in higher education: Some critical issues

153

The Assessment of Instructors' Competence

According to McNeil and Popham (1973), support for the position that the ultimate criterion of a teacher's competence is his impact upon the learner has been offered by a number of individual researchers as well as by profes- sional associations, (e.g. American Educational Research Association AERA, 1952; Astin and Lee, 1966; Biddle and Ellena, 1964; Cohen and Brawer, 1969).

This concept stems from the commonly held definition that:

a teacher is a person engaged in interactive behavior with one or more students for the purpose of effecting a change in these students. The change, whether it is to be attitudinal, cognitive or motor, is intentional on the part of the teacher (McNeil and Popham, 1973).

McNeil and Popham admit, however, that there are reservations in accept- ing pupil change as the chief criterion of teacher effectiveness both due to technical problems in assessing learner growth and to philosophical considera- tions.

Within the limits o f this paper, it is impossible to review the voluminous amount of recent literature which is very critical of present instructional research efforts concentrating on the personal attributes and instructional techniques of the teacher, rather than on pupil change, as the chief criterion of teacher effectiveness. We shall, therefore, limit ourselves to quoting from the concluding remarks by McNeil and Popham (1973):

An evaluation of a teacher is not equivalent to determining the teacher's instruc- tional competency, i.e. the ability to effect desired changes in learners . . . .

Effectiveness in teaching is best evidenced by criterion measures which detect pupil growth as a result of the teacher's instruction . . . .

Measures of long- and short-range instructional objectives for a variety of outcomes are very much needed if two key purposes of assessment are to be fulfilled: instruc- tional accountability and improvement.

Examination of the multi tude of rating and evaluation instruments will reveal that most of them do not address themselves to the problems raised above. The systems which come closest in dealing with the problems of teacher impact on the learner are those which look at the goals of the instruc- tor and check with the students' subjective test imony to what extent these goals were achieved, as for example the IDEA system. In the attitudinal domain this evidence may be sufficient, but in the cognitive and motoric domains, there is very little evidence to support this claim. Until such evi- dence is forthcoming, the proof for teacher effectiveness is remote indeed.

Page 14: Evaluation of instruction in higher education: Some critical issues

154

Concluding Remarks

The purpose of this paper was to point to some critical issues in the area of evaluation of instruction. The recent impressive growth of instructional development agencies and of evaluation of instruction may be deceiving and pacifying to those involved directly in the programs, to administrators and to the public at large. Popham, in his critique of "Higher Education's Com- mitment to Instructional Improvement Programs" (1974) sees them as "meek medicine for a major affliction" and states:

I seriously doubt whether many of the tokenistic and half-hearted efforts currently underway will lead to any genuine improvements in the instructional capabilities of many faculty members. In numerous institutions of higher learning the commitment to improve faculty instructional capabilities is largely political, stemming more from public relations motives than from a genuine desire to bring about increased faculty effectiveness.

He goes on to stress that instructional development agencies must have more power before their efforts toward instructional change will have any impetus. University officials, administrators and faculty must have greater commitment to the improvement of instruction, giving the agency enough personnel and financial resources to evaluate faculty teaching effectiveness and try to improve it.

While Popham addresses himself to the whole field of instructional development, the danger is even greater in the area of evaluation of instruc- tion, because it is in this area that we might fall into the trap of being happy with the large numbers of faculty members that are being evaluated by stu- dents without realizing that this is only the first step in a faculty develop- ment program of a long and tedious process towards improvement. It is our duty, as those concerned with the improvement of teaching in higher educa- tion to point out the complexity of these problems and try and solve them.

Acknowledgement

Invited paper given at the Third International Conference "Improving University Teaching", June 8 -11 , 1977, Newcastle-upon-Tyne, England. Sponsored by the University of Maryland University College in cooperation with Newcastle-upon-Tyne Polytechnic (England).

References

Aleamoni, L. M. (1974). "Typical faculty concerns about student evaluation of instruc- t ion," invited address at the Symposium on Methods of Improving University Teaching at the Technion, Israel Insti tute of Technology, Haifa, Israel.

Page 15: Evaluation of instruction in higher education: Some critical issues

155

Alexander, A. T. and Yelon, S. L. (1972). Instructional Development Agencies in Higher Education. East Lansing, Michigan: Continuing Education Service.

American Educational Research Association, Committee on the Criteria of Teacher Effec- tiveness. (1952). "Report of the committee on the criteria of teacher effectiveness," Review of Educational Research, 22: 238-263.

Astin, A. W. and Lee, C. B. T. (1966). "Current practices in the evaluation and training of college teachers," The Educational Record, 47: 361-375.

Bergquist, W. H. and Phillips, S. R. (1975). "Components of an effective faculty develop- ment program," Journal of Higher Education, 46:177-211 (Also in A Handbook for Faculty Development, Washington, D.C.: The Council for the Advancement of Small Colleges, 1975).

Biddle, B. J. and Ellena, W. J. (eds.) (1964). Contemporary Research on Teacher Effec- tiveness. New York: Holt, Rinehart and Winston.

Biles, B. R. (1976). "Strategies for implementing instructional improvement." Invited address for the Second International Conference on Improving University Teaching, Heidelberg.

Centra, J. A. (1972). Strategies for Improving College Teaching. Washington, D.C.: Amer- ican Association for Higher Education.

Centra, J. A. (1976). Faculty Development Practices in U.S. Colleges and Universities. (PR-76-30), Princeton, N.J.: Educational Testing Service.

Cohen, A. M. and Brawer, F. B. (1969). "Measuring faculty performance," ERIC Clearing- house for Junior College Information. Washington, D.C.: American Association of Junior Colleges.

Costin, F. et al. (1971). "Student ratings of college teaching: reliability, validity and use- fullness," Review of Educational Research, 41, 5.

Dressel, P. L. (1960). "Evaluation of instruction," in Proceedings of Summer Institute on Effective Teaching for Young Engineering Teachers, at the Pennsylvania State University, Lancaster.

Erickson, B. L. (1975). "A survey of the inservice training needs and interest of instruc- tional improvement centers," unpublished doctoral dissertation, University of Massa- chusetts.

Festinger, L. (1957). A Theory of Cognitive Dissonance. Stanford, Ca.: Stanford Univer- sity Press.

Flournoy, D. M. et al. (1972). The New Teachers. San Francisco: Jossey-Bass. Freedman, M. (1973). Facilitating Faculty Development. San Francisco: Jossey-Bass. Gaff, J. G. (1975). Toward Faculty Renewal. San Francisco: Jossey-Bass. Glasman, N. S. (1976). "Evaluation of instructors in higher education, an administrative

function," Journal of Higher Education, 47: 309-326. Group for Human Development in Higher Education (1974). Faculty Development in a

Time of Retrenchment. Change publication. Hoyt, D. P. (1969). Instructional Effectiveness: L Measurement of Effectiveness. Man-

hattan, Kansas: Research Report Number 6, Office of Educational Research, Kansas State University.

Hoyt, D. P. (1973a). "Identifying effective instructional procedures," Improving College and University Teaching, 21: 73-76.

Hoyt, D. P. (1973b). "The Kansas State University program for assessing and improving instructional effectiveness," in A. L. Sockloff (ed.), Proceedings: Faculty Effectiveness as Evaluated by Students. Philadelphia: Measurement and Research Center, Temple University.

Page 16: Evaluation of instruction in higher education: Some critical issues

156

Hoyt, D. P. and Cashin, W. E. (1977). IDEA Technical Report No. 1. Published by the Center for Faculty Evaluation and Development in Higher Education, Kansas State University, Manhattan.

IAU International Association of Universities (1971). Quarterly Bulletin No. 1. IDEA (1975). Information pamphlet, published by the Center for Faculty Evaluation

and Development in Higher Education, Kansas State University, Manhattan. Kosoff, E. (1971-72). "Evaluating college professors by 'scientific' methods," American

Scholar, 41: 79-93. Maslow, A. H. (1954). Motivation and Personality. New York: Harper and Brothers. McNeil, J. P. and Popham, W. J. (1973). "The assessment of teacher competence," in

Travers, R. M. W. (ed.) pp. 218-244 in Second Handbook of Research on Teaching. Chicago: Rand McNally.

Melnik, M. A. (1972). "The development and analysis of a clinic to improve university teaching," submitted to the Graduate School of the University of Massachusetts.

Menges, R. J. (1973). Evaluating Learning and Teaching. San Francisco: Jossey-Bass. Miller, M. T. (1971). "Instructor attitudes toward, and their use of, student ratings of

teachers," Journal of Educational Psychology, 62: 235-239. O'Bannion, T. (1976). "Staff development: a new priority for the seventies," The College

Board Review, 99. Perlberg, A. and O'Bryant, D. C. (1970). "Videotaping and microteaching techniques to

improve engineering instruction," Journal of Engineering Education, 60:741-744. Perlberg, A. et al. (1972a). "Microteaching and videotaping recordings: a new approach

to improving teaching," Journal o f Medical Education, 47. Perlberg, A. (1972b). "Microteaching - part 5, application of research results," Inter-

national Review of Educatio n, 18. Perlberg, A. et al. (1974). "Modification of teaching behavior through the combined use

of microteaching techniques with the Technion Diagnostic System TDS," Journal o f Instructional Science, 3:177-200 .

Perlberg, A. (1975). "Recent approaches on microteaching and allied techniques which can be implemented easily in developing countries," invited paper for UNESCO, Paris.

Perlberg, A. (1976). "The use of laboratory systems in improving university teaching," Higher Education, 5: 135-152.

Popham, W. J. (1974). "Higher education's commitment to instructional improvement programs," Educational Researcher, 3.

Proceedings of First International Conference on Improving University Teaching, Heidel- berg, 1975.

Proceedings of Second International Conference on Improving University Teaching, Heidelberg, 1976.

Rotem, A. and Glasman, N. S. (1977). "Evaluation of university instructors in the U.S.: the Context," Higher Education, 6: 75-92.

Smock, R. and Crooks, T. (1973). "A plan for the comprehensive evaluation of college teaching," Journal of Higher Education, 44: 577-586.

Starks, D. D., Davis, W. K. and Schmalgemeier, W. L. (1973). "The development and use of a goal oriented course evaluation instrument," Paper read at the Annual Meeting of the American Educational Research Association.

Steele, J. M., House, E. R. and Kerins, T. (1971). "An instrument for assessing instruc- tional climate through low-inference student judgements," American Educational Research Journal, 8 : 447-466.

Trent, J. W. and Cohen, A. M. (1973). "Research on Teaching in Higher Education," in R. M. W. Travers (ed.) pp. 997-1071 Second Handbook of Research on Teaching, Chicago: Rand McNally.

Page 17: Evaluation of instruction in higher education: Some critical issues

157

University of Lancaster, (1975). Working party contributed papers of Third International Conference on Higher Education at the University of Lancaster, on "Excellence or equality: a dilemma for higher education," Lancaster, England.

University of Massachusetts at Amherst (Clinic to Improving University Teaching) in cooperation with UNESCO, (I 974). Proceedings of the International Conference on "Improving University Teaching".

Wagner, A. C. (1973). "Changing teacher behavior: a comparison of microteaching and cognitive discrimination training," Journal ofEducationalPsychology, 64: 299-305.

Wilson, M. O. (1967). "Teach me and I will hold my tongue," p. 9 in Lee, C. B. T., Improv- ing College Teaching, Washington, D.C.: American Council on Education.