abstracts of selected management training civil service - eric
TRANSCRIPT
ED 159 186
AUTHOR,TITLE;
INSTITUTION
REPORT NOPUB DATENOTEAVAILABLE FROM
EDRS PRICEDESCRIPTORS
DOCUMENT RESUME
TM 007 372
Gast, IleneAbstracts of Selected Management TrainingEvaluations.Civil Service Commission, Washington, D.C. TrainingLeadership Div.TLP-03-02Mar 7739p.Superintendent of Documents, U.S. Government PrintingOffice, Washington D.C. 20402 (Stock Number733-404/418, price not available)
MF-$0.83 HC-$2.06 Plus Postage.Administrative Personnel; *Annotate& Bibliographies;Course Evaluation; *Educational Programs; *EvaluationMethod's; Glossaries; Leadership Training; ManagementDevelopment; *Management Education; Post SecondaryEducation; *Program Evaluation; Research Design;
IA Supervisory Training
ABSTRACTIntended for evaluators--whether trainers,
psychologists, management consultants or professors--thisbibliography samples findings in management training evaluationbetween 1953 and 1975. It contains 28 abstracts,of representativearticles from journals in applied psychology and personnelmanagement. Each abstract is a one-half to one-page description ofthe training program itself, the evaluation 'design and procedures,and the evaluation findings and implications. Evaluations of trainingvary by organizational setting, managerial function, topics covered,and instructional strategies. Design charlIcteristics of eachevaluation study--namely, type of control, testing schedule, and datacategories--are presented in a table to allow the reader to choose astudy within his or her own scope of interest. ,1ossary ofevaluation terms is appended. (CP)
************************************************************************ Reproductions supplied by EDRS are the best that can be made *
* from the original document. *'***********************************************************************
AbsTRAcTsof
SELECTEd
MANAgEMENT
TRAiNiNqEVALUIONS
Training Leadership Division
SCOPE OF INTEREST NOTICEThe ERIC Facility has assignedthis document for processingto CLIn our judgement, this documentis also of interest to the clearing-houses noted to the right. Index-ing should reflect their specialpoints of view.
U.S. DEPARTMENT OF HEALTH.EDUCATION S WELFARENATIONAL INSTITUTE OF
EDUCATION -
THIS DOCUMENT HAS BEEN REPRO-DLISED EXACTLY AS RECEIVED FROMTAE PERSON OR ORGANIZATION ORIGIN-AT ING IT POINTS OF VIEW OR OPINIONSSTATED. DO NOT NECESSARILY REPRE-SENT OFFICIAL NATIONAL INSTITUTE OF,EDUCATION POSITION OR POLICY
TLP-03-02 (12/77)
AbSTRACTS
ofSE1ECTEd
ANAqEMENT
TRAINiNq
EVALUATiONS
Prepared by
ILENE GASTTraining Leadership Division
U. S. Chia IERViCE COMMISMON
blUREAU Of TRAiNiNq
MARCh 1977
TABLE OF CONTENTS
Introduction
Characteristics of Evaluation Design ii
Characteristics of 'Evaluation Design Table iv
Abstracts I
Glossary
Author Index
Selected References 29
Bihtiogphy 30
INTRODUCTION
Evaluatorstrainers, psychologists, management consultants and professorshave been systematicallyevaluating management training programs for over twenty years. And, also for over twenty years, eval-uators have been publishing 'articles about their findings. Consequently, a wealth of experience hasaccumulated from which anyone interested in evaluation can profit.
You may be interested in the evaluation of management training because you are a trainer and want toevaluate courses yourself. You may need to know more about what constitutes good evaluation design.Perhaps you are a manager who must make decisions that are based on evaluations done by others.You may need help in judging evaluations. Whatever the reason for your interest, you may not havethe time to read all or even a large share of the articles that two decades of steady work has produced.You need a way of judiciously sampling the field.
We have provided that way by abstracting twenty-eight representative articles. In making our choices,we looked at evaluations of training that varied according to the organizational setting in which thetraining was given, according to the managerial functions that were being taught, according to the top-ics that were covered and according to the instructional strategies that were employed.
The table on page iv shows what kind of design was used by each evaluator whose work we have ab-stracted. Using the table will make it easier for you to find the kind of evaluation that you are interest-ed in. It will also make it easier for you to compare the design of one evaluation with that of another.
This publication includes a list of selected references and,,a glossar!, of evaluation terms. Each termthat is defined in the glossary is printed in boldface the first time that it appears in each abstract. Theonly exception to this rule is the term "evaluation," which is not in boldface because it appears as aheading in each abstract.
CHARACTERISTICS OF EVALUATION DESIGN
The table that follows on page iv summarizes each of the evaluations that we have abstracted. It
shows the characteristics of the design that each evaluator used to gather the information and the types
of information that (s)he gathered. (The evaluations are listed chronologically so as to yield a historical
perspective and to reveal any trends that may have developed in the last twenty years).
The design characteristics of an evaluation dictate the strategy that the evaluator uses to ,collect data.
Did (s)he use a control group ,as well as an experimental group? Were the participants picked at random
or by matching? Were they given a pretest, a posttest or both? The table answers each of these ques-
tions for each of the evaluations.An experimental group is the collection of subjects who receive the training. A control group is a
collection of individuals who are similar to the subjects in all relevant ways except one: they do not
receive the training. An evaluator can use a control group in determining whether the training has af-
fected the experimental group. (S)he tests both groups and compares the results. Presumably any
change shown by the experimental group that is not shown by the control group is the result of the
training.The members of the control group should be as much like the members of the experimental group as is
possible. Otherwise it would not be reasonable to determine from a comparison of the two groups what .
the members of the experimental group have gained from the training. For example, suppose that the
members of a control group had each worked with the Federal Personnel Manual for ten years. Sup-
pose further that the members of the experimental group had never seen the Manual and that their
training consists of a one-week introduction to it. If both groups are tested when the experimental
group has completed its training, the experimental group will probably show less knowledge of the
Manual than the control group will. Clearly, however, it would be a mistake to deduce that the experi-
mental group has not profited from the training.
To ensure that the members of the control group are similar to the members of the experimental group,
the members of both groups are usually picked either by matching or by random selection.
To match is to deliberately pick peoplOor the control group whose- backgrounds are the same as those
of the members of the experimental group. Background, for this purpose, can be defined as those fac-
tors that might influence the experiment: age, status, education, etc. To pick by random selection is to
choose participants in such a way that all people who are eligible to be participants have an equal
chance of being included. Furthermore, participants must be chOsen in a way that ensures that the
selection of one participant has no influence on the selection'of any other.
If a control group is not used, it is particularly important that the members of the-eXperimental group
be picked at random. To understand why, consider how valid an evaluation would be if it were an
evaluation of a course that was designed to teach conference leadership and if the course were given
to an experimental group composed of participants who had each successfully led several conferences.
As for the characteristic that has to do with pretests and posttests, if an evaluator has given both to
the subjects, (s)he has given the subjects the same test twice; once before they start the training, and
once after they complete the training. By comparing the results of the pretest to those of the posttest,
the evaluator is able to determine whether the subjects have changed during the time between the
tests. Changes are presumably the_result of the training.
The presence or absence of all of these characteristics influence the accuracy of the evaluation find-
ings. After all, if an evaluation is not\ well designed, it is possible to ascribe effects to training that are
actually caused by other factori. Unfortunately, designing and using an evaluation strategy that incor-
porates all of these characteristics involves time, money, and planning.- Therefore, the designs of many
of- the evaluations that we have abstracted do not include all of these elements.
U
6
The table on page iv also shows what kinds of information were collected in oath evaluation The datais categorized on the table according .to the system suggested by Donald Kirkpat rick;I that is, underthe following headings: reaction, learning, behavior, and results. Reaction and learning measuredwhile the participants are being trained. Reaction refers to whether the participants like the training,including the materials, the instructors, the facilities, the methodology, the media and the content.Learning refers to the facts, skills, and attitudes that the participants gain from the training Behaviorand results are measured after the trainees return to their jobs. Behavior means the trainee's perform-ance on the job, and results refers to the impact that his or her training has on the organization.
, .
lkirkpatrick, Donald L. (ED.). Evaluating Training Programs. Madison. Wisconsin: American Socie t:f for Training andopment, Inc., 1975.
iii
TABLE A: EVALUATION STUDY DESIGN CHARACTERISTICS
Source/ Year
Maie(1953)Blocker. (1955)Goodacre (1955)Moon & Harriton(1958)Voger;-
Buchanan 8.-"unMetter (1959)--Mahoney et al(1960) '
page
Hillman (1962)Good acre (1963)
Blake et al(1964)
"fie et al(1964).
McClelland (1965)M_ i r (1965)
Blake & Mouton(1968)
Albanese (1967)
Schwarz et al(1%8)
Val ici-uet (1968)
ControlGroup
Yes
No
Yes
Yes
No
Yes
23 Yes
No
Yes
No
Yes
2
Yes
No
Yes
40 Yes
40 Yes
Yes
No
No
. 15 Yes
9 Yes
No
Yes
No
Yes
No
43 Yes
Wilson, et al(1968)
Thorley (1969)Schein (1971)
(19Hanci & Slocum72)
Burgoyne (1973),
(1973)GoIri-stein, et al
Margulies (1973)
Ashton & Gibbon(1974)
Ivancevich (1974)Lei(197d4) ecker & Hall
weXte8) y,& NeMerOff(197
Type of Control roeasu rement
Random
Matchedor
Equiva-lency
No
Yes
Yes No
No No
BeforeTog-
AfterTrng
No No Yes-----__--------------,
No yes T es--1.----' No Yes
No Yes
Data Categories
Short-Term
Reaction
Long Term
Learn-ing
Behav-ior Results
No Yes No
NNo No Yes
No Yes 'les
No
Yes
No
No
No rJo Yes No
No Yes
No Yes
No No
No yes Yes No Yes No
No yes Yes YesYes No
yes Yes No Yes Yes
Yes
No
No
No
No
Yes
Yes
No
Yes
No
NoNc
No
YesYes
Yes
Yes
No
Yes
No Yes No
Yes
NoNo Yes
NoNo Yes
No
Yes
No
No
No
Yes.
Yes Yes
No No Yes
No Yes Yes
No Yes Yes
Yes No Yes
No Yes Yes
No No Yes
No
No
No
No
No
No
No
No
No
No
Yes
No
2-McClelland reports three studies, one. of which has as used
category.control
'Dashes. () indicate that study did not specifically address
. .
a matched grouP
tV
No
No
No
Yes
Albanese, Robert. A case study of executive development. Training and Development Journal, 1967'21(1), 28-34.
Albanese, a member of the Ohio State University faculty, evaluated the long term effects of an ongo-ing e;:ecutive development program run by the National Restaurant Association (NRA). His, evaluationdemonstrates how the use of a control group can increase the accuracy of the evaluation results.
TRAININGThe one-week course, given in seven locationsin the U.S., covered basic management con-
improvement of managerialgerial attitudes andbehavi ,r, 'supervisory functions of the managerand standards for restaurant operation. Lee-tures, informal group discussions, case studies,role playing, question and answer sessions, aswell as consultation with faculty members con-veyed course content. The trainees in the ses-sion which Albanese evaluated, were elevenwomen and sixty-one men who managed restau-rants of various types and sizes.
EVALUATION°Albanese began his evaluation in 1961, threeyears after the inception of the NBA's manage-ment course. He selected a control group fromthe NRA membership catalog. Statistical com-Parison of background factors showed that thegroups did not differ significantly, and couldtherefore be compared on evaluation measures.Prior to training, managers in both the experi-mental and control groups received five sets ofquestionnaires with instructions, one to be filledout by the manager and four for his or her sub-ordinates. Subordinates completed the Leadership
AM.
Description Behavior Questionnaire (LDBQ), ameasure of subordinate perception of leadershipbehavioral style. Managers filled out the Respon-sibility Authority Delegation (RAD) scale whichmeasures these elements for individuals whooccupy managerial positions. Only question-naires returned before the end of the sessionwere i Icluded in the analysis.Finally, six Weeks; after training, each respon-dent received a second questionnaire with his orher name typed it directly. This insured thatthe designated person received and completed.the questionnaire and that the questionnairecould be identified upon its return. Question-naires returned after the five-week cut-off pointwere not included in the analysis.
FINDINGSManagers in the experimental group improved intheir perceived ability to take responsibility anddelegate authority. Their subordinates alsoperceived an improvementin managerial use ofauthority. These improvements; however, werenot significantly different from parallel improve=ments in the control group. Because both groupshad an- equivalent amount of change, it wasimpossible to attribute positive effects directly tothe training.
Ashton, David and Gibbon, Brian. Evaluating a training program in the probation service. EuropeanTraining, 1974, 3(1), 62-74.
OM= <....stsAshton and Gibbon apply a mix of formative and summative evaluation techniques to the developmentof a management course for probation officers.
TRAININGBeginning in Feburary 1973, an experimentalcourse was run by the Durham University Busi-
ness School for Probation, Officers in the Dur-ham probation Service. It was the first time thatthe university had taught management concepts
1
to a social service department This type oftraining within the Probatior Service had alsopreviously been limited.
'The course objective was to impart an under-standing of management concepts wad their ap-plication within the probation service context.Subject matter was presented by four me-mbersof the university staff op the university premisesin five one-day sessions, each three to fourweeks apart.
EVALUATIONBecause this was the first ccurse of 'its kindconducted by the university, the evaluators se-lected .a strategy which contained elements offormative, evaluation. They continually gatheredinformation during the development and execu-tion of the course and fed it back to course de-velopers and instructors. In this way evaluatorswere able to achieve the to,istant refinement ofthe course as it evolved.Pricer to training, members of the teaching staffmade several visits to the Probation ServiceOffice to look into the. organiiation's problems.In addition, prospective participants were sent aprecourse questionnaire asking for their own cri-teria of course success. The responses were ana-lyzed and presented to the teaching utaff beforethe course began. The evaluation functionhelped in foranlating teaching policy at the out-set of the program by producing relevant infor-mation which changed assumptions made in theearly stages of the course's development.While the course was in session, two instru-mentsments were used to gauge partieipants' percep-..tions. Session rating scales provided an indica-tion of participants', reactions to the key dimen-sions of the course. A session assessment formasked open-ended questions about the content ofeach session and its perceived benefit to the par-
ticipant. These comments were typed out andpresented to the instructors shortly after the endof each Session. The; information was also fedback to the trainees in' the hope that they wouldfind each others' conirnents valuable and wouldcontinue to'fill out the forms.Thee evaluaiors, the teaching staff, and the Pro-bation ServieWfficer jointly formulated the (1e-sign for a final questionnaire. The main reasonfor this survey was to obtain feedback on theoverall effects of the course, indicating areas ofweakness which could be strengthened for sub-sequent offerings. The questionnaire, sett outthree months after the course had ended, elicitedprimarily personal perceptions from which thevalue of the course to the organization was in-ferred. According to Ashton and Gibbon, "Atfirst glance this questionnaire wo ld appear to bea straight forward 'reactions' valuation tool,but it must be remembered th t respondentswere the toil management strata of the serviceand hence; value to them personally must bealigned with value to the organization.7
FINDINGSOverall, the evaluators found 'that the informa-tion provided by the evaluation effort was ofdual benefit. It served to improve the trainingprogram during its development, as well as help-ing to assess° the value of che ?raining after-wards. The evaluation was especially useful indesignating instructionta goals, establishing rele-vant course .content and providing the necessaryfeedback on course effectiveness.Through the application of formative evaluationtechniques the evaluatm, university staff andProbation Officer were able to combine their tal.-ents and develop a course which addressed theneeds and the interests of their audience.
Blake, Robert, Mounton, Jane S., Barnes; Louis B., and Greiner, Larry. Breakthrough in organizationdevelopment. Harvard Business Review, 1964, 42(6), 1337:155.
External consultants, Barnes and Greiner, used a combination of quantitative and qualitative' measuresto d rmine the effect of Managerial Grid Training upon organizational effectiveness.
NINGEight hundred managers in a 4,000 member divi-sion of a large petroleum company received
2.
Managerial Grid Training, a one-week lab-semi-nar using problem solving and group discussiontechniques and analysis of self assessment ques-
1 0
tionnaires. Exercises concmtrat7:d co interper-sonal relations and settif and iiiev'ng goals.Senior line managers serv,..1 as instructors.
EVALUATIONThe evaluation addressed three main objectives:(1) to determine if train=ng in behavioral scienceconcepts could induce: organi7atiorai changethrough the behavior the trained managers;(2) to determine if introspective analysis dam-ages the mental hth of trainees; and (3) toassess changes in individual and group behavior.To assess the degree of attainment of the statedobjectives, the evaluators used questionnaires,interviews, and observations. Past and presentcompany records were examined in order toseparate program from tionprogram effects. The,data were collected over a four month periodabout one year after the first course offering,while the last offering was in progress. The eval-uators examined productivity and profit indexesfor the periods before and after training. Post-training surveys provided information onchanges in opinions and attitudes, and inter-
views and conversation yielded evidence of be-havioral changes.
FINDINGSDuring the year that the training program was inoperation, the company reported a considerablerise in profits and a decline in costs which theyclaimed were not attributable to price changes inraw materials and finished products or reduc-tions in the labor force. Overall, employee pro-ductivity increased without additional investmentin plant equipment. Other positive effects de-monstrated by the varied measures included acatalog of organizational problems solved duringthe year, increased frequency of meetings, achange in"criteria for management appraisals anda shift towards the 9.9 or high production andhigh people concern as measured on Blake'sManagerial Grid. This study demonstrated thatbehavioral science concepts can be incorporatedinto individual behavior, leading to_more effec-tive group and, utlimately, organizational func-tioning, without any adverse effects on employeemental health.
Blake, Robert R:, and Mouton, Jane S. Some effects of managerial grid seminar training on union andmanagement attitudes towards supervision. Journal of Applied Behavioral Science, 1966, 2(4), 387-400.
Blake and Mouton evaluated the effect of Managerial Grid training on both the production centeredorientation of managers and the employee centered orientation of union leaders.
TRAININGIdentical but separaiv Managerial Grid 'Seminarswere conducted fur thirty-three managementpersonnel- and for twenty-three union officials,all of whom had management or staff responsi-bilities within the local. The training experience,relying mainly upon a laboratory format, alsofeatured pre-course readings, organizational andself-assessment instruments and structuredlearning experiences. The purpose of the Semi-nars was to aid management personnel andunion officials in gaining a fundamental under-standing of modern behavioral science conceptsconcerning sound people-production relation-ships.
0
EVALUATIONMeasurement for the evaluation was taken bothbefore and after training. To assess both groups'attitudinal changes toward supervisory practicesand consequent changes in managerial style, theevaluators used a measure derived from thegoals of the Managerial Grid. The forty-itemforced choke questionnaire assessed beliefsconcerning an individual's supervisory practicesregarding the integration of people in a produc-tion setting. Five distinct managerial styles werenoted by the grid approach and measured by thequestionnaire: maximum concern for both peo-ple and production (9,9), minimum concern forboth production and people (1,1), maximum
3
concern for production and minimum for people(9,1), minimum concern for production and max-imum concern for people (1,9), and a. balanced,but moderate, concern for both (5,5).
FINDINGSInitially, managers scored higher than union rep-resentatives on the sty'es haAing a high prOduc-
tion orientation' and lower on those with a 161.ypeople orientation. No differences betweengroups were found on the moderate style. AfterManagerial Grid training, both groups showedincreased concern for people, but this increasewas more marked for the managers.
Blocker, Clyde E. Evaluation of a human relations training course. Jo_ urnal of the American Society ofTraining Directors, 1955. 9(3), 7-8, 46.
ANEW
Blocker, a university professor, demonstrates the use of previously collected or historical data to as-, sess the results of a human relations training program. ,
TRAININGThis study addresses a leadership training pro-gram conducted for seventy-eight supervisorsfrom all levels of management of an insurancecompany. The purpose of the course was to helpsupervisors understand their personnel-responsi-bilities and give them some techniques whichwould make their leadership more positive andeffective. During ten two-hour sessions spanningan eight-month period, trainees covered supervi-so: responsibilities, leadership, 'motivation.employee cooperation, employee attitudes an._
their effect upon production, counselling, givingorders and discipline. Instructional strategies in-cluded selected readings, films, slides, groupdiscussions and role playing.
EVALUATIONThe evaluation, which took place after the com-pletion of the training program, overcame someof the problems of using only posttraining mea-surements by using historical data. The evaluatorselected fifteen of the seventy-eight participantsat random to take part in the study. Then, basedon observation of on-the-job behavior taken dur-ing the preceding two years, the selected partici-pants were classified into two groups accordingto their style of leadership; eight were classifiedas democratic and seven as authoritarian.
Three months after training, the evaluator exam-ined standard printed interview forms to deter-mine if members of the Sample had experiencedbehavioral change. The organization requiredsupervisors to fill out these forms when he/sheinterviews an employee for any reason. Theseforms contain the reason for the interview, theattitude of the employee, supervisor's Commentsand action taken. All forms were classified ac-cording to the tone of the interview: threateninginterviews were classified as authoritarian; re-warding and constructive ones were seen asdemocratic. To' determine whether any of thefifteen supervisors had changed their behavioralpatterns, Blocker compared the pretraining clas-sifications with those derived from the posttrain-ing examination of interview: forms. Seventypercent of the previously authoritarian group',responded in an authoritarian fashion and eightypercent of the democratic group respondedl_democratically.
rNDINGSBlocker concluded that the course was *able toovercome natural resistance to change found insupervisors. He stressed the importance of topmanagement support of new policy, and theimportance of training on an individual basis.
12
to, "Buchanan, Paul C., and Brunstetter, Phillip H. A research approach to Management improvement: Part _
1, Part II, Journal of the American Society for Training Directors, 1959, 13(l), 9-8, and 13(2), 18-27.
In-house investigators, Buchanan and Brunstetter, evaluated a T-group training program to determineits effects upon organizational climate and individual interpersonal behavior.
TRAINLNGThe three day, two night T-group training pro-gram took place at an inn two hours from the
. work site. Two hundred twenty-fOur managersfrom one large department of the Republic Avia-tion Corp.. attended.The training had two main goals: improving or-ganizational cclimate with respect to the rewardsystem, role larity, morale, inter-level commu-nication and employee utilization; and develop-ingcindividual interpersonal problem solving andbehavioral Skills., Instructors conveyed coursecontent throUgh small group (9-12) lab-type conferences, supplemented by larger group (12-24)factual lectures.
EVALUATIONThree to seven, months after the managers hadbeen trained, they Teceived_a questionnaire ask-
ing them for a comparison of how things werecurrently done within their work unit in relationto how they had been done a year ago. Thequestionnaire, directed at organizational devel-opment, required managers to rate changes inthe effectiveness of various organizational func-tions. A second group of 133 untrained managersfrom another departmental unit, acting as a con-trol group, also completed the questionnaire.
FINDINGSThe researchers determined which organizationalfunctions were controlled by management andcompared the two groups along these dimen-sions. The trained group reported a greaternumber of pcisitive changes.
Burgoyne,cie n G. An action I.research experiment in the evaluation of a management .developmenttk..",-.course. Jopin of Management *idles, 1973; 10(1), 8-14. 0
am= l'
Burgoyne tested an action research design in the evaluation of a university-based management training.program. This- design permits some of the advantages of formative evaluation to lie trans-nitted to anotherwise summative .evaluation study.
TRAININGBurgoyne evaluated a twelve-week management
. course- offered. by the Manchester BuSinessSchkit ip England and taught by the school's.faculty.
EVALUATION
monitors the change brought about_hylgis inter-Nention, which is what Burgoyne-did.The specific action research design employed byBurgoyne is called a-' action researchdesign, in which-the researcher is actually carry-ing out tWc.-inliePendent studies, one of which is"insidePlhe other. First the researcher observesthe-research situation and uses his or her obser-
In this stud9., Burgoyne dernoristrates the apple- .-4ations to influence the situation ("influencingcability ''ty of an "action research" approach to the procedure"). In the second phase the "monitor-evaluation of 'a management training progdm.'the traditional researcher observes and-analyzes
the pther hand, uses the earch findings tothe research situation. The arcetsion--re'searcher, on
bring about change in the research situation
ing procedure3" the researcher-asSesses the con-sequences-0f change brought about through theintervention. This.second study is actually takingplaee.within the framework provided by the firstone..
5
During the first or "influencing" phase of hisresearch BUrgoyne chose tO study twelve of thethirty-six predefined course segments. He interviewed the faculty members responsible forthese segments and used the resulting informa-tion to develop instruments which allowed thesesegments to be compared to future offerings.During the interviews, faculty members desig-nated objectives and criterion measures and pro-jected possible behavior changes which could beexpected if their course segment was successful.About half of these instruments included objec-tive tests. The instruments were administeredboth'before and after training. The posttrainingmeasurement asked, in addition, if participantsexpected subject matter to influence their work,and if s6, would it lead to .a demonstrable effectin the organization. Information was fed back toinstructors after the session was over for use inthe preparation of the next course offering.The second or "monitoring" phase of the studytested the following hypothesis: The impact of
the "influencing" research will manifest itself ina greater increase in average participant rating oflearning for those parts of the course that hadbeen studied than for those parts which had not.When the university_ repeated the course, nine ofthe studied course segments and twelve of theunstudied course segments were taught by theinstructor who had taught them previously. Parti-cipants in both the studied and unstudied groupsrated perceived change in learning on a seven-point scale after they had completed the seg-ment.
FINDINGSA comparison made between the two groupsshowed that participants in the studied groupreported a significantly greater amount of per-ceived learning, thus 'proving- the stated hypoth-esis. The success of this evaluation indicatesthat the action research method has great pofen-tial for the improvement of ongoing courses.
Goldstein, Stanley, Gorman, James, and Smith, Blanchard B. A partnership in evaluation. Training andDevelopment.Journal, 1973, 27(4), 10-14.
While the "one shot case study" (or single posttraining measurement without control group) is general-
ly considered the least acceptable approach to training evaluation, there are some times when financial,organizatiohal and time constraints prevent the use of more rigorous designs. Goldsteih et. al. demon-
.
strate, how under such conditions, evaluators canzapitalize on available resources.
TRAININGIn conjunction with a Kepner-Tregoe research.associate, a training center director and employ-ee development specialist undertook a study .todetermine whether courses had
- been effective in meeting the needs of theNASA Manned Space Craft Center in Texas.Over. five year period, 300 Manned SpaceCraft managers attended a total of sixteen offer-ings of -the problem-solving and decision_rnakingtraining course.
4
EVALUATIONThe objectives of the evaluation were to assesscourse effectiveness, to determine if NASA's
,
6
investment in the course had paid off in: betterproblem solving, and to find out what levels andtypes zof supervisors would benefit from thecourse. To gain the answer to .these questions,the evaluation team,designed a self report ques-tionnaire. After review by NASA executives andpilot testing had been completed, the question-naire was sent to all 250 former course partici-pants remaining with NASA at the time of theevaluation. The ,questionnaire included a, back-,.ground sheet, questions emphasizing changes>inbehavior and job performance, and an optionalsheet- for noting specific incident when courseconcepts had been applied. In addition, the eval-uators obtained a measure of results by estimat-ing the return on investment attributable to thetraining program.
1.
*1
FINDINGS .
The results of the evaluation were encouraging.Program effectiveness was judged by two crite-ria: Perceived value and extent of application tothe job situation. One half of the respondentsreported the course as having significant value.Forty percent reported specific areas of benefit.More than one half reported frequent informaluse of course concepts and slightly fewer thanhalf reported repeated formal applications tomajor job concerns.In order to get an approximation of return onNASA's investment in the training program, theevaluators compared the cost of the course tothe 'typical manager's annual salary. The coursecosts, including participant salaries, facility andprogram expenses equalled about five percent ofa manager's annual salary. By contrasting thisinvestment with payoff, in terms of perceivedgrowth and performance efficiency attributable
to the course, the evaluators estimated that thereturn on investment had been 200 percent thefirst year.Analysis of questionnaire data failed to pinpointa single factor designating those levels and typesof supervisors most likely to succeed in applyingcourse concepts to job concerns. The data did,however, suggest that "the more talented em-ployees received greater course benefit." In ad-dition, the application of course techniques tothe job depended upon the kind and level of po-sition which.the supervisor held and the degreeto which his superior, and peers supported theuse of these new techniqiies.The evaluators concluded with a. discussion ofhow course value can be maximized. Supportiveorganizational factors, applicatiOn of knowledgeon the job, and training employees at a live-inrather than an on-site facility all had an impacton maximimizing course value.
Goodacre, Daniel M. III. Experimental evaluation of training. Journal of Personnel Administration andIndustrial Relations, 1955, 2(4), 143-149.
In this 1955 article, Goodacre expresses the need for 'formal ,evaluation of training by ,training direc-toi-s, then demonstrates how to conduct an experimental training evaluation.,
,
TRAININGThe subjects of Goodacre's experimental studywere 800 individuals holding diverse managerialand supervisory. jobs within the B.F. GoodrichCompany. All subjects had been designated aseligible for training, Subjects were, randoMly di-vided into two groups. One group received train-ing; the second-served as a control.The trained group received, eighteen' ninety -min-ute sessions designed to improve their knowl-.edge and skills in understanding human behav-ior, decision making, employee selection, andjob evaluation. While the conference methodwas the dominant instructional method used,some lectures and discussions were included.
EVALUATIONThe evaluator measured both groups before andafter training. He used attitudinal scales to mea-sure the managers' Confidenceln the applicationof subject matter areas, their attitudes toward
o
15
the company, and' job satisfaction. AChievement .
tests measured learning in subject matter areas, ,
and supervisory ratings measured behavior at-tributable to course and overall job perform-ance.
FINDINGSIn general, the control group did not change onany of the factors measured.' The trained group,however, showed improvements in self confi-dence and learning. The trained ,group also re-ceived slightly higher posttraining supervisoryperformance .ratings than the control group, butthis difference was not as, pronounced as thechanges found by the measures of learning.Goodacre concluded by stressing the need fordata collection strategies "meeting the condi-tions of having ,an adequate criterion, controls,statistical analysis and a built-in experimentaldesign" in order to "provide management withfacts as to the return on its training investmentand aid in further program improvement."
4.
7
Goodacre, Daniel M. Ill. Stimulating man management. Personnel Psychology, 1963, 16(2), 133-143.
Goodacre shows how several control groups can be employed to test simultaneously the relative effec-tiveness of several variations of a new training program.
TRAININGGoodacre assessed a new training program de-signed to change personnel manageMent prac-fiefs through a new performance appraisal sys-tem. The training program consisted of six nine-Vt-minute sessions during which managers_learned and practiced the procedures for thenew performance appraisal system. The programconcentrated on the application of practicesfound in "good developers" such as extensivedelegation and setting high standards.
EVALUATIONSeven hundred managers in four distinct subor-ganizations of a large multi-corporation partici-pated in the study. Prior to training, subordinatesof managers in all four groups completed a sev-enteen item questionnaire measuring those prac-tices stated as occurring in "good developers.!'Three of the groups were then trained while thefourth group served as a control, rhaiiltaining thetraditional trait approach to appraisal used bythe company prior to the study.`The three trained groups were used to test theeffectiveness of variations in the training pro-gram. The first group received training in thenew system. They then were allowed to apply it
on the job and were given feedback in the formof anonymous employee comments which wereculled from the questionnaire which had beenadministered to subordinates. After the feedbacksession they received additional training. Thesecond group received the same treatment asgroup one, but did not receive the additionaltraining. The third group received no feedback,only the initial training in the use of the system.
FINDINGSThe greatest number of changes, as Goodacre I
expected, were recorded in the .first group. Thefl
control grouf:ylowever, was a. close second due I.
to an unplanned occurrenceincreased: Union.pressure upon management to conduct perform-ance appraisals and supply feedback annually.Union membership in the control group was highin comparison, to the three trained groups (72%against 19%, 16% and 23%).Even though the data showed that personnelmanagement practices could be changed througha program of training and -new administrativeprocedures, Gcodacre.was not able to isolate thedifferential effects of the new program..becauseof the unplanned occurrences in the controlgroup.
Hand, Herbert H.and Slocum, John W: A longitudinal study of the effects of a human relations pro-gram on managerial effectiveness. Journal of Applied Psychology, 1972, 56(5), 412-417. . .
The authors conducted this study to determine if a managerial human relations training course couldchange attitudes, avid whether these new attitudes were reflected in organizational effectiveness. Thisarticle reports on the second phaSe of an evaluation two years after training. The first phase, doneninety days after training, reported no significant attitudinal or behavioral changes.
TRAININGA packaged course was conducted 'by the Con-tinuing Education Division of a major - easternuniversity for .a s:nzcialty steel plant in centralPennsylvania. The training provided practicalapplication of human relations principles through
. 8
problem solving methods. The "off the shelf" orstandardized package program used, consisted of,..a total of nine hours of discussion of nianagerialstyles and a total of thirty.hours of experimentallearning experiences using individual' and groupexercises, self assessment deAces and simula-
tions. A third segment of the course provided atotal of three hours of instruction on cnrrentmotivation theory. Training was accomplishedduring twenty-eight ninety minute sessions, be-ginning in September, 1968 and continuingthrough March, 1969.
EVALUATIONThe evaluators, who were members of the uni-versity staff, used a pretest-posttest strategy tomeasure changes in behavior, attitudes and jobperformance. Twenty-one trained managers andtwenty-one controls were'randomly selected fromthe same hierarchical level of the organization toparticipate in the study. A statistical comparisonof corporate demographic factors, such as posi-tion in the hierarchy, time in present positionand educational level, insured comparability ofthe two groups.Measurements were taken prior to training, nine-ty days atter the experimental group had beentrained and again fifteen months later, Duringeach .testing, managers in the experimental andcontrol groups completed the Leadership Opin-ion Questionnaire, a measurement- of leadershipattitudes concerning .work environment and taskperformance. Simultaneously, their subordinatesreceived .the SuperVisory Behavior Description,a companion instrument which provided subordi-nate measurements of the same leadership qUali-ties. At the same time, their superiors completeda job .performance scale developed by_the corpo-
`ration's Director of Training. The traits rated bythis instrument were technical knowledge, func-
tional knowledge, drive/agressiveness, reliability,cooperation and organizing ability.
FINDINGSThe evaluators found that managers in the con-trol grin described themselves as becomingmore impersonal and production centered; theirsubordinates also described them as movingaway from concern for human relations in thetask environment. Evaluators noted a mirrorimage change in the trained group. Significantattitude and performance changes surfaced.Members of the trained group developed morepositive attitudes toward the human relationsaspect of their jobs, and these attitudes led to..positive changes in job performance.
Since the researchers had anticipated" significantchange in : the experimentals group only, theyspeculated, u pon possible explanations for theunexpected change in the control group: :A likelyexplanation was proposed. _Noticing that thetrained managers enjoyed greater success withpeers and subordinates and greater job effective- 'ness, the control group managers reacted byapplying a more autocratic, structured approach,one which they had found successful in-the firmin, the past. Increased managerial control couldhave in turn adversely effected subordinate per-formance, which was ultimately reflected in de-creased managerial effectiveness: Without pre-and posttraining measures this phenomenonmight have remained unnoticed.
Hillman, Harry. Measuring management training, a case study. Training .Directors Journal, 1962, 16(3),27-31.,
Hillman studied a training program designed to foster unifechnTersonnel policies within a large, rapidlyexpanding multiple organization, and to improve the skills of ifs managers in interpersonal relations.
TRAININGA training program was proposed to eliminatethe .widespread inability of managers to manage.Through training, the organization hoped to re-duce. employee turnover and absenteeism, im-prove operating efficiency, improve safety andreduce Workmen's Compensation costs, improveemployee morale and etudes, and retardgrowth of union representation.
The resulting three -day training program cowered: management responsibilities for planning,
_ .organizing, communications, supervision andcontrolling; selection, indoctrination.and trainingof employees; . methods improvement; handlingpersonnel problem cases; safety; and human re-lations. A team of 'two headquarter managersand a selected local manager served as instruc-tors. Course developer's' employed the confer-
9
once technique supplemented by actual caseproblems, role plays, visual aids and films. Thetrainees were branch managers within the firm.
EVALUATIONTo determine whether the goals of training hadbeen met, the evaluators employed a test .ofclassroom learning, measured trainee reactions tocourse and instructor, and assessed impact of
'training on the organization. "How Supervise,"a published questionnaire measuring changes inthe participant's thinking about supervisorypractices, policies and opinions, was adminis-tered at the begiyming and end of the course toobtain a measure of classroom learning. Aftereach session, participants_anonomously complet-ed a five-point rating scale measuring their reac-tions to each conference leader's effectivenessand the interest value, usefulness and potentialfor application of each subject. Once the train-ees had returned to their jobs, -monitoring ofturnover, absenteeism., accident frequency,Workmen's Compensation costs and union gainsprovided- a measure of the impact of- -trainingupon the manization.
FINDINGSThe evaluators decided to discontinue the use of"How Supervise" after consistently favorableresults were obtained from nine consecutivecourse offerings. At that point they were satis-fied that the course was able to transmit learningin the classroom setting.
Overall, ratings obtained from the reaction scalev$.1-;re generally high but conference leaders usingthe ,lecture approach received somewhat lowerrating. When asked for their views, however,participants expressed. the view that basic quali-ties of cc....nmon sense and understanding of peo-ple could not .be taught by this type of 'trainingprogram.
The payoffs of this training program to the orga-nization were high. Turnover dropped thirty per-cent, accidents decreased by fifty percent, andWorkmen's Compensation costs were substan-tially lower. Union activity virtually stopped, anoutcome desired by the organization's top man-agement.
Ivancevich; John M. A 'stUdy of a cognitive training program: Trainer styles and group development.Academy of Management Journal, 1974, 17(3), 428-439.
Ivancevich examined the effects that varying the role-jag _training.
TRAININGA five-day management and organizaiionat be-havior training program served as the stage. forIvancevich's experiment. The course was -designed to teach conceptual knowledge to firstlevel man.;:.,-ement within .a multi-plant manufac-turing company. Text- articles, 'cote plays,' acommunication exercise and questionnaire wereUsed to .teach motivation,, group functioning;leadership, organizational change, behavioral
rs attended.managers . .
science, company trends. Six -ty-four first level ma
EVALUATIONIvancevich explored the effect, of two contrast-ing trainer styles on group development. He
10
.
of the instructor had upon participant groups dur-
hypothesized- that he would find statistically sig-nificant differences between the two groups ingroup cohesion, interparticipant conflict, openessof communication, perceived "interpretations ofpositive productivity and participant attitudestoward training.
With the helpLof the -company's Vice. Presidentof-Production, the Director of Industrial Rela-tions and the Assistant Director of IndustrialRelations,. Ivancevich assigned the sixty-fourmanagers to two matched group's in which mem-bers were not well acquainted with each other.The managers were matched-with respect to age,education level, tenure, and salary rates.
Both groups covered the same topics, receivedidentical instructional and reading materials and
had the same trainer. The difference rested inthe role played by the trainer. In one group,"training group structured," the trainer lecturedat the beginning of each of the eight modulesand specified what the learning expectationswere and how they would be accomplished. Thetrainer played an active and directive role. Heprovided reinforcement for good ideas andsuggestions, gave immediate, constructive feed-back on completed tasks, and felt free to openlydisagree with participants. The time spent on allactivities was closely controlled. In the othergroup, "training group with minimum struc-ture," the trainer played a facilitative role. Heheld lecturing to a minimum and presented histhoughts and ideas via the chalkboard. He ex-pressed opinions only when asked and avoideddisagreements with participants. The group free-ly discussed whatever -ideas and issues theywished to be clarified. The trainer provided a
minimum of feedback and reinforcement, andexerted no control over the sequence of andtime spent on instructional and reading materi-als.
After each module, participants completed a fiftyitem rating scale measuring attitudes towardgroup cohesion, interparticipant conflict, openessof communication, productivity of module ses-sion, and attitude towards trainer. The evaluator
.traced, and examined changes in the developmentof both groups for each individual module.
FINDINGSFor the most part, the structured training ap-proach was provcri to be more effective. Thisgroup experienced more statistically significantiniproVements or group development in the areas:of group cohesion; openess of communication,productivity and attitude toward the trainer.
Leidecker, Joel K., and Hall, James L. The impaCt of management development programs on attitude-formation. Personnel Journal, 1974, 53(7), 507-512.,Iof. .6=.1., .11.-2.M.M.,1 WEN.
Emulating an earlier,study, the .authors set out toovelopment program n participants' attitudes and
management. development programs.
TRAININGLeidecker and Hall studied a ten-week manage-ment development program at the ,Santa ClaraCenter for Leadership Development. The sub-
, jects of this study were drawn, from the partici-pants in these pro,grams beginning with the aca-
, demic year 1971-72 and continuing into the aca-demic year of 1972-73. The participants, who
,were middle managers from diverse kinds offirms in the San Francisco Bay area, ranged inage froth their twenties to their fifties, with themajority in their thirties and forties.
f t
EVALUATIONThrough thei:i reseal-at, the evaluators wantedboth to measure the impact of this program onparticipantSr attitudes and values and to comparetheir findings to a similar study done seven yearsearlier by Schein.' Two instruments were usedto collect the necessary data. The first, the Pub-
,
L
measure the impact of an ongoing management de-st,
values. Their results pointed to new directions for
lic Opinion Questionnaire,, which consisted of aseries of attitude and value statements, deter-mined participants' positions on five dimensions:business and society, general cynicism, manage-ment theories and attitudes, attitudes towardspeople and groups, and individual and organiza-tional relationships. Participants completed itbefore' and after training. A second instrument,an adaption of the Profile' of OrganizationalCharacteristics, determined whether the partici--pant'S- organization was "highly supportive" or"less, supportive." Administered before trainingonly, its use was to gauge the relationship be-tween organizational. climate and permanence ofparticipant's change, if any.
FINDINGSManagers in the present study' overwhelmingly(90%) -responded that they had faith in theirworkers. On this basis, the evaluators suggestedthat future training in -the areas of Participative
1 9
Management, Human Relations and Job Enrich-ment should concentrate on developing the skillsof application rather than teaching the import-ance of these concepts.
The evaluators also found strong support for theprinciple of supportive relationships. The pro-cesses within the organization were no longerviewed as dehumanizing. Rather, the experi-ences provided'by the organization were seen assupportive to the individual, and ones which
'build a sense of personal worth and importance.The strength of this commitment, lacking in aprevious study,2 indicated that managers hadnow adopted the necessary attitudes to changetbe..,organizational relationships. The focus oftraining programs should, therefore. shift frOm
M1111117111
attitude change to concept application.The evaluators compared the present study'sresults with a similar one done by Schein in1967. Schein had studied a twelve-month man-agement education program conducted at M.I.T.The evaluators found that present participantspossessed more positive attitudes toward corpo-rate responsibility than thee earlier group. Fur-thermore, .attitudes of present subjects shiftedpositively in this area, a change not noted in theSchein study. This implies greater receptivity ofsubjects to training programs emphasizing corpo-rate responsibility and social awareness.
;Schein, Edgar H. Attitude change during managementeducation. Administrative Science Quarterly, 1967, 11(4), 601-628.
2lbid.
McClelland, D. Achievement motivation can be developed. Harvard Business Review, 1965, 43(6); 6-24,178.
McClelland cecribes and evalUates his approach to developing entrepreneurial traits in managers:
TRAININGMcClelland reports on three studies of his owndevelopment progiam which was aimed at teach-ing participants how to iMprove their perform-ance as entrepreneurs. One study took place intheU.S., the others .in India. In all programs,.participants were taught about McClelland'sconcept of need achievement (n-ach), its import-ance to entrepreneurs, and how to act and per-.ceive the world like a "hi n-ach" (a person whois highly 'motivated by the need to achieve suc-cess)
He employs four main training methods:, (1)Goal Setting=trainees set goals which are thenused as a target against which they can evaluate
"their progress. Every six months for .two yearsparticipants fill out a report fokm. This recordkeeping process keeps the goal salient. (2) Lan-
. guage of Achievementthe trainee, learns tocode his/her thoughts or fantasies .and talk interms of "hi n-ach" concepts. This in turn influ-ences'behavior. (3) Cognitive-Supports--the per-
,son must re-examine pre-existing attitudes, val-ues and self-concePts and reconcile them withthis new way of thinking and acting. (4) Group
12
SupportsHnstructors and peers provide emo-' tional support.
EVALUATIONIn order to determine whether' or not his coursehad been effective, McClellan_ d compared parti-cipants' pretraining level of entrepreneurial acivity to the level exhibited once they had -re---
Mimed to their jobs. Increased entreprenurialactivity was defined as observable acts Such asadvancement within the organization, or an in-crease in the size or scope of one's own busi-ness.
APPLICATIONSMcClelland describes three applications of histraining program. The following brief summariesaddress the programs and their findings.
Study IA pilot study conducted at a large U.S. corpora-tion involved sixteen executives in .a one-weektraining session. Participants were matched to acontrol grouP on the basis. of length of service,age, job. type and salary level. Two years later
2 0
McClelland conducted a follow-up study to findout which group of men had done better subse-quent to training. On the avera'ge, those who hadattended McClelland's course had been promot-ed faster than their matched controls.
Study IIThe, second study, done in India, evaluated asimilar training program conducted at the SmallIndustries Extension Training Institute. Fifty-two Indians from several communities attendedthe ten-day residential -Seminar. The men wereheads of small businesses, lawyers, bankers andpoliticians. Trainees participated in individualand group sessions, analyzed their self cdteptsbased on psychological test data and learned thelanguage of achievement (positive tknking) andhow to set goals. Ajollow -up period of six toten months ensued.
the traineese. Beforebeen unusually
During the folloW-up period,bled their entrepeneurial ratone third of the group had
training
titrds withac-
tive, the number increased
old business
to twobased
Product line.
a b I etraining. The judgment Wasacts such as expanding aning profit or investigating a new
business, increas-
salaried executivesStudy IIIA third study of thirty-twofrom a variety of -firms in Bombay .replicatedthese results. Before the course only about
Wtwenty to thirty percent of the men were "quiteactive." A follow-up
twocondtuheiZ t after
training revealed thathad attended the course were I:16w "unusuallyactive."
Mahoney, Thomas A., Jercike, Thomas H., and Korman, Abraham. An experiMental evaluation ofmanagement training'..Personnel Psychology., 1960. 13(1), 81-98.
Mahoney, Jerdee and Korman used a multi group experimental design to evaluate the overall effective-
ness of a managerial training program and the relative effectiveness of two types of case studies.
irwr,
TRAININGThe goals of the on-going program were to de-velop within top and subordinate levels of man-agement a knowledge and understanding ofmanagement principles, to develop the ability tol_apply a special approach, to problem-solving andto increase personal responsibility for self devel -'opment. The instructional' techniques used to
"achieve these objectives were case analysis andgroup discussidn supplemented by reading as-
-, signments and lectures." Two approaches to caseanalysis, *ere used. Some.trainees-received thestandard case approach while others receivedthe "boss involved" approach. The latter,ap-proach remlites all'' participants to meet withtheir respective superiors prior ,to training inorder to select a problem ncountered..bY partici-pants in their work. The participant then,analyz--
.
es the problem, prepares .a solution and reportsback to the suPerior after training. The instrik,for selects one. such problem for class discus-sion. The week long course' took place at non-residential facilities near the work site. Trainees
were sixty-four second level managers from onedivision of a large company,
EVALUATION .
:1
To determine whether either case approach wassuperior in achieving ccurse objectives the eval-uators compared participants trained Using each
..:case approach to partiparticipants. wh°'hadsnot as yetbeen trained. The evaluators assigued mana-gers to:=-five groups, atteMpting.t° Provtihdee
of the different functiori al responsibilities withineach group so that managers e°1114 share experi-ences andlearn front one an other, Careful'scheduling -,.)f course sessions enabled. the evalu-ators to have a control grbul); .
The first step was to test prnyanng began.
ieitraining
on, threecnterion measures beforeThe "Management FracticeS'
Presented in the
devel-oped based on course MaterialsQuito"
assess
tWesats
edge of and principlescourse. A standard scoria procedurethe use of sample case
ur enabled
solving .ability. A third .measure ' an attitudinalproblemstudies to
N '13
Seale. was developed to measure responsibilityfor self development.
After the pretest was co mpleted, two instructorstaught two sessions simultaneously. Instructor 1taught Group I using the h"-oss involved" ap-proach; Instructor II used the standard approachwith Group II. Then the instructors taught twomore sessions simultaneously, this time Instruc-tor I used the standard case approach and In-structor II used the "boss involved"
. approach.Instructors I and H taught Groups III and IV,respectively.
A posttest using the. same three criterion, mea-sures was then administered, tr, all five gr'Oups.Group. V, serving as a control group, receivedtraining after the expe riment ended.
FINDINGSAlthough the trainees showed no significant im-provement on the test of knowledge, significant
improvement surfaced on the attitudinaland on one scale of the case study. In seeking a
xpossible explanation-spiana..t;on
tneasurz
to the results, the evalua-tors compared the participants' scores on thecriterion Measures to those of thesinstructors,who had attended an earlier course offering. Ingeneral, the"e instructors scored no F.;-gher afterthe course than the participants had scored be-fore training;g this suggests that either the in-structors. Were not adequately prepared or thecriterion Measures were not relevant criteria forthe evaluation of the course.Analysis also showed no difference between the"boss involved" -approach and the traditionalaPPr in n the achievement Of training objec-tives. No conclusion can be drawn, however,because Nrticipant comments indicated that, in
:I
manyman - .nstances; the. Procedures in the "bossinvolved,' approach had not been followed.
Maier, Nord-tan R.F. An experirnental test of the train.Ing on discussioncussion leaderhip,Relations, 1953, 6(21, 161-173.
effect of Human
Maier sit up an experiment toi discssion leaderhip.
trainee! to use a permissive approachin discussion
determine if supervisors could be
TRAININGAn eight hour training course Was designed to
of group decision-makingPresent the nature andihoiv it can be adapted to dealing with job prob-'els. The instructional methods used were lec-ture, group 'discussion, and role playing. Duringa four-hour 'discussion ueriod, , trainees askedquestions and expressed their opinions and atti-tudes: The role playing problem which followedallowed the trainees to apply their new knowl-edge and allowed the evaluatorato test the train-ees' learning. In the problem, foreman had toconvince three emp loyees to their job)tasks.' '
EVALUATIONTO e'termi e whether the 'training program hadproduced a change in learning, Maier comparedtrainee performance on the role play exercise,toPerformance of. untrained managers. Instead oftraining, managers in the Control group received.
c.
14
on, resistance to Change0 one-halfserved to introduce
hour lecturethe role play.
forty-four grouPs of trained managers and thirty-5sx .groups
which
performed the role play.eachplay Pach role mentioneg
of theseof controls Pe
groups had four membe.sglove. By using iarge
In all,
controlOnditions, Mawr hoped tothe bias which individual per-
sonalities
slumbers °f role PlaYini groups for. 1-.ath'mental andosovnearchotimese-shnme
introduce into the role play setting. By
personalitiesMany- groups the effects of i
other. out.cancel each-ndividuai
,,Vie trained more effective in bring_
parts. :Fifty percent
:were
percent oftheir untrained counter-,the control group and
jog about change
fiftY-nille Pierced of the experimental group "ac-cepted Change,' in the problein setting. niftypercent, of the control group "rejected change."fa the experimentalme group, however, 36.4 per-
ess
cent accepted compromise and Only 4.5 percentrejected change.
A basic difference was found between the twogroups. Untrained leaders attempted to forcechange by using money. Trained leaders viewedthe situation as a problem to be solved by thegr°oP and tended to be more considerate, toler-
ant and open to suggestions.
The evaluator concluded that the eight-hourcourse was sufficient to cause behavioral changein the classroom setting. However, higher man-.agement support was felt to be necessary if thenew bet.aviors were to transfer to the job set-ting.
, . ...._,..Margulies.. Newton. The effects of an organizational sensitivity training program on a measure of selfactualization. Studies iri.Personnel Psychology (Canada), 1973, 5(2),67-74.r ,.. a...The objective of this evaluation was to determine whether or not an extended sensitivity training pro-gram which provides the opportunity for reinforced practicetf learning increases the ability of the. 7-7group.e'verience to develop self actualization in managers:
TRAININGMargulies studied a T-group based: managerialtraining program conducted for middle managersin a large company. This particular sensitivitytraining prograni differed from the usual, week-long approach in that training sessions extendedover a four-month period and specific sessionswere devoted to the transfer of learning to thejob. Margulies wanted to determine if this ap-proach could produce changes in the degree ofself actualization of participants. Fifty middlemanagers were selected to participate in thecomparison on the basis. of interest; ability tobenefit from, the program and lack of interfer-
.' ence from job tension upon participation in theprogram.
\ EVALUATIONMargulies hypothesized that the provision of ex-plicit application' opportunities would be reflect-ed in die,. psychological growth of the partici-pants. To test this, he exposed participants todifferent program variations, testing them:beforeand after training on a measure of self actualiza-tion.
To test his hypothesis, Margulies used a control-group pretest-posttest design. Participants wererandomly assigned to five equal size groups.Four groups received training. The fifth group,serving as a control group, was told that due tothe large number of participants it was necessaryto lc,-.elude them in a sensitivity training programplanned to be held several months later.
Trainees attended a three-hour session eachweek and one full weekend session per monthover a four-month period. One of the trainedgroups was a "pure" T- group'in which instruc-tors made no attempt to link training to job e56-peiiences. The other three trained groups,. hadspecific sessions devoted to practice of specificday to day "appli,"ations" and reinforcement` oflearning.
The Personal Orienthtion Inventory, (POI) ameasure of self actualization, was administeredto all five gioups in an orientation session beforethe program started, and again, once traininghad ended. The self actualizating person is 'onewho functions in ihe present, becoming moreaware of his or her own personal attributes, andpotential, and better able to top that potential.The POI measured self actualization by assesingInner Directedness, or whether the individualwas motivated by self or other; and Time Com-petence, or the individual's ability to function inthe present rather than the past or the future.
FINDINGSThe three trained groups did significantly better uon the posttraining measure than the untrainedcontrol. GroUps had not differed significantly onthe, pretraining measure. The results affirmed the
. value of training in increasing managers abilityto self, actualize.
Further analysis showed that the improvementsmade lay the "applications" groug. s were signifi-
15
cantly greater than those of the "pure" group.The "pure" group made gains only on the Inner
Directedness measure, while the "applications"group gained on.bgth.
Miner, John B. Studies in Management Educarion. New York: Springer, 1965.
Miner addressed the question of:tether motives of management can be changed in the direction of:greater effectiveness and if this change could be induced through training.
TRAININGMiner presented his training course to researchand development managers who were spendingnearly all°their time on research and develop-ment activities and little time in fulfilling their:managerial responsibilities.. Initially the courseconcentrated on 4asons for ineffective subordi-nate performance, shifting in later sessions to theineffective manager. Reasons for ineffective per-formance were presented in light of psychologicaland sociological theory with frequent referrals toresearch findings in those areas.
Training was given during ten ninety-minute ses-sions, conducted at weekly intervals. Minertaught four groups, roughly equivalent in size,
.primarily_using the lecture method with samediscussion when questions arose. Typically,small group discussion followed the liecttires.
EVALUATIONMiner used a pretest-posttest control group designto evaluate the effects of training. He adminis--tered the Miner Sentende Completion Test(MCAT) Ito seventy -two trainees and a contiolgroup of thirty managers in the same work unitwho did not attend the course. The MCAT usesthe sentence completion format to measuremanagerial attitudei toward various aspects ofthe managerial role. Both groups filled out theMCAT during the first week of training andagain once training had been completed.
- 16
FINDINGSOn the posttest, fifty-nire percent of the experi-mental group inproved, twenty percent made nochange, and twenty-one percent dropped inscore. Miner felt that this decrease was not dueto the training, instead attributing it to externalfactors. A similar decrease in the control groupsupported his opinion that organizational changehad been the cause. 4.
Folloiv-up StudyMiner conducted -a follow-up using job relatedcriteria instead of the questionnaire. Fifty-twomanagers who took the course were comparedto forty-nine untrained managers. Both groupswere equivalent in salary and grade level at thetime of training. Performance appraisal data in-dicated that both groups were fairly wellmatched.An examination of posttraining promotion rec-
ords for those remaining with the compatiy dur-ing follow-up, showed the experimental groupsignificantly superior. Within the experimentalgroup eighty-six percent had ,at least one pronio-
while.only fifty-six percent of the controlr,group experienced the same progreSsion. Mineralso examined rehiring recommendations forthose who left the company during' the follow-upperiod. Only thirty percent' of the controls, ascompared to sixty-nine percent the experi-mental subjects who. separated, did so with astatement in their files indicating that their supe-riorS" were sorry to let' them go:
Moffie, Dannie J., Calhoon, Richard, and O'Brien, James. Evaluation of a management developmentprogram. Personnel Psychology, 1964;`17(4), 431-440.
Moffie, Calhoon and O'Brien conducted a study demonstrating, the need for moreethan one measure ofcourse effectiveness. ,
'TRAININGThe consulting firm of Kepner-Tregoe and Asso-ciates Inc. provided their problem solving anddecision-making course to the top three levels ofmanagerial personnel of a large southeasternpapermill. The course lasted a total of twentyhours:"
EVALUATIONThe evaluators collected three distinct types ofdata, all of which were considered in the finaldecision to retain the course: (1) a course.'-endquestionnaire \ obtained trainee reactions to theprogram: trainees were asked for their Opinionsop the extent to which the course met its objec-tives, how the program related to their job relat-ed problems and\ how the program might be im-proved; (2) observers recorded the behavior of
,participants during 'practical work periods intopre-established categories- in order to determinethe effects of training on interpersonal relation-ships; (3) an experimental 'evaluation was con-ducted to determine the' effect of training on
learning. Trainees and a second group of mana-gers from another mill, matched on the basis of
obiographical and psychological test data, re-ceived pre- and post course measures. These in-cluded the Watson-Glaser Critical Thinking Ap-praisal, a standardized test measuring the abilityto logically organize and "analyze problems, andpre- and posttraining case problems developedby the consulting firm.
FINDINGSThe evaluators found no statistkally significantdifference between the trained group and thecontrol group on the experimental measures.They found, however,: that the trainees thoughtthe program to be worthwhile and were applying,the techniques taught in the course to day. to daymanufacturing problems. They concluded byreporting that "Management plans to continueuse of the training program in view of the factthat in specific situations the techniques learnedinihe course appearto be beneficial in the solu-tion of every day manufacturing problems."
Moon, C.G., and Hariton, Theodore. Evaluating an appraisal and feedback training program. Person-nel. 1958, 35(3), 36-41.
Moon and Hariton demonstrate that meaningful results can be obtained from relatively simple measur-ing methods.
TRAINING.
In the spring of 1956, the Engineering Section ofa department of the General Electric Companyinstalted a new appraisal and personnel develop-ment program. The new program featured a re-vised performance appraisal system and a thirty-hour course teaching line -managers to use ap-praisal information in helping subordinates devel-op themselves. The program stressed both theessential principles and interviewing skills neces-sary to feed back appraisal results in a mannerdesigned to stimulate employee growth and de-,
a
veldpment. Fifty managers attended half-daysessions conducted in-house for a period of twoweeks. During these sessions, they received lec-tures and participated in a discussion and a role
EVALUATIONTwo years after the adoption of the new apprais-al system a decision Was made to evaluate itseffectiveness. A consultant from the,EducationalTesting Service and an in-house ev-aluator joinedforces to determine the effects of a new apprais-
17
al and a personnel development program. Ques-tionnaires were sent to sixty-six randomly select-ed employees in the Engineering Section and acontrol group of sixty-seven employees in thecompany's Manufacturing Section. The lattergroup had used a different appraisal system andtheir managers had received no training in feed-back techniques. The questionnaire asked subor-dinates for that i.;:iinions concerning changes intheir superior's job behavior and attitudes. Tendimensions of effective supervision were as-sessed i!-1, the posttralnIng measure. Managerswho attended training also received a question-naire at this time asking their views about howthey themselves had changed.
FINDINGSSubordinates in both the experimental and thecontrol groups expressed that an overall im-provement in their managers' behavior and atti-tudes had occurred as compared to two years
0
ago. The experimental group perceived theirsuperiors significantly higher than the controlson eight out of ten dimenSions. The greatestdifferences in perceptions fell in the areas of giv-ing recognition for good work, providing oppor-tunity for subordinates to express their view-points, and expreSsing concern for their employ-ees', futures - gains clearly attributable. to thetraining. In addition, the 'course participants re-ported that they felt. it easier to talk to., theiremployees about job problems, and were able to'suggest solutions. On the other hand, the majori-ty felt that subordinates did not share any moreof their job problems, with them than they hadpreviously, nor did they feel that working cela-
shijps,with peers had improved.e evaluators concluded, based, on this study,
that the course definitely had a positive. impactOn the engineering managers and their subordi-nates, 'although there were other, factors, notexamined in the study, which miglii::also havehad, an effect.
Schein, 'Virginia. An evaluation of a long term management .training 'program. Training and Develop-. ment Journal, 1971, 24(12), 28-34.
SchA describes a study which attempted to assess the impact of a long-term training program uponparticipant interests, attitudes and persOnality characteristics, and to determine if predictors of thesechanges could bi found.
TRAININGSchein examined an eight-month managementtraining program conducted for college graduatesby the American Management Association_. Ithad three objectives:* (1) to prepare yoimg peo-ple in philosophies, principles, skills and actualtools of management; (2) to make, participantshighly desirable candidates for employment inany kind of organization they wish to enter; and(3) to tielp particiPants gain intellectual maturity,emclional stability, and leaderShip capabilitymore- quickly than could be attained throughother sources.
. The course utilized instructional methods- suchas lectures,- group discussion and films as well asprogrammed instruction, video tape business'
..simulations and T-group sessions. Instructorswere drawn from the ranks of practicing .mana-
18
gets; consultants,_ labor leader's, goVernmentofficials and educators toinstruct in ,their respec-tive areas -of. expertise. The trainees' included inthis study were 124 male members of the firstfour course offerings (1968-1969 term). All werecollege graduatei and ranged in age from 21-31.
EVALUATIONThe purposes of the evaluation were twofold: (1)to determine how the learning experience effect-ed changeS' in, participants' interests, attitudesand personality Characteristics, and (2) to-deter-mine- if 'predictors of these changes could be.found. During the first one, and one-half weeksand again, eight months later, just prior to gradu-ation, participants received a battery of:variedassessment instruments. These instruments 'wereselected to assess changes occurring in the parti-cipants attributable 'to training. Inchided were:
26
the:Study-- of Values; Strong VoCational InterestBlank-,`Guilford-Zimmerman TeMperament Sur7vey; Leadership Opinion Questionnaire; Rotter'sInternal-Eicternal Scale; and three measures de-sigpied tg, measure attitudes toward manage-.itient-Influence Questionnaire, Public Opinion:Qua- ..Jnnaire and a ..portion of the participant
Jrmation Survey.
Further measures were administered only onceduring the first week to 'determine whether or.not they could serve as potential predictors of
'change. These included: Adaptability Test; OtisSelf Administering Test of Mental' Ability; MillerAnalogies Tist; California Test of Personality;Edwards Personal Preference Schedule; GordonPersonal Inventory and Profile;- -BiographicalInventory; -and a portion of the Participant Infor-MatiOn SurVey dealing with previous experience.
r FINDINGSThe evaluators compared pretest scores . with-pOstteit scores on all tests and subteits of the
battery. .Significant change occurred on thirty-two' of the seventy-four subtests administered,confirming the effectiveness of the course:.Changei fell into five general areas: leadershipstyle, attitude toward business, self confidence,need for sociability and, diversity. These changesseemed to correspond roughly with the objec7tives of the program.
After differences in pre- and posttest scoreswere calculated, the potential predictors ofchange were compared to (correlated with) those_measures of change which shoWed significantdifferences between pre- and posttest Scores.Results revealed that predictors of these changescould be identified. Biographical indicators ofchange surpassed intelligence and personalityfactors in their ability to predict change. Thesefindings suggest .that measures of -backgroundand past experience could be 'used. to identifythose individuals most likely to benefit from thisprogram:
Schwarz, Fred C., Stilwell,-William P., and Scanlon, Burt K. Effects of management development onsubordinate behavior. Training and Developnieni =Journal, 1968, 22(4), 38-50; and 22(5), 27-30.
The evaluators demonstrate how a 'time lag design can be used to .obtain a control group when onewould ordinarily not be available. They also point out, the difficulties of this design.
TRAININGA large rnidweStern insurance corporation devel-oped' a. university based general managementseminar, in conjunction with.. the UniverSity ofWisconsin. The course had two..units..The fist,"Effective Executive - Supervision," stresseddealings With people. The second, "The- Man:agement Process," dealt with work aspectk suchas planning, organizing, 'controlling. and-, delega7;lion. Each session lasted;three daYs. rofessors
-4- from the,' University of Wisconsin and otheruniversities, business leaderS, and consultantsinstructed. Workshops in which participantsCould:test .learning were .itressed. Other in-
, stxuctional methdds employed were lectures,,direCted discussions, Case, studieS, buzz groupii,.31.films, and: role .plays. Fifty' -seven managers rep-'resenting #ve levels Of top management attend-ed.
EVALUATION .
An evaluation study was designed by universityperionnel: to assess the. impact of the courseupon management behavior and to identify sub-ordinate reaction to any change. The evaluatorsemployed:a special case of the pretest-posttestcontrol grapp design called the "cycle 7 or time-lag design. As in 'the former design, the prospec-tive. .trainees are divided into two groups theuperiMental group and .the control group.: How-ever, in the time -lag .clesign bOth groups ultimate-ly.receive training.. First .bOth groups take thepretest,. then One:grOup is trained.. Then, -bothgroups stake- the posttest..;..Finally, the secondgroup *selves training and further coinparison
'ititieasureS may be made. This '.procedure allows'for the establishment of a control group withoutdepriving anyone:Of training.' css.
The prospective participants in this study wererandomly divided into two groups and thetrained six months apart. Group "A" startedtraining in September,1965, and Group "B", inFebruary 1966.
Between the two sessions, former and prospec-tive participants were interviewed and asked todescribe critical incidents in their jobs and howthey handled them. No mention of, or connec-tion to the university program was made.. Thecritical incident portion of the evaluation mea-surement assessed the imparGof training uponmanagers' behavior.
4Before and after their managers were trained,subordinates completed the Leadership BehaviorDescription Questionnaire (LBDQ). The instru-ment describes the ledership practices or activi-ties of the superior as perceived by the subordi-nate without assigning value judgments to thebehavior. This provided an overall measure oftraining effectiveness.
FINDINGS.The following concluSions were derived frominterview results. Grotip "A" shoWed a greatertendency' to emphasize personnel events overproduction related events. They also placedgreater emphasis on employee attitudes and hada greater concern for employee development.HoweVer, when control Group "B" mentioned
personnel during interview's they spoke maiplyabout staffing and resolving conflicts, and had agreat concern for production. The groups alsodiffered in their methods and effectiveness ofproblem solving. Group "A" repOrted greatereffectiveness through the use of higher levels ofmotivation, employee involvement and develop-ment; Group "B" used persuasion and sales-manship to-, solve problems. Group "A" alsodisplayed more of a tendency to learn from ev-eryday experiences.
A second comparison -between the two groupswas made b5sed on the results of the subordi-nate questionnaire. ,After. training, Group ""A"was perceived as becoming less active, less de-finitive and, less production-centered. Subordi-nates described the group "B" managers as
, moving more toward structure and becomingmore active. The course, therefore, had oppositeeffects on the two groups. ,
The evaluators suggested that these differenceswere due to the time gap, rather than the train-ing program. Many members of both groups haddaily contact with one another. Because of theinterdependency' of peOple working within theorganization, the time lag deSign allowed the"negative" results of Group "A".'s trainingexperience to influence the outcomes of Group
's experience. The evaluators were, there-fore, unable to reach any definite' conclusionsbased on the results of this study.
Thorley, S. Evaluating an in-company management training program. Trair ng and Development Jour-nal, 1969, 23(9), 48-50.
This study examines an attempt to assess training related behavioral changes in the applied work set-ting.
AAININGThorley describes the evaluation of a trainingcourse given. to 234 managers working in a largeEnglish telecommunications company. The oneweek- course was designed to broaden the knowl-edge of the managers, " expose them to newideas, and to create the sense of functional inter-depiendence. Each day, managers, received 'adifferent topic. Topics included keting, tech-nical manufacturing, finance, and rsonnek
20
4.
Course speakers, for the most part, came ,fromareas of specialization within the company. Thetraining program took place in :the company'ssmall training center in London.
:EVALUATIONThe evaluator devised a questionnaire to collectfrom the' participants possible criteria outside thetraining; situation against which course successcould be measured. The questionnaire was corn-,
00- U
pleted anonomously by participants at the closeof the course. From the responses, the evaluatorderived six categories: introduction of new tech-niques, financial control, Cooperation with otherfunctions, attitude toward job and company,staff development and decision making. To as-sess whether changes in behavior, skills or. atti-tude had resulted from training, interviews witheach trainee's immediate superior were arrangedsix months later. Each superior was given a listof his subordinates who had attended the courseand asked about the performance of the group ineach of the six areas.
FINDINGSOne half the superior's cited ,change in the spe-cific areas of financial control, cooperation with
other functions, attitude toward job and comp-any and decision making. The other half indicatedpositive, changes had occurred in these areas,but could not be specific. One third' reportedchange in staff development; a quarter reportedintroduction of new techniques and ,a quarterperceived a change but couldn't .be specific.
Thorley noted that there might have b-en someimprecision in the measurement technique sedin this study, but he believed this approach topreferable to course ratings, speaker ratings andpost courses reports. It may ,mean more workfor the training, manager, but the results obtainedthrough this approach are directly related to thecourse objectives which had been derived froma perception of training needs.
Valiquet, MiChael I. individual change in a management development program. Journal of Applied Be-havioral Science, 1968, 4(3), 313-325.
Valiquet examined participants on-the-job behavior to determine whether,learning from an experimen-tal organizational development program had resulted in changed attitudes and behavior.
TRAININGThe training prograin under observation was anon-going lab-type organizational developmentprogram with provision for 'guided on-the-job
/application of learning. The primary goal of theprogram was to optimize the company's use ofits human resources and to improve collabora-tion in working toward common goals. To thisend, participants were taught to analyze the con-sequences of their own actions and encouragedto .consider and experiment with alternativemanagement assiimptions, attitudes, and behav-ioral patterns.
EVALUATIONAbout one year.after tall training had .been cbm-pleted, Valiquet undertook -the task of evaluatingthe training effort. He sought to apess the im-pact of the program on the attiffiffes and behav-ior of the participants in terms of the statedtraining objectives. He did this by comparingdatil from trained and untrained managers foreyidence of such changes.
Valiquet selected at random sixty participantsfrom four different company locations. He se-lected these four because in each one, -the train-ing had begun at least one year prior to the eval-nation: Trainees therefore had ample time to in-ternalize the new values, attitudis and behaVior-al skills in the working environment. Also, theresearcher could detefinine how much learninghad "survived the state of early post-trainingeuphoria" and the pressures and- adversities ofthe work situation. I A "matched pair controlgroup" Was obtained by asking each experimen-tal subject to nominate' another manager whoheld an identical or-almost similar functional roleto the subject, but who had not 'participated intraining.
Evaluative data was provided by "describers"Who were individuals who had worked with thesubject for the past year. They were selectedfrom-a mixed list of seven to ten peers, superi-ors and subordinates, submitted by each partici-pant. Five describers selected randomly fromeach participant's list received an Apen-endedquestionnaire asking the individual to describe
any specific change dist had occurred since the.., Previous year. Subjects received a similar ques-
tionnaire to elicit descriptions of their own be_havior.
we're_ categorizedDescribers' reskoSes accord--ing to twenry,ome content sub-Categoriesgrouped into three major categories, found to beorganizationally arid personally relevant in twoearlier studies done by Mtles2 and Bunkers. Dueto pbor overall return rates, final results were
uravailable for only thirty-four trained,-,.. subjects
and fifteen contrely
FINDINGSStatistically slAirocant -differences between theexperimental and control group describers were
obtained on "total number of changes ob-served", "total changes agreed upon by two or
. more observers", and "total number of changesreported the subjects themselves". Valiquetattributes the strong positive results to the facte r e d
bthat the program involved inhouse training, con-ducted with co-workers, by company trainers.These factors facilitated the transfer of actualbehavior to the work situation.
Wiles. M.B. Processes and outcome. Journal of Counsel-
ing2IPsYch°I. °gY' 196°' 7(4).bid
3Bunker, D.R. Individual applications of laboratory train-ing. Journal of Applied Behavioral Science, 1965, 1(2),. 131-
' 148.
Vogels, Captain vAvid :Jr. Evaluation of a management training course. Journal of the American Societyof Training Direttors, 1958, 12(1), 44-51.
Vogels used (Med°noires to test the effectiveness of ament program.
5
TRAININGVogels'describe% 0 management training programfor first and second ievel supervisors in the AirForce. It was designed to improve their supervi-sory' skills and establish two-way communica-tions within tie organization.course covered specific aspects of the supervi-sors' job, work sick atjon improvement, Produc-tivity, problem solving, interpersonal relations,improvement or yelf, and employee develop-ment. Over a one Year period trainees attendedtwenty-five twoloct conferences conducted dur-ing weekly two,,nour sessions. The "develop-mental -approach!, a modified case study ap-proach, and woryohops were the dominant in-structional methoo.
After each conference, trainees compiled a listof job related problems and forwarded
/*heseit to the
course monitors, problems were then dis-cussed with trainees, immediate suPerviSOrS atmonthly meetiNs held by the evaluator andmanagement and changes were agreed upon. In-freqUent meetings -were held with superiors twolevels above the rfainees to discuss unresolvedproblemi. These reSup were, in turn, fed back
'22
combined training and organizational develop-"
to the trainees' immediate superiors. Throughthis circuit, the -evaluator hdped to help establishorganizational commitment and open up -two-way lines of communication throughout organi-zational levels.
EVALUATIONTo test the effectiveness of the combined train-ing and organizational development program, theevaluator examined trainees' and immediatesupervisors' Perceptions while training was stillin progress. A questionnaire administered totrainees asked them how effective they thoughtthe course was and asked them if they perceivedchange in their behavior at the point of question-naire dissemination. A similar questionnaire giv-en to immediate superiors during. one of 'theirfeedback sessions asked for their feelings aboutthe effectiveness of the training program feed-,back sessions.
FINDINGS'The questionnaires showed that while 'both thetrainees and their supervisors were'satisfied withthe training, the trainees were not cognizant of
'1
'their superiors' satisfaction. The feedback technique has only succeeded in opening upward
niines of communication, not in facilitating thedo-wnward flow. The course nonetheless was
found to meet its objectives by improving super-visory skills. It was also responsible for the in-troduction of various management improve-
,. . =.ments: . - ,.
,.Avexley, Kenneth N. and Nemeroff, Wayne F. Effectiveness of positive reinforcements and goals set-ting as methods of management development. Journal of Applied Psychology, 1975, 60(4), 446-450.
e evaluators employ an experimental design to compare, the effectiveness of two variations of a su-Thpervisory skills training program.
TRAININGThe objectives of the training program were todevelop, a mole considerate, democratic style ofleadership of managers working, in a large urbanMedical center: It was hoped that through thetraining of managers- subordinate job satisfactioncould be increased and absenteeism reduced.The training program, had two phases.. Phase Iconsisted of two half-day workshops giVen onconsecutive days.- Instructors relied on role playexercises to teach those behaviors appropriate todemocratic, considerate supefvisors. 'Trainers.k,provided specific feedback to trainees _about theeffectiveness of their performance. Phase II in-yolVed guided application of these techniques onthe. job.
Nine departmentS of the medical center wereselected to participate in the prOgram. All de-partment heads were put thrOugh the trainingexercises in order to insure their understanding,commitment and support of the program. Twen-ty-seven of their immediate' subordinates, three.from each department, then participated in vari-:ations of the two phase program..
EVALUATIONWexley and Nemeroff compared three different'variations in the training program to 'determinewhich of these had the greatest effect on subor-dinate perceptiOns (behayier) and absenteeism(results) within the organizatiOn.
The participating managers were randomly as-signed by department to one of two experimentallitrouPs or the control gionp, such that each de-partment had the same number of managers par-ticipating in each group.j.
Experimental Group I received the followingtreatment or training seqUence. Before each ofthe several role plays, the trainer discussed withthe trainees a list of effective and ineffectivebehaviors specific to the particular exercise,Instructional staff observed the ,trainees' partici-pation and then provided a "delayed appraisal"which incorporated specific feedback with rein-forcement. Trainees were told how well theywere doing, and how much improvement theyhad made. During the second phase, laSting forslit weeks ,back on the job, trainees filled out abehavioral checklist daily, to keep a 'record pftheir Performance: The trainer interviewed train-ees of the end of the first and third weeks to re-view cheaked behaviors and to provide consulta-tion on any particular problem trainees had inthe application 'of neW. behaviors. During thefirst meeting, trainees set goals for review at thesecond meeting. The trainer encouraged them toset as their goals, the application of behaviorsnot previously checked. .
Experimental Group II received the same-treat-ment with the addition of "telecoaching," ordirect reinforcement giVen during exercisesthrough a device worn on the ear of the trainee.The reinforcement was immediate in nature andbehaviorall n context.
The control group received neither "telecoach-ing" nor the behavioral checklist and 'did not setgoals for review, but otherwise experienced thesame training sequence. ,
Sixty days' after completion of the training theevaluators collected measures of managerialbehavior 'and job satisfaction from subordinatesof. the trainees. Between three' and six of each
23
manager's subordinates were randomly selectedto provide inforniation on their 'superior's behav-ior on the .,Leadership Behavior DescriptionQuestionnaire and two scales ,of job satisfaction.Absenteeism records were also examined at thattime.
FINDINGSMembers of both experimental groups improvedslightly more than the control group membersdid. The improvement was, measured in terms ofsubordinates' perceptions of managerial behaviorand of absenteeism within the organization. Im-provements were accomplished without anyundesired detrimental effects to the managers'
, ability to direct the group or emphasize produc-tion goals. The basic treatment provided toGroup I was significantly, more effective in in-creasing subordinate satisfaction than the treat-,ment given Group II, featuring "telecoaChing'7via the ear device.
Post experimental interviews revealed com-plaints that the, ear apparatus had disi-upted theconversational flow of the role plays. In addition.it contradicted the participative managementstyle preached during training, since it only al-lowed for one-way communication.
WilSon, John, E. Mullen, Donald P., and Morton, Robert. Sensitivity training'for individual growthteam training for organizational development: Training and Development Journal, 1968, 22(1), 47-53.
Two different approaches to sensitivity 'training arewIthih,the organizational setting.
TRAININGThe evaluators studied two offerings,of an inter-agency Management development courSe'provid,ed by the training Division :of. the State Person-nel Board of the State of California for upperlevel managers in 'State Service. Consultants.MOrton and Wilson conducted both offerings.The six-day residential course emphasized inter-personal relationships and -leadership''skills. Thefirst prOgrain, co,nducted for forty participants,in November, 1964, used the traditional sensitiv-ity approach. The second, conducted one yearlater for forty-one participants, used a modifiedversion of the Organizational Training Laborato-ry," an approach emphasizing team decisionmaking and problem7solving.
EVALUATIONMullen, a,:management analyst with-the State ofCalifornia, conducted the evaluation. He de-
24
compared to determine their relative effectiveness
signed a questionnaire to determine the mana-gers' perceptions of .the value of their training'experience to theniselves as individuals and .asmanagers in their organizations. The question-naire included specific questions, with scaled re-sponses and an open-ended question asking man-agers to cite things they were now doing thatthey hadn't done prior to the course, both asindividuals and as managers within the organiz::-tion. The questionnaires were sent out sixmonths after the second group of managers hadbeen trained.
FINDINGSThe authors found both courses to be of equalvalue to the managers as individuals. :The Orga-nizational Training Lab, however, proved superi-or for improvements in team work and better, forinterpersonal skills and problem solving.
32
GLOSSARY
BatteryA series of related tests administered together.
BehaviorAn assessment of individual performance on the job. (Se t. Data.)
ClientThe person(s).or organization(s) for which an evaluation is conducted.
Control GroupA group, of persons who do not participate in the training program but .who are similarin all relevant respects to those who do participate. Control groups are used as a basis for compar-ison.
CorrelationA statistical process which shows the degree to which two or more events or objects arerelated to each other. In evaluations of training program effectiveness, correlations may be calcu-,lated to determine the relationships'among factors effecting training results.
Criterion, CriteriaMeasures of training effectiveness which reflect the goals and objectives of thetraining program-They provide a description or image of what should happen, thereby facilitating
. comparisons between what should have happened and what did happen.
DataFactual material from which conclusions may be drawn. When evaluating training, four catego-ries of data may be obtained: Reaction, Learning, Behavior and Results.
DesignA strategy. which the evaluator uses to collect data. The design usually specifies who will bemeasured (experimental group, control group),:and when they will be measured (pretest, posttest).The purpose of the design is to guard against the possibility that something other than the treat-ment causes the :observed effects of the training program.
EvaluationA deliberate process which provides specific reliable information about a selected topic,problem, or question.for purposes of determining value and/Or makingdecisions.
ExperiMentaIDesign7--A data collection strategy which attempts to control as many relevant and irrele-vant variables as poSsible. An experimental design 'gains its rigor by using control groups and ran-dom selection and assignment of individuals to groups.
Experimental GroupA'group of subjects who receive the experimental treatmentin a 'design.
Forced ChoiceA special kind of multiple-choice item which forces the respondent to choose the more`descriptive of two or more equally attractive or un-attractive statements. This type of item is moreoften used in personality and attitudinal measures when social desirability of the possible respon-ses may interfere with the selection of the alternative chosen.
Forniative EvaluationThe process of judging-an instructional package or...process or its componentsduring the developmental period for the purpose of providing persons directly involved with theforMation of the entity with feed-back as to possible improvements. (See Evaluation.)
Historial DataData collected by the organization as part of its normal day4o-day functioning. It caninclude numerical indices such as absenteeism; turnover or production rates, and, organizationaldocuments such as agency, memoranda, auditing reports, program budgets, employee rating forms,supervisOr appraisals and written plans.
Hypothesis-LA statement proposing a plausible relationship between two or more variables.
LearningThe principles, facts, skills and'attitudes that participants gain from training. (See Data.)
. MatchingA pairing of subjects on the basis of background information factors such as age, level of. education or organizational status, followed by random assignment of one member of the pair to
the experimental group and the other to the control group. This process, used when totally randomselection and assignment are impossible, helps prevent the persona characteristics of the subjectsfrom contaminating the evaluation results.
3 3
-25
Open-ended--A question allowing respondents to answer freely in their own words rather than restrict-ing their answers to a few stated alternatives as in a multiple-choice question. Although they aremore difficult to analyze than multiple-choice qUestions, open-ended questions allow for ..a widervariety of responses.
7_Posttest, Postcourse or Posttraining MeasirementA measurement taken after the training program has,
ended. The resulting information can be used to determine whether or not trainees have achievedtraining objectives. If compared to a pretest the posttest provides a measure of change probably t,
attributable to training.
Pietest, Precourse or Pretraining MeasurementA measurement taken before the_ program be-gins, or,during.its early stages. The resulting information may be used to provide instructional de-signerg with a picture of the skills and abilities of the average entering trainee, or may be used togive the instructor an idea of how much the group already knows relative to the learning objec-tives. A pretest also piovides a baseline for comparison against a posttest.
PreceduresInstruments devices used to obtain data for evaluation.
Random Assignment or SelectionThe selection of cases or subjects in such a way that all have anequal probability of being included, and the selection of one subject has no influence on the selec-tion of any other subject.
RatingTbe process of judging someone or something according to predetermined criteria. (See scale).
Reaction--An indication of how well the trainees liked a particular program, including materials, instruc-tors, facilities, methodology, content, etc. (See Data.)
Reliability;--The degree to which a device or instrument (procedure) measures a given characteristicconsistently.
. ResultsThe impact of training on the organization or job environment (See Data')RigorA term used to describe the amount of control exerted by a design and the consequent preci-
Sion that can be attributed to the findings. The more rigorotis the design, the greater is the level ofconfidence that one can have in the findings of an. evaluatiOn.
SampleA -subset of the population, usually selected to be representative of the whole group beingstudied.
-ScaleA graduated. continuum which allows a rater to assign numerical values ranging from low tohigh to a given trait or characteristic. Scales generally have betweenthree and nine categorieswhich may or may not have accompanying deScriptive adjectives or numbers.
Significant, Statistically SignificantA statistically significant event is one that has a low likelihood' ofhappening by chance. Significance does not mean importance; it merely means that a difference,such as the difference between the scores of two groups on a poktest, was due to some difference
. between the two o-grOups rather than due to chance; .
SAjectsIndiViduals selected to participate in any facet of a design. .
.Summative EvaluationL-The process of judging a completed instructional package or process for thepurposenf providing the end user with information as to its demonStrated effectiveness in a given,situation. Based on this information, the user may decide to purchase the entity. (if a potentialuser); or retain it (if a current user). (See Evaluation.)
Treatment=-The training program or a variation in the training program given to an experimental groupin a design. t .
TestA series of questions, exercises or other means of measuring the knowledge, skills, abilities oraptitudes of an individual or group against selected criteria or norms:
ValidityThe degree to which a device or instrument (procedure)measure.
Variable--Something that is capable of ehariging in value over time.trol for (limit the variability of) irrelevant variables so that theobserved.
measures ,what it was intended to
One purpose of a design is to con-effect of relevant variables may be
27..
AUTHOR INDEX
Albanese, R., 1Ashron, D., 1-2Barnes, L. B., 2-3Blake, R., 2-3, 3-4Blocker, C. E., 4Brunstetter, P. H., 5Buchanan, P. C., 5Bunker, D. R. 22nBurgoyne, J. G., 5-6Calhoon, R., 17Gibbon, B., 1-2Goldstein, S., 6-7Goodacre, D. M., 7, 8Gorman, J. 6-7Greiner, L., 2-3Hail, J. L., 11-12Hand, H. H., 8-9HaritOn, T., 17-18Hillman, H., 9-10Ivancevich, J. M., 10-11Jerdee, T. H., 13-14Kirkpatrick, D. L., iiiKorman, A., 13-14Leidecker, J. K., 11-12McClelland, D., 12-13
28 36
Mahoney, T. A., 13-14Maier, N. R. F., 14-15Margulies, N., 15-16Miles, M.B., 22nMiner, J. B., 16Moffie, D. J., 17Moon, C. G.; 17-18Morton, R., 24Mouton, J. S., 2-3, 3-4
'Mullen, D. P., 24Nemeroff,-W. F., 23-24O'Brien, J., 17Scanlon, B. K., 19-20Schein, E. H., 12nSchein, V., 18-19Schwarz, F.C., 19-20Slocum, J. W, 8-9Smith, B. B., 6-7Stilwell, W. P., 19-20
: Thorley, S., 20-21Valiquet, M. I., 21-22Vogels, D., 22-23Wexley, K. N., 23-24Wilson, J.E., 24
SELECTED REFERENCESti
Anderson, Scarvia B., Ball, Samuel, Murphy, Richard T., and Associates. Encyclopedia of EducationalEvaluation: Concepts and Techniques for Evaluating Education and Training Programs. Washington,D.C.: Jossey - Bass Publishers, 1975.
Chabotar, Kent J., and Lad, Lawrence J. Evaluation Guidelines for Training Programs. Lansing, Michi-gan: Training Division, Michigan, Department of Civil Service, State of Michigan, 1974.
Hatry, Harry P., Winnie, RiChard E., and Fisk, Donald M. Practical Program Evaluation for State andLocal Government Officials. Washington, D.C.: The Urban Institute, 1973.
Isaac, Stephan, and Michael, William B. Handbook in Rearch and,Evalustion. San Diego, California:Robert R. Knapp, 1971.
Van Maanen, John. The Process of Program Evaluation. Washington, D.C.: National Training and De-velopment Service, 1973.
4,1
29
BIBLIOGRAPHY OF ARTICLES ABSTRACTED
Albanese, Robert. A case study of executive development. Training and Development-Journal, 1967,21(1), 28-34.
Ashton, David, and Gibbon, Brian. Evaluating a training programme in the probation service. Euro-peararaining, 1974, 3(1), 62-64.
Blake, Robert R., and Mouton, Jane S. Some effects 'of managerial grid seminar training on union andmanagement attitudes towards supervision, Journal of Applied Behavioral Science, 1966, 2(4), 387-406.
Blake, Robert, Mouton, Jane S., Barnes, Louis B., and Greiner, Larry. Breakthrough in organizationdevelopment. Harvard Business Review, 1964, 42(6), 133-135.
Blocker, Clyde E. Evaluation of a human relations training course. Journal of the American Society ofTraining Directors, 1955, 9(3), 7-8, 46.
Buchanan, Paul C., and Brunstetter, Phillip H. A research approach to management improvement: Part.I; IL Journal okthe Asnerican Society of Training Directors, 1959,J3(1), 9-18; and, 13(2), 18-27.
Burgoyn , John G. An action research experioent in- the evaluation of a managementcourse. Journal of Management Studies, 1973, 10(1), 8-14.
Goldstein, Stanley, Gorman, James, and Smith, Blanchard B. A partnership in evaluation.Deyelopment Journal, 1973, 27(4), 10-14.
development
Training and
Goodacre, Daniel M. III. Experimental evaluation of training. Journal of Personnel Administration andIndustrial Relations, 1955, 2(4),,,143-149.
Goodacre, Daniel M. III. Stimulating man management. Personnel Psychology, 1963, 16(2), 133-143.
Hand, Herbert H., and Slocum, John W. A longitudinal study of the effects of a human relations pro-gram on managerial effectiveness. Journal of Applied Psychology, 1972, 56(5), 412417.
-Hillman, Harry. Measuring management training: A case study.. JOurnal of the American Society ofTraining Directors, 1962, 16(3), 27-31.
Invancevich, John M. A study of a cognitive training program: Trainer styles and group development.Academy of Management Journal, 1974, 17(3), 428-439.
Leidecker, Joel K., and 'Hall, James L. The impact of management development pro ms on attitudeformation. Personnel Journal, 1974, 53(7), 507-512.
McClelland; D. Achievement motivation can be developed. Harvard Business RevieW,_1965, 43(6), 6-24,178.
Mahoney, Thomas . A., Jerdee, ThoMas H., and Korman, Abraham. An experimental evaluation omanagement training. Personnel Psychology, 1960, 13(1), 81-98.
Maier, Norman R.F. Ari experimental test of the effect of training on discussion leaderShip. Human Re-;Mons, 1953, 6(2), 161-173.
Margulies, Newton. The effects of an organizational sensitivity training program on a measure of selfactualization. Studies in Personnel Psychology (Canal 1973, 5(2), 46-74.
Miner, John B. Studies in Management Education. New York: Springer, 1965.
Moffie, Dannie J., Calhoon, Richard, and O'Brien, James. Evaluation of a management developmentprogram. Personnel' Psychology' :1964 17(4) 431-440.
f
.30
Moon, C.G.,- and Hariton, Theodore. Evaluating an appraisal and feedback training program. Person-nel, 1958, 35(3) 36-41.
Schein, Virginia. An evaluation of a long term management training program. Training and Develop-ment Journal, 1971, 24(12), 28-34.,
Schwarz, Fred Stilwell, William P., and Scanlon, Burt K. Effects of management development onsubordinate behavior. Training and Development Journid, 1968, 22(4), 38-50; and 22(5), 24-30.
Thorley, S. Evaluating an in-company management training program. Training and Development Jour-nal, 1969, 23(9), 48-50.
Valiquet, Michael I. Individual change in a management development program. Journal of Applied Be-havioral Science, 1968, 4(3), 313-316.
Vogels, Captain Dayid, Jr. Evaluation of a management training course. Journal of the American Socie-ty of Training Directors, 1958, 12(1), 44-51.
Wexley, Kenneth N., and Nemeroff, Wayne F. Effectiveness of positive reinforcement and goal set-ting as methods ofmanagement development. Journal of Applied Psychology, 1975, 60(4), 446-450.
Wilson, John E., Mullen, Donald P., and Morton, Robert. Sensitivity training for individual_growth,team training 'for organizational d velopment. Training and Development Journal; 1968, 22(1), 47-53.
.1
,
39
r
. U. S. GOVERNMENT ?RENTING OFFICE : 1978 733-404/418
a
31