document resume - eric · document resume ed 272 110 he 019 557 author lenning, oscar t. title the...

DOCUMENT RESUME

ED 272 110 HE 019 557

AUTHOR Lenning, Oscar T.TITLE The Outcomes Structure: An Overview and Procedures

for Applying It in Postsecondary EducationInstitutions.

INSTITUTION National Center for Higher Education ManagementSystems, Boulder, Colo.

SPONS AGENCY National Inst. of Education (DHEW), Washington,D.C.

PUB DATE 77NOTE 84p.; For related documents, see HE 019 556 and HE

019 560-561.PUB TYPE Guides - Non-Classroom Use (055) Reports -

Descriptive (141)

EDRS PRICE M701/PC04 Plus Postage.DESCRIPTORS *Classification; College Administration; *College

Attendance; College Planning; Decision Making;Definitions; *Educational Assessment; *EducationalObjectives; Educational Research; Needs Assessment;*Outcomes of Education; *Postsecondary Education;Program Evaluation; Program Implementation;Workshops

IDENTIFIERS *National Center for Higher Educ ManagementSystems

ABSTRACTA summary of the Outcomes Structure developed by the

National Center for Higher Education Management Systems (NCHEMS) ispresented. The Outcomes Structure consists of tnree dimensions: thoseaffected by the outcome (the audience); the type of outcome; and whenthe outcome occurs or is expected to occur (time). Potential uses ofthe Outcomes Structure are considered: communication; stimulation;research; and planning, management, and evaluation (e.g., needsassessment; goal identification, assessment, and translation).Guidelines are provided for using the structure to accomplish thefollowing processes: generating lists of priority outcomes,classifying outcomes, and storing and retrieving outcomesinformation. Suggestions to help trainers and others orient theircampus to the Outcomes Structure are included. Appendices include:the coding scheme for the structure, definitions and outcome measureor indicator examples for the type-of-outcome subcategories, anexample of outcomes classified with the structure using outcomestatements from the NCHEMS Outcomes Inventory, possible copy for abrochure introducing outcomes concepts and the Outcomes Structure;questions and criticisms of the structure posed by particularreviewers and responses from the developers of the structure; andmaterials for campus orientation workshops. (SW)

************************************************************************ Reproductions supplied by EDRS are the best that can be made ** from the original document. *

***********************************************************************

a .MI

"PERMISSION TO REPRODUCE THISMATERIAL HAS BEEN GRANTED BY

cjTO THE EDUCATIONAL RESOURCESINFORMATION CENTER (ERIC)."

a

A

077

U.S. OSPARTMONT OP SOUCATION011a d Reeman end improvement

MICA. RESOURCES INFORMATIONCENTER (ERIC)

t1;147iis document ha been reproduced ISreceived from the Denton or organizationoriginating ft

D wet amnimA hove been made to improvereproduction quality

Pointe of view or olinione stated in this doormint do not wawa/ft represent officialOM position of poky

oa

2

National Center for Higher Education Management SystemsPost Office Drawer P Boulder, Colorado 80302

An Affirmative Action/Equal Opportunity Employer

The mission of the National Center for Higher Education Management

Systems (NCHEMS) is to carry out research, development, dissemination,

and evaluation activities and to serve as a national resource to assist individu-

als, institutions, agencies and organizations of postsecondary education, and

state and federal governments in bringing about improvements in planning

and management in postsecondary education

William ArceneauxLouisiana Commissioner ofHigher Education

Raymond F BacchettiVice Provost, Budget andPlanning, Stanford UniversityMay BrodbeckVice President for AcademicAffairs and Dean of the Faculties,University of IowaHenry J DeckerCommissioner, South DakotaBureau of Finance andManagement

Harold L EnarsonPresident, Ohio State Universit

Patricia GeuderAssociate Professor of English,University of Nevada, Las VegasErvin HarlacherChancellor, The MetropolitanCommunity Colleges

T Edward HollanderChancellor, New Jersey Board ofHigher Education

Hans H JennyVice President for Finance andBusiness, College of Wooster

Joseph F. KauffmanProfessor of EducationalAdministration, University ofWisconsin-Madison

BOARD OF DIRECTORSChairman

Marvin WachmanPresident, Temple University

Chairman-ElectAdrian R Chamberlain

President, Colorado State University

Robert LisenskyPresident, Willamette UniversityDaniel Marvin, JrPresident, Eastern IllinoisUniversity

jay S OlinsVice President, Medical TrainingInstitute, Los AngelesDale P ParnellPresident, San JoaquinDelta College

Frank B Pesci, SrMaryland State Delegate

James A RobinsonPresident, University otWest Florida

Wendell RussellPresident, Federal City College

Stanley F SalwakPresident, University of Maine atPresque Isle

Clarence SchepsErecuiive Vice President,Tulane University

OFFICERSBen LawrenceExecutive DirectorDean P JonesAssociate DirectorWayne R KirschlingAssociate DirectorMelvin D OrihigAssociate DirectorGordon H ZiemerSecretary/Treasurer andAssociate Director

Donald H SmithPresident, BrookdaleCommunity College

Peter SmithPresident, Community College ofVe, mont

Jack F TolbertPresident, The Medic Schools

Lois TorrenceDirector, Bureau of InstitutionalResearch, University ofConnecticut

Barbara S UehlingProvost, University of OklahomaRichard L Van HornVice President for Business Af-fairs, Carnegie-Mellon University

Geraldine P WoodsCalifornia PostsecondaryEducation Commission

Ex OfficioDouglas ConnerExecutive Secretary, AmericanAssociatic r of Collegiate Regis-trars and Admissions Officers,Washington, D.0Ex OfficioJames W WhiteVice President for Administration,American Association ofCommunity and Junior Colleges,Washington, D C

THE OUTCOMES STRUCTURE:AN OVERVIEW AND PROCEDURES FOR

APPLYING IT IN POSTSECONDARYEDUCATION INSTITUTIONS

Oscar T. Lenning

1977

This document is part of a program supported bythe National Institute of Education

National Center for Higher Education Management SystemsP.O. Drawer P Boulder, Colorado 80302

An Affirmative Action/Equal Opportunity Employer

iii

4

This document has been reviewed and approved for publication by the members of the staff andBoard of Directors of the National Center for Higher Education Management Systems.However, this publication does not necessarily reflect official positions or policies of theNational Institute of Education or the National Center for Higher Education ManagementSystems.

iv

TABLE OF CONTENTS

List of Tables vii

List of Figures vii

Pref ace ix

introduction 1

Part I An Overview of the NCHEMS Outcomes Structure 3

The Audience Dimension 4The Type-of-Outcome Dimension 4The Time Dimension 8

Part II The Potential Uses of the Outcomes Structure 9

1. Communication 92. Stimulation 93. Planning, Management, and Evaluation 10

A. Needs Assessment 11

B. Goal Identification 11

C. Goal Assessment 11

D. Goal Translation 12E. Allocation and Utilization Planning and Management 12F. Evaluation 13

4. Research 13

Part Ill Guidelines for Using the Outcomes Structure to AccomplishThree Basic Processes 15

Process 1: Generating Lists of Priority Outcomes 15Outcomes for Whom? 15What Outcomes? 17When Will the Outcomes Occur? 24

Process 2: Classifying Outcomes 24Process 3. Storing and Retrieving Outcomes Information 28

Part IV Orlonting Institutional and Program Faculty and Staff tothe Outcomes Structure and Training Them in Its Use 31

1. Do Your Homework 31

2. Prepare Draft Copy for a Brief Brochure That EffectivelyIntroduces Relevant Outcomes Concepts and the Structure 32

3. Prepare Yourself to Respond to Criticisms That Might BeRaised.. 32

4. Get the Concurrence of Tup-Level Management about theNeed to Proceed Further 32

v

TABLE OF CONTENTS (Continued)

5. Develop Preliminary Plans and Materials for a CampusOrientation Workshop 32

6. Try Out Your Ideas and Draft Materials on Trusted Friends 347. Establish an "Exploration and Planning" Committee

concerning a Campus Orientation Effort 348. Reevaluate Your Ideas, Using the Committee, and Then

Complete Plans and Materials 349. Distribute Brochures and Conduct Workshops as Planned

Then Evaluate 3510. Make Available and Publicize the Availability of Follow-Up

Application Assistance 35

Appendix A The Coding Scheme for the Structure 37

Appendix B

Appendix C

Appendix D

Definitions and Outcome Measure or IndicatorExamples for the Typeof-Outcome Subcategories 39

An Example of Outcomes Classified with theStructure Using Outcome Statements from theNCHEMS Outcomes Inventory 53

Possible Copy for a Brochure or Flier IntroducingOutcomes Concepts and the Outcomes Structure 55

Appendix E

Appendix F

Important Questions and Criticisms of the OutcomesStructure Posed by Particular Reviewers andResponses by the Developers of the Structure 59

Some Materials for Possible Use at the CampusOrientation Workshops 61

References 79

vi

LIST OF TABLES

Table 1 Major Categories of the "Audience" Dimension 4

Table 2 Subcategories of the "Audience" Dimension 5

Table 3 The Major Categories of the "Type -of- Outcome' Dimension 6

Table 4 Ccded Listing of the Second- and Third - Level Subcategories forEach First-Level Category of the Type-of-Outcome Dimension 7

Table 5 Taxonomy of Times to Consider for Collecting UndergraduateStudent Outcomes Data Using Survey Questionnaires 8

LIST OF FIGURES

Figure 1 The Audience Sheets of the Political Science Department CurriculumCommittee at Alpha College After They Had Completed Step C 18

Figure 2 The Audience Sheets of the Political Science Curriculum Committeeat Alpha College After They Had Completed Step 7 19

Figure 3 Tile Audience Sheets of the Political Science Curriculum Committeeat Alpha College After They Had Completed Step 9 21

Figure 4 The First Page of the Outcome List for Students Majoring inPolitical Science That Had Been Developed by the PoliticalScience Curriculum Committee at Alpha College at theCompletion of Step 11 23

Figure 5 The First Page of the Outcome List for Students Majoring inPolitical Science That Had Been Developed by the PoliticalScience Curriculum Committee at Alpha College at theCompletion of the Final Step, Step 13 25

Figure 6 Cautions and Potential Problems with the NCHEMS OutcomesStructure That Were Mentioned by Various Interviewees in theCASC/NCHEMS Project 33

vii

PREFACE

All the major activities conducted by administrators, faculty, and student personnel workers inpostsecondary education institutions are, presumably, aimed toward bringing about certainkinds of "educational outcomes." The activities of planning, budgeting, management,evaluation, research, scholarship, instruction, and other developmental activities for students,and even staff development, ail have educational outcomes as a primary focus. Other groupswithin an educational Institutionsuch as, the secretarial staff and the computer operatorsare also expected to contribute to various educational outcomes.

In this "age of accountability" administrators and others have been especially concernedabout educational outcomes and their measurement. Postsecondary institutions are beingfaced with increasing pressures to define better "what they are all about," and to communicatebetter their purposes and goals to many different audiences. Institutions are also being calledon to provide factual evidence that they and their programs are providing the benefits that wereintended, and that these outcomes are being produced in a cost-effective manner. Because ofthe important needs in the area of educational outcomes, NCHEMS has devoted majorresources and efforts over the last five years toward developing procedures and materials tohelp postsecondary education institutions collect and use outcomes Information. In 1973 anInventory of Outcome Variables and Measures was published. In 1975, the Outcome Measuresand Procedures Manual was published, giving standard definitions and alternative dataacquisition procedures for approximately 50 measures that different postsecondary educationdecision makers (including state legislators) said were most important for their needs. Themost recent development is the NCH EMS Outcomes Structure.

This document presents an overview of the NCHEMS Outcomes Structure, and then providessome basic procedures for using the Structure that can have important, practical Impacts foran institution, program, or other use context. The Outcomes Structure Is a system designed toserve as a framework for organizing information In an effective way for purposes ofclassification, analysis, and decision making. It rests upon significant bodies of previous work,its practical application has been tested in a preliminary way at one public university and eightprivate colleges, it has been formally reacted to by a diverse group of over 100 people, and it isintended to continue its development as time passes.

The current document is designed specifically for those who are trying to introduce and orientthe people on their campus to the Outcomes Structure and its potential, and to train people inhow to apply or use it. It also is expected to be of day-to-day use to administrators, faculty, andother decision makers in postsecondary education institutions. In addition, many of thegeneral procedures presented here are believed to have some applicability to needs of programand resource planners at other levels within postsecondary education, for example, planners atthe state and federal levels and planners for groups or systems of institutions.

This is the second of two documents presenting the NCHEMS Outcomes Structure. The firstone gives an in-depth discussion of the Structure, Its rationale, its development, and itsunderlying empirical support.* It was prepared primarily for use by those wanting a detaileddescription and conceptual discussion of the Outcomes Structure and an In-depthpresentation of the concept of an "educational outcome," including researchers andadministrators or faculty specializing in outcomes. A number of those reviewing drafts of both

Oscar T Lenning, Yong S Lee, Sidney S Micek, and Allan L Service, A Structure for the Outcomes of Postsecondary Education (Boulder, Colo NationalCenter for Higher Education Management Systems, 1977)

ix

9

PREFACE (Continued)

documents expressed the opinion that many to whom this current document is aimed, wouldalso find the first document of interest and use as a supplemental source, and that it should beread prior to reading this document.

The procedures discussed in this document are based on logic and limited, preliminary tests.Although these preliminary tests obtained a number of very positive results, the real value ofthe Structure will not be known until it receives widespread application. As experience withusing the Structure in many different contexts occurs, a revised version of this document willbe published to report any insights gained and any new procedures or techniques found to beuseful.

Many of the people acknowledged in the in-depth conceptual document referred to above alsoreviewed an earlier draft of this document, provided helpful suggestions, and in other wayscontributed to the development of this document. Appreciation is hereby again extended tothem, with special acknowledgment and thanks going to George Barton, Cathleen Bower,Edward Cooper, Jean Endo, Sidney Micek, Edward Myers, Robert Passmore, NancyRenkiewicz, David Witmer, and participants in the CASC/NCHEMS projectincluding TaskForce members who made campus visits and the campuses where the Structure was tried out.

x

10

INTRODUCTION

Having a wide variety of outcomes information without any structure is analogous topossessing a file cabinet in which the contents are arranged randomly. Similarly, withoutagreement on a common language and context for outcomes, information about outcomes canbe misunderstood or be seriously misleading, and it becomes difficult for institutional officialsto communicate succinctly how their institution and program differs from its counterparts. Aneffective outcomes structure provides such a common language.

The sheer volume of potential outcomes information in postsecondary education poses asignificant barrier to using outcome Information in institutional planning and management.Although a number of attempts have been made to develop structural systems for organizingoutcomes information, they have been Inadequate for meeting the practical need forcomprehensiveness and institution -wide use in postsecondary education. Because of thelimitetions in the state of the art and the important planning and management needs that existin this area for postsecondary education, an NCHEMS project was conceived and executedthat aimed to: (1) develop a system that could effectively organize information about the fullrange of potential postsecondary education outcomes for purposes of classification, analysis,and decision making, and (2) develop an extensive set of standard definitions for the differenttypes of educational outcomes. After more than two years of concentrated work, an OutcomesStructure resulted that reviewers outside NCHEMS feel has potential for being of use topostsecondary education decision makers in such areas as identifying educational needs,developing goals, translating goals into more concrete objectives, evaluating institutions andtheir programs, and so forth. Furthermore, preliminary tests of the Structure at one publicuniversity and eight small private colleges were quite encouraging, as reported in the basedocument that gives in -depth information about the Structure, its development, and itsunderlying empirical support (Lenning, Lee, Micek, and Service, 1977). Nevertheless, it isexpected that improvements will be made in the Structure as a result of its use within differenttypes of institutions and various use contexts within those institutions.

The purpose of this document is specifically to provide guidance for those wishing to orientpeople to the Structure and train them in Its use. The first three parts provide information aboutthe Structure and its use which should also be helpful to administrators, faculty, and others oncampus who wish to try using the Outcomes Structure in a practical manner. Part I presents anoverview of the Structure and Its components. Part II describes in some detail the specificpractical uses perceived for the Structure in postsecondary education planning, management,evaluation, and other contexts. Then, Part III lists step-by-step procedures for accomplishingthree basic processes that allow the Structure to be applied as indicated in Part II. Finally, PartIVwhich is not aimed at users directly, as are the first three partsassists trainers and otherswho wish to orient people on their campus to the Structure and itsuse.

PART IAN OVERVIEW OF THE NCHEMS

OUTCOMES STRUCTURE

Postsecondary education outcomes are the endresults of the processes that oci.; withinpostsecondary education institutions andprograms. They include both the direct results ofthese processes and any short-term and long-term consequences of those direct results.Furthermore, some of them may be intendedoutcomes while others are unintendedandwhile outcomes are generically neutral, peoplewill be attaching positive and/or negative valuesto them. In using the NCHEMS OutcomesStructure, information about outcomes isorganized according to three dimensions: (1) thepersons, groups, or things that receive and/orare affected by the outcome(s) or concern, or thatare intended to receive or be affected by it;(2) whether or not the outcome involves a changein statusmaintenance versus changeand thebasic, specific entity that is maintained or

changed; and (3) when the outcome occurs or isintended or expected to occur. These dimensionshave been named "audiei 3," "type-of-outcome,"and "time," respectivel;, and each (except fortime) has been assigned a number of specificallydefined categories and subcategories. it isintended that those dimensions and categoriesnot of serious concern to a particular user of theStructure will be ignored, or will be modified anciadapted to better meet local needs andsituations. (The same is true of the variousprocedures for using the Structure that aredescribed later in this document.)

The Outcomes Structure is illustrated graphicallybelow. For keeping straight the three dimensions,it may help to talk in terms of three Wswho,what, and when. Who is receiving or beingaffected by what outcomes and when?

AUDIENCEDIMENSION

(Who or what receivesor is affected by theoutcome, or is intend-ed to receive or beaffected by it?)

TYPE OF OUTCOME DIMENSION

(What basic entity is, or is intended tobe, maintained or changed?)

TIME DIMENSION

(When is the out-come expected tooccur, or when doesit occur?)

3

12

The Audience Dimension

Table 1 lists the major (first level of detail)categories of the "audience" dimension. Table 2shows subcategories (second-level categories)separately for each of the major categories. Forthose who need a shorthand for referring tospecific categories or for developing filingsystems for particular categories of outcomes in-formation, code numbers are also provided inTable 2 fcr each category and subcategory. (Thecoding scheme for the Structure is discussed indetail in Appendix A.) For many applications ofthe Structure, particular subcategories of thisdimension will need to be further subdivided intoeven finer categoriessuch as, dividing studentsin a program into those majoring in the programand those just taking courses in the program, orinto disabled studer ts and nondisabled students.Such "extending the Structure," as this additionalsubdividing of categories of Rny of the dimensions is called, depends on tile philosophy andpurposes of the user of the Structure and on thecontext in which the app!:nation is taking place

(for example, someone using it in curriculardevelopment will probably need different ad-ditional categories from someone using it inprogram administrative planning). Specificprocedures for extending the Structure are pro-vided in Part III of this document.

The Type -of- Outcome Dimension

Table 3 presents names, definitions, and code(lumbers for the five overall (first level of detail)categories of the "type -of- outcome" dimension.Table 4 provides additional levels of detail by in-dicating names and code numbers for thecategories and subcategories into which each ofthe five broadest categories has been divided.Standard definitions along with example outcomemeasures and indicators are provided in Appen-dix B for each category and subcategory iden-tified in Table 4. As with the audier,:..e dimension,the structure will often need to be extended on thetype-of-outcome dimension. For example, incurriculum development it may be desirable to

Table 1

MAJOR CATEGORIES OF THE "AUDIENCE" DIMENSION

10. Indiyidual/Grnup ClientsThis category refers to persons or groups of persons who aredirect clients of the postsecondary education unit of concern and/or their immediateassociates, such as family and relatives or peers.

20. Interest Based Communities;:his category refers to large groups of people that areidentified as enjties working toward a well-defined interest or mission

30. Geographic-Based CommunitiesThis category refers to large groups of peopledefined on the basis of functional territorial boundaries.

40. Aggregates of PeopleThis category refers to subpopulations of people distinguishedby particular characteristics that mar' hldicate common concerns, needs, or wants, butwho do not necessarily have a conown interest or mission, and therefore do notconstitute communities.

50. Other AudiencesExamples would be the natural environment that is affected byuniversity-sponsored research (which in turn would be expected to have impacts onaudiences such as individuals and communities) and populations of animals (such asthe animals affected by efforts to keep depleted species from becoming extinct or bythe development of veterinary medicines).

4

13

Table 2

SUBCATEGORIES OF THE "AUDIENCE" DIMENSION10 individual/Group Guano' This category (awe to persons or groups of persons who are direct clients of the postsecondary education unit of

concern and/or their immediate associates, such as family and relatives or peers

11 StudentsIndividuals or group* of Individuals who currently we enrolled In the program, institution, or system of postsecondary education

12 For Ar StudentsIndlviduels cr groups of Individuals who formerly were enrolled In the program, insritution, or system of postsecondaryeduadlon

13 Family and Relatives of Students or Former Students

14 Peers and 4ssociates of Students or Former Students

15 Faculty

15 Stall Other than Faculty

17 Other Individual/Group ClientsAn example would be en Individual who is none of the above but is served by an advisory service offeredby the college

20 interest -Based CommunitiesThis category refers to large groups that are Identified u entities working toward a well-defined interest ormission

21 Private Enterprise CommunitiesCommunities when a major purpoee is Miradal remuneration and profitfor example, corporations,smell businesses, and farmers

22 Assoc/10On CommunitiesCommunities where members belong on the basis of affiliation rather than employment, such as unions andprofessional societies

23 Government CommunitiesCommunities dulgrad to administer government regulations and services, such as oft, hall, state departmentof education, and legislative communities

24 Nongovernmental/Public Service Communities Other than the Institution Producing the Outcome Nonprofit service organizations, suchu schools, hospitals, welfare agencies, philanthropic foundations, colleges (other than the college producing the outcome), and reeearchorganizations.

25 Institution or Institutional Unit Producing the OutcomeThe postsecondary education restitution and/or units within that institution thatare perceived as the producer/ f acidtator of the outcomes) of concern

26 Other interest-Baud CommunitiesAn example would be an ad hoc coalition task force of representatives from two or more of the aboveantes.

30 0017r1PRio-Based CommunitiesThis ca, 0 Wens to large groups defined on the basis of functional territorial boundaries

31 Local CommunityA township, city, county, metropolitan wee, or other type of locality having particular boundaries It is not necessarilyrestricted to the lord or Jurisdictional boundary, but the functional one in which the Impact of the institution is (or should be) directly andPhysically felt Thy boundaries will vary with the Institution/program and outcome of concern

32 The State

33 A RegionAn aggregation of states or parts of states

34 The Nation

,15 An International C ,nmunity

36 Other Gocgraphie-Based Communities An example would be a research discovery that affects primarily people liv,ig in the coldestlatitudes, or when 4 snows heav,ly

40 Aggregates of PeopleThis category mien to subpopulations of people distinguished by particular chareetensbes that may indicate commonconcerns, needs, or wants, but who do not necessarily have a common Inrm.rt or mission, and therefore do not constitute communities

41 Ability Level SubpopulationsSubpopulations defined according to level of ability/proficiency on general intellectual functioning Orspecific skillsfor example, gifted, typical, disadvantaged. or skilled, semi-skilled, unskilled

42 Age Subpopulations

43 Educational Level Subpopulations

44 Income Level Subpopulatior _

45 Occupation Sub/mad/at/one

46 Physrcs/ Disability Cone Non Subpopulations

47 &CO Subpopulations

48 Sex Subeopulations

40 Other Such Aggregaars

50 Other ..dime's Examples would be the natural environment that Is affected by university-sponsored research (which in turn would beexpected to have impacts on audiences such u individuals and communities) and populations of animals (such as the animals affected byefforts to keep depleted species from becoming extinct or by the development of veterinary medicines)

5

14

divide "intellectual skills" (code number 2250) in-to the following categories of Bloom's Taxonomy(1956): ability to translate (code 22501), ability tointerpret (code 22502), ability to extrapolate (code22503), ability to apply (code 22504), ability toanalyze (code 22505), ability to synthesize (code22506), and ability to evaluate (code 22507).

One thing should bP made clear. No program orinstitution can hope to focus on outcomes within

all the categories of Table 4, or perhaps even onmany of the categories. Rather, this is a universeof possible outcome categories from which onemust pick and choose those that are most impor-tant, in the light of factors such as philosophy,resources available, and clientele needs. To try todevote primary effort toward accomplishing toomany types of outcomes could have disastrousconsequences.

Table 3

THE MAJOR CATEGORIES OF THE "TYPE-OF-OUTCOME" DIMENSION

CategoryCode Number

The Major "Type-of-Outcome"Category Names and Definitions

1000 Economic OutcomesMaintenance or change in economic characteristics andconditions of individuals, groups, organizations, and communities, e.g., ineconomic access, in economic mobility and independence, in economic security,and in income and standard of living.

2000 Human Characteristic OutcomesMaintenance or change in human makeup andcharacteristics (other than knowledge and understanding) of individuals, groups,organizations, and communities, e.g., aspirations, competenceand skills, affectivecharacteristics, perceptual characteristics, physical and physiological character-istics, personality and personal coping characteristics, recognition and certifica-tion, and social roles.

3000 Knowledge, Technology, and Art Form OutcomesMaintenance or change in theknowledge and understanding, technology, or the art forms and works possessedor mastered by individuals, groups, organizations, and communities, e.g.,discoveries and inventions, technical developments, syntheses and reformulationsof knowledge, new schools of thought in art and works created in those newtraditions, renovation of art works.

4000 Resource and Service Provision OutcomesMaintenance or change in the directresources and services (other than those included above) provided to individuals,groups, organizations, and communities, e.g., providing facilities, events,advisory assistance, analytic assistance, teaching, health care, and leadership.

5000 Other Maintenance and Change OutcomesExamples would be: maintenance orchange in the format, arrangement, activity, or administrative operation of anorganization or institution; maintenance or change in the aesthetic/cultural levelof the local community; maintenance or change in family or community activities,practices, and traditions.

6

15

Table 4

CODED LISTING OF THE SECOND- AND THIRD-LEVEL SUBCATEGORIESFOR EACH F!RST-LEVEL CATEGORY OF THE TYPE-OF-OUTCOME DIMENSION°

CategoryCode Number Entity Being Maintained or Changed

CategoryCode Number Entity Being Maintained or Changed

1000 ECONOMIC OUTCOMES

1100 Economic Access and Independence Ot.tcones1110 Economic Access1120 Economic Flexibility, Adaptability, and Security1130 Income and Standard of Living

1200 Economic Resources and Costs1210 Economic Costs and Efficiency1220 Economic Resources (including employees)

1300 Economic Production1310 Economic Productivity and Production1320 Economic Services Provided

1400 Other Economic Outcomes

2000 HUMAN CHARACTERISTIC OUTCOMES (continued)

2760 Power and/or Authority2770 Job, School, or Life Success2780 Other Status, Recognition, and Certification Outcomes

WOO Social Activities and Roles2810 Adjustment to Retirement2820 Affiliations2830 Avocations' and Social Activities and Roles2840 Career and Vocational Activities and Roles2850 Citizenship Activities and Roles2860 Family Activities and Roles2870 Friendships and Relationships2880 Other Activity and Role Outcomes

2900 Other Human Characteristic Outcomes

2000 HUMAN CHARACTERISTIC OUTCOMES

2100 Aspirations2110 Desires, Alms, and Goals2120 Dislikes, Likes, and Interests2130 Motivation or Drive Level2140 Other Aspirations' Outcomes

2200 Competence and Skills2210 Academic Skills2220 Citizenship and Family Membership Skills2230 Creativity Skills2240 Expression and Communication Skills2250 intellectual Skills2260 interpersonal, Leadership, and Organizational Skills2270 Occupational and Employability Skills2280 Physical and Motor Skills2290 Other Skill Outcomes

2301, Morale, Satisfaction, and Affective Characteristics2310 Attitudes and Values2320 Beliefs, Commitments, and Philosophy of Life2330 Feelings and Emotions2340 Mores, Customs, and Standards of Conduct2350 Other Aff.rctive Outcomes

2400 Perceptual Characteristics2410 Percr-otual Awareness and Sensitivity2420 Perception of Self2430 Perception of Others2440 Perception of Things2450 Other Perceptual Outcomes

2500 Personality and Personal Coping Characteristics2510 Adventurousness and Initiative2520 Autonomy and Independence2530 Dependability and Responsibility2540 Dogmatic/Open-Minded, Authoritarian /Democratic2550 Flexibility and Adaptability2580 Habits2570 Psychological Functioning2580 Tolerance and Persistence2590 Other Personality and Personal Coping Outcomes

2600 Ph ysical and Physiological Characteristics2610 Physical Fitness and Trails2620 Physiological Health2630 Other Pt 'cal or Physiological Outcomes

2700 Status, Recognition, and Certification2710 Completion or Achievement Award2720 Credit Recognition2730 image, Reputation, or Status2740 Licensing and Certification2750 Obtaining a Job or Admission to a Follow-up Program

3000 KNOWLEDGE, TECHNOLOGY, AND ART FORM OUTCOMES

3 ;CO General Knowledge and Understanding3110 Knowledge and Understanding of General Facts and

Terminology3120 Knowledge and Understanding of General Processes3130 Knowledge and Understanding o' General Theory3140 Other General Knowledge and Understanding

3200 Specialized Knowledge and Understanding3211, Knowledge and Understanding of Specialized Facts

and Terminology3220 Knowledge and Understanding of Specialized

Processes3230 Knowledge and Understanding of Specialized Theory3240 Other Specialized Knowledge and Understanding

3300 Research and Scholarship3310 Research and Scholarship Knowledge and

Understanding3320 Research and Scholarship Products

3400 Art Forms and Works3410 Architecture3420 Dance3430 bate and Oratory3440 Drama34b0 Literature and Writing3460 Music3470 Painting, Drawing, and Photography3480 Sculpture3490 Other Fine Arta

3500 Other Knowledge, Technology, and Art Form Outcomes

4000 RESOURCE AND SERVICE PROVISION OUTCOMES

4100 Provision of Facilities and Events4110 Provision of Facilities4120 Provision or Sponsorship of Events

4200 Provision of Direct Services4210 Teaching4220 Advisory and Analytic Assistance430 Treatment, Care, and Referral Services

4240 Provision of Other Services

4300 Other Resouice and Service Provision Outcomes

5000 OTHER MAINTENANCE AND CHANGE OUTCOMES

5100 Aesthetic-Cultural Activities, Traditions, and Conditions

5200 Organizational Format, Activity, and Operation

5300 Other Maintenance and Change

a The fourth level categories, Into which any of the categories listed here can be divided, are "maintenance" (a fourth digit of "1") and "change"(a fourth digit of "2")

7 16

The Time Dimension

No formal categories and subcategories wereassigned to the time dimension because no timecategories would apply across all audiencecategories. Furthermore, even at the broadestlevel, the time categories of interest would be ex-pected to vary depending on the philosophy of theuser of the Structure and the particular need ill hisor her context. Time is so important in planning,however, that it was still made a dimension of theStructureand the categories to use were leftcompletely up to the user.

To illustrate how specialized the taxonomy for thetime dimension may need tc be, Ms. Jean Endo ofthe University of Colorado at Boulder reported tothis writer on recommended times during the yearfor collecting student outcomes and related datausing survey questionnairesas based on theirexperiencesand when during the students'college and alumni career different data shouldbe collected. These can be transformed into a"Taxonomy of Times to Consider for CollectingUndergraduate Student Outcomes Data UsingSurvey Questionnaires," as shown in Table 5.

Table 5

TAXONOMY OF TIMES TO CONSIDER FOR COLLECTING UNDERGRADUATESTUDENT OUTCOMES DATA USING SURVEY QUESTIONNAIRESa

100 Data from Lower Division Students110 Data from Freshman Students

111 Data Collected Prior to Fall Registration112 Data Collected One Month Following the First Day of Classes in the Fall113 Data Collected One Month Prior to First Semester Final Exams114 Data Collected One Month After Spring Semester Classes Begin115 Data Collected One Month Prior to Spring Semester Final Exams116 Other, for Example, in the Middle of the Summer Term

120 Data from Sophomore Students121 Data Collected Prior to Fall Registration122 Data Collected One Month Following the First Day of Classes in the Fall123 Data Collected One Month Prior to First Semester Final Exams124 Data Collected One Month After Spring Semester Classes Begin125 Data Collected One Month Prior to Spring Semester Final Exams126 Other, for Example, in the Middle of the Summer Term

200 Data from Upper Division Students210 Data from Junior Students

211 Data Collected Prior to Fall Registration212 Data Collected One Month Following the First Day of Classes in the Fall213 Data Collected One Month Prior to First Semester Final Exams214 Data Collected One Month After Spring Semester Classes Begin215 Data Collected Ore Month Prior to Spring Semester Final Exams216 Other, for Example, in the Middle of tne Summer Term

220 Data from Senior Students221 Data Collected Prior to Fall Registration222 Data Collected One Month Following the First Day of Classes in the Fall223 Data Collected One Month Prior to First Semester Final Exams224 Data Collected One Month After Spring Semester Classes Begin225 Data Collected One Montn Prior to Spring Semester Final Exams226 Other, for Example, in the Middle of the Summer Term

300 Data from Alumni310 Data Cullected at Graduation320 Data Collected One Year after Graduation330 Data Collected Five Years after Graduation340 Other, for Example, Data Collected 20 Years after Graduation

400 Cther, for Example, Data Collected after Students Have "Dropped Out"

aThese categories and subcategories are based specifically on the data collection experiences of sdf I in the Office of the Vice Chancellor for AcademicAffairs at the Univeraity of Colorado, Boulder Therefore, they may not be entirely appropriate for ether postsecondary institutions

8 17

PART IITHE POTENTIAL USES

OF THE OUTCOMES STRUCTURE

A number of potential uses have been perceivedfor the Outcomes Structure by various people whoreviewed it, some of which have been tested in apreliminary way and found promising. This sec-tion will discuss these proposed uses and provideexamples.

Only time and widespread use in a variety of dif-ferent institutional and other settings will tellwhether the so-far promising poteWial of theStructure will "bear fruit." Such use will alsoprobably suggest modifications for future ver-sions of the Structure, and adaptations in ap-plication procedures that need to be made for dif-ferent types of users and for different types of in-stitutions. The procedures in this document areonly a start, and feedback from users of the Struc-ture is solicited.

1. CommunicationIt is felt that if the terms and their definitionsin the Structure are agreed to by a large num-ber of planners and managers in postsecond-ary educationand those that are not agreedto are modified accordingly in future editionsof the Structureit can serve as a usefulcommon language where the outcome con-cepts and relationships remain constant fromone use of outcomes information to another.In other words, the aim was to develop a"workable language" for more effective com-munication about postsecondary educationoutcomes. Having kept the terms, definitions,and the order of placement as neutral andvalue-free as possible should facilitate such apurpose.

If the Structure terms, definitions, andrelationships do gain widespread acceptance,they should help institutional administratorsand faculty to communicate more clearly andsuccinctly about the goals and outcomes ofthe institution and its programs to many im-portant publicssuch as legislators, com-munity leaders and citizens, employers, foun-

dations and other funders, alumni, andprospective students and their parents. In this"age of accountability," such communicationis essential in order to maintain public supportand to attract the needed students. For example, in a recent prospective student infor-mation needs assessment study coordinatedby this author, involving questionnaireresponses by several thousand relevantpeople and interviews with over 500, clear andconcise information about student outcomesranked very high as information that prospec-tive students need.

The Structure can be just as useful for com-munication about outcomes within the in-stitution. For example, pairs of members of aStudent Learning Outcomes Task Force fromthe Council for the Advancement of SmallColleges (CASC) visited four small privatecolleges to interview administrators, facultymembers, and students about the specificlearning activities emphasized in theirprograms. After discussing activities in eachinterview, categories of the NCHEMS Out-comes Structure were introduced in -n attemptto have the interviewees link their activitiesto outcome areas. The Structure was founduseful for this purpose, but the interviewersalso noted in talking to faculty from differentdisciplines that the Structure served as a"taxonomic device for communicating out-comes across disciplinary lines."

2. StimulationIn the preliminary tryout of the Structure,there was much evidence that the Structurecan stimulate administrators and faculty tostart thinking more about what they are tryingto accomplish for different groups of studentsat their institution or in their programs, and toconsider outcomes in a more systematic andconcrete manner than they have before. Forexample, in the CASC project mentionedabove, the interviewers had a difficult time

9

18

switching thc conversation from the "comfort-able" topic of activities to the "uncomfort-able" topic of outcomes, but the Structurecategories helped them to do so. As the syn-thesizer of the notes from the various inter-viewers, Dr. Edward Lundin stated, "Withoutthe Outcomes Structure categories, thefaculty members could have focused on ac-tivities without any systematic examination ofthe larger purposes."

The CASC Task Fo1 ce conducted anotherproject involving visits to four other smallcolleges, this time jointly with NCHEMS, thatinvolved a two-person CASC/NCHEMS teamgoing on campus to orient administrators,faculty, and students to outcomes conceptsand the Structure through a half-dayworkshop, and then conducting interviewswith individuals in each of those groups forone and one-half days. In the interviews,respondents were first asked to look at theaudience dimension categories and then topoint out those they felt were important toconsider in planning outcomes for the in-stitution or for the particular program ofspecial concern to them. Then they were askedwhich, if any, student subgroups in theirprogram or area of concern had importantenough outcome needs unique to them thatthey should be considered separately in plan-ning for the program or area.

Next the respondents rated the detailedcategories of the type-of-outcome dimensionaccording to which were considered to beessential, which were important, and whichwere less than important as student learningoutcomes for the important (for planning)groups of students in the institution or theprogram of special concern to them. For oneor two of the categories selected as "essen-tial," they were asked to list several specific,observable outcomes that were or should beemphasized for those students, and measuresor indicators that they felt would constituteconcrete evidence as to whether or not thosespecific outcomes occurred. Then they wereasked to identify the specific student ex-periences and activities that they felt wouldmost contribute to the realization of thosepriority learning outcomes that had been iden-tified. Finally, they were asked their Im-

pression of the utility of the Structure now thatthey had gone through this experience.

All of the students, most of the faculty mem-bers outside of education, and many of theadministrators reported that they had neverbefore really thought like this about what theirinstitution or program was trying to accom-plish for its students, and the majority of thesepeople reacted positively to the questionabout utility of the Structure. As one facultymember stated (paraphrasing him), "I'm surethat I have always had in the back of my mindwhat I specifically wanted to accomplish formy students but have never been able tostate it in formal, concrete, and systematicterms like this until now." Another stated, "Ithas made me stop and think about what we aretrying to accomplish, and it is making the ob-jectives in the back of my mind conscious."And as a director of development stated(paraphrasing again), "I now see many con-crete things I never before thought to mentionto foundations, employers, and others in thecommunity that our institution is trying to ac-complish for its students." Other people feltthat the Structure could stimulate people tofocus on the "why of outcomes," to "raisequestions," to "think in a concrete manner,"and to "provide stimulation to guidedirection." Clearly the Structure would seemto have great potential as a device forstimulating a variety of people to think moreabout the outcomes they want accomplished,and in a more systematic and concretemanner than they have previously. This, in turn,could lead to useful and stimulating groupdiscussion about the outcomes desired for aprogram or institution, and about alternativestrategies for bringing them about.

3. Planning, Management, and EvaluationThere are three "basic processes" that involvethe direct application of the Outcomes Struc-ture. These processes in turn can serve impor-tant functions in postsecondary educationplanning, management, and evaluation. Step-by-step instructions for these processes areprovided in Part II of this dicument. In thediscussion to follow, the processes will bereferred to as Process 1, Process 2, andProcess 3. Their respective descriptions are:

10

19

PROCESS 1 Generating lists of priority outcomesthrough application of the OutcomesStructure

PROCESS 2 Classifying outcomes through applicationof the Outcomes Structure

PROCESS 3 Storing and retrieving outcomes informationthrough application of the OutcomesStructure.

Some functions in which these processes maybe useful are:

A. Needs Assessment. In this "time of ac-countability," the impertance of knowingthe outcome wants and needs of prospec-tive clientele groups is increasingly beingrecognized by postsecondary educationplanners and managers. Needs assess-ment, as this process is commonly called,involves (among other activities) surveyingthese client groups and various concernedothers to try to ascertain needs, wants,and expectancies. The Structure couldsuggest areas of needs assessment focusto consider. Furthermore, many of theitems on such surveys are stated inoutcome terms, and the Structure providesa comprehensive universe of outcomecategories that can be used to stimulatethinking about specific outcomes itemsand to ensure that all areas of potential im-portance are considered for coverage in thequestionnaire. A similar universe ofcategories was found to be useful indeveloping the prospective student infor-mation needs assessment questionnairereferred to previously. Carefully thought-out lists of outcomes and classes of out-comes for different audiences resultingfrom either Process 1 or Process 2 can beused to determine outcome items for thesurvey forms.

B. Goal Identification. Another importantplaning function is determining anddeveloping priorities for-goals and objec-tives of the institution or institutional unitof concern. Needs assessment results areuseful input to such a goal identificationprocess, and an application of Processes 1and 2 to needs assessment has already

been mentioned. Some goals (called out-come goals) focus on the outcomes thatare desired and intended for the institution,program, or other unit of concern Others,which are often related to outcome goalsand which some people consider also to beoutcomes because they are a desiredresult of institutional processes, focus onsuch things as levels of resources and par-ticular environmental, climate, and processconditions or levels (process goals). Goalidentification involves taking the broadmission statements of the institution or themissions of institutional units and trans-forming them into the more specific goalsthat are appropriate as based on needsassessment results.'

The "type-of-outcome" dimension of theOutcomes Structure is comprehensive inits coverage of types of outcomes, ishierarchical in nature, and has severallevels of detail. It has broad outcomes atthe first level of detail, and increasinglyspecific outcomes at each succeedinglevel of detail. Therefore, since thedefinition of an "outcome goal" is "desiredoutcome," the Structure is appropriate foruse in the goal-Identification task. (Otheraids in this process are also available, forexample, the Educational Testing ServiceInstitutional Goals Inventory.) And, in ef-fect, that is exactly what Process 1 is allabout. For each important audience of theinstitution or Institutional unit of concern,the process leads to a list of desired,specific priority outcomes. The inter-viewees in the joint CASC/NCHEMSproject cited earlier were in effect iden-tifying their outcome goals for their in-stitution or program through an ab-breviated version of Process 1, but they didnot explicitly involve an integration of for-mal mission with needs-assessment data.

C. Goal Assessment. After a Ilst of goals foran institution or program has been for-mulated, it is desirable to test its adequacy

For a more thorough discussion of the goal Identification process that Includesa discussion of Integrating mission with needs assessment data and of consensusseeking processes concerning goals, see 0 T Lenning and S S Micek, "Definingand Communicating Institutional Mission/Role/Scope and Priorities The Needs ofDifferent Types of Postsecondary Institutions," paper presented at the annualmeeting of the American Educational Research Association, San Francisco, April1976 (ERIC document ED 121378)

11

20

ur coverage. For outcome goals, one way ofdoing this is to use Process 2 to classify allcf the goats in the list, and then to apply (orreapply) Process 1 to those areas of theStructure that seem to be important but towhich none (or few) of the goals areassignedto determine the "holes" or"gaps."

Staff at the University of Colorado atBoulder over a period of years, throughsurveys and interviews of important clien-tele groups, had developed several lists ofintended outcomes that they felt confidentwere comprehensive for their needs.Therefore, NCHEMS hired one of the coor-dinators of that project to evaluate thecoverage of those lists using the OutcomesStructure in the manner described above.To the staff's surprise they found thatseveral outcome areas that they con-sidered to be quite important had beenoverlooked in the development of theirlists. Thus, they are revising their lists ac-cordingly, and also their freshman studentquestionnaire, which was based in part ontheir lists.

Spring Arbor College, one of the partici-pants in the joint CASC/NCHEMS project,used a different method of applying thestructure to achieve similar results. Theytook lists of outcome goals for the institutionand separately for each of a number ofprogramssome of the goals had beendeveloped through use of the InstitutionalGoals Inventory and others had beendeveloped in an open-ended mannerandclassified them into the five broad (first-level of detail) categories of the OutcomesStructure Type-of-Outcome dimension.Then a goal-profile graph that indicated thenumber of goals assigned to each categorywas drawn separately for the institutionand each program. For several programs,they discovered type-of-outcome categorieshaving few goals assigned to them, whichon reflection they felt should have hadmany goals, while one or more other cate-gories ended up with larger percentages ofthe goals than they had anticipated.

D. Goal Translation. In order to determine the

measures or indicators on which datashould be collected for evaluation of theinstitution andlor program, goals must bl?.translated into observable, measurableobjectives. If Process 1 is carried to itsmost detailed level (specific, concrete, ob-servable outcomes), and appropriate in-dicators or measures and desired perfor-mance levels on them are determined, oneiias reached the objective levelwhich isone thing the CASC/NCH EMS project wasattempting (successfully, considering thetime available in each interview) to do. Useof the sample indicators and measureslisted in the right-hand column of AppendixB may be helpful to stimulate thinkingabout what measures, and what objectivelevels on those measures, would be mostappropriate for the goals that were select-ed. A future project at NCHEMS willdevelop relatively comprehensive lists ofmeasures and indicators for selectedcategories of the Structure, from whichplanners and managers can choose thosemost appropriate for them.

E. Allocation and Utilization Planning andManagement. One important administrativeplanning function if_ 'esource allocationdecisions about how the available money,staff, and other resources should beapplied. Goal identification and translationresults are important considerations inmaking allocation decisions. So is the timewhen the outcomes are expected to occurand how long they are expected to last,information that can be determined inProcess 1 and Process 2.

Process 1 or Process 2 may also aid inother planning decisions, such as programactivity planning and curriculum develop-ment. (When interviewees in the CASCINCH EMS project were asked to specify themost important student experiences andactivities to use for the different priorityoutcomes they had specified, they were ina planning phase.) And that planning inturn provides general guidance for day-to-day operations. Another planning functionmentioned under goal translation is whatmeasures or indicators to use for collec-tion of evaluative data, and performance

12

21

levels. If certain outcomes are not expect-ed but seem to have some probability c foccurring, data should also be collected forthem. A related planning decision is whento collect various evaluative data (amanagement function), and the "time" in-formation gathered in Process 1 or 2should aid this decision.

In the joint CASC/NCHEMS project inter-views, use of the Structure for planning atlevels within the institution that had notbeen considered when designing the Struc-ture were emphasized by some. For ex-ample, a number of faculty members felt con-fident that use of the Struct. -e could bequite helpful in developing courses withinthe curriculum. Furthermore, severalstudents and one student personnel ad-ministrator expressed the opinion that theStructure could be useful to students inplanning what they want from their collegecareers.

F. Evaluation. Determining whether the in-tended and planned outcomes actually oc-curred is an evaluation function, as is thedetermination of whether unintended out-comes have occurred. Because it relatesback to the outcome goals (intended out-comes) and the intended quality of in-stitutional and program processes forachieving those goals, success in theevaluation phase depends greatly on thegoal-setting and planning phases. The datafor the measures or indicators selectedduring the planning process, with the helpof Process 1 or Process 2 and Appendix B,have presumably been collected at ap-propriate times (times also decided duringthe planning process through inputstimulated by the Structure "time" dimen-sion). If so, the actual outcomes and out-come levels can be compared to thedesired outcomes and outcome levels, andany unintended outcomes that are observedmay be assessed. Then the evaluationresults and interpretations can be fed backto guide new goal setting and planning. Ifdata are being collected for a large numberof measures pertaining to thousands ofstudents at several or many different timesduring their college careers, computerized

storage of the outcomes information, usingProcess 3, becomes essential. Inter-pretations of the discrepancies found be-tween each important desired outcome andits associated actual outcome can be fedback to the planning process to aid indecisions about modification, revision,replacement, or termination of activities,resources, direction of effort, and so forth.

The evaluation phase involves much morethan just comparing actual outcomes to in-tended outcomes. It includes data andanalyses (plans for which should also havebeen a part of the planning phase) to an-swer questions such as the following:Does the outcome result from this post-secondary education institution or in-stitutional unit? How do we know? Haveresources, including those embodied in theentering students, been accounted for?What is the value of the outcomes as com-pared to their cost?

Evaluation is closely related to the functionmentioned earlier of communicating withfunders and other concerned publics. (Theimage of the institution and its programs iscrucial in attracting funds, students whowill be happy there, and a goo l staff.) It ishoped that more precise evaluative dataabout the institution and its programs willbe available from application of the Struc-ture. With this knowledge, and moreprecise terminology, administrators andfaculty may be able to communicate moresuccinctly what their institution is accom-plishing for individuals and society, andhow their institution and its programs dif-fer from their counterparts.

4. ResearchAll three of the "basic processes" outlined inthe preceding subsection may be useful inboth basic and applied outcomes research.Process 1 results can suggest gaps in thecomprehensive set of outcomes that need tobe researched, either for specific types of out-comes or large areas of outcomes. It can also,when considered with the sample measuresand indicators presented in Appendix B,suggest outcomes and classes of outcomeswhere the development of new measures

13

22

should be especially focused. Results ofstudies found in the literature, pools of out-comes questionnaire items used in variousstudies, pools of outcome measures that havebeen collected, literature references of out-come studies, data collected in large-scalestudies, and so forth, can be classified usingProcess 2. They they can be stored in a com-puter storage bank within locations keyed tothe most detailed classes, and be madeavailable for retrieval in aggregated ordisaggregated form at the proper time for anyanalyses desired (Process 3). In addition to thethree basic processes, the relationshipsamong the various classes and subclasses ofthe Structure itself can provide researchhypotheses that should be tested out. Onereviewer of the Structure reported that hisexamination suggested to him an extensiveoutcomes research agenda that needs to becommenced. In addition to the researchquestions implied above, other possible

questions stimulated by the Structure includethe following: What are the real (as opposed tofolklore) outcomes of postsecondary educa-tion? What outcomes naturally go togetherform clusters or constellations? Whichoutcomes last longest? Which clusters ofoutcomes are most valuable? What indicatorsand measures are most appropriate for thedifferent categories of the Structure, and howcan we go beyond opinions to observing factsand "hard data?" Which processes yieldwhich outcomes for whom? What are therelationships of costs to value added for dif-ferent types of outcomes? What factors im-pact on outcomes of different types, and howcan we understand such relationships? Howdo the outcomes of institutions, programs,and other postsecondary education settingsvary for different audiences? Which outputs(direct outcomes) lead to which impacts (in-direct outcomes)?

14 23

PART IIIGUIDELINES FOR USING THE

OUTCOMES STRUCTURE TO ACCOMPLISHTHREE BASIC PROCESSES

The Outcomes Structure is a framework for use inorganizing and classifying information abouteducational outcomes. It allows placement ofoutcomes along three basic dimensions: audi-ence, type-of-outcome, and time. Unless one hasdecided who or what is to receive or be affectedby the outcome of interestthe audiencedirec-tions for a course of action are missing. The sameis true concerning whether we are striving tomaintain or to change characteristics and con-ditions, and specifically what it is we are trying tomaintain or changethe type-of-outcome.Similarly, planning related to educational out-comes can be adversely affected if we give nothought to when the outcomes should occur. Andevaluation may not be useful if evaluative data arenot collected at the proper time.

This section describes three basic processes foruse of the Structure: (1) generating useful lists ofdesired or expected priority outcomes for an in-stitution or institutional unit, (2) classifying theoutcomes or outcomes information in availablelists, for an institution or institutional unit, anddiscovering where there are important "holes" or"gaps," and (3) storing and retrieving outcomesinformation. As discussed in Part II, theseprocesses can be applied in various ways to aidimportant decision processes in the areas ofneeds assessment, goal identification, goalassessment, goal translation, planning andmanagement, and evaluation. Furthermore, theycan be applied at the institution-wide level, at theprogram or departmental level, and at other levelssuch as planning courses, student planning, statesystem planning.

Process 1: Generating Lists of Priority OutcomesDeveloping a listing of the desired outcomes isespecially important for planning, whether it beplanning for a course, planning for a curricularprogram, planning for an academic department,

planning for an institution, or planning for asystem of institutions. (As mentioned earlier, italso is extremely important for tasks such asevaluation, but these tasks derive and extendfrom the planning function.) In addition, thelisting should give some indication of priority: forexample, which of the outcomes are essential,which are valuable, and which have little or novalue. Process 1 consists of applying the Out-comes Structure to develop such a listing.Decisions (choices) will be made at each of thesteps that follow. "Who decides" will not bediscussed here it depends on local factors suchas organizational arrangement, administrativestyles, and customary procedures. It Is stronglyrecommended, however, that some person haveclear authority and assigned responsibility formaking these decisions and that well-thought-outdecision procedures be developed. In addition, itis important that representatives of each majorconcerned group on campus, including students,be involved through providing input for thesedecisions. There are a number of ways this can bedonefor example, "town hall" meetings, com-mittees, interviews, surveys, card sorts, or theDelphi technique may be used. It also would bedesirable to get input from concerned off-campusgroups such as community leaders, alumni, andmembers of the community at large. Although theuser may wish to modify the procedures fromthose that are suggested, the following aregeneral procedures that evolved from preliminarytests of the Structure. But let it be made clear,local options are encouraged, whether they bemodifications In ,ae procedures for using theStructure or in the content of the Structure itself.For example, in the joint CASCINCHEMS visits tocolleges, time was short so some of the stepswere omitted and all others abbreviated.

Outcomes for Whom? Before beginning to thinkabout the types of outcomes or about specific

15

outcomes, it is important to consider the audi-ence that is meant to receive benefits from or inother ways be affected by the academic in-stitution or the unit of concern within that in-stitution. There are many potential audiences (forexample, students and those who influencestudents, such as parents and counselors; gover-nors, legislators, mayors, and voters; and society,as represented by employers and buyers [that is,consumers] in the market place). The purpose isto choose those which should be of concern inplanning for outcomes. Too often in institutionalplanning, the target audiences for major intendedoutcomes are not formally specified, whichreduces the potential for tailoring the planned in-stitutional activities to fit the characteristics andsituation of the audience. The "audience" dimen-sion of the Structure was developed to facilitatedetermining the people, the communities, or theother entities specifically intended by the in-stitutional or program planners to receive or be af-fected by outcomes of the institution/institutionalunit. The primary categories of the audiencedimension are shown in Table 1 on page 4. (Asmentioned previously, cod^ numbers have beenassigned to the categories, and the codingscheme is discussed in Appendix A.) Note thatthere are five major (first-level) audiencecategories, one being an "other" category. Notonly do "other" categories make the Structuremore flexible and able to meet the needs ofunique institutions, but also they stimulate theuser of the Structure to ask whether there are ad-ditional audiences for the institution's outcomesthat should be considered.

STEP 1 Decide which of the broad audiencecategories in Table 1 are importantrecipients of the outcomes of the in-stitution or institutional unit of concern.

STEP 2 Look at Table 2 on page 5. For eachmajor audience category selected inStep 1, decide which of the sub-categories (second-level) in Table 2 areimportant recipients of the outcomes ofthe institution or institutional unit ofconcern.

STEP 3 It may be helpful when thinking aboutthe outcomes to be even more specificthan the categories listed in Table 2,

concerning who or what are the audieces of concern. Therefore, for eachaudience subcategory selected forfocus, it is suggested that specificsignificant group within the categorybe listed. Examples for the "students"category in Table 2 might be "studentswho entered out of high school" and"older students." (Instead of olderstudents it might be desirable to haveeven more specific groups, such as"veterans," "homemakers," "retrainees,"and "retired persons.") A set of examplesfor students that might be especiallypertinent for a curricular program are"students majoring in the program" and"nonmajors taking courses in theprogram." Examples of more detailedcategories for private enterprise com-munities might be "major corpora-tions in the community" (it could behelpful to list them if there are only a fewor to list the different types), "smallbusiness firms in the community ' (againa listing of different types could beuseful later), and "area farmers." Note inthe last set of examples that we in effectcombined two quite different second-order categories: "local community" and"private enterprise communities."

The lists of more specific audiencetypes generated for the various"audience" subcategories r concern inTable 2 can be used in se.eral ways,whichever seems more helpful to theperson using the Structure: (a) they canbe used as cues t) stimulate thinkingabout specific outcomes for the Table 2categories selected, (b) they can bemade into third-level categories of theaudience dimension to insure morespecificity in categorizing outcomes,and (c) some of them can be handled asin (a) above, and others as in (b) above.

STEP 4 The audience categories on which youwish to focus specific attention indeveloping lists of desired educationaloutcomes have been determined inSteps 1-3. Now print each of thesecategory names at the top of separatesheets of blank paper (one audience

16

25

category per sheet). In addition, for eachcategory, print near the top of the sheetany audience examples that wouldpresumably be helpful in stimulatingthinking about specific desired out-comes for that audience category (assuggested in a of Step 3).

What Outcomes? Now that you have a group ofsheets with all important audience categoryprinted at the top of each, you are ready to proceedto the "type-of-outcome" dimension of the Struc-ture. The purpose now is to determine which out-comes are important for each audience that hasbeen chosen as sufficiently important to mseparate consideration in planning for the in-stitution or program of concern. The type-of-outcome dimension identifies whether or not heoutcome involves a change in status (mainte-nance versus change), and the basic entity that ismaintained or changed (such as knowledge andunderstanding, skills and competencies, at-titudes and values, certification status, income,standard of living, social interactions, and artforms or expression).

STEP 5 Order the sheets according to whichaudiences should be focused on first inlisting specific outcomes, and whichlast. That way, if you run out of time, youwill have completed lists of outcomesfor what you consider to be the most im-portant or relevant audiences for the in-stitution/institutional unit of concern.Criteria that may be helpful in this or-dering process include such factors aswhether the primary impact is likely tobe direct or indirect (the institution oftenhas more control and opportunity to af-fect direct outcomes as compared to in-direct outcomes), relative numbers ofoutcomes expected for each group, howbasic and fundamental the outcomes foreach group seem to be, and the per-ceived importance of each group relativeto the others.

Some people will prefer to focus on theoutcomes for only one audience at atime. Others will find it useful and ef-ficient to focus on the outcomes formore than one audience at a time. It isextremely difficult to focus on outcomes

separately for more than two or threeaudiences at a time, however, based onexperiences in the CASC/NCHEMS jointproject.

STEP 6 The five major (first-level) categories ofthe type-of-outcome dimension, anddefinitions for each, are shown in Table3 on page 6. Across each of the "audi-ence sheets" selected in Step 5 list thetype-of-outcome categories from Table 3that you consider important for thataudience group. To illustrate, this stepis shown in Figure 1 on page 18 for apolitical science department curriculumcommittee at Alpha College (a fictitiouscollege). They had decided to limit theirfocus to two audience groups, "studentsmajoring in political science" and"other students taking political sciencecourses," and thus they had twoaudience sheets on which they wereworking.

STEP 7 Table 4 on page 7 lists the second- andthird-level categories for each first-levelcategory of the type-of-outcome dimen-sion, and definitions for all of thesesubcategories are given in Appendix B.Now go to the first audience sheet anddo the following:

a. For each first level "type" categorylisted on the audience sheet, jotdown on a scratch pad all second-level categories that your analyses(whether they be subjective or objec-tive analyses) indicate to be impor-tant for that audience (referring to thedefinitions of Appendix B as needed).2

b. For each second-level categorydecided on in a, list on the audiencesheet all third-level categories thatyou consider to be important(referring to the definitions of Appendix B as needed).

Repeat a and b for every sheet. Figure 2on page 19 shows the Alpha CollegePolitical Science Committee sheets afterStep 7 was completed.

'If you are unsure about whether a category should be Included, list It Don'twaste time evaluating further at this level because when you get down to the sub-categories for that category It will probably become clear whether or not it shouldbe included In the CASC/NCHEMS test of the Structure, this was a major reason forthe process being slowed

17

26

Figure 2

THE AUDIENCE SHEETS OF THE POLITICAL SCIENCE CURRICULUM COMMITTEE AT ALPHACOLLEGE AFTER THEY HAD COMPLETED STFP 7

Students Maloring in Political Science Sheet 1

2000Human Characteristic Outcomes

2120 - Interests2130 - Desires, Aims, Goals

2220 - Citizenship and Family Membership Skills

2230 - Creativity Skills

2240 - Expression and Communication Skills2250 - Intellectual Skills2260 - Interpersonal, Leadership, and

Organizational Skills2270 - Occupational Skills2310 - Attitudes and Values

2340 Mores, Customs, and Standards of Conduct

2410 - Perceptual Awareness and Sensitivity

2840 - Career and Vocational Roles

2650 Citizenship Roles

3000Knowledge, Technology, id Art Form Outcomes

Knowledge and Understanding of:

3110 - General Facts and Terminology

3120 - General Processes

3130 - Gsneral Theory

3210 - Specialized Facts and Terminology3220- Specialized Processes

3230 Specialized Theory

Other Students in Political Science Courses Sheet 2


2120 - Interests

2130 - Desires, Alms, Goals


2230 - Creativity Skills

2240 - Expression and Communication Skills

2250 - Intellectual Skills2260 - Interpersonal, Leadership, and

Organizational Skills2310 - Attitudes and Values

2340 - Mores, Customs, and Standards of Conduct2410 - Perceptual Awareness and Sensitivity


2850 - Citizenship Roles

3000Knowledge, Technology, and Art Form Outcomes



3120 General Processes3130 - General Theory

19

28

Figure 2

THE AUDIENCE SHEETS OF THE POLITICAL SCIENCE CURRICULUM COMMITTEE AT ALPHACOLLEGE AFTER THEY HAD COMPLETED STEP 7

Students Maioring in Political Science Sheet 1


2120 - Interests

2130 Desires, Aims, Goals

2220 Citizenship and Family Membership Skills2230 - Creativity Skills

2240 - Expression and Communication Skills

2250 - Intellectual Skills2260 - Interpersonal, Leadership, and

Organizational Skills2270 - Occupational Skills

2310 - Attitudes and Values2340 Mores, Customs, and Standards of Conduct

2410 Perceptual Awareness and Sensitivity2840 - Career and Vocational Roles

2850 Citizenship Roles

3000Knowledge, Technology, 7d Art Form Outcomes




3130- General Theory

3210- Specialized Facts and Terminology3220 - Specialized Processes

3230 - Specialized Theory



2120 - Interests

2130 - Desires, Alms, Goals


2230 Creativity Skills2240 - Expression and Communication Skills2250 - Intellectual Skills2260 - Interpersonal, Leadership, and

Organizational Skills

2310 - Attitudes and Values

2340 - Mores, Customs, and Standards of Conduct2410 - Perceptual Awareness and Sensitivity


2850 - Citizenship Roles




3120 General Processes

3130 - General Theory

19

28

STEP 8 Traditionally, maintenance and changehave been considered by many to befundamental purposes of postsecondaryeducation (for example, preserving theculture and improving the condition ofmankind). Too often planners focus onlyon bringing about change, and do notconsider maintenance outcomes in theirplanning. For example, in curriculumdevelopment for advanced courses, itmay be important to build in exercises atperiodic intervals that will keep thestudents from getting "rusty" on impor-tant basic skills that would not other-wise be applied in the advancedprogram. For these reasons, and becausethese two types of outcome goalsrequire quite different orientations andapproaches in planning and manage-ment, they were made fourth-levelcategories of the type-of-outcomedimension. Definitions for eachcategory follow:

1. MaintenanceOutcomes that resultin keeping the status quo; instabilization, reproduction, or preser-vation. Examples include preservingcultural values, restoring artiractsand paintings, keeping up theeducational level of the family, andskill maintenance provided by in-service education.

2. ChangeOutcomes that result inalteration of the status quo, inreorganization, modification, revision(improvement or otherwise), orreplacement. Included are economicand social mobility, degree or cer-tification awarded the student, in-creased knowledge and skill level,new art forms, technological in-novations, medical discoveries, andso forth.

For all of the type-of-outcome categorieslisted on each audience sheet, indicatein some way (for example, by the lettersM, C, and B, or by a fourth-digit code- of1, 2, or 0) whether the category shouldemphasize only maintenance, onlychange, or both maintenance, andchange. In the case of the Alpha College

Political Science Committee, theydecided to focus on change in eachcase. Therefore, they changed the fourthdigit of each outcome code number from'0' (both) to '2' (change).

STEP 9 For some purposes, such as curriculumdevelopment and planning for a course,still more detail may be needed than thethird-level categories of Table 4. Logic orspecialized taxonomies that are avail-able may suggest the need for a set offifth-level categories for some of thetype-of-outcome categories listed on thesheet. For example, using Bloom'sTaxonomy of Educational Objectives(1956), intellectual skills (code number2250) can be divided into: "ability tocomprehend," "ability to apply," "abilityto analyze," "ability to synthesize," and"ability to evaluate." Similarly, physicaland motor skill outcomes (code number2280) could be divided into moredetailed categories using one of the sixpsychomotor taxonomies listed in Len-ning (1977a). As a final example, "at-titudes and values" (code number 2310)could be divided into the followingcategories listed in the NCHEMS Inven-tory of Higher Education OutcomeVariables and Measures (Micek andWallhaus, 1973): intellectual disposi-tion, political attitudes and values,racial/ethnic attitudes and values,personal ethics, social conscience,religious and spiritual attitudes andvalues, and career attitudes and values.

If you determine that more detailedcategories are needed for any of thethird-level categories listed on youraudience sheets, decide what thosecategories should beeither throughapplication of an available taxonomy orthrough open-ended logic. Then replacethose listed categories with the evenfiner categories you have selected.Figure 3 shows the Alpha CollegePolitical Science Committee sheets af-ter Step 9 was completed. Note that theyreplaced "intellectual skills" with fifth-level categories taken from Bloom's

20

29

Figw-3 3

THE AUDIENCE SHEETS OF THE POLITICAL SCIENCE CURRICULUM COMMITTEE AT ALPHACOLLEGE AFTER THEY HAD COMPLETED STEP 9

Students Majoring in Political Science Sheet 1

2002Human Chcractenstic Outcomes

2122 - Interests

2132 - Desires, Aims, Goals

2222 - Citizenship and FamilyMembership Skills

2232 - Creativity Skills2242 - Expression and Communication

Skills2252 - 4Melleetee4-814110

2262 - Interpersonal, Leadership,and Organizational Skills

2272 - Occupational Skills


2342 - Mores, Customs, andStandards of of Conduct

2412 - Perceptual Awarenessand Sensitivity

2842 - Career and Vocational Skills2852 - Citizenship Skills

Ability To:

22521 - TranslatePoliticalScienceLiterature

22522 - InterpretPoliticalScienceLiterature

22523 - ExtrapolatePoliticalScienceLiterature

22524 - ApplyPoliticalScienceLiterature

22525 - AnalyzePoliticalScienceLiterature

22526 - SynthesizePoliticalScienceLiterature

22527 - EvaluatePoliticalScienceLiterature



3112 - General Facts and Terminology3122 - General Processes

3132 - General Theory

3212 - Specialized Facts and Terminology3222 - Specialized Prod s

3232 - Specialized Theory



2122 - Interests

2132 - Desires, Aims, Goals

2222 - Citizenship and FamilyMembership Skills

2232 - Creativity Skills2242 - Expression and Communication

Skills2252 - 40teileettiel-8M41e.

2262 - Interpersonal, dership,and Organizatk lal Skills


2342 - Mores, Customs, andStandards of Conduct

2412 - Perceptual Awarenessand Sensitivity

2842 - Career and Vocational Roles2852 - Citizenship Roles

Ability To:

22521 TranslatePoliticalScienceLiterature

22522 - InterpretPoliticalScienceLiterature

22523- ExtrapolatePoliticalScienceLiterature

22524 - ApplyPoliticalScienceLiterature

22525 - AnalyzePoliticalScienceLiterature

22526 - SynthesizePoliticalScienc;Literature

22527 - EvaluatePoliticalScienceLiterature


Knowledge and Understanding of:3112 - General Facts and Terminology


3132 General Theory

21

30

Taxonomy, and added a fifth digit to thecode number for each of those fifth-levelcategories.

STEP 10 For each detailed "type-of-outcome"category on each of the audiencesheets, on separate sheets of paper youare now ready to list important specificoutcomes you might expect (not justthose you desire) of that type for thataudience. Some users of the Structuremay want to develop long lists of ex-tremely detailed outcomes for eachcategory, while others may want todevelop shorter lists of outcomes thatare less specific. Several suggestionsmight help stimulate your thinking asyou attempt to identify specific out-comes for each category:

a. Think in terms of outcomes youwould consider to be: (1) "positive invalue," (2) "neutral in value," and(3) "negative in value." For example,many might consider "increasedability to analyze problems" to havepositive value and "increased druguse" to have negative value.

b. Think in terms of outcomes that are:(1) easy to measure, and (2) difficultto measure.

c. Think in terms of outcomes that are:(1) productstangible, concrete en-tities that endure with time, such asknowledge, skills, a program com-pleter, a degree, a job, a bookpublished; (2) eventsobservable,tangible transactions or sets ofbehaviors that do not endure withtime, such as a seminar, a concert, a'graduation exercise, being listed inWho's Who in America; and (3) con-ditionsintangible but real circum-stances, such as morale, satisfac-tion, an attitude or belief, an ap-preciation, social equality, achieve-ment, or elimination of polio.

d. Think in terms of outcomes that are:(1) outputsthe direct end products,events, or conditions that are expect-ed to result from the application ofthe institution or institutional unit

processes to transform the variousinputs, such as achievement levels,specialization of knowledge, degrees,program completers, publications,cultural or entertainment events; and(2) impacts the consequencesof outputs and earlier impacts(that is the indirect products, events,or conditions produced), the dif-ferences they make, such as a pro-gram completer's ability to obtainand hold a job, the security and In-come or the prestige the job gives aperson, the increased gross nationalproduct that results from increasedincome of individuals, the increasedstandard of living and quality of lifethat may be associated with in-creased gross national product, Sandso on.

The outcomes listed may be quitebroadlike those examples listedaboveor they may be very specific,such as the percentage scoring above aparticular level on a certain test or inven-tory. (Examples of specific measuresthat could be used for each detailedcategory of the type-of-outcome dimen-sion are given in the right-hand columnof the pages in Appendix B.) Howspecific they should be depends on thephilosophy, needs, and desires of theperson using the Structure.

STEP 11 Now that you have a list of specificdesired outcomes for each importantaudience, it will be useful to separate allthe outcomes listed into priority groups(and to record the priority status of each)such as the following: (1) very importantoutcomes and (2) important outcomes.You may desire to use other categoriesthan these. As with the other steps, itwill be beneficial to get input from majorconcerned groups of people, both onand off the campus.

To illustrate what the lists coming out of Step 11might look like, Figure 4 shows the first page ofintended outcomes for political science majorsdeveloped by the Alpha College Political Science

22

31

Figure 4

THE FIRST PAGE OF THE OUTCOME LIST FOR STUDENTS MAJORING INPOLITICAL SCIENCE THAT HAD BEEN DEVELOPED BY THE POLITICAL SCIENCE

CURRICULUM COMMITTEE AT ALPHA COLLEGE AT THE COMPLETION OF STEP 11

INTERESTS:

1 Increased student interest in the theory, principles, and practices of government and politics

2 Increased student interest in learning about different political concepts, philosophies, and ideologies

3 Increased student interest in entering politics, government, or public administration

4 Increased student interest in international relations and world views

5 Increased student interest in the analysis and solution of social problems

6 Increased student interest in constitutional law and jurisprudence

7. Increased student interest in the dynamics of change in American society

8 Increased student interest in the public economy and its operation.

DESIRES, AIMS, OR GOALS.

1 Increased student desire and intention to prepare for advanced and specialized study in political science.

2. Development by students of plans for political science careers in teaching, government, business, and law

3. Development in students of a desire for public service

4. Development in students of a desire for conducting political analysis and action while they are still studentson campus.

5. Development in students of an aim to be a model in the conduct of citizenship responsibilities.

6. Development in students of an aim to uphold democracy as ou political system.

7. Development in student; of a desire to improve the character of the relationships between citizens andgovernment, and of the values and criteria by which quality of political life is judged

8. Development in students of intentions to uphold citizen rights and liberties, and the separation of churchand state.

CITIZENSHIP AND FAMILY MEMBERSHIP SKILLS

1. Increased student ability to intelligently conduct citizenship responsibilities.

2 Increased student ability to understand the basic political processes and institutions as they operate in differentnational and cultural contexts.

3. Increased student ability to critically analyze public opinion and propaganda.

4. Increased student ability to analyze voting patterns, political participation, techniques of political action,legislative or governmental actions, and governmental bureaucracy.

5. Increased student ability to analyze and solve social problems.

6. Increased student ability to recognize the distinction between what public figures may say and their motivationsfor saying it.

* 7. Increased student ability to deal with the moral dilemma in politics between principle and compromise

CREATIVITY SKILLS

1 Increased student ability to run an innovative, creative political campaign.

Outcomes considered "essential", the others are considered Important

23

32.

Curriculum Committee as it looked when itcame out of Step 11. Note that they put anasterisk by each outcome they considered"essential," and the others were considered"important."

When Will the Outcomes Occur? Time hasalways been considered an especially importantfactor fo, many student outcomes. Particularteaching content and activities do not have mucheffect until the time a student is "ready" for them.Presumably there are times when curriculummaterials and emphases of different types willhave maximum influence. Thus, prerequisite ex-periences and/or courses must often be specifiedfor different advanced courses within a curricu-lum. Another example is that many students maynot be ready to choose a major until they havechosen a vocation.

Outcomes that occur at or prior to graduation aremore often a direct focus of planning than are im-pacts that occur after graduation, because the in-stitution has some control over them. On theother hand, if the relationship of the earlier out-comes to the postgraduate impacts can be ascer-tained, the later impacts may have some influ-ence on priorities. Furthermore, communicationsto the general citizenry about outcomes oftenmust have a primary focus on postgraduate im-pactswhether they are actual or presumedbecause postgraduate outcomes are of specialconcern to the public.

Another "time" consideration is how long theoutcome will endure; for example, is it of longduration or of short duration? Time is also impor-tant for planning and management related to non-student outcomes, such as research and scholar-ship outcomes, art forms and works outcomes,and economic outcomes for society. Because ofits critical importance to postsecondary educa-tion planning and management, "time" was madea dimension of the Outcomes Structure. Nocategories were assigned to this dimension.however, because (1) agreement could not bereached on such categories, (2) time categoriesneeded vary with the audience and type-of-outcome, and (3) the time categories needed by aninstitutional official will vary according tophilosophy and use context. For example, acurriculum developer will undoubtedly need aquite different set of time categories than will a

campus administrator concerned with generalcampus-wide outcomes. It was decided to leaveany specific categorizing (and definitions forthose categories) up to each individual using theOutcomes Structure.

STEP 12 Decide on the categories and sub-categories you need for the "timedimension."

STEP 13 Next to each of the outcome statementsin the lists coming out of Step 11, placethe names (code numbers or letterswould be more efficient) of all "time"categories that apply to that outcome.You are now ready to analyze and to usethe listings of outcomes (one list foreach of your important audiences).

See Figure 5 for an illustration of what the finallistscoming out of Step 13might look like.(Note that the time differentiation in theillustration was whether the outcome should beemphasized in upper-division courses [U], lower-division courses [L], or both upper and lowerdivision courses [B].) Keep in mind, however, thatthis is merely the way a political science curricu-lum committee at Alpha College designed it.You might very well wish to organize your listsdifferently.

Process 2: Classifying Outcomes

Some institutions and institutional units haveperiodically developed lists of desired outcomesor outcome goals and objectives. Taking such alist of outcomes and classifying it through ap-plication of the Outcomes Structure can provideinformation to institutional officials that willassist them as they evaluate the breadth as wellas the adequacy of the list. Such a procedure mayconfirm and stimulate the realization that thedesired outcomes are concentrated in only a fewspecific areas, that some are too broad andgeneral, that some are stated in unduly vagueterms, that some need to be translated into morespecific outcomes (and the Structure can help dothis), that few, if any, of the outcomes are in cer-tain areas of supposed concern to the institutionor institutional unit, that certain outcomes needto be added to the list, and so forth.

24

33

Figure 5

THE FIRST PAGE OF THE OUTCOME LIST FOR STUDENTS MAJORING IN POLITICALSCIENCE THAT HAD BEEN DEVELOPED BY THE POLITICAL SCIENCE CURRICULUM

COMMITTEE AT ALPHA COLLEGE AT THE COMPLETION OF THE FINAL STEP, STEP 13

INTERESTS'

B 1 Increased student interest in the theory, principles, and practices of government and politics

B 2 Increased student interest in learning about different political concepts, philosophies, and ideologies.

U 3 Increased student interest in entering politics, government, or public administration.

U 4. Increased student interest in international relations and world views.

U 5 Increased student interest in the analysis and solution of social problems.

U 6 Increased student interest in constitutional law and jurisprudence

L 7 Increased student interestin the dynamics of change in American society

U 8 Increased student interest in the public economy and its operation.

DESIRES, AIMS, OR GOALS'

U 1 Increased student desire and intention to prepare for advanced and specialized study in political science.

U 2 Development by students of plans for political science careers in teaching, government, business, and law

U 3 Development in students of a desire for public service.

B 4 Development in students of a desire for conducting political analysis and action while they are stillstudents on campus.

B 5 Development in students of an aim to be a model in the conduct of citzenship responsibilities.

L 6 Development in students of an aim to uphold democracy as our political system.

B 7 Development in students of a desire to improve the character of the relationships between citizens andgovernment, and of the values and criteria by which quality of political life is judged.

L 8 Development in students of intentions to uphold citizen rights and liberties, and the separation of churchand state

CITIZENSHIP AND FAMILY MEMBERSHIP SKILLS.

L 1 Increased student ability to intelligently conduct citizenship responsibilities

B 2. Increased student ability to understand the basic political processes and institutions as they operate indifferent national and cultural contexts.

B 3 Increased student ability to critically analyze public opinion and propaganda.

U 4 Increased student ability to analyze voting patterns, political participation, techniques of political action,legislative or governmental actions, and governmental bureaucracy.

B 5. Increased student ability to analyze and solve social problems

L 6. Increased student ability to recognize the distinction between what public fig,Ires may say and theirmotivations for saying it.

U 7. Increased student ability to deal with the moral dilemma in politics between principle and compromise.

CREATIVITY SKILLS

U 1. Increased student ability to run an innovative, creative political campaign

Outcomes considered "essential", the others are considered "important "

KEY

L = Lower Division CoursesU = Upper Division CoursesB = Both Upper & Lower Division Courses

25

34

Although coe person can classify the outcomes ina list, it is suggested that several persons be in-volved. Included should be representatives ofmajor groups having effective control or alegitimate concern about what the educationaloutcomes of the institution/institutional unitshould be (for example, administrators, faculty,students, and graduates).

Rather than write down by each outcome item thename of the Structure category or categories towhich it is assigned, it will be much more efficientand workable to use the code numbers that havebeen assigned to all categories of the Structure.As indicated earlier, each category of eachdimension of the Structure has a code numberassigned to it, and by looking at the code numberone can deduce the level of detail of the categorythat has been assigned. (Again see Appendix Afor a discussion of the coding scheme for theStructure.) Since few, if any, currently availablelists of institution-wide or institutional-unit out-comes reference the time the outcome is expect-ed to occur or give differential indications of whois to receive or be affected by the outcome, thediscussion in this section will focus onclassifying outcomes into categories of the type-of-outcome dimension. If the outcome statementson your list do specify the time when the outcomeis expected or intended to occur or be empha-sized, and the audience or audiences who areto receive that outcome, use the same proceduresto classify on those dimensions as are outlinedbelow for the type-of-outcome dimension.

The procedures presented below are not limitedto use in classifying lists of outcomes and out-come goals or objectives. They can also be usedto classify, in terms of outcomes, such things asresearch findings in the literature, questionnaireitems, institutional and program activities aimedat outcomes, needs, campus environments, andphilosophical orientations.

STEP 1 Note in Table 3 on page 6 that the firstdigit of the code number identifies thefirst-level categories-1 = economicoutcomes, 2 = human characteristicoutcomes, 3 = knowledge, technology,and art form outcomes, and so forth.(The zeros in the succeeding digits ofeach code number indicate that the out-come is not being classified at the more

refined levels of detail that areavailable.) In front of each outcomestatement on your list, write "1" if it isan economic outcome, "2" if it is ahuman characteristic outcome, and soon, until all statements in the list areclassified at this level of detail. (In eachcase, leave enough space after the digitso you can add additional digits.)

STEP 2 Table 4 on page 7 shows the second-and third-level categories applying toeach first-level category of the type-of-outcome dimension. For each outcomestatement on the list that has a "1" infront of it, determine whether the seconddigit should be "1," "2," "3," or "4" byexamining the title of each of the foursecond-level categories for that majortype of outcome (1 = economic accessand independence, 2 = economicresources and costs, 3 = economicproduction, and 4 = other economic out-comes). Each of the second-levelcategories is defined in Appendix B(page 39) and you may need to refer tothose definitions in determining whichsecond-level category specific econom-ic outcomes should be assirned to.Some of the economic outcome state-ments may be too broadly stated tobe classified in any second-leveleconomic outcome category. If theycannot be classified at this level, assignthem a second digit of zero. If such anoutcome statement applies to two of thecategories, you can either assign it twosecond digits (which will give it two dif-ferent code numbers) or give it a zero,whichever you prefer.

Next, for all outcomes in the list thathave a "2" in front of them, determinewhich of the nine secvnd-levelcategories of "human characteristicoutcomes" apply. Assign these out-comes a second digit number in thesame manner that you did for economicoutcomes. If it cannot be classified atthis level, give it a second digit of zero; ifit is an aspiration outcome, give it asecond digit of "1"; if it is a competenceand skill outcome, give it a second digitof "2"; and so forth.

3526

In the same manner as you assigned asecond digit code to the economic andthe human characteristic outcomes onthe list, assign second -digit codes tooutcomes assigned to the three otherfirst-luvel categories. When this is doneyou are ready to classify at the thirdlevel of detail, assuming that you desireto classify at that level of detail. Forsome lists (such as broad overall in-stitutional goals), classifying to thesecond-level of detail may suffice.

STEP 3 Give a third digit of zero to each out-come statement in the list that has asecond digit of zero.

STEP 4 For each outcome statement in the listthat has a "11" in front of it (economicaccess and independence outcomes),determine which of the three third-levelcategories listed in Table 3 (for thatsecond-level category) applies, andassign it a third digit accordingly (forexample, 111 = economic access and113 = income and standard of living). Aswith the second-level categories, eachthird-level category is defined in Appen-dix B. You may need to look at thesedefinitions to classify at this level ofdetail. If the outcome cannot beclassified at this level of detail, assign ita third digit of zero.

In like manner, assign third digits to thecodes for all outcomes on the list thathave "12" in front of them, then to alloutcomes that have "13" in front ofthem, then to all outcomes that have"14" in front of them, then to all out-comes that have "21" in front of them,then to all outcomes that have "22" infront of them, and so forth. Once all ofthe outcomes on the list have beenassigned third-digit codes, you are readyto assign fourth-digit codes (once again,assuming that you desire to classify atthat level).

STEP 5 If you do not desire to classify at thefourth level of detail, give every outcomestatement on the list a fourth digit ofzerc If you do desire to classify at the

fourth level of detail, start with the firstoutcome statement on the list andproceed one-by-one to the last item onthe list, assigning fourth digit codes asfollows:

1 The outcome statement indicatesthat it is a "maintenance" outcome,an outcome that will result in keepingthe status quo, in stabilization, inreproduction, or in preservation.

2 The outcome statement indicatesthat it is a "change" outcome, anoutcome that results in alteration ofthe status quo, in reorganization, inmodification, in revision, or inreplacement.

0 The outcome statement: (a) indicatesthat it is :ncluding both maintenanceand change, or (b) gives no indicationas to whether it is referring to main-tenance or to change.

For an example of a case where these procedureswere actually applied, see Appendix C. All of theoutcomes in the NCH EMS Inventory of HigherEducation Outcome Variables and Measures werelisted on a sheet. Then one or more four-digit codenumbers were assigned to each using theNCHEMS Outcomes Structure and going throughthe five steps that have been described in thissection. Note that the procedures were modifiedslightly. instead of placing the Structure codenumbers in front of the outcomes on the list (asspecified in Step 1), they were placed behind theoutcome statements to facilitate including thedescriptive "type-of-outcome" name for eachStructure category assigned. Modifications of theprocedures outlined here (or of the content of theStructure itself), to meet special local needs orsituations, are encouraged.

Appendix C has another use in addition toillustrating the application of the five steps out-lined above. Those who have in the past appliedthe NCHEMS Inventory to institutional planningfunctions can use Appendix C to transform theirprevious work into Outcomes Structure terms, ifthey desire to do so.

After examining the distribution of code numbers,you may decide that too many outcomes on thelist are not specific enough for your needs, or that

27 36

important outcome areas are not emphasizedenough or are completely ignored by the list. Ifthis is the case, you may wish to expand and inother ways modify the outcomes list. Such ex-pansion and other modifications can be accom-plished by following the procedures outlined forProcess 1 on pages 15-24

Process 3: Storing and Retrieving OutcomesInformation

The statement was made in the introduction tothis document that "having a wide variety of out-comes information without any structure isanalogous to possessing a file cabinet in whichthe contents are arranged randomly." Outcomeinformation to be stored could consist of a pool :

thousands of behavioral objectives that havebeen collected from different curriculumdevelopers, a pool of outcome questionnaireitems that have been collected from differentdevelopers of outcomes instruments, a pool ofdifferent measures that have been developed fordifferent outcome variables, a collection ofreferences from the literature of postsecondaryeducation outcomes, and so on. They could bestored in a filing cabinet, in a microfiche ormicrofilm information storage and retrievalsystem, in a computerized information storageand retrieval system, or in other ways.

In most cases of applying the outcomes Structureto develop an outcomes information storage andretrieval system, use of the coding schemedescribed in Appendix A will be necessary. This istrue especially if one is developing a microficheor computerized system that includes hundredsor even thousands of outcome categories. As isapparent in Appendix A, the storage system canbe a one-dimension system, a two-dimensionsystem, or a three-dimension system. It can be avery simple or a very complex (although fairlystraightforward in concept) system, depending onthe need of the local user.

NCHEMS currently is developing a computerizedinformation bank containing selected informationabout outcomes studies and other literary worksthat can contribute to a synthesis or analysisrelating to the development of outcome indicatorsand measures, correlates of outcome occurrence,research questions relating to outcomes,strategies for increasing or improving outcomes,

impacts that can be expected to result from par-ticular outputs and combinations of outputs, andso forth. This computerized information bank,along with a manual card file of literature abstracts,illustrates the application of the Structure tothe process of developing a storage and retrievalsystem for outcomes information. Based on thisexperience, a number of general steps for thisbasic process are outlined below.

STEP 1 Obtain input from the potential users ofthe system concerning: (a) needs for thesystem, (b) the possible ways they woulduse the system, and (c) criteria thesystem would have to meet to be fullyapplicable to their needs. In addition,logically consider what other demandson the system might be expected todevelop once it becomes accepted andproves itself.

STEP 2 Based on Step 1, outline in list form thepurposes of the System. For example,some of the major purposes for theNCHEMS system that is being devel-oped are: (a) development of a synthe-sis of the literature for each measurein the NCHEMS Outcomes Measuresand Procedures Manual (OM PM) thatrelates to its validity and reliability,(b) expansion of the OMPM to includeadditional measures in the areas alreadycovered and measures in areas not ad-dressed previously, (c) expansion of theresearch and scholarship section of theOutcomes Structure, (d) development oflists of outcome measures and in-dicators separately by audience forselected type-of-outcome categories inthe Outcomes Structure, (e) backuparalysis and support for the develop-ment of new student outcome indicatorsand measures, and (f) analyses leadingto improved application of outcomes in-formation to institutional and programplanning and management.

STEP 3 Based on Step 2, plus convenience,cost, political a;id other considera-tions, determine what basic type ofsystem it should be, for example,manual card files, notebooks oforganized information sheets, filing

28

37

cabinets, a manual or automizedmicrofiche or microfilm system, an on-line computer system, a computerizedtape system, or a computerized disksystem. Combination types should alsobe considered, like the NCHEMS ex-ample referred to above.

STEP 4 For each purpose of Step 2, develop alist of criteria that the system shouldmeet, taking into consideration the inputfrom potential users gathered in Step 1.

STEP 5 Determine whether the Outcomes Struc-ture will be the most desirable basicframework around which to structureand organize the Ming system. If it willnot, decide on a framework to beusedwhether it is a modified versionof the Structure or something com-pletely different.

STEP 6 Determine what specific data should beentered into the master file and the sub-files of the system. For example, at thisstage of development of the NCH EMSsystem (and this may change later) thefollowing items of information abouteach study or other source are being en-tered into the master computer file: asource code, a unique identificationnumber, author/title information, biblio-graphic data, postsecondary educationentity or entities being focused on, typeof study, type(s) of analysis, its applica-

tion focus, its assigned NCHEMS Out-comes Inventory code number, its Out-comes Structure code number. Alsodetermine what form each set of datashould take, for example, should it becoded or written out.

STEP 7 Design the source-identification, data-collection, and data-preparation pro-cedures (such as coding activities,quality control activities like inspectingand editing, and keypunching or otherrecording activities) for the system.

STEP 3 Design the procedures for entering thedata into the system, fitting new datawith the data already there, andmodifying or deleting data later shouldthat become necessary.

STEP 9 Design appropriate procedures formanipulating the data within the system(sorting, matching, aggregating, tabu-lating, reforming, applying analyticaltechniques, and so forth) that will beflexible enough to meet all of theplanned objectives for use of thesystem.

STEP 10 Design appropriate procedures forretrieving the desired and appropriatedata and analytic or synthesized infor-mation from the system, and applyingthose data to the problem at hand.

29

38

PART IVORIENTING INSTITUTIONAL AND

PROGRAM FACULTY AND STAFF TO THEOUTCOMES STRUCTURE

AND TRAINING THEM IN ITS USE

During the CASC Outcomes Learning Task Forceproject and the CASCINCHEMS joint projectdiscussed previously it was found that: (1) Mostpeople on the participating campuses had neverreally considered what they were trying to ac-complish for students in very systematic, con-crete, and specific outcome terms; rather theywere accustomed to thinking solely in terms ofactivities and tasks they wanted to accomplish,although there were undoubtedly outcomereasons "in the back of their minds" for wantingthese activities. (2) People at those campuses,and particularly faculty, found it uncomfortable totry to talk in systematic, concrete outcometerms about what they were trying to accomplishfor students; and a number of them exhibitedinitial resistance to trying to apply the OutcomesStructure to their situation, although the anxietyoften seemed to subside as the interview sessioncontinued. (3) Interviewees had to really strain togo from priority outcome categories to especiallyimportant, specific, concrete, observable out-comes within those categoriesbut with inter-viewer support and assistance they were able todo so successfully. (4) The orientation materialsdistributed to the interviewees ahead of timetended not to be read because they were too bulkyand complex looking, and the workshopalthough helpfulwas not sufficient for orientingcampus personnel to outcomes concepts and theOutcomes Structure. Clearly, small groupdiscussion, or on a one-to-one basis, is needed inaddition to brief, attractive orientation materialsand effectively organized and carried outworkshops. (5) The fact that the CASCINCHEMSproject interviews were more successful in intro-ducing a cut-down version of the type-of-outcomeset of categories (prior to doing to the entire set)than were the CASC project interviews in goingdirectly to the entire set of categories, and that it

was found desirable at Spring Arbor College touse an abbreviated set with no code numbers intheir CASCINCHEMS project follow-up activitieson campus, suggests that the full Structureshould not be presented during the initial stagesof one's on-campus orientation program. (6) TheUniversity of Colorado person, who had workedwith outcomes concepts over a period of severalyears, was able to easily take the OutcomesStructure procedures in this document and applythem effectively and efficiently without any orien-tation or help of any kind. This suggests that oncethey have been properly oriented to outcomesconcepts and the Structure, campus personnelmay be expected to use them effectively and ef-ficiently, although many probably will still needtraining assistance in how to correctly apply themto decision making.

Based on experiences in the trial projectsdiscussed above, plus other experiences atNCHEMS and elsewhere, a set of generalguidelines have been developed and presentedbelow for use by those who wish to introduce out-come concepts and the NCHEMS OutcomesStructure on campus and to have them become anintegral part of the campus planning, manage-ment, and evaluation process. Modification of thesuggested procedures to meet the local situationand conditions is encouraged, and it is hoped thatthe guidelines can be expanded, improved, andmade more specific as input IQ received frompeople trying to orient their campuses to outcomeconcepts and the Structure (or modified versionsof it) in different contexts across the country.

1. Do Your HomeworkBefore you can communicate effectively toothers about outcome concepts and the Out-

31

comes Structure, you must really understandthem yourselfincluding cautions that needto be taken in using themand feel at ease indiscussing them with others. To do this willrequire thorough study of this manual andthinking that goes beyond the manual to thesituation on your campus. It is suggested thatyou also read the conceptual documentwhich is a companion piece to this manual,designed for those who desire a detailed andin-depth discussion of educational outcomesand the Outcomes Structure (Lenning, Lee,Micek, and Service, 1977). Chapter 2 of thatdocument describes six attributes of aneducational outcome plus five other factorsthat are important for really understandingand interpreting a particular outcome; Chapter3 contains detailed background discussionabout each formal dimension of the OutcomesStructure and goes into detail about theStrucwie as a whole; and Chapter 4 lists thenine basic principles followed in developingthe Structure (which were also used toevaluate it) and discusses the available infor-mation that suggests how well these criteriawere met. If you would also like to compare theStructure to other attempts that have beenmade through the years ti structure outcomesand outcome-related concepts such as goals,see Lenning (1977a), which is also availablefrom NCHEMS.

2. Prepare Draft Copy for a Brief Brochure ThatEffectively Introduces Relevant OutcomesConcepts and the Structure.To raise initial awareness on campus aboutuseful outcomes concepts and the Structure,and to bring about curiosity and interest inexploring this further, a short, concise, and at-tractive introductory brochure will be needed.An example brochure or flier from which youcan build is shown in Appendix D; you mayprefer an entirely different concept and ap-proach for your brochure, however.

3. Prepare Yourself to Respond to Criticisms ThatMight Be RaisedYou can expect that various people on yourcampus will think of important criticalquestions to raise about outcome conceptsand the Outcomes Structure, and it is impor-tant that you be prepared to respond openlyand honestly to such questions. Figure 6 listscautions and potential problems that were

mentioned by interviewees in the jointCASC/NCHEMS project. Possible responsesto the criticisms implied or raised directly here,plus criticisms raised by other reviewers of theStructurethat are reprinted from Lenning,Lee, Micek, and Service (1977)are presentedin Appendix E.

4. Get the Concurrence of Top-Level Manage-ment about the Need to Proceed FurtherTop-level administrative p rscnnelforexample, the presidentmay have asked youto examine, become familiar with, and explorethe usefulness of the Structure and outcomesconcepts like those discussed here, in whichcase you should probably now share yourthinking at this point with them and makerecommendations to proceed further or notproceed further. If, however, your interest inthe subject and your exploration in Steps 1-3were at your own initiative and you feelpositive at this point about the usefulness ofthe outcomes concepts and the Structure foryour campus, now would probably be an ap-propriate time to introduce the subject to ap-propriate top-level administrators (in a waythat will interest them in it) and obtain theirpermission to develop some plans for in-troducing outcomes concepts and the Struc-ture to personnel on your campus. Do not gointo too much detail in your presentation, keepit short, and point out the practical usefulnessthat has been found by people on other cam-puses (for example, in the CASC/NCH EMSproject). Take into consideration the localsituation and political realitiesfor example,if management by objectives is "in vogue" onyour campus, show how the Structure cancontribute to that process; but if it is not, em-phasize rather how the Structure can helpcampus people to clarify and stimulate theirthinking about what they are really trying toaccomplish in their programs and to delineatein concrete terms why they are making use ofparticular approaches and activities. At thisstage it is important to have obtainedauthorization from the president, or his or herdesignate, to proceed further.

5. Develop Preliminary Plans and Materials for aCampus Orientation WorkshopFor effective dissemination of outcome con-cepts and the Structure, a well-designed orien-tation workshop will be essential. A possible

32

40

Figure 6

CAUTIONS AND POTENTIAL PROBLEMS WITH THE NCHEMSOUTCOMES STRUCTURE THAT WERE MENTIONED BY VARIOUS INTERVIEWEES

IN THE CASC/NCHEMS PROJECTa

Need to get students and alumni involved in outcc:nd planning also.

Naed to show people how it will help.

This kind of self-analye 1 is very important, but will it make any difference on the campus?

Might work If you have the right setting, like a workshop away from the campus.

Don't use it unless you have the time and energy to change.

One needs time away from day-to-day pressures in order to Implement this.

Resistance to the "bureaucratese" in which the document Is writtenI don't want to go to the trouble of usingit unless you can show me that it is directly fele% nt to my department and its students.

Potentially very time consumingsafeguards should be burl In to insure that the time drain is not too great(find out the outcomes of the system that can be achieved In a short time).

It may promise more than it delivers.

Is the system really hierarchicalis hierarchy the appropriate model?

Need funds for the department to develop long-term workshops to incorporate use of the Structure.

If it is presented properly, students could become interested in the Structure.

I like it. Dynamite! It could be such a long-term thins, however.

Difficult to reach consensus on such thingsneeds a good leader.

Takes too much timewhat is needed is funds for a week's r .aff retreat in a retreat setting.

More useful at small c'lleges than elsewhere because we have more of a chance to see and understand theactivities which lead to outcomes (the why of outcomes) and we get to know our students very well.

Need to start using this In goal setting at the institution-wide level If you start at the department level will goin all directions so that when people get together at the institution-wide level will be so far apart will never beable to reconcile.

It is easier to think of outcomes in specific and concrete terms at the department level, so should startapplying the Structure to reach consensus at that level before you try to reach someconsensus oncollege-vide goals.

Need to have total faculty involvement at the departmental level if you are going to make good use of theStructure.

An excellent instrument if used and the system instructions understood and participated in by most of thefaculty in the small college. The problem may lie in attitudesa strong disposition and loyalty to the liberalstudies and classical structure . . . time constraints, and using the Structure as they envision it. r nce it ilearned and implemented, however, the negatives could be reversed dnd it could speed the processes 4

planning and development.

A problem with the Structure concerns differences In definitions, and it includes jargonese.

As a student majoring In religious education I tend to think in more general terms. Therefore, it is limitingit limits my options. I would prefer a completely open -ended approach.

It would be useful to the extent that it serves rather than enslaves. Minimal outcomes should be emphasized.Should let more happen bevo,...1 the stated outcomes.

I question its real utility, although it is an asset In that it is systematized.

I heard from a staff member who attended the workshop yesterday morning that a lot of good ideaswerepresented, bit that they were perhaps overly Idealistic.

People need to see the connections sooner to their on going tasks. Needed is a week at a retreat settingworking with faculty on institutional goals and objectives, and how to implement them

aReprinted from Leaning (197,,

33

41

agenda, sample overhead transparencies, awork exercise that is meant to stimulatethinking and promote concrete discussion,and a role-playing simulationbased on theworkshops presented at the colleges par-ticipating in the joint CASCINCHEMS trialprojectare provided in Appendix F. Based onthese or your own thinking, develop somepreliminary plans and possible materials foruse in such a workshop on your campus. Alsodevelop preliminary plans for evaluating theworkshop(s) and for using the evaluationdatasee Step 9. A majority of the partici-pants in the CASCINCHEMS workshops foundthem interesting and stimulating, andespecially after certain modifications weremade, so there is a precedent for expectingthat such a workshop can be helpful. However,follow-up discussion and activities with smallgroups a 41 individuals 6% ,Il be important. In theCASCINCHEMS project, it was not until thefollow-up interviews with individuals thatmany of those attending the workshop reallystarted to understand the Structure and itsuse. Interview discussion tended to clarify andreinforce the workshop content for them, andthey felt free to ask questions and to raise per-sonal concerns.

6. Try Out Your Ideas and Draft Materials onTrusted FriendsIt will be important to discuss your ide- 3, andthe draft materials developed in Steps 2 and 5,in depth with a few appropriate trusted friendsand colleagues, preferably some off the

.pus as well as on. If students ale to be in-volved in the workshop, some of them shouldalso be involved in this preliminary tryout.Based on the reactions and suggestionsreceived in these conversations, make desiredmodifications in your preliminary plans andmaterials.

7. Establish an "Exploration and Planning"Committee concerning a Campus OrientationEffortThe Structure and outcome concepts havepotentially important applications within allmajor departments and offices on campus.Therefore, it would be desirable to form an"exploration ,ind planning" committee havingrepresentat.ois from the major groups oncampus (including a student representative). If

possible, try to limit thG size of the committeeto a number of participants that facilitates in-volvement, working together, and effectivedesicion makingfor example, many peoplefeel that a committee with more than 12-15members comes unwieldy. In your letter in-viting them to participate it would be desirableto enclose a copy of the draft brochure, tellthem about your exploration into the topic andyour conclusion that the Structure has muchpotential for being of real value to your cam-pus and its subunits, outline the charge givento the committee, and inform them that thiscommittee and its efforts have beenauthorized by a directive of the president (Aninitial letter of appointment to the committeeshould be received by them from the presidenta few days before your informational letter,which will emphasize to them the importanceof the committee's work and help motivatethem to want to get actively involved.)

8. Reevaluate Your Ideas, Using the Committee,and Then Complete Plans and MaterialsPresent the ideas, preliminary plans, andmaterials developed to the committee in themanner you would use in your orientationprogram, gain their support, and have themhelp you modify and complete the plans asneeded for an efficient and effective programof communication and orientation to the cam-pus community. Because the various impor-tant sectors of the campus are represented onthe committee, the members can providevaluable views on what is needed to make theprogram effective for the sector theyrepresent. Furthermore, they can com-municate their interest in outcomes to theircolleagues and help to carry out the plans fo7their sector of the campus, including the all-important follow-up interviews, consultation,and other assistance.

34

Whether the workshop should be a campus-wide event or consist of a series of workshopsseparately for a single or related sectors of thecampus depends on the local situation (forexample, a very small institution might preferone central workshop). Another possibility isto have a central workshop involving selectedpeople from each sector and then have thesepeople help plan and put on a similarworkshopthat contains more pertinent caseexamplesfor their sectors of the campus.

42

Because on-campus personnel are very busy,having the workshop prior to the start ofschool or between terms, extending it over twodays (for example, from noon to noon), andplacing it in a retreat setting are desirable. If itis necessary to have it during the school year,a carefully selected Saturday may be an op-tion, or splitting it up into an hour or so a weekfor several weeks may be necessary.

It is important to have the workshop(s) en-joyable and yet serious, fast-paced and yet notso rapid-moving that understanding is hin-dered, well organized and disciplined and yetinformal and friendly.

9. Distribute Brochures and Conduct Workshopsas PlannedThen EvaluatePlans for evaluating the reception of and reac-tions to the brochure and workshop(s), fordetermining what more information andclarifications are needed by different people,and for determining the need to makeavailable follow-up implementation andconsultative assistance andlor trainingworkshops to help apply the Structure and

outcomes concepts to actual planning,evaluation, and other pertinent problems,should have been completed in Step 8. Itwould be useful also to have a speciallydesigned questionnaire (tailored to the localsituation) to be completed immediately afterthe workshops or within several days, plusfollow-up interviews with individuals or smallgroups.

10. Make Available and Publicize the Availabilityof Follow -Up Application AssistanceBased on the evaluation of Step 9 anddeliberations of the planning committee,develop, make available, and let unitsthroughout the campus know about theavailability of assistance in applying out-comes concepts and the Structure to par-ticular planning, management, and evaluationactivities. This could include advisory andconsulting assistance, in-depth trainingworkshops for staff who will bq applying out-come concepts and the Structure, and directassistance in applying the Structure and out-come concepts to help meet pertinent needs.

35

43

Appendix A

The Coding Scheme for the Structure

The successful use of the Outcomes Structure forall three basic processes discussed in Part Iii ofthis doJument rests on the fact that for eachdimension there are successive levels of in-creasingly detailed categories. Assigned to eachcategory is a code number that has as many digitsas there are levels of detail in the dimension,where the first digit identifies the first-levelcategory, the second digit identifies the second-level category within the first level category thatwas specified (a zero indicates that none or all ofthem are it Aided), and so forth. As discussed inStep 8 of Process 1 (page 20), the user of theStructure can add additional levels of detail ifdesired, and this adds one digit to each codenumber for every level of detail added. Multi-dimensional categories have code numbers con-sisting of sets of digits (each identifying acategory for one of the dimensions) separated bydecimal points.

The code numbers provide an abbreviated andmore economical way of referring to categories inthe Structure than using descriptive terminology.For example. it is much easier to say or write"13.11132.22" than "longduration change in income and standard of living appearing aftergraduation for the family and relatives of studentsand former students." The coding for eachdimension, and for multi-dimensional categories,is discussed separately in the sections thatfollow.

Coding for the "Audience" Dimension Categories

The "audience" dimension of the Structure refersto the people, the communities, or the things thatreceive or are affected by (or that are intended toreceive or be affected by) the outcome of concern.The categories and subcategories of the audiencedimension (along with definitions where needed)are shown In Tables 1 and 2 on pages 4 and 5.Note that there are two levels of detail, with thefirst digit of each code number identifying thefirst-level categories and the second digit iden-tifying the second-level categories. Note also that

within a level, the subcategories are listed in theorder of a logical (but presumably neutral,valuewise) progression, and numbered con-secutively. A digit of zero i ij;cates that none ofthose categories at this level apply (that this levelis being bypassed) or that all of them apply.

If more detail is needed for any second-levelcategory (and this may often be the case), an ad-ditional level of detail can be added to the dimen-sion, as outlined in the discussion of Step 3 onpage 16. The third-level categories for eachsecond-level category would be represented by athird digit added to the code number. The third-level categories would be ordered within eachsecond-level category according to some logical,but neutral. progression or else alphabetically,and they would then be numbered consecutively.A third digit of zero would represent cases wherethe "audience" is broad or unsegmented at ti:slevee et detail.

Still more levels of detail in the audience dimen-sion could be added if needed. A fourth level ofdetail would necessitate four-digit codes, a fifthievel of detail would n...,:aesitate five-digit codes,and so forth.

Coding for the "Type-of-Outcome" DimensionCategories

The "type-of-outcome" dimension identifieswhether or not the outcome involves a change instatus (maintenance versus change), and thebasic entity that is maintained or changed (suchas knowledge and understanding, skills and com-petencies, attitudes and values, certificationstatus, income, standard of living, social Interac-tions, and art forms or expression). This dimen-sion of the Structure has four levels of detail, withcategories for the first three levels shown inTables 3 and 4 on pages 6-7, and with definitionsfor the categories given In Appendix B (pages 39through 51). Categories and definitions for thefourth level of detail ("maintenance" versus"change") are on page 20.

37

44

Because there are four levels of detail, eachcategory of the type-of-outcome dimension isrepresented by a four-digit code number. Withineach of the first three levels of detail, thecategories are ordered alphabetically (except forthe "other" categories, which are always placedlast), and the digit for each level of detail num-bered accordingly. The two categories for thefourth level of detail go from status quo (1) toalteration of the status quo (2), and once again adigit of zero indicates that no differentiation isbeing made.

If there is a need for more detail, additional levelsof detail can also be added for the type-of-outcome dimension, as discussed in Step 9 onpage 20. A fifth level of detail would necessitate afive-digit code number for the categories of thisdimension, and an extra digit would have to beadded for each other level of detail added.

Coding for "Time" Dimension Categories

The "time" dimension refers to when the outcomeoccurs and/or how long it lasts. Becauseagreement could not be reached on anycategories and the categories needed varydepending on the audience of concern and on thesituation, no categories were assigned to thisdimension of the Structure. After evaluating his orher need, the user should decide how many levelsof detail are needed for this dimension, decide oncategories for each level, order the categories ateach level (separately for each category at thenext higher level) alphabetically or in someneutral and logical progression, and number eachcategory consecutively so that all categories willhave assigned code numbers. An example of thisis Table 5 on page 8.

Coding for Multi - Dimension Categories

Each of the 33 "audience" categories and sub-categories in the Structure can be divided into the224 different categories and subcategories of the"type-of-outcome" dimension, which results in7,392 possible audience/type-of-outcome com-bination categories for which information can besorted. The code number for such a combinationcategory is the audience code number and thetype-of-outcome code number separated by adecimal point. For example, the code number of"change in intellectual skills of students" would

be 11.2252, and the code numbe, of "maintainingthe economic productivity of the local com-munity" would be 31.1311. If one additional levelof detail were added to both dimensions becauseof local needs for additional specificity, therewould be three digits in the set of numbers priorto the decimal point and five digits in the set ofnumbers after the decimal point.

If you add categories for the "time" dimension,the code number for each three-dimensioncategory consists of three sets of numbersseparated by decimal points, where the code forthe "audience" dimension is listed first, the codefor the "type-of-outcome" dimension is listedsecond, and the code for the "time" dimension islisted last. Suppose, for example, that a user ofthe Structure developed the followingstudent/former student categories and sub-categories for "time," and assigned the indicatedcode numbers to each (note that two levels ofdetail are shown):

10. Short-Duration Outcomes

11. Short-durationoutcomes appear-ing at or prior tograduation

12. Short-durationoutcomes appear-ing after gradu-ation

20. Long-Duration Outcomes

21. Long-durationoutcomes appear-ing at or priorto graduation

22. Long-durationoutcomes appear-ing after gradu-ation

The code number for "longduration change inincome and standard of living appearing aftergraduation for the family and relatives of students"would be 13.1132.22, assuming that the user ofthe Structure did not add levels of detail to eitherthe "audience" or the "type-of-outcome" dimen-sion. Similarly, 31.3001.21 would be the codenumber for "longduration knowledge, tech-nology, and art form maintenance outcomesappearing at or prior to graduation of students, forthe local community" (an example would be artstudents at the college helping local communityresidents keep local art objects fromdeteriorating).

38

45

Appendix B

Definitions and Outcome Measure or Indicator ExamplesFor the Type-of-Outcome Subcategories

Page

Economic Outcomes 40

Human Characteristics Outcomes 41

Knowledge, Technology, and Art Form Outcomes 48

Resource and Service Provision Outcomes 50

Other Maintenance and Change Outcomes 51

The code number for each category is given in the left margin (if the focus is on onlymaintenance the fourth digit would become "1," or if the focus is on only change thefourth digit would become "2"). Category definitions are provided in the next column, andseveral examples of possible indicators or measures of such outcomes are provided in theright-hand column for each category. The measures and indicators listed are onlyillustrative examples. The majority of examples given are for individuals, but it should beremembered that composites of these can often serve as indicators for communities andother populations. Future NCHEMS work will include the development of relativelycomprehensive lists of indicators for selected categories of the structure.

39

46

CategoryCode Number 1003 ECONOMIC OUTCOMES'

1100

1110

1120

1130

Economic Access and independence OutcomesOutcomes that relate to the entrance into obtainability, flexibility, and levels andamounts of monetary or pecuniary situations, conditions, and characteristics

(Categories)

Economic AccessThe amount of openness Or ecse of admittanceto economic opportunities, advancement

(Examples of Outcome Measures or Indicators)

Percentage of students obtaining their first full -time job In the fieldof their choice within a specified time after graduation

The number of alternatives for an entry level job open to minoritygroup graduates compared to minority group nongraduates

Economic Flexibility, Adaptability, and SecurityThe amounts ofself-sufficiency, liberty, frugality, thrift, self-government,confidence, certainty, safeguards, stability, and adjustment thatare exhibited in economic matters

Geographic mobility of college graduates compared to those notattending college

Self-report of college graduates about the economic security forthem and their families, and the contribution of college to this

Income and Standard of Living Amount of profits, return oninvestment, necessities and comforts of life, wealth, and othersigns of economic "well-being" that are obtained or possessedIncluded la direct support provided to individuals and thecommunity through local purchases by the educational institutionand through staff salutes and wages.

Amount of annual and lifetime earnings of those attending collegecompared to those not attending college.

Average student and/or former student reported scores on scalesmeasuring perceptions and evaluations of their current and desiredsocio-economic level

1200

1210

1220

Economic Resources and CostsOutcomes that relate to the amount and type of material, energy, effort, people, organization, andother economic assets that we available or that are expenr,..d in economic activities and production

(Categories)

Economic Costs and EfficiencyThe amounts of sacrifice, effort,expenditure, and waste present in economic activities andproduction


The absenteeism and tardiness on-the-job of college graduates ascompared to nonstudents

The number of firms that use the college degree as an Inexpensivescreening device that allows them to hire qualified employees atminimum initial cost to the firm

Economic Rosourrt. is (including employees) The assets availablethat can aid economic production, distribution, and gain

Percentage of college graduates employed in managementpositions within a specified time after graduation

Average number of patents and/or copyrights received per student,former student, and/or faculty member.

1300

1310

1320

Economic ProductionOutcomes that relate to the creation of goods, services, and economic valueissi=wemsec6

(Categories)

Economic Productivity and ProductionThe value of goods andservices that are created or produced by and within specificenterprises of "audiences' or clients of the educational institution,and especially In relation to the resources expended in theenterprise


Percentage of college graduates who can adequately do theirpersonal typing and complete their own income tax forms as aresult of having attended college.

Expert judges' ratings of the amount of Increased workerproduction and higher worker motivation that results from havingattended college

Economic Services ProvidedAmount and type of direct-assistance activities provided by the educational institution or itssubunits in the economic area.

Dollar amount of goods and services bought in the localcommunity by the institution, Its staff, And Its students Numberof hours of consultation in the business area provided to weecompanies and Institutions by the university's college of business.

1400 Other Economic Outcomes An example would be that a company with a large payroll located in this comm. inity rather than anothersimilar community because there is a more prestigious college here

'Many of the subcategories for this category were suggested by the economic outputs classification developed by Goodman (1971)

40

47

CategoryCode Number 2000 HUMAN CHARACTERISTIC OUTCOMES

2100

2110

2120 I

2130

2140

AspirationsLevels, patterns, and directions (in persons, groups, organizations, or communities) of Interests, desires, drives,ambitions, goals, and intentions

(Categories)

Desires, Aims, or Goals Places, conditions, things, or other endsthat Individual!' and/or groups crave, toward which they haveambition, or that they Intend to reach because of Importance tothem.

(Examples of Outcome Measures of indicators)

Changes In observed desires from college entrance to graduation

Changes in the reported aspirations for graduate school as a classproceeds through undergraduate school

Self-repo,1 of changes In goals and aspirations as a result ofcollege

Dislikes, Litos, and InterestsThe persons or types of persons,objects, content areas, occupations and other things andsituations for which there Is a preference or antipathy

The reported likes and dislikes of persons before college ascompared to after graduation, and comparison with such changeover the same period of time for those the same age lot att.- .:ingcollege.

Score or change in score on an Interest Inventory, e 9., StrongVocational Interest Blank, Kuder General Interest Survey, KuderOccupational Interest Survey, ACT Interest Inventory.'

Self-report of changes In interests as a result of college

Motivation or Drive LevelThe intensity of striving toward a goaltnat is elicited by a need or other stimulus

Score or change in score on an instrument that measures "need forachievement" or "achievement motivation," e g , the StrongVocational Interest Blank Academic Achievement Scale, theCollege Student Questionnaire Motivation for Grades Scale, theCalifornia Psychological Inventory Achievement Scales, PersonalValue inventory

Self-report of changes in motivation level as a result of college

Other Aspirational Outcomes

2200

2210

2210

Competence and Skills Levels, patterns, and direction of ability, capability, proficiency, and talent of different kinds

(Categories)

Academic SkillsThe amount of ability or competence In takingtests, earning good grades, persisting in college, etc withoutregard to the amount of cognitive learning that has taken place


Grades earned when the effect of ability, motivation, and otherRuch factors have been controlled

Persistence In college when the effects of ability, motivation, andother such factors have been cancelled out

Score or change in score on a test of study skills, e g , Brown-Holtzman Survey of Study Habits and Attitudes, ComprehensiveTest of Basic Study Skills

Citizenship and Family Membership SkillsThe ability orcompetence to perform relative to the rights, duties, and privilegesof a member of a family, community, state or nation, for example,competence in managing family finances, being an effectiveconsumer, and evaluating political Issues

Self-report of abilities pertaining specifically to citizenship andhome membership that college accentuated

Evaluation by others of citizenship and home memberah:p skillsmastery exhibited

Score or change in score on the Vineland Social Maturity Scales

As an example, one could it graduation compare Interest test scores of college students to a group of their high school classmates not attendingcollege who had similar interests In high school. As another example, one could look at Interest test change scores for college students, adjustedfor Initial Wel.

41

48

CategoryCode Number

2000 HUMAN CHARACTERISTIC OUTCOMES (ccntinued)

2230

2240

2250

2290

2270

2280

2290

Creativity SkillsThe amount of ability or competence in

designing, producing, or otherwise bringing Into existence originalperspectives, explanations, and implementations

Score or change in score on a test that measures originality andcreative ability, e g , Minnesota Teat of Creative Thinking, Test ofCreative Ability, Guilford's Alternate Uses Test, SixteenPersonality Factors Questionnaire Creativity Scale

Evaluation by Judges of creative ability demonstrated in a buildingor forming task

Expression and Communication SkillsThe amount of ability orcompetence In conveying Information, attitudes, or emotions on aone-to-one basis and/or to large or small groups or populations,by whatever media, in order to inform, challenge, uplift, and/orpersuade, etc., and in receiving and interpreting suchcommunicationsthrough reading, writing, speaking, listening,touching, body movement, silence, and cultural arts like acting,painting, sculpturing, singing, playing musical instruments, etc

Score or change in score on tests that measure the ability tocommunicate or express oneself

Judges' rating in a debate or speech contest

Judges' rating of expression In a music, art, or ballet contest

intellectual SkillsThe amount of ability or competence informulating and analyzing problems, comprehending andunderstanding, synthesizing information evaluating information,implementing a solution to a problem, and In locating, retaining,and filtering relevant knowledge

Score or change in score on a test that measures ability to analyzeand solve problems and to make inferences, e.g., California Test ofMental Maturity, Watson-Glazer Critical Thinking Appraisal,California Psychological Inventory Intellectual Efficiency Scale

Self-report of changes in analytical ability as a result of college

Mtorporsonal, Leadership, and Organizational SkillsThe amountof ability or competence In effectively living and interacting withothers, social organizing, being a congenial friend and companion,establishing courses of action for others, and influencing others tofollow

Leadership awards

Self perceptions and evaluation of interpersonal and leadershipability

Perceptions by judges of Interpersonal and leadership skills

Score or change in score on a teat that measures leadership andinterpersonal ability, e g , California Psychological InventoryLeadership Scale, Chapin Social Insight Scale.

Occupational SkillsThe amount of ability or competence in thespecial, unique skills required by particular occupations, and inseeking, gaining, and maintaining a particular level and kind ofemployment

Spatial relations test scores for someone whc Is, or is going to be,an artist

Demonstrated ability In writing FORTRAN or COBOL for someonewho is, or Is going to be, a computer programmer.

Score or change in score on the Bennett MechanicalComprehension Test.

Physical end Motor SkillsThe ability or competence in tasksrequiring physical coordination, dexterity, manipulation, and othermuscular or motor skills

Score or change in score on tests that measure motor skills, e.g ,Crissey Dexterity Test, Minnesota Rate of Manipulation Test

Judges' scores on skill events in athletic competition such asgymnastks, divinii, and figure skating

Other Skill OutcomesExamples are the ability to teacheffectively, to handle one's leisure, etc.

2300

2310

Morale, Satisfaction, end Affective CharactoristicsLevels, patterns, and directions of characteristics typified by emotion

(Categories)

AttitudesThe disposition or tendency to respond eitherpositively or negatively to particular persons or types of persons,things, situations, etc. It Is a predisposition to act In a certain way

ANDVakiesA strong preference based on a conception of what Isdesirable, important, and worthy of esteem. Values affect anIndividual's actions and thoughts toward others.


Score or change in score on an attitude scale, e g , Thurston andChaves Scale for Measuring Attitudes Toward the Church, CollegeStudent Questionnaire Pail I, Adorno Ethnocentrism Scale, Shawand Wright Scales for the Measurement of Attitudes

Self-report of one's attitudes and the effect of college on them.

Score or change in score on an instrument that assesses values,e g , Alport-Vernon-Lindsey Study of Values, Differential ValueProfile, Work Values Inventory.

Self-report of one's values and the effect of college on helping tocisrlfy them

42

49

CategoryCode Number 2000 HUMAN CHARACTERISTIC OUTCOMES (continued)

2320

2330

2340

2350

Be /lets, Commitments, and Philosophy of LifeThe acceptanceand Internalization of particular propoaitlons or declarationa, theparticular things that one in convinced are true The held view ofwhat "man" is, the purposes and reason for a person's existence,and the system of principles and laws that should govern his/herthought, morals, character, and conduct or behavior Included isthe promotion of and the adherence to the conventions, practices,and teachings of religious organizations or sects

Score or change in score on Instrumenta that assess bellefa, e g ,Harvey's Conceptual Syatems Test, Inventory o: Beliefs

Self-report of one's beliefs and commitments and the effect ofcollege on them.

The membership and participation in, and support of, a particularreligious organization or cause prior to as compared with aftercollage

Self-report of one's philosophy of life and the effect of college onclarifying and organizing it

Feelings and EmotionsThe diapoaltion or tendency to respond ornot respond aubjectively to stimuli and the ability ' control or notcontrol such expresalona, I e , feelings of anguish or diatresa,anticipation, anxiety, concern, contentment, empathy, excitement,fear, fruatratlon, happiness and Joy, humor, lethargy, love,pleasure, satisfaction, sorrow, etc

°Demean and acceptance of feelings before college compared toafter college.

Development of an appreciation of different cultures and a widerange of human values as a result of college

Greater reported satiafact ion with life as result of college

Mores, Customs, and Standards of ConductSocial and culturalpractices, rules, and conventions designed to guide personal andcorporate behavior They have strong ethical or moral significanceaccording to tradition and are enforced by social disapproval ofviolations

Self-report of the effect of college on assimilation or internallt Monof the customs of community or society.

Score or change in score on the California Psychological ImintorySocialization Scale.

The adherence to particular mores or social customs prior tocollege as compared to after college.

The amount of subjectivity and emotion guiding one's standards ofconduct prior to college as compared to after college

Other Affective Outcomes

2400

2410

2420

2430

2440

2450

Perceptual CharectoristicsLevels, patteme, and directions of consciousness, awareness, and sensitivity exhibited, and the view(s)or conuipt(s) of self, others, aurroundinge, events, Ideas, etc.

(Categories)

Perceptual Awareness and SensitivityThe amount of conscious-noes or awareness e, or sensitivity to, stimuli that are exhibited byindividuals or groups.

(Examples of Outcome Measures or indicators)

Increased sensitivity to needs and emotional cues provided byothers

Increased alertness to the opportunities confronting one.

Perception of Self The view held about oneself, the character-1st Ica that are perceived, I e., self concept

Development of positive self-regard and self-confidence u a resultof college

Sccre or change In score on a self-co ncept scale, g., AdjectiveCheck Liet, Califomia Psychological inventory Self AcceptanceScale, Tennessee Self Concept Scale.

Perception of OthersThe manner in which other Individuals andparticular groups of others are viewed or perceived, thecharacteristics that are perceived.

Reports by observers about how person's respect for °than haschanged as 'result of college

Sel*-report of how Ones view of others has changed as a result ofcollege

Perception of ThingsThe view one holds (I e., the characteristicsnoted) of Ideas or other things being examined with the physicalaensea

increased respect for the Ideas of others u result of college.

Movement as &result of college experiences from seeing things asall "black and white" to complex "grays "

Other Perceptual Outcomes

43

50

C4tegoryCode Number 2000 HUMAN CHARACTERISTIC OUTCOMES (continued)

2500

2510

2520

2530

2540

2550

2560

2570

Persona lit) and Personal Coping CharacteristicsLevels, patterns, and directions of human conditions, factors, and traits relatedspecifically to the mind and mental processes (other than skills, knowledge, and understanding).

(Categories)

Adventurousness and InitiativeWillingness to take chances andrisks, how daring an individual is, willingness to take a stand orspeak out, willingness and capacity to Initiate personal action or tobecome actively involved


Reports by impartial observers of changes In Initiative that seem tohave resulted from college attendance.

Self-report of the effect of college on one's willingness to take achance, e g , to take an educated guess on an exam

The frequency that one exhibits speaking out on issues as theCollege career progresses

Autonomy and IndependenceThe amount of freedom fromcontrol and influvnce of others that is exhibited

Score or change in score on personality scales that measureautonomy and independence, e g., Sixteen Personality FactorsQuestionnaire Group-Dependent vs Self-Sufficient Scale,Edwards Personality Inventory Independent In His Opinions Scale,College Student Questionnaire Independence Scales, OmnibusPersonality Inventory Autonomy Scale

Self-report of willingness to volunteer or "stand up for one's rights"and the effect of college attendance on such willingness.

Dependability and ResponsibihtyThe amount of reliability,trustworthiness, and accountability for own behavior that leexhibited

Reports by observers of changes In dependability andresponsibility that have occurred during college.

Score or change in score on scales that measure dependability andresponsibility, e g., California Psychological inventory Responsi-bility Scale, Edwards Personality Inventory Assumes Responsi-bility Scale, Sixteen Personality Factors Questionnaire Expedientvs Conscientious Scale

Dogmatism, Authoritarianism, and Open-MindednessTheamount of open-mindedness, assertiveness, unassertiveness,and/or unquestioning obedience to authority that Is exhibited

Reports of expert observers about changes in open - mindednessthat have taken place during college.

Score or change in score on a scale that measures dogmatismand/or authoritarianism, e.g., Rokeach Dogmatism Scale,California Psychological Inventory Dominance Scale, OmnibusPersonality inventory Religious Orientation Scale

Flexibility and AdaptabilityThe amount of adjustment to new andchanging situations and circumstances that is exhibited.

Score or change In score on a scale that measures flexibility, e.g.,California Psychological inventory Flexibility Scale, OmnibusPersonality Inventory Practical Outlook Scale, Sixteen PersonalityFactors Questionnaire Practical vs. Imaginative Scale.

Reports by observers of changes in adaptability and flexibility thathave occurred during college.

Self-report of the effect of college on adaptability and flexibility.

HabitsThe tendency to perform certain actions or to behave Incharacteristic, automatic ways

Observations by others of changes in habit orientation that haveoccurred during attendance

Self-report of changes in habits that have resulted from college.

Psychological FunctioningThe amount of psychologicaladjustment, contact with reality, self-understanding, and self-actualization (optimum self-realization) that le exhibited

The amount of realization of one's actual strengths andweaknesses, and of what is reality.

Score or changes In score on an instrument that measurespsychological adjustment, g., Minnesota Multlohealc PersonalityInventory, Sixteen Personality Factors Questionnaire, MoodyProblem Check List

Reports by expert observers about changes in the psychologicalfunctioning of individuals that have occurred during 0011101attendance

44

51


2580

2590

Tolerance and PersistenceThe amount of endurance, tenacity,forbearance, patience, and restraint that Is exhibited

Observations by others of changes In tolerance and persistenceduring college

Score or changes In score on an Instrument that measurestolerance and persistence, e g , Edwards Personality InventoryPersistence Scale, California Psychological Inventory ToleranceScale

Other Personality and Personal Coping Outcomes

2800

2810

2820

2830

Physical and Physiological CharacteristicsLevels, patterns, and directions of human body traits and processes (other than skillfunctioning)

(Categories)

Physical Fitness and TraitsPhysical and physiologicalcharacteristics such as toughness, .endurance, strength, speed,flexibility or dexterity, physical energy, murilar control, size,vocal characteristics, etc


Score or change in score on physical fitness tests, .g., AAHPERYouth Fitness Tests, Basic Fitness Testa

Self-report of "feeling in better physical shape" as a result ofcollege.

Physiological HealthThe physical well-being of Individuals, howwell the system of normal bodily operations is functioning

t.ledical doctor's health physical examination report at collegeentrance compared to at college graduation

Self-report of the effect of college attendance on how well alumnitake care of their bodies

Other Physical or Physiological Outcomes

2700

2710

2720

2730

Status, Recognition, and CertificationLevels. patterns, and direction concerning recognition of accomplishments, power, prestige,reputation, etc

(Categories)

Completion or Achievement AwardA certificate, diploma, orsome other award for having completed a course or program, forsome demonstrated proficiency, or for accomplishment of sometype


An honorary degree

Graduation diploma.

Alumni achievement award

Sales award or a Job promotion

Danforth Fellowship Award

Being named a Rhodes Scholar

Credit RecognitionFormal or informal acknowledgement of workcompleted or of confidence, trust, approval, etc

Graduate school grades.

Credit hours given for completing a course

By-line credit for a movie, play, book, or article

Financial credit rating Issued by a bank or credit bureau

Image, Reputation, or StatusThe amount of fame, distinction,respect, and standing In the eyes or the profession, thecommunity, or some other group.

Being on the social register.

Being listed In Who's Who.

Oral and written akanowledgernents from others

Being interviewed by the press, radio, or TV

Writing an autobiography that is published or having a biographywritten about you.

2740 Licensing and CertificationFormal written authority that aperson or firm is qual,fied and has met the test to practice someskill or speciality occupation

Entry into the state bar

Passing a cosmetology licensing exam

Being a certified public accountant

An insurance company that has been licensed to sell in a state.

45


2750

2780

2770

2790

Obtaining a Job or Admission to a Follow-Up ProgramSuccessIn being selected for a postgraduate employment position or aspecial educational program at a higher level

Entrance to a university after graduation from a communitycollege

Entrance to law, medical, or graduate school

Being selected by the civil service

Being selected for a company executive position

Being hired in the special field foi which the training applied

Power and/or AuthorityThe amount of acknowledgedauthorization or ability to influence, command, enforce obedience,or set policy as a right of rank, position, delegated jurisdiction,skill, strength, wealth, etc

Appointment or election to a position of authority.

Earning promotion to a position of authority

Influencing important .-..vmmunity or public decisions

Getting acknowledged credit for the important Job having gottendone

Job, School, or Life SuccessEvidence of success In one'soccupation or career, In graduate or professional school, or Insome other aspect of one's life that Is covered In any of the abovecategorise.

Self report of success in career

Teacher's rating of success In graduate school

Employer's rating of overall on-the-job performance

Other Status, Recognition, and Certification Outcomes

2900

2810

2820

2830

2840

2050

Social Activities and RolesLevels, patterns, and directions of social functions assumed and carried out

lam.

(Categories)

Ad/ustmant to RetirementAltering self and lifestyle to meet theneeds and adapt to the limitations of the retirement years


Percentage of college educated retirees reporting productiveretirement years compared to reports of those who never attendedcollege.

Self-report of the effect of having attended college on theretirement years.

AffiliationsFIndIng appropriate organizations and institutions toJoin and associate with, and being accepted by them

Number of affiliations and changes In affiliations for collegegraduates as compared to those never attending college.

Self-report of the effect of having attended college on theaffiliations sought and on the affiliations won

Avocations! and Social Activities and RolesFinding, pursuing,and achieving rswaroing nonwork activities, hobbles, and parts toplay in society, and exhibiting that pattern of behavior that Isexpected of persons having the status that has been earned.

The social roles and avocations of college graduates as comparedto those who never attended college

Self-report of the effect of having attended college on theavocational and social roles sought, and on those practiced.

Career and Vocational Activities and RolesExhibiting thepatterns of behavior expected and/or that are needed for the part Inthe "world of work" that has wen accepted or entered Into.

The career roles of college graduates as compared to those whonever attended college.

Reports of employers concerning the advancement and roles ofcollege trained employees versus the advancement andoccupational roies of those who never attended college.

Citizenship Activities and RolesFacilitating and contributing togovernmental functions and to the overall well-being ofIndividuals, the community, and larger society.

Percent voting in a municipal or state election

Financial and other contributions given to service organizations

Percent running for public offices c; camps fining for someone whois

4653


2880

2870

2880

Family AchwtieS and RolesContributing to and facilitatingfamily functions, I e , parent roles, sibling rclea, Son/daughterroles, etc

The family roles of college graduates as compared to those whonever attended college

Self-report of effect of the college on the roles played in one'sfamily

Friendships and RelationshipsSocially Interacting with andentering into and sustaining Intimate, in-depth, and satisfyassociations with others

Characteristics of friends and relationships of college ediratedpeople versus those never attending college

Self-report of the effect of college on friendships and socialrelationships

Other Activity and Role Outcomes

2900 Other Human Characteristic Outcomes

47 5 4

CategoryCode Number 3000 KNOWLEDGE, TECHNOLOGY, AND AHT FORM OUTCOMES

3100

3110

3120

3130

3140

General Knowledge and Understanding' Familiarity with, analysis and cor, ,7.-ehensinn of, and application of facts and principlesacross broad areas of study breadth of knowledge and understandingas a result of dissemination through educational teaching-learning activities

(Categories)

Knowledge and Understanding of General Facts r7 c1

Terminology Knowlnc about and understanding, and having anadequate vocabulary to be able to describe, the reality, existence,and circumstances of particular sensory (observed, heard, felt,etc ) phenomena, objects, people, products, events, conditions,etc , or components thereof.


Students' scores or changes In score on standardized or classroomtests that measure knowledge and understanding of generalterminology and/or facts. For example, the Miller Analogies Testfocuses entirely on knowledge ano understanding of generalterminology, and finis like the College ..eeel Examination Program(CLEP) or the Graduate Record Exam (GRE) general exam includecoverage of general terminology and facts

Students' self-report of snowledge and understanding aboutgeneral terminology and facts

Knowledge and Understanding of General ProcessesKnowingabout and understanding customs, rules and standards forjudgments, guidelines, processes, methods, procedures,techniques, trends, and other ways of applying anti making use ofterminology and facts.

Students' scores or changes In score on standardized or classroomtests measuring comprehension of general conventions,processes, and methodologies

Students' grades In a general application survey course

Knowledge and Understanding of General TheoryKnowing aboutand understanding principles and generalizations, theoreticalformulations, hyi 'theses, supposition, conjecture, etc

Students' scores or changes in score on standardized or classroomtests measuring comprehension of general theories In a oad fieldof study

Students' grades in a general survey course on theories ofphilosophy

Other General Knowledg Ind Understanding

3200

3210

Specialized Kno *Midge and Understanding' Familiarity with, analysis and comprehension of, and application of facts and principlesin particular specialized fields of studydepth of knowledge and understandingas a resul! of dissemination through educationalteaching /reaming activities.

(Categories)

Knowledge and Understanding of Specialized Facts andTerminologyKnowing shoat and understanding, and having anadequate vocabulary to be able to describe the reality, existence,and circumstances of particular sensory (observed, heard felt,etc ) phenomena, objects, people, products, events ecnditiona,etc , or components thereof


Students' scores or changes in score on standardized or classroomtests that measure knowledge and understanding In a narrow,specialized area of study Professional certification and licensingexams usually focus on this type of knowledge, as do tests like theCollege Level Examination Program (CLEP) Subject exams or theGraduate Record Exam (GRE) area exams.

Students' self-report of knowledge and understanding aboutspecialized terminology and facts

3220

3230

3240

Knowledge and Understanding of Specialized ProcessesKnowingabout and understanding customs, rules and standards forjudgments, guidelines, processes, methods, procedures,techniques, trends, and other ways of applying and making use ofterminology and facts.

Students' scores or change:, in score on standardized or classroomtests measuring comprehension of conventions, prrscesess,methodologies, and techniques unique to particular ^ializedprofessions or disciplines

Students' grades Ina specialized professional course or program.

Knowledge and Understanding of Specialized Theory Knowingabout and understanding principles and geneiellzations,theoretical formulations, hypotheses, supposition, conjecture,etc

Other Specialized Knowledge and Understanding

Qtudents' scores or changes In score on etendardlzed or claw(tests measuring comprehension of specialized theoretitfoi nulations ar models

Students' 1;rades In a course that goes Into depth about one ormore theories or models unique to a specialized discipl'ne orprofession

3300 Research and ScholarshipKnowledge and understanding, techniques, and physical products resulting from basic and appliedresearch and scholarship.

The subcategories used for this category came from Bloom (1956)

4855

CategoryCode Number 3000 KNOWLEDGE, TECHNOLOGY, AND ART FORM OUTCOMES (continued)

3310

3320

(Categories)

Research and Scholarship Knowledge and UnderstandingThediscovery, development, preservation, and professional dissemin-ation of knowledge and understanding resulting from activitiesconducted in basic and applied research and scholarship


Average number of basic research publications, applied researchpublications, textbooks, or monographs etc , per student, formerstudent, and/or faculty member over a specific period of time

Number of 'faculty members and/or former students in thesciences listed In American Men of Science

Research and Scholarship ProductsApplied techniques (forexample, a new therapy treatment in the field of medicine or a newtechnique in the field of music) and physical products (forexample, a new or refined serum) developed from basic and/orapplied research and scholarship.

Average number of patents and /or copyrights received per student,former student, and/or faculty member over a given period of time

Average number of awards and citations received per student,former student, and/or facutiy member (over a given period oftime) for discovery or development of technological products.

3400

3410

3420

3 -....1

3440

3450

Art Forms and WorksReproducing and preserving existing artistic tams and works, and developing new or revised artistic forms and worksis.(Categories)

,-hitroctureOutcomes involving the design for construction ofcalidings, landscape, living complexes, etc.


Number of architectural works completed by students, formerstudents, and/or faculty

Number of awards and other recognitions revived for architecturalworks on the campus commissioned by campus officials.

DanceOutcomes Involving preservatlon or development offorms, works, and performances in tae an of dance.

Number of former students receiving recognitior for performancesIn this area

Number of students Involved In dank.* auditions and publicperformancea.

Debate and OratoryOutcomes Involving preservation ordevelopment of fo- nand performarces in the oratory ans.

Competition record over a period of years of the cortege's debateteam

The average number of graduates each year who go on to somesind of oratorical career

DramaOutcomes involving ti J preservation or development offorms, works, and performances In the professional and amateurtheatrical arts.

The number of students who enter a profess:onel acting career,and the number acting on an amateur basis.

The number of drama performances put on for the local communityeach yew.

Literature and WritingOutcomes involving the preservation ordevelopment of forms and works in the production cl prose, verse,and other writings.

The average number of literary works each year published bystudents, former students, and/or faculty members

The number of students and faculty each year who have entered aformal s'ate or national writing competition

3480

3470

3480

3490

MusicOutcomes involving the preservation or development offorms, works, and performances In the professional and amateurtheatrical arts

The number of musical productions put on each year by the collegathat are open to the public.

The number of studs 'volved In public music recitals and otherperformances

Painting, Drawing, and PhotographyOutcomes Involving theprow. vation or development of forms or works in the graphic andpictorial arts

The number of paintings, and their qu.lity in the campus angallery.

The number of awards won over a certain period of time forpictorial works by students. former students, and facultymembers

SculptureOutcomes involving the preservation or development offorma or works In the carving, chiseling, .;.*sting, modeling, orother sculpturing areas

The number of sculptures that have been commissioned by thecollege and placed throughout the campus

The forms of sculpture that have been developed on the campus

Other Fine Arts

350(/ Other Knowledge, Technology, and Art Form Outcomes

49

5R

CategoryCode Number

4000 RESOURCE AND SERVICE PROVISION OUTCOMES

4100

4110

4120

Provision of Facilities and EventsThe availability, use, and partwipation In campus happenings, buildings, equipment, and otherresources by students, other individuals, and particular groups or communities

(Categories)

Prowsion of FacilitiesAvailability and use of campus grounds,buildings, rooms, equipment, etc


Number of facilities made available to the students during aparticular period of time

Total number of hours each faculty was used by people in thecommunity, and the number of people-hours of use over a specificperiod of time

Provision or Sponsorship of EventsAvailability and participationin happenings on the campus or off that are provided or stimulatedLy the college or one of Its components

The numoer of people who attended athletic events, culturalevents, or other events )4ovided and/or sponsored Ly the collegeIn any one year

The number of column inches of newspaper coverage received byspecific events in local, regional, and national newspapers

4200

4210

4220

4230

Provision of Direct Services The availability, use, and receipt by students, other individuals, and particular groups or communities ofass istam.e, care, or other service

(Categories)

Teaching'Aclvitles and programs designed to instruct and toImpart knowledge, skills, attitudes, etc.

(Examples of Outcome Measures "r Indicators)

Average number of counsel' taught and number of contact hoursper semester In the regular program.Extension courses provided In any one calendar year.

Advisory and Analytic Assistance Activities and programsdesigned for the purpose of (upon request) offering suggestions,recommendations, counsel, information, calculations, andstudies

Number of advisory and analytic assistance services offered tostudents, staff, and/or to the public

Number of person-hours spent by staff in providing this assistanceover a specific period of time.

Treatment, Care, and Referral ServicesHelping and directassistance services, other than those above, provided by theinstitution, Institutional units, and/or Institutional staff

The treatment, care, and referral services offered by the Institutionand Its staff, and health services, day care for children of workingmothers, counseling, crisis referral, and drug treatment and theamount these services are used.

The reported satisfaction of U.0111 of these services with thetreatment and care received.

4240

Provision of Other ServicesAn example would be direct civicleadership provided to the community. Another example would beoffering keypunching service.

4300Other Resource and Service Provision OutcomesAn example would be tne attention and good will the college draws to the localcommunity because ft Is Jaded there.

some people would consider teaching to always be a producer /facilitator activity that leads to outcomes Others would, however, considerteaching to be an output that results from the interaction of faculty, equipment, students, and other educational resources Those who hold thefirst viewpoint should just Ignore this category

...

5750

CategoryCode Number

5000 OTHER MAINTENANCE ANI1 CHANGE OUTCOMES

5100 Aesthetic-Cultural ConditionsPreserving or bringing about changes In tastes, level and kinds of aesthetic-cultural emphasis,aesthetic-cultural availability and opportunities, aesthetic-cultural activity and participation, etc

5200 Organizational F0111141, Activity, and OperafronsFor organizations, groups, and systems (and their components), maintenance orchange in organizational communications, operational methods and interaction, operational effectiveness, organizationalrelationships, organizational arrangement and configuration, organizational activities and programs, and other such organizationalcharacteristic outcomes

5300 Other Maintenance and ChangeOutcomes not covered by any of the other subcategories of "Maintenance" and "Change" in thisdimension of the Outcomes Structure An example might be "destruction of life support in the environment "

51

Appendix C

An Example of Outcomes Classified with the Structure UsingOutcome Statements from the NCHEMS Outcomes Inventory

OUTCOME CATEGORY OFTHE NCHEMS INVENTORY

CORRESPONDING CATEGORY OR CATEGORIESOF THE OUTCOMES STRUCTURE TYPE-OF-OUTCOME DIMENSION'

1 1 1 00 Knowledge Development1 1 1 01 General Knowledge1 1 1 02 Specialized Knowledge

1 1 2 00 Skills Development1.1 2 01 Application of Knowledge Skills1 1 2 02 Critical Thinking and Reasoning Skills1 1 2 03 Creativity Skills1 1 2 04 Communication Skills1 1 2 05 Motor Skills

1 1 3 00 Knowledge and Skills Attitudes, Values, and Beliefs

1 1 3 01 Intellectual Disposition

1 2 1 00 Social Skills Development1 2 1 01 Interpersonal Participation1 2 1 02 Leadership1 2 1 03 Citizenship

1 2 2 00 Development of Social Attitudes, Values, and Beliefs

1.2 2 01 Political1 2 2 02 Racial/Ethnic

1 2 2 03 Personal Ethics1 2 2 04 Social Conscience

1 2 2 05 Socioeconomic Aspirations

1 3 1 00 Student Health Development

1.3 1 01 Physical Health1 3 1 02 Mental Health

1 3 2 00 Student Personal Attitudes, Values, and Beliefs

1 3 2 01 Religious and Spiritual1 3 2 02 Change/Stability

1 3 2 03 Self-Concept

1 4 1 00 Career Preparation

1 4 1 01 Academic Preparation1 4 1 02 Vocational Preparation

1 4 200 Career Attitudes, Values, and Beliefs

1 4 2 01 Achievement Orientation1 4 2 02 Educational Aspiration1 4 2 03 Educational Satisfaction1 4 2 04 Vocational Aspirations

2 0 0 01 Discovery of New Knowledge

2 0 0 02 Interpretation and Application of New Knowledge

0 03 Reorganization 0 New Knowledge

(Continued)

3002 Knowledge, Technology, and Art Form Change Outcomes3102 Change in General Knowledge and Understanding3202 Change in Specialized Knowledge and Understanding

2202 Change in Competence and Skills2252 Change In Intellectual Skills2252 Change In Intellectual Skills2232 Change in Creativity Skills2242 Change In Expression and Communicatis Skills2282 Change In Ph; sical and M.,or Skills

2312 Change In Attitudes and Values2322 Change In Beliefs, Commitments, and Philosophy of Life2122 Change In Desires, Alms, or Goals

2262 Change In Interpersonal, Leadership, and Organizational Skills2262 Change In Interpersonal, Leadership, and Organizational Skills2262 Change in Interpersonal, Leadership, and Organizational Skills2222 Change In Citizenship and Family Membership Skills

2312 Change In Attitudes and Values2322 Change in Beliefs, Commitments, and Philosophy of Life2312 Change in Attitudes and Values2312 Change In Attitudes and Values2432 Change in Perception of Others2312 Change in Attitudes and Values2312 Change in Attitudes and Values2322 Change In Beliefs, Commitments, and Philosophy of Life2102 Change In Aspirations

2622 Change it physiological Health2572 Change in Psychological Functioning2622 Change in Physiological Health2572 Change in Psychological Functioning

2312 Change in Attitudes and Values2322 Change in Beliefs, Commitments, and Philosophy of Life2322 Change in Beliefs, Commitments, and Philosophy of Life2312 Change in Attitudes and Values2442 Change in Perception of Things2422 Change in Perception of Self

2272 Change In Occupational and Employability Skills3102 Change in General Knowledge and Understanding3202 Change In Specialized Knowledge and Understanding2212 Change in Academic Skills2272 Change l:, Occupational and Employability Skills3102 Change in General Knowledge and Understanding3202 Chance im Specialized Knowledge and Understanding2312 Change in Attitudes and Values2322 Change in Beliefs, Commitments, and Philoaphy of Life2132 Change In Motivation or Drive Level2102 Change in Aspirations2332 Change In Feelings and Emotions2102 Change in Aspirations

3312 Change In Research and Scholarship Knowledge and Understanding

3312 Change In Research and Scholarship Knowledge and Understanding

3312 Change in Research and Scholarship Knowledge and Understanding

(Continued)

'More than one Outcomes Structure category may pertain to a single NCHEMS Outcomes Inventory category In order to best conform to that category

53

59

OUTCOME CATEGORY OFTHE NCHEMS INVENTORY (Continued)

CORRESPONDING CATEGORY OR CATEGORIES OF THEOUTCOMES STRUCTURE TYPE-OF- OUTCOME DIMENSION (Continued)

3 1.0 00 Community Development" 2002 Change In Human Characteristics3.1.0 01 Community Educational Development 2002 Change in Human Characteristics3.1.0 02 Facult /Staff Educational Development 2002 Change in Human Characteristics

3 2 0 00 Community ServIce' 4002 Resource and Service Provision Outcomes3 2 0 01 Extension Services 4202 Provision of Direct Services3 2 0 02 Personal Services 4202 Provision of Direct Services3 2 0 03 Extramural Cultural and Recreational Services 4102 Provision of Facilities and Events

4202 Provision of Direct Services3 2 0.04 Financial Impact on the Community 1000 Economic Outcomes

3 3 000 Longer Term Community Effects" 0002 Change Outcomes3 3 0.01 Social Impact 2002 Human Characteristics Outcomes3.3.0.02 Economic Impact 1000 Economic Outcomes

'The "audience" dimension would account for "Community" and the "time" dimension of the Outcomes Structure would account for "longer term"

60

54

Appendix D

Possible Copy for a Brochure or Flier IntroducingOutcomes Concepts and the Outcomes Structure

Introducing the NCHEMS Outcome Structure

A Flexible, Practical Tool to Help Faculty andAdministrators in Their Planning, DecisionMaking, and Communication about EducationalOutcomes

1. What Is an Educational Outcome in Post-secondary Education?Postsecondary education outcomes are theend results of the processes that occur withinpostsecondary education institutions andprograms, such as instructional activities,research and scholarship activities, and com-munity service programs. They include boththe direct results of these processes and anyshort-term and long-term consequences ofthose direct results. Some involve maintenanceof skills, standard of living, and other individ-ual and social conditions, while others in-volve primarily change of different types. Fur-thermore, some of them may be intended out-comes while others are unintended. Althoughoutcomes are generically neutral, people at-tach positive and/or negative values to them,and see them as performing or not performingparticular useful functions for individuals, aparticular community, particular aggregates ofpeople (for example, "handicapped people"),or other entities such as the environment. Onecan focus on outcomes for students, for thelocal community, or for some other group.

2. What Is the NCHEMS Outcomes Structure?The NCHEMS Outcomes Structure is a flexible,integrated, and comprehensive framework andsystem for organizing outcomes and outcomesinformation for purposes of classification,analysis, and decic!on making. It wasdeveloped by the NCHEMS staff over morethan two years of concentrated effort, andrests upon many significant bodies of previouswork. Preliminary trial use and review of thieStructure suggests that it is applicable to thefull range of educational outcomes and has anumber of quite practical uses for faculty andadministrators on college and university cam-puses.

3. Why Was the Outcomes Structure Developed?

Without agreement on a common languageand context for outcomes, it is difficult for in-stitutional faculty and staff to communicatesuccinctlywith each other across disci-plinary lines and with important publicswhattheir institution and/or program is accom-plishing and how it differs from its counter-parts. Such communication is especiallyimportant in this "age of accountability." Priorto now, many people in our institutions havefound it uncomfortable to think and makeplans in terms of outcomes, let alone talk con-cretely in such terms, preferring instead tofocus exclusively on activities, process, and"process quality." Furthermore, the potentialvolume of outcomes information in post-secondary education is quite formidable,which poses a significant barrier to using out-come information in institutional and programplanning and management. Although a numberof attempts have been made in the past todevelop such a system, none of them includedthe full range of possible outcomes (whichlimiad their applicability), they were often in-flexible, and in other ways they provedinadequate to meet some practical needs.

4. Whit Practical Uses Does the OutcomesStructure Have?A. CommunicationThe terms and defini-

tions in the Structure can serve as a work-able common language where the out-come concepts and relationships remainconstant from one use of the outcomes in-formation to another, and it can thusfacilitate effective communication aboutoutcomes.

B. StimulationThe Structure can help createan awareness of postsecondary educationoutcomes and of the need for focusing onthem, plus it can serve as a device that willstimulate people to focus on outcomes in amore systematic and coliciete manner andto discuss them.

C. Planning, Mc-.nagement, and EvaluationThe Structure can serve as a tool to assistdirectly in , he following functions: needsassessment, goal identification, goalassessment, goal translation, resource

55

61

IIMMIe.

allocation, program planning, evaluation,and outcomes information storage andretrieval.

D. ResearchThe Structure can suggest gapsthat need to be researched, research databank applications, hypotheses for study,and so forth.

The Outcomes Structure was designed to be ofuse, in the ways described above, at the in-stitutional and program level. However, a num-ber of faculty who reviewed the Structure andtried it out in a preliminary manner expressedthe opinion that it also could be helpful to in-structors in planning individual courses.Several students and a student personnelworker felt it could be useful in helping stu-dents plan what they want to get out of college.

5. What Is the Makeup of the Outcomes Struc-ture?The Outcomes Structure is a systematic andcomprehensive way to categorize the out-comes of postsecondary education in terms ofthree "dimensions" and to relate outcomes toone another in a logical pattern, or to "struc-ture" them.

The three dimensions are "audience," "type-of-outcome," and "time," and they areillustrated graphically below:

AUDIENCEDIMENSION

(Who or what receive:or is ,Itected by theoutcome or is intended to rece,ve or beal I ected by in

TYPE OF OUTCOME DIMENSION

(What basic entity is or is intended tobe maintained or changed')

TIME DIMENSION

(When is the outcome expected tooccur or Wu en doesIt occur')

The "audience" dimension is divided into thefollowing five broad categories (which are inturn further divided into appropriate sub-categories):1. Individual/Group Clients2. Interest-Based Communities3. Geographic-Based Communities4. Aggregates of People5. Other Audiences

The "type-of-outcome" dimension is also

divided into five broad categories, each ofwhich has additional subdivisions:1. Economic Outcomes2. Human Characteristic Outcomes3. Knowledge, Technology, and Art Form Out-

comes.4. '-iesource and Service Provision Outcomes5. Other Maintenance and Change Outcomes

The categories and subcategories needed forthe "time" dimension vary according to theaudience and type-of-outcome of concern.Therefore, formal categories and sub-categories are not included in the structure forthis dimension.

Standard definitions are provided for eachcategory and subcategory of each dimension,and straightforward procedures are providedfor "extending the Structure" in the caseswhere still more specificity and detail areneeded. 4 coding system is also provided, forthose needing it.

6. Where Can I Find More Information?The following two documents about the Struc-ture are available (from W;HEMS Publications,P.O. Drawer P, Boulder, Colorado 80302):

A. A Structure for the Outcomes of Post-secondary EducationPresents six at-tributes of an educational outcome and fiveother factors that are important for under-standing particular outcomes, and providesdetailed background discussion abouteach. Gives an in-depth description of theStructure, its rationale, the developmentalprinciples and evaluative criteria used, itsdevelopment, and its underlying empiricalsupport.

B. The Outcomes Structure: An Overview andProcedures for Applying It in PostsecondaryEducation InstitutionsProvides an over-view description of the Structure, givesdescriptions and examples of the variousways that the Structure can be used in apractical manner at postsecondary institu-tions; provides step-by-step procedures forthree basic processes using the Structure,which can in turn be applied to a number ofother tasks; and provides guidelines fororienting people on a campus to outcomesconcepts and the Structure, and for trainingpeople to apply them.

56

62

For those who are interested, anotherdocument is also available that pro/ides acompilation of information about each of over80 previous efforts to "structure" educationaloutcomes and related concepts that werefound in the literature: Previous Attempts to"Structure" Educational Outcomes and Out-come Related Concepts. If additional infor-mation or assistance is needed concerning theStructure, contact NCHEMS.

57

63

Appendix E

Important Questions and Criticisms of the Outcomes StructurePosed by Particular Reviewers a

and Responses by the Developers of the Structure

1. By breaking outcomes into component typesand ever more detailed categories, is it notpossible to focus so much on spdcifics thatone loses sight of the overall, combined wholewhich is more than the sum of the parts? Thisis a real danger, but it does not mean it is im-proper or lacks value to focus on the com-ponents that constitute the whole. Rather, itcautions us to constantly, as we examine apart, keep in mind the relationship of the partto the other parts and the whole. It is difficult, ifnot impossible, to adequately describe thewhole without focusing also on each importantcomponent, for example, describing oranalyzing what it means to be an "educatedperson."

2. Could not this attempt to objectify outcomesand develop standard categories anddefinitions lead to a stifling of diversity, in-novation, and changeespecially if the Struc-ture is used by administrators to spell out theoutcomes desired and the process to meetthose outcomes in such specific, precise termsthat there is little room for innovativeness onthe part of faculty members? Institutions andprograms can be unique in the clientele served,the outcomes that are attempted or attained,and the means used to reach chose outcomes.Preliminary use of the Structure shows that theaudience dimension provides a comprehensiveuniverse of categories of potential clients fromwhich to choose, and that the type-of-outcomedimension provides a comprehensive universeof outcomes categories from which to choose.Thus, using the lists of categories as achecklist could stimulate considraration ofspecific unique audiences and types of out-comes that might not have been consideredotherwise.

Although administrators could use the Struc-ture to stifle innovation, they also couldstimulate faculty to be more specific and con-crete in delineating the outcomes intended,

and thus encourage consiut ration of alter-native ways of bringing about the outcomesbased on the desired end results. The Out-comes Structure can clearly be used to helpstimulate such concreteness in thinking, asdemonstrated by the joint CASC/NCH EMSproject.

3. Would not the process of using the Structuretake more of my time and energy than I can af-ford? To use the Structure as an aid todevelopment of a complete program of goals,objectives, and priorities for a complex in-stitution and its programs is an extremelylengthy process. But that process would belengthy whether or not the, structure was used.It is possible that the systematic approachesdeveloped for using the Structure, thedefinitions and concepts outlined, and use ofthe Structure as a checklist of the universe ofalternative possible focuses can shorten theprocess appreciably. Furthermore, certain ap-plications of the Structure do not have to betime consuming to be rsefulfor example,stimulating systematic and more concretethinking by faculty and staff of what they aretrying to accomplish in their programs andcourses, of why they are trying to accomplishthose ends in particular ways, and of how toshow others that they are accomplishing theirgoals. Similarly, experiences at the Universityof Colorado suggest that one who is properlyoriented to outcome concepts can use theStructure to evalliate the coverage of extensivelists of goals relatively quickly and ef -ciently.

4. Does not the use of psychological jargon forsome terms in the Structure diminish itsusefulness as an aid to communication?Jargon of any type should be avoidedwhenever possible, and an attempt was madeto avoid its use in developing the Structure. Butfor certain outcomes, particularly student out-comes, psychological terminology appeared tobe the most generic and descriptive, and the

°Reprinted from Chapter 4 of Oscar T Lenning, Yong S Lee, Sidney S Mlcek, and Allan L Service, A ucture for the Outcomes of Postsecondary Education (Boulder,Colorado National Center for Higher Education Management Systems, 1977)

596 4

most widely accepted. Council for the Advance-ment of Small College visitors to the cam-puses did perceive that the Structure could aidin "communicating outcomes across disci-plinary lines. ' Nevertheless, it is to be hopedthat extensive use of the Structure will sug-gest better terms in some areas of theStructure.

5. goes not the fact that the code numbers aredisconcerting to some people reduce theStructure's usefulness for them? Possibly, butit need not be so. Code numbers for eachcategory were included for two purposes: (1) togive people a shorthand that would allow themto record outcomes and communicate them toknowledgeable others verbally or in writing in amore succinct and efficient manner, and (2) foruse in outcomes information storage andretrieval systems. For other uses, the codescould be ignored. This is what happened infollow-up applications of the Structure atSpring Arbor College. Interestingly, in theCASCINCHEMS interviews, when the respon-dents referred to particular outcome cate-gories, they generally referred to the codenumbers rather than the category nameforexample, "Outcome 2240" rather than "ex-pression and communication skill outcomes."Thus, some people who initially find the codesdisconcerting may later become accustomedto using them and actually find them useful.But users of the Structure are encouraged tomodify it as appropriate for best meeting theircontext and needs.

6. The orientation of the Structure is foreign tothe orientation of most educatorsthey justdon't think this way. Would it not have beenbetter and more acceptable to them if it werecloser to their orientation in its form and con-tent? The authors tried to make the Struc-ture as generic, theoretically and philosoph-ically neutral, and comprehensive in its cover-age as possible, and this guided the develop-ment of the Structure. One person who raisedthis question suggested moving the know-ledge and understanding categories in frontof the human characteristic outcome cate-gories, because "they are more important tomost educators." But such a change wouldhave made the Structure less neutral thanretaining the categories in alphabetical order.The NCHEMS Inventory of Outcome Variablesand Measures, which is the forerunner of theOutcomes Structure, was closer in its orien-tation to that of the typical educator who willuse the Structure. However, that very charac-teristic led to criticism by some that many ofthe terms used were value-laden (it was notphilosophically and theoretically neutral), thatit was too selective in its coverage (not com-prehensive), and that it was too specific in itsorientation and lacked some internal con-sistency (not generic).

6065

Appendix F

Some Materials for Possible Use atthe Campus Orientation Workshops

Page

1. A Possible Workshop Agenda 62-63

2. Sample Transparencies That Could Be Used in Presentingan Overview of Outcome Concepts and the Structure 64-73

3. A Relevant Instrument for a Work Exercise 74-71

4 A Relevant Role-Playing Scenario 78

61

63

Possible Workshop Agenda*

1. A short introduction that outlines why thecampus may have a need to apply outcomeconcepts and the Outcomes Structure.

2. An overview of outcomes concepts and theNCHEMS Outcomes Structure, using transpar-encies such as those included in this appen-dix. Using color, for example, different colorshades of plastic for different transparencies,can add to their attractiveness and attention-getting quality. Be sure to encouragequestions and requests for clarification duringthe presentation.

3. A short overview summary of how the Structurecould be used to develop lists of priority out-comes for one's program, using a handout inwhich are reprinted Figures 1-5 of thisdocument, which show what came out ofvarious stages in the outcome list develop-ment process for the political science programat Alpha College.

4. Conduct a relevant work experience that willstimulate participants to focus on the intendedor needed program (---ials they perceive, waysof reaching those goals, and how outcomesuccess might be documented for theirprogram of concern or responsibility. Have par-ticipants respond quickly to the Student Learn-ing Outcomes Questionnaire included in thisappendixor e dified version of itin termsof the program for which they are primarilyconcerned or responsible, and then have themdiscuss their responses with one another inpreviously assigned small groups. The en-closed questionnaire is a revised version of theform used in the CASCINCHEMS campus in-terviews to stimulate discussion. As with theother workshop materials, it is imperative thatthe stimulation questionnaire decided on betried out in advance and modified accordinglybefore being used at the workshop. For ex-ample, you may find that it should be abbre-viated (any of items 6-50 could be deleted easily)to fit into "le time slot allotted for it in theworkshop.

The portion of each agenda item above that is typed in italics is what could beprinted In the formal agenda sheet to be handed out t,, workshop participants.along with the time frames for each

5. Review of the work exercise, where it is to beshown how the use of the questionnaire andthe follow-up conversation was in fact an ab-breviated application of the outcomes Struc-ture, discussed previously, to one part ofprogram planning.

6. Conduct a role-playing exercise, making use ofthe scenario included in this appendix, whichhas the intent of shifting the group's attentionto institution-wide outcomes, and outcome-oriented planning at that level. People to playthe roles should be selected very carefully andinvitations to play the roles should be made(preferably in person, and they should receivecopies of the scenario at this time) well in ad-vance of the workshop. Furthermore, an orien-tation session with the role players, that in-cludes distribution of appropriate materialsabout the Structure and its use, should be heldahead of time so that they can be preparedadequately to assume their roles.

For the workshop session itself, it would bedesirable to have them seated around a roundtable with large identification place cards thateveryone else at the table can easily read. Inthe CASC/NCHEMS tryouts of this role-playingscenario, the actors had difficulty respondingin terms of their own institution (which eachv as asked to do) because the discussion keptreferring to "Alpha College." Therefore, thescenario sheet distributed probably shouldrefer to your own institution by name (insteadof Alpha College), and the discussion duringthe session then would also refer to your in-stitution by lame. It will probably be desirableto keep the fictitious names (or others deemedmore appropriate and entertaining for yourgroup) for the role participants, however, tokeep it obvious to the group at all times thatthis is a role-playing situation designed to benonthreatening to everyone. (For this reasonalso, a requirement was made for the CASCINCHEMS workshops that none of the roleplayers be playing their real-life role.)

7. Discussion and critique of file role exercise,including: how it related to reality; problemsand issues related to institution-wide out-comes; and desirable modes, procedures, and

62

67

times for institution-wide goal setting for thisinstitution. One person should have beenassigned ahead of time to take notes duringthe role playing, to make observations about itto get the discussion started, and to lead thediscussion.

8. Wrap-up session where the workshop leadersummarizes what happened, ties together thedifferent parts of the workshop into ameaningful whole, and attempts to end on anote that will stimulate further thinking andencourage support, action, and involvementfor campus follow-up efforts in this area (thisworkshop is only meant to be a brief introduc-tion to the whole subject).

63

68

The Six W's

In order to really understand educational outcomes the following must beanswered:

What are the characteristics and makeup of the educational outcome ofconcern?

Which institutional resources and activities aro combined, and in whichways, to bring about the outcome?

For Whom is the outcome intended, or who actually received or was af-fected by it?

Why will, or did, the outcome occur?

Where will, or did, the outcome occur?

When will, or did, the outcome occur?

Eleven Factors That Are Useful inAnswering the Six Questions

What Outcomes?1. '- 71 - Products, Events, and Conditions2. Gliange Status Maintenance and Change3. Focus4. Neutrality5. Measurability6. Output/Impact

Which Way to Obtain Them?7. Producer /Facilitaicr

Outcomes For Whom?8. Audience

Why Will, or Did, They Occur?9. Intended/Unintended

Where Will, or Did, They Occur?10. Functional Area

When Will, or Did, They Occur?11. Time Point in Time and Duration

65

PLACEMENTOFFICE

APPOINTMENTSCHFDULE

GRADUATESCHOOL

WHICH OF THESE DRAWINGSSUGGEST PRIMARILY STUDENTIMPACTS AND WHICH SUGGESTPRIMARILY STUDENT OUTPUTSFOR A COLLEG'

66

71

The NCHEMS Outcomes Structure is asystem and framework designed tohelp people effectively organize anduse information about the full range

of postsecondary education outcomes.

67

72

AUDIENCEDIMENSION

(Who receives oris affected bythe outcome?)

Outcomes Structure

TYPE-OF-OUTCOMEDIMENSION

(What basic entity ismaintained or changed?)

TIMEDIMENSION

(When is the outcomeexpected to occur, orwhen does it occur?)

68

73

The Major (First Level of Detail)Categories for the Audience andTypeof-Outcome Dimension*

A. AUDIENCE DIMENSION1. Individual/Group Clients2. Interest-Based Communities3. Geographic-Based Communities4. Aggregatet, of People5. Other Audiences

B. TYPE-OF-OUTCOME DIMENSION1. Economic Outcooc,2. Human Characteri61ic Outcomes3. Knowledge, Technology, and Art Form Outcomes4. Resource and Service Provision Outcomes5. Other Maintenance and Change Outcomes

*Each of these categories has subcategories that allow you to ;,ous at a more detailedand concrete level.

69

74

AUDIENCE

Individual/GroupClients

Interest-basedCommunities

Geographic-basedCommunities

Human-CharacteristicAggregations

OtherAudiences

The AudiencelType- of -Outcome Matrix

I_LI i-1-7-:r=1---_i_r-4--1--+-4--

I 1 1 1_I-_,__T_I-1 I I__

ll'--/-1--F-4-----r-T-1-1--__4_4_1_4____4_4_1_r_

11-1-4-4---T-1--r-i-1"1.-J- I I 1 -__L-4-J1--1--1 1-7-1-

iill111L-1--1I I I I

. III1-1--r-T-

- --i-1 -1-_ -1--,..-1.-

-4-_L-L-1----r-4-1-1._1 1 1 1

1 i i

-1--r-1--r-- -- I- 1- --

--11--r-1 1 1

1

i

1 1 1

l' 4 1- L -irrT-i----t-4-1-4---'--r-17-_,__I__,____r_ill_i_l__I__-7r-r-1-1---,,,T

_4_1__F_L____1_1_1_1_--T-1r_r-____,__I__4_4_,

.

I

_

11 ,

1 L

1 1.- + 4__1_44.____,___I__r_II

--11-4-4-4----T-1--t--,-__I 1 1 1 _,

-1--r-t-

1

ill,'_L__L__I__,_

1 I 1 I i

4.--L-1-4,t-1-4-1----L I I Ii_r_l_l____,_1_,-T---,, . i_l____,_+_,__L,_,__,_,_i__r

__,_4_,H___4...-r_,_____H_i_i_1 1 1

I

1 I L.

.

I. [ 1-

I

1

1

1-4-4-4--4-1-4-1--,

_4 1 4 1

.-4-1 I-1 ---4-4--Lt1 r-T 1-

_1_4_4_4____T_I__,_f_444-44_,_t_

__,_i_i_l____,_,__4_,_

1 1

I 1 1 1

-4-_L-4_-_-_,.1.--1--r-I 1 i

--r-+-4-11----7--r-t-i-1 r-7-1- __1_i_i_l_t__r__f_i_

1 1 1

1 I

I 1

1i

1

-. . .

_4-_!--t-4.-

1_4_1_4____1_1__L_T_-__1_1_4_1.i 1 1

_i_l_f_l____f_i_+_r___41_1_t_i__I I

1 , I

- 111_1-11.-r+ 1 r

1 1

1

It

7 T TIII!--ir-t-/-1-

-t- t 11 i

_i_I_I__L___4_4_4_7__L 1 I I1-/-1--r---

--4-1_1_11 1___T-4-4-----1--r-1-

I i !

1__L_1

-- -x--1--1--1_1 1

1 i

i

11

_i_4_4_4____4_4__F_I____ht_li_._H__1_1---4-,1_4_1_____4_,-1

1 1

I L

J_

I I-- I

....4.. i......i__L-.

-.

_4_T1

jC

_4_4_4_4___1__!_r_i____1_1_11 1 1 1

7--17-1 r

Economic HumanOutcomes Characteristic

Outcomes

Knowledge,Technology,and Art FormOutcomes

Resource andServiceProvisionOutcomes

TYPE-OF-OUTCOMEr-

Other Maintenanceand ChangeOutcomes

fi\A,17=1" =AN dre

I

AUDIENCE

TYPE-OF-OUTCOME

TIME

0

B.S. Graduates inMechanical Engineering ..

Percentage of GraduatesObtaining Their FirstFull-Time Job in the Fieldof Their Choice ...

Within One Yearafter Graduation

71

76

.0

Uses of the Outcomes Structure

A. COMMUNICATIONB. STIMULATIONC. PLANNING, MANAGEMENT, AND EVALUATION

1. Needs Assessment2. Goal Identification3. Goal Assessment4. Goal Translation5. Planning and Management6. Evaluation

Context/Condition

Resources

Activities

Outcomes

D. RESEARCH

0 PAS

OBJECTIVES

lnterded Actual

77

72

In using the structure, it must beclearly differentiated whether the

focus is on institution-wide outcomesor on program outcomes. It is unwiseto try to focus on both types of out-

comes at the same time when using thestructure.

78

73

STUDENT LEARNING OUTCOMES QUESTIONNAIRESMALL COLLEGE FORM

Small colleges desire to have various impacts on their students. These consequences (outcomes) ofcollege attendance that are planned to happen may or may not be expected to vary with the type ofstudent. This questionnaire will gather information about the particular student outcomes that small-college administrators and faculty members are trying to bring about for: (1) the students throughoutthe college, or (2) the students in the college program area for which they have primary responsibilities.Any students completing the questionnaire should respond in terms of what they as students feel are thedesirable outcomes for their college or their program at the college, and that should be aimed at by hecollege or program. For all respondents (administrators, faculty, or students), please decide whetheryou are going to respond in terms of college-wide outcomes or in terms of outcomes for a particularprogram area, Indicate your decision in Item 1 below, and then respond to the remainder of the question-naire accordingly, by checking () the appropriate boxes and completing the blanks.

1. Please respond to this questionnaire in one of two ways; in terms of outcomes for the college as awhole, or in / s of outcomes for your program within the college. Choose the area most importantto you:

1 2

E7 College-wide outcomes (7.7 Program-area outcomes

2. If you answered "program-area outcomes" for Item 1, what program will you be referring to throughoutthe questionnaire?

(1)

(2-4)

3. What 4s your classification? Check (y) one box and fill in the blank at the right. This informationis needed so that we can describe the diversity of our survey samples. Such identification will bepresented to others only in the form of group statistics.

Administrator LJ (What position? ) (6-8)

Department Head Q (What department?

Other Faculty E7 (What department?

Student [27 (What major?

4. When considering student outcomes, either for your institution or for your program, would you ratherdistinguish between outcomes for student subpopulations, or can you focus meaningfully on desiredoutcomes for "all students i- general"? That is, are outcomes significantly different for: residentvs. commuter; majors vs. non-majors; upper- vs. low -division, etc.?

0 Yes E7 No5. If "yes," please indicate the two groups whose respective outcomes would be most important to plan

for separately, in the spaces below:

Group 1 -- Group 2 -- (9-10

Items 6-55 list some possible student outcome categories. To the right of the page are two columns ofmarking boxes that will allow you to rate the importance of each outcome for the group or groups of studentsyou indicated in Item 4. If you answered "yes" for Item 4, only the Group 1 column should be used. Pleasechoose from the following three categories in making your ratings, and code as indicated.

ESSENTIAL (Student outcomes that must be a direct focus of planning for this institution orprogram)

IMPORTANT (Student outcomes that should be a direct focus of planning for this institutionor program)

NOT NEEDED (May be nice, but is less than important as a focus of planning for thisinstitution or program)

74

79

6. Academic skills

7. Citizenship and family membership skills

8. Creativity skills

9. Expression and communication skills

10. Intellectual skills

11. Interpersonal, leadership, and organizational skills

12. Occupational and employability skills

13. Physical and motor skills

14. General knowledge and understanding

15. Specialized knowledge and understanding

16. Desires, aims, and goals

17. Dislikes, likes, and interests

18. Motivation or drive level

19. Attitudes and values

20. Beliefs, commitments, and philosophy of life

21. Feelings and emotions, for example, appreciations and satisfaction

22. Mores, customs,and standards of conduct

23. Perceptual awareness and perceptions of self, others, and things

24. Physical fitness, traits, and health

25. Adventurousness and initiative

26. Autonomy and independence

27. Dependability and responsibility

28. Dogmatism-openmindedness; authoritarianism-egalitarianism

29. Flexibility and adaptability

30. Habits

31. Psychological functioning

32. Tolerance and persistence

33. Completion or achievement award

34. Credit recognition

35. Economic access, independence,and productivity

36. Image, reputation,or status

37. Licensing and certification38. Obtaining a job or admission to a follow-up program

39. Power and/or authority40. Job, school, or life success (other than above)

41. Adjustment to retirement

42. Affiliations43. Avocational and social activities and roles

44. Career and vocational activities and roles

45. Citizenship activities and roles

46. Family activities and roles

47. Friendships and relationships

48. New knowledge and understanding from research and scholarship

49. Research and scholarship products

50. Production or preservation of art forms and works

51.

52.

53.

54.

55.

Essential E

Important I

Not Needed N

Group 1

E I N0 L7 00 0 00 L7 ClEl 17 170 0 0E I N0 0 1717 0 00 0 0

L7 1_70 17 Cl0E I Nz7 El 00 0 00 0 00 0 017 0 L7E I N

0 O0 L-J

E I N

Q Q 0L7 0L7

0/7 00 17E I N

0 177_ 117L7 v 00 17 00 0 00 0E I N

Q0 L7 00 L7 013 L

L7 L7 L7E I N

Q Q 0E7 0 0C.7_.17 0 00 0

E I N

Q Q 00 0 00 L_70 0 1E I N0 17 00 0 00 0 0

17 1 117 0 0

Group 2

E I N

O 00 0ri 0 00 0 0E I N

O 0 ,00 0 017 0 01.7 0 00 0 0E I N

0 0Q a0 a aQE I N

O Q 0(7 0 00 0 00 0 O0 0 0E I N

Q Q aO 00 L7 O0 0 00 0 0E I N0 0 00 L7 00 00 0 00 L7 0E I N

1170 0 00 0 017 L7L7 L7 Z7E I N

0 000 U 00 00 Z7 L7E I N

0 00 O0 L7 00 0 00 L7 L7E I N

0 0 E70 170 0 C3L 1 17 170 0 0

(16)

(17)

(18)

(19)

(20)

(21)

(22)

(23)

(24)

(25)

(26)

(27)

(28)

(29)

(30)

(31)

(32)

(33)

(34)

(35)

(36)

(37)

(38)

(39)

(40)

(41)

(42)

(43;(44)

(45)

(46)

(47)

(48)

(49)

(50)

(51)

(52)

(53)

(54)

(55)

(56)

(57)

(58)

(59)

(60)

NOTE: IF YOU ARE UNSURE WHAT ANY OF THE ABOVE OUTCOME CATEGORIES MEAN, DEFINITIONS AND EXAMPLES ARE PROVIDED

IN THE MANUAL FOR THE STUDENT LEARNING OUTCOMES QUESTIONNAIRE, AVAILABLE FROM THE CAMPUS COORDINATOR FOR THIS

STUDY.75

s0

For items 56:60, choose one of the outcome categories that you rated "essential" and which you have thoughtmost about prior to this time. Indicate this category in the large box below:

(62-6:

Now try to think of five specific and concrete outcomes for the students of concern that you would considerespecially important for this outcome category, and record them in the blanks provided for Items 56-60. Forexample, "beliefs, commitments, and philosophy of life" could include:developing student commitment to thediscipline; helping students clarify their philosophies of life so that they become coherent, systematic,and internally consistent; helping students incorporate coursework into their philosophy of life; helpingstudents internal izea particular proposition; and demonstrating commitment through action.

56.

57.

58.

59.

60.

For Items 61-65, choose one of , ,tcome categories that you rated "essential" but which you have thoughtthe least about prior to this time. Indicate this category in the large box below:

(64-65:

Now, try to think of five specificand concrete outcomes for the students of concern that you would considerespecially important for this outcome category.

61.

62.

63.

64.

65.

Just because an outcome towards which an institution or program is striving and planning has high priority,there is no guarantee that any success will occur--that the outcome will happen. And in this "age ofaccountability" there are increasing pressures for d, ,umentation that the outcomes colleges and programssay they are bringing about have in fact occurred. In the blanks provided below for Items 56a-55a indicate someobservations or measures (includingself-report measures, but not limited to them ) that you feel would constitutemajor concrete evidence that the particular learning outcome you identified in Items 56-65 as having priorityare, in fact, occurring or did occur (for example, judges' ratings, score on a particular test or exam, observedactions or reactions to carefully planned stimuli, increased

attendance at certain types of campus events,written papers, and observing an oral demonstration by the student):

56a.

57a.

58a.

59a.

60a.

76 81

61a.

62a.

63a.

64a.

65a.

In the blanks provided below for Items 56b-65b, identify some specific student activities and experiences

you especially think would contribute to the realization of the particular student outcomes that you

identified in Items 56-65 (for example, peer tutoring, participation in political campaigns, community

service projects, organized in-class confrontation debates of selected issues, and small discussion groups):

56b.

57b.

58b.

59b.

60b.

61b.

62b.

63b.

64b.

65b.

Age and sex groupings have often been found helpful in interpreting survey data. For Items 66 and 67,

please check (1) the appropriate box.

66. What is your sex?

sQ Female Q Male (66)

67. What is your age:

Q Under 25 Q 25-35 E7 36-50 [7 Over 50 (67)

For students, what is your year in school?

i it 4. ..5

(:7 Freshman Q Sophomore Q Junior Q Senior Q Graduate or (68)Professional

Student

S77

2

Modified Scenario for theRole-Playing Session during the

CASC /NCHEMS On-Campus Workshop

Roles

Nick Knocks College President (Committee Chairperson)Bili Grave Vice-President for FinanceMavis Learned Vice-President for Academic Affaii sEldrige Bridger Vice-President for Student AffairsLeroy Wowsm Vice - President for Public Relations and DevelopmentFredrick O'Tool Faculty representativeMitzi Gainsmore Student representativeFrank Memory Alumni representativeRichard Green Community leader and bank president

The Setting

The Jonathan Doe R...idation has made an offerto Alpha College to provide $25C,000 of un-solicited funds for s *rengthening the on goingprogram of the college during the coming year.The only requirements are that it be f^r items inaddition to those currently budgeted for next year,that it not involve facilities procurement or capitalinvestment, and that the college notify the foun-datie within six mr,nths exactly what the moneywill b-. used to do (if the college does not meet'his requirement, the money will revert to one ofthe flundation's other charities).

Upon receiving word of the good news, AlphaCollege's Board o; Trustees directeci PresidentKnocks to present them in three months with apriority list of student educational outcomes thatthe college should especially strive to improve,and recommendations for funded supplementalactivities and for materials to bring about thedesired improvements during the coming year. lnresponse to tnis request, President Knocks ofAlpha Tried a nine-member comma "' ie represent-ing th major groups and areas of concern oncampy., 'nth himself as chairp..rson. In ao_lition,

an announcement was made to the campuscommunity that anyone interested could sit in cnthe meeting as an observer. (If an observer feltstrongly enough about something beingdiscussed, however, they would be allowed topass their insight or concern on to the committeein a brief fashion.)

At the committee's firs. meeting last week theydiscussed a new Outcomes Structure designed tohelp colleges decide on appropriate outcomes fortheir campus. They decided that special areas ofconcern were: aspirations; competence and skills;morale, satisfaction, and affective characteris-tics; perceptual characteristics; and psycho-logical adjustment characteristics. They alsodecided that at their next meeting they wouldfocus on developing a priority list of concretecompetence and skill outcomes, along with waysof improving those c lcrete outcomes. It wasalso decided that any discussion about the otherfour outcome areas wos:id have to wait until latermeetings of the group. President Knocks is justnow calling their second meeting to order.

78

83

REFERENCES

Bloom, Benjamin S Taxonomy of Educational Oblectives, Handbook 1 Cognitive Domain New York David McKay,

1956

Lenning, Oscar T Previous Attempts to "Structure" Educational Outcomes and Outcome-Related Concepts A

Compilation and Review of the Literature Boulder, Colo National Center for Higher Education Management Systems,

1977a

"Resu,ts of the CASCINCHEMS Small-College Outcomes Identification Project A Summary " Unpublisheddocument Boulder, Colo National Center for Higher Education Management Systems, 1977b

Len ning, Oscar T , Lee, Yong S , Micek, Si d ay S , and Sarv.ce, Allah I_ A Structure for the Outcomes of PostsecondaryEducation Boulder, Colo . National Center for Higher E ation Management Systems, 1977

Lennirg, 0 T, and Micek, S S "Defining and Communicating Institutional Mission/Role/Scope and Priorities The

Nceds of Different Types of Postsecondary Institutions" Paper presented at the annual meeting of the AmericanEducational Research Association, San Franc'sco, California, April 1976 (ERIC Document ED 121376)

Micek, S S , Service, A L , and Lee, Y S Outcome Measures and Procedures Maival Technical Report No 70

Boulder, Colo National Center for Higher Education Management Systems at WICHE, 1975

Micek, Sidney S , and Walihaus, Robert A An Introduction to the Identification and Uses of Higher EducationOutcomes Information Boulder, Colo National Center for Higher Education Management Systems at WICHE, 1973

79

84

document resume - eric · document resume ed 272 110 he 019 557 author lenning, oscar t. title the...

Documents