DOCUMENT RESUME

ED 133 348 TM 005 953

AUTHOR McCray, Emajean; Lottes, John TITLE The Logic of Evaluation: Norm- or

Criterion-Referenced Judgments.PUB DATE Mar 76 NOTE 16p.; Paper presented at the Annual Meeting of the

American Educational Research Association (60th, San Francisco, California, April 19-23, 1976)

EDRS PRICE NF-$0.83 HC-$1.67 Plus Postage. DESCRIPTORS Comparative Analysis; *Criterion Referenced Tests;

Curriculum; Guidelines; Instruction; *Norm Referenced Tests; *Student Evaluation; Test Interpretation

ABSTRACT Principles are established for reaching valid

decisions as to whether evaluative judgments should be norm referenced or criterion referenced. A set of decision rules is presented, in flow chart form, to serve as a guide to the decision-making process. The use of these decision rules is illustrated for a variety of practical situations. The evaluativeframework is founded upon credible principles of teaching and a precise language of curriculum and instruction. These bases include the pragmatic, conceptual, and value aspects of education. (RC)

THE LOGIC OF EVALUATION:

NORM- OR CRITERION-REFERENCED JUDGMENTS

Emajean McCray Department of Research and Evaluation Fort Worth Independent School District

and

John Lottes School of Education

Texas Christian University

March, 1976

This paper has been developed for presentation at the April 1976 American Educational Research Associ-ation conference in San Francisco, California

INTRODUCTION

The Problem And Its Significance

In many practical evaluative situations, a decision must

be made as to whether an evaluative judgment of pupil performance

should be a criterion-referenced judgment or a norm-referenced

judgment. The question with which this paper is concerned is:

How can a justified decision be made as to which is the appro-

priate frame of reference?

The authors' intention is to establish credible prin-

ciples for reaching, in every relevant decision-situation, a

valid decision as to whether an evaluative judgment of pupil per-

formance should be norm-referenced or criterion-referenced.

It is believed that the decision rules developed here can

provide sound bases for practical judgments in an area where

judgments are ordinarily made in the absence of appeal to cre-

dible principles. As an auxiliary outcome, the complete range

of pupil evaluations of relevance to the curricular and in-

structional enterprise will be delineated.

The Organization Of This Paper

Achievement of the stated purpose will be accomplished

through the following course of action. First, concepts of crite-

rion-referenced tests and norm-referenced tests will be clarified;

and the concepts of criterion-referenced judgments and norm-

referenced judgments will be clarified.

Second, the nature of a curricular framework for evalua-

tive judgmentswill be described, and significant interrelation-

ships within that framework will be identified. Also, the

nature of a curricular-free framework for evaluative judgments

will be described. The range of evaluative judgments in refer-

ence to pupil performance will be delineated for each type of

framework.

Third, the logic of the decision as to whether a judgment

should be criterion-referenced or norm-referenced will be set

forth, and its utilization illustrated for all possible decision

situations.

Fourth, some concluding statements will be made to illum-

inate or extend fundamental aspects of the outlined principles.

THE CONCEPTS OF CRITERION-REFERENCED AND NORM-REFERENCED

Criterion-Referenced Tests And Norm-Referenced Tests

The primary difference between criterion-referenced tests

and norm-referenced tests lies with the manner in which the tests

are constructed (i.e. the construction of the test items). Cri-

terion-referenced tests consist of items, each of which is log-

ically valid in reference to some set of concepts or statements.

For example, the test items may be constructed in reference to

a set of explicitly stated objectives.

Norm-referenced tests consist of items selected from a pool

of items on the basis of item statistics obtained by item try-

outs on relevant groups of students. Such tests generally con-

tain items which require performance beyond, as well as below,

the expected level of performance for the intended population.

The items are constructed and selected so as to maximize the

range of scores made by a given group of students; hence, they

represent gradations in knowledge or skills relevant to the

instructional unit.

A test is used as a device to elicit relevant pupil per-

formances. A set of sentences describing the way a pupil per-

forms on a test may appropriately be viewed as a report of the

observed performance of that pupil. The process of making eval-

uative judgments about such reports of observed pupil performance

is the focal point of the following section.

Criterion-Referenced Judgments And Norm-Referenced Judgments

The primary difference between criterion-referenced judg-

ments of pupil performance and norm-referenced judgments lies

with the bases in which the judgments are grounded. A criterion-

referenced judgment of the performance of a pupil is made by com-

paring reports of the observed performance of that pupil against

a set of statements from which the desired manner of performance

can be derived. That set of statements, or some set of derived

statements, constitute the "criteria" which form the frame of

reference for the judgments; hence the label "criterion-ref-

erenced judgments."

A norm-referenced judgment of the performance of a pupil

is made by comparing reports of the observed performance of that

pupil against a set of observation reports of the performances

of other pupils. The set of observation reports of the per-

formances of other pupils constitute the "norms" which form one

aspect of the frame of reference for the judgments; hence the

label "norm-referenced" judgments. Of course, no judgment of

the performance of a pupil can be made in reference to norms

without appeal to values, ethical rules, or some kind of

principles as the grounds for judging the relationship between

the individual performance and the "norm;" in this sense the

use of criteria in making evaluative judgments can never be

evaded.

CURRICULAR AND CURRICULAR-FREE FRAMEWORKS FOR EVALUATIVE JUDGMENTS

The Nature Of A Curricular Framework For Evaluative Judgments

The Structure Of A Curricular Framework

Adequate curricular representation requires articulation

of goals, procedural rules, and qualifying conditions. Curric-

ular goals represent the ends to be achieved by the students.

Curricular procedural rules represent the means by which those

ends are to be achieved. They specify the basic substantive or

content elements with which instructors and students will deal,

and the manner in which the instructors and students will deal

with those substantive elements and the way they will relate to

each other as well. The qualifying conditions represent student

characteristics believed necessary in order to achieve the goals,

through implementation of the procedural rules.

The relationship between a goal-component (ends), a rule-

component (means), and a qualifier-component (conditions under

which the means are believed effective for achieving the ends)

constitute a claim. This kind of claim will be called a curricular

claim; and its generalized form may be represented as follows:

Generalized Curricular Claim

For each pupil X, where X satisfies conditions C; If both the teacher(s) and a set of pupils,

of which X is a member, act under rules R, then X will (probably) attain goal-state S.

The ends, means, and qualifier components of a curricular

claim will be called, respectively: curricular goal-state,

curricular rules, and curricular qualifying conditions.

The curricular rules leave both teacher and pupil some

degrees of freedom to determine the way in which their individual

acts will be performed. The teacher develops strategies for

influencing the pupil toward goal-state attainment, and adapts

these strategies in response to the patterns of individual

pupil acts. Of course a given strategy is reasonable only if the

teacher believes that, under the existing circumstances, implemen-

tation of that strategy will lead to achievement of certain ends.

This entails the requirement that a teacher's strategy must be

imbedded within the rule component of some claim that the teacher

has reason to believe is valid. This sort of a claim will be

called an instructional claim,* whose generalized form may be

represented in the following way:

Generalized Instructional Claim

For each pupil X, where X satisfies conditions I,If the teacher(s) acts under rules T, in relation to a set of pupils of which X is a member, then X will (probably) attain goal-state G.

The ends, means, and qualifier components of an instruction-

al claim will be called, respectively: instructional goal-state,

instructional rules, and instructional qualifying conditions.

* The concepts of "curricular claim" and "instructional claim" are discussed further in "The Validation Of Educational Programs," by Emajean McCray and John Lottes, a paper presented at the 1976 AERA conference.

The Range Of Evaluative Judgments In Reference To A Curricular Framework

Under the preceding curricular framework, the types of

evaluative judgments to be made can be categorized as follows:

(1) judgments of student performance in reference to the curric-

ular qualifying conditions, (2) judgments of student performance

in reference to the curricular rules, (3) judgments of student

performance in reference to the curricular goal-state, (4) judg-

ments of student performance in reference to the instructional

qualifying conditions, and (5) judgments of student performance

in reference to the instructional goal-state.

The Nature Of A Curricular-Free Framework For Evaluative Judgments

The Meaning Of Curricular-Free Framework

All judgments of student performance can be described as

belonging to a universe of evaluative judgments made in reference

to student performance. One class of judgments of student per-

formance was described in the previous section; namely, judgments

in reference to a curricular framework. All judgments of student

performance, which are not made in reference to a curricular

framework, are made in reference to some other framework which

is grounded outside the curricular framework. All such extra-

curricular frameworks for evaluative judgments will be called

curricular-free frameworks.

The Functions Of Curricular-Free Frameworks

Judgments governed by curricular-free evaluative frame-

works are often made about student performances. These judgments

are of varying kinds and can be categorized loosely as follows:

(1) assessment of pupil performance to determine educational

institutional status and (2) assessment of pupil performance

to determine individual status.

Assessment of pupil performance to determine educational

institutional status may be conducted in reference to a school,

district, state, regional or nktional level educational system.

This category would include the assessment of the strengths

and weaknesses of a particular school or of the schools in a

particular geographic area; as well as the assessment as to

whether a particular school or group of schools is effective

in influencing student attainment of institutional goals.

Assessment of pupil performance to determine individual

status would include (a) the determination as to whether an

individual attained institutional goals and (b) the determina-

tion of individual strengths and weaknesses. The latter aspect

would include the diagnosis of individual learning problems

which is a type of judgment often conducted outside the curric-

ular framework. Such diagnosis might yield evaluative judg-

ments as to physical defects, sensory processing weaknesses,

psychological abnormalities, and so forth.

The determination of individual strengths and weaknesses

would also include Judgments made in the selection of personnel

(or students). This process could involve the selection of

students for entry into college, special programs (i.e. special

education, vocational education, programs for the gifted, etc.)

or particular jobs based upon student performance which is

curricular-free.

CRITERION-REFERENCED OR NORM-REFERENCED JUDGMENTS: THE LOGIC OF THE DECISION

The decision rules for determining whether a judgment

should be criterion-referenced or norm-referenced are shown

in CHART A. These decision rules provide the basis for making

rational decisions concerning a wide range of practical educa-

tional situations.

Evaluation within the curricular-instructional claim frame-

work involves judgments which are made against specific criteria.

This situation results from the fact that all activity in the

curricular-instructional realm is goal directed and governed by

rules which apply only when specified guidelines or conditions

are satisfied.

Diagnosis of an individual may be conducted within the

curricular framework or within a curricular-free framework.

When diagnosis occurs within a curricular framework, the evalu-

ative judgments are criterion-referenced, i.e. they are based

upon specific criteria given by some component of the curricular

framework.

Diagnosis of individual pupil performance in the curricular-

free framework should be included in the assessment of individual

strengths and weaknesses (see CHART A), and should require a

criterion-referenced judgment. This same category (i.e. "To

CHART A

EVAL. RE: CURRICULAR yesGOALS?

EVAI.. RE: CURRICULAR

RULES? JUDGMENT

EVAL. RE: CURR. QUALIFYING yes

CONDITIONS?

CRITERION- OR NORM-REFERENCED EVALUATIVE JUDGMENTS OF STUDENT PERFORMANCE: THE LOGIC OF THE DECISION

START

JUDGMENT ASSESSMENT OF PUPIL ASSESSMENT OF PUPILEVAL. RE:

yes no yes RE: CURR/INSTR no PERFORMANCE TO PERFORMANCE TOCURRICULAR

CLAIM CONTEXT? DETERMINE EDUCA— DETERMINE INDIVID-CLAIM? TIONAL INSTITUTION* UAL STATUS

STATUS

USE CRITERION REFERENCE TOUSE TOJUDGMENT

CRITERIONDETERMINE DETERMINE USE

EVAL. RE: REFERENCED yes IF INSTITUTIONAL IF INDIVIDUAL AT- yesCRITERION

yes GOALS ARE TAINS INSTITUTION-INSTRUCTIONAL JUDGMENTREFERENCE ATTAINED? AL GOALS?GOALS? JUDGMENT

USE USE no noCRITERION no yes CRITERION

REFERENCED REFERENCED TO TOJUDGMENT,

EVAL. DETERMINE DETERMINEUSE yesRE: INSTRUC— INSTITUTIONAL INDIVIDUAL STRENGTHS/CRITERION

yesTIONAL QUALIFYING STRENGTHS/WEAK— WEAKNESSES? yesREFERENCED CONDITIONS? USE NESSES?JUDGMENT

NORM— USE REFERENCE

CRITERION JUDGMENT REFERENCE JUDGMENT

*An "educational institution" may have school, district, state, regional or national scope.

USE CRITERION REFERENCEDJUDGMENT

USE CRITERION REFERENCEDJUDGMENT

USENORM-

REFERENCE JUDGMENT

determine individual strengths and weaknesses") also includes

judgments made about individuals for the purpose of counseling

or job selection. These judgments could be either criterion-

referenced or norm-referenced depending upon the purpose. For

example, a student might be selected for a particular job be-

cause of performance judged in reference to a job analysis

(i.e. criterion-referenced judgment); whereas in another case

a student might be selected for a particular job based on judg-

ments of performance relative to that of other students (i.e.

norm-referenced judgment).

CONCLUDING STATEMENTS

This paper has described the nature of a curricular

framework as well as that of a curricular-free framework for

evaluative judgments of student performance. The range of

evaluative judgments in reference to pupil performance was de-

lineated for each type of framework. The logic of the decision

as to whether a judgment should be criterion-referenced or

norm-referenced was set forth in flowchart form; and included

a wide range of practical decisions encountered by educators.

Evaluative judgments within the curricular-free framework

can be either criterion-referenced or norm-referenced. However,

there are no intra-curricular judgments which may be norm-

referenced. All evaluative judgments of student performance

within the curricular framework are criterion-referenced because

they are based upon specific criteria given by some component

housed within the context of a curricular claim.