longitudinal assessment of critical thinking

LONGITUDINAL ASSESSMENT

OF CRITICAL THINKING

IN COLLEGE

WHAT MEASURES ASSESS

CURRICULAR IMPACT?

Marcia Mentkowski

Glen Rogers

Office of Research & Evaluation

Alverno College

Findings reported here are

based upon a study funded by

a grant from the National

Institute of Education (NIE-G-

77-0058), as one part of a

larger research project. Tamar

Ben-Ur assisted with the

statistical analyses reported in

this paper.

Paper presented at the annual

meeting of The Mid-Western

Educational Research

Association in Chicago,

October 1985.

Longitudinal Assessment

of Critical Thinking in

College

What Measures Assess

Curricular Impact?

Marcia Mentkowski Glen Rogers

Educational Research and Evaluation

ALVERNO COLLEGE

Milwaukee, WI

This publication is available from:

Alverno College Institute

3400 South 43rd Street

PO Box 343922

Milwaukee, WI 53234-3922

Phone: 414-382-6000

www.alverno.edu

Graphic Design: Lynn Chabot-Long, Project Specialist, Educational Research and Evaluation

Copyright 1985. Alverno College Institute, Milwaukee, Wisconsin. All rights reserved under U.S., International and

Universal Copyright Conventions. Reproduction in part or whole by any method is prohibited by law.

Longitudinal Assessment of Critical Thinking in College Page i

ABSTRACT

Longitudinal results from a college outcomes study were used to judge one widely used and three new

measures of critical thinking. Each measure was reviewed for how well it measured longitudinal vs.

cross-sectional change, for whether it was associated with progress in the curriculum, for its association

to background variables (e.g., High school GPA), and for whether it assessed change in critical thinking

for both traditional-age and older adult students. Two of the newer measures are better college

outcomes measures on these criteria. Interrelationships among the measures suggest that critical

thinking is made up of multiple components, so researchers and educators are cautioned to improve

both production and recognition measures.

INTRODUCTION

Objectives

This study explores (1) the validity of a set of four critical thinking measures, (2) the possibility of an

expanded domain of critical thinking abilities, (3) the degree and pattern of change in critical thinking

abilities in an undergraduate population, and (4) the relationship between progress in the curriculum,

which is outcome-centered and performance-based, and change in critical thinking abilities over the

course of four years.

Perspective and Theoretical Background

Educators and researchers alike are strengthening their commitment to critical thinking as a college

outcome. The American Association of Colleges, in a recent report redefining the baccalaureate, has

identified critical thinking as one of the outcomes all colleges should prepare students to demonstrate

(Integrity in the college curriculum, February 1985). In 1984, major conferences at Wingspread, Harvard,

Sonoma State and The University of Chicago addressed critical thinking assessment. Accrediting agencies

(e.g., North Central, COPA) are also calling for assessment of broad college outcomes. Recently,

attention has turned to both the assessment of individual student learning and to the institutional

evaluation of student outcomes (Marchese, 1985; Ewell, 1980; Marchese, 1985, Mentkowski & Doherty,

1984).

The need for measures that assess broad college outcomes and that are not limited to amount of

knowledge is critical. But, how are administrators and faculty to do so without instruments that

contribute to cross-college measurement, and that have face validity for liberal arts faculty? Many

liberal arts faculty do not just focus on knowledge measured by recognition tasks like SATs or GREs, but

instead focus on higher order cognitive processes that are measured by production tasks and that have

been conceptualized as central to newer definitions of critical thinking (Lipman, 1984; Nickerson, 1984;

Paul, 1984; Sigel, 1984; Sternberg, 1983; Winter & McClelland, 1978). Moreover, there is a reciprocal

relation between assessment and teaching. For example, Frederiksen (1984) observes in the American

Psychologist that use of the newer production measures could also encourage teaching of higher level

cognitive skills and provide practice with feedback. Several researchers have designed newer measures,

but how well do they work as college outcomes measures?

METHOD

Critical Thinking Measures

A total of 12 cognitive-developmental, learning style and generic ability measures were administered in

a longitudinal study of college outcomes (Mentkowski & Doherty, 1983). Four of these are the focus of

this analysis of critical thinking measures. Of these, two involve, at least partially, the more typical

recognition tasks whereby the participant must select the correct answer:

Longitudinal Assessment of Critical Thinking in College Page 2

(1) Watson and Glaser’s Critical Thinking Appraisal (Watson & Glaser, 1964) is a standardized, easily

used recognition measure of component critical thinking abilities. The Form ZM subscales

assessing quality of inferences, recognition of assumptions, and deductive reasoning were

administered in this study. These subscales are henceforth designated as the Inference,

Recognition of Assumptions, and Deduction subscales, respectively. The interpretation subscale

and the evaluation of argument subscale were not administered.

(2) Test of Cognitive Development (Renner, Fuller, Lockhead, Tomlinson-Keasey & Campbell, 1976)

is a newer series of paper and pencil tasks designed to measure formal operational thinking as

defined by Fiaget. Although participant responses for several tasks are scored from multiple

choice answers, written justification of the answers often is considered in scoring as well.

Scoring of the flexibility of rods task focuses upon participants’ written explanation of their

reasoning. The theoretical justification for the test suggests sophisticated cognitive processes

are being measured.

The other two critical thinking measures administered are pure production measures:

(3) Analysis of Argument (Stewart, 1977a, 1977b) is designed to measure flexibility in arguing (with

consistency) for opposite positions on a specified issue. In response to the stimulus, an

emotional one-sided essay, participants write two essays. The “Attack” essay is scored for

whether it has a central organizing principle and whether it focuses on the faulty logic of the

stimulus essay. The “Defense” essay is scored for whether it reflects a modified or qualified

endorsement of the counter-attitudinal stimulus essay.

(4) Test of Thematic Analysis (Winter, 1976; Winter & McClelland, 1978) is designed to measure the

ability to form complex concepts and communicate them. The task requires participants to

compare and contrast two sets of essays according to the themes in the two essay sets. Set A

includes three stimuli essays about four sentences long, as does Set B. Appendix A gives the

brief titles of the nine scoring criteria.

Study Design and Inventory Administration

All undergraduates who entered a women’s college in 1976 and 1977 were recruited as volunteer

participants in the longitudinal study. In 1977, a Weekend College timeframe was offered for the first

time. Thus, the 1977 cohort includes students from both the weekend and weekday timeframes. The

longitudinal cohorts were assessed at entrance, two years after entrance, and three and one-half years

after entrance.

As a cross-sectional comparison group, the entire 1978 weekday graduating class was recruited for

assessment at graduation. In order to control for attrition effects, the entrance scores of the students in

the 1977 weekday longitudinal cohort who did not graduate were deleted when they were cross-


sectionally compared to the scores of the 1978 graduates. Measures were administered in large group

sessions.

Data Source, Attrition, and Participation

Alverno College is a women’s college. At the time of the study, the students were predominantly

Caucasian and from one midwest state. Many were first-generation college students. The college

traditionally has served working class students from a large urban area, and has not been highly

selective.

At the first assessment, all of the women who entered in 1976 and 1977 were recruited as participants.

To be considered eligible for recruitment for later longitudinal assessments, the students had to be

currently enrolled and to have completed the prior assessments on at least a subset of the inventories.

The weekend college attracted older students in increased numbers. Longitudinal data analyzed for this

study include all students who participated on all three occasions for a particular instrument. Between

83% to 99% of the students participated at each assessment for at least a subset of the inventories. The

longitudinal data pool, n = 208, included both traditional-age (17–19 years; n = 108) and older students

(20–55 years; n = 100). The graduating class used as a cross-sectional comparison group also included

both traditional age (n = 45) and older students (n = 15).

Since not all participants completed all of the inventories, the number of observations per inventory

varies. The proportion of older and younger participants remains about the same. The lowest number of

replicated observations across three assessments occurred for the Analysis of Argument “Attack” essay,

n = 133, and “Defense” essay, n = 130. This is accounted for by a delayed decision to include the

inventory in the study, which resulted in a reduced number of observations at time 1 for this inventory.

For the other inventories reported here, between 181 and 194 participants completed all three

assessments for the particular inventory. The vast majority of the participants completed all of the

instruments. For the purposes of obtaining internal reliability estimates only, the Analysis of Argument

scores were used for all those who completed the Analysis of Argument inventory at the time of

assessment, even if they did not complete all three assessments. Thus, between 183 and 189

participants were available for these time 2 and time 3 reliability analyses.

Main Analyses Employed

Raw change over occasions of assessment was studied with multivariate analysis of variance for

repeated measures and unequal n’s in an Age X Time factorial design, with the repeated measures on

the Time of assessment. In this analysis, both the linear and the quadratic effects of Time were tested

with orthogonal polynomial contrasts. Weights were used to adjust for the unequal lengths of Time

between the intervals. The association of the critical thinking measures with rate of progress in the

curriculum was measured with correlational analyses. Alpha coefficients were computed to determine

the internal reliability of measures at each time of assessment. The amount of variance accounted for by

scores from the previous administration of each measure was computed to determine test-retest


predictability. Mentkowski and Strait (1983) have previously reported data analyses of this data set,

which are sometimes referenced here. The present analyses extend the previous analyses by exploring

the Age X Time interaction directly, by focusing entirely upon unadjusted scores, by exploring the

internal reliability of the summated measures, and by analyzing the component criteria for the Test of

Thematic Analysis and Analysis of Argument measures.

RESULTS AND CONCLUSIONS

Reliability

Analysis of Argument post-test scores showed no predictability from Analysis of Argument pre-test

scores. The inter-item reliability analysis of the “Attack” essay’s five scoring criteria (one was dropped

because of multicollinearity) yielded unacceptably low reliability coefficients at times two and three (see

Table 1), even though at least 184 students participated at these administrations. Although we believe

researchers need to accept lower reliability estimates for production measures, the coefficients for the

“Attack” essay’s scoring criteria at time two and time three do not inspire confidence in the current

measurement. Since there appeared to be sufficient variability in the scores for each scoring criterion,

we surmise that our relatively low internal reliability coefficients might be improved by increasing the

number of scoring criteria. Alternatively, a consistently unitary construct may not yet underlie the six

scoring criteria for the “Attack” essay. The internal reliability of the “Defense” essay’s four scoring

criteria was relatively high at each time of assessment (see Table 1).

The Test of Cognitive Development showed good internal reliability for the five item scale (see Table 1).

In addition, pretest scores for the Test of Cognitive Development also accounted for a high proportion of

the variance in the Test of Cognitive Development post-test scores. Averaging across the percent of

variance accounted for by time one in predicting time two and by time two in predicting time three,

29.87 of the variance (R Square) was statistically predictable.

The Test of Thematic Analysis appeared less internally reliable. The low internal reliability may be

partially due to the lack of variance in some of the nine scoring criteria, which were dichotomously

scored. The reliability coefficients for the scale were improved by removing just one scoring criterion,

“making exceptions or qualifications.” Another scoring criterion was removed because of its poor

correlation with the remaining seven criteria and its limited variance. This negatively scored criterion,

“Affect,” is scored if the participant makes a comparison that is based upon her emotional reaction to

the story. In order to prevent inappropriate extreme scores, two scoring criteria that yielded scores with

an extremely restricted variance were also removed, leaving 5 of 9 of the Test of Thematic Analysis

scoring criteria for a summated scale. These procedures yielded somewhat improved internal reliability

coefficients (see Table 1). Appendix A documents the scoring criteria included and excluded from this

exploratory 5 of 9 summated criteria-measure for the Test of Thematic Analysis.

Although internal reliability coefficients were not computed for the Critical Thinking Appraisal subscales,

the proportion of variance accounted for by pre-test scores for each interval confirms the test-retest


reliability of these subscales. Averaged across the two intervals, pretest scores account for 26.3% of the

Inference post-test scores, for 19.17 of the Recognition of Assumptions post-test scores, and for 31.97

of the Deduction post-test scores. Although the relatively high predictability of the post-test scores from

the pretest scores supports the reliability of this set of Critical Thinking Appraisal measures, the lower

predictability of the post-test scores for the Test of Thematic of Analysis may be only due to individual

differences in response to educational or other influences.

A Face Validity Issue

A qualitative examination of the “Defense” essays written by the students suggested that the students

were role playing the emotional and faulty arguments they were asked to defend. This face validity

concern is consistent with the scoring of the “Defense” essays. At each time of assessment, the scoring

of the “Defense” essays according to the four scoring-criteria yielded very little variance among

students. In the case of the first assessment, for example, 937 “totally” endorsed the presented

argument (which was probably counter to their attitude); 917 presented new arguments supporting the

counter-attitudinal statement; 47 made a modified endorsement of it; and 47 accepted a particular part

of it. In other words, in the “Defense” essay, almost all of the entering freshman did exactly opposite of

what would yield a high score in the standard scoring. Our qualitative observation that the students

were role playing the bad arguments they were asked to defend is also consistent with the lack of

instructions to dissuade against role playing in the Analysis of Argument inventory. We also observe that

our students encounter numerous role playing exercises in their education.

Trends through Time

Our concerns about the face validity of the “Defense” essays and about the internal reliability of the

“Attack” essay scoring criteria are supported by the previously conducted analyses on this data set (see

Mentkowski and Strait, 1983). We know from these analyses that neither the “Attack” essay’s

summated measure nor the “Defense” essay’s summated measure yielded any statistically significant

differences across the times of assessment. Mentkowski and Strait (1983) also found that these

summated measures were not related to measures of progress in the college’s curriculum. The lack of

internal reliability for the “Attack” essay’s scoring criteria suggested an exploratory analysis with the

individual criteria as the unit of analysis. None of these individual scoring criteria yield statistically

significant differences for the Time of the assessment, however.

Given the exploratory nature of this research, the Test of Thematic Analysis was analyzed in more than

one way. Not only was a total score for the Test of Thematic Analysis included (in this case, the 5 of 9

criteria summated measure), but also, three of the scoring criteria derived from this measure were

separately analyzed. Thus, analysis of variance was also performed on (1) the criterion of making

exceptions or qualifications to one’s ascriptions, (2) the criterion of giving examples for observations,

and (3) the negatively scored criterion of affective reaction. It should be noted that the examples

criterion is a component of the Test of Thematic Analysis summated measure (5 of 9 criteria). As is

noted below, analysis of this summated measure and analysis of its component scoring criterion,


examples, yield very similar results. This is probably largely because the scores yielded by the examples

scoring criterion actually contribute the bulk of the variance to the summated score. Thus, like the

summated scale, the examples scoring criterion does not correlate positively with the exceptions

criterion at any of the times of measurement. Again, the direction of association is always negative,

even though, in this case, it is not statistically significant.

The Test of Thematic Analysis scoring criterion for affect-based comparisons was scored for only 16% of

the essay comparisons. Because of low variance, a Friedman’s distribution-free one-way analysis of

variance procedure for time of assessment was again conducted. It showed no statistically significant

difference, Chi Square (2, 194), F < 1.

The Age X Time repeated measures analysis of variance procedure for unequal n’s and unequal intervals,

which tested for the linear and quadratic effects of Time, were performed on the following seven

measurements: the Test of Cognitive Development summated measure, the Test of Thematic Analysis

summated measure (5 of 9 criteria), the examples and exceptions scoring criteria for the Test of

Thematic Analysis, and the three subscales of the Critical Thinking Appraisal. This subset of seven

measurements, which includes both summated scales and single item scores, yielded many effects. The

Age X Time analyses yielded main effects for Age on 4 of 7 of these measurements (see Table 2). They

also yielded main effects for linear Time on all seven of these measurements. A main effect for quadratic

Time was revealed on the Test of Thematic Analysis (5 of 9 criteria) and the Recognition of Assumption

subscale of the Critical Thinking Appraisal (see Table 2). The lack of any interaction effects between Age

and Time (all F values less than 1) suggests that these main effects of linear Time, and quadratic Time

are relatively general for both age groups.

First, looking at those measures not associated with Age effects, we see that scores on the Test of

Cognitive Development and the scores on the Deduction subscale of the Critical Thinking Appraisal

improve linearly with Time (see Table 3). Recall that the analysis of variance procedure (see Table 2)

adjusted for unequal interval lengths.

Next, looking at those measures with Age main effects (see Table 4), older age students show a fairly

constant advantage on these measures at each time of assessment. There is no statistically significant

interaction between Age and Time for scores yielded by the exceptions scoring criterion despite the

apparent convergence of the means at time three for the two age groups.

The quadratic effect of Time for the Test of Thematic Analysis summated score (5 of 9) is reflected in a

drop on this measure at time three (see Table 4), which is confirmed by a posteriori analyses. There is

also a linear decrease with Time (see Table 2). The examples scoring criterion, which is a component of

this summated measure, likewise shows a linear (see Table 2) decrease with Time (see Table 4). In

contrast, the exceptions scoring criterion shows a linear (see Table 2) increase with Time (see Table 4).

Thus, by exploring the scoring criterion components of the Test of Thematic Analysis, we have found

trends in opposite directions.


The Inference subscale of the Critical Thinking Appraisal shows a linear (see Table 2) increase with Time

(see Table 4). Like the linear increases with Time for the Deduction subscale (see Table 3), these

increases occur generally across our two age cohorts. The Recognition of Assumptions subscale shows

both a linear and quadratic effect (see Table 2). Examination of Table 4 shows an increase on the

Recognition of Assumptions subscale at the second interval, which is confirmed by a posteriori analyses.

Relation of Measures to Curricular Progress

Are these critical thinking and exploratory single item measures that have shown change as a function of

time of assessment sensitive to curricular impact? In preparation for examining the relation between

curriculum progress and performance on the critical thinking measures, we tabulated the measures of

curriculum progress according to two separate cumulative subtotals: one subtotal for the first two years

since enrollment and the other subtotal for the subsequent year and a half. The one curriculum progress

subtotal corresponds to the interval of time between the first and second fieldings of the critical

thinking measures, and the other curriculum progress subtotal corresponds to the interval of time

between the second and third fieldings of the critical thinking measures.

We reasoned that students showing a high degree of curriculum progress in the intervals of the study

would at the end of the intervals show higher performance on the critical thinking measures. Thus,

progress in the curriculum up to the second assessment was correlated with the critical thinking

measures at the second assessment, and progress in the curriculum between the second and third

assessments was correlated with the critical thinking measures at the third assessment.

The measures of curriculum progress employed are conceptually well suited for showing the kind of

student development relevant to growth of critical thinking. The curriculum of Alverno College is

abilities-based; On over 100 assessments Alverno students demonstrate to criteria each of eight broad

abilities set by the faculty (eg. communication, analysis, problem solving, and valuing). These

assessments lead to the credentialing of the student on the sequential and developmentally arranged

“competence level units.” The number of competence level units completed and the number of course

credits completed were employed as two separate measures of progress in the curriculum.

Examination of Table 5 shows that the Test of Cognitive Development scores at time two are positively

correlated with cumulative progress in the curriculum. At time three, the Test of Cognitive Development

scores do not correlate with the intervening progress in the curriculum (see Table 5).

The Test of Thematic Analysis showed a somewhat complex relation to progress in the curriculum. The

Test of Thematic Analysis summated scale (5 of 9 criteria) was negatively associated with curriculum

progress at both the second and third assessments.

While the examples scoring criterion performs similarly to the summated scale, of which it is a

component, the exceptions scoring criterion yields different results. Thus, the examples criterion is

negatively associated with curriculum progress at the second assessment, and, for at least one of the


measures of curriculum progress, is also negatively associated with curriculum progress at the third

assessment (see Table 5). Again in contrast, the exceptions criterion shows a small positive relation to

progress in the curriculum at the third assessment (see Table 5).

For the three Critical Thinking Appraisal subscales only one of the twelve correlations with progress in

the curriculum is statistically significant (see Table 5). The Test of Cognitive Development and the Test of

Thematic Analysis and its single item subcomponents were related to progress in the curriculum, but the

Critical Thinking Appraisal measures generally were not. Why not?

Level of Thinking, Mastery of That Level, and Habituality of Use

We believe the lack of relationship of the Critical thinking Appraisal with the measures of curriculum

progress flows from the multiple choice format of the Critical Thinking Appraisal. Before we clarify why

we suspect this, we need to distinguish between some broad performance characteristics. From a

developmental perspective we might describe one performance as demonstrating a higher level of

thinking than another. Higher levels of thinking are perhaps more complex, broader in scope, and so on.

Also, a sophisticated thought process could be based upon the coordination of several thought

processes, and might be predicated upon the development of other thought processes.

We can conceptually distinguish the sophistication or level of a thought process from other performance

characteristics. An individual can be said to have mastered a particular thinking process when, at a

certain frequency, they can demonstrate the process of thinking when it is explicitly requested. But,

even if an individual has mastered a level or kind of thinking, they might not habitually employ it. For

example, they may not be called on to use it, they may have opposing tendencies, or they may not know

it is expected of them. The distinctions between sophistication of a thinking process, mastery of a

thinking process, and habitual use of a thinking process may help explain why some Critical Thinking

measures that showed development during the college years also showed a relationship to curriculum

progress, while others did not.

Because of the recognition character of the Critical Thinking Appraisal, we do not feel confident in it as a

measure of well-practiced mastery. College students can probably recognize higher levels of thinking

than they can systematically produce. Performance on the Critical Thinking Appraisal may have more to

do with the breadth of the exposure to the thinking processes the test taps, than with a well-practiced

mastery of the thinking processes. Thus, although we are able to show gains on all of the Critical

Thinking Appraisal subscales, these robust gains may indicate that it is relatively easy to show progress

on these recognition based measures.

Even those students progressing slowly in the curriculum may have a breadth of exposure to the

thinking processes required by the Critical Thinking Appraisal. Indeed, the breadth of exposure that

slower progressing students have compared to that of faster progressing students may be of equal value

in recognizing the “correct” answers on the Critical Thinking Appraisal. This assumption would help


account for the finding that rate of progression in the curriculum is not generally related to subsequent

performance on the Critical Thinking Appraisal subscales.

Nonetheless, we must note that the Recognition of Assumptions subscale gains were most strongly

associated with the second interval in the study. This suggests that a sophisticated critical thinking

process was being measured. And, theoretically, the sophistication required to recognize assumptions of

arguments does seem high enough to account for delayed development in the curriculum. If a later

curriculum exposure is responsible for the delayed development on the Recognition of Assumptions

subscale, it would seem that this argues for expecting to find a relationship with curriculum progress.

But, could it be that both the slower and faster progressing students were both far enough along in the

curriculum to be exposed sufficiently to this thinking process as it was required by the recognition test?

We think so, and can note that the variance in the rate of progression of our students was not

inordinately large.

In explaining the relationship of the Test of Thematic Analysis to progress in the curriculum, we

conversely focus on its characteristic of being a production measure. We believe these curriculum-linked

changes on the Test of Thematic Analysis and its single item criterion-scores are consistent with an

interpretation that focuses on the assumed relationship between production measures and the

measurement of either broadly based mastery or habituality. More specifically, we suspect that mastery

of a way of thinking or habituality in a way of thinking develops incrementally and would not be

sensitive to minimal curriculum exposure. As a result, a range of progress in the curriculum would be

likely to be associated with differential performance on a production measure, which is what we found.

The response format of the Test of Cognitive Development is not purely either recognition or

production, and so it is inappropriate to speculate on how the results for this measure reflect upon our

thesis that production measures capture more habitual tendencies or more well-practiced abilities. We

do, however, gain greater confidence in this measure as a measure of college outcomes because of its

relation to progress in the curriculum. Furthermore, the historical development of this measure in the

Research on Formal Operations does suggest the instrument as a measure of sophisticated and well-

practiced thinking.

Inter-Relation of Critical Thinking Measures/Background

Variables

More generally, we find other evidence that these various instruments are measuring different aspects

of higher-order thinking. First, the Test of Thematic Analysis (the purest production measure) is related

to Age, but the Test of Cognitive Development is not. Second, the Test of Cognitive Development and

the Critical Thinking Appraisal subscales are related to high school GPA at entrance, but the Test of

Thematic Analysis is not. Finally, although these measures are correlated with one another positively

(see Table 6), previously reported factor analyses (see Mentkowski & Strait, 1983) supports distinctions

between the summated measures of critical thinking for the first two assessments.


Comparison with Cross-Sectional Results

We are compelled to offer another word of caution. Our own cross-sectional analyses would have led to

different conclusions than our longitudinal analyses. The cross-sectional analyses, which controlled for

attrition, showed a “gain” only on the “Defense” score for the Analysis of Argument instrument, F(1,125)

- 7.26, p<.01, and the Inference subscale of the Critical Thinking Appraisal, F(1, 127) - 12.75, p<.001.

Moreover, because the cross-sectional comparison involves the comparison of existing groups (which

differed on high school GPA, for example), even these “gains” would have been potentially spurious.

When we did control statistically for high school GPA in our cross-sectional comparison, the cross-

sectional Inference “gains” were no longer statistically significant. We place greater faith in our

longitudinal changes, of course. We point up this discrepancy between the longitudinal and cross-

sectional results, because we believe it is an object lesson in the problems of interpretation from cross-

sectional findings, even though each method has its place.

Interpreting the New Production Measures

The task of more specifically interpreting changes on Test of Thematic Analysis remains. We believe the

decline on the Test of Thematic Analysis summated measure (5 of 9 criteria) can be attributed to the

decline on the examples criterion. We interpret the tendency of students to give less examples, but to

make more exceptions, as a positive outcome. The increasing tendency of students to make exceptions

in their analysis of the essays suggests that the thinking of the students may be becoming more abstract.

The decreasing tendency of the students to give examples of their abstractions seems somewhat

puzzling at first. In order to explain this, we make the following observations.

Faculty at Alverno College are pioneers in the writing across the curriculum movement. They explicitly

teach students to write for a particular audience, and they have developed standards of performance on

this dimension. Since the format of the Test of Thematic Analysis suggests that researchers are the

audience, we imagine them increasingly reasoning as follows as they repeatedly encounter the Test of

Thematic Analysis: “These researchers assuredly are aware of what the four sentence essays say. I don’t

need to give examples from these essays to make my points, because they already know the examples.

They would be bored by the obviousness of the examples, which would be about as long as the set of

essays.”

We believe styles of writing developed across school programs, as well as across disciplines, may vary.

Thus, we encourage a complex interpretation of the decline and gain on the Test of Thematic Analysis

scores. Future modifications of the instrument should consider, at a minimum, providing directions

which identify a more explicit audience

Although the findings for the Analysis of Argument instrument cast some doubt upon its suitability for

use at this college as an outcomes measure, we believe modifications in the instrument may overcome

these difficulties.


In regard to the instructions for the “Defense” essay, if they advised against role playing, this might

improve the “Defense” essay’s measurement properties. In effect, students are penalized for role

playing “unsophisticated” thinking, and for empathizing with the perspective of the essay. In regard to

the “Attack” essay’s instructions, we note that they do not specifically request students to analyze the

adequacy of the arguments in the stimulus essay. Instead the instructions ask the students to “argue

against” the stimulus essay. The stimulus essay was an emotional one-sided essay that presented an

unsupported position likely to be counter to the attitudes of the students. In their argument against this

stimulus essay, the students in the present study predominantly tended to present the opposing

position, which would be their own opinion, as opposed to attacking the logic of the essay. The

emotional one-sided stimulus essay may have elicited an emotional need to present the opposing

position.

Thus, with respect to the lack of change on the Analysis of Argument “Attack” scores, one might

speculate that these students have not specifically and habitually learned to give precedence to a logical

analysis of the flaws in their opponent’s arguments; instead, their predominant tendency may have

been to give a positive statement of the opposite position. If the instructions had specifically requested

an analysis of the merits and flaws of the stimulus essay, we may have found developmental differences

on this measure. We are proposing a distinction here between the ability to write, when requested, an

intellectual critique in an emotional environment versus the habitual tendency to write a purely

intellectual critique in an emotional environment.

Implications for Standardization of Measures

We note that Winter, McClelland, and Stewart (1981) reported both longitudinal and cross-sectional

gains for the Analysis of Argument and Test of Thematic Analysis at “Ivy” College. These findings

contrast somewhat with our findings at Alverno College, even though we used the same scoring

procedures. We tend to discount the cross-sectional gains for Analysis of Argument we found because of

our lack of longitudinal results, while we feel encouraged by our perhaps unique bi-directional

longitudinal changes on the single criterion-scores of the Test of Thematic Analysis. Although we would

like to have been able to demonstrate the robustness of the standard scoring and administration

procedures developed at “Ivy” college, we have not been able to show cross-college equivalence for

these measures.

The likelihood that different educational strategies are used in different colleges has wide implications

for using scoring schemes that are assumed to be standard, and requires further consideration. For

instance, colleges may differ in how writing is taught and in how often role playing is pedagogically used,

and these and other stylistic differences in instruction may cause students to interpret differently the

instructions on these production measures. This confounding may greatly complicate the cross-

institutional measurement properties of these measures. Not only may differing educational contexts

elicit different interpretations of how to perform well on an instrument, but also these different

interpretations may be equally reasonable if the instrument itself does not communicate to the student

the preferred type of performance.


It is also possible that these differing educational strategies reflect differing conceptions of critical

thinking outcomes. For example, at Alverno, role playing is often explicitly used to encourage students

to take tolerantly the perspective of another person or culture. Students are encouraged to consider the

cultural or personal assumptions they make. In contrast, the scoring procedures standardized by “Ivy”

college for the Analysis of Argument “Defense” essay is based upon another conception. The ideal

student portrayed in these scoring criteria. will insist upon maintaining a position consistent with her

own, even as she obligingly defends the good points present in an opponent’s position. There is no

necessary recognition of the cultural supports for one’s own beliefs or of the perhaps equal validity of

another’s views. Another example of possibly different conceptions of critical thinking can be illustrated

with the scoring of the “Attack” essay. The “Ivy” college scoring procedures reflect a preference for a

logical analysis of the stimulus essay. The present college, however, also encourages students to express

their own values and positions as well as encouraging logical analysis.

If either the conception of critical thinking differs across colleges, or, else, the students’ interpretation

based upon their educational context differs, even the direction of “gain” or “decline” on a criterion may

also differ. This concern suggests that a single college population—however reputable—may not serve

well as the major source for standardizing scoring criteria.

In many instances, it would be desirable to try to insure that these production measures elicit similar

interpretations of the task by students from different colleges. Here a distinction needs to be made

between the respondent’s ability to conform to a standard versus the respondent’s predominant

tendencies in unstructured situations. We do not rule out the development of several standardized

scoring systems, or even the particularization of scoring systems to a college’s conception of critical

thinking. What we earnestly recommend for production measures of ability, however, is that they offer

to the respondent a clear idea of the standards that will be applied to their productions, if the

researcher intends to represent their performance as their ability to conform to a standard.

A more unstructured stimulus may be used when the researcher is interested in the predominant

tendencies of the respondents. In such cases, the researcher reeds to be especially sensitive to the

distinction between necessary habits supporting an ability and stylistic divergence. We feel that the

Analysis of Argument and Test of Thematic Analysis do elicit habitual tendencies, but are not yet

sensitive to stylistic divergence. Their instructions draw forth the respondent’s habitual tendencies, but

the standardized scoring reflects a preferred response that does not accommodate alternative modes

and styles of critical thinking.

In general, we recommend fielding both types of production measures, those measuring the ability to

demonstrate mastery to standards and those measuring habitual tendencies. We also recommend

keeping these types of measures separate and distinct. Otherwise, it may not be possible to unconfound

them.


Educational/Scientific Importance of the Study

Researchers are encouraged by these results to continue development of measures which ask students

to show the process of their thinking in addition to showing comprehension of knowledge, concepts or

generalizations. Critical thinking appears best understood as comprised of several dimensions, which at

this college develop differently across the college years. Researchers developing production measures

need to distinguish in their measurement between habitual tendencies versus the ability to show

mastery to a standard when requested. At present, the Analysis of Argument and Test of Thematic

Analysis tend to be similar to projection measures in that they do not specifically guide responses and,

as a result, tend to measure habitual tendencies. Our results suggest that these instruments need

further refinement if they are to be used as standardized cross-college outcome measures. We were not

able to show their scoring criteria to be internally reliable.

We remain confident in the potential usefulness of the Test of Thematic Analysis and Analysis of

Argument to a wide variety of colleges once they have undergone further development to either tailor

them to the institution’s own educational goals and definitions, or to improve their generic cross-

institutional equivalence. Even at this early stage of development, we have found the findings from the

instruments useful in suggesting the habitual tendencies of our students. The Test of Thematic Analysis

appeared sensitive to changes in their habits of thought.

We urge educators to support researchers developing production measures and not to rely entirely on

traditional, though efficient measures that may not work well as measures of well-practiced mastery.

Production measures seem better suited to the measurement of the kind of mastery obtained by the

consistent practice of higher-order thought and of the kind of habituality of thought processes that

would lead to their use across situations. We have suggested that the Recognition of Assumptions

subscale may tap a relatively sophisticated thought process, but we also note that some sophisticated

thought processes, for example, as used in the integration of perspectives, may require a production

measure. We encourage researchers to broaden their scoring schemes to include the kind of critical

thinking practiced by adults already in the working world (Arlin, 1975; McClelland, 1973).

Educators should be encouraged by our conclusion that critical thinking develops in college. The Alverno

students not only showed gains on the Piagetian-based Test of Cognitive Development, but also their

performance on this measure has been linked with their prior progress in the curriculum (cf.

Mentkowski & Strait, 1983). These students also showed gains on each of the three subscales of the

Critical Thinking Appraisal that we fielded. This finding is tempered somewhat by our inability to show a

direct relationship between performance on this measure and progress in the curriculum. Performance

on the Test of Thematic Analysis did appear to be linked to progress in the curriculum. In this regard, the

Alverno students may have developed the habit of making more exceptions or qualifications to their

analyses. Although they also may have weakened in their habit of giving examples to their analyses, we

suspect this finding is context bound.


So far, we are able to support the usefulness of only two of three new measures of critical thinking. We

are cautioned in interpreting these study results. For example, although this study generally confirmed

the usefulness of the Test of Thematic Analysis, a comparative cross-sectional study of seven colleges

was able to demonstrate change on the Test of Thematic Analysis for only two of the seven: “Ivy”

College, a famous and highly selective one and Alverno College (see Winter, McClelland, and Stewart,

1981). Why? We have already noted that the instrument criteria and instructions were standardized by

a single institution, “Ivy” college We also note that the findings for Alverno reported in that study, which

were based upon a cross-sectional comparison of entering versus graduating students, have not been

confirmed by our longitudinal results. But, even if the summated results from “Ivy” college and the

results from the present analysis of the scoring criteria for the Test of Thematic Analysis are taken as

suggestive of possible differences, we have yet to demonstrate robust cross-college findings. We note

that at Alverno College, faculty have identified critical thinking abilities as college outcomes and

developed teaching strategies and instruments to teach and assess them. Perhaps, colleges cannot

expect change on production measures without such an explicit curriculum or without highly selective

student bodies. If so, it means that researchers and educators must work together at both instrument

and curriculum development at a range of colleges.


REFERENCES

Arlin, P. (1975). Cognitive development in adulthood: A fifth stage? Developmental Psychology, 11(5), 602-606.

Ewell, P. (1984). The self-regarding institution: Information for excellence. Boulder, CO: National Center for Higher

Education Management Systems.

Fredericksen, N. (1984). The real test bias: Influences of testing on teaching and learning. Paper presented at a

conference on Teaching Thinking Skills, Wingspread Conference Center, Racine, WI.

Lipman, M. (1984). Philosophy and the cultivation of reasoning_. Paper presented at a conference on Teaching

Thinking Skills, Wingspread Conference Center, Racine, WI.

Marchese, T. (1985). Learning about assessment. AAHE Bulletin, 38(1), 10-13.

McClelland, D. (1973). Testing for competence rather than for “intelligence.” American Psycologist, 28, 1-14.

Mentkowski, M., & Doherty, A. (1984). Abilities that last a lifetime: Outcomes of the Alverno experience. AAHE

Bulletin, 36(6), 1-6, 11-14

Mentkowski, M., & Doherty, A. (1983, revised 1984). Careering after college: Establishing the validity of abilities

learned in college for later careering and professional performance. Final report to the National Institute of

Education: Overview and summary. Milwaukee, WI: Alverno Productions.

Mentkowski, M., & Strait, M. (1983). A longitudinal study of student change in cognitive development, learning

styles, and generic abilities in an outcome-centered liberal arts curriculum. Final report to the National

Institute of Education, research report number six. Milwaukee, WI: Alverno Productions.

Nickerson, R. (1984). Teaching thinking: What is being done and with what results? Cambridge, MA: Bolt Beranek

and Newman, Inc.

Paul, R. (1984). The concept of critical thinking: An analysis, a global strategy, and plea for emancipatory reason.

Rohnert Park, CA: Sonoma State University Center for Critical Thinking and Moral Critique.

Renner, J., Fuller, R., Lockhead, J., Johns, J., Tomlinson-Keasey, C. & Campbell, T. (1976). Test of Cognitive

Development. Norman, OK: University of Oklahoma.

Sigel, I. (1984). Reflection on thinking about thinking: The educational discovery of the 80’s? Paper for presentation

at a conference on Teaching Thinking Skills, Wingspread Conference Center, Racine, WI.

Sternberg, R. (1983). How can we teach intelligence? Philadelphia, PA: Research for Better Schools, Inc.

Stewart, A. (1977a). _Analysis of argument:, An empirically-derived measure of intellectual flexibility. Boston:

McBer and Company.

Stewart, A. (1977b). Scoring manual for stages of psychological adaptation to the environment. Unpublished

manuscript, Department of Psychology, Boston University.

Watson, G., & Glaser, E. (1964). Critical Thinking Appraisal. New York: Harcourt, Brace, Jovanovich.


Winter, D. (1976). The Test of Thematic Analysis. Boston: McBer and Company.

Winter, D., & McClelland, D. (1978). Thematic analysis: An empirically derived measure of the effects of liberal arts

education. Journal of Educational Psychology, 70, 8-16.

Winter, D., McClelland, D., & Stewart, A. (1981). A new case for the liberal arts: Assessing institutional goals and

student development. San Francisco: Jossey-Bass.


Table 1: Inter-Item Reliability, Chronbach’s Alpha, for Each Time

of Assessment1

Measure Time 1 Time 2 Time 3

Test of Cognitive Development NA .472 .54

Test of Thematic Analysis (all 9criteria) .28 .11 .19

Test of Thematic Analysis (5 of 9criteria) .36 .17 .35

Analysis of Argument Attack items .33 .17 .09

Analysis of Argument Defense items .79 .46 .75

1 For the reliability analyses, each scoring criteria was used as an item. For the Test of Cognitive

Development, 5 scores were used as items. For the Analysis of Argument “Attack” essay, 5 scoring

criteria were also available. For the Analysis of Argument “Defense” essay, 4 criteria were coded, but

only 3 were used in the reliability analysis because of multicollinearity.

NA The reliability coefficient is not available because only total scores were keypunched.

2 At time 2, the Test of Cognitive Development reliability coefficient is based only on the 1977 cohort,

because only the total score for 1976 cohort was keypunched.


Table 2: Age by Time Repeated Measures ANOVAs for Linear and

Quadratic Contrasts

Measures Age

Main Effect

Linear Time

Main Effect

Quadratic Time

Main Effect

Test of Cognitive Development F(1,189) < 1 F(1,189) = 14.7*** F(1,189) = 1.7

Test of Thematic Analysis

(5 of 9 criteria) F(1,192) = 22.4*** F(1,192) = 3.9* F(1,192) = 5.1*

Test of Thematic Analysis exception F(1,192) = 3.6 F(1,192) = 14.0*** F(1,192 < 1

Test of Thematic Analysis example F(1,192)-16.4*** F(1,192) = 16.6*** F(1,192) < 1

Inference Subscale of Critical

Thinking Appraisal F(1,180) = 6.6* F(1,180) = 19.1*** F(1,180) < 1

Recognition of Assumptions Subscale

of Critical Thinking Appraisal F(1,180) = 4.9* F(1,180) = 4.1* F(1,180) = 6.8*

Deduction Subscale of Critical

Thinking Appraisal F(1,179) < 1 F(1,179) = 19.6*** F(1,179) < 1

1 Multivariate analysis of variance testing combined effects of linear and quadratic quadratic time were

statistically significant for all reported main effects. Both linear and quadratic tests of the Age by

Time interaction failed to reach to reach statistical significance (all F values less than 1).

* p <.05

** p < .01

*** p < .001


Table 3: Means for Linear Time Main Effect Ignoring Age

Measures Time 1 Time 2 Time 3

Test of Cognitive Development 11.45 12.24 12.37

Deduction Subscale of Critical Thinking Appraisal 16.10 16.64 17.16


Table 4: Means for Time of Assessment Broken Down for the Age

Main Effect

Measure Time 1 Time 2 Time 3

Test of Thematic Analysis (5 of 9 criteria)

Age 17 to 19 1.09 1.19 .91

Age 20 to 55 1.56 1.57 1.32

Test of Thematic Analysis, exception

Age 17 to 19 .25 .36 .48

Age 20 to 55 .37 .48 .49

Test of Thematic Analysis, example

Age 17 to 19 .32 .22 .16

Age 20 to 55 .50 .41 .29

Inference Subscale of Critical Thinking Appraisal

Age 17 to 19 8.97 9.56 9.58

Age 20 to 55 9.87 10.45 10.93

Recognition of Assumptions Subscale of Appraisal

Age 17 to 19 10.96 10.52 11.18

Age 20 to 55 11.26 11.35 12.01


Table 5: Correlation of Critical Thinking Measures with Progress

in the Curriculum On “Competence Level Units” (CLU’s)1 and On

Credits Achieved

Measures Time 2

CLU's

Time 2

Credits

Time 3

CLU's

Time 3

Credits Test of Cognitive Development .21** .15* .03 .08

Test of Thematic Analysis (5 of 9 criteria) –.20** –.31*** –.17** –.29***

Test of Thematic Analysis, exception –.04 –.09 .10 .13*

Test of Thematic Analysis, example –.21** –.29*** –.10 –.25***

Inference Subscale of Critical Thinking Appraisal .05 –.07 –.08 –.15*

Recognition of Assumption Subscale of Critical Thinking

Appraisal .09 –.03 –.11 –.07

Deduction Subscale of Critical Thinking Appraisal .04 .06 –.11 –.02

1 Alverno Students demonstrate to criteria on over 100 assessments each of 8 broad abilities, which

have been set by the faculty (e.g. communication, analysis, problem solving, etc.). It is these

assessments that lead to the credentialing of the student on the sequential and developmental

arranged “competence level units.”

* p < .05

** p < .01

*** p <.001


Table 6: Correlations between the Critical Thinking Measures at

Time One

Measures

Test of Cognitive

Test of

Cognitive

Development

CTA

Inference

Subscale

Recognition of

Assumptions

Subscale

CTA

Deduction

Subscale Test of Cognitive Development .28*** .21** .35***

Test of Thematic Analysis (5 of 9 criteria) .38*** .20** .16* .29***

Test of Thematic Analysis, exception –.05 .10 .17** .06

Test of Thematic Analysis, example .24** .08 –.01

.17*

Inference Subscale of Critical Thinking

Appraisal .29*** .29***

Recognition of Assumptions Subscale of

Critical Thinking Appraisal

.38***

Deduction Subscale of Critical Thinking

Appraisal

* p < .05

** p < .01

*** p <.001


Appendix A: Test of Thematic Analysis Criteria Included Versus

Not Included

Criteria Included in 5 of 9 Total Score

Making “Direct Compound Comparisons” positively scored

Giving “Examples” positively scored

Using an “Analytic Hierarchy” positively scored

“Redefinition” for Scope or Clarity positively scored

Comparing “Apples and Oranges” negatively scored

Criteria Excluded From 5 of 9 Total Score

Making “Exceptions” or “Qualifications” positively scored

“Subsuming Alternatives” positively scored

“Affect” negatively scored

“Subjective Reaction” negatively scored

longitudinal assessment of critical thinking

Documents