accountability and need for cognition effects on contrast, halo, and accuracy in performance ratings

This article was downloaded by: [University of Haifa Library]On: 20 May 2013, At: 07:57Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH,UK

The Journal of Psychology:Interdisciplinary and AppliedPublication details, including instructions forauthors and subscription information:http://www.tandfonline.com/loi/vjrl20

Accountability and Needfor Cognition Effects onContrast, Halo, and Accuracy inPerformance RatingsJerry K. Palmer a & Jack M. Feldman ba Eastern Kentucky University Department ofPsychologyb Georgia Institute of Technology Department ofPsychologyPublished online: 07 Aug 2010.

To cite this article: Jerry K. Palmer & Jack M. Feldman (2005): Accountability andNeed for Cognition Effects on Contrast, Halo, and Accuracy in Performance Ratings,The Journal of Psychology: Interdisciplinary and Applied, 139:2, 119-138

To link to this article: http://dx.doi.org/10.3200/JRLP.139.2.119-138

PLEASE SCROLL DOWN FOR ARTICLE

Full terms and conditions of use: http://www.tandfonline.com/page/terms-and-conditions

This article may be used for research, teaching, and private study purposes.Any substantial or systematic reproduction, redistribution, reselling, loan,sub-licensing, systematic supply, or distribution in any form to anyone isexpressly forbidden.

The publisher does not give any warranty express or implied or make anyrepresentation that the contents will be complete or accurate or up todate. The accuracy of any instructions, formulae, and drug doses should be

http://www.tandfonline.com/loi/vjrl20

http://dx.doi.org/10.3200/JRLP.139.2.119-138

http://www.tandfonline.com/page/terms-and-conditions

http://www.tandfonline.com/page/terms-and-conditions

independently verified with primary sources. The publisher shall not be liablefor any loss, actions, claims, proceedings, demand, or costs or damageswhatsoever or howsoever caused arising directly or indirectly in connectionwith or arising out of the use of this material.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

The Journal of Psychology, 2005, 136(2), 119–137

119

Accountability and Need for CognitionEffects on Contrast, Halo, and Accuracy

in Performance Ratings

JERRY K. PALMERDepartment of Psychology

Eastern Kentucky University

JACK M. FELDMANDepartment of Psychology

Georgia Institute of Technology

ABSTRACT. In the present study, the authors investigated the effects of accountabilityand need for cognition on contrast errors, halo, and accuracy of performance ratingsexamined in good and poor performance context conditions, as well as in a context-freecontrol condition. The accountability manipulation reduced the contrast effect and alsomodified rater recall of good ratee behavior. Accountability reduced halo in ratings andincreased rating accuracy in a poor performance context. Accountability also interactedwith need for cognition in predicting individual rater halo.

Key words: accuracy, contrast, halo, need for cognition, performance ratings

WE INVESTIGATED THE EFFECTS OF ACCOUNTABILITY on contrasterrors and rating accuracy in performance ratings and examined the interactiveeffects of accountability and need for cognition (Cacioppo & Petty, 1982) onthe same. Contrast errors in performance appraisals occur when previouslyviewed and judged (context) performances influence ratings of a subsequentlyviewed and judged target performance such that ratings of the target are dis-placed away from the evaluative level of context performance ratings (Maurer& Alexander, 1991). This contrast error has proven very difficult or impossibleto eliminate in laboratory settings (Maurer, Palmer, & Ashe, 1993; Wexley,Sanders, & Yukl, 1973).

Theoretical works by Tesser (1978), Kunda (1990), and Feldman and Lynch(1988), however, suggest that prior attempts to eliminate contrast errors have

Address correspondence to Jerry K. Palmer, Department of Psychology, Cammack 127,Eastern Kentucky University, Richmond, KY 40475; [email protected] (e-mail).

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

failed because the experimental manipulations altered the intensity, rather thanthe direction, of raters’ thoughts when performing ratings. This distinctionbetween intensity and direction of cognitive processing (Kunda) is central to ourpresent arguments and study. We believe that past attempts to reduce contrasterrors have merely increased the intensity of processing without altering itsdirection and that altering the direction instead might reduce the severity of con-trast as well as other accompanying errors in performance ratings and interviewdecisions (Palmer, Feldman, & Maurer, 1998).

Accountability

Accountability in a performance appraisal context means that there are con-sequences that depend on some aspect of the ratings or judgments given (Feld-man, 1994). In a performance appraisal setting, a rater’s reactions to account-ability include efforts to enhance or protect his or her self-image and attempts tomaximize rewards and minimize punishments (Lerner & Tetlock, 1999; Tetlock,1985). In each case, accountability motivates the rater to engage in thoughtful, asopposed to automatic, processing (Feldman, 1994; Fiske & Neuberg, 1990),although the outcome of such processing depends on the nature of the account-ability contingencies. The influence of additional processing on the direction ofthought and on ratings depends on the goals of the rater. A goal to enhance one’sself-image may result in deliberate distortion in the form of leniency in ratings(Klimoski & Inks, 1990; Longnecker, Sims, & Gioia, 1987) or in the develop-ment of arguments that support the ratings (Eagly & Chaiken, 1993), for exam-ple, in an effort to appear not easily swayed. Accountability thus directs, as wellas intensifies, thinking (Kunda, 1990).

When accountability contingencies motivate the rater to produce moreaccurate ratings, the rater will carefully attend to ratee behavior or search mem-ory for information relevant to the judgment task rather than rely on heuristicsor summary judgments (Feldman & Lynch, 1988). This works in a fashion anal-ogous to the effects of different levels of personal involvement on evaluatingarguments (Petty & Cacioppo, 1986) in which more involved raters processarguments more deeply. Accountable (for accuracy) raters are more likely touse complex rules in choosing among alternative possibilities (Arkes, 1991;Tetlock, 1983a) and be more critical in evaluating evidence about other persons(Tetlock, 1983b).

For example, Tetlock and Kim (1987) found that raters who learned beforea personality prediction task that they would be held accountable for their judg-ments produced more integratively complex impressions (e.g., impressions con-taining more perspectives and connections between perspectives) and made moreaccurate behavioral predictions. This was not true for raters informed of beingheld accountable after completing the task, or for unaccountable control partici-pants. In fact, holding raters accountable after ratings are rendered causes raters

120 The Journal of Psychology

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

to engage in mental effort aimed at justifying their ratings (Longnecker et al.,1987; Tetlock, Skitka, & Boettger, 1989). Tetlock’s findings have implicationsfor accountability in a performance appraisal setting because raters may becomeaware of accountability factors at two different times: prior to observation andratings, and after ratings have been completed (Feldman, 1994). The latter wouldlikely occur, for example, when a rater learns that his or her ratings will be usedfor important organizational decisions of which he or she was previouslyunaware.

Simonson and Nye (1992) found that accountability did not increase thequality of decisions for which the correct response was unclear. Instead, account-ability likely improves decision making when raters correctly anticipate whichresponses will appear rational or normative (Kunda, 1990; Simonson & Nye,1992). Thus, it seems possible that accountability might decrease contrast errorsif raters learn of being held accountable before observation and ratings and ifthey are held accountable to an accuracy criterion.

A consideration of the methods used to manipulate accountability supportsthis idea. A typical manipulation involves asking raters to publicly justify theirratings to the experimenter or to the ratee as opposed to confidential ratings. Forexample, Simonson and Nye (1992), Tetlock and Boettger (1989), and Tetlockand Kim (1987) told raters to expect to justify ratings in a post-rating interview.Tetlock et al. (1989) asked raters to explain and justify ratings to another raterwhile their explanations were audiotaped. Another approach involves tellingraters to expect a meeting with the ratee after ratings (Klimoski & Inks, 1990). Itis probably safe to assume that most raters are motivated by the desire to appearcompetent and avoid criticism (Kunda, 1990).

Contrast of Performance Ratings

An examination of findings from studies on reducing contrast effects sug-gests that accountability may achieve what other experimental manipulationshave not been able to do. In these studies (e.g., Loveland & Palmer, 2001;Maurer et al., 1993), participants viewed and rated two good or two poor lec-turers and then viewed and rated an average lecturer. The context lecturers’ per-formances consisted of all good or all poor behaviors, and the average targetlecturer’s performance consisted of both good and poor behavior instances.The average lecturer received good ratings in the poor performance context andpoor ratings in the good performance context. Maurer and Alexander (1991)and Palmer, Maurer, and Feldman (2002) demonstrated that the polarized per-formance context directs raters’ attention to context-discrepant target rateebehavior. In other words, when viewing the average lecturer performance,raters attended to good target behavior in a poor context and poor target behav-ior in a good context. Thus, differential attention drove the contrast errors inratings (Maurer & Alexander).

Palmer & Feldman 121

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

In both the Maurer et al. (1993) and Loveland and Palmer (2001) studies, theexperimental manipulations, diary use, and group discussion increased raters’thoughts about the behavior they had already noticed. Maurer et al. found thatuse of behavioral diaries to record and review behavioral information prior to rat-ings magnified contrast errors. Loveland and Palmer found that group discussionfollowing observation and prior to ratings did the same. In both studies, raterswho had attended to poor behavior (good context) thought more about the poorbehavior attended to, and raters who had attended to good behavior (poor con-text) thought more about the good behavior they had attended to. Their impres-sions, already good or poor, became more extreme following additional thought.In short, the experimental manipulations in both studies increased the intensityof raters’ thoughts while not altering the direction. This explanation of the diaryor group-induced increase in contrast is consistent with findings by Tesser (1978)as well as research on polarization effects on attitudes (Lamm & Myers, 1978;Myers & Lamm, 1976).

A strong, evaluatively polarized performance context thus directs process-ing, specifically, attention to context-discrepant behavior that would influencesubsequent recall. Would, however, an accountability contingency override theinfluence of the performance context, redirect processing, and reduce the con-trast error? The research on accountability, specifically the effect of account-ability on processing when raters are aware of accountability contingencies, sug-gests it might.

Hypothesis 1: Accountability will reduce contrast error in ratings.

To be consistent with Simonson and Nye (1992), we told raters in theaccountable condition in the present study that they would be held accountableto produce ratings that would be similar to those produced by an expert rater. Wewanted to give accountable raters a general idea of the judgment standards theywere expected to use and for which they would be held accountable. If account-ability to an accuracy standard produces changes in the direction of raters’ atten-tion and thought, then support for this would be demonstrated by a reduction ofthe contrast in behavior recalled by accountable raters.

Hypothesis 2: Accountability will reduce contrast in behavior recall.

Positive Halo in Performance Ratings

Positive halo in performance appraisals occurs when a rater fails to distin-guish between conceptually distinct but overlapping performance dimensionswhen performing ratings (Feldman, 1994; Fisicaro, 1988) and stems from therater’s use of a general impression to make ratings on multiple dimensions(Lance, LaPointe, & Fisicaro, 1994). An example would be the case in which a


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

supervisor may give an employee all or virtually all good or all poor ratings forseveral different performance dimensions even though the ratee’s performancemay vary along those dimensions.

A strong, evaluatively polarized performance context has been shown to pro-duce halo in addition to producing contrast errors in ratings (Palmer et al., 1998).This is because the context causes raters to attend to a good or poor context-dis-crepant subset of the target ratee’s behavior and thus form a strong good or poorgeneral impression of the ratee. This general impression is highly accessiblefrom memory (Srull & Wyer, 1989), reflected by the fact that raters perform rat-ings of the target performance much faster when a prior, strong evaluativelypolarized performance context exists (Palmer, Jennings, & Thomas, 2001). Thus,if accountability reduces the effect of context on rater attention and recall, thenit might reduce positive halo as well.

Hypothesis 3: Accountability will reduce positive halo in ratings.

Rating Accuracy

Researchers on performance appraisal rating accuracy have typically com-pared ratings to some accuracy criterion (e.g., Maurer, Palmer, & Lisnov, 1995).In the typical study, raters’ ratings are compared with hypothetical “true” ratings,with “true” ratings operationalized as judgments made by expert raters. Forexample, differential accuracy (Cronbach, 1955) is computed as the sum of devi-ations of dimension ratings from the corresponding dimension “true” or “expert”values, squared and summed across dimensions, producing an accuracy index foreach rater. A low value for this index indicates higher rating accuracy, as inaccu-racy is reflected in the degree to which raters’ ratings deviate from the “true” or“expert” ratings. Contrast of ratings is generally accompanied by inaccuracy ofratings for normative raters (Maurer et al., 1995). Thus, we might expect that amanipulation reducing contrast would concomitantly increase rating accuracy.

Hypothesis 4: Accountability will result in greater rating accuracy.

Need for Cognition

A strong performance context does not produce a rating contrast for allraters, and accountability likely would not improve ratings for all raters. Further-more, many raters return accurate ratings in the presence of a strong, polarizedcontext (Maurer et al., 1995). Some raters apparently attend to all relevant, ratherthan just context-discrepant, aspects of ratee behavior or use appropriate stan-dards and produce accurate ratings. It is possible that some individual differencemechanism moderates the context-rating relationship. Given the evidence for theinfluence of the amount and direction of cognitive processing on judgments, the


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

need for cognition (Cacioppo & Petty, 1982), defined as the “tendency for anindividual to engage in and enjoy thinking” (p. 116) is worth examining.

In comparison with individuals low in need for cognition, individuals highin need-for-cognition recall more information to which they have been exposed(Cacioppo, Petty, & Morris, 1983), are influenced by argument quality ratherthan by peripheral cues such as message source favorability (Petty & Cacioppo,1986), and generate more thoughts when performing tasks such as formingimpressions (Lassiter, Briggs, & Bowman, 1991). In short, on the one hand,individuals high in need for cognition are less likely to act like cognitive misersand more likely to engage in effortful processing, regardless of the situation(Cacioppo, Petty, Feinstein, & Jarvis, 1996); individuals high in need for cog-nition are also less likely to resort to heuristics or prior judgments when per-forming ratings. On the other hand, individuals low in need for cognition areexpected to engage in controlled processing and produce more accurate ratingswith less error only when directed to do so by some contingency, such asaccountability.

Hypothesis 5: The positive effects of accountability on ratings, halo, and accuracywill be moderated by individual differences in need for cognition.

Method

Participants, Design, and Materials

Undergraduate students (N = 120) from a large university received classcredit to participate in the present study. The design included two performancecontext conditions, good and poor; a context-free or “random” control condition;as well as accountable versus not accountable conditions. Each condition con-tained 20 randomly assigned participant raters.

Stimuli and Measures

We used five videotaped lecturers developed by Maurer and Alexander(1991) and Palmer et al. (2002) as context and target stimulus performances. Onepair of performances was of poor quality and a second pair was good; these werethe context performances. To maximize control, the first and second pairs ofgood and poor performances were identical except for the quality of the perfor-mance. The lecturer, his or her attire, and content of the lecture did not differ. Theother performance was neutral or average in nature, consisting of a mixture ofgood and poor behaviors; this was the target performance.

Raters performed ratings using scales from Maurer & Alexander (1991) thatconsisted of nine dimensions of lecturer behavior (e.g., confidence with thematerial) and used a 7-point Likert-type rating scale with endpoints ranging from


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

1 (very poor) to 7 (very good). We used a free recall diary-like instrument(DeNisi, Robbins, & Cafferty, 1989; Maurer et al., 1993) to measure raters’memory for ratee behavior. Raters recorded behaviors corresponding to each ofthe nine dimensions separately. We used the 18-item Need For Cognition Scale(Cacioppo et al., 1996) to measure need for cognition.

Procedure

We gave raters in the accountability condition the following instructionsprior to observation:

We are interested in gathering the most accurate ratings possible for the perfor-mances you will view. We have collected ratings from a group of “expert” raters whoare experts in evaluating lecturer behavior. We are interested in determining whetherstudents can evaluate these lectures similarly to the experts. In addition, we are alsointerested in examining raters’ decision-making strategies. Therefore, following theratings, we will interview each of you and ask you to justify your rating to the exper-imenter and the other raters. In other words, we wish to identify your rationale foryour rating of the performance and why you think your ratings are similar to thoseof the “experts.”

We told raters in the not accountable condition that the experimenter wasonly interested in collecting ratings on the performances and that all ratings wereconfidential. After each set of ratings, raters recorded, observed, and rememberedbehaviors in a free-recall diary organized by dimension and then rated eachbehavior from –5, (extremely poor lecturer behavior) to +5 (extremely good lec-turer behavior) (Maurer et al., 1993). After completing the final ratings, raterswere given the need for cognition questionnaire, debriefed, and dismissed.

Results

Reliability (alpha) for the nine target lecturer dimension ratings was .93 andranged from .72 to .92 for the context lecturers’ dimension ratings. Thus, all fur-ther analyses were computed on the rating means of the nine dimensions. Weconducted t tests to compare ratings for context lecturer performances againstcontext-free ratings of the target performance (the control condition) to verify themanipulation of context (Kravitz & Balzer, 1992). In other words, we did this toensure that the poor-context lecturers were really poor and the good-context lec-turers were really good, when compared to the average lecturer. The results of allfour t tests were significant (p < .05), indicating that the performance context dif-fered across the three conditions.

We then used planned contrasts to test the hypotheses. We chose plannedcontrasts because they test precise questions regarding the hypothesized patternor order found in a set of means (Rosenthal & Rosnow, 1988). For example, the


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

hypothesis that µ1 is greater than both µ2 and µ3 (µ1 > µ2 = µ3) could be tested withweights of 2, –1, and –1, and it could be mathematically shown that a t test com-paring two groups is equivalent to a planned contrast with weights of 1 and –1.An ANOVA, on the other hand, tests only whether any difference exists within aset of means.

Contrast weights used in the present study are given in Table 1 and reflect(a) a test of accountability; (b) linear and quadratic trends, with conditionsordered good context, control, and poor context; and (c) accountability by linearand quadratic interaction contrasts.

Rating Means

Rating means are given in Figure 1. A contrast effect would be demon-strated by a linear trend in the target ratings. A Contrast × Accountability inter-action, supporting Hypothesis 1, would be demonstrated by a Linear ×Accountability trend. The linear trend was significant, t(114) = 10.28, p < .001,r2 = .48, indicating a contrast effect in the ratings. The quadratic trend was alsosignificant, t(114) = 4.89, r2 = .17, reflecting the fact that the contrast wasstronger in the poor context condition, consistent with prior research (Maureret al., 1993). The Linear × Accountability interaction contrast was significant,t(114) = 3.82, p < .001, r2 = .11, supporting Hypothesis 1. The contrast effectwas weaker when participants were held accountable (see Figure 1).

Behavior Recall

Hypothesis 2 held that accountability would reduce the contrast in behaviorrecorded. We computed indices of positive and negative ratee behavior attendedto separately by dividing the sum of the ratings of the recorded behaviors by thenumber of behaviors recorded. Prior research indicated that poor behaviorattracts attention more than does good behavior, and thus we computed the analy-ses separately. Mean negative and positive behavior recorded are presented inFigures 2 and 3.

Support for Hypothesis 2 would be demonstrated by a significant Account-ability × Linear trend interaction. For negative behavior recorded, the linear,t(114) = 6.39, p < .001, r2 = .26, and quadratic, t(114) = 2.96, p < .01, r2 = .07,contrasts were significant. Negative target ratee behaviors were noticed aboutequally between the good context and control conditions, but very few poorbehaviors were noticed in a poor context, and this pattern accounted for the sig-nificant linear and quadratic contrasts. For positive behavior recorded, the linear,t(114) = 5.46, p < .001, r2 = .21, and Accountability × Linear, t(114) = 1.83, p <.05, r2 = .03, contrasts were significant; the contrast of good behavior remem-bered was reduced by the accountability manipulation. This result was expected,and Hypothesis 2 was partially supported.


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013


TA

BL

E 1

. Pla

nned

Con

tras

t W

eigh

ts

Acc

ount

able

Not

acc

ount

able

Poor

Goo

dPo

orG

ood

Con

tras

tco

ntex

tC

ontr

olco

ntex

tco

ntex

tC

ontr

olco

ntex

t

Acc

ount

able

vs.

not

acc

ount

able

11

1–1

–1–1

Lin

ear

tren

d–1

01

–10

1Q

uadr

atic

tren

d–1

2–1

–12

–1L

inea

r ×

Acc

ount

abili

ty–1

01

10

–1Q

uadr

atic

×A

ccou

ntab

ility

–12

–11

–21

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

Halo

Hypothesis 3 held that halo would be weaker for accountable raters. Halohas been defined as the average absolute discrepancy between the rater’s overallrating and each dimension rating (Balzer & Sulsky, 1992). Thus, the measure isan index of the degree to which ratings vary across dimensions and is appropri-ate for the present study because it allows analysis of halo at the individual level.Because halo is an index of variability, low numerical scores indicated high (pos-itive) halo, and high numerical scores indicate low halo.

Halo means are presented in Figure 4. We found a significant contrast foraccountability, t(114) = 3.00, p < .01, r2 = .07, which supported Hypothesis 3. AsFigure 4 shows, halo was weaker when raters were held accountable. Further-more, the linear, t(114) = 2.20, p < .05, r2 = .04, quadratic, t(114) = 6.16, p <.001, r2 = .25, and Linear × Accountability, t(114) = 1.88, p < .05, r2 = .03, con-trasts were also significant. The linear trend reflects the stronger halo in a poorcontext, and the quadratic trend reflects the fact that halo is weaker when nostrong performance context exists. Both findings are consistent with priorresearch (Palmer et al., 1998). The Linear × Accountability contrast reflects the


5.5

5.0

4.5

4.0

3.5

3.0

2.5

Mea

n R

atin

g

Prior Good Control Prior Poor

Context Condition

Not Accountable

Accountable

FIGURE 1. Rating means.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

fact that the effect of accountability on halo was strong in the good context con-dition, but weak in the poor context condition.

Accuracy

Differential accuracy (Cronbach, 1955) was defined as the squared sum ofthe deviations of each rating dimension from the dimension’s hypothetical (nor-mative) true score, with true scores taken from previous research (Maurer et al.,1995). In other words, the index assesses the degree to which raters’ judgmentsof rating dimensions for the target lecturer deviate from the true target lecturerperformance level. The measure is thus an index of rating deviation from a rat-ing criterion, and high numeric scores indicate low accuracy.

Accuracy means are presented in Figure 5. The accountability, t(114) = 2.33,p < .05, r2 = .05, linear, t(114) = 5.77, p < .001, r2 = .23, quadratic, t(114) = 6.23,p < .05, r2 = .25, Accountability × Linear, t(114) = 2.90), p < .01, r2 = 07, andAccountability × Quadratic, t(114) = 2.42, p < .05, r2 = 05, contrasts were all sig-nificant. Accuracy was greater when raters were held accountable, supporting


3.5

3.0

2.5

2.0

1.5

Neg

ativ

e B

ehav

ior

Rec

orde

d


Context Condition

Not Accountable

Accountable

FIGURE 2. Negative behavior recalled.

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

Hypothesis 4. A poor context produced greater inaccuracy than did a good con-text, consistent with previous research and accounting for the significant linearand quadratic contrasts. The profound inaccuracy for not accountable, poor-con-text raters, relative to accountable poor-context raters, was responsible for thesignificant interaction contrasts. Accountable raters in the poor context conditionproduced inaccurate ratings relative to the good context and control conditions,but were still far more accurate than the not accountable raters.

Need for Cognition

Coefficient alpha for the need for cognition measure was .89. Hypothesis 5held that a Need for Cognition × Accountability interaction would occur. Highneed for cognition raters should produce ratings with less contrast, less halo, andgreater accuracy regardless of whether or not they are held accountable. Weexpected the accountability manipulation to reduce the discrepancy between highand low need-for-cognition participants by causing raters low in need for cogni-tion to exert greater intensity and more appropriate direction of thought withrespect to the rating task.


3.5

3.0

2.5

2.0

1.5

Posi

tive

Beh

avio

r R

ecor

ded


Context Condition

Not Accountable

Accountable

FIGURE 3. Positive behavior recalled.

4.0

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

We used moderated regression to test this hypothesis. In Step 1, we enteredneed for cognition as a continuous variable and context and accountability wereentered as dummy variables. In Step 2, we entered the Need for Cognition ×Accountability interaction, and support for the hypothesis would be demonstratedby a significant increment R2 at Step 2. Only the increment R2 for halo was sig-nificant (∆R2 = .023, p < .05). Figures 6 and 7 present the predicted halo values,computed as recommended by Cohen (1983) for not accountable and accountableraters. Raters low in need for cognition produced ratings with higher halo acrossthe context conditions (see Figure 6). However, the difference between raters highand low in need for cognition disappeared when raters were held accountable,supporting Hypothesis 5 (see Figure 7).

Discussion

Hypothesis 1 was supported. Holding raters accountable reduced the contrasteffect of ratings. The best explanation, and consistent with Maurer and Alexander(1991), is that accountability caused raters to attend to and recall both good andpoor ratee behavior rather than just a context-discrepant behavior subset. Attention


1.1

.9

.8

.6

.5

Mea

n H

alo


Context Condition

Not Accountable

Accountable

FIGURE 4. Mean halo.

1.2

1.0

.7

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

was not measured in the present study, but recall was. Hypothesis 2 held thataccountability would reduce the contrast in behaviors recalled. This hypothesiswas supported for positive behaviors. Raters were more likely to remember goodtarget behaviors if the context was poor and poor target behaviors if the contextwas good. However, the recall contrast was reduced by the accountability manip-ulation. Thus, it appears that one method by which accountability reduced the con-trast effect was to reduce the contrast in behaviors attended to and recalled, con-sistent with Maurer and Alexander (1991) and Simonson and Nye (1992).

The accountability manipulation had no effect on negative behaviorsremembered across contexts. It may be that it is more difficult to alter the per-ceptual salience of negative information. Poor information is more diagnostic(Feldman & Lynch, 1988) and is difficult to ignore. Holding a rater accountablethus reduces the contrast in good behaviors noticed but fails to modify the effectof the poor context.

Hypothesis 3 was supported; accountability reduced halo in ratings. However,this effect was more pronounced in the good context condition. Accountability hadless effect on halo in the poor context condition. Again, poor information, whether


6

4

3

2

1

Mea

n A

ccur

acy


Context Condition

Not Accountable

Accountable

FIGURE 5. Mean accuracy.

7

5

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

contextual or observed, has a stronger effect on processing. Previous researchresults have shown that the effect of a poor context on ratings of a target perfor-mance is much stronger than that of a good context (Maurer et al., 1993), and theresults of the present study agree with this result.

Hypothesis 4 was supported. Accountability increased rating accuracy. Thiswas primarily the result of the increased accuracy in the poor context condition;accuracy changed little in the good context condition. The best explanation forthis is a floor effect; accuracy for good context ratings was already similar toaccuracy of context-free ratings and, therefore, little improvement could beexpected in this condition. Improved accuracy in the poor context conditionprobably resulted from the reduction in the contrast of ratings. Our accuracyindex was a measure of the distance between aggregated, specific dimensionalratings and their corresponding accuracy criteria. A contrast effect shifts alldimension ratings away from their criterion scores and thus, the reduction of con-trast resulted in more accurate ratings.

Hypothesis 5 was supported for halo. Raters high in need for cognition pro-duced ratings with low positive halo whether held accountable or not, and this is


1.0

.8

.6

.4

.2

Pred

icte

d H

alo


Context Condition

Not Accountable

Accountable

FIGURE 6. Predicted halo, not accountable raters.

1.2Need For Cognition

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

consistent with research on need for cognition (Petty & Cacioppo, 1986). Raterslow in need for cognition produced ratings with low halo only if held account-able. The best explanation is that accountability caused these raters to engage inthoughtful processing. Their halo levels were similar to those of raters high inneed for cognition, who engage in thoughtful processing habitually. The onlygroup not engaging in thoughtful processing, whether by habit or as a result ofaccountability, were the not accountable raters low in need for cognition, whoproduced high levels of positive halo relative to the other three groups. Theseraters, not directed to think in any particular direction, probably recalled and usedonly their general impression to make specific ratings and probably attended pri-marily to context-discrepant behavior.

Accountability reduced the magnitude of contrast, but accountability aloneis insufficient. The results from prior research and the present study indicate thatat least two qualifications must be added. First, accountability works only whenraters are explicitly told, prior to the rating task, that they will be held account-able for their ratings (Tetlock & Kim, 1987). Second, raters will think as account-ability directs them and will provide ratings consistent with the contingencies of


1.0

.8

.6

.4

.2

Pred

icte

d H

alo


Context Condition

Not Accountable

Accountable

FIGURE 7. Predicted halo, accountable raters.

1.2

1.1

.9

.7

.5

.3

Need For Cognition

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

the accountability requirements (Kunda, 1990). Contrast was reduced by anaccountability manipulation that held raters accountable for ratings approximat-ing those given by an “expert” rater. It is likely that when raters are not given astandard against which they know their ratings will be held accountable, theywill use whatever standards come to mind during observation and judgment andform justifications for their ratings later.

One shortcoming of the present study is that although raters were heldaccountable to an external standard, it probably was not identical acrossaccountable raters. Whereas not accountable raters probably compared the tar-get lecturer with whatever standards came to mind (and the most accessible ofthese would be the context lecturers), accountable raters probably varied intheir conceptions of how the “expert” rater would rate the target lecturer. It isentirely possible that the use of a common standard, such as that used in frameof reference training (Woehr, 1994), in combination with the use of account-ability, would serve to further decrease the contrast. Future research shouldaddress this possibility. Additionally, although our theory concerns attention,we measured behavior recall. We feel that the assumption that a rater whoattends to good behavior will recall good behavior is, in the present case, ten-able. Nonetheless, recall is not always driven by attention (Feldman, 1981), andthus a more direct measure of attention is needed to conclusively demonstratethat accountability influences rater attention.

REFERENCES

Arkes, H. R. (1991). Costs and benefits of judgment errors: Implications for debiasing.Psychological Bulletin, 110, 486–498.

Balzer, W. K., & Sulsky, L. M. (1992). Halo and performance appraisal research: A crit-ical examination. Journal of Applied Psychology, 77, 975–985.

Cacioppo, J. T., & Petty, R. E. (1982). The need for cognition. Journal of Personality andSocial Psychology, 42, 116–131.

Cacioppo, J. T., Petty, R. E., & Morris, K. J. (1983). Effects of need for cognition on mes-sage evaluation, recall, and persuasion. Journal of Personality and Social Psychology,45, 805–818.

Cacioppo, J. T., Petty, R. E., Feinstein, J. A., & Jarvis, W. B. (1996). Dispositional differ-ences in cognitive motivation: The life and times of individuals varying in need for cog-nition. Psychological Bulletin, 119, 197–253.

Cohen, J. (1983). Applied multiple regression/correlation analysis for the behavioral sci-ences (2nd ed.). Hillsdale, NJ: Erlbaum.

Cronbach, L. J. (1955). Processing affecting scores on “understanding of others” and“assumed similarity.” Psychological Bulletin, 52, 177–193.

DeNisi, A. S., Robbins, T. L., & Cafferty, T. P. (1989). Organization of information usedfor performance appraisals: Role of diary-keeping. Journal of Applied Psychology, 74,124–129.

Eagly, A. H., & Chaiken, S. (1993). The psychology of attitudes. New York: HarcourtBrace Jovanovich.

Feldman, J. (1981). Beyond attribution theory: Cognitive processes in performanceappraisal. Journal of Applied Psychology, 66, 127–148.


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

Feldman, J. (1994). On the synergy between theory and application: Social cognition andperformance appraisal. In R. S. Wyer & T. K. Srull (Eds.), Handbook of social cogni-tion (2nd ed., Vol. 2, pp. 339–397). Hillsdale, NJ: Erlbaum.

Feldman, J., & Lynch, J. (1988). Self-generated validity and other influences of measure-ment on belief, attitude, intention, and behavior. Journal of Applied Psychology, 73,421–435.

Fisicaro, S. A. (1988). A reexamination of the relation between halo and accuracy. Jour-nal of Applied Psychology, 73, 239–244.

Fiske, S. T., & Neuberg, S. L. (1990). A continuum of impression formation, from cate-gory-based to individuating processes: Influence of information and motivation onattention and interpretation. In M. P. Zanna (Ed.), Advances in experimental social psy-chology (Vol. 23, pp. 1–74). San Diego, CA: Academic Press.

Klimoski, R., & Inks, L. (1990). Accountability forces in performance appraisal. Organi-zational Behavior and Human Decision Processes, 45, 194–208.

Kravitz, D. A., & Balzer, W. K. (1992). Context effects in performance appraisal: Amethodological critique and empirical study. Journal of Applied Psychology, 77,24–31.

Kunda, Z. (1990). The case for motivated reasoning. Psychological Bulletin, 108,480–498.

Lamm, H., & Myers, D. G. (1978). Group-induced polarization of attitudes and behavior.In L. Berkowitz (Ed.), Advances in experimental social psychology (pp. 145–195). SanDiego, CA: Academic Press.

Lance, C. E., LaPointe, J. A., & Fisicaro, S. A. (1994). Tests of three causal models of halorater error. Organizational Behavior and Human Decision Processes, 57, 83–96.

Lassiter, G. D., Briggs, M. A., & Bowman, R. E. (1991). Need for cognition and the per-ception of ongoing behavior. Personality and Social Psychology Bulletin, 17, 156–160.

Lerner, J. S., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psy-chological Bulletin, 125, 255–275.

Longnecker, C. O., Sims, H. P., Jr., & Gioia, D. A. (1987). Behind the mask: The politicsof employee appraisal. The Academy of Management Executive, 1, 183–193.

Loveland, J., & Palmer, J. (2001, April). Group polarization and interview decisions:Contrast effects revisited. Paper presented at the 16th annual conference of the SocietyFor Industrial and Organizational Psychology, San Diego, CA.

Maurer, T., & Alexander, R. (1991). Contrast effects in behavioral measurement: Aninvestigation of alternative process explanations. Journal of Applied Psychology, 76,3–10.

Maurer, T., Palmer, J., & Ashe, D. (1993). Diaries, checklists, evaluations, and contrasteffects in measurement of behavior. Journal of Applied Psychology, 78, 226–231.

Maurer, T., Palmer, J., & Lisnov. S. (1995). Distinguishing context effects from contexterrors in judgments of behavior. Journal of Applied Social Psychology, 26, 1637–1651.

Myers, D. G., & Lamm, H. (1976). The group polarization phenomenon. PsychologicalBulletin, 83, 602–627.

Palmer, J. K., Feldman, J., & Maurer, T. (1998, April). Contrast, halo, and accuracy: Sys-tematic relationships explored. Paper presented to the 13th annual conference of theSociety for Industrial and Organizational Psychology, Dallas, TX.

Palmer, J. K., Jennings, T., & Thomas, A. (2001, June). Processing speed mediates therelationship between context and halo. Paper presented at the 13th annual conferenceof the American Psychological Society, Toronto, Canada.

Palmer, J. K., Maurer, T. J., & Feldman, J. M. (2002). Contrast and prior impressioneffects on attention, judgment standards, and ratings: Contrast effects revisited. Journalof Applied Social Psychology, 32, 2575–2597.


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

Petty, R. E., & Cacioppo, J. T. (1986). The elaboration likelihood model of persuasion. InL. Berkowitz (Ed.), Advances in experimental social psychology (Vol. 19, pp.123–205). San Diego, CA: Academic Press.

Rosenthal, R., & Rosnow, R. (1988). Contrast analysis: Focused comparisons in theanalysis of variance. New York: Cambridge University Press.

Simonson, I., & Nye, P. (1992). The effect of accountability on susceptibility to decisionerrors. Organizational Behavior and Human Decision Processes, 51, 416–446.

Srull, T. J., & Wyer, T. K. (1989). Person memory and judgment. Psychological Review,96, 58-83.

Tesser, A. (1978). Self-generated attitude change. In L. Berkowitz (Ed.), Advances inexperimental social psychology (Vol. 11, pp. 290–338). San Diego, CA: AcademicPress.

Tetlock, P. E. (1983a). Accountability and complexity of thought. Journal of Personalityand Social Pschology, 45, 74–83.

Tetlock, P. E. (1983b). Accountability and the perseverance of first impressions. SocialPsychology Quarterly, 46, 285–292.

Tetlock, P. E. (1985). Accountability: The neglected social context of judgment andchoice. In B. Staw & L. Cummings (Eds.), Research in organizational behavior (Vol.1, pp. 297–332). Greenwich, CT: JAI.

Tetlock, P. E., & Boettger, R. (1989). Accountability: A social magnifier of the dilutioneffect. Journal of Personality and Social Psychology, 57, 388–398.

Tetlock, P. E., & Kim, J. I. (1987). Accountability and judgment processes in a personal-ity prediction task. Journal of Personality and Social Psychology, 52, 700–709.

Tetlock, P. E., Skitka, L., & Boettger, R. (1989). Social and cognitive strategies for copingwith accountability: Conformity, complexity, and bolstering. Journal of Personality andSocial Psychology, 57, 623–640.

Wexley, K .N., Sanders, R. E., & Yukl, G. A. (1973). Training interviewers to eliminatecontrast effects in employment interviews. Journal of Applied Psychology, 57,233–236.

Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training onthe recall of performance information. Journal of Applied Psychology, 79, 525–534.

Original manuscript received April 10, 2003Final revision accepted July 12, 2004


Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

Powerful

keyword

search.

Easy to

use.

Find

articles

quickly.

OUR ARCHIVES

The Journal of Psychology is published by Heldref Publications,a division of the nonprofit Helen Dwight Reid Educational Foundation

now available online.

www.heldref.org

Dow

nloa

ded

by [

Uni

vers

ity o

f H

aifa

Lib

rary

] at

07:

57 2

0 M

ay 2

013

accountability and need for cognition effects on contrast, halo, and accuracy in performance ratings

Documents