“An Experimental Study of Self-Relevance in Information

Processing”

Seda Ertaç

University of ChicagoJune 30, 2007

Questions:1. Do individuals process information as they

should? How Bayesian are they? 2. Does this depend on the relevance of the

information to the self?

• Self-serving use of information?

Study Bayesian updating with different types of information

The Experiment

MAIN IDEA: Compare two theoretically equivalent updating problems (within-person)

Processing of information when information is:1. Self-relevant (relative performance feedback)

a) Addition task (11+25+34+40+91=?)b) Verbal task (GRE verbal)

2. Irrelevant to the self (a statistical urn problem)

DESIGN:

Performance Rounds

Task Performance Stage (Piece-rate compensation)

Submit Initial Estimates of Relative Performance (top, middle, or bottom of the distribution)

Receive performance feedback (“top” vs. “not top”, some new sessions with bottom/not bottom)

Submit revised estimates of performance

Accurate beliefs compensated using a quadratic scoring rule.

Belief Elicitation in the Performance Rounds:

Assign probabilities to each of the following three states:

MIDDLE 60%

BOTTOM 20%

TOP 20%

Non-Performance Rounds

Computer randomly picks one of three states (top, middle, bottom) according to a probability distribution->Comes from each subject’s own submitted priors in the task rounds

Subject sees the prior probabilities of each state being picked

Assigns a probability to each of the three states

Feedback is received (top/not top)

Revised probabilities about the three states are submitted.

Beliefs compensated using a quadratic scoring rule.

Procedures:UCLA undergraduates200 participants. 62% female.29% econ/business, 36% natural

sciences+engineering, 35% other social sciences

10 participants in each group. Depending on session: 16 or 24 rounds played1,5 hours

Initial Assessments in the Performance Rounds

1. More positive self-assessments in the addition task than the verbal (individual and group)

2. No difference in “confidence in assessment” across tasks

3. Women have less positive self-assessments than men, especially in the verbal task.

Performance Assessment

Index

Overconfidence

Index

Addition 1.05 (0.44) 0.05 (0.58)

Verbal 0.86 (0.45) -0.14 (0.556)

Accuracy of Judgment

0

20

40

60

80

100

120

140

160

Underconfident Correct Overconfident

Addition

Verbal

A c c u r a c y o f J u d g m e n t , A d d it io n

0

0 . 1

0 . 2

0 . 3

0 . 4

0 . 5

0 . 6

U n d e rc o n fid e n t C o r re c t O ve r c o n fid e n t

Per

cent

age

M a le s

F e m a le s

A c c u r a c y o f J u d g m e n t , V e r b a l

0

0 . 1

0 . 2

0 . 3

0 . 4

0 . 5

0 . 6

U n d e rc o n fid e n t C o r re c t O ve r c o n fid e n t

Per

cent

age

M a le s

F e m a le s

Response to Information in the Performance Rounds

When the information “not top” is received:

Actual-Bayesian Posteriors for the state “MIDDLE” (Bias)Addition Average: -0.026 (0.14)

Average Absolute: 0.10Verbal Average: -0.04 (0.14)

Average Absolute: 0.10

Response to Information in Performance Rounds

The probability assigned to “middle” is lower than it should be (z=-3.88, t=-3.75)

The bias is very significantly different from zero. (p=0.0000)

Relation between initial “self-confidence” and updating:

BIAS IN THE POSTERIOR FOR MIDDLE:When learn “not top”

When learn “not bottom”

Confident -0.005 -0.05

Not Confident

-0.045 0.004

Bias In the Non-Performance Rounds:

Non-performance rounds with objective priors

Bias AbsoluteBias

All “interesting” casesN=376

-0.004 0.075

Same initial priorsN=139

-0.001 0.045

Comparison of Performance and Non-Performance Rounds

Absolute bias is greater in the performance rounds (bias=0.10 vs. bias=0.7)

t=3.46, p=0.0005Restricting attention to “risk-neutral” cases,

we get (bias=0.10 vs. bias=0.045): t=6.15, p=0.000

Conservative View of BU:In the non-performance task, errors occur in either direction at

the same frequency.Performance rounds:

77.5% revised state correct19% revised state reflects overuse

3.5% revised state reflects underuseNon-Performance Rounds

85.5 % revised state correct 7% revised state reflects overuse

7.5 % revised state reflects underuse

Comparison of Performance and Non-Performance Rounds (within person)

• Look at cases where priors are exactly the same and feedback is also the same, also exclude the “trivial” cases of updating.

• Average difference=Pr(mid)NP-Pr(mid)P=0.044

• Both Wilcoxon and t-tests confirm that the probability attributed to “middle” is higher in the non-performance rounds (z=3.36, t=3.81)

• Are subjects better Bayesian updaters with self-relevant information?

The absolute bias is significantly higher in the performance rounds. (t=6.98, p=0.000)

Pr(mid)NP-Pr(mid)P

Absolute bias difference

0.045 0.052

N=109

Task 3—Belief Updating under AmbiguityOnly one of the objective probabilities is revealed.

MIDDLE X %

BOTTOM ?%

TOP ?%

Frequency of Types of Errors

Treatments

Type of Bias in Pr(mid)

Task Non-Performance

Ambiguity

(+) 35% 42% 41%

0 5% 12% 10%

(-) 60% 46% 49%

Does the QSR work? 1440 instances where we know true and submitted priors. 685 of them have the exact same priors for the 3 states.

In 30% of the deviations, the highest submitted probability is higher than the highest true probability, and the lowest submitted probability is lower.

In 9% of the deviations, the reverse is true. Risk-aversion does not seem to hold. (Data on ind.risk preferences->to be analyzed)Experience seems to help in reducing deviations

Summary:Not much support for use of information in a

self-serving way. If anything, individuals pay undue attention to

performance information (may be because they are not confident enough in their priors: ambiguity?).

Confidence and direction of information seems to also matter.

Individuals seem to be better Bayesian updaters when processing information irrelevant to themselves.

Posterior Probability DifferencePerformance vs. Non-Performance Rounds

(ProbNP-ProbP)

Gender and Use of Information • There is no significant difference between the

genders in terms of information processing in the statistical problem.

• No significant difference when they get the information top/not top (still, women more likely to go to “bottom”).

• Significant difference when the information is “not bottom” (men are much more likely to go to top, bias 0.05 versus ~0).

Gender and Use of Information(continued)

Does this come from confidence?

Among the subgroup of self-confident people, there is no significant difference.

A Conservative Measure of BU:

Revised Choice

Bayesian Choice

Middle Middle~Bottom

Bottom

Middle 162 (80%)

33(16%)

8(4%)

Middle~Bottom

10(43.5%)

3(13%)

10(43.5%)

Bottom 1(9%)

2(18%)

8(73%)

77% of the time people choose the correct state.

Initial Assessments in the Performance Rounds

• Overconfidence at the group level?• Overconfidence at the individual level?

States perceived as most likely by the subjects:

Choices Top Top~Middle Middle Middle~Bottom Bottom

Addition 19% 5% 61% 9% 6%

Verbal 10% 6 % 50% 17% 17%

Bias (restricted to RN)

Bias Absolute Bias (restricted to RN)

Absolute Bias

Performance N/A -0.035 N/A 0.10

Non-performance

~0 ~0 0.045 0.07

Ambiguity -0.015 -0.015 0.09 0.09

SessionsFirst Set of Sessions: 1-7, order: Performance+Non-PerformanceFeedback Type: Top/Not TopMore Sessions to Control For Some Issues:Sessions 9-11Perf+Non-Perf+AmbiguityFeedback Type: Bottom/Not BottomSessions 12-15Statistical+Task+Ambiguity Feedback Type: Top/Not TopSession 16: Accurate beliefs not compensated

Quadratic Scoring Rule:If really in the top:Payoff=50+100 pT-50(pT

2+pM2+pB

2)

If really in the middle:Payoff=50+100 pM-50(pT

2+pM2+pB

2)

If really in the bottom:Payoff=50+100 pB-50(pT

2+pM2+pB

2)

----------------------------------------------------- Payoffs are min. if assign 1 to a wrong statement, max. if

assign 1 to a true statement Submitting true beliefs is optimal for a risk-neutral

expected utility maximizer.

TOTAL EARNINGS=EARNINGS FROM PERFORMANCE (piece rate per question solved) +EARNINGS FROM PRE-INFO BELIEFS+

EARNINGS FROM POST-INFO BELIEFS

The Importance of Good Relative Performance in the Verbal Task, Females

The Importance of Good Relative Performance in the Addition Task, Females

Survey Responses and Gender 65% of men, 50% of women think addition is

more reflective of overall ability. 23% of men, 31% of women think addition is

easier. 73% of men, 63% of women say they enjoyed

the addition task more. Men care about the addition task more than

women do. Women and men are not different in their level

of caring about performance in the verbal task.

The Importance of Good Relative Performance in the Verbal Task, Males

The Importance of Good Relative Performance in the Addition Task

Hypotheses:1. Bayesian Updating:

Pr(T|“Top”)=1 Pr(T|“Not Top”)=0 Pr(M|“Not Top”)=PrM/(PrM+PrB) Pr(B|“Not Top”)=PrB/(PrM+PrB)Likewise for bottom/not bottom. 2. Same posteriors in the performance and non-

performance rounds, if the priors and the received information are the same.

Some Notes on Survey Responses

23% of men and 31% of women find the addition task to be more difficult.

65% of men think that addition is more indicative of general intelligence, whereas 50% of women do so.

The Importance of Good Relative Performance in the Verbal Task

The Importance of Good Relative Performance in the Addition Task, Males

Summary of Survey Answers

Men do better in the addition task (z=2.71)No significant difference in the verbal task. Men (women) get negative feedback 74%

(83%) of the time in the addition task, and 80% (79.5%) of the time in the verbal task. (in 74% of the instances where they are faced with feedback).

Things to be Done:Using the Existing Data:Individual-level analysis:• Beliefs• How they use information (do we have

consistently good “Bayesians”?)• Stated importance of relative performance

Design Issues• Order Effects (non-task first?)• Type of feedback (positive vs. negative)• Objective vs. subjective priors (ambiguity?)• Potential issues with the quadratic scoring

rule

=>Additional Sessions

End-of-Experiment Survey• Which task do you think is most reflective of a

person’s overall ability?• Rank tasks in terms of difficulty for you.• Rank tasks in terms of enjoyability for you.• How important was it to you to be better than

others in task X (0 to 10 scale)?• Did the positive(negative) information you

received affect your morale? (-10 to 10 scale)• Did the positive (negative) information you

received affect your subsequent performance? (-10 to 10 scale)

• Gender, major

Taking only the subsample of subjects that say they understood perfectly does not change the results about information processing.

Task Performance

Average # of questions solved:Men Women

Addition 4.95 4.30

Verbal 5.14 4.87

A Conservative Measure of BU, Non-Task:Revised Choice

Bayesian Choice

Middle Middle~Bottom

Bottom

Middle 227 (80%)

24(16%)

1(4%)

Middle~Bottom

7(43.5%)

27(13%)

0(43.5%)

Bottom 3(9%)

9(18%)

31(73%)

87% of the time people choose the correct state. (contrast with 77 in the task case)

Order Effects? Performance in the non-performance updating

task seems to be independent of order, no significant difference (p=0.40)

Performance in the performance updating task is also not significantly different. (p=0.75)

No clear trend in the data as rounds progress

Session with Accuracy Not Compensated

No significant difference in initial confidence levels (people taking seriously)

The incidence of submitting the objective priors is significantly higher.

Belief Updating When Accuracy is not Compensated8% bias with addition task, 13% with verbal task. (Same

direction, higher than in the case of incentives) Highly significant (p=0.004 for verbal, p=0.03 for

addition)On average, 0 bias in non-performance task. Not

significant. Likewise in the ambiguity. This suggests that risk-loving cannot be the whole

reason why people put too much probability to the bottom

Survey ResultsOverwhelming majority indicated high clarity

of instructions.1 person said he/she did not understand at all. 57% thinks addition more important for

ability.74% find the verbal task more difficult

Value of Information Information increases “confidence” in

estimation.

Information is always significantly valuable for payoffs.

No significant difference between task performance and statistical, but value higher for ambiguity rounds.

Related Work:Theoretical: Belief-dependent utility and self-serving use of info: e.g.

Koszegi (2005), Benabou and Tirole (2002), Eliaz (2002).

Experimental:Experiments on Bayesian updating: e.g. Dave and Wolfe

(2005), Jones and Sugden (2001), Ertac (2005)Most similar experimental design:Clark and Friesen (2003)Real effort task, predict future performance, compensated

by a quadratic scoring rule.