overconfidence in judgment: why experience might not be a good teacher tom stewart september 24,...
TRANSCRIPT
Overconfidence in judgment: Why experience might not be a good teacher
Tom Stewart
September 24, 2007
2
Einhorn, H. J., & Hogarth, R. M. (1978). Confidence in judgment: Persistence of the illusion of validity.
Psychological Review, 85(5), 395-416.
“How can the contradiction between the considerable evidence on the fallibility of human judgment be reconciled with the seemingly unshakable confidence people exhibit in their judgmental ability? In other words, why does the illusion of validity persist?” (p. 396)
12
0
50
100
0 50 100
Judgment
"Tru
th"
r = .50
Criterion
threshold
Action is appropriate
Action is inappropriate
13
Correct rejections
Hits
r = .50Decision threshold
Criterion
threshold
False alarms
Misses
0
50
100
0 50 100
Judgment
"Tru
th"
14
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
r = .50Decision threshold
Criterion
threshold
False alarms
Misses
15
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
r = .50Decision threshold
Criterion
threshold
False alarms
Misses
16
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
r = .95Decision threshold
Criterion
threshold
Misses
False alarms
17
Research on judging contingencies between x and y based on information in 2x2 tables suggests that people
focus on frequency of Hits.
This may be due to the difficulty people have in using disconfirming information.
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
Misses
False alarms
18
How do people learn to make decisions if feedback (knowledge of results) is
incomplete?• Selective feedback example – selection task
– If an employer chooses not hire an applicant, she will not learn how that applicant would have performed.
• Selective feedback example – detection task– If a customs officer chooses not to conduct a
search of an airline passenger entering the country, he will not learn whether the passenger is smuggling goods into the country.
19
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
Misses
False alarms
Knowledge of results: Full feedback
20
Correct rejections
Hits
0
50
100
0 50 100
Judgment
"Tru
th"
Misses
False alarms
Knowledge of results: Selective feedback
21
Typical results
Miss
Correct rejection
Hit
False Alarm
Judgment
"Tru
th"
Miss
Correct rejection
Hit
False Alarm
Judgment
"Tru
th" Correct decision Reject Select
Select 23 27 50Reject 44 6 50
67 33 1000.710 = proportion correct decisions
Decision
Cases 100Base rate 0.500
Correlation 0.700
Correct decision Reject SelectSelect 13 37 50Reject 37 13 50
50 50 1000.740 = proportion correct decisions
Decision
Full feedback
Selective feedback
22
Possible explanation
• Encoding of cases when no feedback is available. Two possibilities (not exhaustive):– Positivist – People assume that when feedback is
missing accuracy is the same as when feedback is present.
– Constructivist (optimistic) – People assume perfect accuracy when feedback is missing.
Elwin, E., Juslin, P., Olsson, H., & Enkvist, T. (2007). Constructivist Coding: Learning From Selective Feedback. Psychological Science, 18(2), 105-110.
23
Correct decision Reject SelectSelect 23 27 50Reject 44 6 50
67 33 1000.710 = proportion correct decisions
0.500 = base rate
Correct decision Reject Select
Select 12 27 39
Reject 55 6 61
67 33 1000.818 = subjective proportion correct decisions
0.392 = subjective base rate
DecisionPositivist encoding
Correct decision Reject Select
Select 0 27 27
Reject 67 6 73
67 33 1000.940 = subjective proportion correct decisions
0.270 = subjective base rate
DecisionConstructivist encoding
Selective feedback – possible types of encoding
Objective results
Subjective results – Constructivist (optimistic)
encoding
Cases 100Base rate 0.500
Correlation 0.700
Subjective results – Positivist encoding
= subjective encoding
24
Encoding and values affect the cutoff Subjective encoding
If people assume they are correct when they don’t get feedback, the cutoff will move up (fewer cases selected).
Values of the four outcomes– There is evidence that people value hits more than
other outcomes.– This could result in selecting more cases.– However, if people pay attention to the positive hit
rate, they might select fewer cases.
Einhorn and Hogarth, 1978
25
Hits as a function of selection rate
0
0.2
0.4
0.6
0.8
1
00.10.20.30.40.50.60.70.80.91
Selection rate
Hit rate Proportion correct decisions Proportion of hitsP
rop
ort
ion
(Hit rate is number of hits divided by number of positive decisions.)
Hit rate
Proportion correct
Proportion of all decisions that are hits
Base rate 0.500Correlation 0.700
Note that hit rate can be high even if accuracy is not.
Correct decision Reject SelectSelect 41 9Reject 49 1
Decision
Full feedback results
Correct decision Reject SelectSelect 0 50Reject 0 50
Decision
Full feedback results
26
0
0.2
0.4
0.6
0.8
1
1.2
1.4
Cutoff
E.V
.
Plot of expected value vs. decision cutoff
Base rate 0.500Correlation 0.700
Full feedback, objective expected value
Selective feedback, constructivist encoding, subjective expected
value
Selective feedback, positivist encoding, subjective expected
value
Correct decision Reject Select
Select -1 2Reject 1 -1
Payoff matrix assumes greater value for hits
27
Summary: Selective feedback increases confidence while reducing performance
• Research suggests that, with limited feedback, people will learn to select fewer cases.
– This results in a decision bias that increases the error rate.• Other research suggests that people pay more attention to hits
than to other outcomes.– This could result in either more cases being selected in order to
increase the number of hits, or fewer cases to increase the hit rate.• The constructivist encoding hypothesis can account for the
experimental results.*• Furthermore, with constructivist encoding subjective
performance will be better than objective performance, accounting for overconfidence.
• It appears that while selective feedback results in more decision errors, it may not affect the accuracy of judgment.
*Of course, this does not prove that people are actually doing constructivist encoding, and there are certainly individual differences.
29
Confidence
• People pay attention to positive hit rate.– Inability to use disconfirming information– Limited feedback when action not taken
• Positive hit rate is often high, even when accuracy is not.– Positive hit rate can always be increased by
reducing selection rate/increasing threshold.– Treatment effects increase positive hit rate, and
this increase is greater for high selection rates.
30
If people judge their skill by the true positive rate, what affects that rate?
• Base rate
• Correlation
• Selection rate
• Treatment effects
Illustrate with spreadsheet C:/Documents and Settings/Tom/My Documents/aaDocuments/AAPRJCTS/2005/NSF-TR-SDT-Feedback/6-Talks/T-R-
634Assignment-Einhorn-Hogarth-treatment.xls
31
Treatment effectG
old
Sta
ndard
hd
False negative
True negative
True positive
False positive
Judgment
r = .50