probability assessment: how do we get "good" numbers for litigation decision tree...

34 Coquito Court, Menlo Park • California 94028 • Phone 650.854.1914 • www.litigationriskmanagement.com • [email protected]

LitigationRisk Management

Institute

Bruce L. Beron, Ph.D.

PROBABILITY ASSESSMENT HOW DO WE GET "GOOD" NUMBERS FOR THE ANALYSIS?

Bruce L. Beron, Ph.D.

© 2015 Bruce Beron Prob Enc 9/13 Risk ManagementLitigation

Institute

It is not easy to assess uncertainties.

People do not wish to acknowledge uncertainty In fact, most people seek certainty

-Trained to provide "the answer"-Wish to appear knowledgeable and certain-More comfortable dealing with/planning for certainty

-Even if they do consider uncertainty, there are biases in the way we think about it


Institute

Biases are systematic discrepancies between subject's knowledge and experience and statements. Two classes:

•Motivational Biases• Subject consciously distorts responses• Result from fear of punishment or desire for reward

•Cognitive Biases• Discrepancy occurs because of manner in which

subject subconsciously processes information•The use of clarity to judge distance


Institute

There are six cognitive biases to be aware of:

•Availability• Easier to think of is perceived as more likely

•Adjustment• Being too certain of our knowledge

• Almanac Questionnaire•Anchoring• Locking onto a particular number

• Population, Area of Pentagon•Recent Data• Discard previous experience too easily

• Tom W.•Implicit Conditioning• Impression of coherence• Unstated assumptions

•World View• God is on our side!• The other side doesn't have a case!

The first five of these are discussed in detail in the Kahnmenn and Tversky article.


Institute

Two kinds of uncertainties and they must be assessed differently

Discrete events• There are a finite number of possible outcomes• Try to assess as two state events

• Makes it hard to mix issues• Permits sensitivity analysis

• If many outcomes, use modified version of continuous encoding

•Continuous outcomes• There is an infinite number of possible outcomes• Answer is in the form of a number (non-integer)• Must be converted to discrete events to put in decision tree• Assess properly, then convert to discrete set of outcomes


Institute

Several steps are necessary to prepare a subject for the encoding process

Define variable/issue/question so that it is unambiguous• - It meets clairvoyant test

Make sure you know which outcome(s) is (are) good/desirable, which bad/unfavorable and which neutralUse a scale with which subject is comfortable and familiarExplore for unstated assumptions


Institute

Several steps are necessary to prepare a subject for the encoding process (cont'd)

Explore subject's knowledge of, experience with, and ways of thinking about the uncertain variableBalance any recent or specific information with previous experiencePostulate extreme outcomes and request explanatory scenarios from a retrospective viewpoint• Put your subject in the future and have him/her imagine an

extremely unfavorable* outcome• Ask the subject for an explanation of how this outcome came

to be• Have them make an exhaustive list of all the possible reasons• Repeat the process for an extremely favorable outcome

*Remember your experiences with the almanac questions!


Institute

Research shows that the best way to get "correct" assessment is to use the wheel. For a discrete uncertainty:

You could win $10,000 personally on either of these outcomes, after the event happens. But you must choose now which has the better odds.

Wheel Event

The jury finds liability.

OR

If the damages are greater than $X millions, you win.

If the pointer stopsin the orange, you win.


Institute

Here is the procedure for using the wheel to encode the probability of a discrete event:

Start with almost all orange, so that no matter how likely the subject thinks the event is, orange is a better choice.• Ask the subject to choose (he/she will choose the wheel).

Then show almost no orange, so the event will be more likely than the orange.• Ask the subject to choose (he/she will choose the event).

Change the amount of orange and ask the subject which he/she prefers.• If they choose the event, increase the orange.• If they choose the wheel, reduce the orange.

When the subject is indifferent between the wheel and the event, read the probability on the back of the wheel.• This is the encoded/assessed probability for the event to put into

the decision tree.


Institute

Before encoding a continuous distribution, reassure the subject that:

Not testing them or trying to trick themShould not worry about being consistent with their previous answers• Consider each question by itself

You want their "gut" feeling on questions, not "intellectual / head"response


Institute

For continuous uncertainty, ask a series of questions regarding discrete assessments

You could win $10,000 personally on either of these outcomes, after the event happens. But you must choose now which has the better odds.

Wheel Event

The jury finds damages greater than $X millions.

OR

If the damages are greater than $X millions, you win.

If the pointer stopsin the orange, you win.


Institute

Procedure for using the wheel to encode the probability of a continuous uncertainty:

Set the 1% and 99% end points for the distribution• Set the wheel so that only 1% shows orange• Ask the subject to review the list of reasons why the variable may

have an extremely low (unfavorable) outcome• Now ask the subject to choose a value so low, that there is only one

chance in a hundred (1%) that it will have a value that low or lower• Pick an extremely low value• Which do they think is more likely, the wheel or the outcome being less than the value picked?

•If they pick the wheel, increase the value•If they pick the outcome being less, lower the value•When they are indifferent, you have established the 1%tile of the distribution

• Repeat the above process for the value of the 99% end point• There is a 99% chance that the outcome will have this value or less• Equivalently, there is only a 1% chance that the outcome will exceed this value


Institute

Probability Encoding Form

Variable Definition: Encoded by:Date:

Expert:

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Value

Less-ThanProbability

1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0

ExcessProbability


Institute

Using the question as posed on the previous page, do the following:

Choose a number close to the 1% end pointSet the wheel to almost all orangeAsk the subject if he/she would rather bet on the wheel or the event (the outcome being less than the chosen value). They will, of course, choose the wheel.

• It is almost all orange, and it is very unlikely that the outcome will be less than the chosen value.

Now set the wheel so that almost no orange is showing (~1%).Again ask the subject whether they would prefer the wheel or the event (the outcome being less than the chosen value). They should choose the event.

• If they do not, they are either confused, or there has been a change in their 1% endpoint. This is okay. Note this new value and repeat the procedures on this page with a slightly higher value.

Adjust the wheel and ask the question until the subject is indifferent between the wheel and the event.Record the probability at the chosen value and label it with a 1. (see page 15)Plot the results out of the subjects viewDo not lead the subject!


Institute

Continue asking questions using these guidelines until "reasonable" curve develops

Repeat the procedure from the previous pageMove back and forth from the lower end of the distribution to the upper end, roughly working your way in from the ends• Avoid the 50% point and anchoring as long as possible

• Remember the population of Turkey!

Randomly alternate between "less than X" and "greater than X"• When recording results of "greater than" question, subtract the

result from 1 before plotting


Institute

Draw a smooth curve through the points*

You can ignore points that are clearly off the curve because the subject may have misunderstood a question or changed their thinking in the course of the questioning.


Institute

Distribution is a less-than probability distribution. Non-intuitive, initially hard to interpret. Extremely useful analytically.

There is a 65% (.65) chance that the variable being encoded will have a value less than 580.

580

.

Value

Cum

ulat

ive

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Excess Probability


Institute

Confirm results of encoding session with own judgment and that of subject/expert

Review any inconsistenciesAsk questions like:• There is a 10% chance that the outcome will be less than x

(read from graph)• There is a 50% chance that outcome will be between y and z

Use your own judgment about reasonableness of the distributionIs subject willing to bet his/her own money based on assessment?


Institute

Histogram view of distribution is often easier and more intuitive to review

.

Value

Cum

ulat

ive

Prob

abili

ty

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Excess Probability

0 100 200 300 400 500 600 700 800 900 1000

The histogram height is!the difference between!the values of the less-than!curve at the sides of the bin!

.

Less

-Tha

n Pr

obab

ility


Institute

Most distributions look like the S-shaped curves in the examples

•Beware of:• Straight line distributions!

• Implies that variable equally likely anywhere between end points• The world almost never works that way

• Bi-modal distributions• Usually implies an undisclosed underlying/conditioning event

If hard limit (damages can't go below zero), curve may look something like:

Value

1.00

0.75

0.50

0.25

0.00



Institute

In order to convert our continuous distribution to a discrete distribution:Draw two horizontal lines at "appropriate" less-than probabilities - divides distribution into three regions — High, Medium, and Low.

Draw vertical line in each region at point where area below and to left of the curve is equal to the area above and to right.

.

Value

Cum

ulat

ive P

roba

bilit

y

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100% 0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Excess Probability

0 100 200 300 400 500 600 700 800 900 1000

LowRange

MediumRange

HighRange

Equal Areas

Equal Areas

Equal Areas

501!

.

Less

-Tha

n Pr

obab

ility

.!

.25!

.50!

.25!

High=740!

Medium=500!

Low=265!


Institute

Not always time to do full encoding. A quick way to encode distribution — the values A, B, and C are very close to the 10th, 50th, and 90th percentiles

A quick assessment would ask for values for (in order):

• 1 %tile• 99 %tile• 10 %tile• 90 %tile• 25%tile• 75%tile• 50 %tile

Plot the points and draw the curve to verify that the distribution is "reasonable."


1.00

0.75

0.50

0.25

0.00A B C

0.90

0.10

.25

.50

.25

C=90%tile

B=50%tile

A=10%tile

Value


Institute

Probability assessment allows us to measure expert differences and to resolve them.

Separately assessed distributions indicate possible differences of opinionImportance of disagreement can be tested through sensitivity analysisIf differences critical, sharing underlying information may result in consensusUnresolvable differences referred to decision maker

probability assessment: how do we get "good" numbers for litigation decision tree...

Law