an info-gap approach to modelling risk and uncertainty in bio-surveillance having imperfect...

An Info-gap Approach to Modelling Risk and Uncertainty in Bio-surveillance having Imperfect Detection rates

Prof. David R. Fox

Acknowledgements:

• Prof. Yakov Ben-Haim (Technion, Israel)

• Prof. Colin Thompson (University of Melbourne)

Risk versus Uncertainty

1. risk = hazard x exposure or

risk = likelihood x consequence

2. Duckworth (1998):• is a qualitative term • cannot be measured • is not synonymous with probability • “to ‘take a risk’ is to allow or cause exposure to the

danger”

3. is the chance, within a specified time frame, of an adverse event with specific (negative) consequences

Risk

Insignificant Minor Moderate Major Catastrophic

Almost

Certain H H E E E

Likely M H H E E

Possible L M H E E

Unlikely L L M H E

Rare L L M H H

The AS4360:1999 Risk Matrix

CONSEQUENCE

LIK

EL

IHO

OD

Risk versus Uncertainty

• Development and adoption of a ‘standard’ risk metric seems a long way off (never?);

• Bayesian methods are becoming increasingly popular, although acceptance may be hampered by biases and lack of understanding;

• More attention needs to be given to appropriate statistical modelling. In particular:

- model choice- Parameter estimation- Distributional assumptions- ‘Outlier’ detection and treatment- robust alternatives (GLMs, GAMs, smoothers etc).

Risk

Uncertainty

• Severe uncertainty → almost no knowledge about likelihood

• Arises from:

- Ignorance

- Incomplete understanding

- Changing conditions

- Surprises

• Is ignorance probabilistic?

Ignorance is not probabilistic – it is an info-gap

Shackle-Popper Indeterminism

Intelligence

• What people know, influences how they behave

Discovery

• What will be discovered tomorrow cannot be known today

Indeterminism

• Tomorrow’s behaviour cannot be modelled completely today

Knightian Uncertainty

Frank Knight • Nov 7 1885 – Apr 15 1972• Economist• Author (Risk, Uncertainty and Profit)

Knightian Uncertainty

• Differentiates between risk and uncertainty

→ unknown outcomes and known probability distributions

• Different to situations where pdf of a random outcome is known

Dealing with Uncertainties

Strategies

• Worst-case

• Max-Min (utility)

• Min-Max (loss)

• Maximize expected utility

• Pareto optimization

• “Expert” opinion

• Bayesian approaches

• Info-Gap

Info-Gap Theory (Ben-Haim 2006)

• Is a quantitative, non-probabilistic approach to modelling true Knightian uncertainty;

• Seeks to optimize robustness / immunity to failure or opportunity of windfall;

• Contrasts with classical decision theory which typically seeks to maximize expected utility;

An info-gap is the difference between what is known and what needs to be known in order to make a reliable and responsible decision.

Components of an Info-Gap Model

1. Uncertainty Model

• Consists of nominal values of unknowns and an horizon of uncertainty

2. Performance requirement

• Inequalities expressed in terms of unknowns

2. Robustness Criterion

• Is the largest for which the performance requirements in (2) are met realisations of unknowns in the uncertainty model (1)

• ‘Unknowns’ can be probabilities of adverse outcome

0

Robustness and Opportuneness

Uncertainty

Pernicious

Propitious

Robustness and Opportuneness

Robustness (immunity to failure)

is the greatest horizon of uncertainty at which failure cannot occur

Opportuneness (immunity to windfall gain )

is the least level of uncertainty which guarantees sweeping success

Note: robustness/opportuneness requires optimisation but not of the performance criterion.

Robust satisficing vs direct optimization

Alternatives to optimization:

• Pareto improvement – an alternative ‘solution’ which leaves one individual better off without making anyone else worse off.

• Pareto optimal – when no further Pareto improvements can be made

• Principle of good enough – where quick and simple preferred to elaborate

• Satisficing (Herbert Simon, 1955) – to achieve some minimum level of performance without necessarily optimizing it.

Robust satisficing

Decision is preferred over if robustness of is > robustness of

at the same level of reward; i.e

> if , ,

where is reward required.

c c

c

q q q q

q q q r q r

r

c

Thus, if is the set of all feasible decision vectors ,

a robust-satisficing decision is one which maximizes

robustness on and satisfices performance at .

i.e

= arg max

c

cq

q

r

q r

c

,

Note: usually (although not necessarily) dependes on

c

c c

q r

q r r

Robust satisficing

Fractional Error Models

• Best estimate of uncertain function U(x) is U(x)

-Although fractional error of this estimate is unknown

• The unbounded family of nested sets of functions is a fractional-error info-gap model:

~

, : ; 0U u u x u x u x u x

IG Models : Basic Axioms

All IG models obey 2 basic axioms:

1. Nesting

2. Contraction

, is nested if <

, ,

U u

U u U u

0, is a singleton set containing

its center point 0,

U u

U u u

i.e when horizon of uncertainty is zero, the estimate is correctu

An IG application to bio-surveillance

• Thompson (unpublished) examined the general sampling problem associated with inspecting a random sample of n items (containers, flights, people, etc.) from a finite population of N using an info-gap approach.

• The info-gap formulation of the problem permitted the identification of a sample size n such that probability of adverse outcome did not exceed a nominal threshold, when severe uncertainty about this probability existed.

• Implicit in this formulation was the assumption that the detection probability (ie. the probability of detecting a weapon, adverse event, anomalous behaviour etc.) once having observed or inspected the relevant item / event / behaviour was unity.

Surveillance with Imperfect Detection

I – the event that an object is inspected;

W – the event that an object is a security threat (eg. the object is a weapon, the person is a terrorist, the behaviour is indicative of malicious intent);

D – the event that the security breach is identified / detected.

Furthermore, we assume that only inspected objects are classified as either belonging

to D orD . We thus have I D D and hence

P I P D P D


Arguably, the more important probability is

P W D

and not P W

Define: detection efficiency = P DW P W

nP I

N


Can show (see paper), that:

1 1

, ,1 1

P W D p

For 100% inspections:

11, ,

1P W D p

Furthermore:

, , 1, , 0< 1p p


Performance criterion:

, ,

1, ,

p

p

1 1 1

, ,1 1 1

i.e.


Fractional error model:

, : max 0, 1 min 1, 1

, , , 0 max 0, 1 min 1, 1

U

Robustness function:

, , ,

, max : min , ,d dU


Example

• Dept. of Homeland Security intelligence → attack on aircraft imminent

• Nature / mode of attack unknown

• All estimates (detection prob., prob. of attack etc.) subject to extreme uncertainty.

Let , with 0.7 and 0.05

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

lambda=0.4lambda=0.5lambda=0.6lambda=0.7lambda=0.8lambda=0.85lambda=0.9lambda=0.975min performance requirement

lambda=0.4lambda=0.5lambda=0.6lambda=0.7lambda=0.8lambda=0.85lambda=0.9lambda=0.975min performance requirement

Robustness

Per

form

ance


0.4

0.5

0.6

0.7

0.8 0.85

0.9

0.975


Comparison with a Bayesian Approach

Assume

~ (0.98,97.0225) and ~ (14,6) [0.5,1.0]beta beta

0 0.2 0.4 0.6 0.80

1

2

3

4

phi

pdf

0 0.05 0.1 0.150

20

40

60

80

100

theta

pdf

psi

Probability

1.000.950.900.850.800.750.700.650.60

1.0

0.8

0.6

0.4

0.2

0.0

0.8

0.783

0.827

0.86

0.88

0.91


an info-gap approach to modelling risk and uncertainty in bio-surveillance having imperfect...

Documents

uncertainty slide

uncertainty severe uncertainty

uncertainty model

modelling risk

fox slide

profit knightian uncertainty

true knightian uncertainty

uncertainty unknown