taking uncertainty into account: bias issues arising from uncertainty in risk models

Taking Uncertainty Into Taking Uncertainty Into Account:Account:

Bias Issues Arising from Uncertainty in Risk Bias Issues Arising from Uncertainty in Risk ModelsModels

John A. Major, ASAJohn A. Major, ASAGuy Carpenter & Company, Inc.Guy Carpenter & Company, Inc.

N=20 observationsN=20 observations T = sample mean; T = sample mean; =1 true mean=1 true mean MLE EP curve:MLE EP curve:

q-exceedance point (PML, VaR)q-exceedance point (PML, VaR)

XX.01.01 = 4.605 actual = 4.605 actual

Example: Exponential Example: Exponential DistributionDistribution

TxxQ exp)(ˆ

)ln(ˆ qTX q

Sampling Distribution of TSampling Distribution of T

Estimated PDFsEstimated PDFs

Client QuestionsClient Questions

What is the 1 in 100-yr PML (1% VaR)?What is the 1 in 100-yr PML (1% VaR)? What is probability of exceeding 4.605?What is probability of exceeding 4.605? Can you give me an EP curve to answer Can you give me an EP curve to answer

these and similar questions?these and similar questions? Does sampling error affect the answer?Does sampling error affect the answer? Can I get unbiased answers?Can I get unbiased answers?

3 Kinds of Bias3 Kinds of Bias

““dollar” or X-bias:dollar” or X-bias: the average of PML dollar estimatesthe average of PML dollar estimates

““probabilistic” or P-bias:probabilistic” or P-bias: the average the average truetrue exceedance probability of exceedance probability of estimatedestimated PML points PML points

““exceedance” or Q-bias:exceedance” or Q-bias: the average estimated exceedance the average estimated exceedance

probabilityprobability

qq vsXE Xˆ

qvsXQE qˆ

qvsQE qXˆ

Exponential MLE is X-Exponential MLE is X-unbiasedunbiased

TE

qq qqTEXE X)ln()ln(ˆ

Exponential MLE is X-Exponential MLE is X-unbiasedunbiased

for small qfor small q Expected actual risk is greater than Expected actual risk is greater than

nominalnominal Uncertainty increases risk!Uncertainty increases risk!

Exponential MLE is P-biasedExponential MLE is P-biased

qXQE q ˆ

Exponential MLE is P-biasedExponential MLE is P-biased

Correcting for P-biasCorrecting for P-bias

Predictive distributionPredictive distribution ““Prediction interval” in regressionPrediction interval” in regression

Mix randomness and uncertaintyMix randomness and uncertainty integrate model pdf over parameter integrate model pdf over parameter

distributiondistribution Exponential model:Exponential model: Predictive result:Predictive result:

TxxQ exp)(

n

nT

xxQ

1)(

Predictive vs. Model DensityPredictive vs. Model Density

Which to use?Which to use? MLE curve is X-unbiasedMLE curve is X-unbiased

no uncertainty adjustment, but...no uncertainty adjustment, but... on average, gets right $ answeron average, gets right $ answer

Predictive curve is P-unbiasedPredictive curve is P-unbiased ““takes uncertainty into account” and...takes uncertainty into account” and... on average, reflects true exceedance pron average, reflects true exceedance pr

But they disagree...But they disagree... and it gets worse...and it gets worse...

for small qfor small q Expected estimated risk is greater than Expected estimated risk is greater than

the true risk (at the specified threshold)the true risk (at the specified threshold) Uncertainty now causes risk to be Uncertainty now causes risk to be overoverstated!stated!

Exponential MLE is Q-biasedExponential MLE is Q-biased

qQE q Xˆ

Exponential MLE is Q-biasedExponential MLE is Q-biased

Correcting for Q-biasCorrecting for Q-bias

Minimum Variance Unbiased EstimatorMinimum Variance Unbiased Estimator standard procedure in classical statisticsstandard procedure in classical statistics

Rao-Blackwell TheoremRao-Blackwell Theorem Expectation of unbiased estimator, Expectation of unbiased estimator,

conditional on sufficient statisticconditional on sufficient statistic Exponential model:Exponential model: MVUE result:MVUE result:

TxxQ exp)(

1

1)(

n

nT

xxQ

MVUE vs. Model DensityMVUE vs. Model Density

ParadoxParadox

Say we get an estimated T=1 (correct)Say we get an estimated T=1 (correct) MLE says XMLE says X.01.01=4.605, Pr{X>4.605}=1%=4.605, Pr{X>4.605}=1%

Predictive: XPredictive: X.01.01=5.179 is p-unbiased=5.179 is p-unbiased risk is greater than MLE answer because risk is greater than MLE answer because

impact of uncertaintyimpact of uncertainty MVUE: Pr{X>4.605}=.69% is q-unbiasedMVUE: Pr{X>4.605}=.69% is q-unbiased

risk is less because MLE tends to overstate risk is less because MLE tends to overstate exceedance probabilityexceedance probability

How the Paradox ArisesHow the Paradox Arises

ConclusionsConclusions

Uncertainty induces bias in estimatorsUncertainty induces bias in estimators Biases operate in different directionsBiases operate in different directions

depends on the question being askeddepends on the question being asked There is no monolithic “fix” for taking There is no monolithic “fix” for taking

uncertainty into accountuncertainty into account Predictive distribution fixes p-bias, Predictive distribution fixes p-bias, while making q-bias worsewhile making q-bias worse

RecommendationsRecommendations

First: Show modal estimates (MLE etc.)First: Show modal estimates (MLE etc.) Second: Show effect of uncertaintySecond: Show effect of uncertainty

Keep uncertainty distinct from randomnessKeep uncertainty distinct from randomness Sensitivity testing w.r.t. parameters Sensitivity testing w.r.t. parameters Confidence intervals on estimatorsConfidence intervals on estimators

Third: Adjust for bias only as necessaryThird: Adjust for bias only as necessary Carefully attend to the question askedCarefully attend to the question asked Advise that bias adjustment is equivocalAdvise that bias adjustment is equivocal