timothy aman, fcas maaa managing director, guy carpenter miami statistical limitations of...
TRANSCRIPT
![Page 1: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/1.jpg)
Timothy Aman, FCAS MAAAManaging Director, Guy Carpenter Miami
Statistical Limitations of Catastrophe Models
CAS Limited Attendance SeminarNew York, NY
18 September 2006
![Page 2: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/2.jpg)
2
Introduction
Given the limited Atlantic hurricane sample size, speakers discuss the limitations of predictive modeling from three perspectives:
– A frequentist (broker) approach using bootstrapping techniques
– A Bayesian (modeler) approach incorporating new events into a prior assumption framework
– A practical (insurer) approach reconciling the politics of actual claims experience with model-based expectations
![Page 3: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/3.jpg)
3
Introduction
When cat models first came out, loss estimates at various return periods AND upper confidence bounds around those loss estimates were regularly shown as output
Over the course of time, fewer and fewer output summaries have focused on confidence bounds and uncertainty
This panel attempts to remind us of the magnitude of that uncertainty, from various perspectives
![Page 4: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/4.jpg)
4
Outline
Definitions
A frequentist approach
An update
Statistical limitations of cat models
![Page 5: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/5.jpg)
5
Definitions
![Page 6: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/6.jpg)
6
Definitions
Frequentist: One who believes that the probability of an event should be defined as the limit of its relative frequency in a large number of trials
– Probabilities can be assigned only to events
– Need well-defined random experiment and sample space
Bayesian: Probability can be defined as degree to which a person believes a proposition
– Probabilities can be applied to statements
– Need a prior opinion (ideally, based on relevant knowledge)
![Page 7: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/7.jpg)
7
Definitions
A bootstrap sample is obtained by randomly sampling n times, with replacement, from the original data points [Efron]
Bootstrap methods are computer-intensive methods of statistical analysis that use simulation to calculate standard errors, confidence intervals, and significance tests [Davison and Hinkley]
![Page 8: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/8.jpg)
8
Definitions
In statistics bootstrapping is a method for estimating the sampling distribution of an estimator by resampling with replacement from the original sample
– Most often with the purpose of deriving robust estimates of standard errors and confidence intervals of a population parameter
The bootstrap technique assumes that the observed dataset is a representative subset of potential outcomes from some underlying distribution
– Random subsamples from the observed dataset are themselves representative subsets of potential outcomes
![Page 9: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/9.jpg)
9
A frequentist approach
![Page 10: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/10.jpg)
10
A frequentist approach
David Miller: “Uncertainty in Hurricane Risk Modeling and Implications for Securitization”, (Guy Carpenter, 1998)
– CAS Forum 1999, Securitization of Risk
David Miller “thought experiment”
– Create multiple catastrophe simulation models, each based on a simulated historical event set
![Page 11: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/11.jpg)
11
A frequentist approach
Miller’s approach
– Frequency is historical number of hurricanes over time period Assume distributed Poisson
– Conditional severity is based on bootstrap technique Assume stationary climate Each bootstrap replication represents an equivalent realization
of the historical record, and consists of random draw, with replacement, of N hurricanes from the observed record
Confidence intervals can then be determined from the boostrap replications
![Page 12: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/12.jpg)
12
A frequentist approach
Miller’s approach
– Essentially, each bootstrap replication represents a new catastrophe simulation model, created as if the observed historical event set had been the replicated rather than the actual event set
– “Blended” approach Severity distribution is calculated using a given catastrophe
model This severity distribution is fit to a parametric model (Beta
distribution) New parametric severity distribution is fit for each bootstrap
replication Use fitted parametric distribution for severity
![Page 13: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/13.jpg)
13
A frequentist approach
Miller’s conclusions for hurricane loss 90% confidence intervals for three US nationwide portfolios (personal, commercial, and specialty)
– Low return periods (<10 years) Lower bound is 0 Upper bound diverges (as multiple of mean)
– Remote return periods (>80 years) Lower bound 0.5 times mean estimate Upper bound 2.5 times mean estimate
![Page 14: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/14.jpg)
14Return Period (Years)
L̂L )95(.
L̂L )05(.
A frequentist approach
![Page 15: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/15.jpg)
15
An update
![Page 16: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/16.jpg)
16
An update
With the addition of more years of hurricane data, how have relative confidence intervals changed?
![Page 17: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/17.jpg)
17
An update
Suppose we want to estimate “100-year loss” to a portfolio
Suppose we have a reliable sample of 100 years of data
– We might have seen a 100-year loss in the sample (63% of samples, assuming Poisson frequency)
– We might not (37% of samples)
Now suppose we have a reliable sample of 110 years of data
– The above probabilities are revised to 67% and 33%
…and so on…
With a sample of 300 years, the probabilities are 95% and 5%
With a sample of 450 years, the probabilities are 99% and 1%
![Page 18: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/18.jpg)
18
An update
Bootstrap from cat model output
– Simulate datasets using cat model event sets
– “Direct” approach Eliminates need to specify, fit, and re-fit conditional severity
distributions
– Determine relative confidence intervals at various return periods
![Page 19: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/19.jpg)
19
An update
For a given return period n…
Mean
– Generate samples of n years
– Identify largest element of each sample year
– Take the average over all sample years of the largest observation in each year
Confidence intervals
– Capture through repeated experiment the distribution of the above mean
– Take the 5th and 95th sample percentiles of the maximum value across all sample years
– Obtain 90% confidence interval around mean estimate
![Page 20: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/20.jpg)
20
An update
90% confidence intervals around 100-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 21: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/21.jpg)
21
An update
90% confidence intervals around 100-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 22: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/22.jpg)
22
An update
90% confidence intervals around 100-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 23: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/23.jpg)
23
An update
90% confidence intervals around 100-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 24: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/24.jpg)
24
An update
Now a look at the 250-year level…
![Page 25: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/25.jpg)
25
An update
90% confidence intervals around 250-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 26: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/26.jpg)
26
An update
90% confidence intervals around 250-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 27: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/27.jpg)
27
An update
90% confidence intervals around 250-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 28: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/28.jpg)
28
An update
90% confidence intervals around 250-year loss
0.00
0.50
1.00
1.50
2.00
2.50
3.00
Miller Model A Model B Model C
5%ile 95%ile
![Page 29: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/29.jpg)
29
Statistical Limitations of Cat Models
![Page 30: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/30.jpg)
30
Statistical Limitations of Cat Models
John Major: “Uncertainty in Catastrophe Models: Part I: What is it and where does it come from?” and “Part II: How bad is it?”, (Guy Carpenter, 1999)
![Page 31: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/31.jpg)
31
Statistical Limitations of Cat Models
Sources of uncertainty in catastrophe modeling
1. Limited data sample For example, estimating 250-year EQ losses with only 100
years of detailed data
2. Model specification error For example, Poisson frequency (iid assumption)
3. Nonsampling error Identification of all relevant factors For example, global climate change
4. Approximation error For example, limited simulations and discrete event sets
![Page 32: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/32.jpg)
32
Statistical Limitations of Cat Models
Cat models are collections of event scenarios
– Discrete approximations, with probabilities attached to each scenario
– Not exhaustive
– Limited perils
– Calibrated using historical experience Recalibrated as required, based on research and actual event
experience
![Page 33: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/33.jpg)
33
Worldwide Property Catastrophe Insured Losses
$0
$10,000
$20,000
$30,000
$40,000
$50,000
$60,000
$70,000
$80,000
$90,000
'85 '87 '89 '91 '93 '95 '97 '99 '01 '03 '05*
USA Non-US
* Preliminary estimate. Source: Swiss Re Sigma
![Page 34: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/34.jpg)
34
Statistical Limitations of Cat Models
Uncertainty factors due to limited sample size are substantial
Data quality can add significantly to uncertainty
Are we capturing all material factors?
Scientific input can be used to reduce uncertainty
– Hazard sciences (meteorology, seismology, vulcanology)
– Engineering studies
![Page 35: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/35.jpg)
35
Statistical Limitations of Cat Models
Factors potentially influencing relative confidence interval widths
– Larger data sample / destabilizing recent experience
– Improvements in science / weakening of stationary climate assumption
– Improvements in technology
– Differences in modeled portfolios
– Negative Binomial frequency
– Increased awareness of factors contributing to uncertainty
Further exploration of the general factors influencing relative confidence interval widths is material for another presentation
![Page 36: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/36.jpg)
36
Statistical Limitations of Cat Models
Relative widths of individual company confidence intervals will depend on specifics
– Geographical scope e.g., US hurricane, Peru earthquake, UK flood
– Insured portfolio e.g., Dwellings, Petrochemical facilities, Hotels
– Financial variables e.g., Excess policies, EQ sublimits, Business interruption
Further exploration of the portfolio-specific factors influencing relative confidence interval widths is material for another presentation
![Page 37: Timothy Aman, FCAS MAAA Managing Director, Guy Carpenter Miami Statistical Limitations of Catastrophe Models CAS Limited Attendance Seminar New York, NY](https://reader035.vdocuments.mx/reader035/viewer/2022081516/5697c02e1a28abf838cda4a4/html5/thumbnails/37.jpg)
37
Statistical Limitations of Cat Models
“Don’t believe the cat model point estimates too much, but don’t believe them too little.”