model risk cultures - actuaries.chb9f3124e... · davos, 5 september 2014 . acknowledgments m bruce...
TRANSCRIPT
![Page 1: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/1.jpg)
Model Risk Cultures
Dr Andreas Tsanakas Cass Business School, City University London
Annual Meeting of the Swiss Association of Actuaries
Davos, 5 September 2014
![Page 2: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/2.jpg)
Acknowledgments
● M Bruce Beck, Warnell School of Forestry and Natural
Resources, University of Georgia
● Michael Thompson, International Institute for Applied
Systems Analysis
● The Model Risk Working Party of the Institute and
Faculty of Actuaries
![Page 3: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/3.jpg)
What is model risk? Office of the Comptroller of the Currency (2011)
[T]he potential for adverse consequences from decisions
based on incorrect or misused model outputs and reports.
Model risk occurs primarily for two reasons:
● The model may have fundamental errors and may
produce inaccurate outputs when viewed against the
design objective and intended business uses. […]
● The model may be used incorrectly or
inappropriately.
![Page 4: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/4.jpg)
Questions
● Are there model risks not associated with model
error?
● How do different ways of using models generate
model risks?
● What is the upside of model risk?
● What sorts of governance responses do different
model uses/risks necessitate?
● And what of Risk Culture?
![Page 5: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/5.jpg)
Types of model risk
● Technical
○ Coding errors, approximations
○ Arising from model uncertainty
● Behavioural
○ Information rejection
○ Loss of accountability
● Systemic
○ Market use of same models (CAT models, ESGs)
○ Endogenous model risks (VaR trading limits)
○ Can also be result of model-free strategies
![Page 6: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/6.jpg)
Model uncertainty (A)
A. Model uncertainty refers to all the possible ways in
which the model used diverges from some true but
unknown model of the process under consideration
● Under definition A., we cannot know the extent of
model uncertainty, but we can sometimes make
informed guesses
![Page 7: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/7.jpg)
Model uncertainty (B)
B. Model uncertainty refers to all the potential ways in
which the model’s inputs, outputs and structure may
change under plausible changes in assumptions
● Under definition B., the extent of model uncertainty is
revealed by sensitivity analysis
○ Materiality is application specific
○ Plausibility is not only a technical matter
○ See Beck (2014) for in-depth discussion
![Page 8: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/8.jpg)
Sensitivity of annuity value to model choice (70 year old male, discount at 3%; Richards et al, 2013)
0
0.5
1
1.5
2
2.5
11 12 13 14
Pro
bab
ility
Den
sity
Annuity Value
LC
DDE
LC(S)
CBD Gompertz
CBD Pspline
APC
2DAP
![Page 9: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/9.jpg)
Perception of models
Confidence / low concern for model uncertainty
Diffidence / high concern for model uncertainty
High legitimacy
of modelling
Low legitimacy
of modelling
![Page 10: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/10.jpg)
Perception of models
Confident model users
Known knowns
Confidence / low concern for model uncertainty
Diffidence / high concern for model uncertainty
High legitimacy
of modelling
Low legitimacy
of modelling
![Page 11: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/11.jpg)
Perception of models
Confident model users
Known knowns
Conscientious modellers
Known unknowns
Confidence / low concern for model uncertainty
Diffidence / high concern for model uncertainty
High legitimacy
of modelling
Low legitimacy
of modelling
![Page 12: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/12.jpg)
Perception of models
Confident model users
Known knowns
Uncertainty avoiders
Unknown unknowns
Conscientious modellers
Known unknowns
Confidence / low concern for model uncertainty
Diffidence / high concern for model uncertainty
High legitimacy
of modelling
Low legitimacy
of modelling
![Page 13: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/13.jpg)
Perception of models
Expert decision makers
Unknown knowns
Confident model users
Known knowns
Uncertainty avoiders
Unknown unknowns
Conscientious modellers
Known unknowns
Confidence / low concern for model uncertainty
Diffidence / high concern for model uncertainty
High legitimacy
of modelling
Low legitimacy
of modelling
![Page 14: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/14.jpg)
Confident model users: optimality
● Model drives decisions
● Focus on using model to
identify and exploit
opportunities
● Information discordant with
model is ignored
● Risk: world behaves very
differently to what model
predicts
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious modellers
![Page 15: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/15.jpg)
Conscientious modellers: fitness for purpose
● Model may inform some
decisions but not others
● Focus on stating uncertainties /
delimiting fitness of purpose
● While uncertainty is high,
overall paradigm is appropriate
● Risk: constraints on model
use / suboptimal strategy
● Risk: paradigm is wrong
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious modellers
![Page 16: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/16.jpg)
Uncertainty avoiders: robustness
● Models are never good enough
to drive decisions
● Focus on adverse scenarios,
structural changes,
interconnectedness of risks
● Conservative strategies that are
robust to model uncertainty
● Risk: highly suboptimal
decisions
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious modellers
![Page 17: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/17.jpg)
Expert decision makers: intuition
● Decisions driven by
management expertise alone
[But model may be manipulated
to produce convenient answers]
● Information discordant with
intuition is ignored
● Risk: missing dangers and
opportunities that model
could identify
● [Risk: loss of accountability]
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious modellers
![Page 18: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/18.jpg)
On individuals
● The classification reflects ways of thinking about
models, not individual psychological profiles
● People may hold different views depending e.g. on
context and professional affiliation
○ Or indeed entertain conflicting views
● Each perspective challenges the other three
○ Twelve challenges in total
![Page 19: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/19.jpg)
On individuals
● The classification reflects ways of thinking about
models, not individual psychological profiles
● People may hold different views depending e.g. on
context and professional affiliation
○ Or indeed entertain conflicting views
● Each perspective challenges the other three
○ Twelve challenges in total
● People may be responsive to those who hold different
views or instead...
![Page 20: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/20.jpg)
What “we” may say about “them”
Expert decision makers
Confident model users
“Naïve”
Uncertainty avoiders
Conscientious modellers
![Page 21: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/21.jpg)
What “we” may say about “them”
Expert decision makers
“Stone-age cynics”
Confident model users
Uncertainty avoiders
Conscientious modellers
![Page 22: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/22.jpg)
What “we” may say about “them”
Expert decision makers
Confident model users
Uncertainty avoiders
“Unhelpful scaremongers”
Conscientious modellers
![Page 23: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/23.jpg)
What “they” may say about “us”
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious
modellers
“Obstructive
worriers”
![Page 24: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/24.jpg)
What “they” may say about “us”
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious
modellers
“Irrelevant
navel-gazers”
![Page 25: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/25.jpg)
What “they” may say about “us”
Expert decision makers
Confident model users
Uncertainty avoiders
Conscientious
modellers
“Missing the
forest for the
tree”
![Page 26: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/26.jpg)
Model specification and delivery (“us” vs. Confident model users)
● Technical modellers now recognize how model
complexity (“granularity”) increases sensitivity
○ E.g. in hierarchical dependence modelling
● Management information, reporting requirements,
model embedding, require more detail
● Statistical inference, sensitivity analysis, may point out
the need for less detail
![Page 27: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/27.jpg)
Paradigm challenge (“us” vs. Uncertainty avoiders)
● Economic theory is flawed (rational agents,
frictionless markets, linear valuation functionals...)
● Endogeneity of financial risk
● Claims generating processes undergo unobservable
structural changes
● But how can we quantify uncertainty without employing
a paradigm?
● How can we think outside the box, without a box?
![Page 28: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/28.jpg)
Input/output feedback (“us” vs. Expert decision makers)
● We know that portfolio risk can greatly vary with small
changes in hard-to-validate dependence assumptions
● Statistically plausible risk metrics (reserve estimates,
portfolio VaRs...), when linked to decisions, can be
commercially unviable
● Model uncertainty allows choice of dependence
parameters that give reasonable outputs
● Reasonableness is a social construct
![Page 29: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/29.jpg)
Input/output feedback (“us” vs. Expert decision makers)
“Both groups have difficulty recognizing the ways in which
the process subtly interweaves truth seeking and
advantage seeking, leaving each somewhat compromised
by the other, even as each somewhat serves the other.”
(March, 1994)
![Page 30: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/30.jpg)
Basket of 5 defaultable securities p=5%, Beta-binomial dependence
Number of
Defaults
Probability
(corr=0)
Multiplier
(corr=1%)
Multiplier
(corr=5%)
Multiplier
(corr=10%)
5 0.0000313% 4.44 84.68 508.53
4 0.00297% 2.45 16.25 51.42
3 0.113% 1.54 4.13 7.49
2 2.14% 1.12 1.48 1.70
1 20.4% 0.96 0.84 0.71
0 77.4% 1.01 1.02 1.04
![Page 31: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/31.jpg)
Hegemony eats itself
“Why didn't rating agencies build in some cushion for this
sensitivity to a house-price-depreciation scenario? Because if
they had, they would have never rated a single mortgage-
backed CDO.” (Salmon, 2009)
“Where is the liquidity crisis supposed to come from?”
somebody asked in the meeting. No one could give a good
answer. [...] In their eyes, we were not earning money for the
bank. Worse, we had the power to say no and therefore prevent
business from being done. Traders saw us as obstructive and
a hindrance to their ability to earn higher bonuses.”
(Confessions of a risk manager, Economist, 2008)
![Page 32: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/32.jpg)
Echoes of Cultural Theory Thompson et al (1990); Ingram et al (2012)
![Page 33: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/33.jpg)
Echoes of Cultural Theory Thompson et al (1990); Ingram et al (2012)
1. In the presence of deep uncertainties, risk is socially
constructed. What we consider to be risky depends on
moral and political considerations.
![Page 34: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/34.jpg)
Echoes of Cultural Theory Thompson et al (1990); Ingram et al (2012)
1. In the presence of deep uncertainties, risk is socially
constructed. What we consider to be risky depends on
moral and political considerations.
2. There are four fundamental ways of perceiving risk
(rationalities / Risk Cultures), linked to different ways of
organising social and economic relations (ways of life).
![Page 35: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/35.jpg)
Echoes of Cultural Theory Thompson et al (1990); Ingram et al (2012)
1. In the presence of deep uncertainties, risk is socially
constructed. What we consider to be risky depends on
moral and political considerations.
2. There are four fundamental ways of perceiving risk
(rationalities / Risk Cultures), linked to different ways of
organising social and economic relations (ways of life).
3. Each of these Risk Cultures represents a distinct form of
human wisdom. But they are also mutually incompatible –
consensus is not possible.
![Page 36: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/36.jpg)
Echoes of Cultural Theory Thompson et al (1990); Ingram et al (2012)
1. In the presence of deep uncertainties, risk is socially
constructed. What we consider to be risky depends on
moral and political considerations.
2. There are four fundamental ways of perceiving risk
(rationalities / Risk Cultures), linked to different ways of
organising social and economic relations (ways of life).
3. Each of these Risk Cultures represents a distinct form of
human wisdom. But they are also mutually incompatible –
consensus is not possible.
4. An institution committed to only one of these Risk Cultures
is not viable. Each way of life needs the presence of all
others to be viable.
![Page 37: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/37.jpg)
Myths of nature
“Irrelevant
navel-gazers”
Nature benign
Known knowns
Nature capricious
Unknown knowns
Nature perverse/tolerant
Known unknowns
Nature ephemeral
Unknown unknowns
![Page 38: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/38.jpg)
Clumsy solutions
● Lack of consensus means that viable solutions will
have to be clumsy rather than elegant
● Different rationalities need to be represented, have
access to decision making and be responsive to
each other
![Page 39: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/39.jpg)
Clumsy solutions
● Lack of consensus means that viable solutions will
have to be clumsy rather than elegant
● Different rationalities need to be represented, have
access to decision making and be responsive to
each other
● The people of Davos have managed such
pluralism for many years (Thompson, 2002)
○ Private property
○ Wald- and Alpgenossenschaften
○ Avalanche control
○ Casual labour
![Page 40: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/40.jpg)
Model governance
● Current best-practice
○ Model inventory and documentation standards
○ Improved articulation of model limitations
○ Requirements and restrictions on model use
○ Independent model validation process
![Page 41: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/41.jpg)
Model governance
● Current best-practice
○ Model inventory and documentation standards
○ Improved articulation of model limitations
○ Requirements and restrictions on model use
○ Independent model validation process
● “Culturally aware governance”
○ Multi-way challenge
○ Accessibility and responsiveness
![Page 42: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/42.jpg)
Regulatory guidance: 1-way challenge
Expert decision makers
Confident model users
Uncertainty avoiders
Office of the Comptroller of the
Currency
![Page 43: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/43.jpg)
Regulatory guidance: 1-way challenge
Expert decision makers
Confident model users
Conscientious modellers
Haldane & Madouros:
The Dog and the Frisbee
![Page 44: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/44.jpg)
Challenges to “conscientious modelling”
● Conditions for embedding in decisions and attracting
investment in the model
○ Model user friendly, addresses business issues,
released on time
○ Test output against expert opinion and
commercial implications
○ Extended peer review
![Page 45: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/45.jpg)
Challenges to “conscientious modelling”
● Conditions for embedding in decisions and attracting
investment in the model
○ Model user friendly, addresses business issues,
released on time
○ Test output against expert opinion and
commercial implications
○ Extended peer review
● But we also need strong challenges to model-bashing
and manipulation
![Page 46: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/46.jpg)
The quest for openness
● Should we (?) be able to openly say this:
“Assumptions are consistent with empirical evidence
and best modelling practice. Model uncertainty
remains high. The precise model calibration is such
that standard outputs are also consistent with
senior management’s perspective of a commercially
reasonable capital requirement.”
![Page 47: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/47.jpg)
The quest for openness
● Should we (?) be able to openly say this:
“Assumptions are consistent with empirical evidence
and best modelling practice. Model uncertainty
remains high. The precise model calibration is such
that standard outputs are also consistent with
senior management’s perspective of a commercially
reasonable capital requirement.”
● If we acknowledge the legitimacy of such concerns,
will this improve scientific integrity?
○ To ask hard questions, we need to be unafraid of
the answers
![Page 48: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/48.jpg)
Yes but...
● Why would Expert Decision Makers choose to
manipulate models
○ Why use a model at all?
● In the face of deep uncertainties, what exactly do
Conscientious Modellers have to offer?
○ Apart from: “sorry can’t answer this question”...
![Page 49: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/49.jpg)
Yes but...
● Why would Expert Decision Makers choose to
manipulate models
○ Why use a model at all?
● In the face of deep uncertainties, what exactly do
Conscientious Modellers have to offer?
○ Apart from: “sorry can’t answer this question”...
● If we had better data / models / software etc, could
we all agree what to do?
![Page 50: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/50.jpg)
Decision principles
● To make a decision we need
○ Technical (probability) assessment
○ Decision principle
● Disagreement can be in respect of either element
● Sensitivity of decision to model changes reveals
materiality of model uncertainty
● Rigid application of some sanctioned decision principle
is a form of hegemony
![Page 51: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/51.jpg)
Fields of contestation (how we often think)
Science
contested
Principle
contested
Simple problems No No
Model risk Yes No
Ecstasy v horse-riding No Yes
Climate change Yes Yes
![Page 52: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/52.jpg)
Fields of contestation (how things are)
Science
contested
Principle
contested
Simple problems No No
Model risk Yes No Yes
Ecstasy v horse-riding No Yes Yes
Climate change Yes Yes
![Page 53: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/53.jpg)
Rebellious rationalities
● Conscientious modellers
○ Re-define purpose of modelling!
○ (Protect integrity and need for expertise)
● Uncertainty avoiders
○ Change decision principle!
○ (Protect enterprise or market)
● Expert decision makers
○ Rig the model to get the answers I want!
○ (Survive by manipulation)
![Page 54: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/54.jpg)
Conclusions
● Governance should encourage multi-way challenges
to model (non-)use as legitimate
○ Are all perspectives heard?
○ Do all perspectives respond to others?
● Not easy to distinguish contestation of models from
contestation of decision principles
● How does model governance improve accountability?
● Acknowledging the political nature of risk
○ Less politics worse science
![Page 55: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/55.jpg)
References
1. Beck (2014), “Handling uncertainty in environmental models at the
science-policy-society interfaces,” in Boumans, Hon, and Petersen,
(eds), Error and Uncertainty in Scientific Practice, Pickering &
Chatto, pp 97-135.
2. Haldane and Madouros (2012) “The dog and the frisbee”, Federal
Reserve Bank of Kansas City’s 36th economic policy symposium.
3. Ingram, Tayler, and Thompson (2012), “Surprise, surprise: from
neoclassical economics to e-life”, ASTIN Bulletin 42 (2), 389-411.
4. March (1994), A Primer on Decision Making: How Decisions
Happen, Simon & Schuster.
![Page 56: Model Risk Cultures - actuaries.chb9f3124e... · Davos, 5 September 2014 . Acknowledgments M Bruce Beck, Warnell School of Forestry and Natural ... Model may inform some decisions](https://reader033.vdocuments.mx/reader033/viewer/2022042400/5f0e60817e708231d43ef31d/html5/thumbnails/56.jpg)
References
5. Office of the Comptroller of the Currency (2012), “Supervisory
guidance on model risk management”, Report OCC 2011-22,
Board of Governors of the Federal Reserve System
6. Richards, Currie, and Ritchie (2013) “A Value-at-Risk framework
for longevity trend risk” British Actuarial Journal 19(01), 116-139.
7. Thompson (2002), “Man and nature as a single but complex
system,” in Munn (ed), Encyclopedia of Global Environmental
Change, Vol 5, Wiley, pp 384-393.
8. Thompson, Ellis, and Wildavsky (1990), Cultural Theory, Westview
Press.