quantile uncertainty and value-at-risk model risk

Risk Analysis DOI: 10.1111/j.1539-6924.2012.01824.x

Quantile Uncertainty and Value-at-Risk Model Risk

Carol Alexander1,∗ and Jose Marıa Sarabia2

This article develops a methodology for quantifying model risk in quantile risk estimates. Theapplication of quantile estimates to risk assessment has become common practice in many dis-ciplines, including hydrology, climate change, statistical process control, insurance and actu-arial science, and the uncertainty surrounding these estimates has long been recognized. Ourwork is particularly important in finance, where quantile estimates (called Value-at-Risk)have been the cornerstone of banking risk management since the mid 1980s. A recent amend-ment to the Basel II Accord recommends additional market risk capital to cover all sourcesof “model risk” in the estimation of these quantiles. We provide a novel and elegant frame-work whereby quantile estimates are adjusted for model risk, relative to a benchmark whichrepresents the state of knowledge of the authority that is responsible for model risk. A simu-lation experiment in which the degree of model risk is controlled illustrates how to quantifyValue-at-Risk model risk and compute the required regulatory capital add-on for banks. Anempirical example based on real data shows how the methodology can be put into practice,using only two time series (daily Value-at-Risk and daily profit and loss) from a large bank.We conclude with a discussion of potential applications to nonfinancial risks.

KEY WORDS: Basel II; maximum entropy; model risk; quantile; risk capital; value-at-risk

1. INTRODUCTION

This article focuses on the model risk of quantilerisk assessments with particular reference to “Value-at-Risk” (VaR) estimates, which are derived fromquantiles of portfolio profit and loss (P&L) distri-butions. VaR corresponds to an amount that couldbe lost, with a specified probability, if the portfo-lio remains unmanaged over a specified time hori-zon. It has become the global standard for assessingrisk in all types of financial firms: in fund manage-

1ICMA Centre, Henley Business School at the University ofReading, Reading, RG6 6BA, UK.

2Department of Economics, University of Cantabria, Avda. de losCastros s/n, 39005-Santander, Spain; [email protected]

∗Address correspondence to Carol Alexander, Chair of Risk Man-agement, ICMA Centre, Henley Business School at the Uni-versity of Reading, Reading, RG6 6BA, UK; Phone: +44 1183786431; [email protected].

ment, where portfolios with long-term VaR objec-tives are actively marketed; in the treasury divisionsof large corporations, where VaR is used to assess po-sition risk; and in insurance companies, who measureunderwriting and asset management risks in a VaRframework. But most of all, banking regulators re-main so confident in VaR that its application to com-puting market risk capital for banks, used since the1996 amendment to the Basel I Accord,3 will soon beextended to include stressed VaR under an amendedBasel II and the new Basel III Accords.4

The finance industry’s reliance on VaR hasbeen supported by decades of academic research.Especially during the last 10 years there has beenan explosion of articles published on this subject.Popular topics include the introduction of new VaR

3 See Basel Committee on Banking Supervision(1).4 See Basel Committee on Banking Supervision(2,3).

1 0272-4332/12/0100-0001$22.00/1 C© 2012 Society for Risk Analysis

2 Alexander and Sarabia

models,5 and methods for testing their accuracy.6

However, the stark failure of many banks to set asidesufficient capital reserves during the banking crisis of2008 sparked an intense debate on using VaR modelsfor the purpose of computing the market risk capitalrequirements of banks. Turner(18) is critical of themanner in which VaR models have been applied andTaleb(19) even questions the very idea of using statis-tical models for risk assessment. Despite the warn-ings of Turner, Taleb, and early critics of VaR modelssuch as Beder(20), most financial institutions continueto employ them as their primary tool for market riskassessment and economic capital allocation.

For internal, economic capital allocation pur-poses VaR models are commonly built using a“bottom-up” approach. That is, VaR is first assessedat an elemental level; for example, for each individ-ual trader’s positions; then is it progressively aggre-gated into desk-level VaR and VaR for larger andlarger portfolios; until a final VaR figure for a port-folio that encompasses all the positions in the firmis derived. This way the traders’ limits and risk bud-gets for desks and broader classes of activities can beallocated within a unified framework. However, thisbottom-up approach introduces considerable com-plexity to the VaR model for a large bank. Indeed, itcould take more than a day to compute the full (oftennumerical) valuation models for each product overall the simulations in a VaR model. Yet, for regula-tory purposes VaR must be computed at least daily,and for internal management intra-day VaR compu-tations are frequently required.

To reduce complexity in the internal VaR systemsimplifying assumptions are commonly used, in thedata generation processes assumed for financial assetreturns and interest rates and in the valuation mod-els used to mark complex products to market everyday. For instance, it is very common to apply nor-mality assumptions in VaR models, along with log-normal, constant volatility approximations for exotic

5 Historical simulation(4) is the most popular approach amongstbanks(5) but data-intensive and prone to pitfalls(6). Other popu-lar VaR models assume normal risk factor returns with the Risk-Metrics covariance matrix estimates(7). More complex VaR mod-els are proposed by Hull and White(8), Mittnik and Paolella(9),Ventner and de Jongh(10), Angelidis et al.(11), Hartz et al.(12),Kuan et al.(13), and many others.

6 The coverage tests introduced by Kupiec(14) are favored by bank-ing regulators, and these are refined by Christoffersen(15). How-ever Berkowitz et al.(16) demonstrate that more sophisticatedtests such as the conditional autoregressive test of Engle andManganelli(17) may perform better.

options prices and sensitivities.7 Of course, there isconclusive evidence that financial asset returns arenot well represented by normal distributions. How-ever, the risk analyst in a large bank may be forcedto employ this assumption for pragmatic reasons.

Another common choice is to base VaR calcu-lations on simple historical simulation. Many largecommercial banks have legacy systems that are onlyable to compute VaR using this approach, commonlybasing calculations on at least three years of dailydata for all traders’ positions. Thus, some years af-ter the credit and banking crisis, vastly over-inflatedVaR estimates were produced by these models longafter the markets returned to normal. The implicitand simplistic assumption that history will repeat it-self with certainty—that the banking crisis will recurwithin the risk horizon of the VaR model—may wellseem absurd to the analyst, yet he is constrained bythe legacy system to compute VaR using simple his-torical simulation. Thus, financial risk analysts are of-ten required to employ a model that does not complywith their views on the data generation processes forfinancial returns, and data that they believe are inap-propriate.8

Given some sources of uncertainty a Bayesianmethodology(21,22) provides an alternative frame-work to make probabilistic inferences about VaR,assuming that VaR is described in terms of a set ofunknown parameters. Bayesian estimates may be de-rived from posterior parameter densities and pos-terior model probabilities which are obtained fromthe prior densities via Bayes theorem, assuming thatboth the model and its parameters are uncertain. Ourmethod shares ideas with the Bayesian approach, inthe sense that we use a “prior” distribution for thequantile probability, in order to obtain a posteriordistribution for the quantile.

The problem of quantile estimation under modeland parameter uncertainty has also been stud-ied from a classical (i.e., non-Bayesian) point ofview. Modarres et al.(23) considered the accuracyof upper and extreme tail estimates of three rightskewed distributions (log-normal, log-logistic, andlog-double exponential) under model and parameter

7 Indeed, model risk frequently spills over from one business lineto another, for example, normal VaR models are often employedin large banks simply because they are consistent with the geo-metric Brownian motion assumption that is commonly appliedfor option pricing and hedging.

8 Banking regulators recommend three to five years of data for his-torical simulation and require at least one year of data for con-structing the covariance matrices used in other VaR models.

Quantile Uncertainty and VaR Model Risk 3

uncertainty. These authors examined and comparedperformances of the maximum likelihood and non-parametric estimators based on the empirical or aquasi-empirical quantile function, assuming four dif-ferent scenarios: the model is correctly specified; themodel is mis-specified; the best model is selected us-ing the data; and no form is assumed for the model.Giorgi and Narduzzi(24) have studied quantile esti-mation for a self-similar time series and the uncer-tainty that affects their estimates. Figlewski(25) dealswith estimation error in the assessment of financialrisk exposure. This author finds that, under stochasticvolatility, estimation error can increase the probabil-ities of multi-day events such as three 1% tail eventsin a row, by several orders of magnitude. Empiricalfindings are also reported using 40 years of daily S&P500 returns.

The term “model risk” is commonly applied toencompass various sources of uncertainty in statis-tical models, including model choice and parame-ter uncertainty. In July 2009, revisions to the BaselII market risk framework added the requirementthat banks set aside additional reserves to cover allsources of model risk in the internal models usedto compute the market risk capital charge.9 Thus,the issue of model risk in internal risk models hasrecently become very important to banks. Finan-cial risk research has long recognized the impor-tance of model risk. However, following some earlywork(26,27,28,29,30,31) surprisingly few papers deal ex-plicitly with VaR model risk. Early work(32,33) inves-tigated sampling error and Kerkhof et al.(34) quan-tify the adjustment to VaR that is necessary forsome econometric models to pass regulatory back-tests. Quantile-based risk assessment has also beenapplied to numerous problems in insurance and ac-tuarial science: see Reiss and Thomas,(35) Cairns,(36)

Matthys et al.,(37) Dowd and Blake(38) and many oth-ers. However, a general methodology for assessingquantile model risk has yet to emerge.

This article introduces a new framework for mea-suring quantile model risk and derives an elegant, in-tuitive and practical method for computing the riskcapital add-on to cover VaR model risk. In additionto the computation of a model risk “add-on” for agiven VaR model and given portfolio, our approachcan be used to assess which, of the available VaRmodels, has the least model risk relative to a givenportfolio. Similarly, given a specific VaR model, ourapproach can assess which portfolio has the least

9 See Basel Committee on Banking Supervision, Section IV.(3)

model risk. However, outside of a simulation envi-ronment, the concept of a “true” model against whichone might assess model risk is meaningless. All wehave is some observable data and our beliefs aboutthe conditional and/or unconditional distribution ofthe random variable in question. As a result, modelrisk can only be assessed relative to some bench-mark model, which itself is a matter for subjectivechoice.

In the following: the definition of model riskand a benchmark for assessing model risk is dis-cussed in Section 2; Section 3 gives a formal defi-nition of quantile model risk and outlines a frame-work for its quantification. We present a statisti-cal model for the probability α that is assigned, un-der the benchmark distribution, to the α quantile ofthe model distribution. Our idea is to endogenizemodel risk by using a distribution for α to gener-ate a distribution for the quantile. The mean of thismodel-risk-adjusted quantile distribution detects anysystematic bias in the model’s α quantile, relativeto the α quantile of the benchmark distribution. Asuitable quantile of the model-risk-adjusted distribu-tion determines an uncertainty buffer which, whenadded to the bias-adjusted quantile gives a model-risk-adjusted quantile that is no less than the α quan-tile of the benchmark distribution at a predeterminedconfidence level, this confidence level correspond-ing to a penalty imposed for model risk; Section 4presents a numerical example on the application ofour framework to VaR model risk, in which the de-gree of model risk is controlled by simulation; Sec-tion 5 illustrates how the methodology could be im-plemented by a manager or regulator having accessto only two time series from the bank: its aggregatedaily trading P&L and its corresponding 1% VaR es-timates, derived from the usual “bottom up” VaRaggregation framework; Section 6 discusses the rele-vance of the methodology to nonfinancial problems;and Section 7 summarizes and concludes.

2. MODEL RISK AND THE BENCHMARK

We distinguish two sources of model risk:“model choice,” that is, inappropriate assumptionsabout the form of the statistical model for the ran-dom variable; and “parameter uncertainty,” that is,estimation error in the parameters of the chosenmodel. One never knows the “true” model exceptin a simulation environment, so assumptions aboutthe form of statistical model must be made. Param-eter uncertainty includes sampling error (parameter


values can never be estimated exactly because onlya finite set of observations on a random variable areavailable) and optimization error (e.g., different nu-merical algorithms typically produce slightly differ-ent estimates based on the same model and the samedata). We remark that there is no consensus on thesources of model risk. For instance, Cont(39) pointsout that both these sources could be encompassedwithin a universal model, and Kerkhof et al.(34)

distinguish “identification risk” as an additionalsource.

Model risk in finance has been approached intwo different ways: examining all feasible models andevaluating the discrepancy in their results, or speci-fying a benchmark model against which model riskis assessed. Papers on the quantification of valua-tion model risk in the risk-neutral measure exemplifyeach approach: Cont(39) quantifies the model risk of acomplex product by the range of prices obtained un-der all possible valuation models that are calibratedto market prices of liquid (e.g., vanilla) options; Hulland Suo(40) define model risk relative to the impliedprice distribution, that is, a benchmark distributionimplied by market prices of vanilla options. In thecontext of VaR model risk the benchmark approach,which we choose to follow, is more practical than theformer.

Some authors identify model risk with the de-parture of a model from a “true” dynamic process:see Branger and Schlag(41) for instance. Yet, outsideof an experimental or simulation environment, wenever know the “true” model for sure. In practice,all we can observe are realizations of the data genera-tion processes for the random variables in our model.It is futile to propose the existence of a unique andmeasurable “true” process because such an exerciseis beyond our realm of knowledge.

However, we can observe a maximum entropydistribution (MED). This is based on a “state ofknowledge,” that is, no more and no less than the in-formation available regarding the random variable’sbehavior. This information includes the observabledata that are thought to be relevant plus any subjec-tive beliefs. Since neither the choice of sample northe beliefs of the modeller can be regarded as objec-tive, the MED is subjective. For our application toVaR we consider two perspectives on the MED, theinternal perspective where the MED would be set bythe risk analyst himself, or else by the Chief Risk Of-ficer of the bank, and the external perspective wherethe MED would be set by the regulator.

Shannon(43) defined the entropy of a probabilitydensity function g(x), x ∈ R as

H(g) = −Eg[log g(x)] = −∫R

g(x) log g(x)dx.

This is a measure of the uncertainty in a probabilitydistribution and its negative is a measure of informa-tion.10 The maximum entropy density is the functionf (x) that maximizes H(g), subject to a set of con-ditions on g(x) which capture the testable informa-tion.11 The criterion here is to be as vague as possible(i.e., to maximize uncertainty) given the constraintsimposed by the state of knowledge. This way, theMED represents no more (and no less) than the in-formation available. If this information consists onlyof a historical sample on X of size n then, in ad-dition to the normalization condition, there are nconditions on g(x), one for each data point. In thiscase, the MED is just the empirical distribution basedon that sample. Otherwise, the testable informa-tion consists of fewer conditions, which capture onlythat sample information which is thought to be rele-vant, and any other conditions imposed by subjectivebeliefs.

Our recommendation is that banks assess theirVaR model risk by comparing their aggregate VaRfigure, which is typically computed using the bottom-up approach, with the VaR obtained using the MEDin a “top-down” approach, that is, calibrated directlyto the bank’s aggregate daily trading P&L. Typicallythis P&L contains marked-to-model prices for illiq-uid products, in which case their valuation model riskis not quantified in our framework.

From the banking regulator’s perspective whatmatters is not the ability to aggregate and disaggre-gate VaR in a bottom-up framework, but the ade-quacy of a bank’s total market risk capital reserves,which are derived from the aggregate market VaR.Therefore, regulators only need to define a bench-mark VaR model to apply to the bank’s aggregate

10 For instance, if g is normal with variance σ 2, H(g) = 12 (1 +

log(2π) + log(σ )), so the entropy increases as σ increases andthere is more uncertainty and less information in the distribu-tion. As σ → 0 and the density collapses the Dirac function at 0,there is no uncertainty but −H(g) → ∞ and there is maximuminformation. However, there is no universal relationship betweenvariance and entropy and where their orderings differ entropy isthe superior measure of information. See Ebrahimi et al.(42) forfurther insight.

11 A piece of information is “testable” if it can be determinedwhether F is consistent with it. One of piece of information isalways a normalization condition.


daily P&L. This model will be the MED of the regu-lator, that is, the model that best represents the reg-ulator’s state of knowledge regarding the accuracy ofVaR models.

Following the theoretical work of Shannon(43),Zellner(44), Jaynes(45), and many others it is com-mon to assume the testable information is given bya set of moment functions derived from a sample, inaddition to the normalization condition. When onlythe first two standard moments (mean and variance)are deemed relevant, the MED is a normal distribu-tion(43). More generally, when the testable informa-tion contains the first N sample moments, f (x) takesan exponential form. This is found by maximizing en-tropy subject to the conditions

μn =∫R

xng(x)dx, n = 0, . . . , N,

where μ0 = 1 and μn, n = 1, . . . , N are the momentsof the distribution. The solution is

f (x) = exp

(−

n=N∑n=0

λnxn

),

where the parameters λ0, . . . λn are obtained by solv-ing the system of nonlinear equations

μn =∫

xn exp

(−

n=N∑n=0

λnxn

)dx, n = 0, . . . , N.

Rockinger and Jondeau(46), Wu(47), Chan(48,49),and others have applied a simple four-moment MEDto various econometric and risk management prob-lems. Perhaps surprisingly, since tail weight is an im-portant aspect of financial asset returns distributions,none consider the tail weight that is implicit in the useof an MED based on standard sample moments. Butsimple moment-based MEDs are only well-definedwhen N is even. For any odd value of N there will bean increasing probability weight in one of the tails.Also, the four-moment MED has lighter tails than anormal distribution, due to the presence of the termexp[−λ4x4] with non-zero λ4 in f (x). Indeed, themore moments included in the conditions, the thin-ner the tail of the MED. Because financial asset re-turns are typically heavy-tailed it is likely that thisproperty will carry over to a bank’s aggregate dailyP&L, in which case we would not advocate the use ofsimple moment-based MEDs.

Park and Bera(50) address the issue of heavytails in financial data by introducing additional pa-rameters into the moment functions, thus extendingthe family of moment-based MEDs. Even with just

two (generalized) moment conditions based on oneadditional parameter they show that many heavy-tailed distributions are MEDs, including the Studentt and generalized error distributions that are com-monly applied to VaR analysis—see Jorion(32) andLee et al.(51), for example. Since our article concernsthe estimation of low-probability quantiles we shallutilize these distributions as MEDs in our empiricalstudy of Section 5.

There are advantages in choosing a parametricMED for the benchmark. VaR is a quantile of aforward-looking P&L distribution, but to base pa-rameter estimates entirely on historical data limitsbeliefs about the future to experiences from the past.Parametric distributions are frequently advocated forVaR estimation, and stress testing in particular, be-cause the parameters estimated from historical datamay be changed subjectively to accommodate be-liefs about the future P&L distribution. We distin-guish two types of parametric MEDs. UnconditionalMEDs are based on the independent and identi-cally distributed (i.i.d.) assumption. However, sinceMandelbrot(52) it has been observed that financial as-set returns typically exhibit a “volatility clustering”effect, thus violating the i.i.d. assumption. Thereforeit may be preferable to assume the stochastic processfor returns has time-varying conditional distributionsthat are MEDs.

Volatility clustering is effectively captured bythe flexible and popular class of generalized con-ditional heteroskedastic models (GARCH) modelsintroduced by Bollerslev(53) and since extended innumerous ways by many authors. Berkowitz andO’Brien(54) found that most bottom-up internal VaRmodels produced VaR estimates that were too large,and insufficiently risk-sensitive, compared with top–down GARCH VaR estimates derived directly fromaggregate daily P&L. Thus, from the regulator’s per-spective, a benchmark for VaR model risk based ona GARCH process for aggregate daily P&L withconditional MEDs would seem appropriate. Filteredhistorical simulation of aggregate daily P&L wouldbe another popular alternative, especially when ap-plied with a volatility filtering that increases its risksensitivity: see Barone-Adesi et al.(55) and Hull andWhite(56). Alexander and Sheedy(57) demonstratedempirically that GARCH volatility filtering com-bined with historical simulation can produce very ac-curate VaR estimates, even at extreme quantiles. Bycontrast, the standard historical simulation approach,which is based on the i.i.d. assumption, failed many oftheir backtests.


3. MODELING QUANTILE MODEL RISK

The α quantile of a continuous distribution F ofa real-valued random variable X with range R is de-noted

qFα = F−1(α). (1)

In financial applications the probability α is oftenpredetermined. Frequently it will be set by seniormanagers or regulators and small or large values cor-responding to extreme quantiles are very commonlyused. For instance, regulatory market risk capital isbased on VaR models with α = 1% and a risk hori-zon of 10 trading days.

In our statistical framework F is identified withthe unique MED based on a state of knowledge Kwhich contains all testable information on F . Wecharacterize a statistical model as a pair {F, K},where F is a distribution and K is a filtration whichencompasses both the model choice and its parame-ter values. The model provides an estimate F of F ,and uses this to compute the α quantile. That is, in-stead of (1) we use

qFα = F−1(α). (2)

Quantile model risk arises because {F, K} �= {F,K}.First, K �= K, for example, K may include the beliefthat only the last six months of data are relevant tothe quantile today; yet K may be derived from an in-dustry standard that must use at least one year of ob-served data in K;12 and second, F is not, typically, theMED even based on K, for example, the execution offirm-wide VaR models for a large commercial bankmay present such a formidable time challenge that Fis based on simplified data generation processes, asdiscussed in the introduction.

In the presence of model risk the α quantile ofthe model is not the α quantile of the MED, that is,qF

α �= qFα . The model’s α quantile qF

α is at a differentquantile of F and we use the notation α for this quan-tile, that is, qF

α = qFα , or equivalently,

α = F(F−1(α)). (3)

In the absence of model risk α = α for every α. Oth-erwise, we can quantify the extent of model risk bythe deviation of α from α, that is, the distribution ofthe quantile probability errors

e(α|F, F) = α − α. (4)

12 As is the case under current banking regulations for the use ofVaR to estimate risk capital reserves - see Basel Committee onBanking Supervision.(1)

If the model suffers from a systematic, measurablebias at the α quantile then the mean error e(α|F, F)should be significantly different from zero. A signifi-cant and positive (negative) mean indicates a system-atic over (under) estimation of the α quantile of theMED. Even if the model is unbiased it may still lackefficiency, that is, the dispersion of e(α|F, F) may behigh. Several measures of dispersion may be used toquantify the efficiency of the model, including theroot mean squared error (RMSE), the mean absoluteerror (MAE), and the range.

We now regard α = F(F−1(α)) as a random vari-able with a distribution that is generated by our twosources of model risk, that is, model choice and pa-rameter uncertainty. Because α is a probability ithas range [0, 1], so the α quantile of our model, ad-justed for model risk, falls into the category of gen-erated random variables. For instance, if α is param-eterized by a beta distribution B(a, b) with density(0 < u < 1)

fB(u; a, b) = B(a, b)−1[ua−1(1 − u)b−1], (5)

a, b ≥ 0, where B(a, b) is the beta function, then theα quantile of our model, adjusted for model risk, is abeta-generated random variable:

Q(α|F, F) = F−1(α), α ∼ B(a, b).

Beta generated distributions were introduction byEugene et al.(58) and Jones(59). They may be charac-terized by their density function (−∞ < x < ∞)

gF (x) = B(a, b)−1 f (x)[F(x)]a−1[1 − F(x)]b−1,

where f (x) = F ′(x). Several other distributionsD[0, 1] with range the unit interval are available forgenerating the model-risk-adjusted quantile distribu-tion; see Zografos and Balakrishnan(60) for example.Hence, in the most general terms the model-risk-adjusted VaR is a random variable with distribution:

Q(α|F, F) = F−1(α), α ∼ D[0, 1]. (6)

The mean E[Q(α|F, F)] of Q(α|F, F) quantifiesany systematic bias in the quantile estimates: for ex-ample, if the MED has heavier tails than the modelthen extreme quantiles qF

α will be biased: if α is closeto zero then E[Q(α|F, F)] > qF

α and if α is close toone then E[Q(α|F, F)] < qF

α . This bias can be re-moved by adding the difference qF

α − E[Q(α|F, F)]to the model’s α quantile qF

α so that the bias-adjustedquantile has expectation qF

α .The bias-adjusted α quantile estimate could

still be far away from the maximum entropy α


quantile: the more dispersed the distribution ofQ(α|F, F), the greater the potential for qF

α to de-viate from qF

α . Because financial regulators requireVaR estimates to be conservative, we adjust for theinefficiency of the VaR model by introducing an un-certainty buffer to the bias-adjusted α quantile byadding a quantity equal to the difference betweenthe mean of Q(α|F, F) and G−1

F (y), the y quantile ofQ(α|F, F), to the bias-adjusted α quantile estimate.This way, we become (1 − y)% confident that themodel-risk-adjusted α quantile is no less than qF

α .Finally, our point estimate for the model-risk-

adjusted α quantile becomes:

qFα +{qF

α −E[Q(α|F, F)]}+{E[Q(α|F, F)]−G−1F (y)}

= qFα + qF

α − G−1F (y), (7)

where {qFα − E[Q(α|F, F)]} is the “bias adjustment”

and {E[Q(α|F, F)] − G−1F (y)} is the “uncertainty

buffer.”The total model-risk adjustment to the quantile

estimate is thus qFα − G−1

F (y), and the computationof E[Q(α|F, F)] could be circumvented if the de-composition into bias and uncertainty componentsis not required. The confidence level 1 − y reflectsa penalty for model risk which could be set by theregulator. When X denotes daily P&L and α is small(e.g., 1%), typically all three terms on the right handside of (7) will be negative. But the α% daily VaRis −qF

α , so the model-risk-adjusted VaR estimate be-comes −qF

α − qFα + G−1

F (y). The add-on to the dailyVaR estimate, G−1

F (y) − qFα , will be positive unless

VaR estimates are typically much greater than thebenchmark VaR. In that case there should be a neg-ative bias adjustment, and this could be large enoughto outweigh the uncertainty buffer, especially wheny is large, that is, when we require only a low degreeof confidence for the model-risk-adjusted VaR to ex-ceed the benchmark VaR.

4. NUMERICAL EXAMPLE

We now describe an experiment in which a port-folio’s returns are simulated based on a known datageneration process. This allows us to control the de-gree of VaR model risk and to demonstrate thatour framework yields intuitive and sensible resultsfor the bias and inefficiency adjustments describedearlier.

Recalling that the popular and flexible class ofGARCH models was advocated by Berkowitz and

O’Brien(54) for top–down VaR estimation we as-sume that our conditional MED for the returns Xt

at time t is N (0, σ 2t ), where σ 2

t follows an asymmetricGARCH process. The model falls into the categoryof maximum entropy ARCH models introduced byPark and Bera(50), where the conditional distributionis normal. Thus it has only two constraints, on theconditional mean and variance.

First the return xt from time t to t + 1 and itsvariance σ 2

t are simulated using:

σ 2t = ω + α(xt−1 − λ)2 + βσ 2

t−1, xt |It ∼ N (0, σ 2t ),(8)

where ω > 0, α, β ≥ 0, α + β ≤ 1 and It = (xt−1,

xt−2, . . .).13 For the simulated returns the parametersof (8) are assumed to be

ω = 1.5 × 10−6, α = 0.04, λ = 0.005, β = 0.95, (9)

and so the steady-state annualized volatility of theportfolio return is 25%.14 Then the MED at time t isFt = F(Xt |Kt ), that is, the conditional distribution ofthe return Xt given the state of knowledge Kt , whichcomprises the observed returns It and the knowledgethat Xt |It ∼ N (0, σ 2

t ).At time t , a VaR model provides a forecast

F t = F(Xt |Kt ) where Kt comprises It plus the modelXt |It ∼ N (0, σ 2

t ). We now consider three differentmodels for σ 2

t . The first model has the correct choiceof model but uses incorrect parameter values: insteadof (9) the fitted model is:

σ 2t = ω + α(xt−1 − λ)2 + βσ 2

t−1, (10)

with

ω = 2 × 10−6, α = 0.0515, λ = 0.01, β = 0.92. (11)

The steady-state volatility estimate is therefore cor-rect, but since α > α and β < β the fitted volatilityprocess is more “jumpy” than the assumed variancegeneration process. In other words, compared withσt , σt has a greater reaction but less persistence to in-novations in the returns, and especially to negativereturns since λ > λ.

The other two models are chosen because theyare commonly adopted by financial institutions,having been popularized by the “RiskMetrics”

13 We employ the standard notation α for the GARCH return pa-rameter here; this should not be confused with the notation α

for the quantile of the returns distribution, which is also standardnotation in the VaR model literature.

14 The steady-state variance is σ 2 = (ω + αλ2)/(1 − α − β) and forthe annualization we have assumed returns are daily, and thatthere are 250 business days per year.


Table I. Sample Statistics for Quantile Probabilities. The meanof α and the RMSE between α and α. The closer α is to α and thesmaller the RMSE, the less model risk there is in the VaR model.

α AGARCH EWMA Regulatory

0.10% Mean 0.11% 0.16% 0.23%RMSE 0.07% 0.14% 0.37%

1% Mean 1.03% 1.25% 1.34%RMSE 0.42% 0.64% 1.22%

5% Mean 4.97% 5.44% 5.27%RMSE 1.03% 1.31% 2.66%

methodology introduced by JP Morgan in the mid1990s—see RiskMetrics(7). The second model uses asimplified version of (8) with:

ω = λ = 0, α = 0.06, β = 0.94. (12)

This is the RiskMetrics exponentially weightedmoving average (EWMA) estimator in which asteady-state volatility is not defined. The thirdmodel is the RiskMetrics “Regulatory” estimator inwhich:

α = λ = β = 0, ω = 1250

250∑i=1

x2t−i . (13)

A time series of 10,000 returns {xt }10,000t=1 is simu-

lated from the “true” model (8) with parameters (9).Then, for each of the three models defined above weuse this time series to (a) estimate the daily VaR,which when expressed as a percentage of the port-folio value is given by -�−1(α)σt , and (b) computethe probability αt associated with this quantile underthe simulated returns distribution Ft = F(Xt |Kt ). Be-cause �−1(αt )σt = �−1(α)σt , this is given by

αt = �

[�−1(α)

σt

σt

]. (14)

Now for each VaR model we use the simulateddistribution to estimate α at every time point, using(14). For α = 0.1%, 1%, and 5%, Table I reports themean of α and the RMSE between α and α. Thecloser α is to α and the smaller the RMSE, the lessmodel risk there is in the VaR model. The Regula-tory model yields an α with the highest RMSE, forevery α, so this has the greatest degree of model risk.The AGARCH model, which we already know hasthe least model risk of the three, produces a distribu-tion for α that has mean closest to the true α and thesmallest RMSE. These observations are supportedby Fig. 1, which depicts the empirical distributionof α and Fig. 2, which shows the empirical densities

Fig. 1. Density of quantile probabilities. Empirical distribution ofαt derived from (14) with α = 1%, based on 10,000 daily returnssimulated using (8) with parameters (9).

Fig. 2. Density of model-risk-adjusted daily VaR. Empirical den-sities of the model-risk-adjusted VaR estimates F−1(α) with α =1%, based on the 10,000 observations on α whose density is shownin Figure 1.

of the model-risk-adjusted VaR estimates F−1(α),taking α = 1% for illustration. Here and henceforthVaR is stated as a percentage of the portfolio value,multiplied by 100.

A point estimate for model-risk-adjusted VaR(RaVaR, for short) is computed using (7). Becausewe have a conditional MED, the benchmark VaR(BVaR, for short) depends on the time it is mea-sured, and so does the RaVaR. For illustration, weselect a point when the simulated volatility is at itssteady-state value of 25%—so the BVaR is 4.886,3.678, and 2.601 at the 0.1%, 1%, and 5% levels,respectively. Drawing at random from the pointswhen the simulated volatility was 25%, we obtainAGARCH, EWMA, and Regulatory volatility fore-casts of 27.02%, 23.94%, and 28.19%, respectively.These volatilities determine the VaR estimates thatwe shall now adjust for model risk.


Table II. Components of the Model-Risk Adjustment. The biasand the 95% uncertainty buffer, for different levels of α, derivedfrom the mean and 5% quantile of the empirical distribution of

Q(α|F, F). the bias is the difference between the benchmarkVaR (which is 4.886, 3.678, and 2.601 at the 0.1%, 1%, and 5%levels) and the mean. The uncertainty buffer is the difference

between the mean and the 5% quantile. UB means “uncertaintybuffer” and Q is the quantile.

α AGARCH EWMA Regulatory

0.1% Mean 4.919 4.793 4.9615% Q 4.447 4.177 3.912Bias −0.033 0.093 −0.075UB 0.472 0.615 1.049

1% Mean 3.703 3.608 3.7355% Q 3.348 3.145 2.945Bias −0.025 0.070 −0.056UB 0.355 0.463 0.789

5% Mean 2.618 2.551 2.6415% Q 2.366 2.224 2.082Bias −0.017 0.050 −0.040UB 0.252 0.327 0.559

Table II summarizes the bias and the uncertaintybuffer, for different levels of α, based on the em-pirical distribution of Q(α|F, F).15 It reveals a gen-eral tendency for the EWMA model to slightly un-derestimate VaR and the other models to slightlyoverestimate VaR. Yet the bias is relatively small,since all models assume the same normal form asthe MED and the only difference between them istheir volatility forecast. Although the bias tends toincrease as α decreases it is not significant for anymodel.16 Beneath the bias we report the 5% quantileof the model-risk-adjusted VaR distributions, sincewe shall first compute the RaVaR so that it is no lessthan the BVaR with 95% confidence.

Following the framework introduced in the pre-vious section we now define:

RaVaR(y) = VaR + {BVaR − E[Q(α|F, F)]}+ {

E[Q(α|F, F)] − G−1F (y)

}where {BVaR − E[Q(α|F, F)]} is the bias adjust-ment and {E[Q(α|F, F)] − G−1

F (y)} is the uncertaintybuffer.

15 Similar results based on the fitted distributions are not reportedfor brevity.

16 Standard errors of Q(α|F, F) are not reported, for brevity. Theyrange between 0.157 for the AGARCH at 5% to 0.891 for theRegulatory model at 0.1%, and are directly proportional to thedegree of model risk just like the standard errors on the quantileprobabilities given in Table I.

Table III. Computation of 95% RaVaR. The volatility σt

determines the VaR estimates for α = 0.1%, 1%, and 5%,respectively, as �−1(α) σt . Adding the bias shown in Table II

gives the bias-adjusted VaR. Adding the uncertainty buffer givenin Table 2 to the bias-adjusted VaR yields the

model-risk-adjusted VaR estimates (RaVaR) shown in the thirdrow of each cell. BA means “bias-adjusted.”

AGARCH EWMA Regulatory

α Volatility 27.02% 23.94% 28.19%0.10% VaR 5.277 4.678 5.509

BA VaR 5.244 4.772 5.434RaVaR 5.716 5.387 6.483

1% VaR 3.972 3.522 4.147BA VaR 3.948 3.592 4.091RaVaR 4.303 4.055 4.880

5% VaR 2.809 2.490 2.932BA VaR 2.791 2.540 2.892RaVaR 3.043 2.867 3.451

Table III sets out the RaVaR computation fory = 5%. The model’s volatility forecasts are in thefirst row and the corresponding VaR estimates are inthe first row of each cell, for α = 0.1%, 1%, and 5%,respectively. The (small) bias is corrected by addingthe bias from Table II to each VaR estimate. Themain source of model risk here concerns the poten-tial for a large (positive or negative) errors in thequantile probabilities, that is, the dispersion of thedensities in Fig. 2. To adjust for this we add to thebias-adjusted VaR the uncertainty buffer given in Ta-ble II. This gives the RaVaR estimates shown in thethird row of each cell.

Since risk capital is a multiple of VaR, the per-centage increase resulting from replacing VaR byRaVaR(y) is:

% risk capital increase = BVaR − G−1F (y)

VaR. (15)

The penalty (15) for model risk depends on α, exceptin the case that both the MED and VaR model arenormal, and on the confidence level (1 − y)%. Ta-ble IV reports the percentage increase in risk capi-tal due to model risk when RaVaR is no less thanthe BVaR with (1 − y)% confidence. We considery = 5%, 15%, and 25%, with smaller values of y cor-responding to a stronger condition on the model-riskadjustment. We also set α = 1% because risk capitalis based on the VaR at this level of significance underthe Basel Accords. We also take the opportunity hereto consider two further scenarios, in order to verifythe robustness of our qualitative conclusions.

The first row of each section of Table IV reportsthe volatilities estimated by each VaR model at a


Table IV. Percentage Increase in Risk Capital from Model-RiskAdjustment of VaR. The percentage increase from VaR toRaVaR based on three scenarios for each model’s volatility

estimate at time t . In each case the benchmark model’sconditional volatility was 25%.

AGARCH EWMA Regulatory

Scenario 1 27.02% 23.94% 28.19%95% 8.40% 14.46% 21.29%85% 5.23% 9.45% 13.93%75% 3.08% 6.53% 9.14%Scenario 2 28.32% 27.34% 22.40%95% 8.02% 12.66% 26.79%85% 4.98% 8.27% 17.53%75% 2.94% 5.71% 11.50%Scenario 3 23.18% 26.34% 28.66%95% 9.80% 13.14% 20.94%85% 6.09% 8.59% 13.70%75% 3.59% 5.93% 8.99%

point in time when the benchmark model has volatil-ity 25%. Thus for scenario 1, upon which the resultshave been based up to now, we have the volatili-ties 27.02%, 23.94%, and 28.19%, respectively. Forscenario 2 the three volatilities are 28.32%, 27.34%,and 22.40%, that is, the AGARCH and EWMAmodels over-estimate and the Regulatory modelunder-estimates the benchmark model’s volatility.For scenario 3 the AGARCH model slightly under-estimates the benchmark’s volatility and the othertwo models over-estimate it.

The three rows in each section of the Table IVgive the percentage increase in risk capital thatwould be required were the regulator to choose95%, 85% or 75% confidence levels for the RaVaR.Clearly, for each model and each scenario, theadd-on for VaR model risk increases with the degreeof confidence that the regulator requires for theRaVaR to be at least as great as the BVaR. Atthe 95% confidence level, a comparison of the firstrow of each section of the table shows that riskcapital would be increased by roughly 8–10% whenbased on the AGARCH model, whereas it wouldbe increased by about 13–14.5% under the EWMAmodel and by roughly 21–27% under the Regulatorymodel. The same ordering of the RaVaRs applies toeach scenario, and at every confidence level. That is,the model-risk adjustment results in an increase inrisk capital that is positively related to the degree ofmodel risk, as it should be.

Finally, comparison between the three scenariosshows that the add-on will be greater on days when

the model under-estimates the VaR than it is on dayswhen it over-estimates VaR, relative to the bench-mark. Yet even when the model VaR is greater thanthe benchmark VaR the add-on is still positive. Thisis because the uncertainty buffer remains large rela-tive to the bias adjustment, even at the 75% level ofconfidence. However, if regulators were to require alower confidence for the uncertainty buffer, such asonly 50% in this example, then it could happen thatthe model-risk add-on becomes negative.

5. EMPIRICAL ILLUSTRATION

How could the authority responsible for modelrisk, such as a bank’s local regulator or its ChiefRisk Officer, implement the proposed adjustmentfor model risk in practice? The required inputs toa model-risk-adjusted VaR calculation are two dailytime series that the bank will have already beenrecording to comply with Basel regulations: one se-ries is the aggregate daily trading P&L and the otheris the aggregated 1% daily VaR estimates corre-sponding to this trading activity. From the regionaloffice of a large international banking corporation wehave obtained data on aggregate daily P&L and thecorresponding aggregate VaR for each day, the VaRbeing computed in a bottom–up framework based onstandard (un-filtered) historical simulation. The dataspan the period 3 September 2003 to 18 March 2009,thus including the banking crisis during the last quar-ter of 2008. In this section the bank’s daily VaR willbe compared with a top–down VaR estimate basedon a benchmark VaR model tuned to the aggregatedaily trading P&L, and a model-risk-adjusted VaRwill be derived for each day between 3 September2007 and 18 March 2009.

When the data are not i.i.d. the benchmarkshould be a conditional MED rather than an uncon-ditional MED. To illustrate this we compute the timeseries of 1% quantile estimates based on alternativebenchmarks. First we employ the Student t distribu-tion, which maximizes the Shannon entropy subjectto the moment constraint17.

E[log(ν2 + (X/λ)2)] = log(ν2) + ψ

(1 + ν2

2

)

−ψ

(ν2

2

).

17 Here ψ(z) denotes the digamma function, λ is a scale parameterand ν the corresponding shape parameter. See Park and Bera[49],Table I for the moment condition.


Second, we consider the AGARCH process (8)which has a normal conditional distribution for theerrors. We also considered taking the generalized er-ror distribution (GED), introduced by Nelson(61), asan unconditional MED benchmark. The GED hasthe probability density (−∞ < x < ∞)

f (x; λ, ν) = ν−1/ν

2�(1 + 1/ν)λexp

(−1

ν|x/λ|ν

),

and maximizes the Shannon entropy subject to themoment constraint E[ν−1|X/λ|ν] = ν−1/ν

2�(1+1/ν) . This ismore flexible than the (always heavy-tailed) Studentt because when ν < 2 (ν > 2) the GED has heavier(lighter) tails than the normal distribution. We alsoconsidered using a Student t conditional MED withthe AGARCH process, and a symmetric GARCHprocess, where λ = 0 in (8), with both Student t andnormal conditional distributions. However, the un-conditional GED produced results similar to (andjust as bad as) the unconditional Student t . Alsoall four conditional MEDs produced quite similarresults. Our choice of the AGARCH process withnormal errors was based on the successful resultsof the conditional and unconditional coverage teststhat are commonly applied to test for VaR modelspecification—see Christoffersen(15). For reasons ofspace, none of these results are reported but they areavailable from the authors on request.

We estimate the parameters of the two selectedbenchmark models using a “rolling window” frame-work that is standard practice for VaR estimation.Each sample contains n consecutive observationson the bank’s aggregate daily P&L, and a sam-ple is rolled forward one day at a time, each timere-estimating the model parameters. Fig. 3 com-pares the 1% quantile of the Student t distributionwith the 1% quantile of the normal AGARCH pro-cess on the last day of each rolling window. Alsoshown is the bank’s aggregate P&L for the daycorresponding to the quantile estimate, between 3September 2007 and 18 March 2008. The effect of thebanking crisis is evidenced by the increase in volatil-ity of daily P&L which began with the shock collapseof Lehmann Brothers in mid September 2008. Beforethis time the 1% quantiles of the unconditional Stu-dent t distribution were very conservative predictorsof daily losses, because the rolling windows includedthe commodity crisis of 2006. Yet at the time of thebanking crisis the Student t model clearly underesti-mated the losses that were being experienced. Evenworse, from the bank’s point of view, the Student t

model vastly over-estimated the losses made duringthe aftermath of the crisis in early 2009 and wouldhave led to crippling levels of risk capital reserves.Even though we used n = 200 for fitting the uncondi-tional Student t distribution and a much larger sam-ple, with n = 800, for fitting the normal AGARCHprocess it is clear that the GARCH process capturesthe strong volatility clustering of daily P&L far betterthan the unconditional MED. True, the AGARCHprocess often just misses a large unexpected loss,but because it has the flexibility to respond the verynext day, the AGARCH process rapidly adapts tochanging market conditions just as a VaR modelshould.

In an extensive study of the aggregate P&Lof several large commercial banks, Berkowitz andO’Brien(54) found that GARCH models estimatedon aggregate P&L are far more accurate predictorsof aggregate losses than the bottom–up VaR figuresthat most banks use for regulatory capital calcula-tions. Fig. 3 verifies this finding by also depicting the1% daily VaR reported by the bank, multiplied by−1 since it is losses rather than profits that VaR issupposed to cover. This time series has many fea-tures in common with the 1% quantiles derived fromthe Student t distribution. The substantial losses ofup to $80 m per day during the last quarter of 2008were not predicted by the bank’s VaR estimates, yetfollowing the banking crisis the bank’s VaR was fartoo conservative. We conclude, like Berkowitz andO’Brien(54), that unconditional approaches are muchless risk sensitive than GARCH models and for thisreason we choose the normal AGARCH rather thanthe Student t as our benchmark for model risk assess-ment below.

Fig. 4 again depicts −1× the bank’s aggregatedaily VaR, denoted −VaRt in the formula below.Now our bias adjustment is computed daily usingan empirical model-risk-adjusted VaR distributionbased on the observations in each rolling window.Under the normal AGARCH benchmark, for a sam-ple starting at T and ending at T + n, the daily P&Ldistribution at time t , with T ≤ t ≤ T + n is N(0, σ 2

t ),where σt is the time-varying standard deviation ofthe AGARCH model. Following (3) we set αt =�

(− σ−1t VaRt

)for each day in the window and then

we use the empirical distribution of αt for T ≤ t ≤T + n to generate the model-risk-adjusted VaR dis-tribution (6). Then, following (7), the bias adjust-ment at T + n is the difference between the mean ofthe model-risk-adjusted quantile distribution and thebenchmark VaR at T + n.


-100

-80

-60

-40

-20

0

20

40

60

80

Daily P&L Student t Normal AGARCH Daily VaR

Fig. 3. Daily P&L, daily VaR, and two potential benchmark VaRs. The bank’s daily P&L is depicted by the grey line and it’s “bottom-up”daily VaR estimates by the black line. The dotted and dashed lines are the Student t (unconditional MED) benchmark VaR and the normalAGARCH (conditional MED) benchmark VaR.

Since the bank’s aggregate VaR is very conser-vative at the beginning of the period but not largeenough during the crisis, in Fig. 4a positive bias re-duces the VaR prior to the crisis but during the cri-sis a negative bias increases the VaR. Having ap-plied the bias adjustment we then set y = 25% in(7) to derive the uncertainty buffer correspondingto a 75% confidence that the RaVaR will be noless than the BVaR. This is the difference betweenthe mean and the 25% quantile of the model-risk-adjusted VaR distribution. Adding this uncertaintybuffer to the bias-adjusted VaR we obtain the 75%RaVaR depicted in Fig. 4 which is given by (7). Thisis more variable than the bank’s original aggregateVaR, but risk capital is based on an average VaR fig-ure over the last 60 days (or the previous VaR, if thisis greater) so the adjustment need not induce exces-sive variation in risk capital, which would be difficultto budget.

6. APPLICATION TO NONFINANCIALPROBLEMS

Quantile-based risk assessment has become stan-dard practice in a wide variety of nonfinancial dis-ciplines, especially in environmental risk assessmentand in statistical process control. For instance, appli-cations to hydrology are studied by Arsenault andAshkar(62) and Chebana and Ouarda(63), and otherenvironmental applications of quantile risk assess-ments include climate change (Katz et al.(64) andDiodato and Bellocchi(65)) wind power(66), and nu-clear power(67). In statistical process control, quan-tiles are used for computing capability indices(68),for measuring efficiency(69) and for reliability analy-sis(70).

In these contexts the uncertainty surround-ing quantile-based risk assessments has been thesubject of many papers(36,71,72,73,74,75). Both model


-100

-80

-60

-40

-20

0

20

40

60

80

Daily VaR Bias 75% RaVaR

Fig. 4. Aggregate VaR, bias adjustment, and 75% RaVaR. The bank’s daily VaR estimates are repeated (black line) and compared with thebias-adjustment (grey line) and the final model-risk-adjusted VaR at the 75% confidence level (dotted line) based on the normal AGARCHbenchmark model.

choice and parameter uncertainty has been con-sidered. For instance, Vermaat and Steerneman(76)

discuss modified quantiles based on extreme valuedistributions in reliability analysis, and Sveinssonet al.(77) examine the errors induced by using a sam-ple limited to a single site in a regional frequencyanalysis.

As in banking, regulations can be a key driverfor the accurate assessment of environmental riskssuch as radiation from nuclear power plants. Never-theless, health or safety regulations are unlikely toextend as far as requirements for regular monitor-ing and reporting of quantile risks in the foreseeablefuture. The main concern about the uncertainty sur-rounding quantile risk assessment is more likely tocome from senior management, in recognition thatinaccurate risk assessment could jeopardize the rep-utation of the firm, profits to shareholders and/or thesafety of the public. The question then arises: if it is a

senior manager’s knowledge that specifies the bench-mark distribution for model risk assessment, whyshould this benchmark distribution not be utilized inpractice?

As exemplified by the work of Sveinsson et al.(77),the time and expense of utilizing a complete sam-ple of data may not be feasible except on a few oc-casions where a more detailed risk analysis is per-formed, possibly by an external consultant. In thiscase the most significant source of model risk in reg-ular risk assessments would stem from parameter un-certainty. Model choice might also be a source ofrisk when realistic model assumptions would leadto systems that are too costly and time-consumingto employ on a daily basis. For instance, Merriket al.(74) point out that the use of Bayesian sim-ulation for modeling large and complex maritimerisk systems should be considered state-of-the-art,rather than standard practice. Also in reliability


modeling, individual risk assessments for variouscomponents are typically aggregated to derive thetotal risk for the system. A full account of compo-nent default codependencies may require a lengthyscenario analyses based on a complex model (e.g.,multivariate copulas with nonstandard marginal dis-tributions). This type of risk assessment might not befeasible every day, but if it could be performed on anoccasional basis then it could be used as a benchmarkfor adjusting everyday quantile estimates for modelrisk.

Generalizations and extensions to higherdimensions of the benchmark model could be im-plemented. A multivariate parametric MED for thebenchmark model can be obtained using similararguments to those used in the univariate case. Inan engineering context, Kapur(78) have consideredseveral classes of multivariate MED. Zografos(79)

characterized Pearson’s type II and VII multivari-ate distributions, Aulogiaris and Zografos(80) thesymmetric Kotz and Burr multivariate families andBhattacharya(81) the class of multivariate Liouvilledistributions. Closed expressions for entropies inseveral multivariate distributions have been pro-vided by Ahmed and Gokhale(82) and Zografos andNadarajah(83).

A major difference between financial and nonfi-nancial risk assessment is the availability of data. Forinstance, in the example described in the previoussection the empirical distributions for model-risk-adjusted quantiles were derived from several years ofregular output from the benchmark model. Clearly,the ability to generate the adjusted quantile distri-bution from a parametric distribution for α, suchas the beta distribution (5), opens the methodologyto applications where relatively few observations onthe benchmark quantile are available, but there areenough to estimate the parameters of a distributionfor α.

7. SUMMARY

This article concerns the model risk of quantile-based risk assessments, with a focus on the risk ofproducing inaccurate VaR estimates because of aninappropriate choice of VaR model and/or inaccu-racy in the VaR model parameter estimates. We de-velop a statistical methodology that provides a practi-cal solution to the problem of quantifying the regula-tory capital that should be set aside to cover this typeof model risk, under the July 2009 Basel II propos-als. We argue that there is no better choice of model

risk benchmark than a maximum entropy distribu-tion since, by definition, this embodies the entiretyof information and beliefs, no more and no less. Inthe context of the model risk capital charge under theBasel II Accord the benchmark could be specified bythe local regulator; more generally it should be spec-ified by any authority that is concerned with modelrisk, such as the Chief Risk Officer. Then VaR modelrisk is assessed using a top–down approach to com-pute the benchmark VaR from the bank’s total dailyP&L, and comparing this with the bank’s aggregatedaily VaR, which is typically derived using a compu-tationally intensive bottom–up approach that neces-sitates many approximations and simplifications.

The main ideas are as follows: in the presenceof model risk an α quantile is at a different quan-tile of the benchmark model, and has an associatedtail probability under the benchmark that is stochas-tic. Thus, the model-risk-adjusted quantile becomes agenerated random variable and its distribution quan-tifies the bias and uncertainty due to model risk.A significant bias arises if the aggregate VaR esti-mates tend to be consistently above or below thebenchmark VaR, and this is reflected in a signifi-cant difference between the mean of the model-risk-adjusted VaR distribution and the benchmark VaR.Even when the bank’s VaR estimates have an in-significant bias, an adjustment for uncertainty is re-quired because the difference between the bank’sVaR and the benchmark VaR could vary consider-ably over time. The bias and uncertainty in the VaRmodel, relative to the benchmark, determine a riskcapital adjustment for model risk whose size will alsodepend on the confidence level regulators require forthe adjusted risk capital to be no less than the riskcapital based on the benchmark model.

Our framework was validated and illustrated bya numerical example which considers three commonVaR models in a simulation experiment where thedegree of model risk has been controlled. A furtherempirical example describes how the model-risk ad-justment could be implemented in practice given onlytwo time series, on the bank’s aggregate VaR and itsaggregate daily P&L, which are in any case reporteddaily under banking regulations.

Further research would be useful on backtestingthe model-risk-adjusted estimates relative to com-monly used VaR models, such as the RiskMetricsmodels considered in this article. Where VaR mod-els are failing regulatory backtests and thus be-ing heavily penalized or even disallowed, the top–down model-risk-adjustment advocated in this article


would be very much more cost effective than imple-menting a new or substantially modified bottom–upVaR system.

There is potential for extending the methodol-ogy to the quantile-based metrics that are commonlyused to assess nonfinancial risks in hydrology, cli-mate change, statistical process control and reliabilityanalysis. In the case that relatively few observationson the model and benchmark quantiles are avail-able, the approach should include a parameterizationthe model-risk-adjusted quantile distribution, for in-stance as a beta-generated distribution.

ACKNOWLEDGMENTS

The authors would like to thank to the as-sociate editor and two anonymous reviewers formany constructive comments that improved the orig-inal version. The second author thanks Ministeriode Economıa y Competitividad, Spain (ECO2010-15455) for partial support.

REFERENCES

1. Basel Committee on Banking Supervision. Amendment tothe capital accord to incorporate market risks. Bank for In-ternational Settlements, Basel, 1996.

2. Basel Committee on Banking Supervision. International con-vergence of capital measurement and capital standards: A re-vised framework. Bank for International Settlements, Basel,2006.

3. Basel Committee on Banking Supervision. Revisions to theBasel II market risk framework. Bank for International Set-tlements, Basel, 2009.

4. Hendricks D. Evaluation of Value-at-Risk models using his-torical data. FRBNY Economic Policy Review, 1996:39–69.

5. Perignon C, Smith D. The level and quality of Value-at-Riskdisclosure by commercial banks. Journal of Banking and Fi-nance, 2010; 34(2):362–377.

6. Pritsker M. The hidden dangers of historical simulation. Jour-nal of Banking and Finance, 2006; 30:561–582.

7. RiskMetrics. Technical Document. Available from the Risk-Metrics website, 1997.

8. Hull J, White A. Value at risk when daily changes in marketvariables are not normally distributed. Journal of Derivatives,1998; 5(3):9–19.

9. Mittnik S, Paolella M. Conditional density and Value-at-Risk:Prediction of Asian currency exchange rates. Journal of Fore-casting, 2000; 19:313–333.

10. Venter J, de Jongh P. Risk estimation using the normal in-verse Gaussian distribution. Journal of Risk, 2002; 4:1–23.

11. Angelidis T, Benos A, Degiannakis S. The use of GARCHmodels in VaR estimation. Statistical Methodology, 2004;1(2):105–128.

12. Hartz C, Mittnik S, Paolella M. Accurate Value-at-Risk fore-casting based on the normal-GARCH model. ComputationalStatistics and Data Analysis, 2006; 51(4):2295–2312.

13. Kuan C-M, Yeh J-H, Hsu Y-C. Assessing Value-at-Riskwith CARE: The conditional autoregressive expectile mod-els. Journal of Econometrics, 2009; 150(2):261–270.

14. Kupiec P. Techniques for verifying the accuracy of riskmeasurement models. Journal of Derivatives, 1995; 3(2):73–84.

15. Christoffersen P. Evaluating interval forecasts. InternationalEconomic Review, 1998; 39:841–862.

16. Berkowitz J, Christoffersen P, Pelletier D. Evaluating VaRmodels with desk-level data. Management Science, 2011; 57:2213–2227.

17. Engle R, Manganelli S. CAViaR: Conditional autoregressiveValue at Risk by regression quantile. Journal of Business andEconomic Statistics, 2004; 22:367–381.

18. Turner L. The Turner review: A regulatory response to theglobal banking crisis. Financial Services Authority, London,2009.

19. Taleb N. The black swan: The Impact of The Highly Improb-able. London: Penguin, 2007.

20. Beder T. VAR: Seductive but dangerous. Financial AnalystsJournal, 1995; 51(5):12–24.

21. Bernardo JM, Smith AFM. Bayesian Theory. Chichester, UK:Wiley, 1994.

22. O’Hagan A. Kendall’s Advanced Theory of Statistics. Vol 2B:Bayesian Inference. London: Edward Arnold, 1994.

23. Modarres R, Nayak TK, Gastwirth JL. Estimation of upperquantiles model and parameter uncertainty. ComputationalStatistics and Data Analysis, 2002; 39:529–554.

24. Giorgi G, Narduzzi C. Uncertainty of quantile estimates inthe measurement of self-similar processes. In Proceedingsof the International Workshop on Advanced Methods forUncertainty Estimation Measurements, AMUEM, Sardagna,Trento, Italy, 2008.

25. Figlewski S. Estimation error in the assessment of financialrisk exposure. SSRN-id424363, version of June 29, 2003.

26. Derman E. Model risk. Risk, 1996; 9(5):139–145.27. Simons K. Model error: Evaluation of various finance models.

New England Economic Review, 1997, 17–28.28. Crouhy M, Galai D, Mark R. Model risk. Journal of Financial

Engineering, 1998; 7: 267–288.29. Green T, Figlewski S. Market risk and model risk for a finan-

cial institution writing options. Journal of Finance, 1999; 54:1465–1499.

30. Kato T, Yoshiba T. Model risk and its control. Monetary andEconomic Studies, 2000, 129–157.

31. Rebonato R. Managing Model Risk. Vol. 2 in Alexander C.(ed.). Mastering Risk. UK: Pearson, 2001.

32. Jorion P. Measuring the risk in value at risk. Financial Ana-lysts Journal, 1996; 52:47–56.

33. Talay D, Zheng Z. Worst case model risk management. Fi-nance and Stochastics, 2002; 6:517–537.

34. Kerkhof J, Melenberg B, Schumacher H. Model risk and cap-ital reserves. Journal of Banking and Finance, 2010; 34:267–279.

35. Reiss R, Thomas M. Statistical Analysis of Extreme Val-ues with Applications to Insurance, Finance, Hydrology andOther Fields. Basel: Birkhauser Verlag, 1997.

36. Cairns A. A discussion of parameter and model uncertainty ininsurance. Insurance: Mathematics and Economics, 2000; 27:313–330.

37. Matthys G, Delafosse E, Guillou A, Beirlant J. Estimatingcatastrophic quantile levels for heavy-tailed distributions. In-surance: Mathematics and Economics, 2004; 34: 517–537.

38. Dowd K, Blake D. After VaR: The theory, estimation and in-surance applications of quantile-based risk measures. Journalof Risk and Insurance, 2009; 73(2): 193–229.

39. Cont R. Model uncertainty and its impact on the pricing ofderivative instruments. Mathematical Finance, 2006; 16(3):519–547.

40. Hull J, Suo W. A methodology for assessing model risk and itsapplication to the implied volatility function model. Journalof Financial and Quantitative Analysis, 2002; 37(2):297–318.


41. Branger N, Schlag C. Model risk: A conceptual framework forrisk measurement and hedging. Working Paper, EFMA BaselMeetings. Available from SSRN, 2004.

42. Ebrahimi N, Maasoumi E, Soofi E. Ordering univariate dis-tributions by entropy and variance. Journal of Econometrics,1999; 90:317–336.

43. Shannon, C. A mathematical theory of communication. BellSystem Technical Journal, 1948; 27 (July): 379–423 and (Oct):623–656.

44. Zellner A. Maximal data information prior distributions. In:Aykac A. and Brumat C. (eds). New methods in the Ap-plications of Bayesian Methods. Amsterdam: North-Holland,1977.

45. Jaynes E. Papers on Probability, Statistics and StatisticalPhysics. In: Rosenkrantz R.D. (ed.), Dordrecht: Reidel Pub-lishing Company, 1983.

46. Rockinger M, Jondeau E. Entropy densities with an appli-cation to autoregressive conditional skewness and kurtosis.Journal of Econometrics, 2002; 106:119–142.

47. Wu X. Calculation of maximum entropy densities with appli-cation to income distribution. Journal of Econometrics, 2003;115(2):347–354.

48. Chan F. Modelling time-varying higher moments with maxi-mum entropy density. Mathematics and Computers in Simu-lation, 2009; 79(9):2767–2778.

49. Chan F. Forecasting value-at-risk using maximum entropy. Inproceedings of the 18th World IMACS / MODSIM Congress,Cairns, Australia, 2009.

50. Park S, Bera A. Maximum entropy autoregressive condi-tional heteroskedasticity model. Journal of Econometrics,2009; 150(2):219–230.

51. Lee M-C, Su J-B, Liu H-C. Value-at-risk in US stock indiceswith skewed generalized error distribution. Applied FinancialEconomics Letters, 2008; 4(6):425–431.

52. Mandelbrot B. The variation of certain speculative prices.Journal of Business, 1963; 36:394–419.

53. Bollerslev T. A conditionally heteroskedastic time seriesmodel for speculative prices and rates of return. Review ofEconomics and Statistics, 1987; 69(3):542–547.

54. Berkowitz J, O’Brien J. How accurate are Value-at-Riskmodels at commercial banks? Journal of Finance, 2002;55:1093–1111.

55. Barone-Adesi G, Bourgoin F, Giannopoulos K. Don’t lookback. Risk, 1998; 11(8):100–103.

56. Hull J, White A. Incorporating volatility updating into the his-torical simulation method for Value-at-Risk. Journal of Risk,1998; 1:5–19.

57. Alexander C, Sheedy E. Developing a stress testing frame-work based on market risk models. Journal of Banking andFinance, 2008; 32(10):2220–2236.

58. Eugene N, Lee C, Famoye F. The beta normal distributionand its applications. Communications in Statistics, Theoryand Methods, 2002; 31:497–512.

59. Jones M. Families of distributions arising from distributionsof order statistics. Test, 2004; 13(1):1–43.

60. Zografos K, Balakrishnan, N. On families of beta- and gen-eralized gamma-generated distributions and associated infer-ence. Statistical Methodology, 2009; 6(4):344–362.

61. Nelson DB. Conditional heteroskedasticity in asset returns: Anew approach. Econometrica, 1991; 59:347–370.

62. Arsenault M, Ashkar F. Regional flood quantile estimationunder linear transformation of the data. Water Resources Re-search, 2000; 36(6):1605–1610.

63. Chebana F, Ouarda T. Multivariate quantiles in hydro-logical frequency analysis. Environmetrics, 2011; 22(1):63–78.

64. Katz R, Parlange M, Naveau P. Statistics of extremes inhydrology. Advances in Water Resources, 2002; 25:1287–1304.

65. Diodato N, Bellocchi G. Assessing and modelling changesin rainfall erosivity at different climate scales. Earth SurfaceProcesses and Landforms, 2009; 34(7):969–980.

66. Bremnes J. Probabilistic wind power forecasts using lo-cal quantile regression. Wind Energy, 2004; 7(1):47–54.

67. Rumyantsev A. Safety prediction in nuclear power. AtomicEnergy, 2007; 102(2):94–99.

68. Anghel C. Statistical process control methods from the view-point of industrial application. Economic Quality Control,2001; 16(1):49–63.

69. Wheelock D, Wilson P. Non-parametric, unconditional quan-tile estimation for efficiency analysis with an applicationto Federal Reserve check processing operations. Journal ofEconometrics, 200; 145:209–225.

70. Unnikrishnan Nair N, Sankaran P. Quantile-based reliabilityanalysis. Communications in Statistics - Theory and Methods,2009; 38(2):222–232.

71. Mkhandi S, Kachroo R, Guo S. Uncertainty analysis of floodquantile estimates with reference to Tanzania. Journal of Hy-drology, 1996; 185, 317–333.

72. Pinson P, Kariniotakis G. On-line assessment of predictionrisk for wind power production forecasts. Wind Energy, 2004;7(2):119–132.

73. Noon B, Cade B, Flather C. Quantile regression reveals hid-den bias and uncertainty in habitat models. Ecology, 2009;86(3):786–800.

74. Merrick J, Van Dorp R, Dinesh V. Assessing uncertaintyin simulation-based maritime risk assessment. Risk Analysis,2005; 25(3):731–743.

75. Peng L, Qi Y. Confidence regions for high quantiles of a heavytailed distribution. Annals of Statistics, 2006; 34(4):1964–1986.

76. Vermaat M, Does R, Steerneman A. A modified quantile es-timator using extreme-value theory with applications. Eco-nomic Quality Control, 2005; 20(1):31–39.

77. Sveinsson O, Salas J, Boes D. Uncertainty of quan-tile estimators using the population index floodmethod. Water Resources Research, 2003; 39(8), 1206,doi:10.1029/2002WR001594

78. Kapur JN. Maximum Entropy Models in Engineering. NewYork: John Wiley, 1989.

79. Zografos K. On maximum entropy characterization of Pear-son’s type II and VII multivariate distributions. Journal ofMultivariate Analysis, 1999; 71:67–75.

80. Aulogiaris G, Zografos K. A maximum entropy characteriza-tion of symmetric Kotz type and multivariate Burr distribu-tions. Test, 2004; 13:65–83.

81. Bhattacharya B. Maximum entropy characterizations of themultivariate Liouville distributions. Journal of MultivariateAnalysis, 2006; 97:1272–1283.

82. Ahmed NA, Gokhale DV. Entropy expressions and their es-timators for multivariate distributions. IEEE Transactions onInformation Theory, 1989; 35:688–692.

83. Zografos K, Nadarajah S. Expressions for Renyi and Shannonentropies for multivariate distributions. Statistics and Proba-bility Letters, 2005; 71:71–84.

quantile uncertainty and value-at-risk model risk

Documents