model validation for equity...
TRANSCRIPT
Model Validation for
Equity Derivatives
Bernhard Hientzsch
Head of Model, Library, and Tools Development
WFS Model Validation & ApprovalMarket Risk ManagementWells Fargo Securities
Presentation in Course on Managing Model Risk, New York
October 25, 2011
11
Agenda
� Introduction
– Outline
� Modeling and Pricing of Equity Linked Instruments
– Introduction
– Types of Equity Linked Instruments
– Stochastic Models & Pricing Approaches
– Calibration
� Model Validation: Mandates, Interpretations, and Approaches
– Regulatory Mandates and Guidance
– Model Validation: Interpretations and Approaches
� Validating Equity Linked Instrument Models & Case Study
– Testing, Validation, and Benchmarking Environments
– Issues, Reviews, Experiments, and Tests
– Example Case Study
� Conclusion and References
– Conclusion
– References
222
Introduction
33
This Session
� This session is designed to introduce you to
– Some features of equity linked instruments,
– Some modeling and pricing approaches for equity linked
instruments,
– Regulatory mandates and guidance and various approaches for
model risk management and model validation,
– How pricing methods and models for equity linked instruments can
go wrong (and right) and suggest some validation tests, and
– Present a simple case study.
44
About Wells Fargo
Wells Fargo & Company (NYSE: WFC) is a nationwide, diversified,
community-based financial services company with $1.2 trillion in
assets. Founded in 1852 and headquartered in San Francisco,
Wells Fargo provides banking, insurance, investments, mortgage,
and consumer and commercial finance through more than 10,000
banking stores, 12,000 ATMs, the Internet (wellsfargo.com and
wachovia.com), and other distribution channels across North
America and internationally. With 278,000 team members,
Wells Fargo serves one in three households in America.
Wells Fargo & Company was ranked #19 on Fortune’s 2009
rankings of America’s largest corporations. Wells Fargo’s vision is
to satisfy all our customers’ financial needs and help them
succeed financially.
55
About Bernhard Hientzsch
� Now: Director, Head of Model, Library, and Tools Development
(for Model Validation and Approval). Responsible for
development of tools, models, and libraries for benchmarking
and validation, covering eventually all asset classes.
� Previously:
– (A)VP validating equity, hybrid, and credit.
– Consulting (U.S./Germany), e.g., back testing futures strategies,
nuclear medicine, congress planning & organization.
– Postdocs (nonlinear problems and computational electromagnetics;
computational plasma physics – magnetohydrodynamics; SEM &
domain decomposition) and Ph.D. (Mathematics), Courant Institute
at NYU. Took wide variety of courses and TAed for finance.
666
Modeling and Pricing of
Equity Linked
Instruments
77
Pricing and Calibration Process
Pricing According to Calibrated Model
Prices Other Greeks
Downstream Systems or Reports: General Ledger, Account Statements, Risk Systems or Reports, Hedging, Exercise or Trading Strategies, etc.
Contract Data for Derivative
Market Data for Derivative
Market Data
Contract Data
Contract Prices
Calibration
Model Parameters
Adjustment from Calibration to Pricing Environment
Adjusted Parameters
88
Modeling and Pricing of Equity Linked
Instruments
� To sensibly present model validation and model risk tests,
reviews, and experiments for equity linked instruments in the
appropriate context and more detail, useful to quickly review:
– Pricing and calibration process for such instruments in general.
– Some types and features of equity linked instruments.
– Some classes of stochastic models used for such instruments.
– Numerical pricing approaches for such models (PDE, simulation).
– Calibration.
99
Equity Linked Instruments: Overview
� The next few slides present a short and necessarily incomplete
overview of some features of equity linked instruments, such as:
– When can instrument be exercised, if at all?
– Who holds embedded options?
– What rights do the embedded options grant?
– Barrier features.
– Example payouts.
1010
Equity Linked Instruments: Types of Exercise
� European: Rational exercise decision of party already
incorporated into final payout.
� Bermudan: Party can exercise one or several embedded
options at a finite discrete number of intermediate times.
� American: Party can exercise one or several embedded options
at any time in some given time intervals.
� Automatic Exercise: Instrument exercises itself automatically
(i.e., changes itself into a different instrument or payout). E.g.,
autocallable feature cancels instrument and pays out rebate
upon some event, such as, hitting barrier.
1111
Equity Linked Instruments: Who has the
Options
� Holder of the Derivative/Note (e.g., American vanilla option).
� Issuer of the Derivative/Note (e.g., embedded call option in
callable note).
� Both (Game options, e.g., convertible bond or exchangeable
bond with both embedded conversion option and embedded call
option).
� Third-Party Option.
1212
Equity Linked Instruments: Option Type
� Receive specified payment in cash or other securities in
exchange for the note (e.g., American vanilla, conversion
option in convertible bond - right to get certain amount of
underlying, embedded put in convertible bond - right to give
back bond to issuer in exchange for given put price).
� Obligation to accept specified payment in cash or other
securities in exchange for the note (e.g., embedded call in
convertible bond - obligation to return bond to issuer in
exchange for given call price, if issuer calls).
� Right to choose or change terms of the contract (e.g., Chooser
option or Shout option).
1313
Equity Linked Instruments: Barriers
� Knock-in/Fade-in/Knock-out/Fade-out/Parisian/others
– Instrument turns automatically into another instrument, potentially
after rebate payment,
– Either abruptly (e.g., knock-in or knock-out) or gradually (i.e.,
terms of the instruments are changing, e.g., fader)
– Once the underlying enters or remains for a specified time (e.g.,
Parisian)
– Within a specified trading range, such as, above and/or below given
barriers.
� All of the above can be observed continuously (in given time
intervals) or discretely (at given times).
1414
Equity Linked Instruments: Payouts
� European: Call, Put, or other given function of price of underlying at
maturity.
� Asian: Instead of price of underlying, take average over times or time
interval, and plug into payoff formula.
� Lookback: Payoff formula uses observed maximum and/or minimum
of price of underlying over sequence of times or time period(s).
� Forward Starting: Some terms fixed in future based on market data.
� Variance Swaps, Futures, or Options: Payoff depends on realized
variance.
� Cliquets.
� And many others.
1515
Stochastic Models for Equity Derivatives
� For equity factors/underlyings:
– Black-Scholes (constant or time-dependent volatility/rates).
– Local Volatility (time- and underlying-dependent local volatility).
– Stochastic Volatility (one or more stochastic processes).
– Jump terms, possibly with stochastic intensity.
� Credit factor(s) (for issuer of derivative & of underlying):
– Deterministic Spread, possibly underlying-dependent.
– Jumps (to Default), possibly with stochastic intensity.
� Interest rate factor(s):
– Deterministic Yield Curve.
– Stochastic Short Rate, Few-Factor Model, or HJM/BGM (*MM) type models.
� And correlation or other dependence modeling among all those factors.
� Resulting in a system of SDEs (in some cases, forward-backward SDEsfor underlying and hedging strategy)
1616
Pricing Approaches For Stochastic Models
� PDE type approach: Obtain partial differential/integral
equation(s)/inequality(ies)/complementarity problems for
instrument values (Feynman-Kac and related) and solve.
� If distribution or transition probabilities underlying payout are
given explicitly (e.g., Black-Scholes) or available with
reasonable effort (e.g., characteristic function known, e.g., for
affine models and Heston): Compute integral (explicitly, FFT,
quadrature).
� Simulation of stochastic model (with appropriate exercise
strategies, if applicable) and computing expectations of
functionals on simulated paths.
Note that some conditions must hold to apply some approaches or for them to have meaningful or consistent results. For an example, see [HS00].
1717
PDE Type Approach
� PDE with jump, final, and/or boundary conditions for options
with European and Bermudan exercise (if no jumps). Only 1D
for each underlying for Local Volatility Models, several D for
each underlying for Stochastic Volatility Models. Possibly
different characters/properties in different directions. (≤3-4D).
� PDE with additional (integral) terms, for European and
Bermudan exercises (if jumps). (≤2-3D).
� PD (in)equality(ies)/complementarity problems, when
continuously exercisable options are active, possibly involving
integral terms. (≤2D).
1818
Numerical Methods for PDE Type Problems, I
� Space discretization:
– Finite difference (FD) (sparse and structured).
– Finite/Spectral Elements (F/SEM)/discontinuous Galerkin (block
sparse and structured).
– Dimensional splitting methods, such as, ADI/ADE.
� Time discretization: Usually FD, although others do exist.
– Explicit: Fast, direct. Stability conditions might force use of small
time step.
– Implicit: Solve linear equations, use structure. Usually stable (but
CN only neutrally so).
� Integral terms from jumps introduce dense matrices, requiring
more effort or tricks in application and solution of the system.
Possibly implicit-explicit splitting.
1919
Numerical Methods for PDE Type Problems, II
� For inequality(ies)/complementarity problems, inequality constraints can be treated by:
– Time-stepping unconstrained system and projecting after each step.
– Solving inequality system iteratively with projected iterative method.
– Introduce Lagrangian multiplier and solve extended system.
– Inequality solvers using special structure.
– Penalization: include constraint into formulation of partial differential equation in penalty term, which makes equation nonlinear and requires penalty parameter, but reduces it to an equation.
2020
Simulation/Expectation Approach
� European exercise, final/pricing distribution given explicitly or
by model: Simulation of distribution or model.
� Bermudan exercise for one party (American exercise as limit or
approximated between steps): Simulation with some exercise
strategy. If exercise strategy is non-anticipatory, one-sided
bound (Longstaff-Schwartz).
� Dual formulations for martingales can also be simulated if
martingale can be constructed (Andersen-Broadie). Can give
other-sided bounds.
� Bermudan exercise for both parties: Extensions of above
methods to several exercise strategies and associated
martingales (Lvov et al, Beveridge-Joshi, and others).
2121
Simulation: Exercise Strategies
� Instrument value is maximum over holder exercise strategies and/or minimum over issuer exercise strategies of expectation over solution of stochastic model assuming those exercise strategies.
� Various dual formulations give dual minimum respective maximum problems over martingales and remaining exercise strategies. Can construct martingales as trading strategies from exercise strategies (Andersen-Broadie).
� To get exercise strategies, move backwards from last possible exercise time before maturity and build approximation of continuation value. At previous exercise time, compare exercise value and continuation value estimate to decide whether to exercise.
2222
Simulation: Estimating Continuation Values
� Choose basis functions, e.g., low-order polynomials in all appropriate observable factors (price of underlying, interest rate, CDS rate) and perform least-square regression over computed forward paths, to estimate continuation value.
� For each exercise time, starting from last, regress sampled continuation values from paths by least squares, get coefficients for basis functions and thus, continuation value estimates, also on paths. Repeat until first possible exercise time.
� Given continuation value estimates for exercise times, easy to implement exercise strategy.
� Conditional continuation value estimates and regressions can produce different (and better adapted) estimates for different regions of state space.
2323
Formulations and Example Instrument
Features
� Preferable numerical formulations depend on features and
instrument properties:
– For barriers, discretely observed barriers are more straightforward
for simulation or MC and continuously observed barriers are more
natural for FDM, FEM, and PDE, while opposite combinations
require more work, some new algorithmic parts, and additional
tests.
– For some payouts, actual payout or ideal payout is continuous while
approximation is discrete (e.g., approximating a log contract with a
strip of options), and for some, actual payout is discrete while
approximation is continuous (e.g., variance swap is defined on
discrete realized variance, but often approximated as the integral).
2424
Calibration of Models: Methods
� Interpolation or direct formula (e.g., Dupire’s formula for local
volatility).
� Bootstrapping or iterative formula (e.g., for yield curves).
� Formulation of single inverse problem and its solution (e.g.,
alternative inverse methods for Dupire’s PDE).
� Forward/backward adjoint approaches for optimization.
� Black-Box or Grey-Box optimization over calibration pricers:
LM, differential evolution, and others (prices and possibly
gradients needed).
2525
Calibration and Use of Model
� What do we believe about the model?
– Calibrate model once, valid forever. Single calibration.
– Can observe some information about true model now but more
information about true model will be revealed in future. Calibrate
once and then update.
– Can identify some (good) approximation of true model now and
have some understanding how model and/or parameters will evolve
locally in time. Regular recalibration and stickiness assumptions.
– Can identify some approximate model allowing us relative pricing
now and in the next instant. Continuous recalibration.
262626
Model Validation:
Mandates,
Interpretations,
and Approaches
2727
Model Validation: Mandates, Interpretations,
and Approaches
� In the next slides, we will quickly introduce:
– Some regulatory mandates and guidance, including the recent new
guidance by the OCC and the Federal Reserve, OCC 2011-12.
– Some different interpretations and approaches to model validation,
ranging from narrow to wide.
– Some thoughts on validation and validated models.
2828
Introduction
� Since this is not a talk about the regulatory background nor
about how various accords and regulations affect model
validation and treat model risk, only a few slides on:
– Common themes from the regulations and regulatory
pronouncements.
– U.S. regulators' statements and rules.
– Basel II Accord and the Basel Committee, with some references.
– Most detailed guidance: OCC 2011-12 and OCC Bulletin 2000-16.
2929
Common Themes
� Reading regulations, bulletins, accords, and principles, one
regularly reads that models should:
– Be based on appropriate assumptions and methods.
– Be implemented appropriately and perform as intended.
– Be independently (from FO) validated and/or approved by qualified
personnel.
– Have their assumptions and implementations assessed and
challenged in the validation (Basel II) providing an effective
challenge (Basel, April 2009, and OCC 2011-12, April 2011).
� (Risk) management should be aware of weaknesses (and
strengths) of the models.
3030
U.S. Regulators: Regulations and Statements
� Some of the regulations or other pronouncements of the
regulators regarding model validation can be found in:
– OCC 2011-12: Supervisory Guidance on Model Risk Management,
April 4, 2011
– OCC Risk Bulletin 2000-16: Model Validation, May 30, 2000.
– Model Governance, FDIC Supervisory Insights, Winter 2005.
– Valuation and Modeling Processes, in BANK HOLDING COMPANY
SUPERVISION MANUAL, Board of Governors of the Federal Reserve
System, Section 2128.06.03 (Section date: July 2008).
– Section on Validation on pages 69319-69320 in Final Rule on Risk-
Based Capital Standards – Basel II, 72 FR 235, pages 69288-
69445, codified in various parts of 12 CFR.
3131
Basel II Accord and Its Revisions and
Enhancements
� The Basel II framework and the Basel committee touch on
model validation for various models used in banks and on
model risk, e.g.:
– Comprehensive Version of the Basel II Framework, June 2006:
(see, e.g., 695 and 718). http://bis.org/publ/bcbs128.htm
– Supervisory Guidance for Assessing Banks' Financial Instrument
Fair Value Practices. April 2009: (see, e.g., page 6).
http://bis.org/publ/bcbs153.htm
– Revisions to the Basel II Market Risk Framework. July 2009: (see
page 23: model validation standards, pages 27-29: valuation
methodologies, including adjustments for model risk).
http://bis.org/publ/bcbs158.htm
3232
OCC Bulletin 2000-16: Model Validation
� OCC 2000-16 was first official pronouncement for general model
validation. It requires national banks to have a formal model
validation policy and model validation/validators.
� It introduces three model components: (1) Input, (2) Processing, and
(3) Reporting component. All three and their interplay must be
validated (not all necessarily by model validation).
� It introduces three generic procedures:
– Independent review of logic, concepts, and/or theory.
– Comparison against other models.
– If possible, comparison of model predictions against subsequent real-world
events (“back-testing”).
� OCC 2011-12 confirms most things from OCC 2000-16 but extends
guidance to model risk control in the entire firm and process.
33
OCC 2011-12: Model Risk Management
� Widens definition of model.
� Instead of concentrating only on appropriate model
validation, OCC 2011-12 broadens scope and
requires the effective management of model risk in
entire organization and model life cycle, including:
– Sound development, implementation, and use.
– Rigorous model validation.
– Appropriate governance and control mechanisms, such as
management oversight, policies & procedures, control, and
compliance.
– Appropriate incentives and organizational structure.
� Commensurate with bank’s risk exposures, business
activities, and complexity and extent of model use.
34
OCC 2011-12: Effective Challenge of Models
� Guiding principle of model validation and other activities to
manage model risk.
� Effective challenge is critical analysis by objective parties
– Who are competent and informed enough to identify model
limitations and assumptions (competence: theoretical and technical
expertise and knowledge, modeling skills, familiarity with business
and model use).
– Who have the right incentives and are willing to challenge
(separation; independence: no stake in outcome or development;
corporate culture; compensation practice).
– And who have influence and authority, as judged by actions and
outcomes, to ensure that found issues are addressed (explicit
authority; stature within organization; commitment and support
from higher management).
35
OCC 2011-12: Model Development &
Implementation
� Clear statement of purpose of model, including
intended use.
� Documentation of design, theory, logic, and
methodology underlying model.
� Rigorous assessment of data quality, suitability, and
relevance, and documentation thereof.
� Developmental testing including developmental
evidence showing such testing.
� Model use as additional, but potentially biased,
source of tests and challenges.
36
OCC 2011-12: Model Documentation
� Design, theory, and logic underlying the model should be (1) well documented, (2) generally supported by published research, and (3) supported by sound industry practice.
� Model methodology and implementation, including (1) mathematical specification and statement, (2) numerical techniques and approaches, and (3) used approximations; should be explained in detail with particular attention to merits and limitations.
� Comparison with alternative theories & approaches.
� Why model is appropriate for purpose, conceptually sound, and mathematically and statistically correct.
� Should be sufficiently detailed so that parties unfamiliar with model can understand how model operates, its limitations, and its key assumptions.
� Incentives for effective and complete model documentation.
� Also need adequate documentation of other model risk management activities, including model validation.
37
OCC 2011-12: Model Development Testing
� Test components as well as overall functioning to ensure
expected/intended performance.
� Test accuracy, stability, robustness of model over range of
input values (including extreme or stressed), for variety of
products or applications intended to assess potential
limitations, impact of assumptions, and to identify situations in
which model performs poorly.
� Test on actual circumstances under variety of market
conditions, including scenarios not ordinarily expected.
� Upstream/downstream tests.
� Test plans: Planning, design, and execution of testing.
� Appropriate documentation: Summary results with commentary
and evaluation, detailed analysis of informative samples, other
testing activities and results.
38
OCC 2011-12: Model Validation
� Model validation:
– Tries to verify whether model performs as expected according to
design objective and business use.
– Helps to ensure sound models.
– Identifies potential limitations and assumptions and asses impact.
– Is aspect of effective challenge: needs staff with appropriate
incentives, independence, competence, and influence.
– Should cover all model components (input, processing, output).
– Applies equally to in-house models or third-party models.
– Should be commensurate with bank’s overall use of models,
complexity and materiality of models, and size and complexity of
operations.
� If model developers or users perform part of validation/testing,
need critical review by independent party conducting additional
activities to ensure proper validation.
39
OCC 2011-12: Model Validation Processes, I
� Initial full validation and periodic full revalidation:
– Judges quality of model theory, design, methods, construction,
implementation, and reliability, to evaluate soundness. Includes
review of documentation and empirical (&developmental) evidence.
– Documentation and testing should convey understanding of model
limitations and assumptions. Validation reports should have clear
executive summary with model purpose and model and validation
results, including major limitations and key assumptions.
– Ensure that model design is well informed, carefully considered,
and consistent with published research and sound industry practice.
– Critical analysis of model aspects: theoretical construction, key
assumptions, data, specific mathematical approaches, and other
modeling choices, by reviewing developmental evidence and
conducting additional analysis and testing as necessary.
– Comparison to alternative theories and approaches. Benchmarking.
– Assessment of key assumptions and analysis of their impact on
model outputs and potential limitations. Sensitivity analysis.
40
OCC 2011-12: Model Validation Processes, II
� Periodic reviews of models (at least annually) whether model
works as intended and validation activities are sufficient.
Potential outcomes: (1) Affirmation of previous validation, (2)
suggested updates, or (3) call for additional validation.
� Material changes require (re-)validation of affected model.
� Ongoing monitoring of model limitations and results, including
process verification, benchmarking, and rerunning (or
extending) model development tests or model validation tests,
sensitivity analyses, as appropriate, judged from empirical
evidence, model or market changes, or theoretical research.
� Outcomes analysis compares model outputs to corresponding
actual outcomes, for instance back-testing.
� Perform similar and appropriate processes for vendor or third-
party models.
4141
Interpretations and Approaches to Model Validation
Narrow: Review Model/Methodology
� Qualitative and theoretical review: Is the model, methodology,
and approach adequate for the intended purpose?
� Check do's and don'ts.
� Review model documentation and model descriptions, not
implementations.
� (See, e.g. [ZWP-0006].)
4242
Interpretations and Approaches to Model Validation
Broader: Review of Implementations
� As before, review model, methodology, and approach.
� Also review and test the implementation of the model to be used by the front-office.
� This could involve:
– Code reviews.
– Stress-tests: tests on alternative or experimental realities.
– Back-tests: tests on historical pricing environments.
– Tests on special cases with easier solution or that allow alternative solution techniques.
– Tests against identical implementations, alternative implementations, or alternative models.
� (For some of the points, see [ZWP-0006].)
4343
Interpretations and Approaches to Model Validation
Widest: Review of Whole Process
� In addition, consider (for some of the points, see [ZWP-0006]):
– Model use: Which strategies or decisions are supported by model
or results, if any? E.g.: Arbitrage, hedging, buy & sell signals,
exercise strategy, reference price for bookkeeping or accounting.
• For mostly inventory based strategies: different risks, different
techniques.
• For statically OTC hedged positions: Credit risk, not really equity risk.
– Market data or other data sources used? Quality of data?
– How is the model calibrated? Model assumptions?
– What might break process or require special treatment (f.i., credit
events)?
4444
Validation? Review? Approval?
� We call our subject model validation (& approval).
� Similar to research in natural sciences: Even though always
working on validating models, a model is never validated for all
eternity.
� Tomorrow or in few years, could find out that model breaks
sometimes, performance degrades, or there is some scientific
or financial reason to no longer use it.
� Thus, instead of “validated”, potentially say that the model is
currently approved for a particular use, given the current
knowledge, tests, experience, and risk level.
454545
Validating Equity Linked
Instrument Models
&
A Case Study
4646
Validating Equity Linked Instrument Models
� In this part of the presentation, we will discuss:
– How scriptable testing and validation environments interfacing to
pricing systems can simplify testing and validation.
– How scriptable validation and benchmark environments interfacing
to validation or benchmark pricers and/or models can simplify
validation and benchmarking.
– How to apply previously discussed guidance, approaches, and
generic reviews, to previously described pricing processes for
equity linked instruments.
– Finally, a simple case study illustrating some of the points.
4747
Testing, Validation, and Benchmarking
Environments
� A scriptable environment is useful for efficient testing and
validation offering various interfaces to pricing systems under
review and to data sources. Should offer functions to retrieve,
analyze, change, and (re)submit data.
� Similarly, a scriptable interface and environment for pricing
programs and libraries (from model validation or third parties)
is useful to allow easy access to validation or benchmark
pricers for models on similar or simplified pricing environments.
� Under Windows, specific .NET/COM scripting language (f.i., VBA
or IronPython) or general scripting language (f.i., Python)
combined with C++ or C# (or others) validation pricers
implementing common interface.
4848
Issues, Reviews, Experiments, and Tests:
Overview
� In this part, we will talk about potential issues and concerns
and related reviews, experiments, and tests. In particular, we
touch:
– Potential qualitative reviews.
– Data and process reviews. Code review.
– Potential numerical issues in pricing and modeling.
– Calibration issues and tests.
– Various possible implementation and use tests.
– Validation plan.
4949
Potential Qualitative Reviews
� Qualitative review of proposed models and pricing approaches
to them can include:
– Theory: Might not apply - conditions or assumptions not satisfied.
Error in derivation. Gaps in derivation: Unchecked and/or
unidentified assumptions.
– Model Choice: Model does not exhibit some salient features from
market important to instrument pricing. Model exhibits features
that contradict market behavior or are implausible. (E.g., no LVMs
for Cliquets).
– Choice of Formulation: Outside of domain of validity or where it
does/might not give same/correct results. Formulation not
appropriate for model features.
– Documentation: Complete? Correct? Comprehensible?
Background? Theory? Tests?
5050
Data and Process Reviews
� Look at and check quality and “cleanness” of input data.
� Recognition and handling of input exceptions, e.g., if market
data (feeds) change format or source, IT services stop or have
problems.
� Handling/reporting of model pricing or calibration (or other)
failures?
� Tests and exceptions handling for flow of model outputs to
downstream systems and reports?
These reviews might be more appropriate for production support or oversight rather than for model validation. (Usually such things are implemented during system development.) However, need clear responsibility for such tests.
5151
Contract Data Review, I
� Another important part of pricing system and model review is
to understand how financial instruments on trading books are
actually represented in the systems.
� Representation and modeling depend on model purpose:
– Model supposed to reflect market or market participants: As long
as some use simple(r) model with appropriate data, potentially with
added spreads, such simple model could predict at which price
those participants could buy or sell.
– If model is to reflect cost in manufacturing, hedging, or arbitraging
instrument, all potentially important or exploitable features should
be covered.
5252
Contract Data Review, II
� Thus, review contract data from authoritative prospectus or
confirmation for agreement with data in pricing system:
– Which features are missing or incompletely reflected?
– What kind of impact?
– Reason for omission: Current pricing model or capabilities? Small
impact? Features that issuer and/or trading desk will not use?
– Easier to later extend model and have information available instead
of having to update all contract data with added items.
5353
Contract Data Review, III
� Contract representation might change upon event, e.g., knock-
in changes option into knocked-in option.
� Contract representation might be updated for fixings or similar.
� Equity issuer might undergo corporate change, f.i.:
– Issuer acquired for mix of stock, debt, and/or cash: Underlying
turns into mix of stock, debt, and/or cash.
– Issuer spins off subsidiary or distributes stock as dividend:
Underlying turns into combination of several stocks.
� System must be able to take such things into account and
make provisions for appropriate pricing or marking, if
necessary, or otherwise be set up to avoid issue.
5454
Algorithm and (Outside) Code Review
� Review of algorithms and code is often suggested.
� True, helps spot “obvious” mistakes (second set of eyes).
� But it is software quality assurance: Programmers of pricing
system should perform this, not model validators.
� For simple enough problems, code review might be effective
(but exhausting and uninspiring).
� Code review is notoriously unreliable: Code with small but
consequential mistake might look perfect, correct code might
look horrible. Code review is NP-hard.
� For any kind of assurance that code performs as promised,
need requirements, unit tests, consistency tests, and/or
similar. But, again, this would be good software development
style, not outside review.
5555
Numerical Issues (Non-MC Formulations), I
� Standard implementations of special functions (in libraries)
might not be accurate enough and lead to erroneous prices.
(See [EM05] for negative option prices caused by bad
approximation of normal cdf.)
� Discretization methods and their properties: Stability?
Convergence? Convergence order? Smooth convergence
(extrapolation)? Lowest acceptable resolution? Error
estimates?
� Properties preserved/violated in discretization (f.i., martingale,
mean, conservation of probability).
� Condition numbers and other things affecting stability and
precision of numerical computations.
� Numerical boundary conditions and boundary placement.
5656
Numerical Issues (Non-MC Formulations), II
� Direct solvers: Condition numbers, behavior at finite accuracy,
assumptions of solvers (positive definiteness).
� Iterative solvers: Stopping criterion, speed (or failure) of
convergence, accuracy.
� Splitting methods (implicit-explicit or dimensional splitting):
Splitting error? Stability conditions/handling of cross terms?
Boundary conditions for split problem?
� Nonlinear solvers: Local convergence to local solution, might
get stuck or misbehave, or might result in badly behaved linear
systems.
� Finite difference approximations for derivatives or gradients:
Step too small: Numerical noise and numerical precision issues.
Too large: Approximation bad.
5757
Numerical Issues: Monte-Carlo/Simulation
� Approximation of stochastic model behavior, f.i. positivity & behavior close to zero (f.i., variance process in Heston).
� Convergence of discrete time approximation.
� Required number of paths for simulation (and for regression). Number and kind of basis functions.
� Quality of error estimates. Also, low discrepancy numbers and similar require different kinds of error estimates.
� If random or low-discrepancy numbers: Potential issues for high dimensions or certain kind of approaches.
� Greeks: Many approaches and issues. Detailed knowledge of mathematics and/or model might be necessary.
� For complex problems, hard to test accuracy or quality.
5858
Calibration Issues and Risk
� If model calibrated against market data, how strongly does
calibration depend on calibration assumptions?
� Hamida and Cont: Recovering volatility from option prices by
evolutionary optimization. J. Comp. Finance 8(4), 2005: Find
different local volatility surfaces that reprice market options.
Wide range of local volatility surfaces. Very wide for short
maturities and smaller for longer.
� Detlefsen and Haerdle: Calibration Risk for Exotic Options.
Working Paper, 2005: Calibrated Heston and Bates parameters
depend strongly on premiums or volatilities, objective
functions, and other details. Model prices of exotics might
differ substantially.
5959
Calibration Results as Model Test
� As introduced before, different understandings and uses of
calibrated model (calibrate once vs. constant recalibration)
make different assumptions.
� Those different assumptions can be tested by calibration
studies and investigating how calibrated parameters change,
and, potentially, how pricing changes (which might change
less).
� Quality of calibration over time will tell us whether model can
at least be good approximation of current market if not truly
dynamically consistent model.
6060
Testing Prices and Outputs
� For same model formulation and type, compare pricing system output and validation pricers (identical/similar formulation and approach, identical/similar model, and identical/similar data). Convenient environment allowing data manipulation helps.
� Use models just outside pricing framework to look at specific issues, f.i., compare local volatility models to Heston and Bates (for stochastic volatility/jumps).
� If available, use fundamentally different type of model to see range.
� Use (1) historical market data (“back-test”), (2) historical or constructed stress scenarios (“stress-test”), or (3) given assumed data generating process, data produced by that process.
6161
Testing Numerical Processes
� For special functions, try different implementations. See
whether Maple and/or Mathematica or high precision packages
allow spot checks.
� Check intermediate variables in closed form solutions to detect
overflow/underflow/loss of accuracy.
� For any discretization or approximation, check for stability,
oscillation, discontinuities or corners, and/or pollution AND
convergence with respect to discretization parameters.
� For spot checks, look at entire numerical solution.
� Possibly cross-validate many approaches for special case.
� Environment and pricer setup allowing easy plug-and-play
exchange of components or algorithmic choices comes in very
handy.
6262
Testing Model Use
� There might be other ways to judge quality of model and formulation besides output comparison.
� Model might be used for hedging, manufacturing derivatives, or arbitrage, and that use can be simulated.
� Thus, simulate how well use would have performed (f.i., P&L) in back-tests, stress-tests, or given data generating process.
� Alternatively, historical pricing performance (or at least all the data needed to perform historical pricing and marks) might be stored somewhere and/or P&L attributions and explanations using that model might be computable or stored - good to identify particular risks and dependencies (and also to identifyunexplained behavior).
6363
Validation and/or Testing Plans, I
� Having all these potential issues, problems, reviews, and tests,
not every potential issue should be investigated and every
possible test should be run for each validation.
� Depending on instrument and its features and chosen
formulation, some reviews are more relevant.
� Size of holding (and thus impact of model) will determine how
material the model risk is and thus, how thorough the review
should be.
� Actual model use will guide you to which risks and issues are
most important.
� Previous validation for other similar instruments (or of same
instrument, for revalidation) will suggest avenues of
investigation.
6464
Validation and/or Testing Plans, II
� A few, general tests in the beginning might allow you to select
right level of review or places to investigate.
� Historical pricing performance and/or P&L attributions and/or
explains allow review of average and normal performance as
well as identification of potential issues.
� Chart a course and select battery of sensible tests, make plan,
and execute it.
� At least sometimes, you should challenge or stress test the
model.
6565
Example Case Study: CEV Local Volatility
Model
� Constant elasticity of volatility is one of the simplest models
with non-constant local volatility and a skew.
– The stochastic model is:
(1)
– For negative β, the process might get absorbed at zero and a point
mass might develop there.
– Closed form formula for European calls and puts in terms of two non-
central chi-squared distribution functions (Davydov-Linetsky,
Schroeder).
– Some tests with displaced CEV (local volatility shifted to left):
(2)
( ) dWSdtqrS
dS β+−=
( ) ( ) dWSdtqrS
dS β∈++−=
6666
Why CEV
� CEV allows us to look at, at least, three issues:
– Influence of skew and non-constant local volatility in an easy case.
– Possibility of default as absorption at zero (and how methods
handle this).
– Some of the theory (such as, PDE formulation gives same result as
numeraire/expectation formulation gives same result as explicit
solution distribution) no longer applies.
� While still having closed form solutions for some cases.
6767
List of Tests
� Compare PDE pricer with flat volatility, standard boundary
conditions, and standard settings, against closed form solution.
� Compare American or exotic option prices between flat
volatility (Black-Scholes setting) and CEV setting: Set Black-
Scholes implied volatility for Black-Scholes American or exotic
option pricer to implied volatility computed from European
option pricer.
� Test different maturities, strikes, β, etc. Show dependence only
for strike and β for one setting here.
� Compare implied volatility slice to market data.
6868
Validating PDE Pricer Against Closed Form
5
6
7
8
9
10
11
-7 -6 -5 -4 -3 -2 -1 0
Put Price
beta
PDE Solver for CEV Closed Form Formula (noncentral chi^2)
6969
Comparing to Some Market Implied Volatility, 1Y
0.05
0.1
0.15
0.2
0.25
0.3
0.5 0.6 0.7 0.8 0.9 1 1.1 1.2 1.3 1.4
Volatility
Strike/Spot
MSFT (shifted down 0.25) SPY (shifted down 0.11)beta = -1 beta = -2beta = -3 beta = -4
7070
Absolute Price Differences CEV vs. BS, 1Y
-0.16
-0.14
-0.12
-0.1
-0.08
-0.06
-0.04
-0.02
0
0.02
60 70 80 90 100 110 120 130 140AmCEV -AmConst
Strike
beta = 0 beta = -1 beta = -2 beta = -3 beta = -4
7171
Relative Price Differences CEV vs. BS, 1Y
-5.00%
-4.00%
-3.00%
-2.00%
-1.00%
0.00%
1.00%
2.00%
3.00%
60 70 80 90 100 110 120 130 140
(AmCEV -AmConst)/AmCEV
Strike
beta = 0 beta = -1 beta = -2 beta = -3 beta = -4
727272
Conclusion and
References
7373
Some Last Thoughts
� If you are the validator,
– Communicate and advocate for what you are doing. Plans and a
good process can help with that.
– Get buy-in and understanding from front-office and/or management.
Don't be afraid to learn about what you are validating, model use,
and approaches in your firm.
– Front-office or quants might have run particular tests or evaluated
certain risks and issues with their models. Talk to them and find
out. You will still need to validate, but this can give you insight into
or comfort with front-office models.
– Validation efforts and their depth have to match materiality of
impact and risk.
7474
Conclusion
� I hope that this presentation introduced you to
– Equity linked instruments, their modeling, and pricing.
– Regulatory mandates and guidance and various approaches for
model validation.
– Potential issues and concerns for implementations of models for
equity linked instruments.
– Potential tests, experiments, and approaches to model validation
for equity linked instruments.
� Thank you for your attention.
7575
References
� Only papers which are cited several times in short form are
given.
– ZWP-0006: Martini and Henaff, Model Validation: Theory, practice,
and perspectives. Zeliade White Paper ZWP-0006, Zeliade
Systems, March 2010
– HS00: Heath and Schweizer, Martingale versus PDEs in Finance:
An Equivalence Result with Examples. J. Applied Probability, (37)
947, 2000.
– EM05: Eben Mare, Learning Curve: Verification of Mathematics
Implemented in Financial Derivatives Software. Derivatives Week,
September 19, 2005, Page 6-7.
� Ask me for other references.
7676
Disclaimer
� This presentation is for informational purposes only, is not an offer,
solicitation, recommendation or commitment for any transaction or to
buy or sell any security or other financial product, and is not intended
as investment advice or as a confirmation of any transaction. Any
market price, indicative value, estimate, view, opinion, data or other
information herein is not warranted as to completeness or accuracy, is
subject to change without notice, and Wells Fargo Securities accepts
no liability for its use or to update or keep it current. Any views or
opinions are those of the individual presenter, not necessarily of Wells
Fargo Securities.
� Wells Fargo Securities is the trade name for the capital markets and
investment banking services of Wells Fargo & Company and its
subsidiaries, including Wells Fargo Securities, LLC, member FINRA and
SIPC.