model validation a raroc case study
DESCRIPTION
TRANSCRIPT
![Page 1: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/1.jpg)
Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.Copyright (c) 2007 Standard & Poor’s, a division of The McGraw-Hill Companies, Inc. All rights reserved.
Model ValidationA RAROC Case Study
Rocky IeraciDirectorStandard & Poor’s Risk Solutions
February 27th, 2008
![Page 2: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/2.jpg)
2.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
• Model Validation Framework
• Case Study
• Summary
• Q & A
Agenda
![Page 3: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/3.jpg)
3.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Model Validation Framework
Model Development /Revisions
Enha
ncem
ents
Model Enhancement Opportunities
(Based on Validation Activities)
Validation Protocols
Governance Framework
Confirmation of Model Operation
Validation of Conceptual and Theoretical Soundness
Outcome Analysis
(Backtesting & Benchmarking)
Enha
ncem
ents
![Page 4: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/4.jpg)
4.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Model Validation Framework
Conceptual Soundness
• Review of Conceptual and Theoretical Soundness of model assumptions, inputs, outputs, functions and overall methodology
• “Developmental Evidence” – Focus is on design and construction- Can the model be expected to work as intended?
- Consistency between model and business objectives
- “Statistical” vs. “Expert Judgment”
• A consistent assessment of two dimensions of model risk: - Model error potential (i.e. potential errors in estimation), and
- Impact of model errors (i.e. what is bottom line impact of errors)
![Page 5: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/5.jpg)
5.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Model Validation Framework
Model Operations
• Confirmation of model operation: on-going monitoring of model and surrounding processes, which may include:
- Key Performance Indictors
- Exceptions monitoring (i.e. Overrides)
- Verification of “replicability”, appropriate use of model, data integrity
Outcome Analysis / Performance Testing
• Review of the model’s historical and relative performance, including:
- Backtesting – Predicted versus realized outcomes
- Benchmarking - Uses alternative models, methodology or data to draw inferences about the suitability of the predicted estimates, risk factors, or segmentations prior to observation of actual outcome
- Importance of tolerance levels and remedial action policy
![Page 6: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/6.jpg)
6.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study Validation of RAROC Model
![Page 7: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/7.jpg)
7.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Overview of Typical RAROC Model
• Many FI’s have adopted some variant of a RAROC model in their loan adjudication and performance measurement processes
• In it’s simplest form,RAROC = Risk Adjusted Return / Marginal Capital
• Some uses of RAROC models include:- Accept/Reject decisions
- Loan pricing
- Structuring (I.e. collateral coverage)
- Compare profitability across business segments (compensation link?)
• Having the appropriate methodology is critical as it has a direct impact on the bottom line. Both refusing “Good Credits” and accepting “Bad Credits” puts you at a competitive disadvantage.
![Page 8: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/8.jpg)
8.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Overview of Typical RAROC Model
A closer look at our example RAROC equation:
Total Revenue – Overhead - Expected Loss - TaxesRAROC =
Marginal Credit Capital where,
Time Horizen = One year forward estimate of profitability
Total Revenue = Expected 1st yr Spread Revenue + Upfront Fees
Overhead = Non-Interest Expense (fixed charge applied per segment)
Expected Loss = Obligor PD * Facility LGD * Loan Exposure
Taxes = Jurisdiction specific tax payable on loan income
Marginal Credit Capital = 1st yr credit capital based on one factor VaR model (similiar to Basel II)
on
> Hurdle
Hurdle = Cost of capital + risk premium applied to all deals
![Page 9: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/9.jpg)
9.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
PD& PD Migration
Obligor Risk Rating
Process
L G D[Loss Given Default]
StructureDebt Below
Industry
EA D[Expos Given Default]
TermLoan Type
Loan Amount
Term
U L[Unexpected Loss]
Diversification
Target Debt Rating[for Portfolio]
Portfolio Capital Model
CorporatePolicy EC
(Economic Capital)
Total Revenues
- Overhead
- Expected Loss
- Taxes
Non-Interest Expense
L G DEAD
PD
NetIncome
Typical RAROC
Schematic
R A R O C
Case Study – Overview of Typical RAROC Model
![Page 10: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/10.jpg)
10.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Conceptual Soundness Review
![Page 11: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/11.jpg)
11.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Conceptual Soundness
Suitability of Inputs
• Concerned with alignment of inputs with expected output
- PD used is “TTC”, but we want one year outlook of profitability?
- LGD used is consistent with average loss expected. How does this align with RAROC measure?
- EAD based on expected utilization. For revolving loans how is expected utilization estimated? Not an alignment issue, but one of consistency.
Total Revenue – Overhead - Expected Loss - TaxesRAROC =
Marginal Credit Capital
PD, LGD, EAD
![Page 12: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/12.jpg)
12.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Conceptual Soundness
Suitability of Inputs, con’t• Marginal Credit Capital – Horizon is one year
• Best estimate of how much capital FI needs to hold to maintain target rating over following year?
- Rating Philosophy will, in part, dictate what the credit capital figure represents.
PIT CapitalCapital requirement over the next year.It is Cyclical.
TTC CapitalAvg capital requirement over a cycle. It is a-cyclical.
Business Cycle (time)
Bottom of the Cycle CapitalMax capital requirement over a cycle.It is a-cyclical
Cap
ital
![Page 13: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/13.jpg)
13.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Conceptual Soundness
Suitability of Inputs, con’t• Risk Rating Philosophy – Point in Time or Through the Cycle?
- Expansion:
- Downturn:
• Consider PD of cycle sensitive group of credits- TTC PD < PIT PD in downturn Actual EL likely > than estimated - Will Marginal Capital compensate?
Rate HurdleCapital Marginal
Loss ExpectedRev Fee Expected Rev Spread Expected≥
−+=RAROC
↓
↓
↓
Rate HurdleCapital Marginal
Loss ExpectedRev Fee Expected Rev Spread Expected≥
−+=RAROC
↑ ↑
↑
![Page 14: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/14.jpg)
14.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Conceptual Soundness
Suitability of Inputs, con’t• Anything missing on Revenue front?
- Return on invested capital
- Relationship revenues – controversial and difficult to get right
• Can be a subjective factor in final loan decision
• Can be an estimate that is built into profitability equation
• Important to track outcome of estimates to avoid ‘gaming’
• What does this all mean for our RAROC model?
- Interpretation issue – expect one year forward assessment of risk based profitability, but we get ???
![Page 15: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/15.jpg)
15.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Other Issues• What are the impact of the term assumptions?
• If we only use the first year’s expected profit and marginal capital, we introduce a bias against long term deals. Some ways to counter this:
- RAROC calculated relative to term of deal.
- Use average forward marginal capital
- Accept bias and mitigate as much as possible (Term Premium)
Case Study – Conceptual Soundness
Term to Maturity
(Expected) 1 year M
arginal Capital
1 year Deal
Multi Year Deal
![Page 16: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/16.jpg)
16.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Conceptual Soundness
Other Issues, con’t• How is Total Capital allocated down to deal level?
- Assumptions: Contribution to UL, Tail, Expected shortfall
- What about Exposure and Industry concentration?
- How does this impact our RAROC and resulting loan decision?
• Operational Risk
- As we move towards SME world, it can become a significant component of total losses
• Regulatory Capital
• Differentiated Hurdle Rates
• Price Taker vs. Price Setter?
![Page 17: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/17.jpg)
17.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Model Operation & Performance Testing
![Page 18: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/18.jpg)
18.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Confirmation of Model Operations
• In this context, confirmation of model operation would primarily entail setting up a report that tracks KPIs, for example:
- Ratio of rejected vs. accepted deals
- Average RAROC by segment for new deals, including trends
- # of below hurdle deals accepted – ‘exceptions monitoring’
- Initial RAROC estimate vs. current using actual balances
• Replicability Test – Take sample of deals and apply RAROC model, do we get same result?
• Particularly with legacy systems, data integrity test should be performed periodically. Are we using the “right” inputs when calculating RAROC?
![Page 19: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/19.jpg)
19.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Outcome Analysis: Backtesting
• Backtesting of input along three dimensions:1) Discriminatory Power
• Rank Order Tests
• GINI Coefficient
• KS and other statistical measures
2) Accuracy of Estimates• Statistical accuracy tests;
• Importance of Confidence boundaries;
• Correlation of Risk Factors - adjustments required
3) Confirmation of Philosophy• Point in Time vs. Through the Cycle - Are predicted outcomes following
intended behavior?
• Rating Migration patterns – Mobility Metric
![Page 20: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/20.jpg)
20.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Outcome Analysis: Backtesting
• Inputs fine, but do combination of these inputs produce accurateforward looking picture of risk based profitability?
• Static portfolio analysis – Total expected vs. realized economic profit
Capital MarginalRate Hurdle Expenses Expected- Revenue Expected ×≥
UsageCapitalRate Hurdle Expenses Realized- Revenue Realized ×≥
Expected Economic Profit, (Sum of individual deals)
Realized Economic Profit, (Portfolio)
![Page 21: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/21.jpg)
21.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Case Study – Outcome Analysis: Benchmarking
• Benchmarking of RAROC methodology (also CS Validation)
- Selection of representative peer sample or ‘best practice’ banks
- Sources of benchmark information may include:• Publicly available information
• Collective expert knowledge within organization
• Survey research (i.e. questionnaire on risk based pricing)
• Benchmarking RAROC results
- Consortium studies: FIs run a sample of representative deals through their pricing model and results shared anonymously
• Benchmark alternative risk based pricing models
- Analysis conducted at portfolio/segment level
- Alternative model provides more realistic estimate of profitability?
![Page 22: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/22.jpg)
22.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Summary
• Importance of Model Validation function
• Continuous improvement process
• Many models do not seem overly complex but underlying concepts can be very technical and complex
• “Hidden” assumptions and their impact on results often overlooked
• These “simple” models can impact the bottom line significantly
![Page 23: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/23.jpg)
23.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Q & A
![Page 24: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/24.jpg)
24.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Contact Information
Rocky Ieraci, Director
Standard & Poor’s Risk Solutions
Tel:(416) 507-3208
E-mail: [email protected]
Web site: www.risksolutions.standardandpoors.com
![Page 25: Model Validation A RAROC Case Study](https://reader031.vdocuments.mx/reader031/viewer/2022012309/54c6e9394a79590e788b45ff/html5/thumbnails/25.jpg)
25.Permission to reprint or distribute any content from this presentation requires the prior written approval of Standard & Poor’s.
Analytic services and products provided by Standard & Poor’s are the result of separate activities designed to preserve the independence and objectivity of each analytic process. Standard & Poor’s has established policies and procedures to maintain the confidentiality of non-public information received during each analytic process.