validation - issues, challenges, observations - febraban (june 2009)

7/28/2019 Validation - Issues, Challenges, Observations - Febraban (June 2009)

1/20

Validation in Risk Management:, ,

.Division of Banking Supervision and Regulation

Board of Governors of the Federal Reserve System

**************************

Febraban 2nd Annual Operational RiskConference

June 5, 2009


2/20

Agenda

Validation more than models

U.S. Final Rule

requirements Framework validation

Data validation

Supervisory review

I think ou should be moreThis presentation contains the views and

2

explicit here in step number 2,

represent a policy statement of the Federal

Reserve Board of Governors.


3/20

Validation

Validation in this context is a set of processesor activities designed to assess whether riskmanagemen oo s are wor ng as n en e Applies to quantitative and qualitative tools (not just

models

Validation is a control mechanism that helps

to assess uncertainty in risk management Tools for assessing risk always contain inherent

uncertainty

remove it)

3


4/20

Characteristics of Validation

Validation:

management tools

Is a process, not an event

(including underlying components)

quantitative and qualitative assessment

4


5/20

Governance Framework

Board &SeniorMgmt

BusinessLine Mgmt

Risk Management Approaches

Oversight

s

Controls

y n

Validation

l

Roles&

onsibiliti

olicies

entories

ecurity

hange

ontrol

aIntegrit

umentatio

rocess

rification

elopment

vidence

utcomes

nalysis

Res In Da

t

Doc

Ve Dev

O

Internal Validation

5


6/20

Components of Validation

Validation activities generally fall into three

categories:

Review of the development process and conceptual foundation

Ongoing monitoring of process, which includes verifyingsoundness of approach and comparing to benchmarks

Analysis of outcomes (such as backtesting)

All three types of validation activities should be

process

Applied to the process that translates inputs into outputs, as well

u u u v Key exercises for analyzing and checking outcomes

include sensitivity analysis and stress testing

6


7/20

Keys to Successful Validation

Senior management is responsible for ensuring thatvalidation is working properly

Should set clear expectations for acting on validation results Validation tends to be more effective if considered

Good practice to have a firm-wide validation policy

Many institutions have various levels of validationconcep ua par a u Can be appropriate when full validation is not possible

But lack of full validation should always be transparent

a oo as no een u y va a e , ere s ou e grea eruncertainty about the estimates it produces

7


8/20

Keys to successful validation (cont.)

Objectivity in validation is highly important Does not necessarily have to mean structural independence

alleviate potential conflicts of interest On its own, independent structure does not translate to effectivevalidation

Documentation is vital to good validation Without it, evidence of objective validation is lacking

Effective validation re uires com etent staff with theright skil ls, the right resources, the right incentives,and appropriate stature within the organization Need abilit to effectivel challen e develo ers

Need to identify issues and have a process for resolution

8


9/20

Model Validation Techniques

Benchmarking

Benchmarking involves assessing the consistency of

the estimated parameters with those obtained by otherestimation techniques, and potentially using other data.

Backtesting

post) values with estimated (ex ante) parameters for acomparable and homogeneous data set.

Judgmental Comparison Comparison with other judgments and models

omparson wt ot er externa n ormaton

9


10/20

Validating Vendor Models

Vendor models should be subject to the same

validation standards as internal models

Validation includes not just the vendor model itselfbut also application of the vendor model to the

n v ua an , ts exposures, an rs pro e

Documentation standards also apply

Two key questions regarding vendor models:

Does the model do what it is desi ned to do?

Does the model do what the bank is employing it todo?

10


11/20

U.S. Final Rule Requirements

The [bank] must validate, on an ongoing basis, its advancedsystems. The [bank]s validation process must ben epen en o e a vance sys ems eve opmen ,

implementation, and operation, or the validation processmust be subjected to an independent review of itsadequacy and effectiveness. Validation must include:

i. An evaluation of the conceptual soundness of (including

systems;

ii. An ongoing monitoring process that includes verification

of processes and benchmarking; and

iii. An outcomes analysis process that includes back-testing.

11


12/20

Framework Validation

Examples:

A broad assessment of the conceptual soundness of theAMA framework (e.g., Do the tools, processes, andoversight systems the institution has designed andim lemented to measure and mana e o erational risk fitwith the quantitative techniques designed to quantify

operational risk?). re a e r s managemen oo s su ec o ongo ng

monitoring and evaluation of their effectiveness? Doesthe institution have robust mechanisms to periodically test

whether a particular tool or process remains longereffective?

12


13/20

Data Validation

Examples:

Reconcile to source s stems

Review by a risk control function

Does internal audits participation as an internal control

compromise its independence?

Review by business line peers

Comparison with other data elements

Comparison with experience or expertise

Backtesting of BEICFs to internal loss data

Assumptions are also part of the data13


14/20

Framework for Supervisory Review

SupervisoryEvaluation of

Model Oversight

SupervisoryExamination ofModel Controls

SupervisoryReview of Model

Validation

Model Developmental

Documentation

Inventories Data Integrity ProcessVerification

Securi ty & OutcomeChange Control Analysis

14


15/20

Pillar 2

Pillar 2:

Supervisory

Review and

Pillar 3:

Market Discipline

Pillar 1:

Minimum Capital

Requirements

n erna ap aAdequacy

Assessment

Process

15

(ICAAP)


16/20

Pillar 2 Supervisory Review & ICAAP

Supervisory guidance issued 7-15-08

Including, but not limited to: credit, market, operational,interest rate, and liquidity risks

Set and assess capital adequacy goals in

relation to risk Ensure the integrity of the ICAAP

ICAAP is not just an economic capital calculation

16


17/20

Implications for Operational Risk

Overall process for operational risk may be

based heavil on AMA

Pillar 2 approach may use a different confidencelevel (99.9th percentile for Pillar 1) or insuranceoffset (limited to 20% of operational risk capital forPillar 1)

e s no as ru es- ase as e

AIRB

o prescr e ormuas Institutions design framework that best fits their

17


18/20

Pillar 2 Validation Required

Each bank should ensure that the components

of its ICAAP, including any models and their

inputs, are subject to the banks validationpolicies and procedures.

A bank should be cognizant that both

their own inherent biases and assumptions

that affect risk assessment. Accordin l a

bank should recognize the biases andassumptions embedded in, and the limitations

of, the approaches used.

18


19/20

Supervisory References

Risk Model Validation ; OCC Bulletin 2000-16

http://www.occ.treas.gov/OCC_current.htm

,

http://www.fdic.gov/regulations/examinations/supervisory/insights/siwin05/index.html

,

Advanced Measurement (AMA) and Internal Ratings Based (IRB)

Approaches, Committee of European Bank Supervisors, April

2006 htt ://www.c-ebs.or / etdoc/5b3ff026-4232-4644-b593-

d652fa6ed1ec/GL10.aspx

Supervisory Guidance: Supervisory Review Process of Capital

Ade uac Pillar 2 Related to the Im lementation of the Basel II

Advanced Capital Framework, July 2008http://www.federalreserve.gov/newsevents/press/bcreg/bcreg200

80715a1.pdf

19


20/20

Kenneth [email protected]

validation - issues, challenges, observations - febraban (june 2009)

Documents