comply and compete: model management best practices
TRANSCRIPT
» insights
Comply and Compete: Model Management Best PracticesImprove model validation and tracking practices to prepare for a regulatory audit and boost decision performance
Regulators worldwide have become increasingly concerned about the soundness of decision
making and capital adequacy within financial services. As a result, they are taking an even closer look
at how financial institutions manage risk and use scoring models.
Regulations demand rigorous documentation across the model lifecycle, with particular vigilance
in monitoring ongoing model performance and use in production. While the Fed/OCC and Basel
regulators have recently issued guidelines on what they expect, they are often just that—guidelines.
The onus falls on financial institutions to develop a rigorous model management framework to
satisfy audit requirements.
While increased scrutiny from regulators carries an added burden, financial institutions can take
the opportunity to make improvements that translate into increased analytic performance and risk
reduction. This paper highlights best practices in model management, focusing on nine critical areas
on the radar of regulators:
1. Review credit risk policies regularly.
2. Prepare a suitable data sample.
3. Ensure segmentation transparency.
4. Choose the right model type.
5. Validate model effectiveness.
6. Track performance.
7. Monitor overrides.
8. Defend decision strategies.
9. Document thoroughly.
Number 55
www.fico.com Make every decision countTM
This paper outlines best practices that not only help you prepare for a regulatory audit, but also evaluate and refine model performance to boost portfolio profitability.
April 2014 www.fico.com page 2
Comply and Compete: Model Management Best Practices
» insights
Financial institutions are now using predictive models on an increasingly broader scale, to
measure capital reserve requirements and manage complex decisions across the credit account
lifecycle. The greater complexity and number of models in use makes it even more difficult to
meet the latest regulatory guidelines, particularly around model tracking and validation.
Globally, the Basel Framework attaches great importance to model validation. Under Basel,
financial institutions calculating capital requirements must fully understand how models used
in internal ratings-based and risk-weighted asset calculations are developed; conduct regular,
ongoing validation of such models; and prove they are responding to findings from their
ongoing analyses. Basel also sets strict requirements for governance and documentation.
Basel is intended as a set of guidelines,
allowing individual countries to implement
the regulations to fit their needs. While the
core principles remain the same regardless of
country, there are nuances in interpretation
by regulatory authorities and even individual
regulators assigned to a particular institution.
Similarly, in the US, the Board of Governors
of the Federal Reserve System (Fed) and the
Office of the Comptroller of the Currency
(OCC) recently released updated guidelines
for financial institutions using predictive
models across their business. Model
validation still dominates the new guidance,
but it is more clearly defined as within the
broader scope of model management and
governance. As with Basel, the Fed/OCC
guidelines can be highly subjective and open
to interpretation.
While implementing new regulations is
proving an enormous undertaking, the
usefulness of required tracking, monitoring
and documenting is not limited to
compliance. Improving processes also
enables institutions to evaluate and refine
model performance and risk management
practices in ways that control losses and boost
portfolio profitability. In this paper, we outline
best practices for preparing for a regulatory
audit with this dual goal of compliance and
competitive advantage in mind.
» The Compliance Challenge
Best Practices for Compliance
Key Takeaway
1. Review credit risk policies regularly
Conduct policy reviews every six months since these have a direct impact on your bottom line.
2. Prepare a suitable data sample
Be able to demonstrate that your sampling techniques are complete, responsible and relevant.
3. Ensure segmentation transparency
The best automated tools help ensure transparency for regulators, while enabling performance improvements.
4. Choose the right model type
Select a model type that’s appropriate for the decision type and available data, as well as one that ensures transparency and palatability.
5. Validate model effectiveness
It is essential to revalidate models on an ongoing basis— minimum once a year, but more often in a dynamic economy.
6. Track performance Employ standard reporting and analysis that provide critical insight into the health of models that drive your critical decisions.
7. Monitor overrides Monitor overrides carefully, and be able to prove that they are based on clear and consistent guidelines.
8. Defend decision strategies
Since decision strategies have become increasingly complex, interactive strategy exploration functionality is critical for tracking strategies, strategy changes and results.
9. Document thoroughly
Automate production and review of standard reports, and document any findings.
www.fico.com page 3
Comply and Compete: Model Management Best Practices
» insights » insights
Regulatory compliance must be central to credit risk policies, which reflect an organization’s broader
objectives in terms of risk appetite. Because policies have a direct impact on the bottom line,
reviewing policies every six months is considered good business practice. In the US, the OCC/Fed
guidance requires a review of policies at least annually.
A thorough review should ask:
• Does each policy serve a purpose? Policies have a tendency to become part of corporate
culture. Regular policy reviews ensure you are not retaining a policy when it is no longer useful in
the current business environment or overlooking newly emerging requirements.
• Are your policies defensible? Examine your policy requirements closely to determine if they
are truly indicative of risk and empirically defensible.
• Are your policies consistent? Policies should be consistent throughout an account lifecycle
and across channels, taking into consideration that you may need to interact differently with
clients in person, over the internet or through a call center.
• Are your policies redundant/do gaps exist? Review policy performance to eliminate
redundancies, reduce contradictions and identify gaps or overlaps. Gaps can result in excessive
risk being introduced into your portfolio. In contrast, redundant risk mitigation can limit your
opportunity to book good quality accounts. In other words, both extremes can impact your
bottom line.
Be sure to document the reasons behind each policy you observe. You may also consider hiring an
independent third party to review your policies within the context of industry benchmarks, and
challenge policies accepted as the norm within your organization.
1. Review Credit Risk Policies Regularly
Manage growing model portfolios. Financial institutions are using predictive models on an increasingly broader scale. The challenge is
finding an efficient way to manage and track models in production—which can number in the thousands for large lenders—and ensure these
models are still performing well.
Respond to regulatory requests. By implementing consistent, repeatable processes at every stage of the model lifecycle, lenders can
respond to regulator requests promptly with sufficient documentation and a complete audit trail.
Retain transparency. Models should be strong predictors, but they should also be easy to understand, defend and explain. This is important, for
example, when explaining to a regulator why a particular characteristic or segmentation was used.
Determine what to prioritize. By putting in place a model inventory and reporting schedule, financial institutions can more easily monitor
high-impact models and identify ones most in need of review and redevelopment.
Keep up with documentation. Solutions that standardize documentation and automate workflow enable financial institutions to
demonstrate compliance and respond more rapidly to management and regulatory requests.
Top Model Management Challenges
www.fico.com page 4
Comply and Compete: Model Management Best Practices
» insights » insights
Since improper sampling can result in poorly suited models, regulators require you demonstrate
that your sampling and model validation techniques are complete, responsible and relevant.
This holds true for both the initial validation after you develop the model, as well as your
ongoing model validations. Basel guidelines require that you use relevant internal or external
data when testing and validating models.
For your initial validation, the sample you use to validate a model should be independent of the
development sample. This can inform whether a model is over-fit to training data, and provides
a more realistic benchmark for how it would perform in production.
For ongoing validation of models, we recommend that you:
• Avoid sampling when possible. It is best to use all records from a given time period to
validate the model. When seasonality is an issue, choose a scoring window outside of that
time period.
• If sampling, ensure a representative and adequate sample. Insufficient sample size
can lead to poor conclusions in the model validation. When possible, select a random
sample that adequately represents all subpopulations of interest. Keep in mind the
economic, market and product situation during the timeframe in which you pull your
sample, since this may impact the accuracy of your results.
• Be aware of data bias. It is highly likely that your data will
be biased, for example, by the decision strategies you apply
to new applicants. For accounts scoring well above your
cutoff, you should expect a reliable odds-to-score relationship.
However, near the cutoff score, this pattern may reasonably
weaken or even reverse due to the influence of well-chosen
overrides. Furthermore, because business outcomes are not
known for rejected applicants, you won’t see as strong of
a separation between goods and bads compared with the
development sample. Be prepared to defend these influences
to regulators.
Regulators will also inquire about your data hygiene processes.
You should understand and document the accuracy of data
sources, inputs, outputs, transformations and calculations
for both model development and validation. Be prepared to
demonstrate how you treat outliers and missing values. FICO
recommends validating the reliability and quality of data
sources yearly.
Regulators require that you clearly document how you segmented the subpopulations within
your model and how you determined the unique actions you took against each subpopulation.
You also need to demonstrate that your segmentation supports your business objectives.
3. Ensure Segmentation Transparency
2. Prepare a Suitable Data Sample
“Enhancements to model oversight and validation processes are re-investments in a bank’s intellectual property. This not only aligns with current regulatory guidance, but truly gives a bank solid support for treating this asset as capital today and will provide tomorrow’s differentiator in competitive advantage.”
— Pasquale Lapomarda, Retail Credit Risk Analytics Manager,
TD Bank
www.fico.com page 5
Comply and Compete: Model Management Best Practices
» insights
Regulators will ask whether you
defined your subpopulation
empirically or by a domain expert,
and how your segmentation fits
in with your decision strategies.
In some countries, you may
also need to demonstrate that
your segmentation does not
discriminate based on age, race or
gender. Basel mandates that if you
segment by product, you must do
it under the umbrella of Residential
Mortgage (RM), Qualified Revolving
Retail Exposures (QRRE) and Other
Retail (OR).
The key to successful segmentation
is in identifying the right variables
to split a population into actionable
segments. Automated tools and
techniques now make this process
significantly faster and easier. The
best solutions deliver optimized
segmentation schemes that
substantially improve a model’s
precision, while maintaining a
transparent and interpretable
scoring solution for use with
regulators.
Remember that in order to validate
and track a segmented modeling
solution effectively, you will want to
evaluate the entire system, as well
as the individual segment models.
Financial institutions should select a model type appropriate for data type and decision area,
and one that will provide robust predictions. For both business and regulatory purposes, also
consider the following at model design time when choosing a model type:
• Transparency. Your model type should be easy to understand and explain, for both
regulators and customers. Look for interpretable features that allow you to identify and
explain what is driving a score result. A risk model should include reason codes, which many
regulators require you give to customers when an adverse action is taken based on the score.
• Palatability. Regulators will ask about model outcomes, so it is important that model
scores and reason codes have a high degree of face validity. Palatability is about intuition
4. Choose the Right Model Type
New tools, such as the Segmented Ensembles module in FICO® Model Central™ Solution, make the process of finding optimal segmentation schemes much quicker and easier. Segmented Ensembles automates the process of searching the countless combinations of segmentation variables, split points and sequencing to find the best segmented model system.
Figure 1: Automate the discovery of predictive segments
www.fico.com page 6
Comply and Compete: Model Management Best Practices
» insights
and common sense, not complex
mathematics. Does your model behave
intuitively from a business context and
is it directionally correct? For instance, as
length of good credit history increases,
does the risk score improve?
• Ease of engineering. During
development, you may need to engineer
or fine-tune a model to ensure it will
address your identified business goal. This
may require you to substitute or remove
predictive characteristics that may be
contentious in order to address regulatory
requirements or customer concerns.
You may wish to alter variable binnings
or apply pattern constraints to smooth
noisy data and improve the model’s
robustness.
Once you have developed a model, you need to validate that it works according to your business
objectives. You also need to revalidate on an ongoing basis—once a year at a minimum, but more
often in a dynamic economy. Validation evaluates your model’s behavior over a range of input
values and identifies any segments where the model has degraded.
Overall, you should:
• Strive for clarity, consistency. Regulators want to see that you validate on a consistent
basis, and that your process is repeatable. Regulators also want to know what threshold
metrics you’ve put in place and actions you are taking (such as more frequent reassessment,
recalibration or rebuilding) when a model falls below an identified threshold.
• Create a supervisory review. In the US, the OCC/Fed requires your validation processes
be reviewed by parties independent of those developing the model and designing and
implementing the validation process. Globally, Basel puts an equally strong emphasis
on governance. An independent reviewer should have the authority to challenge the
recommendations of model developers. In addition, the reviewer should be given the
authority to sign off on the final determination, in order to ensure that input is considered
carefully rather than summarily overruled.
• Never validate in a standalone environment. Models and the scores they produce
rarely operate in a vacuum; rather, they are intimately tied to business rules and decision
strategies. Test a model’s interactions with these other elements and simulate the impact of
an updated model with respect to your customer portfolio.
• Include standard performance measures. Your validation checklist should include
standard measures (K-S, divergence, ROC area, Gini coefficient, etc.), along with metrics that
ensure the model rank-orders by score range. Measure both the model performance, as well
as the score and attribute stability. You should evaluate this at least quarterly, and ideally
monthly, in order to quickly identify changes.
5. Validate Model Effectiveness
Figure 2: Ensuring transparency in model type
Characteristic Points
Length of Credit History in Months
Less than 12 12
12–23 35
24–47 60
48 or more 75
Number of Credit Accounts with Balance > 0
0–1 65
2 55
3–4 50
5–7 40
8+ 30
Scorecards are commonly used model types in risk modeling because they are highly transparent and interpretable. This sample section of a scorecard shows how points are assigned for different values within a category of information. It’s easy to see how each factor relates to an individual’s score.
www.fico.com page 7
Comply and Compete: Model Management Best Practices
» insights
Regulators want you to demonstrate that you fully understand the philosophy around your
validation process. You will need to document information such as the boundaries of your model’s
effectiveness and responses to a variety of market conditions.
Over time, many factors can impact model performance. These include shifts in population makeup or
behavior, economic changes, and changes to credit and collection policies. Regulators expect you to
monitor models on a continual basis so you can recalibrate and rebuild them in a timely manner and
modify your strategies accordingly.
6. Track Performance
Figure 3: Use a variety of measures to assess model performance
100%
0%HighLow SCORE
% O
F PO
PULA
TIO
N
Bads
Goods
How far apart?
How much overlap?
DIVERGENCE KOLMOGOROV-SMIRNOV STATISTIC(K-S)
RECEIVER OPERATING CHARACTERISTIC(ROC)
100%
0%hgiHwoL SCORE
CU
MU
LATI
VE %
OF
AC
CO
UN
TS
GoodsBads
K-S
100%
0%%001%0 CUMULATIVE % OF TOTAL POPULATION
CU
MU
LATI
VE %
OF
BAD
S
ROC Area
Random
Divergence, K-S and ROC area are three useful measures of a score’s predictive power. Divergence measures the separation of the score distributions between outcome classes (e.g., “good” vs. “bad” accounts). K-S quantifies the maximum separation between the score distributions. ROC measures how well the score classifies across the entire population.
Figure 4: Using dashboards for ongoing model health checks
Technology tools such as FICO® Model Central™ Solution can be used to get an overall view of the health of the models across the organization. This should include functionality such as automated alerts indicating shifts in model perform-ance or the distribution of scores and attributes.
www.fico.com page 8
Comply and Compete: Model Management Best Practices
» insights
Tracking outcomes is vital to understanding how well business strategies are performing. This
requires capturing what was known at the time of a decision, what actions were taken and what the
resulting outcomes were.
Automating the analysis provides faster feedback about predictions and assumptions, and makes
it easier to identify and adjust for emerging trends and market fluctuations. Alerts can be used to
identify when performance has shifted out of the target range.
For a more detailed overview of several key model tracking reports, review the FICO white paper:
Effective Tracking and Reporting Is Key to Precise Risk Management. Besides aiding in compliance,
regularly producing these reports allows you to quickly detect deteriorating model effectiveness and
emerging portfolio delinquency changes. This lets you modify approval and collection strategies more
quickly, and avoid future losses.
Anytime you override a score, regulators will require that you document and monitor that decision
carefully. Your overrides should be based on clear and consistent guidelines. Regulators will ask
questions such as: What is your cutoff for an override? What authority level do you require for override
approval? How many overrides are you doing every month? What is your policy for authorizing an
override?
Each override reason should be assigned an identifying code for tracking in order to evaluate an
underwriter’s decisions. Use codes that allow for efficient and effective analysis. Strive to eliminate
vague codes such as “general” or “miscellaneous.”
Reasons for high-side overrides (accounts that score above the cutoff, but are declined) should be
examined carefully to make sure you are not turning away potentially good customers and that no
disparate impact is evident. Disparate impact is a fairness test that looks at whether certain minority
groups are impacted by the decisions made
differently than the majority, even if the scoring
models themselves do not evaluate race or other
characteristics that define the minority.
You should also analyze reasons for low-side
overrides (accounts that scored below the cutoff,
but are approved). If the percentage is significant or
unexpected, you should re-evaluate your override
policy and follow up to see that it is being applied
correctly.
If an underwriter is constantly overriding a score,
find out why. Does the underwriter have the proper
understanding of how scores work? Or is the model
deteriorating at an extent to which the underwriter
feels it is no longer accurate?
When tracking the business performance of
overrides, a general rule of thumb is: An account
booked as a result of a low-side override should
perform no worse than one at the score cutoff.
7. Monitor Overrides
Figure 5: Monitor overrides with an Override Tracking Report
Override Code
Reasons for Decline Total Low Side High Side
1 Previous Derogatory (internal) 2 2
2 Previous Derogatory (external) 34 34
3 Debt Ratio high (>45%) 53 53
4 Debt Ratio Low (<10%) 21 21
5 Deposit Accounts w/ >$10K 3 3
6 Bank Customer >10 years 5 5
99 Miscellaneous 25 11 14
Total Overrides 143 40 103
The Override Tracking Report enables you to pinpoint the reason for overrides. Strive to eliminate vague codes such as “general” or “miscellaneous.” The report should be produced quarterly at a minimum, and more often in a volatile economy.
www.fico.com page 9
Comply and Compete: Model Management Best Practices
» insights
No matter how complex your decision strategies, regulators will expect you to explain and defend
them with empirical results. Regulators will want to know how you develop, track and implement
your strategies. You must also show the results of your strategies, including the realized losses, gains
and exposures arising from your decisions. Most importantly, regulators will want to know how you
balance the need to increase profits with the need to contain risk.
Carefully document all your strategy decisions, as well as changes to those strategies. You should
document what your subpopulations are, what actions you’ve taken and where cutoff scores are
applied. You must also demonstrate that your segments are homogeneous and actionable.
Regulators might also ask you to pull performance and score information on an isolated
subpopulation, which they define. They may ask you to pull a sample of your declines to make sure
you are not engaging in regulatory violations, such as disparate impact or redlining (discrimination
based on where a person lives).
Since decision strategies have become increasingly complex, with hundreds or even thousands of
nodes, an automated solution is essential to track such strategies, strategy changes and business
results. You also want a solution that can simulate various “what-if” scenarios so you can understand
the projected results of a decision, fine-tune decision strategies, balance risk and profits, and
optimize business performance.
Decision trees are visually complex, sometimes having hundreds or even thousands of nodes. Advanced visualization solutions allow strategies to be viewed in ways that make it easier to explain and defend them, both within an organization and to regulators. In this graphic, FICO’s Decision Graph offers multiple simplified views of the same collections treatment strategy.
Even in a simple strategy, multiple paths lead to the same action. Notice that both Letter and Phone Queue “interrupt” each other—in other words, they aren’t always adjacent on the decision tree graph. This makes it difficult to understand the pathways leading to a single action. This problem would be exacerbated in a real-world decision tree, which is 10–200 times this size.
Here, we see which paths lead to Phone Queue and which lead to Letter. Every action is presented only once on the graph, making the strategy more digestible.
Figure 6: Improved visualization tools aid in transparency of decision strategies
A strong strategy visualization tool also allows you to focus on a single action and see all the pathways that lead to that action.
8. Defend Decision Strategies
www.fico.com page 10
Comply and Compete: Model Management Best Practices
» insights
Regulators worldwide place tremendous importance on documentation and oversight. When a
regulator asks you for proof of when you last ran a model validation, who reviewed the results and
what action was taken, you need the right tools in place to quickly retrieve the supporting evidence.
With that in mind, you should keep an
inventory of every model within your
operating environment, cataloguing its
purpose, usage and restrictions on use.
List the types and sources of inputs. Your
documentation should be detailed enough
so that anyone unfamiliar with the model can
understand how it operates, its limitations and
your key assumptions. You also should be able
to retrieve documentation for any vendor-
supplied models, and demonstrate that you
understand it.
Your inventory should indicate if a model
is functioning properly. It should include a
description and dates of any updates, and a
list of policy exceptions. It should also include
names of individuals responsible for validation,
a list of validation plans, findings of validations
performed and any actions taken as a result.
You should also have a complete audit trail of
who modified a model and for what purpose,
with a traceable path to the outcome of
each modification. All annotations should
be digitally captured and attributed to an
individual, and the sequence of any changes
should be apparent.
By putting in place a technology that
automatically ensures documentation and
validation processes are managed correctly
and consistently, financial institutions can
ensure that highly trained analysts can
focus on ad hoc regulator queries and new
model developments, rather than being
consumed with producing standard validation
and tracking reports. And by centrally
documenting a model’s design and limitations,
you reduce the risk of misapplying a model.
9. Document Thoroughly
FICO® Model Central™ Solution provides a complete environment for managing predictive models in a reliable, automated and integrated way. The solution:
• Presents a management dashboard of overall model health, alerting personnel to performance degradation for action before business decisions are impacted.
• Creates a standardized process for easy management and monitoring of models, freeing resources to focus on other pressing business tasks and issues.
• Coordinates model validation, tracking tasks and management reporting, storing complete and annotated audit trails to satisfy compliance requirements.
• Deploys new models quickly and efficiently—up to 50% faster—speeding time to value and improving return on investment.
• Integrates models from various programming languages into one environment, further saving time and IT resources.
• Improves business outcomes dramatically with the integration of simulation and testing capabilities, and optimization of strategy decisions.
Learn more about FICO Model Central Solution.
Improving Model Management
ModelData Mart
Tracking
Monitoring
OngoingValidation
ManagementReporting
Alerts
DecisionSimulation
DecisionExecution
ScoringServices
DecisionOptimization
Development&
Calibration
Deployment&
Verification
ModelData Mart
ADVANCED
PR
OFESSIONAL
DEC
IS
IONING
DEV
ELOPMENT
FO
UN
DA
TION
FICO Model Central Solution is available in three tiers: Foundation for validation, monitoring, management reporting, alerting and administration; Professional Development, which includes Foundation services plus full model development and deployment capabilities; and Advanced Decisioning, which includes professional services and capabilities for testing and rapid learning adaptation.
Comply and Compete: Model Management Best Practices
» insights
The Insights white paper series provides briefings on best practices, research findings and product innovations from FICO. To subscribe, go to www.fico.com/insights.
Financial institutions today must operate in a highly regulated world. Although new regulations
come with their share of overhead, once a financial institution has the people, processes and
technology in place for proper model management and validation, it can go from merely
complying with new regulations to proactively improving model performance. Stronger models,
in turn, drive better decision making to improve business results.
» Conclusion
FICO, Model Central and “Make every decision count” are trademarks or registered trademarks of Fair Isaac Corporation in the United States and in other countries. Other product and company names herein may be trademarks of their respective owners. © 2014 Fair Isaac Corporation. All rights reserved.2811WP 04/14 PDF
For more information North America Latin America & Caribbean Europe, Middle East & Africa Asia Pacific www.fico.com +1 888 342 6336 +55 11 5189 8222 +44 (0) 207 940 8718 +65 6422 7700 [email protected] [email protected] [email protected] [email protected]