the validation of credit rating and scoring models … · the validation process literature review...

The validation processLiterature review

Methodological proposals

The validation of Credit Rating and ScoringModels

Raffaella Calabrese

[email protected] of Milano-Bicocca

Swiss Statistics MeetingGeneva, SwitzerlandOctober 29th, 2009

Raffaella Calabrese Validation of internal rating systems

Outline

1 The validation process

2 Literature reviewCumulative Accuracy Profile CurveReceiver Operating Characteristic Curve

3 Methodological proposalsCurve of Classification Error Costs and Error Costs

Outline

Preliminary definitions

Credit rating and scoring models estimate the creditobligor’s worthiness and provide an assessment of theobligor’s future status.

The discriminatory power of a rating or scoring modeldenotes its ability to discriminate ex ante betweendefaulting and non-defaulting borrowers.

The validation process assesses the discriminatory powerof a rating or scoring model

Cumulative Accuracy Profile CurveReceiver Operating Characteristic Curve

Each borrower is characterized by two random variables:

the score S assigned to the borrower is a continuous r. v.with support (−∞,∞)

the Bernoulli r.v. B represents the borrower’s state at theend of a fixed time-period

1, the borrower’s state is default (d );0, the borrower’s state is non default (n).

The conditional distribution functions of S given a value of Bare denoted respectively by Fd(·) and Fn(·).

The distribution function of the score S is

F (s) = pFd(s) + (1 − p)Fn(s)

where p is the probability of default p = P[B = d ].

The accuracy (AC) is

AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)

The distribution function of the score S is

F (s) = pFd(s) + (1 − p)Fn(s)

where p is the probability of default p = P[B = d ].

The accuracy (AC) is

AC = pFd(s) + (1 − p)[1 − Fn(s)] = 2pFd(s) − F (s) + (1 − p)

Actual default Actual non defaultPredicted default True Default (TD) False Default (FD)(below s) Type II errorPredicted non default False Non Default (FN) True Non Default(above s) Type I error (TN)

hit rate F̂d(s) =TDNd

false alarm rate F̂n(s) =FDNn

F̂ (s) = p̂F̂d(s) + (1 − p̂)F̂n(s) =TD + FDNd + Nn

where p̂ =Nd

Nd + Nn

where p̂ =Nd

Nd + Nn

where p̂ =Nd

Nd + Nn

where p̂ =Nd

Nd + Nn

Cumulative Accuracy Profile (CAP) Curve andAccuracy Ratio (AR)

curve: CAP(u) = Fd [F−1(u)], u ∈ (0, 1)

synthetic index (BCBS, 2005):

AR =aR

aR + aQAR ∈ [0, 1]

optimal cut-off score (Hong, 2009): the intersection of theCAP curve and the iso-performance tangent line

Fd(s) =1

2p[F (s) + AC + p − 1]

drawbacks:- dependence on the sample relative frequency of defaulted

borrowers;- the type II error and the costs of wrong classification are

ignored.

AR =aR

aR + aQAR ∈ [0, 1]

Fd(s) =1

2p[F (s) + AC + p − 1]

ignored.

AR =aR

aR + aQAR ∈ [0, 1]

Fd(s) =1

2p[F (s) + AC + p − 1]

ignored.

AR =aR

aR + aQAR ∈ [0, 1]

Fd(s) =1

2p[F (s) + AC + p − 1]

ignored.

Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)

Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)

Receiver Operating Characteristic (ROC) Curve andArea Under the Curve (AUC)

Curve: ROC(u) = Fd [F−1(u)], u ∈ (0, 1)

Synthetic index (BCBS, 2005):

0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]

Optimal cut-off score (Hong, 2009): the intersection of theROC curve and the iso-performance tangent line

Fd(s) =1 − p

pFn(s) +

(AC + p − 1).

drawbacks: the costs of wrong classification are ignored.

0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]

Fd(s) =1 − p

pFn(s) +

(AC + p − 1).

0Fd [Fn(s)]dFn(s) AUC ∈ [0.5, 1]

Fd(s) =1 − p

pFn(s) +

(AC + p − 1).

Methodological proposalsCurve of Classification Error Costs and Error Costs

Curve of Classification Error Costs (CEC) and ErrorCosts (EC)

Curve:

C[u] =CFN

2{1 − Fd [F−1(u)]} +

2Fn[F−1(u)] u ∈ (0, 1)

Synthetic index:

EC∗ =

0C[F (s)]dF (s)

EC =EC∗ − EC∗

P − EC∗

REC ∈ [0, 1]

where EC∗

R and EC∗

P are respectively the error costs of therandom and perfect models.

Curve:

C[u] =CFN

2{1 − Fd [F−1(u)]} +

2Fn[F−1(u)] u ∈ (0, 1)

Synthetic index:

EC∗ =

0C[F (s)]dF (s)

EC =EC∗ − EC∗

P − EC∗

REC ∈ [0, 1]

where EC∗

R and EC∗

P are respectively the error costs of therandom and perfect models.

Optimal cut-off score: the value s that satisfies

2[1 − Fd(s)] +

2Fn(s)

= maxs

CFD−

Point measure (Zenga, 2007): U(c) =

µ (c)+µ (c)

where c ∈ (−∞,+∞) and

µ (c) =1

−∞

2[1 − Fd(s)] +

2Fn(s)

dF (s)

+µ (c) =

11 − F (c)

2[1 − Fd(s)] +

2Fn(s)

dF (s)

Optimal cut-off score: the value s that satisfies

2[1 − Fd(s)] +

2Fn(s)

= maxs

CFD−

Point measure (Zenga, 2007): U(c) =

µ (c)+µ (c)

where c ∈ (−∞,+∞) and

µ (c) =1

−∞

2[1 − Fd(s)] +

2Fn(s)

dF (s)

+µ (c) =

11 − F (c)

2[1 − Fd(s)] +

2Fn(s)

dF (s)

Bibliography

Basel Committee on Banking Supervision (2005). Studies on theValidation of Internal Rating Systems. Working paper 14. Basel,BIS.

Hong C. S. (2009). Optimal Threshold from ROC and CAPCurves. Communications in Statistics, 38, 2060-2072.

Zenga M. (2007). Inequality curve and inequality index based onthe ratio between lower and upper arithmetic means. Statistica &Applicazioni, V, 3-27.

the validation of credit rating and scoring models … · the validation process literature review...

Documents