Proceedings of the IV Brazilian Conference on Neural Networks - IV Congresso Brasileiro de Redes Neuraispp. 269-274, July 20-22, 1999 - ITA, São José dos Campos - SP - Brazil

A Neural Network Based Solution for t he Credit Risk Assessment Problem

Germano C. VasconcelosPaulo J. L. Adeodato

Domingos S. M. P. MonteiroDepartamento de Informática,

Universidade Federal de Pernambuco,Cx. Postal 7851, 50732-970, Recife-PE, Brazil

E-mails: {gcv,pjla,dsmpm} @di.ufpe.br

Abstract

The automation of decision making in financialmarkets is one of the major application areas of neuralnetworks. Risk analysis is one of the problems where thetechnique has been efficiently applied. This paperinvestigates a solution to a credit analysis problem in arather peculiar environment, characterized by astabilized economy but subject to a high interest rate,namely the Brazilian market. A neural network basedcredit scoring system has been developed for the retailbusiness in Brazil and its performance has beenevaluated against that attained by a traditionaldiscriminant analysis system. Extensive experimentalresults carried out with a database of 18,000 consumersof a leading Brazilian supermarket chain clearlyindicate that a better solution is found with theconnectionist based system.

1. Introduction

In the globalized economy, efficiency and low costare fundamental aspects in making a product or servicebecome competiti ve in the market. The huge volume oftransactions turns the information processingautomation into a crucial factor for cost reduction, highspeed and high quality standards. The scientific andtechnological advance has modified the characteristicsof managerial work. Automation has reached activitiesnormally thought of as “ reserved for humans” and manyskeptics about this have changed their opinions as aresult of the relevant successes achieved by state-of-the-art computer solutions applied nowadays. In the past,people tended to think that financial market analysisrequires knowledge, experience and intuition andwondered how this activity could be automated.However, in developed countries the automation offinancial market analysis has been steadily growingalong with the scientific and technological advances [2].In Brazil, the automation of this task was not possibleuntil recently due to the high rates of inflation (at least10% per month). This has only become feasible after

the stabilization of the Brazilian economy with PlanoReal which lowered inflation to less than 10% per yearsince 1994. Since then several financial institutionshave already automated part of their decision supportsystems. But the demand for automation is stil l growingwith new services being offered (e.g. personal leasing)and the need to expand the services to a broader market.

A major application of neural network [8,9,10] basedsystems to finance is risk analysis [2,3,4,7]. It involvesthe estimation of several aspects concerning acredit/insurance applicant: credit limit, expected profit,mean time for tardy payments, concession of credit,insurance coverage, etc.

This paper focuses on the problem of credit analysisparticularly the decision on the concession of creditcards to customers [6]. A case study has been carriedout on the actual database of a leading supermarketchain in Brazil which has its own credit card operator.The company had already automated the creditevaluation process and was interested in improving itsperformance and flexibilit y. This case of study isparticularly interesting because of the peculiar aspectscurrently associated with the Brazil ian market : astabil ized economy with low inflation rates (less than1% per month) but with very high monthly interest rates(more than 5%). This has influenced the percentage ofbad payers in the market. The system presented here hasattained a better performance than the company’s creditscore system based on linear discriminant analysis on aset of customer independent data retained at thecompany for evaluation.

The paper is organized as follows. Section 2characterizes the credit concession problem. Section 3particularizes the problem to the actual case investigatedand Section 4 presents the solution adopted. Section 5presents the results of the neural network based systemand compares it to the existing credit score system.Finally, Section 6 presents final remarks on thelimitations of the solution achieved, on the future workfor improvement and on new desired features for such asystem.

2. The Credit Concession Problem

When a person or a company asks for financial creditthere are several ways to evaluate the application.Financial institutions define different cost functions tobe optimized by the decision system. They may wish tomaximize their profits, minimize the risks of non-payment or minimize the delays in repayments, etc.Nevertheless, the main goal is to decide whether or notto give credit to the applicant [7]. This paper willconcentrate on this aspect of the problem. The blockdiagram in Figure 1 illustrates the process.

In the context of the credit evaluation problemdescribed in this paper, the credit appli cant suppliesinformation through an application form which ischecked by the institution for the veracity of some fields(input variables) such as social security number, annualincome and permanent address (for personal loans).Additional financial information for that social securitynumber is then obtained by the institution from creditsecurity database bureaus (SPC, SERASA).

All the information about the applicant is fed to thecredit concession system which awards a scalar score tothat applicant. If the score is above a certain thresholddefined by the institution, the application is approved;otherwise, it is refused 1.

After the credit concession, the institution keeps trackof the applicant’s financial transactions and, accordingto the institution’s criteria, labels the customer as goodor bad payer [3].

REFUSED

BAD PAYERS

ACCEPTED

CREDITCONCESSION

SYSTEM

GOOD PAYERS

Figure 1

As a technique based on statistics, neural networks[1,8,9] requires a large database to extract knowledgefrom. However, usuall y this is not a problem once thatbefore automating the process, the institutions alwaysmake decisions relying on human expertise. So at thetime of the implementation of the automatic systems,

1 The rejection score interval corresponding to casessolved by human experts has been omitted from thispresentation

the institutions already have a large database with thehistory of their customers and their classification as debtpayers according to the institution’s poli cies.

The large database provided for extraction ofknowledge, however, is biased in the sense that onlythe applicants who were given credit are followed up asdebt payers. Therefore only one type of decision errorcan be measured in such a system: the concession ofcredit to bad payers. Considering that the creditconcession system maps a multidimensional space intoa scalar value (score), the types of error caused by thethresholded decision poli cy can be illustrated byFigure 2.

Figure 2 shows the hypothetical class-conditionalprobabilit y density functions for good (G) and bad (B)payers plotted against the score obtained in the creditconcession system implemented at the institution. Thethreshold (LB) defined by the poli cy of the institutiondetermines the decision error regions. The area ofregion EB2 measures the error of type-II correspondentto bad payers who received credit whereas the area ofregion EB1 measures the error of type-I correspondent togood payers who have not received credit. Errors oftype-I are not observable because of the bias in thedatabase.

B

P(G|cs)

P(B|cs)

Credit Score (cs)

EB1EB2

LB

G

Figure 2

If another credit concession system is to be comparedto the existing one for the biased database, two types oferror can be measured. This can be illustrated in theprevious plot with another threshold (LR) for the newsystem. Figure 3 presents the types of error that occurwhen the new threshold (LR) is higher than the existingone (LB). The area of region ER2 measures the error oftype-II correspondent to bad payers who would receivecredit. The area of region ER1 measures the error oftype-I correspondent to good payers who would notreceive credit but were good payers and were followedup (observable error).

Credit Score (cs)

G

B

P(G|cs)

ER2

ER1

LRLB

P(B|cs)

Figure 3

One way of evaluating the performances of thesystems is to compare the sum of the observable areasof error regions (ER1 + ER2) of the proposed system(Figure 3) against that (EB2) of the existing system(Figure 2). The system with smaller error would havethe better performance. Thus, the advantage of the newsystem can be defined as ∆ = EB2 - (ER1 + ER2) in thecomparison. This simple measure does not take intoaccount the cost of the types of error despite being wellknown that the cost of the error of giving credit to a badpayer (type-II) is much higher than that of not givingcredit to a good payer (type-I).

There are hypothetically two ways of achieving aperformance better than that of the existing system. Thefirst is to assume that the new system maps the inputvariables into a scalar producing class-conditionalprobabilit y density functions similar to those producedby the existing system also assuming that the existingthreshold is smaller than the optimal decision point(intersection of the curves). The new system could thenhave a higher threshold and reduce the overall error asillustrated in Figure 3. This situation however, may notbe the case; the threshold may be beyond the optimaldecision point, not giving credit to much more goodpayers than it should.

The second and more plausible solution is that thenon-linear mapping of the neural network based system(as opposed to the linear mapping of the lineardiscriminant analysis based system) produces class-conditional probabilit y density functions more separatedthan those produced by the existing system (with lessarea of intersection under the curves), thus reducing thedecision errors for thresholds near the optimal value.Figure 4 illustrates this situation.

Figure 4

3. The Actual Case Study

The actual problem to be solved was to evaluate theperformance of the neural network-based system on theretailer’s database for applicants who had receivedcredit in recent years (after Plano Real). All of them hadbeen approved by the system based on lineardiscriminant analysis. This financial institution is acredit card company with 500.000 customers.

The database consisted of a set of 18,000 customersfrom which 6,000 had their labels withheld by theinstitution for performance evaluation. This apparentlylarge database turned out to be small since from the12,000 labeled examples available for development,there were only 450 customer labeled as bad payers bythe institution criteria. This strong unbalance in theclasses (3.75% of bad payers and 96.25% of goodpayers) required a maximum performance error of3.75% from the proposed system.

The unbalance in the classes also suggested that theinstitution’s system was operating with a thresholdbeyond the optimum level; that is, to the right of theintersection of the distributions in Figure 1. But thatcould not be proved because the was no follow up onthe applicants with credit refused.

The system had also to cope with other featureswhich make the problem slightly more complex. Thereare missing data. Some fields in the application formneed not to be fill ed (non-obligatory). There are alsoseveral non-numerical fields in the form the applicantsfill . The truth of the information on the form is onlyverified for particular fields.

Credit Score (cs)

P(B|cs) P(G|cs)

4. The Solution Adopted

Given the complexity of the problem and the highaccuracy needed, the system could not be trivial toachieve the expected goal.

4.1. Data preparation

The data of this case of study contains the featuresgenerally found in data mining problems. Therefore itrequired several levels of data preparation: datacleansing, statistical data analysis, number scaling andattribute encoding [5,7].

The first problem encountered on the data was thelack of standard fill ing of attribute fields such as city,borough, profession etc. requiring a long non-automateddata cleansing.

Another difficulty faced was missing data which isinherent to this problem once that some fields in theapplication form need not to be fill ed (non-obligatory).Statistical analysis helped on completing these blanks.

The reliabil ity of the data is assured only for a fewparticular fields of the application form which havetheir veracity confirmed through documents.

There are also several non-numerical fields in theform the applicants fil l. Random variables converted theattributes into numerical values in a way to optimize thedata representation for the classifiers. The encoding wastypically 1-out-of-n for fields with few options (e. g.marital status). For fields with many different ways offill ing (e. g. town, profession etc.), statistical analysishelped in grouping all rare events in a single value. Inthis case, the k-out-of-n encoding was used; acompromise between the bad distance mapping of thedense binary code and the large dimension of the 1-out-of-n code.

Some fields (e. g. time in the present job: years andmonths) needed to be combined for expressing theirinformation in a simpler way (months).

Numerical data were all scaled to the fall in theinterval between zero (0) and one (1).

4.2. The architecture

Considering that the data made available by thecompany had only a small fraction of bad customers(only 3.75% of the total) and that no information aboutthe refused applications was supplied, the system hasbeen developed as a 2-stage classifier described belowand shown in Figure 5.

The first stage consisted of a k-nearest neighbors-like(k-NN) classifier to separate credit appli cants in twogroups: applicants "close" to the bad payers go forfurther evaluation whereas those "far" from bad payersare classified as good applicants to receive credit. The

output threshold of the this classifier can be adjustedaccording to the institutions poli cies to accept or refusecredit applications.

The credit appli cants clustered around the bad payersthen pass through a finely tuned Multi-Layer Perceptron(MLP) dedicated to the separation of a more restrictedset with a much more balanced expected ratio ofgood/bad payers.

ACCEPT

REF USE

First StageClassifier

Second StageClassifier

Figure 5

4.3. Training and testing

The system was trained on 6,000 examples and testedon the remaining 6,000 from the 12,000 made availableto the developers.

5. Results

The results were presented to the institution in theworksheet supplied to the developers with the 6,000unlabeled customers. For each customer, a classificationlabel was given indicating whether or not that customershould have received credit. To reach this binarydecision, the test set consisting of 6,000 examples wasused for the purpose of setting the threshold of the firststage classifier. The goal was to maximize advantage ofthe system in terms of number of applications (asdefined in section 2).

Figure 6 shows the process of selecting the thresholdbased on the test data. As the threshold increases, thesystems starts detecting bad payers in the set but soonthe system starts refusing credit to good payers. The aimis to optimize the difference between the number of badpayers detected and the number of good payers withcredit refused. The threshold for this optimal netnumber of customers in the test set was then used forthe deciding about the 6,000 customers on the setretained by the institution. In fact 3 different thresholdswere chosen to account for statistical variations on thedatabase once that the number of bad payers in thesample was too small.

-40

-30

-20

-10

0

10

20

30

40

50Good Refused (1)

Bad Detected (2)

Difference (2-1)

1-NN distance

Figure 6

The behavior observed in Figure 6 suggests that theneural network mapping into a scalar separates theclass-conditional probabilit y density functions of badand good payers from the biased database and that itallows the threshold to be placed either to the left or tothe right of its optimal value.

Table 1 below shows the performance (Det = baddetected, Ref = good refused, Dif = difference betweenthe two) of the system on both test set and the data setretained by the institution for the 3 thresholds chosen.The proposed system performed well both on the 6,000examples retained by the developers and on the 6,000retained by the institution. Should the cost of the errorsbe taken into account (as mentioned in section 3), theperformance of the neural network based system wouldhave been far better than has already been.

Table 1: System Performance

Test Set Retained SetDet Ref Di f Det Ref Di f

Thresh ol d- 1 10 0 10 17 0 17Thresh ol d- 2 15 0 15 23 2 21Thresh ol d- 3 17 6 11 25 6 19

After the institution’s evaluation of the neuralnetwork based system, the developers had access to theactual status of the payers in the retained data set.Figure 7 shows the results obtained on that data set inthe same way as shown in Figure 6 for the test set. Thesimilar behavior observed confirms the power of thesolution offered and the statistical representativeness ofthe sample.

Good Refused (1)

Bad Detected (2)

Difference (2-1)

0

5

10

15

20

25

30

1-NN distance

Figure 7

An interesting feature of the new system wasobserved some months later. Several customers whohad been mistakenly classified as bad payers by the newsystem (classification errors) became bad payers afterthe initial evaluation of the system. This power could bebetter analyzed to predict when a customer is going toturn into a bad payer if more information is madeavailable.

6. Final Remarks

This article has presented a neural network basedsystem for automatic support to credit analysis in a realworld problem. The proposed system reached a higherperformance than the existing one based on lineardiscriminant analysis despite the huge unbalance amongthe classes which required a maximum of less than 4%classification error from the proposed system. That wasnot an unexpected result considering that k-NN andneural networks implement non-linear mappingswhereas the existing system is constrained to linearones.

The proposed system is also better in terms offlexibilit y for adapting to gradual changes in theapplicants’ behavior during the usage of the system. Theexisting system, however, is still important forexplaining the reasons for the decisions made and foradapting to drastic changes in the economy (economiclaw changes, stock market crashes, etc.).

Since the proposed system achieved a performancebetter than the credit score system, the institution isgoing to operate both systems in parallel giving credit tothe applicant whenever one of the systems decides for itand the other does not refuse strongly. Thus, a muchbetter performance evaluation wil l be carried out. Also,an automatic combination of both systems through aneural network can be used for a better decision makingwhere the best of each system will be taken intoaccount.

It should be emphasized that the second condition“ ...and the other does not refuse strongly.” for parallel

operation is crucial for the neural network based systembecause it has not been trained on any data fromapplicants with credit score below the threshold. Thisguarantees that the company’s credit score system wil lprevail in those cases. The proposed solution has onlybeen tested in a cascade combination of the twosystems. The developers have asked the institution tocreate a database for storing data about the refusedapplications for them to develop a more general andpowerful neural network based system under theunsupervised learning paradigm.

References

[1] D. Rumelhart, B. Widrow, and M. Lehr,The basic ideas in neural networks,Communications of The ACM, Vol. 37, (3)1994.

[2] The Economist, pp.75-77, 15 April 1995.[3] E. Gately, Neural Networks for Financial

Forecasting, John Wiley & Sons, Inc., 1996.[4] A. Ainslie and X. Drèze, Data Mining: Using

Neural Networks as a Benchmark for ModelBuilding, 1996.

[5] K. M. Decker and S. Focardi, TechnologyOverview: A Report on Data Mining,. CSCS-ETH, 1995.

[6] W. K. Schrickel Credit Analysis,. Atlas, 1995 (inPortuguese).

[7] J. P. Bigus, Data Mining with Neural Networks,McGraw-Hill, 1996.

[8] S. Haykin, Neural Networks: A ComprehensiveFoundation. McMil lan College PublishingCompany Inc, 1994.

[9] R. Beale and T. Jackson, Neural Computing: AnIntroduction, Adam Hilger, IOP Publishing Ltd,1991.

[10] D. Rumelhart, B. Widrow, and M. Lehr,Neural networks: applications in industry,business and science, Communications of TheACM, Vol. 37, (3) 1994.