performance evaluation of fuzzy rule-based systems with ......gerald schaefer hisao ishibuchi aston...

Performance Evaluation of Fuzzy Rule-Based Systems

with Class Priority for Medical Diagnosis Problems

Tomoharu Nakashima Yasuyuki Yokota

Osaka Prefecture University Osaka Prefecture University

Naka-ku Gakuen-cho 1-1, Sakai Naka-ku Gakuen-cho 1-1, Sakai

Osaka 599-8531, Japan saka 599-8531, Japan

[email protected] [email protected]

Gerald Schaefer Hisao Ishibuchi

Aston University Osaka Prefecture University

Aston Triangle Naka-ku Gakuen-cho 1-1, Sakai

Birmingham B4 7ET, U.K. Osaka 599-8531, Japan

[email protected] [email protected]

Abstract—In this paper we examine the performance of

fuzzy rule-based systems with classification priority for

medical diagnosis problems. The assumption in this pa-

per is that a classification priority is given a priori for each

class in a pattern classification problem. Our fuzzy rule-

based system consists of a set of fuzzy if-then rules that are

automatically generated from a set of given training pat-

terns. The consequent class of fuzzy if-then rules are de-

cided based on the number of covered training patterns for

each class. We apply the fuzzy classifier with class prior-

ity to two medical diagnosis problems: appendix diagnosis

and breast cancer diagnosis, and compare its performance

with that of a conventional fuzzy rule-based systems.

I. Introduction

While in the past fuzzy rule-based systems have beenapplied mainly to control problems [1], [3], more recentlythey have also been used in pattern classification prob-lems. There are many approaches to the automatic gen-eration of fuzzy if-then rules from numerical data for pat-tern classification problems [4]-[9].

Pattern classification research typically focusses on theminimisation of misclassification. However, in real worldproblems it is also necessary to consider differenct casesof misclassification. For example, in the case of a creditapproval problem where the task is to decide whetherto give an applicant a loan or not, if the applicant isnot appropriate for the loan but was decided to be givena loan, the loss incurred by this misclassifcation is thefinanced money plus any interest. On the other hand, ifthe applicant would have been able to pay back but wasnot given the loan, the incurred loss by this misjudgementis just the amount of lost interest.

Let us consider another example in which different mis-classification scenarios need to be taken into account. Inmedical diagnosis of cancer two kinds of misclassifica-tions have to be considered. One is a misclassification ofa benign tumor as malignant while the other is a malig-nant tumor mistakenly diagnosed as benign. Obviously,

the number of occurances of both misclassification casesshould be as small as possible. However, it is obviousthat the latter case (malignant tumor diagnosed as be-nign) is more serious than the first case (benign tumordiagnosed as malignant).

In order to handle situations such as the ones men-tioned the concept of misclassification cost can be in-troduced [10]. Several researchers have developed algo-rithms that consider classification cost in the construc-tion of classification systems. For example, Domingos[11] proposed MetaCost which converts error-based clas-sification systems into cost-sensitive ones. In MetaCost,multiple error-based classification systems are generatedfrom multiple sets of resampled training patterns withreplacement. The final classification for an unknown pat-tern is made by voting among the generated classifiers.MetaCost can be viewed as a wrapper approach to cost-sensitive pattern classification because it can be appliedto any classification system.

There is still a problem that often cost-sensitive classi-fication is performed under the assumption that misclas-sification costs are given numerically. In real world prob-lems however, it poves difficult to estimate these numer-ical costs. Thus most cost-sensitive learning algorithmscannot be applied to many real world problems.

In order to overcome this problem, we employ an as-sumption that can be more applicable to such problems,namely that only the classification priority is given foreach class. For the example of medical diagnosis, thecorrect identification of malignant tumors is more impor-tant than that of benign ones. Expressed differently, theclassification priority for malignant cases is higher thanthat of benign cases so that the misclassification of ma-lignant tumors as benign is minimised even though theother misclassifications may increase.

In this paper we will evaluate the effectiveness of fuzzyrule-based systems with classification priority for two

Proceedings 21st European Conference on Modelling and SimulationIvan Zelinka, Zuzana Oplatková, Alessandra Orsoni ©ECMS 2007ISBN 978-0-9553018-2-7 / ISBN 978-0-9553018-3-4 (CD)

1.0

1.0

0.0

(a) Two fuzzy sets

Attribute value

Membership

Membership

Membership

Membership

1.0

1.0

0.0

(b) Three fuzzy sets

Attribute value

1.0

1.0

0.0

(c) Four fuzzy sets

Attribute value1.0

1.0

(d) Five fuzzy sets

Attribute value

A1 A2 A4 A5A3

Fig. 1. An example of antecedent fuzzy sets.

medical diagnosis problems. The reminder of the paper isorganised as follows: First we explain conventional fuzzyrule-based systems in Section II. Next, the descriptionof fuzzy rule-based systems with classification priority isprovided in Section III. We the examine the performanceof the fuzzy rule-based systems for the medical diagnosisproblems in Section IV while we conclude the paper withSection V.

II. Conventional Fuzzy Classification System

In the literature various methods have been proposedfor fuzzy classification [12]-[17]. In this paper we use thefuzzy-rule generation method proposed by Ishibuchi et

al. [14] as the basis of our classification system.

A. Pattern Classification Problems

Let us assume that our pattern classification prob-lem is an n-dimensional problem with C classes and m

given training patterns xp = (xp1, xp2, . . . , xpn), p =1, 2, . . . ,m. Without loss of generality each attribute ofthe given training patterns is normalised into a unit inter-val [0, 1]. That is, the pattern space is an n-dimensionalunit hypercube [0, 1]n.

We use fuzzy if-then rules of the following type as basisof our classification system:

Rule Rj : If x1 is Aj1 and . . . and xn is Ajn

then Class Cj with CFj ,

j = 1, 2, . . . , N, (1)

where Rj is the label of the j-th fuzzy if-then rule,Aj1, . . . , Ajn are antecedent fuzzy sets on the unit in-terval [0, 1], Cj is the consequent class (i.e. one of the C

given classes), CFj is the grade of certainty of the fuzzyif-then rule Rj , and N is the total number of rules. Asantecedent fuzzy sets we use triangular sets as in Figure 1where we show various partitions of the unit interval intoa number of fuzzy sets.

B. Generating Fuzzy If-Then Rules

In our classification systems, we specify the consequentclass and the grade of certainty of each fuzzy if-then rulefrom the given training patterns [14]. In [18] it is shownthat the use of the grade of certainty in fuzzy if-thenrules allows us to generate comprehensible fuzzy rule-based classification systems with high classification per-formance.

The consequent class Cj and the grade of certaintyCFj of fuzzy if-then rules are determined in the follow-ing manner:

[Generation Procedure of Fuzzy If-Then Rule]Step 1: Calculate βh(Rj) for Class h (h = 1, 2, . . . , C) as

βh(Rj) =∑

xp∈Class h

µj1(xp1) · . . . ·µjn(xpn), (2)

where βh(Rj) is the sum of the compatibilityof training patterns from Class h with Rj , andµji(·) is the membership function of the fuzzy setAji.

Step 2: Find Class h that has the maximum value ofβh(Rj):

βh(Rj) = max{β1(Rj), . . . , βC(Rj)} (3)

If two or more classes take the maximum value,the consequent class Cj of the rule Rj can notbe determined uniquely. In this case, specify Cj

as Cj = φ.Step 3: If a single class takes the maximum value (i.e. if

Cj 6= φ), let Cj be Class h and specify the gradeof certainty CFj as

CFj =β

h(Rj) − β

C∑

h=1

βh(Rj)

, (4)

where

β =

∑

h6=h

βh(Rj)

c − 1. (5)

The number of fuzzy if-then rules depends on how eachattribute is partitioned into fuzzy sub-sets. For example,when we divide each attribute into three fuzzy sub-setsin a ten-dimensional pattern classification problem, thetotal number of fuzzy if-then rules is 310 = 59049. Thegrade of certainty CFj can be adjusted by a learningalgorithm [10].

C. Fuzzy Reasoning

Using the rule generation procedure outlined above wecan generate fuzzy if-then rules as in (1). After both theconsequent class Cj and the grade of certainty CFj aredetermined for all N rules, a new pattern x is classified

by the following procedure:

[Fuzzy Reasoning Procedure for Classification]Step 1: Calculate αh(x) for Class h, h = 1, 2, . . . , C, as

αh(x) = max{µj(x) · CFj |Cj = Class h}, (6)

where

µj(x) = µj1(x1) · . . . · µjn(xn). (7)

Step 2: Find Class h′ that has the maximum value ofαh(x):

αh′(x) = max{α1(x), . . . , αC(x)}. (8)

If two or more classes take the maximum value,then the classification of x is rejected (i.e. x isleft as an unclassifiable pattern), otherwise as-sign x to Class h′.

III. Proposed Fuzzy Classification

In this section we describe a fuzzy rule-generationmethod for constructing cost-sensitive fuzzy classificationsystems that explicitly utilise costs of training patternsin the rule-generation process.

The assumption in this paper is that a classificationpriority is given a priori together with a set of trainingpatterns. In the generation of fuzzy if-then rules, first wecount the number of covered training patterns by a fuzzyif-then rule for each class. A training pattern is coveredby a rule if the compatibility of the training pattern withthe rule is larger than zero. Then the consequent classis determined as the class with the highest priority thathas at least one covered training patterns.

As in the last section let us assume that we have m

training patterns xp, p = 1, 2, . . . ,m for an n-dimensionalC-class pattern classification problem. We also assumethat a classification priority rc is given a priori for Class c,c = 1, 2, . . . , C, where rc is a natural number and the clas-sification priority decreases as the value increases. Theprocedure of generating a fuzzy if-then rule Rj with classpriorities rc, c = 1, 2, . . . , C, is summarized as follows:

[Generation Procedure of Fuzzy If-Then Rule]Step 1: Count the number of covered training patterns

njc from Class c by the j-th fuzzy if-then rule Rj .

The p-th training pattern xp is covered by Rj ifthe following equation holds:

µj(xp) > 0.0, (9)

where

µj(xp) = µj1(xp1) ·µj2(xp2) · . . . ·µjn(xpn). (10)

Step 2: Find Class h that has the highest classificationpriority among the class with nj

c > 0:

h = arg minc

{rc|njc > 0, c = 1, 2, . . . , C}. (11)

Step 3: Specify the grade of certainty CFj as

CFj =β

h(Rj)

C∑

h=1

βh(Rj)

. (12)

We modified the specification of the degree of certaintyin (12) as the conventional specification in (4) makes thedegree of certainty negative in some cases.

The fuzzy reasoning procedure for classifying unseenpatterns is exactly the same as the conventional fuzzyrule-based classification described in Subsection II-C.

IV. Computational Experiments

A. Two-dimensional synthetic problem

First we show the effect of introducing classificationpriority for a two-dimensional synthetic pattern classi-fication problem shown in Figure 2 where 1,000 circlepatterns (◦) are uniformly distributed in the unit space[0, 1]2 except in the subspace [0, 0.1]2. On the other hand,square patterns (2) are normally distributed with a meanvector of (0, 0) and a variance of 0.12 for both attributes.It is assumed that the classification priority of squarepatterns is higher than that of circle patterns.

x1

x2

1

0 1

Fig. 2. Two-dimensional synthetic problem.

We examined the classification boundaries obtainedfrom applying our proposed method by changing thenumber of fuzzy partitions for each attribute and com-pare it then to the boundaries generated by the conven-tional fuzzy rule-based systems described in Section II.

The obtained classification boundaries by the proposedmethod and the conventional method are shown in Fig-ure 3 and Figure 4 respectively. We can see that allsquare patterns are correctly classified by the proposedmethod for all the four fuzzy partitions in Figure 3. Wecan also see from Figure 4 that all square patterns aremisclassified when the number of fuzzy partitions is two(L = 2) although the number of misclassification de-creases as the number of fuzzy partitions increases. Theclassification priority is hence successfully incorporatedin our proposed fuzzy rule-based classification system.

(a) L=2 (b) L=3

(c) L=4 (d) L=5

1

0 1

x2

x1

1

0 1

x2

x1

1

0 1

x2

x1

1

0 1

x2

x1

Fig. 3. Classification boundaries by the proposed method.

(a) L=2 (b) L=3

(c) L=4 (d) L=5

1

0 1

x2

x1

1

0 1

x2

x1

1

0 1

x2

x1

1

0 1

x2

x1

Fig. 4. Classification boundaries by the conventional method.

B. Medical diagnosis problems

We then applied the fuzzy classifiers to two medical di-agnosis problems: appendix and breast cancer (availablefrom UCI Machine Learning Repository [19]). Table Ilists the details of the datasets.

Since the classification priority is not given in the datasets in Table I, we specify it manually in order to generatea synthetic situation where the classification priority inthe problems is given a priori. For medical diagnosisproblems it is natural to assume that the classificationof malignant patterns has higher priority than that of

TABLE I: Medical diagnosis problems.

Data sets Classes Patterns AttributesAppendix 2 106 7

Breast cancer 2 683 9

benign patterns. Thus, we assign the higher classificationpriority to malignant patterns and the lower classificationpriority to benign patterns (i.e. r1 = 1 for malignant andr2 = 2 for benign in Section III).

We specified the number of fuzzy partitions for eachattribute as L = 3, 4, . . . , 15. Since both the conventionaland the proposed fuzzy rule-based classification systemsare deterministic, we need to examine the performance ofthe two fuzzy rule-based classification systems just once.We show the classification results of the proposed methodin Table II for the appendix diagnosis and in Table III forthe breast cancer diagnosis. Tables IV and V shows theclassification results obtained by the conventional fuzzyclassifierfor comparison. All tables show the classificationaccuracies for the entire data set, for malignant patterns,and for benign patterns. Bold numbers in Tables II andIII indicate that better results were obtained comparedto the corresponding results obtained by the conventionalfuzzy rule-based classification system in Tables IV and V.

We can see that 100% classification accuracy for malig-nant patterns with the higher classification priority wasobtained by the proposed method for all the fuzzy par-titions L = 3, 4, . . . , 15 although the classification accu-racy for benign patterns by the proposed method is nothigher than that of the conventional method. This isbecause the classification priority of malignant cases ishigher than that of benign ones. Thus, the proposedclassifier focusses on the correct classification of malig-nant patterns more than that of benign patterns.

Next, we examined the performance of the proposedfuzzy rule-based classification systems on test data. Ten-fold cross-validation was used where the data set is di-vided into ten disjoint subsets, one subset used as testdata and the other nine subsets used as training pat-terns. A single trial is completed when all ten subsetshave been used as test data once. Again, we examined theperformance of the proposed and the conventional fuzzyrule-based classification systems with the fuzzy partitionsL = 3, 4, . . . , 15. Tables VI and Table VII show the clas-sification results of our proposed classification systemswhile Tables VIII and IX give results for the conven-tional classifiers. Again. bold numbers show where supe-rior results were obtained compared to the correspondingconventional fuzzy classification system.

It is apparent from Tables VI to IX that the classifica-tion priority is successfully incorporated in the proposedfuzzy rule-based classification systems not only for train-ing data but also for test data.

TABLE II: Classification accuracy by the proposed method for

appendix training data.

L Total malignant benign3 37.74 100 22.354 38.68 100 23.535 64.15 100 55.296 59.43 100 49.417 75.47 100 69.418 70.75 100 63.539 86.79 100 83.5310 87.74 100 84.7111 89.62 100 87.0612 96.23 100 95.2913 94.34 100 92.9414 98.11 100 97.6515 98.11 100 97.65

TABLE III: Classification accuracy by the proposed method for

breast cancer training data.

L Total malignant benign3 80.82 100 45.194 88.87 100 68.205 90.04 100 71.556 93.85 100 82.437 96.19 100 89.128 94.58 100 84.529 94.14 100 83.2610 99.71 100 99.1611 94.29 100 83.6812 95.46 100 87.0313 98.10 100 94.5614 95.46 100 87.0315 95.46 100 87.03

V. Conclusions

In this paper we proposed and examined the perfor-mance of fuzzy rule-based systems with classification pri-ority for medical diagnosis problems. The assumptionhere is that the classification priority is given a priori foreach class. The results of the computational experimentsshowed the effectiveness of the proposed classifier.

References

[1] Lee. 1990, “Fuzzy Logic in Control Systems: Fuzzy LogicController Part I and Part II,” IEEE Trans. Syst., Man, Cy-beretics, Vol. 20:404–435.

[2] C.T. Leondes. 1999, “Fuzzy theory Systems,” Techniques andApplications. Academic Press, San Diego, Vol.1–4, 1999.

[3] M. Sugeno. 1985, “An Introductory Survey of Fuzzy Control,”Information Science, Vol. 30, No. 1/2:59–83.

[4] H. Ishibuchi, T. Nakashima, M. Nii, “Fuzzy IF-THEN Rulesfor Pattern Classification,” Fuzzy If-Then Rules in Compu-tational Intelligence: Theory and Applications, Kluwer Aca-demic Publishers, pp. 267–295, May 2000.

[5] H. Ishibuchi, T. Nakashima, T. Morisawa, ”Voting in FuzzyRule-Based Systems for Pattern Classification Problems,”Fuzzy Sets and Systems, Vol.103, No.2, pp.223–238, April1999.

TABLE IV: Classification accuracy by the conventional method

for appendix training data.

L Total malignant benign3 79.25 0 98.824 85.85 38.10 97.655 88.68 47.62 98.826 90.57 57.14 98.827 92.45 61.90 1008 94.34 71.43 1009 93.40 66.67 10010 93.40 66.67 10011 94.34 71.43 10012 95.28 76.19 10013 96.23 80.95 10014 96.23 80.95 10015 95.28 76.19 100

TABLE V: Classification accuracy by the conventional method

for breast cancer training data.

L Total malignant benign3 95.46 98.42 89.964 96.19 98.20 92.475 96.49 98.65 92.476 96.49 98.87 92.057 96.63 98.42 93.318 96.34 98.42 92.479 96.34 98.65 92.0510 100 100 10011 96.63 99.10 92.0512 97.07 98.87 93.7213 98.24 99.55 95.8214 97.22 99.10 93.7215 97.22 99.10 93.72

[6] H. Ishibuchi, T. Nakashima. 1999a, “Performance evaluationof fuzzy classifier systems for multi-dimensional pattern classi-fication problems,” IEEE Trans. on Syst., Man, Cybernetics,Part B Vol. 29:601–618.

[7] H. Ishibuchi, T. Nakashima. 1999b, “Improving the perfor-mance of fuzzy classifier systems for pattern classificationproblems with continuous attributes,” IEEE Trans. on Indus-trial Electronics, Vol. 46, No. 6:1057–1068.

[8] H. Ishibuchi, K. Nozaki, N. Yamamoto, H. Tanaka. 1995, “Se-lecting fuzzy if-then rules for classification problems using ge-netic algorithms,” IEEE Trans. on Fuzzy Systems, Vol. 3,No. 3:260–270.

[9] Y. Yuan, H. Zhang. 1996, “A genetic algorithms for generatingfuzzy classification rules,” Fuzzy Sets and Systems, Vol. 84,No. 1:1–19.

[10] R. O. Duda, P. E. Hart, D. G. Stork, Pattern Classification,Wiley Interscience, 2001.

[11] P. Domingos, “Metacost: A General Method for MakingClassifiers Cost Sensitive,” In Proceedings Fifth InternationalConference on Knowledge Discovery and Data Mining, 1999,pp. 155–164.

[12] K. Nozaki, H. Ishibuchi, H. Tanaka. 1996, “Adaptive fuzzyrule-based classification systems,” IEEE Trans. on Fuzzy Sys-tems, Vol. 4, No. 3:238–250.

[13] G. J. Klir, B. Yuan. 1995, Fuzzy Sets and Fuzzy Logic,Prentice-Hall.

[14] H. Ishibuchi, K. Nozaki, H. Tanaka. 1992, “Distributed repre-

TABLE VI: Classification accuracy by the proposed method for

appendix test data.

L Total malignant benign3 36.73 95.00 22.364 35.00 90.00 21.535 63.27 85.00 57.786 55.64 75.00 51.117 67.09 66.67 67.508 63.09 68.33 62.649 77.37 63.33 81.2510 75.37 61.67 79.0311 73.73 53.33 78.7512 82.18 58.33 88.4713 78.18 38.33 87.2214 83.91 60.00 90.6915 81.09 53.33 88.33

TABLE VII: Classification accuracy by the proposed method for

breast test data.


sentation of fuzzy rules and its application to pattern classifi-cation,” Fuzzy Sets and Systems, Vol. 52, No. 1:21–32.

[15] M. Grabisch. 1996, “The representation of importance and in-teraction of features by fuzzy measures,” Pattern RecognitionLetters, Vol. 17:567–575.

[16] M. Grabisch, F. Dispot. 1992, “A comparison of some methodsof fuzzy classification on real data,” Proc. of 2nd Intl. Conf.on Fuzzy Logic and Neural Networks, 659–662.

[17] M. Grabisch, and J.-M. Nicolas. 1994, “Classification by fuzzyintegral: performance and tests,” Fuzzy Sets and Systems,Vol. 65, No. 2/3:255–271.

[18] H. Ishibuchi, T. Nakashima. 2001, “The Effect of rule weightsin fuzzy rule-based classification systems,” IEEE Trans. onFuzzy Systems, Vol. 9, No. 4:506–515.

[19] D.J. Newman, S. Hettich, C.L. Blake, and C.J. Merz,“UCIRepository of machine learning databases,”1998.

TABLE VIII: Classification accuracy by the conventional method

for Appendix (Test data).

L Total malignant benign3 78.46 0 97.784 84.09 30.00 97.785 86.82 40.00 97.646 84.00 40.00 94.177 83.09 35.00 94.178 85.09 55.00 93.069 87.73 60.00 95.2810 85.00 50.00 94.1711 84.18 50.00 93.0612 86.00 55.00 94.1713 84.09 35.00 95.5614 87.82 55.00 96.6715 83.09 50.00 91.94

TABLE IX: Classification accuracy by the conventional method

for Breast W (Test data).


performance evaluation of fuzzy rule-based systems with ......gerald schaefer hisao ishibuchi aston...

Documents