iwlcs'2007: fuzzy-ucs: preliminary results

Fuzzy-UCS: Preliminary Results

Albert Orriols-Puig1,2Albert Orriols PuigJorge Casillas2

Ester Bernadó-Mansilla1

1Research Group in Intelligent SystemsEnginyeria i Arquitectura La Salle, Ramon Llull University

2Dept. Computer Science and Artificial IntelligenceUniversity of Granada

Motivation

Michigan-style LCSs for supervised learning. Eg. XCS and UCS

Evolve online highly accurate models– Evolve online highly accurate models

– Competitive to the most-used machine learning techniques• (Bernadó et al, 02; Wilson, 02; Bacardit & Butz, 04; Butz, 06; Orriols & Bernadó, 07)

Main drawback: Intepretability of the rule sets– Number of rules or classifiers

• Reduction schemes (Wilson, 02; Fu & Davis, 02; Dixon et al., 03)

Intervalar representation of continuous attributes: [l u ]– Intervalar representation of continuous attributes: [li, ui] . Semantic-free variables

Slide 2GRSI Enginyeria i Arquitectura la Salle

Framework

Genetic Fuzzy SystemsChange the rule representation to fuzzy rules– Change the rule representation to fuzzy rules

– Provide a robust, flexible, and powerful methodology to deal with noisy imprecise and incomplete datanoisy, imprecise, and incomplete data.

Michigan-style Learning Fuzzy-Classifier Systems (LFCS)– (Valenzuela-Radón, 91 & 98)

– (Parodi & Bonelli, 93)

– (Furuhashi, Nakaoka & Uchikawa, 94)

– (Velasco, 98)

– (Ishibuchi, Nakashima & Murata, 99 & 05): First LFCS for pattern classification

– (Casillas, Carse & Bull, 07) Fuzzy-XCS


Aim

Propose Fuzzy-UCSAccuracy based Michigan style LFCS– Accuracy-based Michigan-style LFCS

– Supervised learning scheme

– Derived from UCS (Bernado & Garrell, 2003)

• Introduction of a linguistic fuzzy representation

• Modification of all operators that deal with rules

– We expect:We expect:• Achieve similar performance than UCS

• Higher interpretability since we would deal with linguistic rules• Higher interpretability since we would deal with linguistic rules

• Lower number of fuzzy rules in the final population


Outline

1 D i i f F UCS1. Description of Fuzzy-UCS

2 Experimental Methodology2. Experimental Methodology

3. Results3. Results

4. Conclusions


Description of UCS1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i

p4. Conclusions

Rule representation:– Binary variables: {0 1 #}– Binary variables: {0, 1, #}

– Continuous variables: [li, ui]

E– Eg:IF x1 Є [l1, u1] ^ x2 Є [l2, u2] … ^ xn Є[ln, nn] THEN class1

– Matching instance e: for all ei: li ≤ ei ≤ ui

– Set of parameters:• Accuracy

• Fitness

• Numerosity• Numerosity

• Experience

• Correct set size


Description of UCS1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i

p4. Conclusions

Environment

M t h S t [M]P bl i t

Stream ofexamples

Population [P]

1 C A acc F num cs ts exp3 C A acc F num cs ts exp5 C A acc F num cs ts exp

Match Set [M]Problem instance+

output class

1 C A acc F num cs ts exp2 C A acc F num cs ts exp3 C A acc F num cs ts exp4 C A acc F num cs ts exp

Population [P]

Classifier

6 C A acc F num cs ts exp…

correct set4 C A acc F num cs ts exp5 C A acc F num cs ts exp6 C A acc F num cs ts exp

…

ClassifierParameters

UpdateMatch set generation

Correct Set [C]

correct setgeneration

ExperienceCorrectacc #

=Selection, Reproduction, mutation

Deletion 3 C A acc F num cs ts exp6 C A acc F num cs ts exp

…

Correct Set [C]

p

νaccFitness =Genetic AlgorithmIf there are no classfiers in [C], covering is triggered


Description of Fuzzy-UCSp y

Describe the different components1. Rule representation and matching1. Rule representation and matching

2. Learning interaction

3 Di t3. Discovery component

4. Fuzzy-UCS in test mode


Description of Fuzzy-UCS1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i

p y

Rule representation

4. Conclusions

Rule representation– Linguistic fuzzy rules

– E.g.: IF x1 is A1 and x2 is A2 … and xn is An THEN class1

Disjunction of linguistic

All i bl h th ti

Disjunction of linguistic fuzzy terms

– All variables share the same semantics

– Example: Ai = {small, medium, large}

IF x1 is small and x2 is medium or large THEN class1

– Codification:


IF [100 | 011] THEN class1


p y

How do we know if a given input is small, medium or large?

4. Conclusions

g p , g– Each linguistic term defined by a membership function

Belongs to medium with a degree of 0 8Belongs to medium with a degree of 0.8

Belongs to small with a degree of 0 2

ei

Belongs to small with a degree of 0.2

Triangular-shaped membership functions

Attribute valuemembership functions



p y

Matching degree uAk(e) [0,1]

4. Conclusions

g g A ( ) [ , ]

k: IF x1 is small and x2 is medium or large THEN class1

Example: (e1, e2)

0 8

0.2 0.2

0.8

T-conorm: bounded sum

e1 e2

max ( 1, 0.8 + 0.2) = 1

T-norm: productk


uAk(e) = 1 * 0.2 = 0.2


p y4. Conclusions

The role of matching changes:• UCS: A rule matches or not an example (binary function)• Fuzzy-UCS: A rule matches an example with a certain degree



p y4. Conclusions

Each classifier has the following parameters:1 Weight per class w1. Weight per class wj

• Soundness with which the rule predicts the class j.

• The class value is dynamic and corresponds to the class j with higher w• The class value is dynamic and corresponds to the class j with higher wj

2. Fitness:• Quality of the rule

3. Other parameters directly inherited from UCS:• numerosity

• correct set size

• experience



p y4. Conclusions

Learning interaction:– The environment provides an example e and its class c

– Match set creation: all classifiers that match with uAk(x) > 0

– Correct set creation: all classifiers that advocate cCorrect set creation: all classifiers that advocate c

– Covering: if there is not a classifier that maximally matches e• Create the classifier that match the input example with maximumCreate the classifier that match the input example with maximum

degree.

• Generalize the condition with probability P#

A2A1 A3For each variable:



p y4. Conclusions

Parameters’ UpdateExperience:– Experience:

– Sum of correct matching per class j cmj:



p y4. Conclusions

Parameters’ UpdateUse cm to update of the weights per each class:– Use cm to update of the weights per each class:

• Rule that only matches instances of class c:

• wc = 1

• For all the other classes j: wj = 0

– Calculate the fitness

Pressuring toward rules that correctly match instances of

only one classonly one class



p y4. Conclusions

Discovery componentSteady state niched GA– Steady-state niched GA

– Roulette wheel selectionInstances that have a highergmatching degree have more

opportunities of being selected



p y4. Conclusions

Class inference of a test example eWinner rule– Winner rule

• Inference: Select the rule that maximizes uAk(e) · Fk

• Reduction: Only keep in the final population that rules that maximize uA

k(e) · Fk at least for one training example

– Average vote• Inference: All experienced rules vote for the class they predict. The

most voted class is returned.

R d ti O l k i d l ith iti fit i th• Reduction: Only keep experienced rules with positive fitness in the final population


Experimental Methodology1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i

p gy4. Conclusions

Methodology– Compare Fuzzy-UCS to UCS, C4.5, and SMO.

– 10-fold cross-validation

– Averages over 10 runs

– 5 linguistic labels

#Inst #Fea #Re #In #No #Cl %Min %Max %MisAtt#Inst #Fea #Re #In #No #Cl %Min %Max %MisAtt

bal 625 4 4 0 0 3 7,8 46,1 0

bpa 345 6 6 0 0 2 42 58 0ph-c 303 13 6 0 7 2 45,5 54,5 15,4

irs 150 4 4 0 0 3 33,3 33,3 0

pim 768 8 8 0 0 2 34,9 65,1 0

thy 215 5 5 0 0 3 14 60 0

wbcd 699 9 0 9 0 2 34,5 65,5 11,1


, , ,

wne 178 13 13 0 0 3 27 39,9 0

Results1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i4. Conclusions

• 1st objective: Competitive in terms of performance• 1st objective: Competitive in terms of performance

Significant difference of Fuzzy-UCS with winner rule: Significant difference of Fuzzy-UCS with average vote:

Parameters:iter = 100,000, N = 6,400, F0 = 0.99, v=10, {θGA, θdel, θsub} = 50,x =0.8, u=0.04, P#=0.6, 5 linguistic labels



• 2nd objective: Improve the interpretability

Example of rules evolved by UCS for iris

• 2nd objective: Improve the interpretability

Example of rules evolved by UCS for iris

Example of rules evolved by Fuzzy-UCS for iris – Linguistic terms: {XS, S, M, L, XL}



Number of rules

WRule Avote UCS

b l 80 1293 1392bal 80 1293 1392bpa 60 1849 2075h-c 145 4441 2265h c 145 4441 2265irs 18 477 624pim 166 3344 2908thy 32 1142 856wbcd 136 3018 1111wne 101 3984 3618


Conclusions and Further Work1. Description of Fuzzy-UCS2. Experimental Methodology3. Results4 C l i4. Conclusions

Conclusions– We proposed a Michigan-style LFCS for supervised learning– We proposed a Michigan-style LFCS for supervised learning

– Competitive with respect to UCS, SMO, and C4.5

I t i t f i t t bilit ith t t UCS– Improvement in terms of interpretability with respect to UCS

Further work– Evolve more reduced populationsp p

– Enhance the comparison with new real-world problems

– Compare to other LFCSCompare to other LFCS

– Exploit the incremental learning approach to dig large datasets


Fuzzy-UCS: Preliminary Results

Albert Orriols-Puig1,2Albert Orriols PuigJorge Casillas2

Ester Bernadó-Mansilla1

1Research Group in Intelligent SystemsEnginyeria i Arquitectura La Salle, Ramon Llull University

2Dept. Computer Science and Artificial IntelligenceUniversity of Granada

iwlcs'2007: fuzzy-ucs: preliminary results

Education