multiobjective genetic fuzzy systems - accurate and interpretable fuzzy rule-based classifier design...

Multiobjective Genetic Fuzzy Systems- Accurate and Interpretable Fuzzy Rule-Based

Classifier Design -

Hisao Ishibuchi Osaka Prefecture University, Japan

1. Introduction to Fuzzy Rule-Based Classification - Is Fuzzy Rule-Based Classification a Popular Research Area?

2. Fuzzy Rule-Based Classifier Design- Accuracy Improvement- Scalability to High-Dimensional Problems- Complexity Minimization

3. Multiobjective Fuzzy Rule-Based Classifier Design- Formulation of Multi-objective Problems- Accuracy-Complexity Tradeoff Analysis- Maximization of Generalization Ability

4. Current Hot Issues and Future Research Directions- Search Ability of EMO for Fuzzy System Design - Definition of Interpretability of Fuzzy Systems- Explanation Ability of Fuzzy Rule-Based Systems- Various Classification Problems: Imbalanced, Online, ...

Contents of This PresentationAccurate and Interpretable Fuzzy Rule-Based Classifier Design

Application Areas of Fuzzy Systems - Fuzzy Control - Fuzzy Clustering - Fuzzy ClassificationControl and clustering are well-known application areas !

Question: Is “fuzzy classification” popular ?

Application Areas of Fuzzy Systems

Web of Science

Application Areas of Fuzzy Systems - Fuzzy Control - Fuzzy Clustering - Fuzzy ClassificationControl and clustering are well-known application areas !

Question: Is “fuzzy classification” popular ?

Application Areas of Fuzzy Systems

Control, Clustering, Classification

Web of Science

Application Areas of Fuzzy SystemsFuzzy Control: Well-Known Application Area

9421 Fuzzy Control Papers

Web of Science

Application Areas of Fuzzy SystemsFuzzy Control: Well-Known Application Area

Takagi-Sugeno Model (1985)

3801ANFIS (1993)

1983

1896

CC Lee: Fuzzy Logic Controller (1990)

Web of Science

Application Areas of Fuzzy SystemsFuzzy Clustering: Well-Known Application Area

2968 Fuzzy Clustering Papers

Web of Science

Application Areas of Fuzzy SystemsFuzzy Clustering: Well-Known Application Area

819

Pal NR, Pal SK

Bezdek JC

Pal NR, Bezdek JC

Web of Science

Application Areas of Fuzzy SystemsFuzzy Classification: Well-Known?

4144 Fuzzy Classification Papers

Web of Science

Application Areas of Fuzzy SystemsFuzzy Classification: Well-Known?

389Dubois D, Prade H

Dubois D, Prade H

Fuzzy Min-Max NN

Fuzzy If-Then Rules

Web of Science

Application Areas of Fuzzy SystemsControl, Clustering, and Classification

9421 (Fuzzy Control)

2968 (Fuzzy Clustering)

4144 (Fuzzy Classification)Web of Science

Application Areas of Fuzzy SystemsControl, Clustering, and Classification

9421 (Fuzzy Control)

2968 (Fuzzy Clustering)

4144 (Fuzzy Classification)

FUZZ-IEEE 1992

IEEE Trans FS 1993FUZZ-IEEE 1992

FUZZ-IEEE 1992

IEEE Trans FS 1993

IEEE Trans FS 1993

Web of Science


2. Fuzzy Rule-Based Classifier Design- Accuracy Maximization- Scalability to High-Dimensional Problems- Complexity Minimization



Contents of This Presentation Accurate and Interpretable Fuzzy Rule-Based Classifier Design

This Presentation: Accurate and Interpretable Fuzzy Rule-Based Classifier Design

Web of Science


Fitness Function

w1 Accuracy (S) w2 Complexity (S)

Accuracy Maximization and Complexity Minimization


w1 Accuracy (S) w2 Complexity (S)

The number of correctly classified training patterns

The number of selected fuzzy rules

Fuzzy Rules for Classification Accurate and Interpretable Fuzzy Rule-Based Classifier Design

Basic FormIf x1 is small and x2 is small then Class 2

1.0

small medium large

0.0 1.0x1

Class 1 Class 2 Class 3

0.0x

2

largem

edium

small



If x1 is small and x2 is medium then Class 2

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small




If x1 is small and x2 is large then Class 1

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small





. . .

If x1 is large and x2 is large then Class 3

High InterpretabilityEasy to Understand !

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small

Classification Boundary Accurate and Interpretable Fuzzy Rule-Based Classifier Design

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small




. . .


High InterpretabilityEasy to Understand !

Fuzzy Rules for ClassificationBasic form does not always have high accuracy




. . .


High InterpretabilityLow Accuracy

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small

Fuzzy Rules for ClassificationAnother form has a rule weight (certainty)

1.0

0.0 1.0x1


0.0x

2

Basic FormIf x1 is small and x2 is medium

then Class 2

Rule Weight VersionIf x1 is small and x2 is medium

then Class 2 with 0.158

small medium large

largem

edium

small

Classification BoundaryAccurate and Interpretable Fuzzy Rule-Based Classifier Design

1.0

0.0 1.0x1


0.0x

2


then Class 2



small medium large

largem

edium

small

Fuzzy Rules with Rule Weights Accurate and Interpretable Fuzzy Rule-Based Classifier Design

1.0

0.0 1.0x1


0.0x

2


then Class 2



small medium large

largem

edium

small

H. Ishibuchi et al., Fuzzy Sets and Systems (1992)Web of Science

Fuzzy Rules in This PresentationFuzzy Rules with Rule Weights

1.0

0.0 1.0x1


0.0x

2


then Class 2



small medium large

largem

edium

small

Use of Rule Weights: Controversial Issue(1) Rule weight adjustment can be replaced with membership learning.

D. Nauck and R. Kruse, Proc. of FUZZ-IEEE 1998 (1998).Google Scholar

Fuzzy Rules in This PresentationFuzzy Rules with Rule Weights

1.0

0.0 1.0x1


0.0x

2


then Class 2



small medium large

largem

edium

small

Use of Rule Weights: Controversial Issue(2) Membership learning can be partially replaced with weight adjustment.

H. Ishibuchi and T. Nakashima, IEEE Trans. FS (2001)Google Scholar

Fuzzy Rules for ClassificationAnother Form: Multiple Consequents

1.0

0.0 1.0x1


0.0x

2


then Class 2



Multiple ConsequentsIf x1 is small and x2 is medium

then Class 1 with 0.579, Class 2 with 0.421, Class 3 with 0.000

small medium large

largem

edium

small

Fuzzy Rules for ClassificationAnother Form: Multiple Consequents

1.0

0.0 1.0x1


0.0x

2


then Class 2



Multiple ConsequentsIf x1 is small and x2 is medium

then Class 1 with 0.579, Class 2 with 0.421, Class 3 with 0.000

small medium large

largem

edium

small

O. Cordon et al., IJAR (2001)Google Scholar

Other Forms of Fuzzy RulesHandling of Classification as Function Approximation

Integer ConsequentIf x1 is small and x1 is large then y = 1

If x1 is large and x2 is large then y = 3

Binary ConsequentIf x1 is small and x1 is large

then (y1, y2, y3) = (1, 0, 0)

If x1 is large and x2 is large

then (y1, y2, y3) = (0, 0, 1)

1.0

0.0 1.0x1


0.0x

2

small medium large

largem

edium

small

Accuracy ImprovementUse of Fine Fuzzy Partition

S M L

0.0 1.0x1


MS ML

1.00.0

x2

LM

SM

LM

S

Accuracy ImprovementHow to choose an appropriate partition ?

1.0

0.0 1.0

1 2 1.0

0.0 1.0

3 54

1.0

0.0 1.0

6 987 1.0

0.0 1.0

a edb c

1.0

0.0 1.0

f kjg ih1.0

0.0 1.0

l rpm on q

Too Fine Fuzzy Partition ==> Over-Fitting (Poor Generalization Ability)

S M L

0.0 1.0x1


MS ML

1.00.0

x2

LM

SM

LM

S

S M L

0.0 1.0x1


MS ML

1.00.0

x2

LM

SM

LM

S

Accuracy ImprovementHow to choose an appropriate partition ?

One Idea: Use of All Partitions (Multiple Fuzzy Grid Approach)

1.0

0.0 1.0

1 2 1.0

0.0 1.0

3 54

1.0

0.0 1.0

6 987 1.0

0.0 1.0

a edb c

1.0

0.0 1.0

f kjg ih1.0

0.0 1.0

l rpm on q

Ishibuchi et al., Fuzzy Sets and Systems (1992)Web of Science

0.0 1.0x1

Class 1 Class 2 Class 31.00.0

x2

Accuracy ImprovementLearning of Membership Functions

0.0 1.0x1


x2


Various learning methods such as neuro-fuzzy and genetic-fuzzy methods are available.

D. Nauck and R. Kruse, Fuzzy Sets and Systems (1997).Web of Science

0.0 1.0x1


x2


Various learning methods such as neuro-fuzzy and genetic-fuzzy methods are available.

Interpretability is degraded.

D. Nauck and R. Kruse, Fuzzy Sets and Systems (1997).Web of Science

1.00.0 x1


x2

Accuracy ImprovementUse of Independent Membership Functions

Each fuzzy rule can be generated and adjusted independently from other rules. ==> High Accuracy

Membership functions can be heavily overlapping. ==> Poor Interpretability

1.00.0 x1


x2

Accuracy ImprovementUse of Independent Membership Functions

Each fuzzy rule can be generated and adjusted independently from other rules. ==> High Accuracy

Membership functions can be heavily overlapping. ==> Poor Interpretability

S. Abe and M. S. Lan, IEEE Tras. on FS (1995)Web of Science

If x is A then Class 2


A

A: Multi-dimensional Fuzzy Set (Membership Function)

Accuracy ImprovementUse of Multi-Dimensional Membership Functions



A



Fuzzy rules are flexibility. ==> High Accuracy

Each membership function is multi-dimensional. ==> Poor Interpretability



A



S. Abe et al., IEEE TSMC-C (1998)

S. Abe et al., IEEE TSMC-C (1999)

Web of Science

largesm

all

0.0 1.0x1


x2

small large

Accuracy ImprovementUse of Tree-Type Fuzzy Partitions

If x1 is small then Class 2.

If x1 is large and x2 is small

then Class 3.

If x1 is large and x2 is large

then Class 1.

x1 is small x1 is large


Accuracy ImprovementUse of Tree-Type Fuzzy Partitions

Web of Science

Difficulty of High-Dimensional ProblemsExponential Increase of Fuzzy Rules



. . .


Number of Fuzzy Rules: 2-D Problem: 3x3 3-D Problem: 3x3x3 4-D Problem: 3x3x3x3 5-D Problem: 3x3x3x3x3

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small

1.00.0 x1


x2

Scalability ImprovementUse of Independent Membership Functions

Fuzzy rules are generated in the multi-dimensional space. => No Exponential Increase



A


Scalability ImprovementUse of Multi-Dimensional Membership Functions

Fuzzy rules are generated in the multi-dimensional space. => No Exponential Increase

Scalability ImprovementUse of Tree-Type Fuzzy Partitions



An appropriate stopping condition prevents the exponential increase in the number of fuzzy rules.

Web of Science

x1 x2 x3 x4 x5

subsystem

subsystemsubsystem

subsystem

Scalability ImprovementHierarchical Fuzzy Systems

subsystem

x1 x2 x3 x4 x5

subsystem subsystem

subsystemsubsystem

subsystem

Scalability ImprovementHierarchical Fuzzy Systems

K. Simojima et al., Fuzzy Sets and Systems (1995)Web of Science

Accuracy & Scalability Improvement ==> Poor Interpretability

- Use of Fine Fuzzy Partition (Accuracy)

- Use of Rule Weights (Accuracy)

- Membership Function Learning (Accuracy)

- Fuzzy Rules with Independent Membership Functions (Accuracy and Scalability)

- Multi-Dimensional Fuzzy Systems (Accuracy and Scalability)

- Tree-Type Fuzzy Partitions (Accuracy and Scalability)

- Hierarchical Fuzzy Systems (Accuracy and Scalability)

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

Class 1 Class 2 Class 3 1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


Four Rules Nine Rules

Rule Selection (Nine Rules ==> Four Rules)

Complexity MinimizationFuzzy Rule Selection

Four Rules Nine Rules

Rule Selection (Nine Rules ==> Four Rules) The same classification boundaries are generated.

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small



Four Rules


Nine Rules

Rule Selection (Nine Rules ==> Four Rules) The same classification boundaries are generated.

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


H. Ishibuchi et al., IEEE Trans. on FS (1995)

Web of Science


Seven Rules Nine Rules

Use of “Don’t Care” Conditions Nine Rules ==> Seven Rules (If x1 is large then Class 3)

Class 1 Class 2 Class 31.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

Complexity MinimizationUse of “Don’t Care” Conditions

Seven Rules Nine Rules

Nine Rules ==> Seven Rules (If x1 is large then Class 3)Slightly different classification boundaries are obtained.


Class 1 Class 2 Class 3 Class 1 Class 2 Class 31.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

Nine Rules ==> Seven Rules (If x1 is large then Class 3)Slightly different classification boundaries are obtained.



small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

H. Ishibuchi et al., IEEE Trans. on SMC Part B (1999)Google Scholar

The use of “Don’t Care” conditions can be viewed as an scalability improvement method. If x1 is small and x10 is ...



small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


The use of “Don’t Care” conditions can be also viewed as input selection for each rule (rule-wise input selection).



small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


1.00.0 x1


x2

Complexity MinimizationMerging Similar Membership Functions

1.00.0 x1


x2

Similar Membership Functions


1.00.0 x1


x2


==> One Membership Function


1.00.0 x1


x2




M. Setnes et al., IEEE Trans. on SMC Part B (1998)

Google Scholar

1.00.0 x1


x2




- Interpretability is improved.- Accuracy is degraded.

1.00.0 x1


x2





Interpretability-Accuracy Tradeoff

Complexity MinimizationProjection of Multi-Dimensional Fuzzy Sets


Interpretability-Accuracy Tradeoff

Aq2

Aq1

0.0 1.00.0

1.0

x1

x2 Aq

If x is Aq then ...

If x1 is Aq1 and x2 is Aq2 then

Change of Fuzzy Rule Form

Interpretability-Accuracy Tradeoff (Accuracy-Simplicity Tradeoff)

Err

or

Sm

all

Lar

ge

ComplexitySimple Complicated

Interpretable

fuzzy system

Accurate

fuzzy system

Accuracy MaximizationMain Research Direction Since the Early 1990s

Err

or

Sm

all

Lar

ge


Interpretable

fuzzy system

Accurate

fuzzy system

Accuracy Maximization - Fuzzy Neuro Learning - Genetic Fuzzy Optimization

Possible Difficulties

Accuracy Maximization

- Poor Interpretability

- Overfitting to Training Data



- Poor InterpretabilityPoor Interpretability

- Overfitting to Training Data

Difficulty in Accuracy Maximization

Test data accuracy

Training data accuracy

Accuracy maximization Overfitting

Complexity

Err

or

S*0

Accuracy-Complexity Tradeoff

Curve Fitting

0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Complexity

Err

or

y

x

Feasible Area


Curve Fitting

Complexity

y

x0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Err

or


Curve Fitting

0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Complexity

y

x

Err

or


Curve Fitting

0 0.2 0.4 0.6 0.8 1

0.2

0.4

0.6

0.8

1

Complexity

y

x

Err

or

Test Data Error

Training Data Error



- Poor Interpretability

- Overfitting to Training DataOverfitting to Training Data

In the design of fuzzy systems, emphasis should be placed on their linguistic interpretability.

Interpretability maintenance while maximizing the accuracy.

Accuracy and Interpretability MaximizationActive Research Direction Since the Late 1990s

Compromise between Accuracy and Complexity(Search for a good accuracy-complexity tradeoff)

Err

or

Sm

al lL

arg

e


Interpretable

fuzzy system

Accurate

fuzzy system



Some Ideas - Aggregated Objective Function: To combine the error

minimization and the complexity minimization into a single scalar fitness function





- Constraint Condition: To use constraint conditions on the position and the shape of membership functions






- Two-Step Fuzzy System Design: 1st Step: Search for accurate and complicated fuzzy rule-based systems. 2nd Step: Simplification of obtained fuzzy rule-based systems.


Accuracy Maximization and Complexity Minimization

Google Scholar



Basic Idea - Aggregated Objective Function: To combine the error



- Two-Step Fuzzy System Design: 1st Step: Search for accurate and complicated fuzzy rule-based systems. 2nd Step: Simplification of obtained fuzzy rule-based systems.


Aggregated Objective Function - To combine the error minimization and the complexity

minimization into a single scalar objective function



Aggregated Objective Function - To combine the error minimization and the complexity

minimization into a single scalar objective function

Example: Combination of the average error rate and the number of fuzzy rules

Example of a scalar objective function: Weighted sum

)()()( Complexity2Error1 SfwSfwSf



Err

or

Sm

all

Lar

ge


Interpretable

fuzzy system

Accurate

fuzzy system

Fuzzy systems were automatically generated, trained, and simplified.

Difficulty in Weighted Sum Approach

Sensitivity to the weight vector:

The obtained system strongly depends on the specification of the weight vector.

Complexity

Err

or

Test data accuracy

S*0


Minimize w1·Error + w2·Complexity

When the weight for the complexity minimization is large:

A simple system is obtained.


Complexity

Err

or

Test data accuracy

S*0



When the weight for the error minimization is large:

A complicated system is obtained.



Complexity

Err

or

Test data accuracy

S*0



When the two weights are appropriately specified:

A good system is obtained. But the best complexity is not always found.

Multiobjective Fuzzy System DesignCurrently An Active Research Issue

Basic Idea To search for a number of non-dominated fuzzy systems with

respect to the accuracy maximization and the interpretability maximization (instead of searching for a single fuzzy system).

Aggregation Approach

Multiobjective Approach

)}(),({Minimize ComplexityError SfSf



Basic Idea To search for a number of non-dominated fuzzy systems with

respect to the accuracy maximization and the interpretability maximization (instead of searching for a single fuzzy system).

Aggregation Approach

Multiobjective Approach

)}(),({Minimize ComplexityError SfSf


Search for Pareto Optimal Fuzzy Rule-Based Systems


Err

or

Sm

all

Lar

ge


Interpretable

fuzzy system

Accurate

fuzzy system





Contents of This PresentationAccurate and Interpretable Fuzzy Rule-Based Classifier Design

Evolutionary Multiobjective Optimization of Fuzzy Rule-Based Systems Bibliography Page

Created and Maintained by Marco Cococcioni (Univ. Pisa)

Evolutionary Multiobjective Optimization of Fuzzy Rule-Based Systems Bibliography Page Created and Maintained by Marco Cococcioni (Univ. Pisa)





Contents of This Presentation

Multi-Objective Fuzzy Rule-Based Systems (Fuzzy Rule* OR Fuzzy Rule-Based System*) AND (Multi-Objective OR

Multiobjective OR Two-Objective OR Three-Objective OR Multiple Criteria)

H. Ishibuchi et al., Fuzzy Sets and Systems (1997)

Web of Science

Multi-Objective Fuzzy Rule Selection for Fuzzy Rule-Based Classifier Design

Two Objectives: - The number of correctly classified training patterns - The number of selected fuzzy rules


Multi-Objective Fuzzy Rule-Based Systems (Fuzzy Rule* OR Fuzzy Rule-Based Systems) AND (Multi-Objective OR


Web of Science

Multi-Objective Fuzzy Rule Selection for Fuzzy Rule-Based Classifier Design

Two Objectives: - The number of correctly classified training patterns - The number of selected fuzzy rules


H. Ishibuchi et al., Information Science (2001)

Three Objectives: - The number of correctly classified training patterns - The number of selected fuzzy rules - Total number of antecedent conditions (Total rule length)



Web of Science

Multi-Objective RBF Function Design for Function Approximation Problems

Multi-Objective RBF Function Design for Function Approximation Problems

Multi-Objective Neural Network Design and Learning

Multiobjective Neural Networks

0Complexity

Err

or

Multiobjective Decision Trees (GP)

0Complexity

Err

or



Web of Science

Paper Title: Industrial Applications of Fuzzy-Logic at General-Electric

Web of Science

Recent Publications on Multi-Objective Fuzzy System Design

Web of Science

Paper Title: A Multiobjective Evolutionary Approach to Concurrently Learn Rule and Data Bases of Linguistic Fuzzy-Rule-Based Systems

Web of Science

Paper Title: A Multiobjective Evolutionary Approach to Concurrently Learn Rule and Data Bases of Linguistic Fuzzy-Rule-Based Systems

1. Univ Granada 2. Univ Pisa

Web of Science

Paper Title: Learning concurrently partition granularities and rule bases of Mamdani fuzzy systems in a multi-objective evolutionary framework

Web of Science

Paper Title: Learning concurrently partition granularities and rule bases of Mamdani fuzzy systems in a multi-objective evolutionary framework

1. Univ Pisa

Web of Science

Multi-Objective Fuzzy System Design ResearchActive Geographical RegionsGoogle Maps

Complexity0

Err

orMany non-dominated fuzzy systems can be obtained along the tradeoff surface by a single run of an EMO algorithm.

EMO: Evolutionary Multi-Objective Optimization

Results of Multi-Objective SearchNon-Dominated Fuzzy Rule-Based Systems

A Single Fuzzy Rule-Based System

Accuracy-Complexity Tradeofffor Training Data and Test Data

Complexity

Test data

S*0

Training data

Err

or

The obtained non-dominated fuzzy systems show the tradeoff between the complexity and the training data accuracy (not the tradeoff between the complexity and the test data accuracy).

Accuracy-Complexity Tradeofffor Training Data and Test Data

Complexity

Test data

S*0

Training data

Err

or

The obtained non-dominated fuzzy systems show the tradeoff between the complexity and the training data accuracy (not the tradeoff between the complexity and the test data accuracy).

- Tradeoff for test data accuracy should be examined.- This can be done since we have many fuzzy systems.

Example: Obtained Rule Sets (Heart C)

Training data accuracy Testing data accuracy

Number of rules

Err

or r

ate

(%)

0 2 4 6 8 10 12 1410

20

30

40

50

60

Number of rulesE

rror

rat

e (%

)0 2 4 6 8 10 12 14

30

40

50

60

70

80

Obtained rule sets help us to find the optimal complexity of fuzzy systems. (Rule sets with six, seven and eight rules may be good)

A rule set with eight fuzzy rules

Some human users may prefer simpler rule sets.

A Rule Set with High-Generalization Ability

Number of rules

Err

or r

ate

(%)

0 2 4 6 8 10 12 1410

20

30

40

50

60

Number of rules

Err

or r

ate

(%)

Class 1(0.23)

Class 1(0.81)

Consequent

Class 2(0.63)

Class 2(0.20)

Class 2(1.00)

Class 3(0.35)

Class 3(0.24)

x1 x3 x4 x6 x7 x8 x10 x11 x12

R1

R2

R3

R4

R5

R6

R7

R8

Class 1(0.46)

A very simple rule set with only two fuzzy rules

A Rule Set with High Interpretability

Number of rules

Err

or r

ate

(%)

0 2 4 6 8 10 12 1410

20

30

40

50

60

Number of rules

Err

or r

ate

(%) 10x

1R

2R

Consequent

Class 1(0.26)

Class 2(1.00)

11x

DC

DC

1. Large Search Space: Difficulty in Search

The search space exponentially increases with the number of attributes (i.e., with the dimensionality of the pattern space).

Why is the fuzzy system design difficult?

Number of Fuzzy Rules



. . .


Number of Fuzzy Rules: 2-D Problem: 3x3 3-D Problem: 3x3x3 4-D Problem: 3x3x3x3 5-D Problem: 3x3x3x3x3

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small

Number of Fuzzy Rules

Use of Don’t CareIf x1 is small and x2 is small then Class 2


. . .

If x1 is large and x2 is don’t care then Class 3

Number of Fuzzy Rules: 2-D Problem: (3+1)x(3+1) 3-D Problem: (3+1)3

4-D Problem: (3+1)4

5-D Problem: (3+1)5

1.0

small medium large

0.0 1.0x1


0.0x

2

largem

edium

small

Search Space Size: Large

Example: Classification problem with 50 attributes and 3 linguistic values for each attribute

Number of Fuzzy Rules and Number of Rule Sets

1.0

small

medium large

0.0 1.0x1

0.0x

2

large

mediu

msm

all

Class 1 Class 2 Class 3The total number of fuzzy rules (i.e., antecedent condition combinations):

(3+1) x ... x (3+1) = 450 = 2100

The total number of fuzzy rule sets with 20 rules (i.e., combinations of 20 fuzzy rules):

NC20 ~ 22000 where N = 2100

Search Space Size: Large

Example: Classification problem with 50 attributes and 1-7 fuzzy partition for each attribute

Number of Fuzzy Rules and Number of Rule Sets

The total number of fuzzy rules (i.e., antecedent condition combinations):

(1+2+ ... 7) x ... = 2850 > 2400

The total number of fuzzy rule sets with 20 rules (i.e., combinations of 20 fuzzy rules):

NC20 ~ 28000 where N > 2400

1.0

0.0 1.0

1 2 1.0

0.0 1.0

3 54

1.0

0.0 1.0

6 987 1.0

0.0 1.0

a edb c

1.0

0.0 1.0

f kjg ih1.0

0.0 1.0

l rpm on q


The search space exponentially increases with the number of attributes (i.e., with the dimensionality of the pattern space). It is likely that the entire tradeoff curve can not be covered well by the obtained non-dominated fuzzy rule-based systems.


Complexity0

Err

or

0 Complexity

Err

or

Poor Diversity Poor Convergence




2. Possibility of Over-Fitting: Difficulty in Learning

The improvement in the training data accuracy does not always mean the improvement in the test data accuracy.


The improvement in the training data accuracy does not always mean the improvement in the test data accuracy. This means that the fitness function improvement does not always lead to better fuzzy rule-based classifiers (when the training data accuracy is used in the fitness function) .





Complexity0

Err

or

Poor Diversity



Test Data Accuracy

Training Data Accuracy

Training Data Accuracy Improvement


Complexity0

Err

or

Poor Diversity Diversity Improvement



Test Data Accuracy

Complexity0

Err

or

Test Data Accuracy

Training Data Accuracy Training Data Accuracy


Complexity0

Err

or

Poor Convergence



Test Data Accuracy


Training Data Accuracy Improvement


Complexity0

Err

or

Poor Convergence Convergence Improvement



Test Data Accuracy

Complexity0

Err

or

Test Data Accuracy

Training Data AccuracyTraining Data Accuracy

Recent Studies: Improvement in training data accuracy leads to Improvement in test data accuracy.

Poor Diversity Diversity ImprovementComplexity0

Err

or

Test Data Accuracy

Complexity0

Err

or

Test Data Accuracy

Training Data Accuracy Training Data Accuracy

Gacto MJ, Alcala R, Herrera F (2009) Adaptation and application of multi-objective evolutionary algorithms for rule reduction and parameter tuning of fuzzy rule-based systems, Soft Computing 13 (5): 419-436

Ishibuchi H, Nakashima Y, Nojima Y (2009) Performance evaluation of evolutionary multiobjective optimization algorithms for multiobjective fuzzy genetics-based machine learning, Proc. of FUZZ-IEEE 2009.

Our Experimental Results

MoFGBML Algorithm (Framework: NSGA-II) Multi-Objective Fuzzy Genetics-Based Machine Learning

Ishibuchi & Nojima, IJAR 2007

NSGA-II Basic Setting - Population size: 200 individuals - Termination Condition: 2000 generations - Multiple Fuzzy Partitions: Granularities 1-5

Three Variants of MoFGBML Setting - Diversity Improvement Method (Mating, EJOR 2008) - Termination Condition: 20000 generations - Multiple Fuzzy Partitions: Granularities 7

Google Scholar

Experimental Results (Glass)

5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data

Number of fuzzy rules

Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

NSGA-II Basic Setting


Improvement


5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020 5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

NSGA-II Basic Setting NSGA-II with Mating


5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020 5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

NSGA-II 20,000 Generations NSGA-II with Mating

5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

2,000 generations

20,000 generations

5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 30205 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020


5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

NSGA-II Granularities 1-7 NSGA-II with Mating

1-5 granularities

1-7 granularities

Our Experimental ResultsSimple Changes of Objectives (GEFS 2010, Spain)

Original Formulation f1(S): Error Rate (%) f2(S): Number of Fuzzy Rules

Simple Modification g1(S) = f1(S) - f2(S) g2(S) = f2(S) + f1(S)

5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

f1(S)

f2(S)

g2(S)

g2(S)


5 10 15 20 25 30 350.10

0.15

0.20

0.25

0.30

0.35


Err

or r

ate

on tr

aini

ng d

ata

(%) 35

30

25

20

15

10 5 10 15 20 25 30 35

0 0.01 0.1

5 10 15 20 25 30 350.14

0.16

0.18

0.20

0.22

0.24

0.26 0 0.01 0.1


Err

or r

ate

on tr

aini

ng d

ata

(%) 26

24

22

20

18

14 5 10 15 20 25 30 35

16

(a) Glass data. (b) Diabetes data.

Original Problem Original Problem


Original Formulation f1(S): Error Rate (%) f2(S): Number of Fuzzy Rules

Simple Modification g1(S) = f1(S) - f2(S) g2(S) = f2(S) + f1(S)

5 10 15 20 25 300.1

0.2

0.3

0.4

0.5

0.6Test Data

Training Data


Err

or r

ate

(%)

60

50

40

30

20

10 5 10 15 25 3020

f1(S)

f2(S)

Four-Objective g1(S) = f1(S) - f2(S) g2(S) = f1(S) + f2(S) g3(S) = f2(S) - f1(S) g4(S) = f2(S) + f1(S)

Our Experimental ResultsFour-Objective Formulation (GEFS 2010, Spain)

(a) Glass data. (b) Diabetes data.

Original Problem Original Problem

5 10 15 20 25 30 350.10

0.15

0.20

0.25

0.30

0.35


Err

or r

ate

on tr

aini

ng d

ata

(%) 35

30

25

20

15

10 5 10 15 20 25 30 35

0 0.01 0.1

5 10 15 20 25 30 350.14

0.16

0.18

0.20

0.22

0.24

0.26 0 0.01 0.1


Err

or r

ate

on tr

aini

ng d

ata

(%) 26

24

22

20

18

14 5 10 15 20 25 30 35

16

Handling of Interpretabilityin Our Former Studies[1] H. Ishibuchi et al. (1995) Selecting fuzzy if-then rules for

classification problems using genetic algorithms, IEEE TFS.ISI Web of Knowledge Citations: 273 times

[2] H. Ishibuchi et al. (1997) Single-objective and two-objective genetic algorithms for selecting linguistic rules for pattern classification problems. Fuzzy Sets & Systems.ISI Web of Knowledge Citations: 107 times

[3] H. Ishibuchi et al. (2001) Three-objective genetics-based machine learning for linguistic rule extraction. Information Sciences.ISI Web of Knowledge Citations: 94 times

Interpretability Maximization = Complexity Minimization - Minimization of the number of fuzzy rules - Minimization of the number of antecedent conditions

Interpretability of Fuzzy Systems

Ishibuchi et al. (1995, 1997, 2001)Interpretability Maximization = Complexity Minimization - Minimization of the number of fuzzy rules - Minimization of the number of antecedent conditions



Many other factors are related to the interpretability



Many other factors are related to the interpretability

Special Sessions and Many Related Papers - IFSA 2009 Conference - ISDA 2009 Conference (4 Papers with Interpretability in Their Titles)

- FUZZ-IEEE 2010 Conference

1.0small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


Another Issue in InterpretabilityExplanation of Classification Results

Explanation AbilityThe ability of fuzzy rule-based systems to explain why a new pattern is classified as a particular class.

Example: Classification of a pattern : xA = (0.05, 0.05)

xA

(2) Rule Set 2: Four Rules

Comparison between Rule Sets 1 and 2

(1) Rule Set 1: Nine Rules

Classification of : xA = (0.05, 0.05)


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xA xA

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


R1: If x1 is small and x2 is small then Class 2

Comparison in Explanation CapabilityResponsible Rules for Classification

R5: If x1 is medium and x2 is medium then Class 2

xA xA

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small




R5: If x1 is medium and x2 is medium then Class 2

xA xA

R1 seems to be a better explanation for the classification of xA.

(2) Rule Set 2: Four Rules



Rule Set 1 seems to have higher explanation ability while Rule Set 2 is simpler than Rule Set 1.


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xA xA


(4) Rule Set 4: Three Rules


Classification of : xA = (0.05, 0.05)

Class 1 Class 2 Class 3 Class 1 Class 2 Class 3

small or medium large

largem

edium or sm

all

0.0 1.0x1

1.00.0

x2

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xA xA


largem

edium or sm

all

0.0 1.0x1

1.00.0

x2

R1234: If x1 is small or medium and x2 is small or medium then Class 2



1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


xA xA


largem

edium or sm

all

0.0 1.0x1

1.00.0

x2

R1234: If x1 is small or medium and x2 is small or medium then Class 2



1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


xA xA

R1 seems to be a better explanation for the classification of xA.

Comparison between Rule Sets (1) and (4)

(4) Rule Set 4: Three Rules


Rule Set 1 seems to have higher explanation ability while Rule Set 4 is simpler than Rule Set 1.



largem

edium or sm

all

0.0 1.0x1

1.00.0

x2

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xA xA

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


Classification CapabilityAnother Example xB= (0.95, 0.50)

Classification of : xB = (0.95, 0.50)

xB


(3) Rule Set 3: Seven Rules


Classification of : xB = (0.95, 0.50)


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xB xB


R8: If x1 is large and x2 is medium then Class 3

R789: If x1 is large then Class 3

1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xB xB




1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xB xB

Which is a better explanation for the classification of xB between R8 and R789 ?




1.0

small medium large

0.0 1.0x1

0.0x

2

largem

edium

small


small medium large

0.0 1.0x1

0.0x

2

largem

edium

small

xB xB

Which is a better explanation for the classification of xB between R8 and R789 ? It is a very difficult question for me to answer.

We have a lot of different types of classification problems where fuzzy rule-based classifiers have not been well-utilized and have a large potential usefulness:

1. Imbalanced Data2. Semi-Supervised Learning3. Active Learning4. On-Line Learning5. . . .6. . . .

Fuzzy Classifiers on Various Problems


IFSA 2009 Invited Talk


IFSA 2009 Invited Talk

Machine Learning

Basic Advanced

Fuzzy


ISDA 2009 Invited Talk by Hisao Ishibuchi

Machine Learning

Basic Advanced

Fuzzy

MoFuzzyMoML





Conclusions

Yes !





Conclusions

It is not always high.





Conclusions

We have still a lot of interesting research issues.

Appendix: Comparison of the Two Approaches

Two-objective maximization problem

EMO Approach Weighted Sum ApproachExperimental results of a single run of each approach

f1: Total profit from knapsack 1

f 2: T

otal

pro

fit f

rom

kna

psac

k 2 2000th generation

50th generation20th generation

Pareto front

16000 17000 18000 19000 2000016000

17000

18000

19000

20000

21000

f1: Total profit from knapsack 1

f 2: T

otal

pro

fit f

rom

kna

psac

k 2 2000th generation

50th generation20th generation

Pareto front

16000 17000 18000 19000 2000016000

17000

18000

19000

20000

21000

Appendix: Two-Dimensional Antecedent Fuzzy Sets

Aq

Aq2

Aq1

0.0 1.00.0

1.0

x1

x2

Aq2

Aq1

0.0 1.00.0

1.0

x1

x2 Aq

(a) A two-dimensional fuzzy vector. (b) An ellipsoidal antecedent fuzzy set.

Appendix: Interval Rules vs Fuzzy Rules

(a) Interval Rules (b) Fuzzy Rules

S L

Class 1 Class 2

0.0 x1 1.0

0.0

x2

1.0

M

SL

M

S L

SL

Class 1 Class 2

0.0 x1 1.0

0.0

x2

1.0

M

M

multiobjective genetic fuzzy systems - accurate and interpretable fuzzy rule-based classifier design...

Documents

fuzzy classification

fuzzy classificationfuzzieee

prade h fuzzy

fuzzy logic controller

max nn fuzzy

wellknown application

popular research area

ieee trans fs