Dr. Edwin Lughofer

Evolving Fuzzy Systems (EFS)

Methodologies, Advanced Concepts and Applications

Overview

Dr. Edwin LughoferDepartment of Knowledge-Based Mathematical Systems

Johannes Kepler University, Linz, Austria

Dr. Edwin Lughofer

Properties of Fuzzy Systems • Models which build upon the concept of Fuzzy Logic (going back to Zadeh: truth values are fuzzified in [0,1] by MFs) •Tradeoff between

» Universal Approximation: Models with arbitrary degree of non-linearity by piecewise local approx.

» Interpretability(Readable Linguistic Terms and Rules)

• (Dark) Gray-Box Models

• Similarities Between Certain Architectures with Certain Types of Neural Networks

• Old-School (70ties, 80ties): Knowledge-Based Input

• New-School (90ties, 00s): Data-Driven Learning with Machine Learning/Optimization Techniques (genfis2, LOLIMOT, FMCLUST, ANFIS)

Dr. Edwin Lughofer

The Idea of Evolving Fuzzy Systems

Evolving = Evolutionary

Building Up Fuzzy Systems in (Single-Pass) Incremental and Evolving Manner

with On-line Recordings/Data-Streams

Dr. Edwin Lughofer

Concepts in Evolving Fuzzy Systems

Incrementality accounts for a step-wise (sample or block) processing of data and model building, omitting time-intensive re-training (on-line capability) Adaptivity accounts for (recursively) adapting parameters with newly loaded samples Evolving means that structural components(rules, neurons) are added on demand due to new system states, operating conditions etc. Single-Pass = Sample is loaded, sent into the incremental learning engine and discarded immediately, afterwards (achieving low computation time and virtual memory demand)

Dr. Edwin Lughofer

Incremental Learning Example (Movement of Decision Boundary)

Dr. Edwin Lughofer

Some (Industrial) Requirements

• Fast Online Identification of Models/Classifiers (from scratch), usage in On-line Plausibility Analysis/FD• Updating and extending already available models => improvement of generalization capability of models, preventing extrapolation => Higher Process Safety• Hybrid Modelling: Refining Knowledge-based Models with Data• Extracting models from Huge Data Bases (no loading of all data at once possible)• Enhanced Human-Machine Interaction Scenarios• Appropriate handling of drifts/shifts in data streams(gradual forgetting, out-smoothing over time)

Dr. Edwin Lughofer

Evolving Modelling FrameworkInclusion of target values from measurement/feedback;

Never include estimated/predicted values from models (risk of self-error propagation)

Dr. Edwin Lughofer

EFS Model Architectures

• Takagi-Sugeno Type: most frequently used, best tradeoff between accuracy and interpretability • Neuro-Fuzzy Approaches: fuzzy components embedded in neural network architecture (layers)• Mamdani Fuzzy Systems (linguistic consequents)• Fuzzy Classifiers

» Single Model Architectures » Multi-Model Architectures » [All-Pairs] (in work)

Dr. Edwin Lughofer

Definition of a (specific) Takagi-Sugeno Fuzzy Model

Usage of product t-norm and Gaussian funcs as fuzzy sets

C rules

Linear Hyper-planes, in case of polynomials with higher order TSK models

Dr. Edwin Lughofer

Incremental Learning of Consequent Parameters (in TS fuzzy models)Most of the EFS approaches exploit local learning by a recursive fuzzily weighted least squares (consequ. func. for each rule separately)

Fuzzy Weight

Objective Function

Inverse Hessian Matrix

Dr. Edwin Lughofer

Local Verus Global Learning (All Params in All Rules at Once)

• More Flexibility during learning: Rules can be added, pruned or merged by not ‚disturbing‘ the recursive parameter update of the others(Global learning requires a spec. Handling of the inverse Hessian in such cases as size changes)• O(C(p+1)^2) versus O((C(p+1))^2)• More stability as dealing with smaller Hessian matrices • Local feature weighing capability (per local region)• Better Interpretability (piecewise linear approx. along real trend of curve)

Dr. Edwin Lughofer

Comparison Global vs Local Estimation: Consequent Functions

Consequent Functions obtained by global estimation -> not interpretable

Consequent Functions obtained by local estimation -> interpretable (piecewise local tendencies of

variables (gradients))

Dr. Edwin Lughofer

Success Cases of Consequ. Adaptation

Model Refinement

Dr. Edwin Lughofer

Success Cases of Consequ. Adaptation

Closure of Data Hole

Dr. Edwin Lughofer

Success Cases of Consequ. Adaptation

Slight Range Extension

Dr. Edwin Lughofer

Failure Cases (Consequ. Adaptation)

Strong Changing Behavior

Bad Approx.

Dr. Edwin Lughofer

Failure Cases (Consequ. Adaptation)

Significant Range Extension

Bad Approx.

Dr. Edwin Lughofer

Failure Cases (Consequ. Adaptation)

FAZIT: Consequent adaptation not sufficient => Evolving Components Required

Bad Approx.

Dr. Edwin Lughofer

Advanced Issues in Consequent Adaptation Orthogonal Regressors: If regressors could be made orthogonal (in incremental manner synchronously to the model updates), dynamic feature selection (discarding, adding features)would be possible as each consequent parameter can be estimated separately

Possibility: via Principal Fuzzy Component Regression using iPCA + EFS

Jth Consequent in Ith Rule

Dr. Edwin Lughofer

Advanced Issues in Consequent Adaptation Alternative Optimization Function: including punishment terms as least squares is biased towards over-fitting (fits error on training data)

Dr. Edwin Lughofer

Possible Alternatives

In-Sample Error:

Rule-Based Constrained Least-Squares (Solution in Batch exists: SparseFIS)

Generalized Total Least Squares (Modeling error in inputs)

No incremental/recursive optimization method exists for these

Dr. Edwin Lughofer

Incremental Learning of Non-Linear Antecedent Parameters

Update of only one

param in one cycle

Recursive LM (used in EFP)

Dr. Edwin Lughofer

EFS Approaches (Alphabetically)

DENFIS (Kasabov) ePL (Lima, Ballini and Gomide) eTS (Angelov and Filev) FLEXFIS (Lughofer) SAFIS (Rong et al.) SEIT2FNN (Juang, Tsao) SOFNN (Leng et al.) SONFIN (Juang and Lin)

Bottom Line: Most of these are using TS fuzzy systems and R(FW)LS; antecedent learning and rule evolutionare performed with different concepts

Dr. Edwin Lughofer

FLEXFIS: Antecedent Learning (AL) and Rule Evolution (RE)

AL and RE is performed in the Clustering Space => each cluster is associated with one rule (local region) evolving version of Vector Quantization (basic) (eVQ, Lughofer 2008): Three central steps: Checking whether new sample fits into current cluster

parition (distance to nearest cluster higher than vigilance) Movement of winning = nearest cluster towards current

sample

Update of ranges of influence of winning cluster

Dr. Edwin Lughofer

Example Rule Evolution / Update

Movement Horizon (bounded by vigilance)

Dr. Edwin Lughofer

Cluster Projection => Forming Fuzzy Sets and Rules

Dr. Edwin Lughofer

Extensions to eVQ

Deletion of Cluster Satellites

Usage of Mahalanobis Distance (Ellip. in Arb. Pos.) New Distance Measure Concept (to Surface instead of Center) Integration of Feature Weights for Classification (preventing generation of superfluous clusters due to large distances in unimportant dimensions)

Dr. Edwin Lughofer

Feature Weight Integration (Example)

Dr. Edwin Lughofer

Combining Antecedent with Consequent Learning

Problem: structural change in Fuzzy Systems in each inc. learning cycle => Convergence to Optimality in LS sense not guaranteed Integration of Correction Terms (before RFWLS)

No explicit calculation of correction terms achieved so far, but partial results: Decreasing sequence of correction terms towards 0 In a Quasi-monotonic fashion => Sum of correction

terms bounded

Dr. Edwin Lughofer

Block-size versus Accuracy (Example)

Dr. Edwin Lughofer

Success of Update with FLEXFIS

Update Params only Update Structure + Params

Dr. Edwin Lughofer

Success of Update with FLEXFIS

Update Params only Update Structure + ParamsUpdate Structure + Params

Dr. Edwin Lughofer

Variants of FLEXFIS (FLEXFIS Family)

FLEXFIS for Regression (previous Slides)Edwin Lughofer. FLEXFIS: A Robust Incremental Learning Approach for Evolving

TS Fuzzy Models, IEEE Trans. On Fuzzy Systems, vol. 16 (6), pp. 1393-1410, 2008

FLEXFIS for Classification using Single-Model Architectures (FLEXFIS-Class SM)

FLEXFIS for Classification using Multi-Model Architectures (FLEXFIS-Class MM) K TS-Fuzzy Models based on Indicator Entries for K Classes One-versus-rest classificationEdwin Lughofer, Plamen Angelov and Xiaowei Zhou.Evolving Single- and Multi-Model Fuzzy

Classifiers with FLEXFIS-Class, in Proceedings of FUZZ-IEEE 2007, pp. 363-368, London, UK, 2007

eVQ-Class (Clustering-based Classifier)Edwin Lughofer, On-line Evolving Image Classifiers and Their Application to Surface

Inspection, Image and Vision Computing, Vol. 28 (7), pp. 1065-1079, 2010

Singleton Consequents

Dr. Edwin Lughofer

FLEXFIS-Class MM: Solving Masking Effect of Linear Regression by Indicator

Linear Regr. By Indicator matrix, 3 classes => red class nowhere

maximal (masking)

Fuzzy regression by indicator matrix, 3 classes

Dr. Edwin Lughofer

eVQ-Class: Classification Strategy Variant B (distance and class weight)

Dr. Edwin Lughofer

Advanced Concepts (For more Process-Safety and Robustness) • Instabilities in the learning process (ill-posed problems)

• Solution by regularization (Tichonov, sparsity constraints, parameter choice rules)

• Upcoming drifts and shifts in on-line data streams» Solution by Drift detection (rule ages) and reaction

(gradual forgetting) (Lughofer and Angelov, ASOC, 2010)

• Situations causing an unlearning effect (in steady states) (Lughofer, EFS’06, Book Chapter EIS 2010)

» Solution by Updating only significantly firing rules• Outliers and faults in on-line data streams

» Rule base procrastination » Features characterizing outlook/behav. of clusters/rules

Dr. Edwin Lughofer

Drift Reaction - Example

Conventional Update =>

cluster joins both distribution

Update with forgetting => Cluster covers new

distribution (old samples

completely forgotten

Dr. Edwin Lughofer

Outlier and Faults in Regression Modelling (Example)

Single Outlier Systematic Faults

Dr. Edwin Lughofer

Advanced Concepts (For more Process-Safety and Robustness)

Uncertainties in the models due to high noise levels or lack of data Adaptive local error bars in regression problems (for

evolving TS fuzzy models) (Lughofer, Guardiola EFS’ 08) Concept of conflict and ignorance, confidence levels

in class responses account for conflict, maximal firing degree of rules for ignorance

Extrapolation situations for new samples to be predicted (regression problems) (Lughofer, 2006/2010) Omittance of reactivation of inner Fuzzy Sets

Dr. Edwin Lughofer

Adaptive Local Error Bars (Example)

Constant Error bands Adaptive Local error bars

Dr. Edwin Lughofer

Reactivation Inner Sets

Dr. Edwin Lughofer

Improving Performance and Increasing Useability

Dynamic (On-line) rule split-and-merge Strategies overcoming local redundancies (Lughofer and Bouchot, IEEE SMC-B

2011, submitted) and changing distributions over time

Dynamic soft dimension reduction with incremental feature weighting (Lughofer, Fuzzy Sets and Systems, 2010) Contributes to changing feature importance levels = weights during

incremental learning in smooth manner

Active and semi-supervised learning (for reducing annotation effort of operators) (Lughofer, ICMLA 2010)

Incremental Classifier Fusion (Sannen, Lughofer et al, IDA, 2010): exploiting diversity of classifiers for boosting performance

Alternatives to Incremental/Evolving Learning: Lazy Learning

Dr. Edwin Lughofer

Problem of Rule Update/Movement

Dr. Edwin Lughofer

Example Merged Solutions (Strongly Overlapping Rules)

Merging formula of two clusters

Dr. Edwin Lughofer

Motivation Soft Dimension Reduction

Task: Combining Dimensionality Reduction with On-Line Evolution of Classifiers in an Incremental andHomogenous Manner

• One Possibility: A Priori Feature Selection (as Filters, Expert-Knowledge) keeping fixed during Incremental Learning Phase => static, no exchange of inputs

• Another Poss. : Incremental Feature SelectionSynchronously to Update of Fuzzy Classifiers

Problem with Inc. Feature Selection: How to exchange input features in EFC dynamically, on-the-fly without re-training phase??

Input Features Cycle #1: 1, 5, 9, 15, 20

Input Features Cycle #2: 1, 5, 9, 12, 20Exchange of Feat #12 with #15 in the list

of 5 most important ones

Dr. Edwin Lughofer

Our Approach SummaryExploiting Generalization Concept of IncrementalFeature Selection = Incremental Feature Weighting=> Assigning Weights in [0,1] to Features According to

their Importance Levels (0 = Not important, 1 = veryimportant) – measured by LOFO and Dy-Brodley‘s sep. criterionAchievements (Lughofer, Fuzzy Sets and Systems 2010):

• Soft Dimension Reduction (Features whose weightsare approaching 0 are nearly discarded)

• Dynamic Dimension Reduction (Feature Weightsare incrementally updated)

• Smooth Learning Behavior: sample-wise inc. learning changes the impact of features step-wisecontinuously

Dr. Edwin Lughofer

Interpretability Issues (To be handled in current FWF/DFG Project) Reduction of Unnecessary Complexity

Methods for improving Interpretability of EFS Linguistic interpretability (readability,

understandability of the rules and granules, consistency)

Visual interpretability (tracking development of fuzzy sets, rules over time, visual transparency in GUI)

Reliability issues (how reliable is the evolved model in its predictions)

Dr. Edwin Lughofer

Applications (Overview) On-line System Identification and Prediction Multi-Channel Measurement Systems (static) Prediction Models for NOx Emission (static+dynamic+mixed) Prediction Models for Resistance Value (dynamic)

On-line Fault and Anomaly Detection In stationary data (from multi-channel system) In time series data (dynamic modelling) In audio data (noise detection, analogue tapes)

Visual Inspection Systems On-line Image Classification in Surface Inspection Problems Emotion-based perception modelling (for visual textures)

Potential Future Applications Chemometric Modelling (high-dim spectral data) Enhanced Human-Machine Interaction Systems

Dr. Edwin Lughofer

Results NOx Prediction Models(Comparison Physical versus EFS)

Dr. Edwin Lughofer

Results Non-Linear Dynamic SysID

Dr. Edwin Lughofer

Results Rolling Mills (Prediction of Resistance Values)

Drift of the Error as no forgetting of older stitches takes place

No drift of the error as forgetting older stitches

achieves more flex.

Dr. Edwin Lughofer

Online FD Framework

Anomaly Detection in Res Signal

FLEXFIS

Adaptive Local Error Bars

Dr. Edwin Lughofer

On-line Image Classification Framework (for Surface Inspection)

HMI 1: Incorporating On-Line Operator‘s(s‘) Feedback into Classifier: Inc./Evolv Classifiers

HMI 2: Handling Input from Multiple Operators

HMI 3: Handling Variable Levels of Detail in Operators Inputs

HMI 4: Accomodating Partial Confidence of Operators

HMI 5: Operator Insight and Enhanced Interaction (Model Level)

E. Lughofer, J.E. Smith, M. A. Tahir, C. Eitzinger, D. Sannen, „Human–Machine Interaction Issues in Quality ControlBased on On-Line Image Classification“, IEEE Trans. On Systems, Man and Cybernetics, part A: Systems and

Humans, vol. 39(5), pp. 960-971, 2009

Deviation to Master

Automatic ROI extraction

Dr. Edwin Lughofer

CV Accuracies Surface Inspection Problems (class. into good and bad)

Dr. Edwin Lughofer

Accuracies on Separate Test Data Static vs. Evolved Image Classifiers

Dr. Edwin Lughofer

Impact of Incremental Feature Weighting

Evolution of Test Accuracies

for Egg Data Set

Number of features with weights more than 0.05 resp. 0.8

Dr. Edwin Lughofer

Impact Incremental Classifier Fusion

Evolution of Test Accuracies using 3 base

classifiers

Evolution of Test Accuracies using

ensemble classifiers (exploiting diversity of

base cl.)

Dr. Edwin Lughofer

List of References• Basic Algos (FLEXFIS):• Edwin Lughofer. FLEXFIS: A Robust Incremental Learning Approach for Evolving TS Fuzzy Models, IEEE

Transactions on Fuzzy Systems, vol. 16 (6), pp. 1393-1410, 2008• Edwin Lughofer. Extensions of Vector Quantization for Incremental Clustering, Pattern Recognition, vol. 41(3),

pp. 995-1011, 2008• Edwin Lughofer, Plamen Angelov, Xiaowei Zhou. Evolving Fuzzy Classifiers using Different Model

Architectures, Fuzzy Sets and Systems, vol.159 (23), pp. 3160-3182, 2008• Edwin Lughofer, Plamen Angelov and Xiaowei Zhou, Evolving Single- and Multi-Model Classifiers with

FLEXFIS-Class, in Proceedings of FUZZ-IEEE 2007, pp. 363-368, London, UK, 2007• Edwin Lughofer. Evolving Vector Quantization for Classification of On-line Data Streams, Proc. of the

International Conference on Computational Intelligence for Modelling, Control and Automation (CIMCA), pp. 780-786, Vienna, 2008

• Advanced Concepts in EFS:• Edwin Lughofer, On-line Incremental Feature Weighting in Evolving Fuzzy Classifiers, Fuzzy Sets and

Systems, to appear• Edwin Lughofer and Plamen Angelov, Handling Drifts and Shifts in On-Line Data Streams with Evolving Fuzzy

Systems, Applied Soft Computing, in press• Davy Sannen, Edwin Lughofer and Hendrik van Brussel, Towards Incremental Classifier Fusion, Intelligent Data

Analysis, Vol. 14 (1), pp. 3-30, 2010• Edwin Lughofer. Towards Robust Evolving Fuzzy Systems, book chapter in Evolving Intelligent Systems –

Methodologies and Applications, editors: Plamen Angelov, Dimitar Filev and Nik Kasabov, John Wiley and Sons, 2010, pp. 87-126

• Edwin Lughofer. Process Safety Enhancements for Data-Driven Evolving Fuzzy Models. In Proceedings of the International Symposium on Evolving Fuzzy Systems (EFS), pp. 42-48, Lake District, UK, September 2006 (awarded as best paper)

• Edwin Lughofer, Jean-Luc Bouchot, On-line Elimination of Local Redundancies in Evolving Fuzzy Systems, IEEE Transactions on Systems, Man and Cybernetics, part B: Cybernetics, submitted

Dr. Edwin Lughofer

List of References (cont‘d)

• Applications:

• Edwin Lughofer, On-line Evolving Image Classifiers and Their Application to Surface Inspection, Image and Vision Computing (special issue of on-line pattern recognition and machine learning techniques for computer vision), Vol. 28 (7), pp. 1065-1079, 2010

• Edwin Lughofer, Carlos Guardiola, Vicente Macian, Erich Peter Klement, Identifying Static and Dynamic Prediction Models for NOx emissions, in revision, Applied Soft Computing

• E. Lughofer, J.E. Smith, M. A. Tahir, C. Eitzinger, D. Sannen, Human–Machine Interaction Issues in Quality Control Based on On-Line Image Classification, IEEE Trans. On Systems, Man and Cybernetics, part A: Systems and Humans, vol. 39(5), pp. 960-971, 2009

• Christian Eitzinger, Wolfgang Heidl, Edwin Lughofer, Stefan Raiser, James E. Smith, Muhammad A. Tahir, Davy Sannen and Hendrik van Brussel, Assessment of the Influence of Adaptive Components in Trainable Surface Inspection Systems, Machine Vision and Applications, Vol. 21 (5), pp. 613-626, 2010

• Edwin Lughofer and Carlos Guardiola. On-Line Fault Detection with Data-Driven Evolving Fuzzy Models. Control and Intelligent Systems, vol. 36 (4), pp. 307-317, 2008

• D. Sannen, M. Nuttin, J.E. Smith, M.A. Tahir, P. Caleb-Solly, E. Lughofer, C. Eitzinger. An On-line Self-adaptive and Interactive Image Classification Framework, in International Conference on Computer Vision Systems 2008, A. Gasteratos, M. Vincze, J.K. Tsotsos (Eds.), Springer Lecture Notes 5008, pp. 171-180, Santorini Island, Greece

• Edwin Lughofer and Carlos Guardiola. Applying Evolving Fuzzy Models with Adaptive Local Error Bars to On-line Fault Detection, in Proc. of the 3rd International Workshop on Genetic and Evolving Fuzzy Systems, GEFS 2008, pp. 35-40, Witten-Bommerholz (Germany) (awarded as best paper finalist)

Dr. Edwin Lughofer

Monography

Edwin Lughofer

Evolving Fuzzy Systems Methodologies Advanced Concepts Applications

To be submitted to Springer (Heidelberg) in November 2010 (approx. 150 figures, 250 images, 460 pages)

Dr. Edwin Lughofer

Thank you for your attention!