General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.

Users may download and print one copy of any publication from the public portal for the purpose of private study or research.

You may not further distribute the material or use it for any profit-making activity or commercial gain

You may freely distribute the URL identifying the publication in the public portal If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.

Downloaded from orbit.dtu.dk on: Sep 01, 2021

Influence of the Migration Process on the Learning Performances of Fuzzy KnowledgeBases

Akrout, Khaled; Baron, Luc; Balazinski, Marek; Achiche, Sofiane

Published in:Annual Meeting of the North American Fuzzy Information Processing Society Conference

Publication date:2007

Document VersionPublisher's PDF, also known as Version of record

Link back to DTU Orbit

Citation (APA):Akrout, K., Baron, L., Balazinski, M., & Achiche, S. (2007). Influence of the Migration Process on the LearningPerformances of Fuzzy Knowledge Bases. In Annual Meeting of the North American Fuzzy InformationProcessing Society Conference (pp. 478-483). IEEE.

Influence of the Migration Process on the LearningPerformances of Fuzzy Knowledge Bases

Khaled Akrout, Luc Baron, Marek Balazinski and Sofiane AchicheDepartment of Mechanical Engineering

Ecole Polytechnique of MontrealP.O. box 6079, succ. CV

Montreal, Quebec, Canada, H3C 3A7E-mail: Khaled.Akrout@..., Luc.Baron@polymtl.ca

E-mail: Marek.Balazinski@ ..., Sofiane.Achiche@polymtl.ca

Abstract- This paper presents the influence of the process of modules on the left and the knowledge discovery modulesmigration between populations in GENO-FLOU, which is an on the right. The latter includes a Mamdani's type fuzzyenvironment of learning of fuzzy knowledge bases by genetic system module (described in section III) and a genetic learningalgorithms. Initially the algorithm did not use the processof migration. For the learning, the algorithm uses a hybrid module (descrlbed insecton IV) that follows the Pittsburghcoding, binary for the base of rules and real for the data base. approach. The left part includes several data preprocessingThis hybrid coding used with a set of specialized operators of such as preparation, formatting, analysis, clustering and featurereproduction proven to be an effective environment of learning. selection modules. In this paper, we investigated the introduc-Simulations were made in this environment by adding a process tion of a migration process within the genetic learning moduleof migration. While varying the number of populations, the onon e i n a gnumber of generations and the rate of migration or simply the of Geno-Floumigration of the best elements, on various types of problems. Ingeneral, simulations show a significant improvement of the results GENO-FLOU1 Anl Interactive LEarning Environment for Fuzzy Knowledge Bdasobtained with migration. The variation of these parametersmakes it possible to conclude on the dominating importance ofthe number of migrant generations. Data F Qdeaure

Clustering Selecton Genetic Leanii ngMoule Modul.e MXNoduleI. INTRODUCTION

Nowadays fuzzy logic is increasingly used in decision-aidedsystems because it offers several advantages over other tradi- L D0FKBM . .0 0FKB*tional decision-making techniques. The fuzzy decision supportsystems (FDSS) can easily deal with incomplete or/and impre-cise knowledge applied to either linear or nonlinear problems. rearation

. Analysis Fuzzy SystemMafalo4l.. Anle ModuleIn general, FDSS requires a fuzzy knowledge base (FKB) in Module Moduleorder to support the decision-making process of end-users. The LD =Learning Data FKB =Fuzzy Knowledge BaseFKB can be either manually created by a human expert orautomatically learned from a set of experimental data. Since Data preparation and mining pr Knwledge disoey part

the manual creation is very long and tedious, many research Fig. 1. Architecture of GENO-FLOU.works have been conducted toward the automatic learningof fuzzy rules alone, e.g., [1], [2], or the fuzzy sets alone,e.g., [3], [4]. More recently, other research works proposed to II. DATA PREPARATION AND MINING MODULESperform a simultaneous learning of both fuzzy rules and sets,including their quantity and distribution over each premises As shown in Fig. 1, the data preparation module is aimedand conclusion, e.g., [5], [6]. This learning process of FKBs is to account for different format of learning data sources, whileperformed with a genetic algorithm (GA) along the Pittsburgh the analysis module can be used to perform statistical andapproach [7], [8]. multi-regression analysis. The clustering module is aimed

In this paper, we use the Geno-Flou, an integrated learning to get ride of superfluous repetition, and hence, result in aenvironment, which allow the automatic learning of FKBs data reduction. Two main different clustering algorithms havefrom different preprocessed sets of experimental data. GENO- been implemented [16]: AGNES (for Agglomerate NESting)FLOU is a strategic research project sponsored by the Natural algorithm and the K-mean algorithm. Finally, the featureSciences and Engineering Research Council (NSERC) of selection module is aimed to determine the most appropriateCanada. As shown in Fig. 1, the architecture of GENO-FLOU input variable that observe the desired output. This selectionis made of two main parts: the data preparation and mining can be manually done or automatically done using another

1-4244-1214-5/07/$25.00 ©B2007 IEEE 478

GA that performs the supervision of the input learning data CRI i.e.: V-supA-A-A; V-sup*-*-*; 1-supA-A-A; and Z-together with the FKB learning along the Pittsburgh approach. sup*-*-* are expressed as

III. Fuzzy SYSTEM MODULE Y/ k=1 SUP (5)A rule-based approach to the decision making using fuzzy k=1 {zi E Xi IN

logic techniques may consider imprecise vague language as *t( *t (XI XI) *t (Xk. Xk,yk)))a set of rules linking a finite number of conclusions. The N( ) (knowledge base of such systems consists of two components: where *t (.) denotes the t-norm of (.) defined as either A or *.a linguistic terms base and a fuzzy rules base. The former is These variants of CRI mechanisms allow us to obtain differentdivided into two parts: the fuzzy premises (or inputs) and the conclusions represented as the membership function Y'. In thisfuzzy conclusions (or outputs). In general, the fuzzy systems fuzzy system module, there are three defuzzification methods:are either of type: SISO (Single Input and Single Output); the center of gravity (COG); the mean of maxima (MOM);MISO (Multiple Inputs and Single Output); and finally MIMO and the height method (HM). All the results presented in(Multiple Inputs and Multiple Outputs). For the initial version this paper are obtained with the X-sup*-*-* CRI and COGof GENO-FLOU, we limit ourself to fuzzy systems of N as defuzzification. This initial implementation of the fuzzymultiple inputs and one single output (i.e., MISO). Moreover, system module is based on the FDSS Fuzzy-Flou softwarefor the sake of simplicity, we consider general non-symmetric previously developed at Ecole Polytechnique (Canada) andtriangular fuzzy sets on the premises and sharp-symmetric the Technical University of Silesia in Gliwice (Poland). Astriangular fuzzy sets on the conclusion. The representation shown in Fig. 2, the graphic interface of this software is madeof such imprecise knowledge by means of fuzzy linguistic of two parts. The left window shows the rules base system,terms makes it possible to carry out quantitative processing while the right window shows the fuzzy membership functionsin the course of inference that is used for handling uncertain distributed on the two premises and the conclusion of this(imprecise) knowledge. This is often called approximate rea- simple FKB.soning [9]. Such knowledge can be collected and deliveredby a human expert (e.g. decision-maker, designer, processplaner, machine operator). This knowledge, expressed by (k 2

1,2, K) finite heuristic fuzzy rules of the type MISO, maybe written in the form: Pri

k if ,iSXk iS Xkth Y '1 4li97rr .rrRMIso 1 is X and ...XfN then y S yk, (1) 2:. .

where {X~}N1 denote values of linguistic variables {X}IN1 ii - -(conditions) defined in the following universe of discourse WilaRl .m 12I40r. 4 Ir 7 a{Xi}f-L1 and Yk stands for the value of the independent lI [1AO ml

linguistic variable y (conclusion) in the universe of discourse ml _: :Y. The global relation aggregating all rules from k I to K 'KtRIr.1dig IUUis given as

I II 1 L44ao t nkm 3L E eIta COA

where the sentence connectivealso denotes any t- or s-norm(e.g., min (A) or max (V) operators) or averages. For a given Fig. 2. Graphic interface of the FDSS Fuzzy-Flou software.set of fuzzy inputs {X,}N (or observations), the fuzzy outputY' (or conclusion) may be expressed symbolically as:

Y- (XI) XI)*.. )XI )oR, (3) IV. GENETIC LEARNING MODULE-(1: 2:- N)The GAs are powerful stochastic optimization methods

where o denotes a compositional rule of inference (CRI), e.g, presented first in the early 70's by John Holland [10]. Thesethe sup-A or sup-prod (sup-*). Alternatively, the CR1 of methods are based on the analogy of the mechanics of naturaleq.(3) is easily computed as genetics, and imitates the Darwinian survival-of-the-fittest

Y' = XN o ... o (X2 ° (XI O R)). (4) approach. A skilled FKB must satisfy two contradictory objec-tives: 1) the FKB must accurately reproduce the set of learning

In this fuzzy system module, there are four variants of CRI: data, while interpolating or extrapolating fair conclusions forthe sentence connective also can be either V or sum (Z); other data sets; 2) the FKB must be of low complexity suchthe compositional operator is the supremum (sup) of either A as measured by the number of fuzzy rules. This minimalistor *, denoted supA and sup*; while the sentence connective objective for the learning of FKBs allow a higher level ofanid and the fuzzy relation are always identical to the second generalization, while obtaining FKBs of size still manageablepart of the latter. For the sake of brevity, all four variants of by a human being.

479

As shown in Fig. 3, the genetic learning module follows where each Gxi is given asthe Pittsburgh approach. Each individual of the populationis a potential FKB. The method uses four basic operations: GxXi,x K.}' (8)reproduction; mutation; evaluation and natural selection, in summitl sUmmit2 summitKiorder to iteratively improve the population of FKBs at eachgeneration toward multiple optima simultaneously. with Ki being the number of fuzzy sets on the premise i (or the

conclusion). The two fuzzy sets located at the extreme limitsPopulation of Fuzzy Knowledge Bases are not included in the coding since they are always there. It

{ 0==J3 FKB1 FKJ 0000 FKB1 } is noteworthy that Ki can be different for each premise i andi FKB F~FKB KB * 0 0 FKB f

can also be different, for the i, from an FKB to another FKB.GA + + IAs a result, the genotype Gsets has a variable length.

Reprofiducti Natura lS A fuzzy rule, namely ri, is coded as an ordered lists ofinteger numbers pointing from one fuzzy set on each premise

MutaIon EvalutIon I LD toward one fuzzy set of conclusion. Therefore, the genotype-------------- --------------- of the fuzzy rulesisdefined as

Grules { rl r, * rK (9)Fuzzy System rulel rule2 ruleK

Ffififidffi iiifiihdd ngie Ddifffidufinwhere K is the number of active fuzzy rules. It is noteworthyFuzzification Inference' Engin Defzzification -that two FKBs may have different values of K, and thus,

different numbers of fuzzy rules. As a result, the genotypeFig. 3. Fuzzy system learning based on Pittsburgh approach. Grules has also variable length.

B. FKB ReproductionA. FKB Coding

For the sake of implementation simplicity, most of the As shown in Fig. 3, the reproduction is obtained from

research works with GAs have been conducted with Binary a multi-crossover mechanism which apply three crossover

Coded Genetic Algorithms (BCGA), thus resulting in poor operators on different parts of the genotype G.accuracies of results. Many optimization problems require real First, the crossover on Gsets is implemented with thenumber representation rather than binary, and hence, a Real blended crossover a (BLX-a) [13], where a controls theCoded Genetic Algorithms (RCGA) is more appropriate. The exploitation/exploration level, knowing that:representation of an FKB requires the coding of the following . in the exploitation zone, the offspring inherits behaviorsinformation: close to those of his parents;

1) the number of fuzzy sets and their locations as expressed . in the exploration zone, the offspring is a result of anby the location of the summit along all premises and exploration, therefore his attributes will be distant fromalong the conclusion; his parent's average.

2) the fuzzy rules connecting one fuzzy set of each premise In order to avoid any bias in either direction (exploitationto one fuzzy set on the conclusion. or exploration) the value of a is set to 0.5, which pro-

Apparently, the first item has to deal with a list of real numbers vides offsprings in the zone named the relaxed exploitation(the location of the fuzzy set summit), while the second item zone, as shown in Fig. 4. This multi-combinative strategyis a list of integers (pointers to fuzzy sets). Consequently, may overcome some aspects of the premature convergencean FKB must use a combination of both RCGA and BCGA encountered in the learning FKBs with the Pittsburgh approach[11], and hence, we call this combination a Hybrid Coded [14]. We have also studied the different scheduling explo-Genetic Algorithm (HCGA). We have performed a comparison ration/exploitation strategies for the learning of FKBs [15],study between HCGA and BCGA in [12]. In this context, the which can be summarized as follows. The exploration at thegenotype of an FKB, namely G, is defined as early stage, relaxing in the middle stage and exploitation at

G GsetsG Gruies }6) the end of the evolution is the most promising strategy.{ Csets, Cruies }' (6) Second, the crossover on Grules is implemented by a simple

where Gsets and Cruies are respectively the real coding of crossover. The operation is performed by inverting the end partfuzzy sets and the binary coding of fuzzy rules. The latter of Gsets of the parents at a randomly selected crossover site.Csets has many sublists as the number of premises N together Third, the fuzzy set reducer is a mechanism aiming to increasewith the conclusion, i.e., the simplicity level of FKBs by randomly selecting a fuzzy set

on a premise and erasing it together with the correspondingCGsets-{ Cx1 ,... i ,.. CXN Cxc }' (7) fuzzy rules. This mechanism allows one to try more simple

premisei premisei premiseN concluhsion FKB solutions (with less complexity).

480

Relaxe Exploit opi Pop Pp 3

| Expliwation EFxploitatin |xrioration|

Iminv-ILa | xi I mj| lFrt8X; Ica | ( popo P.p. Pop4 )

Fig. 6. Circular migration between populations

Fig. 4. Blended crossover a (BLX-a).

3) Single Migration: The single migration is similar to

C. FKB Mutation circular migration, but we extract at each generation only thebest FKB from each population and then transfer it into theAs shown in Fig. 3, the random uniform mutation is next population of the list.

implemented by toggling one bit of an arbitrary FKB at arandomly selected mutation site. V. LEARNING UNDER MIGRATION

D. FKB Natural Selection In order to investigate the influences of the different mi-The natural selection is performed on the population by gration strategies over the learning speed and fitness, we

keeping the most promising FKBs based on their fitness. have conducted several learning from the same dataset. TheThe first generation starts with p randomly generated FKBs. population size is set to 100 FKBs. In all cases, the initialAt each generation, the most promising FKBs have more FKBs are randomly generated.probabilities to reproduce themselves than others with lower Figure 7 shows the fitness evolution of the best FKB amongfitness. This elitist strategy is crucial in GA, since it provides the 16 populations with 0%, 5% and 10% of random migrationa direction of enhancement. The multi-crossover mechanism rate. Apparently, the random migration slightly accelerate theallows to double the size of the population at each generation, learning speed and fitness. However, after 100 generationsand hence, select only the first p FKBs that are kept for the random migration mechanism does not help anymore thethe future generations in order to keep the population size learning process.constant.

E. Migration between Populations 1 Best Fitness

Three types of migration mechanisms between populationshave been implemented, while keeping the population sizeconstant.

1) Random Migration: As shown in Fig. 5, it concistes toextract at each generation a pourcentage of randomly selectedFKBs from each population and then redistributed them toother populations, while keeping the population size constant. 0.9

5%

* (Ens) : generation0.8 I I I I I I I I

0 20 40 60 80 100 120 140 160 180

Fig. 7. Best fitness in terms of generation for 0%, 5% and 10% of randomPop4.....P ......P 0 migration rate

Fig. 5. Random migration between populations As expected, the random migration produces the best learn-ing performances, while the circular migration performs well

2) Circular Migration: As shown in Fig. 6, the populations and the single migration produces the same results as havingare placed into acircular list. At each generation apourcentage no migration mechanism at all. This differences betweenof randomly selected FKBs are extracted from each population the migration mechanisms appears only for low number ofand transfered in the next population of the list, generation.

481

Best Fitness1 1 Best Fitness

0.90 8,9,16

migration rate(%) generation0.8 - U0.8

0 1 2 3 4 5 6 7 8 9 0 20 40 60 80 100 120 140 160 180Fig. 8. Best fitness in terms of migration rate

Fig. 9. Best fitness in terms of generation for 2, 4, 8 and 16 populations

Figure 8 shows the fitness evolution of the best FKB amongthe 16 populations at the 16th generation. Apparently, the best of Canada under grants RGPIN-203618, RGPIN-105518 andresult is obtained at 5% of random migration rate. STPGP-269579.

Figure 9 shows the fitness evolution of the best FKB throughthe generation for different number of population from 2 to 16 REFERENCES(from the lower to the upper curves) with a random migrationrate of 5%. At the 30th generation, the migration mechanism [1] G. Castellano, G. Attolico and A. Distante, "Automatic generation ofallows to increase the fitness by 2% to 4.5%. Apparently, the fuzzy rules for reactive robot controllers", Robotics Autonomous Systems,best result is obtained for 16 populations. 22, pp. 133-149, 1997.

[2] G. Casillas, 0. Cord6n and F. Herrera, "A methodology to improve adhoc

VI. CONCLUSIONSdata-driven linguistic rule learning methods by inducing cooperationV*.CONCLUSIONS among rules", Technical report #DECSAI-OOO101, University of Granada,

In this paper, we have investigated the use of different Spain, 2000.[3] H. Nomura, I. Hayashi and N. Wakami, "A self-tuning method of fuzzymigration mechanisms in a genetic-based learing system of reasoning by genetic algorithm" in Proceedings of the 7th Interna-

FKBs from dataset. Three migration mechanisms have been tional Conference on Genetic Algorithms, Morgan Kaufmann Publishers,implemented in Geno-Flou, an interactive teaching environ- pp. 236-245, 1992.

ment for FBs.Teknoledgediscoeryprtofthesstem [4] J.R. Valesco, S. Lopez and L. Magdalena, "Genetic fuzzy clustering forment for FKB s. The knowledge discovery part of the system the definition of fuzzy sets" in Proceedings of the 6th IEEE Internationalfollows the Pittsburgh approach, while the data preparation Conference on Fuzzy Systems, FUZZ IEEE, III, pp. 1665-1670, 1997.and mining part is interactive. The learning uses a hybrid [5] L. Baron, S. Achiche and M. Balazinski, "Fuzzy decision system knowl-coded genetic Algorithm to learn the fuzzy sets and rules edge base genaration using a genetic algorithm", International Journal

of Approximate Reasoning, vol. 28, no. 2-3, pp. 125-148, 2001.information. Its uses a multi-crossover mechanism together [6] s. Achiche, M. Balazinski and L. Baron, "Tool wear monitoring usingwith the different migration mechanisms to accelerate the genetically-generated fuzzy knowledge bases", Engineering Applicationslearning process. The migration mechanisms do not acelerate of Artificial Intelligence, vol. 15, no. 3-4, pp. 303-314, 2002.

[7] S.F. Smith, "A learning system based on genetic adaptive algorithms",the learning process for high number of generation (say more Doctoral dissertation, Department of computer science, University ofthan 100), while it enhanced the fitness by 2% to 4.5% at Pittsburgh, USA, 1980.30 generations. The ramdom migration mechanism is most [8] S.F. Smith, "Flexible learning of problem solving heuristics through

adaptive search" in Proceedings of the 8th International Joint Conferenceusefull one, when used with low migration rate such as 5%. on Artificial Intelligence, Morgan Kaufmann Publishers, pp. 422-425,CConsequently, we recommand the use of random migration 1983.mechanism, when the number of generation is low (say 30) [9] L.A. Zadeh, "Outline of new approach to the analysis of complex

systems and decisions processes", IEEE Transactions of Systems, Mansuch as in online applications. and Cybernetics, no. 3, pp. 28-44, 1973.[10] J. Holland, "Genetic algorithms and the optimal allocation of trials",

ACKNOWLEDGMENT SIAM Journal on Computing, vol. 2, no. 2, pp. 88-105, 1973.[11] 0. Cordon, F. Herrera, F. Hoffmann, and L. Magdalena, Genetic fuzzy

The authors want to acknowledge the financial support systems: evolutionary tuning and learning of fuzzy knowledge bases.from the Natural Sciences and Engineering Research Council Singapore: World Scientific, 2001.

482

[12] S. Achiche, L. Baron and M. Balazinski, "Real/binary-like coded versusbinary coded genetic algorithms to automatically generate fuzzy knowl-edge bases: a comparative study", Engineering Applications of ArtificialIntelligence, vol. 17, no. 4, pp. 313-325, 2004.

[13] F. Herrera and M. Lazano, "Gradual distributed real-coded geneticalgorithms", IEEE Transactions on Evolutionary Computation, vol. 4,pp. 43-63, 2000.

[14] S. Achiche, M. Balazinski and L. Baron, "Multi-Combinative Strategyto Avoid Premature Convergence in Genetically-Generated Fuzzy Knowl-edge Bases", in Proceeding of the IEEE International Conference onFuzzy Systems, Banff, Canada, june 2004.

[15] S. Achiche, L. Baron and M. Balazinski, "Scheduling Explo-ration/Exploitation Levels in Genetically-Generated Fuzzy KnowledgeBases", Jouranl of Theoretical and Applied Mechanics, vol. 42, no. 3,pp. 417-444, 2004.

[16] A. Przybylo, S. Achiche, M. Balazinski and L. Baron, "EnghancingFuzzy Learning with Data Mining Techniques", in Proceeding of the AI-MECH,Gliwice, Poland, nov. 2004.

[17] S. Achiche, L. Baron, M. Balazinski and M. Benaoudia, "Chips ImageProcessing to Predict Pulp Brightness using Fuzzy Logic Techniques", toappear in Proceeding of the 2005 Annual Meeting of the Pulp and PaperTechnical Association of Canada, Montreal, Canada, febuary 2005.

483