[ieee 2010 third international workshop on advanced computational intelligence (iwaci) - suzhou,...

8
A Novel Updating Strategy for Associative Memory Scheme in Cyclic Dynamic Environments Yong Cao and Wenjian Luo Abstract—Associative memory schemes have been developed for Evolutionary Algorithms (EAs) to solve Dynamic Optimization Problems (DOPs), and demonstrated powerful performance. In these schemes, how to update the memory could be important for their performance. However, little work has been done about the associative memory updating strategies. In this paper, a novel updating strategy is proposed for associative memory schemes. In this strategy, the memory point whose associated environmental information is most similar to the current environmental information is first picked out from the memory. Then, the selected memory individual is updated according to the fitness value, and the associated environmental information is updated according to the matching degree between environmental information and individuals. This updating strategy is embedded into a state- of-the-art algorithm, i.e. the MPBIL, and tested by experiments. Experimental results demonstrate that the proposed updating strategy is helpful for associative memory schemes to enhance their search ability in cyclic dynamic environments. I. INTRODUCTION VOLUTIONARY Algorithms (EAs) have been successfully used to solve stationary optimization problems [1]. However, there are many challenges when EAs are used to solve Dynamic Optimization Problems (DOPs). The optima of DOPs could change over time [2-7], and this requires EAs to track the moving optima. However, the convergence character of the traditional EAs makes them hard to adapt to dynamic environments. In order to improve the performance of EAs in dynamic environments, four typical approaches have been developed for them. The first one is to generate diversity after the environment changes, e.g. using the hypermutation operator [8, 9]. The second one is to maintain the population diversity throughout the whole run, and the most classical example is the random immigrants scheme [2, 10]. The multi-population strategy [11-16] is the third effective approach for EAs. The last one is the memory scheme [17, 18]. Based on the above four typical approaches, many effective hybrid schemes [19-29] have also been developed to enhance the search ability of EAs for DOPs. This work is partly supported by the 2006-2007 Excellent Young and Middle-aged Academic Leader Development Program of Anhui Province Research Experiment Bases. Yong Cao and Wenjian Luo (the corresponding author) are with the Nature Inspired Computation and Applications Laboratory, School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, Anhui, China. Both authors are also with the Anhui Key Laboratory of Software in Computing and Communication, University of Science and Technology of China, Hefei 230027, Anhui, China (e-mail: [email protected], [email protected]). Among these approaches, memory schemes have been paid more and more attention for their effective performance in dynamic optimization, especially in cyclic environments. There are two primary memory schemes to store useful information of environments, i.e. the implicit memory scheme [30-36] and the explicit memory scheme [18, 37-46]. For implicit memory schemes, genotype representations are used to memorize the useful information implicitly. For example, the diploidy or multiploidy representations [30-32] and dualism mechanisms [33-36] are often used for implicit memory schemes. While for explicit memory schemes, the useful information (good solutions and/or environmental information) is stored explicitly in memory and reused later. There are two kinds of explicit memory schemes, i.e. direct memory schemes [17, 18] and associative memory schemes [37, 42, 43]. In direct memory schemes, only good solutions are stored in the memory. While in associative memory schemes, the associated environmental information is also memorized by the memory. In order to keep the memory individuals fresh, one of the memory individuals should be selected and replaced by the current best individual at regular intervals. Generally, the most similar memory individual to the current best individual is often picked out. A good updating strategy could lead the memory to store appropriate information and make the memory individuals distribute in reasonable areas. However, little work has been done for the memory updating strategies, especially for the associative memory schemes. In 1999, four memory updating strategies were discussed by Branke [17]. For the first memory updating strategy, the least important memory individual is selected to be replaced. The importance of a memory individual is evaluated by its age, fitness or contribution to the diversity. For the second one, the individual which has least contribution to the memory variance will be replaced. For the third one, the most similar memory individual will be replaced if the new individual is better. For the last one, two memory individuals with the minimum distance are first picked out, and then remove the less fit one. Among these strategies by Branke [17], the third strategy is a simple and effective memory updating strategy, and it has been widely used in both direct memory schemes and associative memory schemes. Besides four memory strategies by Branke [17], another memory updating strategy is proposed in [45]. When the memory is updated with this strategy [45], the memory individual closest to the best individual is not removed. The current information is memorized by moving the memory individual towards the current best individual. E 32 Third International Workshop on Advanced Computational Intelligence August 25-27, 2010 - Suzhou, Jiangsu, China 978-1-4244-6337-4/10/$26.00 @2010 IEEE

Upload: luo

Post on 10-Mar-2017

212 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

A Novel Updating Strategy for Associative Memory Scheme in Cyclic Dynamic Environments

Yong Cao and Wenjian Luo

Abstract—Associative memory schemes have been

developed for Evolutionary Algorithms (EAs) to solve Dynamic Optimization Problems (DOPs), and demonstrated powerful performance. In these schemes, how to update the memory could be important for their performance. However, little work has been done about the associative memory updating strategies. In this paper, a novel updating strategy is proposed for associative memory schemes. In this strategy, the memory point whose associated environmental information is most similar to the current environmental information is first picked out from the memory. Then, the selected memory individual is updated according to the fitness value, and the associated environmental information is updated according to the matching degree between environmental information and individuals. This updating strategy is embedded into a state-of-the-art algorithm, i.e. the MPBIL, and tested by experiments. Experimental results demonstrate that the proposed updating strategy is helpful for associative memory schemes to enhance their search ability in cyclic dynamic environments.

I. INTRODUCTION 1.

VOLUTIONARY Algorithms (EAs) have been successfully used to solve stationary optimization problems [1]. However, there are many challenges

when EAs are used to solve Dynamic Optimization Problems (DOPs). The optima of DOPs could change over time [2-7], and this requires EAs to track the moving optima. However, the convergence character of the traditional EAs makes them hard to adapt to dynamic environments.

In order to improve the performance of EAs in dynamic environments, four typical approaches have been developed for them. The first one is to generate diversity after the environment changes, e.g. using the hypermutation operator [8, 9]. The second one is to maintain the population diversity throughout the whole run, and the most classical example is the random immigrants scheme [2, 10]. The multi-population strategy [11-16] is the third effective approach for EAs. The last one is the memory scheme [17, 18]. Based on the above four typical approaches, many effective hybrid schemes [19-29] have also been developed to enhance the search ability of EAs for DOPs.

This work is partly supported by the 2006-2007 Excellent Young and Middle-aged Academic Leader Development Program of Anhui Province Research Experiment Bases.

Yong Cao and Wenjian Luo (the corresponding author) are with the Nature Inspired Computation and Applications Laboratory, School of Computer Science and Technology, University of Science and Technology of China, Hefei 230027, Anhui, China. Both authors are also with the Anhui Key Laboratory of Software in Computing and Communication, University of Science and Technology of China, Hefei 230027, Anhui, China (e-mail: [email protected], [email protected]).

Among these approaches, memory schemes have been paid more and more attention for their effective performance in dynamic optimization, especially in cyclic environments. There are two primary memory schemes to store useful information of environments, i.e. the implicit memory scheme [30-36] and the explicit memory scheme [18, 37-46]. For implicit memory schemes, genotype representations are used to memorize the useful information implicitly. For example, the diploidy or multiploidy representations [30-32] and dualism mechanisms [33-36] are often used for implicit memory schemes. While for explicit memory schemes, the useful information (good solutions and/or environmental information) is stored explicitly in memory and reused later. There are two kinds of explicit memory schemes, i.e. direct memory schemes [17, 18] and associative memory schemes [37, 42, 43]. In direct memory schemes, only good solutions are stored in the memory. While in associative memory schemes, the associated environmental information is also memorized by the memory.

In order to keep the memory individuals fresh, one of the memory individuals should be selected and replaced by the current best individual at regular intervals. Generally, the most similar memory individual to the current best individual is often picked out. A good updating strategy could lead the memory to store appropriate information and make the memory individuals distribute in reasonable areas. However, little work has been done for the memory updating strategies, especially for the associative memory schemes.

In 1999, four memory updating strategies were discussed by Branke [17]. For the first memory updating strategy, the least important memory individual is selected to be replaced. The importance of a memory individual is evaluated by its age, fitness or contribution to the diversity. For the second one, the individual which has least contribution to the memory variance will be replaced. For the third one, the most similar memory individual will be replaced if the new individual is better. For the last one, two memory individuals with the minimum distance are first picked out, and then remove the less fit one. Among these strategies by Branke [17], the third strategy is a simple and effective memory updating strategy, and it has been widely used in both direct memory schemes and associative memory schemes.

Besides four memory strategies by Branke [17], another memory updating strategy is proposed in [45]. When the memory is updated with this strategy [45], the memory individual closest to the best individual is not removed. The current information is memorized by moving the memory individual towards the current best individual.

E

32

Third International Workshop on Advanced Computational Intelligence August 25-27, 2010 - Suzhou, Jiangsu, China

978-1-4244-6337-4/10/$26.00 @2010 IEEE

Page 2: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

In this paper, a novel memory updating strategy is proposed for associative memory schemes. In this strategy, the memory point whose environmental information is most similar to the current environmental information is picked out to be replaced. Here, a point includes an individual and its associated environmental information. The individual and the environmental information of the selected memory point are replaced separately. For the replacement of the individual, the one with the better fitness will be stored in the memory. As to the replacement of the environmental information, if the current environmental information matches with the stored individual at a higher degree, it will be stored to replace the selected memory environmental information. In the experiments, three dynamic test problems from [43] are adopted to test the performance of the novel memory updating strategy. Experimental results demonstrate that the proposed memory updating strategy is helpful to improve the performance of associative memory schemes in cyclic environments.

The rest of this paper is organized as follows. A typical associative memory scheme (i.e. the MPBIL [43]) and its memory updating strategy are reviewed in Section II. Section III describes the proposed novel associative memory updating strategy. Experiments are carried out in Section IV, and the experimental results are given. Finally, some discussions and conclusions are given in Section V and Section VI, respectively.

II. RELATED WORK In recent years, some effective associative memory schemes have been developed for EAs. For example, in [37], Ramsey and Grefenstette proposed a Case-based Initialization of Genetic Algorithm for a robot control problem. The GA with associative memory scheme (AMGA) and the memory-enhanced PBIL (MPBIL) are proposed in [42] and [43], respectively. The environment is detectable in the Case-based Initialization of Genetic Algorithm [37], but undetectable in the AMGA [42] and the MPBIL [43]. The allele distribution vector and the probability vector are taken as the environmental information for the AMGA and the MPBIL, respectively. In this paper, the proposed memory updating strategy is incorporated into the MPBIL [43] to test its performance.

The MPBIL [43] is proposed by introducing the associative memory scheme into the population-based incremental learning algorithm (PBIL) [47]. For convenience, the procedure of the MPBIL is shown in Algorithm 1.

In the MPBIL, the environmental information (i.e. the probability vector) and samples are interdependent. At generation t, the new population is sampled by the probability vector ))(),...,(()( 1 tPtPtP l= , where l is the encoding length. For each candidate solution ),...,( 1 lssS = , its sampling method is shown as Eq. (1) [43].

⎩⎨⎧ <

=otherwise

tPrandifs i

i ,0)()0.1,0.0(,1, },...,1{ li = (1)

As for the probability vector, it undergoes learning and mutation process in the MPBIL. At generation t, if no environmental change is detected, the probability vector

)(tP will learn towards the current best sample ))(),...,(()( 1 tbtbtB l= as the follow equation, where α is

the learning rate [43].

)()()1(:)1( tbtPtP iii ×+×−=+ αα , },...,1{ li = (2)

In every generation of the MPBIL, the mutation is performed on the probability vector. For each bit of the probability vector, the mutation happens with a certain probability, and the mutation progress is described as the Eq. (3) [43].

⎪⎩

⎪⎨

<+−×=>−×

=5.0 ,)0.1(5.0 ,5.0 ),0.1(

'

immi

ii

imi

i

PPPPPP

Pδδ

δ, },...,1{ li = (3)

In this equation, mδ is named as the mutation shift to control the severity of the mutation.

Algorithm 1. The MPBIL [43] // initialization 1: Set 0:=t and )10,5(: randtM = .

Set the probability vector )5.0,,5.0(:)( …=tP . Set the memory set φ=:)(tM .

Sample solutions for the sample set )(tS by )(tP . Evaluate the samples in )(tS .

2: REPEAT // the updating progress 3: IF Mtt = THEN

Update the memory set. )10,5(: randttM += .

// the retrieving progress 4: IF an environmental change is detected THEN

Denote the best memory point as ),( MM PB and the current best sample as BC .

IF )()( CM BfBf > THEN MPtP =:)( .

// the evolution progress 5: ELSE

Learn )(tP towards the current best sample.

Mutate the current working probability vector )(tP .

Use )(tP to generate a new sample set )(tS . Evaluate the samples of )(tS .

)(:)1( tPtP =+ ; )(:)1( tStS =+ . 1: += tt .

6: UNTIL the termination condition is satisfied. In the MPBIL, the environmental information (i.e. the

probability vector) and its associated best sample are stored in the memory, and reused later. In every generation of the MPBIL, memory samples are reevaluated to detect the environmental change. If an environmental change is

33

Page 3: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

detected, the stored information could be retrieved to enhance the search ability in the new environment. The fittest memory sample is selected to compete with the current best sample. If the selected memory sample is fitter than the current best sample, its associated probability vector is used to create new generation samples by replacing the current probability vector.

The memory in the MPBIL is updated at stochastic intervals. When the memory is updated, if the memory set is not full, store the current best sample and the current working probability vector into the memory directly. Otherwise, the memory sample closest to the current best sample in terms of the Hamming distance is picked out and compared. If the current best sample has a better fitness value than the selected memory sample, the memory sample and its associated probability vector are replaced by the current best sample and the current probability vector, respectively. This updating strategy is just the third strategy among those strategies discussed by Branke in [17], and for convenience, it is named as the individual based updating strategy in this paper.

III. THE PROPOSED MEMORY UPDATING STRATEGY For an explicit memory scheme, the memory space is often limited. An effective memory updating strategy is needed to make full use of the limited memory space. In direct memory schemes, the memory only stores good individuals. When to update the memory, the memory individual which is most similar to the current best individual will be picked out and replaced. In binary-encoded problems, the similarity is often calculated by the Hamming distance between two compared individuals. However, in associative memory schemes, besides good individuals, the associated environmental information is also stored in the memory. When to update the memory, the associated environmental information should be considered and it could be helpful for improving the performance of the updating strategy.

To our best knowledge, there is no associative memory updating strategy that considers the environmental information when to update the memory. In the individual based updating strategy, the memory point whose associated sample is closest to the current best sample in terms of Hamming distance is taken as the most similar memory point. If the current best sample is better than the selected memory sample, it will be stored in the memory together with the current environmental information (i.e. the working probability vector) to replace the selected memory point. Throughout the whole updating process, the environmental information is stored or replaced just following with its associated sample.

From the Eq. (1) and Eq. (2), it can be observed that, given a probability vector ),...,( 1 lPPP = , any sample

),...,( 1 lssS = could be generated by P with a fixed probability. This generation probability reflects the matching degree between the sample S and the environment represented by the probability vector P . The bigger generation probability means that the sample S matches the

environment represented by the probability vector P at a higher degree. This matching degree between the samples and the probability vector P could be helpful to organize the memory in the updating process.

According to Eq. (1), for each index },...,1{ li ∈ (l is the

encoding length), the probability vector P can generate “1” with the probability of iP , or generate “0” with the probability of iP−1 . Therefore, the generation probability of each bit in the sample S can be calculated with Eq. (4). Furthermore, the matching degree between the sample S and the probability vector P can be worked out with Eq. (5).

},...,1{,0,11,

),( lisPsP

PsProbii

iiii =

⎩⎨⎧

=−=

= (4)

∏==

l

iii PsProbPSeMatchDegre

1),(),( (5)

In this paper, a novel updating strategy for associative memory schemes is proposed, and this strategy is named as the environmental information based updating strategy. In this updating strategy, both finding the most similar memory point and conducting replacement are based on the matching degrees between probability vectors and samples. The process of the environmental information based updating strategy is shown in Algorithm 2.

Algorithm 2. The environmental information based updating

strategy 1 IF it is time to update the memory THEN 2 Denote the current best sample as BC, the working

probability vector as CP . 3 IF the memory set is not full THEN 4 Store BC and CP into the memory directly. 5 ELSE6 Find the memory point ),( MM PB which has the highest

),( MC PBeMatchDegre in the memory set. 7 IF )()( MC BfBf > THEN 8 CM BB =: . 9 IF ),(),( MCCC PBeMatchDegrePBeMatchDegre > THEN 10

CM PP =: . 11 ELSE 12 IF ),(),( MM MC PBeMatchDegrePBeMatchDegre > THEN

13 CM PP =: . In the environmental information based updating

strategy, the matching degree between the memorized environmental information and the current best sample is used to find the most similar memory point. If the memory environmental information and the current best individual have a high matching degree, the stored environment could be similar to the current environment. Therefore, the

34

Page 4: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

memory point whose associated probability vector matches to the current best sample with the highest degree is taken as the most similar one.

After the most similar memory point is found, the replacement process will be done. In the environmental information based updating strategy, the stored probability vector and the associated sample are replaced separately.

If the current best sample is better than the selected memory sample in terms of the fitness value, the memory sample will be replaced by it. In these cases, the current probability vector is stored to replace the selected memory probability vector if and only if its matching degree with the current sample is higher than that of the selected memory probability vector.

If the current best sample is worse than the selected memory sample, the memory sample remains unchanged, while the associated memory probability vector could be replaced. If the matching degree between the current probability vector and the selected memory sample is higher than that of the selected memory probability vector and the memory sample, the memory probability vector will be replaced by it.

This separated replacing process makes sure that the stored probability vector and the sample have a relatively higher matching degree.

When the memory is updated with the environmental information based updating strategy, if the memory set is not full, the current best sample and the current working probability vector are stored directly as the individual based updating strategy does. Otherwise, the environmental information based updating strategy stores the current best sample and the current working probability vector as the above description.

IV. EXPERIMENTS

A. Experimental Settings In this paper, the proposed environmental information based updating strategy is tested by replacing the individual based updating strategy in the MPBIL [43].

In [43], three dynamic test functions, denoted as DDUF1, DDUF2 and DDUF3 respectively, are constructed with a DOP generator [33, 34] based on three 100-bit binary-encoded stationary functions. These three stationary functions are the OneMax function, the Plateau function and the deceptive function, respectively. For each DDUF, three kinds of dynamic environments are constructed, i.e. the cyclic environment, the cyclic with noise environment and the random environment. These constructed dynamic test functions are adopted in the experiments to test the performance of the proposed environmental information based updating strategy.

The MPBIL with the environmental information based updating strategy is named as the eiuMPBIL. In this paper, the eiuMPBIL is compared with the MPBIL in dynamic environments. The common settings of the eiuMPBIL are the same as the MPBIL as description in [43]. In both of the MPBIL and eiuMPBIL, the size of the memory is set to 10. Including the memory size, the total population size is set to

100. As to the evolutionary parameters, the learning rate α and the mutation shift mδ for the probability vector are set to 0.25 and 0.05, respectively. The mutation probability is set to 0.02.

Experiments are carried out in three kinds of dynamic environments based on the DDUFs (i.e. the DDUF1, DDUF2 and DDUF3). For each experiment, the algorithms run totally 50 times with 5000 generations in each run. For fair, the MPBIL and the eiuMPBIL use 50 identical random seeds for each experiment. Every τ generations in a run, the environment changes with the change severity ρ . As the setting in [43], the parameter τ is set to 10 and 25 respectively, and the parameter ρ is set to 0.1, 0.2, 0.5 and 1.0 respectively. For the cyclic and cyclic with noise environments, the environment changes circularly among

ρ/2 base states. The average fitness value of the best sample across every

generation in each run is taken as the off-line performance for an algorithm, and it is shown as follows [43].

∑ ∑= =

=T

t

R

r

trBfRT

F1 1

)1(1 (6)

In this equation, T denotes the total generations in a run, R denotes the total run times in each experiment, and tr

Bf is the fitness value of the best individual in the tth generation of the rth run.

B. Experimental Results The experimental results are given in Table I and Table II. In this paper, both of the MPBIL and eiuMPBIL are implemented with C++ tools. The source codes are available from authors on request.

Like the settings in [43], the one-tailed, 98 freedom degrees and 0.05 significance level t-test is used to analyse the results of the two algorithms, i.e. the MPBIL and the eiuMPBIL. In Table I, the t-test results on DDUFs for three kinds of environments are given. In this table, five kinds of results, i.e. “s+”, “+”, “s−”, “−” and “~”, are obtained by comparing the eiuMPBIL with the MPBIL. For each comparison, if the eiuMPBIL is better or significantly better than the MPBIL, the t-test result will be marked as “+” and “s+”, respectively. Conversely, If the eiuMPBIL is worse or significantly worse than the MPBIL, “−” and “s−” will be obtained, respectively. Furthermore, “~” means that the performance of the eiuMPBIL and the MPBIL is completely equal.

In Table II, the off-line performance of both MPBIL and eiuMPBIL on DDUFs in three kinds of environments are given. The standard variance of the off-line performance is shown in the parentheses.

From Table I and Table II, some empirical results could be obtained as follows.

Firstly, from the results in cyclic environments, it can be seen that the eiuMPBIL performs significantly better than the MPBIL in most cases of the cyclic environments. This demonstrates that the proposed environmental information based updating strategy is beneficial for the associative

35

Page 5: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

memory scheme to solve DOPs in cyclic environments. In the environmental information based updating strategy, the environmental information is considered when the memory is updated. Comparing with the individual based updating strategy, the stored environmental information and the

associated sample could maintain a higher matching degree. When to retrieve the memory in the eiuMPBIL, the memory sample could represent its associated environmental information more reasonably, and more appropriate memory point could be retrieved for DOPs.

TABLE I

THE T-TEST RESULTS ON DDUFS IN THREE DYNAMIC ENVIRONMENTS

t-test Result DDUF1 DDUF2 DDUF3 Cyclic ρ 10=τ 25=τ 10=τ 25=τ 10=τ 25=τ

eiuMPBIL• MPBIL

0.1 s+ s+ + s+ s+ s+ 0.2 s+ s+ s+ s+ + s+ 0.5 + s+ s+ s+ + − 1.0 s+ + s+ s+ + −

Cyclic with Noise ρ 10=τ 25=τ 10=τ 25=τ 10=τ 25=τ

eiuMPBIL• MPBIL

0.1 s- + s- − s- − 0.2 s- s- − s- − s- 0.5 s- + − − s- s- 1.0 − + − + s- −

Random ρ 10=τ 25=τ 10=τ 25=τ 10=τ 25=τ

eiuMPBIL• MPBIL

0.1 − ~ + ~ − + 0.2 s- + − + s- s- 0.5 s- s- s- s- s- s- 1.0 + + s+ s+ + +

TABLE II

THE OFF-LINE PERFORMANCE OF THE MPBIL AND EIUMPBIL ON DDUFS IN THREE DYNAMIC ENVIRONMENTS

Environments Cyclic Cyclic with noise Random Functions τ ρ MPBIL eiuMPBIL MPBIL eiuMPBIL MPBIL eiuMPBIL DDUF1 10 0.1 90.925 (0.2551) 91.183 (0.3198) 66.934 (0.0903) 66.729 (0.0797) 74.899 (0.1655) 74.892 (0.1599)

0.2 90.680 (1.2893) 91.902 (1.2680) 65.585 (0.0955) 65.456 (0.1151) 66.829 (0.0787) 66.607 (0.1018) 0.5 98.264 (0.0960) 98.313 (0.0900) 66.900 (0.1215) 66.764 (0.1091) 63.822 (0.0493) 63.724 (0.0322) 1.0 98.986 (0.0351) 99.055 (0.0173) 71.904 (0.3752) 71.698 (0.4606) 99.019 (0.0198) 99.049 (0.0143)

25 0.1 91.204 (0.1572) 91.720 (0.2246) 78.987 (0.1971) 78.992 (0.1985) 86.599 (0.0774) 86.599 (0.0774) 0.2 91.173 (0.5280) 91.539 (0.4269) 71.377 (0.2117) 71.200 (0.1990) 76.432 (0.1355) 76.438 (0.1425) 0.5 96.330 (16.928) 97.784 (5.4794) 77.355 (0.5096) 77.365 (0.4972) 67.865 (0.0749) 67.687 (0.0624) 1.0 99.568 (0.0065) 99.578 (0.0043) 86.068 (0.3201) 86.097 (0.3518) 99.583 (0.0037) 99.585 (0.0036)

DDUF2 10 0.1 78.657 (2.2070) 78.957 (1.7956) 40.847 (0.1732) 40.684 (0.2147) 50.820 (0.6143) 50.872 (0.5305) 0.2 77.704 (7.5016) 79.726 (6.1974) 38.836 (0.1553) 38.739 (0.2316) 40.341 (0.2014) 40.211 (0.1658) 0.5 86.884 (11.370) 90.192 (5.6535) 40.593 (0.3373) 40.472 (0.3308) 37.021 (0.0885) 36.919 (0.0917) 1.0 83.591 (88.623) 90.585 (42.762) 46.960 (0.9335) 46.662 (0.7884) 83.076 (104.92) 90.829 (30.708)

25 0.1 79.566 (1.2447) 80.906 (1.4747) 55.965 (1.0511) 55.816 (0.9259) 71.281 (0.7343) 71.281 (0.7343) 0.2 79.453 (1.5393) 80.212 (1.2725) 46.674 (0.4493) 46.250 (0.4977) 52.909 (0.6636) 52.938 (0.6313) 0.5 88.543 (37.194) 91.833 (13.036) 53.894 (1.9753) 53.871 (2.1028) 42.399 (0.1788) 42.243 (0.2151) 1.0 93.238 (20.207) 95.981 (7.4727) 69.361 (2.8077) 69.528 (2.2356) 93.522 (27.871) 95.804 (11.172)

DDUF3 10 0.1 78.777 (1.0817) 79.729 (1.4344) 52.452 (0.0782) 52.324 (0.1066) 55.191 (0.0942) 55.149 (0.1182) 0.2 84.381 (1.6736) 84.646 (0.6578) 51.313 (0.0791) 51.250 (0.0772) 51.285 (0.0342) 51.168 (0.0574) 0.5 86.546 (0.0803) 86.566 (0.0640) 52.540 (0.1441) 52.408 (0.1299) 50.361 (0.0528) 50.284 (0.0346) 1.0 87.313 (0.0045) 87.314 (0.0042) 60.385 (0.3452) 60.087 (0.4170) 87.310 (0.0039) 87.311 (0.0037)

25 0.1 79.096 (0.8408) 80.068 (0.9519) 59.378 (0.2900) 59.327 (0.3573) 63.571 (0.0767) 63.572 (0.0765) 0.2 84.119 (2.9486) 84.881 (0.3982) 56.754 (0.2158) 56.529 (0.1803) 57.035 (0.0798) 56.821 (0.1618) 0.5 86.725 (0.0931) 86.721 (0.0890) 61.355 (0.9342) 60.888 (0.7752) 55.041 (0.0679) 54.856 (0.0892) 1.0 87.325 (0.0035) 87.324 (0.0039) 71.940 (0.6588) 71.931 (0.6461) 87.336 (0.0037) 87.339 (0.0028)

Secondly, from the results in the cyclic with noise and

random environments, the performance of the eiuMPBIL is a little worse than that of the MPBIL in most cases. This demonstrates that the environmental information based updating strategy may produce negative effects in cyclic

with noise and random environments. This will be discussed in Section V.

From the above analysis, it can be seen that the proposed environmental information based updating strategy is an effective updating strategy for associative memory schemes

36

Page 6: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

in cyclic dynamic environments.

V. DISCUSSIONS For a memory scheme, the updating strategy could be very important for its performance in dynamic environments. However, the recent work pays little attention to the memory updating strategies, especially to associative memory schemes.

In this paper, an effective memory updating strategy, i.e. the environmental information based updating strategy, is proposed for associative memory schemes. With this strategy, the stored samples and their associated environmental information have high matching degrees. The memory memorizes accurate environmental information for the associated samples. If a new environment is similar to one of the stored environment, the similar memory point is easy to be found by its associated environmental information. Furthermore, the accurate environmental information has a high level convergence to its associated sample. When the memory point is retrieved, the retrieved environmental information could track the new optimum quickly. However, if the optimum of a new

environment stays far away from the stored solutions, the relatively low level diversity could be harmful to the search ability. Such cases are often happen in cyclic with noise and random environments. In fact, this could be the reason that the eiuMPBIL performs worse than the MPBIL in some cases of cyclic with noise and random environments.

In order to compare the diversity of the stored probability vector in the MPBIL and the eiuMPBIL, a simple way is adopted to calculate the diversity of the probability vector

),...,( 1 lPPP = (l is the encoding length) [46]. If a bit value

of the probability vector P is closer to 0.5, the corresponding sample bit generated by it will be more stochastic. Therefore, the diversity of the probability vector P can be calculated by Eq. (7) [46]. In this equation, a smaller Div means the higher diversity.

∑ −==

l

iiP

lPDiv

1

2)5.0(1)( (7)

TABLE III THE DIVERSITY OF THE MPBIL AND EIUMPBIL

Environments Cyclic Cyclic with noise Random Functions τ ρ MPBIL eiuMPBIL MPBIL eiuMPBIL MPBIL eiuMPBIL DDUF1 10 0.1 0.2393 0.2398 0.2326 0.2336 0.2366 0.2366

0.2 0.2372 0.2389 0.2301 0.2307 0.2319 0.2329 0.5 0.2400 0.2449 0.2262 0.2270 0.2251 0.2257 1.0 0.2409 0.2466 0.2309 0.2324 0.2402 0.2465

25 0.1 0.2427 0.2434 0.2400 0.2400 0.2421 0.2421 0.2 0.2420 0.2433 0.2361 0.2375 0.2392 0.2393 0.5 0.2414 0.2456 0.2370 0.2376 0.2335 0.2349 1.0 0.2422 0.2468 0.2400 0.2405 0.2420 0.2467

DDUF2 10 0.1 0.2389 0.2392 0.2326 0.2336 0.2360 0.2361 0.2 0.2371 0.2385 0.2301 0.2308 02316 0.2326 0.5 0.2391 0.2440 0.2267 0.2276 0.2265 0.2270 1.0 0.2392 0.2452 0.2301 0.2319 0.2395 0.2454

25 0.1 0.2422 0.2430 0.2394 0.2396 0.2418 0.2418 0.2 0.2416 0.2428 0.2359 0.2377 0.2386 0.2390 0.5 0.2412 0.2447 0.2356 0.2373 0.2339 0.2353 1.0 0.2418 0.2465 0.2397 0.2404 0.2420 0.2464

DDUF3 10 0.1 0.2388 0.2401 0.2320 0.2330 0.2346 0.2355 0.2 0.2393 0.2444 0.2290 0.2306 0.2302 0.2312 0.5 0.2399 0.2468 0.2273 0.2290 0.2261 0.2277 1.0 0.2456 0.2456 0.2322 0.2332 0.2456 0.2456

25 0.1 0.2421 0.2440 0.2394 0.2401 0.2421 0.2421 0.2 0.2418 0.2462 0.2357 0.2381 0.2371 0.2391 0.5 0.2417 0.2472 0.2354 0.2379 0.2340 0.2360 1.0 0.2460 0.2460 0.2419 0.2419 0.2460 0.2460

The diversity of the probability vector is recorded when

the environmental change is detected. The average diversity values across the whole runs of the MPBIL and eiuMPBIL is calculated by Eq. (8). In this equation, R is the total run times. CT denotes the change times of the environment in a run. CP is the working probability vector after the environment changes.

∑ ∑== =

R

r

CT

ctCPDiv

CTRD

1 1))(1(1 (8)

The results of the average diversity are shown in Table III. From this table, it can be seen that the diversity of the stored probability vector in the eiuMPBIL is lower than that in the MPBIL. In order to solve this problem, some

37

Page 7: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

improvement ways should be studied in the future. Besides the MPBIL, the environmental information

based updating strategy could also be beneficial for other associative memory schemes. For example, it could be extended and embedded into the AMGA [42] to improve the environmental adaptability for DOPs, especially in cyclic environments. This is another interesting work in the future.

Additionally, the proposed environmental information based updating strategy should be tested on the dynamic real-valued multimodal problems in the future.

Finally, the memory updating strategy should be paid enough attention in the future. To devise novel effective memory updating strategies could be very interesting and significant.

VI. CONCLUSION In recent years, some effective memory schemes have been developed for EAs to solve DOPs. In these schemes, how to update the memory could be very important for their environmental adaptability in dynamic environments. However, little work has been done for memory updating strategies, especially for associative memory updating strategies.

In associative memory schemes, the good solution and its associated environmental information are both stored in the memory. When the memory is updated, the environmental information should also be considered.

In this paper, a novel memory updating strategy, i.e. the environmental information based updating strategy, is proposed for associative memory schemes. In this strategy, the matching degrees between environmental information and individuals are used to find the most similar memory point and do replacement. Based on three dynamic test functions, the environmental information based updating strategy is tested by embedding it into the MPBIL. The experimental results demonstrate that the proposed environmental information based updating strategy is an effective updating strategy for associative memory schemes, especially in cyclic dynamic environments.

REFERENCES [1] K. A. D. Jong, Evolutionary Computation: A Unified Approach.

London, England: The MIT Press, 2006. [2] J. J. Grefenstette, "Genetic Algorithms for Changing

Environments," in Proc. Parallel Problem Solving From Nature. Amsterdam, Netherlands, 1992, pp. 137-144.

[3] N. Mori and H. Kita, "Genetic Algorithms for Adaptation to Dynamic Environments - A Survey," in Proc. the 2000 IEEE International Conference on Industrial Electronics, Control and Instrumentation. Nagoya, Japan, 2000, pp. 2947-2952.

[4] Y. Jin and J. Branke, "Evolutionary Optimization in Uncertain Environments - A Survey," IEEE Transactions on Evolutionary Computation, vol. 9, no. 3, pp. 303-317, 2005.

[5] J. Branke, Evolutionary Optimization in Dynamic Environments. Dordrecht, The Netherland: Kluwer Academic Publishers, 2002.

[6] K. Trojanowski and Z. Michalewicz, "Evolutionary Algorithms for Non-Stationary Environments," in Proc. the 8th Workshop: Intelligent Information Systems. Poland, 1999, pp. 229-240.

[7] J. Branke and H. Schmeck, "Designing Evolutionary Algorithms for Dynamic Optimization Problems," in Advances in evolutionary computing: theory and applications: Springer-Verlag New York, 2003, pp. 239-262.

[8] H. G. Cobb, "An Investigation into the Use of Hypermutation as an Adaptive Operator in Genetic Algorithms Having Continuous, Time-Dependent Nonstationary Environments," NRL-MR-6760, United States, 1990.

[9] R. W. Morrison and K. A. De Jong, "Triggered Hypermutation Revisited," in Proc. the 2000 IEEE Congress on Evolutionary Computation (CEC'2000). California, CA, USA, 2000, pp. 1025-1032.

[10] H. G. Cobb and J. J. Grefenstette, "Genetic Algorithms for Tracking Changing Environments," Australian Electronics Engineering, vol. 27, no. 2, pp. 523-530, 1994.

[11] F. Oppacher and M. Wineberg, "Reconstructing the Shifting Balance Theory in a GA: Taking Sewall Wright Seriously," in Proc. the 2000 IEEE Congress on Evolutionary Computation (CEC'2000), 2000, pp. 219-226.

[12] R. K. Ursem, "Multinational GAs: Multimodal Optimization Techniques in Dynamic Environments," in Proc. the 2000 Genetic and Evolutionary Computation Conference (GECCO 2000). Riviera Hotel, Las Vegas, USA, 2000, pp. 19-26.

[13] J. Branke, T. Kaussler, C. Smidt, and H. Schmeck, "A Multi-population Approach to Dynamic Optimization Problems," in Proc. the 4th International Conference on Adaptive Computing in Design and Manufacture (ACDM'00), P. IC, Ed. Plymouth, England, 2000, pp. 299-307.

[14] R. K. Ursem, "Multinational Evolutionary Algorithms," in Proc. the 1999 IEEE Congress on Evolutionary Computation (CEC'99). Washington, DC, USA, 1999, pp. 1633-1640.

[15] R. Mendes and A. S. Mohais, "DynDE: a Differential Evolution for Dynamic Optimization Problems," in Proc. the 2005 IEEE Congress on Evolutionary Computation (CEC'2005). Piscataway, United States, 2005, pp. 2808-2815.

[16] M. C. du Plessis and A. P. Engelbrecht, "Improved Differential Evolution for Dynamic Optimization Problems," in Proc. the 2008 IEEE Congress on Evolutionary Computation (CEC'2008), 2008, pp. 229-234.

[17] J. Branke, "Memory Enhanced Evolutionary Algorithms for Changing Optimization Problems," in Proc. the 1999 IEEE Congress on Evolutionary Computation (CEC'99). Washington, DC, USA, 1999, pp. 1875-1882.

[18] S. Yang, "Explicit Memory Schemes for Evolutionary Algorithms in Dynamic Environments," in Evolutionary Computation in Dynamic and Uncertain Environments. vol. 51: Springer Berlin / Heidelberg, 2007, pp. 3-28.

[19] S. Yang, "Genetic Algorithms with Elitism-Based Immigrants for Chaning Optimization Problems," in Proc. EvoWorkshops 2007: Applications of Evolutionary Computing, S. B. Heidelberg, Ed, 2007, pp. 627-636.

[20] S. Yang, "Memory-Based Immigrants for Genetic Algorithms in Dynamic Environments," in Proc. the 2005 Genetic and Evolutionary Computation Conference (GECCO 2005). Washington, DC, US, 2005, pp. 1115-1122.

[21] S. Yang, "Genetic Algorithms with Memory- and Elitism-based Immigrants in Dynamic Environments," Evolutionary Computation, vol. 16, no. 3, pp. 385-416, 2008.

[22] X. Yu, K. Tang, and X. Yao, "An Immigrants Scheme Based on Environmental Information for Genetic Algorithms in Changing Environments," in Proc. the 2008 IEEE Congress on Evolutionary Computation (CEC'2008). HongKong, China, 2008, pp. 1141-1147.

[23] S. Yang and R. Tinós, "A Hybrid Immigrants Scheme for Genetic Algorithms in Dynamic Environments," International Journal of Automation and Computing, vol. 4, no. 3, pp. 243-254, July 2007.

[24] R. Tinós and S. Yang, "A Self-organizing Random Immigrants Genetic Algorithm for Dynamic Optimization Problems," Genetic Programming and Evolvable Machines, vol. 8, no. 3, pp. 255-286, 2007.

[25] L. T. Bui, J. Branke, and H. A. Abbass, "Multiobjective Optimization for Dynamic Environments," in Proc. the 2005 IEEE Congress on Evolutionary Computation (CEC'2005), 2005, pp. 2349-2356.

[26] L. T. Bui, J. Branke, and H. A. Abbass, "Diversity As a Selection Pressure in Dynamic Environments," in Proc. the 2005 Genetic and Evolutionary Computation Conference (GECCO 2005). Washington, DC, US, 2005, pp. 1557-1558.

38

Page 8: [IEEE 2010 Third International Workshop on Advanced Computational Intelligence (IWACI) - Suzhou, China (2010.08.25-2010.08.27)] Third International Workshop on Advanced Computational

[27] H. Cheng and S. Yang, "Genetic Algorithms with Elitism-Based Immigrants for Dynamic Shortest Path Problem in Mobile Ad Hoc Networks," in Proc. the 2009 IEEE Congress on Evolutionary Computation (CEC'2009), 2009, pp. 3135-3140.

[28] S.-K. Lee and B.-R. Moon, "Genetic Algorithm with Adaptive Elitism-Based Immigrants for Dynamic Optimization Problems," in Proc. the 11th Annual conference on Genetic and evolutionary computation. Montreal, Québec, Canada, 2009, pp. 1865-1866.

[29] S. Yang, H. Cheng, and F. Wang, "Genetic Algorithms with Immigrants and Memory Schemes for Dynamic Shortest Path Routing Problems in Mobile Ad Hoc Networks," IEEE Transactions on Systems, Man, and Cybernetics Part C: Applications and Reviews, 2009.

[30] D. E. Goldberg and R. E. Smith, "Nonstationary Function Optimization using Genetic Algorithms with Dominance and Diploidy," in Proc. the 2th International Conference on Genetic Algorithms. USA, 1987, pp. 59-68.

[31] K. P. Ng and K. C. Wong, "A New Diploid Scheme and Dominance Change Mechanism for Non-Stationary Function Optimization," in Proc. the 6th International Conference on Genetic Algorithms, 1995, pp. 159-166.

[32] J. Lewis, E. Hart, and G. Ritchie, "A Comparison of Dominance Mechanisms and Simple Mutation on Non-stationary Problems," in Proc. the 5th International Conference on Parallel Problem Solving from Nature, 1998, pp. 139-148.

[33] S. Yang, "Non-Stationary Problem Optimization Using the Primal-Dual Genetic Algorithm," in Proc. the 2003 IEEE Congress On Evolutionary Computation (CEC'2003), 2003, pp. 2246-2253.

[34] S. Yang and X. Yao, "Experimental Study on Population-Based Incremental Learning Algorithms for Dynamic Optimization Problems," Soft Computing, vol. 9, no. 11, pp. 815-834, 2005.

[35] H. Wang and D. Wang, "An Improved Primal-Dual Genetic Algorithm for Optimization in Dynamic Environments," in Proc. the 13th International Conference on Neural Information Processing (ICONIP 2006). Hong Kong, 2006, pp. 836-844.

[36] H. Wang, S. Yang, W. H. Ip, and D. Wang, "Adaptive Primal-Dual Genetic Algorithms in Dynamic Environments," IEEE Transactions on Systems, Man, and Cybernetics Part B: Cybernetics, 2009.

[37] C. L. Ramsey and J. J. Grefenstette, "Case-Based Initialization of Genetic Algorithms," Australian Electronics Engineering, vol. 27, no. 2, pp. 84-91, 1994.

[38] K. Trojanowski and Z. Michalewicz, "Searching for Optima in Non-stationary Environments," in Proc. the 1999 IEEE Congress on Evolutionary Computation (CEC'99). Washington, DC, USA, 1999, pp. 1843-1850.

[39] N. Mori, H. Kita, and Y. Nishikawa, "Adaptation to Changing Environments by Means of the Memory Based Thermodynamical Genetic Algorithm," Transactions of the Institute of Systems, Control and Information Engineers, vol. 14, no. 1, pp. 33-41, 2001.

[40] S. Yang, "Population-Based Incremental Learning with Memory Scheme for Changing Environments," in Proc. the 2005 Genetic and Evolutionary Computation Conference (GECCO 2005). Washington, DC, US, 2005, pp. 711-718.

[41] S. Yang, "A Comparative Study of Immune System Based Genetic Algorithms in Dynamic Environments," in Proc. the 2006 Genetic and Evolutionary Computation Conference, 2006, pp. 1377-1384.

[42] S. Yang, "Associative Memory Scheme for Genetic Algorithms in Dynamic Environments," in Proc. EvoWorkshops 2006: Applications of Evolutionary Computing. Heidelberg, Germany, 2006, pp. 788-799.

[43] S. Yang and X. Yao, "Population-Based Incremental Learning With Associative Memory for Dynamic Environments," IEEE Transactions on Evolutionary Computation, vol. 12, no. 5, pp. 542-561, 2008.

[44] S. J. Louis and Z. J. Xu, "Genetic Algorithms for Open Shop Scheduling and Re-scheduling," in Proc. the 11th International Conference on Computers and Their Applications (ISCA) 1996, pp. 99-102.

[45] C. N. Bendtsen and T. Krink, "Dynamic Memory Model for Non-Stationary Optimization," in Proc. the 2002 IEEE Congress on Evolutionary Computation (CEC'2002), 2002, pp. 145-150.

[46] Y. Cao and W. Luo, "Novel Associative Memory Retrieving Strategies for Evolutionary Algorithms in Dynamic Environments," in Proc. the 4th International Symposium on Intelligence

Computation and Applications (ISICA 2009). Huangshi, China, 2009, pp. 258-268.

[47] S. Baluja, "Population-Based Incremental Learning: A Method for Integrating Genetic Search Based Function Optimization and Competitive Learning," CMU-CS-94-163, United States, 1994.

39