online learning in estimation of distribution algorithms for dynamic environments

Departamento de Engenharia de Computação e Automação

Industrial

Faculdade de Engenharia Elétrica e de Computação

Unicamp

Online learning in estimation of distribution algorithms for dynamic environments

André Ricardo GonçalvesFernando J. Von Zuben

2

Optimization in dynamic environments

Estimation of distribution algorithms

Mixture model and online learning

Proposed method: EDAOGMM

Experimental results

Concluding remarks and future works

References

Outline

3







References

Outline

4

Optimization in dynamic environments World is dynamic!

New events arrive and others canceled at a scheduling problem;

Vehicles must reroute around heavy trac and road repairs;

Machine breakdown occurs during a production run.

5

Optimization in dynamic environments Dynamic optimization algorithm should be able to react to the new

environment, updating the internal model and generating new candidate solutions;

Evolutionary algorithms (EAs) appear as promising approaches, since they maintain a population of solutions that can be adapted by means of a balance between exploration and exploitation of the search space;

EAs approaches: GA, PSO, AIS, EDAs, among others;

However, to be applied in dynamic environments, they must be adapted.

6







References

Outline

7

Estimation of distribution algorithms Estimation of distribution algorithms (EDA) are

evolutionary methods that use estimation of distribution techniques, instead of genetic operators.

The key aspect in EDAs is how to estimate the true distribution of promising solutions. Dependence trees, Bayesian networks, mixture models, etc.

Classication of EDAs based on complexity of probabilistic model.

8


9







References

Outline

10

Mixture model and online learning Mixture models are flexible estimators;

In optimization, they are able to capture the multimodality of the search space;

Learning methods, such as Expectation-Maximization (EM), are computationally efficient;

In optimization of dynamic environments, the model tends to change constantly;

EM with online learning appear as a promising approach to model dynamic environments.

11

Mixture model and online learning Online learning

Fast adaptation model to the new data coming from the environment;

Approach proposed by (Nowlan,1991) stores the relevant information in a vector of sufficient statistics;

Exponential decay (γ) of the data importance to the model.

12







References

Outline

13


EDA with online Gaussian mixture model (EDAOGMM)

Employs an incremental and constructive mixture model (low computational cost);

Self-adjusts the components number by means of BIC;

Model tends to adapt to the multimodality of search space;

Employs a “random immigrants” approach to promote population diversity;

14


15

Proposed method: EDAOGMM Selection method:

Stochastic selection aids to preserve the population diversity; η parameter defines the balance between exploration and explotation.

Diversity control: Stochastic selection; Random immigrants; Controlled reinitializations (δ parameter).

Components number control: Incremental and constructive approach; Removal of overlapped components (ε parameter).

16

Proposed method: EDAOGMM New population is composed by 3 subpopulations (dependent

of the η parameter): Sampled by the mixture model; Best individuals; Random immigrants.

Overlapped components is a redundant representation of a promising region Remove the component with lower mixture coefficient; Check the overlap using the ε parameter.

17







References

Outline

18

Experimental results Moving Peaks benchmark (MPB) generator plus a rotation method (Li &

Yang, 2008);

Fitness surface are composed by a set of peaks that changes your positions, heights and widths over time;

Maximization problem in a continuous space;

Seven types of change (T1-T7): small step, large step, random, chaotic, recurrent, recurrent with noise and random with dimensional changes;

There are parameters to control the multimodality of the search space, severity of changes and the dynamism of the environment;

Range of search space: [-5,5];

Problem dimensions: 10 and [5-15].

19

Experimental results Six dynamic environments settings were considered:

20

Experimental results Contenders proposed algorithms in the literature:

Improved Univariate Marginal Distribution Algorithm - IUMDA (Liu et al., 2008);

Tri-EDAG (Yuan et al., 2008); Hypermutation Genetic Algorithm - HGA (Cobb,1990).

Two EDAs and a GA developed for dynamic environments.

21

Experimental results Free parameters settings:

22

Experimental results Comparison metrics:

Offline error Average of the absolute error between the best solution found

so far and the global optimum (known) at each time step t.

23

Experimental results Scenarios 1 and 2

24


25


26







References

Outline

27

Concluding remarks and future works EDAOGMM outperforms all the contenders, particularly in high-

frequency changing environments (Scenarios 1 and 2);

EDAOGMM has a fast convergence because it can explore several peaks simultaneously;

We can detect a less prominent performance in low frequency scenarios (5 and 6), indicating that, once converged, the EDAOGMM reduces its exploration power;

So, a continued control to avoid premature convergence is desirable.

28

Concluding remarks and future works Future works:

Incorporate a continued convergence control mechanism;

Compare EDAOGMM with other algorithms designed to deal with dynamic environments;

Increment the experimental tests aiming at investigating scalability and other aspects related to the relative performance of the proposed algorithm;

Performs a parameter sensitivity analisys.

29







References

Outline

30

References S. Nowlan, “Soft competitive adaptation: neural network learning

algorithms based on fitting statistical mixtures,” Ph.D. dissertation, Carnegie Mellon University, Pittsburgh, PA, USA, 1991.

C. Li and S. Yang, “A generalized approach to construct benchmark problems for dynamic optimization,” in Proc. of the 7th Int. Conf. on Simulated Evolution and Learning, 2008.

X. Liu, Y. Wu, and J. Ye, “An Improved Estimation of Distribution Algorithmin Dynamic Environments,” in Fourth International Conference on Natural Computation. IEEE Computer Society, 2008, pp. 269–272.

B. Yuan, M. Orlowska, and S. Sadiq, “Extending a class of continuous estimation of distribution algorithms to dynamic problems,” Optimization Letters, vol. 2, no. 3, pp. 433–443, 2008.

H. Cobb, “An investigation into the use of hypermutation as an adaptive operator in genetic algorithms having continuous, time-dependent nonstationary environments,” Naval Research Laboratory, Tech. Rep., 1990.

Departamento de Engenharia de Computação e Automação

Industrial

Faculdade de Engenharia Elétrica e de Computação

Unicamp

Online learning in estimation of distribution algorithms for dynamic environments

André Ricardo GonçalvesFernando J. Von Zuben

online learning in estimation of distribution algorithms for dynamic environments

Entertainment & Humor