Rotated Test Problems for Assessing the Performance ofMulti-objective Optimization Algorithms

Antony W. IorioSchool of Computer Science and I.T., RMIT

UniversityMelbourne VIC 3000 Australia

iantony@cs.rmit.edu.au

Xiaodong LiSchool of Computer Science and I.T., RMIT

UniversityMelbourne VIC 3000 Australia

xiaodong@cs.rmit.edu.au

ABSTRACTThis paper presents four rotatable multi-objective test prob-lems that are designed for testing EMO (Evolutionary Multi-objective Optimization) algorithms on their ability in deal-ing with parameter interactions. Such problems can besolved efficiently only through simultaneous improvementsto each decision variable. Evaluation of EMO algorithmswith respect to this class of problem has relevance to real-world problems, which are seldom separable. However, manyEMO test problems do not have this characteristic. Theproposed set of test problems in this paper is intended toaddress this important requirement. The design principlesof these test problems and a description of each new testproblem are presented. Experimental results on these prob-lems using a Differential Evolution Multi-objective Opti-mization algorithm are presented and contrasted with theNon-dominated Sorting Genetic Algorithm II (NSGA-II).

Categories and Subject DescriptorsG.1.6 [Mathematics of Computing]: Numerical Analy-sis—Optimization

General TermsExperimentation, Performance, Measurement

KeywordsParameter interactions, Multi-objective Optimization

1. INTRODUCTIONTraditionally, multi-objective problems have been solved

using classical techniques, but such approaches typically havemany limitations with respect to the types of problems theycan solve, and may not even be able to find optimal solu-tions for problems which have no known functional repre-sentation. A relatively recent approach to solving multi-

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.GECCO’06, July 8–12, 2006, Seattle, Washington, USA.Copyright 2006 ACM 1-59593-186-4/06/0007 ...$5.00.

objective problems has been the application of populationbased Evolutionary Algorithms (EAs). There is a naturalsynergy between a population based evolutionary approach,which searches a number of points in the decision space si-multaneously, and a multi-objective problem, which has aPareto-optimal set of solutions. Evolutionary approachescan provide a set of solutions which closely approximatesthe Pareto-optimal set. Despite their apparent successes,there is a disparity between how we evaluate EMO algo-rithms with test problems with specified properties, and thecharacteristics of many real-world problems.

1.1 Test problems for EMO AlgorithmsTypically, in order to assess the performance of EMO al-

gorithms in a variety of fitness landscapes, test problemswith a variety of characteristics are employed. For instance,one may wish to evaluate the performance of an algorithmwith respect to the concavity of the Pareto-optimal front, orone may employ test problems with discontinuous Pareto-optimal fronts, sparsely distributed solutions near the Pareto-optimal front, or non-uniform mappings between the de-cision and objective space. The ZDT series of test prob-lems [13], built on the framework proposed in [1], exhibitsome of these characteristics. In [1] the problem featuresthat may cause difficulty for an EMO algorithm are dis-cussed, and a framework is proposed for construction of testproblems exhibiting such features. Recently, there has beenfurther progress in the construction of test problems formulti-objective optimization. Although many approacheshave focussed on the construction of only two-objective prob-lems, other approaches deal with the construction of testproblems with more than two objectives in [6] and [3]. Aframework is also proposed in [9] for constructing arbitraryuser specified Pareto-optimal sets in the decision space, whichmap to Pareto-optimal fronts in the objective space. In thearea of multi-objective combinatorial optimization, instancegenerators have been proposed for the quadratic assignmentproblem [8]. These generators are useful for studying para-meter interactions in combinatorial problems.

1.2 Problems with parameter interactionsBefore discussing the previous work in the area of test

problems with parameter interactions, it is necessary to de-fine what we mean by a linearly separable problem. Parame-ter interactions occur in a problem because the parametersin the problem are not linearly separable. The geometricinterpretation of what constitutes a linearly separable prob-

683

lem is where the n parameters of a problem in n-dimensionalspace can be separated by an n−1 dimensional hyperplane.We will see in Section 2 how linear separability occurs witha simple ellipsoid function, and how rotation can make thefunction non-separable.

Previous work in [12] demonstrated the importance ofevaluating single objective Evolutionary Algorithms on rota-ted problems in order to test their rotationally invariant be-haviour, the contention being that evaluations of EAs shouldbe independent of any coordinate system. Furthermore, inthe EMO domain, NSGA-II was demonstrated to performpoorly on a simple rotated problem [2] and the importanceof parameter interactions in multi-objective problems wasdiscussed in [7]. If EMO algorithms can fail on such a sim-ple problem, but succeed when the problem is aligned withthe principle coordinate system of the decision space, thenobviously many reported results on such test problems arepotentially misleading. In [12] a strong case is presented forall EA evaluations to be independent of a particular coordi-nate system, and our contention is that this should equallybe true for the evaluation of EMO algorithms.

Recently, the issue of parameter interactions in multi-objective problems has garnered more interest; in [3] thenon-separability of multi-objective problems is demonstratedto be an important characteristic lacking in existing multi-objective test suites, and a method is described for con-structing problems with parameter interactions for an arbi-trary number of objectives. In [11] a differential evolutionmulti-objective algorithm was evaluated on a number of theproposed non-separable and separable problems proposedin [3].

1.3 OverviewEach of the real-coded ZDT test problems has a Pareto-

optimal set in the decision space which is aligned with theprinciple coordinate system. This makes the problem easier,because the objectives can be solved in stages. The purposeof this paper is to propose a complementary series of testproblems to the ZDT series of problems, using the approachdescribed in [1], and to provide some further insight intorotated multi-objective problems. The primary motivationfor this is that most real world problems have parameterinteractions, and ideally we would also like a variety of testproblems which have parameter interactions as well. Theframework employed for constructing the ZDT problems iscommonly used by practitioners, and it is easily employedin the construction of test problems.

The proposed problems in this paper can be arbitrarilyrotated on any axis of the decision space. Through a ro-tation of the coordinate system, parameter interactions canbe introduced to the problem. When the proposed problemsare rotated in the decision space, the Pareto-optimal set isrotated accordingly. In Section 2.1 we will see that when anysolution in the set is perturbed independently with respectto any of the decision variables, it can only be perturbed tonon-dominated solutions which are not Pareto-optimal.

The proposed series of rotated problems presented in thispaper will provide a means of assessing the performance ofEMO algorithms which is not biased towards a particularorientation of the problem with respect to the coordinateaxes. This provides EMO practitioners with a reliable indi-cation of the performance of their algorithm irrespective ofthe orientation of the problem in the decision space.

In the following section we will describe in more detail howrotation makes single objective and multi-objective prob-lems harder to optimize. In Section 3 we describe the pro-posed problem suite, and how the problems were constructed.For a comparison with the NSGA-II algorithm, a new Dif-ferential Evolution approach to EMO is briefly introducedin Section 4 along with the experiments conducted. This isfollowed by a discussion of the results and the utility of theproposed problem suite in Section 5 and concluding remarksin Section 6.

2. ROTATED PROBLEMSParameter interactions can be introduced in other types

of problems, such as combinatorial problems. For the pur-poses of this study, we are only considering multi-objectiveproblems with real-valued decision variables, where one ormore objectives have non-linear parameter interactions.

Before elucidating upon rotated multi-objective problems,we will first consider the effect of rotation on a simple singleobjective ellipsoid minimization problem defined in Equa-tion (1). Figure 1(a) also presents a contour plot of thefunction.

f(x1, x2) = x21 + a0x

22 (1)

f(x1, x2) = x21 + a1x1x2 + a0x

22 (2)

The ellipsoid problem in Equation (1) has a global mini-mum located at x1 = 0 and x2 = 0, at the origin O, of theprinciple coordinate axes. It is apparent from the contourplot in Figure 1(a), that this function is aligned with theprinciple coordinate axes, and is linearly separable with re-spect to the two decision variables. The two components,x2

1 and a0x22, of Equation (1), can be solved as independent

minimization problems. A search algorithm only needs toperturb the variables x1 and x2 independently in order tofind the global optimum for this problem. If the ellipsoidfunction from Equation (1) is rotated away from the prin-cipal coordinate axes (Figure 1(b)), the decision variablesbecome non-separable through the introduction of parame-ter interactions in Equation (2). Parameter interactions areintroduced through the term a1x1x2. Progress towards theglobal optimum can only proceed efficiently by making si-multaneous improvements with respect to all parameter val-ues.

In Figure 1, the contour represents a region of constantfitness. The point A in Figure 1(a) can be perturbed alongthe x1 and x2 axes, and any location along the dashed lineswill be an improvement over any point along the contour. InFigure 1(b) it is apparent that progress from perturbing therotated point A′ will be lower. This is because the intervalof potential improvement for each of the decision variables isreduced, and as a result, the progress of the search throughindependent perturbations will be reduced.

Furthermore, the search can easily be trapped on the linesegment that bisects the ellipsoid lengthwise, as can be seenin Figure 2. Any point on this line segment can only move toanother point which evaluates to a better solution, by mak-ing simultaneous improvements on each decision variable.For example, in Figure 2(a), the point A can be indepen-dently perturbed on the x1 axis to find the global minimumlocated at the origin of the coordinate system. The samepoint A′, in Figure 2(b), after rotation cannot progress to apoint of improved fitness by only moving along the direction

684

x2

x1

x2

x1

A′

A

(a) (b)

OO

Figure 1: The contour plots demonstrate how rota-tion can reduce the interval of possible improvementrepresented by the dashed line segments. Point Ain (a) and A′ in (b) represent the same point on thecontour plot before and after rotation respectively.The contour represents a region of constant fitness.

x2

x1

x2

x1

A′

A OO

(a) (b)

Figure 2: The contour plots demonstrate how rota-tion can trap points along the line segment bisectingthe ellipsoid. Point A in (a) and A′ in (b) representthe same point on the contour plot before and afterrotation respectively.

of the principle coordinate axes because any such perturba-tion will be to a point of lower fitness in the objective space.Typically the basin of attraction can be found comparativelyeasily, but the search can become trapped when the problemis non-separable. Only a simultanenous improvement in allparameters will result in the discovery of fitter solutions inthis situation. On these types of problems, the small muta-tion rates frequently used in Genetic Algorithms are knownto be even less efficient than a random search [12]. Evolu-tionary Strategies have been relatively successful at solvingthese types of problems, but they require the learning of ap-propriate correlated mutation step sizes and it can be rathercomputationally expensive when the decision space dimen-sion becomes large [10].

2.1 Rotated Multi-objective ProblemsAlthough one might intuitively expect that in the multi-

objective domain the situation encountered is the same, thesituation is not quite as simple. In order to highlight the dif-ference between rotated single-objective and multi-objectiveproblems, we will consider a simple multi-objective problem.This will also facilitate our understanding of the effect ofrotation on multi-objective problems where non-dominatedsolution sets are sought by a search algorithm. The situationis analogous to the single objective domain, where a searchalgorithm with independent perturbations on each decisionvariable will have trouble finding more optimal solutions.

(a)

-0.4-0.2 0 0.2 0.4x1-0.4-0.2 0 0.2 0.4

x2

-0.3-0.2-0.1

0 0.1 0.2 0.3

f1(x1,x2)

(b)

-0.4-0.2 0 0.2 0.4x1-0.4-0.2 0 0.2 0.4

x2

-0.3-0.2-0.1

0 0.1 0.2 0.3

f1(y1,y2)

(c)

-0.4-0.2 0 0.2 0.4x1-0.4-0.2 0 0.2 0.4

x2

0 5 10 15 20 25 30

f2(x1,x2)

(d)

-0.4-0.2 0 0.2 0.4x1-0.4-0.2 0 0.2 0.4

x2

0 5 10 15 20 25 30

f2(y1,y2)

Figure 3: The effect of a 45-degree rotation on thex1x2 plane of problem R1. Before rotation, the func-tions are aligned with the coordinate system ((a)and (c)), and after rotation they are not ((b) and(d)).

Figure 3 shows a simple bi-objective optimization problemwith a 2-dimensional decision space. This is problem R1,defined in Section 3.2.

The problem is characterised by a slightly inclined troughin objective f2. Objective f1 is a plane with a gradient slop-ing in an opposing direction to the incline of objective f2.The Pareto-optimal set is represented by a line segment bi-secting the decision space in objective f2 and f1 respectively.The decision space is subject to a rotation matrix R.

Consider the contour plot of a non-rotated version of thisproblem in Figure 4, where a point, O, is a member of thePareto-optimal set. If the point O is perturbed in the direc-tion of �OA, it is towards points which evaluate lower withrespect to objective f1 (Figure 4(a)) and higher with re-spect to objective f2 (Figure 4(b)). If it is perturbed in

the direction of �OD, it will be towards points which evalu-ate higher with respect to objective f1 and lower with re-spect to objective f2. Such perturbations are with respectto the parameter x1 only, and this is the only such pertur-bation required in order to discover other Pareto-optimalsolutions which are located on the line segment bisectingthe contour plots for objectives f1 and f2. It is apparentthat such a Pareto-optimal solution can easily be perturbedtowards other Pareto-optimal solutions when the problemis aligned with the principle coordinate axes. However, af-ter the problem has been rotated, it becomes more difficultto find Pareto-optimal solutions through independent per-turbations of individuals. Consider Figure 5, where pointO′ represents the point O after rotation. Through indepen-dent perturbations of decision space parameters, the pointO can perturb to other non-dominated solutions in the di-rection of �O′A′, �O′B′, �O′C′, and �O′D′. A perturbation inthe direction of �O′A′ leads to points which evaluate loweron objective f1, but higher on objective f2. This is similarlytrue for perturbations in the direction of �O′B′. A perturba-tion in the direction of �O′C′ leads to points which evaluatehigher on objective f1, but lower on objective f2. This is

685

B

C

(a)

x2

O

B

C

(b)

x1

x2

D

Pareto-optimal set Pareto-optimal set

minimizef2(y1, y2)

minimizef2(y1, y2)

A

minimizef1(y1, y2)

x1DOA

Figure 4: The contour plot of non-rotated R1 in(a) represents function f1, and the contour plot in(b) represents objective function f2. The dashedlines represent regions of constant value with respectto the objective function evaluation. Smaller dashsizes represent lower evaluations on the objectivefunctions. The point O, in the Pareto-optimal set,can be perturbed to other Pareto-optimal points byperturbing the decision variable x1.

also true for perturbations in the direction of �O′D′. Unfor-tunately each of these perturbations leads to non-dominatedsolutions which skew away from the Pareto-optimal set. Thesituation becomes even worse if the perturbation extends to

�C′E′ or �C′F ′, because individuals in this region evaluatehigher with respect to objective f1 and objective f2, andare dominated by the point at O′, as a result. In actuality,the problem in Figure 3 that this analysis is based on, has anextremely small region relative to the feasible space, wherenon-dominated solutions can be located in the direction of

�O′C′ and �O′D′. Non-dominated solutions in these direc-tions only becomes increasingly likely as the orientation ofthe problem approaches alignment with the principle coor-dinate axes. As the orientation of the Pareto-optimal frontaligns with the axis x1, the line vector �O′C′ extends furtherand more non-dominated solutions can be discovered in thisregion more easily. Secondly, such non-dominated solutionswill be close to the Pareto-optimal set. This is similarly truefor the line vector �O′D′, as the Pareto-optimal front alignswith the axis x2. In other words, a rotation of the problemwhich results in the Pareto-optimal set not being alignedwith any principle coordinate axis, makes it difficult to dis-cover other Pareto-optimal solutions when only independentperturbations of decision variables can occur.

In the presence of only independent perturbations, thereis a tendency for points to be discovered in the direction oflower f1 evaluations, and higher f2 evaluations, pushing thenon-dominated solution set away from the Pareto-optimalset and degrading the search over time. As a result of thisbehaviour, the search can become trapped in the Pareto-optimal region and fail to find more non-dominated solu-tions in the Pareto-optimal set. Progress in covering thePareto-optimal front becomes extremely slow. This effectwas also apparent in [4] and [2], where the NSGA-II pro-duced poor coverage of the Pareto-optimal front on the rota-ted uni-modal multi-objective problem. Any multi-objectiveoptimization algorithm which is not rotationally invariant,will exhibit such behaviour.

E ′

F ′

B′

A′

C ′

D′O′

(b)

Pareto-optimal set

minimizef2(y1, y2)

minimizef2(y1, y2)

x1

x2

A′

B′

C ′

D′ F ′

E ′

O′

(a)

Pareto-optimal set

minimizef1(y1, y2)

x1

x2

Figure 5: The contour plot of rotated R1 in (a) rep-resents function f1, and the contour plot in (b) rep-resents objective function f2. After rotation, thepoint O′, in the Pareto-optimal set, can be per-turbed to other Pareto-optimal points by perturbingthe decision variables x1 and x2 simultaneously alongthe line segment which bisects both feasible regions.

3. CONSTRUCTION OF ROTATED TESTPROBLEMS

Each of the test problems proposed was designed using theframe work proposed in [1]. The g function is responsiblefor affecting convergence to the Pareto-optimal front. Theh function specifies the shape of the Pareto-optimal front,whether the problem is rotated or not. The Pareto-optimalset can be determined by setting the g function to 1.0, andevaluating f1 over the range of feasible solutions. Diversityis also affected by the f1 function.

In order to construct a problem with parameter interac-tions, the problem must have at least one non-linear func-tion. With this consideration, the rotatable test problemswe have proposed in this paper will have at least one non-linear function in at least one of the objective functions.Each of the test problems also sets f1 and f2 to large val-ues if f1 is outside the ranges specified in the problem de-scriptions. This is important because rotation may push afunction evaluation outside the range desired by the exper-imenter [1].

In order to construct a rotated problem, one must also becareful that under rotation the problem can still evaluate toa meaningful result. One should avoid situations where arotation transformation results in decision variables whichtake negative values, and are then subjected to a squareroot function for instance. This can be achieved by avoid-ing functions which would result in such a situation, or byoffsetting the variable value within the function so a neg-ative value never results. It should also be noted that themethodology for performing a rotation can be applied toother problems from the literature, and is not limited tothe ZDT problems. The only considerations that need tobe addressed are related to making sure the evaluation ofthe rotated vector yields a result that can still be evaluated.The Pareto-optimal set can still be determined by the meansspecified within the test problem construction methodologyemployed by the user.

686

3.1 Generating uniform random rotationsIn order to achieve a completely unbiased assessment of

an algorithm on a problem which is rotated, one must guar-antee a uniformly distributed random rotation. Algorithm1 outlines the procedure for generating a random orthnor-mal basis which is used to introduce parameter interdepen-dencies into a problem by rotating the parameter vector.This is a transformation which does not change the fitnesslandscape of the problem domain because it is an isometrytransformation; it preserves distances between points, andas a result it also preserves angles. A uniform distribution

Algorithm 1 Algorithm for generating a uniformly randomrotation matrix. R is an m × m rotation matrix, where Ri

is the ith row of R, and R(j) is the j th column of R. N (0, 1)is a normal distribution with a mean of 0 and variance of 1.

for i = 0 to m dofor j = 0 to m do

R(j)i = N (0, 1)

end forfor j = 0 to m do

R(j)i =

R(j)i

‖Ri‖end for�d = Ri

for all j such that 0 ≤ j ≤ i do

n =‚‚‚�d

‚‚‚p = �d · Rj

for k = 0 to m do

dk =dk−p·R(k)

j

n2

end forend forRi =

�d

‖�d‖end for

of points on the surface of a hypersphere are possible whenthe orthonormal basis R1 , ..., Rm ∈ R

m, is used to rotate apoint in m-dimensional space. This technique also makes itpossible to randomly and uniformly rotate a decision spacevector �x, by using matrix multiplication R�x, so that thereis no bias for any particular coordinate axis. The rotationmatrix is used to rotate about the origin of the principlecoordinate axes, in the decision space.

3.2 Rotated Test ProblemsIn this section, four rotated test problems are introduced.

These problems can be arbitrarily rotated in the decisionspace. Each of these problems has at least one objectivewhich is non-linear, and through a rotation of the coordinatesystem, parameter interactions can be introduced.

Problem R1 was first proposed by Deb [1]. It is charac-terised by a valley in objective f2. The Pareto-optimal setis situated along the length of this valley as well, and whenthe problem is subject to a rotation the valley can trap anon-rotationally invariant search from progressing along it,as was explained in Section 2.1. The function f1 is linear

0

1

2

3

4

5

-1 -0.5 0 0.5 1

f2

f1

R2

Figure 6: R2 Pareto-optimal front and feasible re-gion

and f2 is non-linear.

f1(y)= y1

f2(y) = g(y)h(f1(y), g(y))

h(f1(y), g(y)) = exp

„−f1(y)

g(y)

«

g(y) = 1 + 10(m − 1) +

mXi=2

ˆy2

i − 10 cos(4πyi)˜

y = Rx, −0.3 ≤ xi ≤ 0.3, for i = 1, 2, ..., m

|f1| ≤ 0.3

9>>>>>>>>>>>>>>=>>>>>>>>>>>>>>;

R1

Problem R2 is similar to the ZDT3 problem, and has aPareto-optimal front which is not continuous. R2 presentsa difficulty to an optimization algorithm, because it has tolocate a number of discontinuous Pareto-optimal fronts, andmaintain solutions in each of those fronts. When R2 is ro-tated, an optimization algorithm which only searches inde-pendently along the principle coordinate axes will generatenon-dominated solutions which skew away significantly fromthe Pareto-optimal front. The reason for this behaviour isthat perturbed solutions have to travel quite far along theprinciple coordinate axes before an independent perturba-tion can generate a solution which dominates the currentnon-dominated set. The function f1 is linear and f2 is non-linear. The g function is not multi-modal over the specifiedrange. The feasible space and Pareto-optimal front of thisproblem is shown in Figure 6.

f1(y)= y1

f2(y) = g(y)h(f1(y), g(y))

h(f1(y), g(y)) = 1.0 + exp

„−f1(y)

g(y)

«+

„f1(y) + 1.0

g(y)

«(sin(5πf1(y)))

g(y) = 1 + 10(m − 1) +

mXi=2

ˆy2

i − 10 cos(πyi)˜

y = Rx, −1.0 ≤ xi ≤ 1.0, for i = 1, 2, ..., m

|f1| ≤ 1.0

9>>>>>>>>>>>>>>>>>>=>>>>>>>>>>>>>>>>>>;

R2

Decision space variables which increment at a regular in-terval, evaluate with non-regular intervals in the objectivespace on Problem R3, making it hard to find a uniform dis-tribution along the Pareto-optimal front. The density of

687

0

1

2

3

4

5

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

f2

f1

R3

Figure 7: R3 Pareto-optimal front and feasible re-gion

solutions is lower towards lower f1 values. Problem R3 issimilar to the ZDT6 problem. f1 and f2 are non-linear func-tions. The g function is not multi-modal over the specifiedrange. The feasible space and Pareto-optimal front of thisproblem is shown in Figure 7.

f1(y) = 1.0 − exp(2.0y1) sin6(6πy1)/9.0

f2(y) = g(y)h(f1(y), g(y))

h(f1(y), g(y)) = 1.0 −„

f1(y)

g(y)

«2

g(y) = 1 + 10(m − 1) +mX

i=2

ˆy2

i − 10 cos(πyi)˜

y = Rx, −1.0 ≤ xi ≤ 1.0, for i = 1, 2, ..., m

0.3 ≤ f1 ≤ 1.0

9>>>>>>>>>>>>>>=>>>>>>>>>>>>>>;

R3

Problem R4 is based on the Schwefel function [12], where alocal front is located far from the global minimum. Pointsare easily trapped by this deceptive front. R4 is difficult, andobjective f2 is characterised by a number of valleys, includ-ing the highly deceptive valleys far from the true Pareto-optimal front. These valleys correspond to the modalitiesgenerated by function g. Each of these valleys can trappoints in a sub-optimal non-dominated front. The feasiblespace and Pareto-optimal front of this problem is shown inFigure 8.

f1(y)= y1

f2(y) = g(y)h(f1(y), g(y))

h(f1(y), g(y)) = exp

„−f1(y)

g(y)

«

g(y) = 1.0 + 0.015578(m − 1.0)+mX

i=2

(y2i − 0.25(yi sin(32.0

p|yi|)))

y = Rx, −1.0 ≤ xi ≤ 1.0, for i = 1, 2, ..., m

|f1| ≤ 1.0

9>>>>>>>>>>>>>>>>=>>>>>>>>>>>>>>>>;

R4

4. EXPERIMENTSAn algorithm for optimizing a multi-objective problem

which is not aligned with the principle coordinate axes musthave the property of rotational invariance. Secondly, likeother EMO algorithms it must maintain good coverage and

0

0.5

1

1.5

2

2.5

3

3.5

-1 -0.5 0 0.5 1

f2

f1

R4

Figure 8: R4 Pareto-optimal front and feasible re-gion

spread as well as convergence towards the Pareto-optimalfront.

We have used a rotationally invariant Non-dominated Sort-ing Differential Evolution algorithm with Directional Con-vergence and Spread (NSDE-DCS) for the purposes of con-trasting the performance of NSGA-II on the proposed ro-tated problems [5]. Each of the test problems described inthe previous section used a 10 dimensional decision spacefor this study. For the NSDE-DCS, F was set to 0.8 and Kwas set to 0.4. These settings were also employed in a pre-liminary study of rotational invariance [4]. The NSGA-II1

experiments used a mutation rate of 0.1 and crossover rateof 0.9. ηc and ηm are parameters within the NSGA-II whichcontrol the distribution of the crossover and mutation prob-abilities and were assigned values of 10 and 50 respectively.For both algorithms, a population size of 100 individuals wasemployed, and 50 runs of each algorithm were conducted, foreach test problem.

Experiments were conducted on each of the test prob-lems. Rotations were performed in the decision space, us-ing a random uniform rotation matrix generated using thetechnique described in Section 3.1. In 10-dimensions thereare 45 planes of rotation, introducing parameter interactionsbetween each parameter with every other. A new randomuniform rotation matrix was generated for each run of eachalgorithm.

5. DISCUSSIONThe results of our experiments are presented in Figures 9

and 10. For each run of the NSGA-II and NSDE-DCSwe have plotted the final non-dominated solutions set af-ter 1000 generations. The purpose of presenting these plotsis to demonstrate the type of behaviour one can expect fromtwo types of algorithms; a rotationally invariant EMO anda non-rotationally invariant EMO, on a variety of rotatedmulti-objective problems. The plots demonstrate the diffi-culty in convergence towards the Pareto-optimal front, aswell as the discovery of non-dominated solutions which arenot Pareto-optimal, which occurs when the EMO algorithmis not rotationally invariant.

From Figure 9 it is apparent that the majority of runs, ofthe rotationally invariant NSDE-DCS, converge and cover

1The variant of NSGA-II used in this study is the origi-nal NSGA-II available from the Kanpur Genetic AlgorithmsLaboratory site at http://www.iitk.ac.in/kangal/

688

the Pareto-optimal front of R1. For the discontinuous R2,and the non-uniformly mapped R3, some runs closely ap-proximated the Pareto-optimal front more or less. The dif-ference in performance, with the NSGA-II, is striking oneach of these problems. In Figure 10 it is apparent thatof the 50 runs of the NSGA-II algorithm, not a single runsucceeded in covering the Pareto-optimal front of R1. ForR2, and R3, only a few runs managed to find near Pareto-optimal solutions. Non-dominated solutions had a tendencyto skew away from the Pareto-optimal front on R2 whenthe NSGA-II was employed, which is consistent with thedescription in Section 3. In particular, complete coverage ofthe Pareto-optimal front of R2 was not possible with NSGA-II over the 50 runs. Problem R3 was the only proposedproblem which had a non-linear f1 and f2 function, and itpresented significant difficulties for NSGA-II with respect toconvergence to the Pareto-optimal front, and the spread ofsolutions generated.

R4 is a highly deceptive multimodal problem, with a lo-cal front which is close to the Pareto-optimal front withrespect to fitness. This local front maps to a region of thedecision space which is far from the global optimal Pareto-optimal front. Over the 50 runs of each algorithm, theNSDE-DCS demonstrated far better coverage of this frontthan the NSGA-II, although both demonstrated difficulty inconverging to the Pareto-optimal front, and had a tendencyto become trapped in a local front.

6. CONCLUSIONEMO algorithms should be invariant under a coordinate

rotation in order to efficiently optimize complex multi-objectiveproblems with many parameter interactions. Many currentresults reported in the literature are with respect to prob-lems which do not exhibit complex parameter interactions,although such interactions are characteristic of many real-world problems [7].

We have demonstrated four problems with different char-acteristics which maintain the same fitness landscape un-der an arbitrary rotation in the decision space. We havecompared NSGA-II with a rotationally invariant algorithm,NSDE-DCS, in order to contrast the difference in behaviour.It is apparent that a rotationally invariant scheme, suchas the NSDE-DCS, demonstrates superior performance onproblems with significant parameter interactions, comparedwith the NSGA-II. The primary reason for the poor perfor-mance of NSGA-II on the problems presented in this paper,is that NSGA-II uses a non-rotationally invariant crossoveroperator.

Although we have demonstrated a small range of prob-lems, many others are conceivable, including problems withmore than two objectives. Increasing the number of ob-jectives results in rotated problems which are potentiallymore challenging for non-rotationally invariant EMO algo-rithms, because there will be more non-dominated solutionsto choose from which are not optimal. It is relatively easy tofind such non-dominated solutions, but it is harder to findthe Pareto-optimal set.

This paper has proposed a starting point for re-evaluatingmany current EMO algorithms, and new algorithms whichhave yet to be proposed. with respect to the important topicof rotational invariance.

7. REFERENCES[1] K. Deb. Multi-objective genetic algorithms: Problem

difficulties and construction of test problems.Evolutionary Computation, 7(3):205–230, 1999.

[2] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan. Afast and elitist multi-objective genetic algorithm:Nsga-ii. IEEE Trans. Evol. Comput., 6(2):182–197,2002.

[3] S. Huband, L. Barone, L. While, and P. Hingston. Ascalable multi-objective test problem toolkit. InEvolutionary Multi-Criterion Optimization. ThirdInternational Conference, EMO 2005, Lecture Notesin Computer Science, volume 3410, pages 280–295,Mexico, 2005.

[4] A. Iorio and X. Li. Solving rotated multi-objectiveoptimization problems using differential evolution. InAI 2004: Advances in Artificial Intelligence: 17thAustralian Joint Conference on Artificial Intelligence,Cairns, Australia, page 861, Heidelberg, 2004.

[5] A. Iorio and X. Li. Incorporating directionalinformation within a differential evolution algorithmfor multi-objective optimization. In Proceedings of the2006 Genetic and Evolutionary ComputationConference (GECCO-06), 2006.

[6] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler.Scalable test problems for evolutionary multi-objectiveoptimization. Technical Report TIK-Report No. 112,Computer Engineering and Networks Laboratory(TIK), Swiss Federal Institute of Technology (ETH),Zurich, 2001.

[7] J. Knowles. Parego: A hybrid algorithm with on-linelandscape approximation for expensive multi-objectiveoptimization problems. IEEE Trans. Evol. Comput.,10(1):50–66, 2005.

[8] J. Knowles and D. Corne. Instance generators and testsuites for the multi-objective quadratic assignmentproblem. In Evolutionary Multi-CriterionOptimization. Second International Conference, EMO2003, Lecture Notes in Computer Science, volume2632, pages 295–310, Portugal, 2003.

[9] T. Okabe and Y. Jin. On test functions forevolutionary multi-objective optimization. In ParallelProblem Solving from Nature - PPSN VIII, LectureNotes in Computer Science, volume 3242, pages792–802, Birmingham, 2004.

[10] K. Price. Differential Evolution, chapter 2, pages79–108. McGraw-Hill, London UK, 1999.

[11] T. Robic. Performance of demo on new test problems:A comparison study. In Proceedings of the FourteenthInternational Eletrotechnical and Computer ScienceConference ERK 2005, pages 121–124, 2005.

[12] R. Salomon. Re-evaluating genetic algorithmperformance under coordinate rotation of benchmarkfunctions: A survey of some theoretical and practicalaspects of genetic algorithms. Bio. Systems,39(3):263–278, 1996.

[13] E. Zitzler, K. K. Deb, and L. Thiele. Comparison ofmulti-objective Evolutionary Algorithms: Empiricalresults. Evolutionary Computation, 8(2):173–195,2000.

689

1

1.5

2

2.5

3

-0.3 -0.2 -0.1 0 0.1 0.2 0.3

f2

f1

R1

Pareto-frontNSDE-DCS

0

1

2

3

4

5

-1 -0.5 0 0.5 1

f2

f1

R2

Pareto-frontNSDE-DCS

0

0.2

0.4

0.6

0.8

1

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

f2

f1

R3

Pareto-frontNSDE-DCS

0

0.5

1

1.5

2

2.5

3

3.5

4

-1 -0.5 0 0.5 1

f2

f1

R4

Pareto-frontNSDE-DCS

Figure 9: 50 runs of NSDE-DCS (Generation 1000)

1

2

3

4

5

6

7

8

9

10

-0.3 -0.2 -0.1 0 0.1 0.2 0.3

f2

f1

R1

Pareto-frontNSGA-II

0

5

10

15

20

25

30

-1 -0.5 0 0.5 1

f2

f1

R2

Pareto-frontNSGA-II

0

5

10

15

20

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

f2

f1

R3

Pareto-frontNSGA-II

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

-1 -0.5 0 0.5 1

f2

f1

R4

Pareto-frontNSGA-II

Figure 10: 50 runs of NSGA-II (Generation 1000)

690