This article was downloaded by: [The University of British Columbia]On: 09 December 2014, At: 17:39Publisher: Taylor & FrancisInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Engineering OptimizationPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/geno20

The benefits of adaptiveparametrization in multi-objectiveTabu Search optimizationTiziano Ghisu a , Geoffrey T. Parks a , Daniel M. Jaeggi a , JeromeP. Jarrett a & P. John Clarkson aa Engineering Design Centre, Department of Engineering ,University of Cambridge , UKPublished online: 16 Jul 2010.

To cite this article: Tiziano Ghisu , Geoffrey T. Parks , Daniel M. Jaeggi , Jerome P. Jarrett & P.John Clarkson (2010) The benefits of adaptive parametrization in multi-objective Tabu Searchoptimization, Engineering Optimization, 42:10, 959-981, DOI: 10.1080/03052150903564882

To link to this article: http://dx.doi.org/10.1080/03052150903564882

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Engineering OptimizationVol. 42, No. 10, October 2010, 959–981

The benefits of adaptive parametrization in multi-objective TabuSearch optimization

Tiziano Ghisu*, Geoffrey T. Parks, Daniel M. Jaeggi, Jerome P. Jarrett and P. John Clarkson

Engineering Design Centre, Department of Engineering, University of Cambridge, UK

(Received 10 June 2009; final version received 9 December 2009 )

In real-world optimization problems, large design spaces and conflicting objectives are often combined witha large number of constraints, resulting in a highly multi-modal, challenging, fragmented landscape. Thelocal search at the heart of Tabu Search, while being one of its strengths in highly constrained optimizationproblems, requires a large number of evaluations per optimization step. In this work, a modification of thepattern search algorithm is proposed: this modification, based on a Principal Components’Analysis of theapproximation set, allows both a re-alignment of the search directions, thereby creating a more effectiveparametrization, and also an informed reduction of the size of the design space itself. These changes makethe optimization process more computationally efficient and more effective – higher quality solutions areidentified in fewer iterations. These advantages are demonstrated on a number of standard analytical testfunctions (from the ZDT and DTLZ families) and on a real-world problem (the optimization of an axialcompressor preliminary design).

Keywords: multi-objective optimization; Tabu Search; principal components’ analysis; adaptiveparametrization

Nomenclature

�i design vector component increment� matrix made up of the eigenvalues of XXT

a design vector in principal components’ spaceC covariance matrixD matrix of design changesI identity matrixU matrix made up of the eigenvectors of XXT

V matrix made up of the eigenvectors of XTXX matrix of design vectorsx design vectorc design constraintDF Diffusion FactorDH De Haller number

*Corresponding author. Email: tg269@cam.ac.uk

ISSN 0305-215X print/ISSN 1029-0273 online© 2010 Taylor & FrancisDOI: 10.1080/03052150903564882http://www.informaworld.com

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

960 T. Ghisu et al.

f objective functionfred fraction of npca during which design space reduction occursg Lagrangian functioniiter number of evaluationsilocal number of continuous unsuccessful optimization stepsk minimum number of active PCsKoch Koch factorn total number of stagesnd number of ‘dormant’ optimization stepsnmtm MTM sizenpca number of iterations performed before a rotation of the parametrizationnregions factor defining the number of regions for diversificationnstm STM sizenvar number of design variablesPR pressure ratior number of optimization steps since the last change of coordinatesSM surge marginSPR static pressure rise coefficientSS initial step size (fraction of range)SSRF step size reduction factort number of active PCsxi design vector componentα exponent regulating the design space dimensionality reductionδ mean step size (as a fraction of range)m mass-flow through the compressorη compressor isentropic efficiencyλ eigenvalue (for the covariance matrix)� rotation matrixφ principal componentAS Approximation Set

1. Introduction

Real-world optimization tasks are often characterized by a complex design problem, with a largenumber of design variables, many local optima and various constraints generating a highly frag-mented design space, making it difficult to locate the truly optimal design(s). In recent years,the use of meta-heuristic optimization techniques has rapidly expanded in recognition of theireffectiveness in dealing with complex, real-world optimization problems to which traditional,gradient-based methods are ill suited. Examples of meta-heuristic techniques are SimulatedAnnealing (SA), Genetic Algorithms (GAs), Evolution Strategies (ESs) and Tabu Search (TS).Since the first multi-objective adaptation of GA in 1985, significant effort has been devoted to thisarea of research (Deb 2001), with multi-objective adaptations of ES (Knowles and Corne 1999),SA (Suppapitnarm et al. 2000) and TS (Baykasoglu et al. 1999, Caballero et al. 2004, Jaeggiet al. 2008) also presented in recent years.

Relatively little attention – compared to the other multi-objective meta-heuristics – has beenpaid to Tabu Search (Jones et al. 2002). The authors’ interest in TS is motivated by their interestin aerodynamic shape design problems. These tend to be optimization problems with large designspaces – see Duvigneau andVisonneau (2001), Harvey (2002), Kellar (2002), Oyama et al. (2002),Gaiddon and Knight (2003), Jameson (2003), Kipouros et al. (2008) as representative examples

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 961

– which, in combination with the often multi-modal nature of their objective functions and thelarge number of associated constraints, generate a highly fragmented search landscape that ischallenging for any optimization algorithm. Harvey (2002) found TS to be particularly effectivefor this application domain: the local search algorithm at its heart, by applying small changesto the design vector, is able to navigate the complex design space successfully, in circumstanceswhere the larger changes applied by other Evolutionary Algorithms (EAs) tend to generate largenumbers of unfeasible designs. Similar comments have been made by Kroo (2004) and Keskin andBestle (2006), who highlight the limitations of EAs when tackling high-dimensionality, highlyconstrained optimization problems. Like other meta-heuristics, the TS algorithm does not requireany gradient information, providing flexibility and robustness in solving complex optimizationproblems, without being adversely affected by the presence of noisy errors (Duvigneau andVisonneau 2003).

Jaeggi et al. (2008) developed the Multi-Objective TS (MOTS) algorithm used as the startingpoint for this work and a variant (PRMOTS) with an additional variable selection scheme basedon a path-relinking approach, and tested these on a set of five standard test functions from the ZDTfamily (Zitzler et al. 2003), obtaining performance comparable with that given by a leading GA(NSGA-II by Deb et al. 2002a). They concluded that, while NSGA-II might be a better choice forproblems with a relatively small number of variables, MOTS and PRMOTS can be good choicesfor large, highly constrained optimization problems.

Irrespective of the specific optimization scheme in use, three modules are needed to construct anautomatic optimization environment: the evaluation tool(s), the geometry modeller and the opti-mizer itself. The geometry modeller (or parametrization scheme) represents the ‘critical enablingfactor’ (Shahpar 2004) for an efficient exploration of the design space: it is important to ensurethat the parametrization scheme is able to cover all feasible designs, in order not to lose anypotentially good designs, but also that the minimum number of parameters is used, since theseaffect the size of the design space and thus the time required to solve the optimization problem(the number of design alternatives grows exponentially with the number of variables).

The dimensionality of the design space is a particular issue for TS, which is based on a localsearch algorithm. While this local search is key to TS’s success in tackling highly constrained,multi-modal optimization problems, it also means that the number of evaluations required peroptimization step is roughly proportional to the number of design variables, making the algorithmvery expensive for high-dimensionality problems. Means of reducing this computational costwithout reducing solution quality are thus of considerable interest.

In the remainder of this article, a methodology for adaptively modifying the design spaceparametrization, based on a Principal Components’Analysis (PCA) of the current Pareto-optimalset found during the optimization process, is presented. This allows a re-alignment of the searchdirections, creating a more effective parametrization, and a temporary, informed reduction of thedesign space dimensionality, leading to a faster and more effective search of the available designspace.

2. The basic multi-objective Tabu Search implementation

TS is a meta-heuristic method designed to help a search negotiate difficult regions of the searchspace (e.g. escape from local minima) by imposing restrictions (Glover and Laguna 1997). InJaeggi et al. (2008)’s implementation the local search phase at its heart is conducted with theHooke and Jeeves (H&J) algorithm (Hooke and Jeeves 1961): a suitable increment is chosen foreach variable and the value of the objective function is calculated in turn for x ′

i = xi + �i andx ′

i = xi − �i while keeping the other variables at their base values. The best allowed move ismade. The Short Term Memory (STM) records the last nstm visited points, which are tabu and

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

962 T. Ghisu et al.

thus cannot be revisited. The effect of the STM is that the algorithm behaves like a normal hill-descending algorithm until it reaches a minimum, then it is forced to climb out of the hollow andexplore further.

Two other important features of the TS algorithm are intensification and diversification. Inten-sification is associated with the Medium Term Memory (MTM) where the best solutions locatedare stored. Diversification is associated with the Long Term Memory (LTM), which records theareas of the search space that have been searched reasonably thoroughly by dividing the designspace into a number of sectors and recording how many times each sector has been visited. In thecurrent implementation, this is achieved by dividing each design variable into nregions and count-ing the number of solutions evaluated in those regions (Jaeggi et al. 2008). On diversification thesearch is restarted in an under-explored region of the design space.

The extension to multi-objective problems is straightforward: the MTM contains the set of non-dominated solutions found, while at every H&J move, in the absence of a single non-dominatedsolution, a random move is selected from among the set of non-dominated new designs. Thediscarded designs are not lost: they are stored in the MTM, if appropriate, and can then beselected during intensification.

3. An optimal parametrization scheme

The aim is to find the best linear orthogonal transformation (or orthonormal for unicity) of theoriginal set of coordinates. This is equivalent to finding the orthogonal matrix � (��T = I) sothat the transformation

a = �Tx (1)

is optimal. Geometrically this is equivalent to rotating the system of coordinates so that the newbasis is optimal in some sense. The result is the well-known Karhunen–Loeve (KL) expansion,which is particularly important in Pattern Recognition, and is usually referred to as Proper Ortho-gonal Decomposition, Principal Components’Analysis or Empirical Orthogonal Functions (Kirby2002).

Mathematically, the best eigenvector φ(1) can be defined as the one that maximizes the projectionof the data (or its mean square):

maxφ(1)

〈(φ(1), x)2〉 (2)

subject to

(φ(1), φ(1)) = 1, (3)

where ( , ) represents the scalar product operator and 〈 〉 represents summation over all the elementsof the data set.

This constrained optimization problem can be solved via the Lagrange multipliers technique:it is equivalent to maximizing the following function:

g1(λ1, φ(1)) = 〈(φ(1), x)2〉 − λ1[(φ(1), φ(1)) − 1] (4)

the stationary points of which are given by

∂g1

∂φ(1)= ∂g1

∂λ1= 0. (5)

Noting that (φ(1), x)2 = (φ(1), xxTφ(1)) and defining the covariance matrix C = 〈xxT〉,Equation (4) can be rewritten as

g1(λ1, φ(1)) = (φ(1), Cφ(1)) − λ1[(φ(1), φ(1)) − 1]. (6)

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 963

The stationary points of this equation are found from

∂g1

∂φ(1)= 2Cφ(1) − 2λ1φ

(1) = 0 (7)

or

Cφ(1) = λ1φ(1). (8)

The other components of this optimal basis can be found by solving a problem equivalent tothat in Equation (4) with the further constraint of orthogonality with respect to the previouslydetermined elements – this constraint is immediately satisfied because of the symmetry of matrixC (Kirby 2002). The elements of the optimal basis can then be found by solving the eigenvectorsproblem

Cφ(i) = λiφ(i). (9)

The KL basis has a property that can prove important in the solution of an optimization problem:the variances along each coordinate direction are proportional to the associated eigenvalues of thematrix C. This gives an immediate criterion for choosing the most significant components, i.e.those that have the greatest effect on the objective functions. The same approach used in PatternRecognition to produce a simpler, more accurate model of the problem under consideration (byneglecting the less significant variables) can be used in optimization for determining both the mostefficient system of coordinates to describe the design space and a reduced design space containingpotentially the best designs.

3.1. Integration with Tabu Search

While it is obvious that knowledge of the optimal parametrization can lead to a significantly fasterexploration of the design space, this requires knowledge of the Pareto front, the finding of whichis precisely the aim of the optimization process. However, it can be anticipated that, as the searchproceeds and better designs are found, the approximation set (the current Pareto-optimal set) willadvance and gradually become a better approximation to the true Pareto front. In the same waythat the approximation set represents an approximation to the Pareto front, it is likely that PrincipalComponents (PCs) based on analysis of the approximation set will be an approximation to theoptimal parametrization, becoming increasingly reliable as the approximation set approaches thetrue Pareto front. If this conjecture is verified, PCA conducted on the approximation set (thecontents of the MTM) as the search proceeds will provide a better parametrization than the initialone and thus facilitate a faster exploration of the most important regions of the design space.

In addition to the parameters already required by the original MOTS algorithm (see Table 1) andestablishing the number of unsuccessful optimization steps required before intensification, diver-sification and step size reduction moves are performed, a further parameter has been introduced,defining how often the process of determining an improved parametrization will be conducted.Over the first npca optimization steps, the optimizer will search the design space in the orig-inal coordinate system, after which the optimization will be continued in the newly definedparametrization, updated every npca optimization steps. A flowchart of the modified algorithm isgiven in Figure 1.

At every optimization step, a set of new candidate solutions will be generated by projecting theH&J move in the new coordinate system onto the original system of reference:

D = δ�I, (10)

where I is the identity matrix, � is the matrix containing the eigenvectors of the covariancematrix, δ is the mean of the ratios between the design variable step-sizes and the corresponding

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

964 T. Ghisu et al.

Table 1. Tabu Search parameter settings.

Parameter Description Value

nstm Size of STM 20nregions Divide search space into nvar × nregions regions 2intensify Perform intensification when ilocal = intensify 10diversify Perform diversification when ilocal = diversify 20restart Reduce step sizes and restart when ilocal = restart 50SS Initial step-sizes (as a percentage of variable range) 10%SSRF Step-sizes are multiplied by this factor at restart 0.5npca Perform PCA when r = npca 100k Minimum number of active PCs nvar/10α Coefficient regulating design space reduction 3fred Reintroduce all PCs when r/npca > fred 0.9

variable ranges and D is a matrix the columns of which correspond to the design changes in theoriginal system of coordinates, which need, in turn, to be added to and subtracted from the currentsearch point.

3.2. An informed design space reduction

As mentioned in Section 3, not only does PCA identify the most important search directions (thosethat maximize the mean squared projection of the Pareto-optimal designs onto the orthogonal basisgiven by the directions themselves), but it also provides a measure of their relative importance:the eigenvalues of the covariance matrix represent the variances calculated along each PC (Kirby2002). The latter offer a criterion on which an informed reduction of the design space dimension-ality can be based. While reducing the number of active design variables can significantly speedup the optimization process, completely neglecting some PCs is potentially dangerous, as the rel-ative importance of a design variable might vary with location in the design space. Furthermore,the optimal number of variables will be a function of the specific optimization problem.

The approach adopted is therefore to keep the generic ith PC ‘dormant’ for a number ofoptimization steps nd,i after each parametrization updating, calculated based on the correspondingeigenvalue λi :

nd,i = npca

[max

(0, 1 − λi

λk

)]α

(11)

whereλk is the kth largest PC eigenvalue andα is a positive exponent that regulates the design spacereduction: the smaller α is the more the search is biased towards the most important directions.This corresponds to reducing matrices I and D in Equation (10) to the first t columns, with t beinggiven by the number of PCs for which

(1 − λi

λk

)α

<r

npca, (12)

where r is the number of optimization steps since the last change of coordinates.Thus, in this approach, the k most important PCs (those with the largest eigenvalues) are always

active. This is in recognition of the importance of distributed computing environments for largescale optimization problems, where reducing the number of active PCs to less than the numberof simulations that can be run in parallel would not reduce the overall wall-clock time for theoptimization, only the number of design analysed.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 965

Figure 1. Flow diagram of PC-MOTS.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

966 T. Ghisu et al.

Figure 2. Number of active PCs for different optimization problems.

To avoid PCs corresponding to null eigenvalues being completely ruled out of the optimization,all the PCs are reintegrated after frednpca optimization steps in the new coordinate system.

Figure 2 compares the number of active PCs during an optimization for two extreme cases andfor the real-world optimization problem presented in Section 4.2. One extreme is the case whereall the PCs have the same importance (i.e. the kth and the last eigenvalues are equivalent); the otherthe case where less than k eigenvalues are non-zero. In the first case, no design space reduction isperformed and the optimization proceeds in the full PC space (or in the initial parametrization if theoptimal directions correspond to the original ones). In the second case, only the first k directionsare explored for frednpca optimization steps and the remaining (n − k) PCs are included forthe following (1 − fred)npca optimization steps, until PCA is performed again. In the real-worldoptimization, the number of active PCs is controlled by the eigenvalues’ relative importance andthe PCs corresponding to null eigenvalues are explored only in the last (1 − fred)npca optimizationsteps. The number of active PCs increases with the number of optimization steps after each PCA(by design through Equation 11), while the relative importance of the PCs changes after each PCAin light of the new improved parametrization and of the eigenvalues of the covariance matrix C. Inthe example shown in Figure 2 the number of PCs with non-zero eigenvalues increases after eachPCA (at least in the first 500 optimization steps), demonstrating the importance of not excludingdirections corresponding to null eigenvalues in the early phases of the search.

4. Testing procedure

In order to verify the performance gains achievable through the proposed adaptive change ofdesign space parametrization, a rigorous testing procedure was undertaken: the original MOTSimplementation by Jaeggi et al. (2008) and the new version (PC-MOTS) – modified to includePCA and the adaptive change of parametrization – were compared on the basis of a numberof standard functions, widely used to test the performance of multi-objective optimizers (Zitzleret al. 2003, Deb et al. 2002b). Some of these functions had already been used by Jaeggi et al. (2008)to compare the performance of their MOTS implementation with a leading multi-objective GA(NSGA-II by Deb et al. (2002a)).

In addition, in recognition of the well-known limitations of these test functions in mimickingreal-world optimization problems (Okabe et al. 2004), the two TS variants were also tested

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 967

on a real-world optimization problem: the preliminary aerodynamic design of a seven-stageintermediate pressure compressor from a three-spool gas turbine engine.

4.1. Analytical test functions

The performance of the two algorithms was compared on two families of analytical test functions.The first is the ZDT family (Zitzler et al. 2003), which contains problems with two objectivesand numbers of variables ranging from 10 to 30, designed to have Pareto-optimal sets whichare difficult for optimization algorithms to locate. The second is the DTLZ family, introducedby Deb et al. (2002b) to provide a number of benchmark problems to test the efficacy of meta-heuristic algorithms in handling optimization problems with more than two objectives. Both setsof problems have some shortcomings (the locations of their Pareto-optimal sets form continuousregions (Okabe et al. 2004) and their relevance to real-world problems is debatable) but theyare simple to implement, well studied, fast to compute and provide a good basis for assessingoptimizer performance. In the case of the DTLZ family, the test fuctions are scalable to have anynumbers of variables and objectives, making them ideal for testing the capabilities of the proposedmodification of the search algorithm, which should offer greater benefit in large, complex designspaces. Problems with 60 variables and four objectives were selected for this work.

The test functions are summarized in Tables 2 (ZDT family) and 3 (DTLZ family).

Table 2. ZDT test functions.

Name nvar nobj Objective functions Variable bounds

ZDT1 30 2 f1(x) = x1 x ∈ [0.0, 1.0]f2(x) = g(x)

[1 −

√x1

g(x)

]

g(x) = 1 + 9

∑nvari=2 xi

(nvar − 1)

ZDT2 30 2 f1(x) = x1 x ∈ [0.0, 1.0]

f2(x) = g(x)

[1 −

(x1

g(x)

)2]

g(x) = 1 + 9

∑nvari=2 xi

(nvar − 1)

ZDT3 30 2 f1(x) = x1 x ∈ [0.0, 1.0]f2(x) = g(x)

[1 −

√x1

g(x)− x1

g(x)sin(10πx1)

]

g(x) = 1 + 9

∑nvari=2 xi

(nvar − 1)

ZDT4 10 2 f1(x) = x1 x1 ∈ [0.0, 1.0]f2(x) = g(x)

[1 −

√x1

g(x)

]xi ∈ [−5.0, 5.0]

g(x) = 1 + 10(nvar − 1) + s(x) i = 2, . . . , nvar

s(x) = ∑nvari=2 [x2

i − 10 cos(4πxi)]ZDT6 10 2 f1(x) = 1 − exp(−4x1) sin6(6πx1) x ∈ [0.0, 1.0]

f2(x) = g(x)

[1 −

(f1(x)

g(x)

)2]

g(x) = 1 + 9

[ ∑nvari=2 xi

(nvar − 1)

]0.25

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

968 T. Ghisu et al.

Table 3. DTLZ test functions. In this work, nvar = 60 and M = nobj = 4.

Name Objective functions Variable bounds

DTLZ1 f1(x) = 12 x1x2 · · · xM−1(1 + g(x)) x ∈ [0.0, 1.0]

f2(x) = 12 x1x2 · · · (1 − xM−1)(1 + g(x))

.

.

.

fM−1(x) = 12 x1(1 − x2)(1 + g(x))

fM(x) = 12 (1 − x1)(1 + g(x))

g(x) = 100(|x| + ∑

xi∈x (xi − 12 )2 − cos(20π(xi − 1

2 )))

DTLZ2 f1(x) = cos(x1π2 ) cos(x2

π2 ) · · · cos(xM−1

π2 )(1 + g(x)) x ∈ [0.0, 1.0]

f2(x) = cos(x1π2 ) cos(x2

π2 ) · · · sin(xM−1

π2 )(1 + g(x))

.

.

.

fM−1(x) = cos(x1π2 ) sin(x2

π2 )(1 + g(x))

fM(x) = sin(x1π2 )(1 + g(x))

g(x) = ∑xi∈x (xi − 1

2 )2

DTLZ3 f1(x) = cos(x1π2 ) cos(x2

π2 ) · · · cos(xM−1

π2 )(1 + g(x)) x ∈ [0.0, 1.0]

f2(x) = cos(x1π2 ) cos(x2

π2 ) · · · sin(xM−1

π2 )(1 + g(x))

.

.

.

fM−1(x) = cos(x1π2 ) sin(x2

π2 )(1 + g(x))

fM(x) = sin(x1π2 )(1 + g(x))

g(x) = 100(|x| + ∑

xi∈x (xi − 12 )2 − cos(20π(xi − 1

2 )))

DTLZ4 f1(x) = cos(x1001

π2 ) cos(x100

2π2 ) · · · cos(x100

M−1π2 )(1 + g(x)) x ∈ [0.0, 1.0]

f2(x) = cos(x1001

π2 ) cos(x100

2π2 ) · · · sin(x100

M−1π2 )(1 + g(x))

.

.

.

fM−1(x) = cos(x1001

π2 ) sin(x100

2π2 )(1 + g(x))

fM(x) = sin(x1001

π2 )(1 + g(x))

g(x) = ∑x100i

∈x (x100i − 1

2 )2

DTLZ5 f1(x) = cos(θ1) cos(θ2) · · · cos(θM−1)(1 + g(x)) x ∈ [0.0, 1.0]f2(x) = cos(θ1) cos(θ2) · · · sin(θM−1)(1 + g(x))...

fM−1(x) = cos(θ1) sin(θ2)(1 + g(x))

fM(x) = sin(θ1)(1 + g(x))

θi = π4(1+g(x)

(1 + 2g(x)xi )

g(x) = ∑xi∈x (xi − 1

2 )2

DTLZ6 f1(x) = x1 x ∈ [0.0, 1.0]f2(x) = x2...

fM−1(x) = xM−1fM(x) = (1 + g(x))h(f1, f2, . . . , fM−1, g)

g(x) = 1 + 9|x|

∑xi∈x xi

h = M − ∑M−1i=1

[fi

1+g(1 + sin(3πfi))

]DTLZ7 fj (x) = 1

[ nM

∑j nM

i=(j−1) nM

xi for j = 1, . . . , M x ∈ [0.0, 1.0]subject to fM(x) + 4fj (x) − 1 ≥ 0 for j = 1, . . . , M − 1

2fM(x) + minM−1i,j=1,i �=j [fi(x) + fj (x)] − 1 ≥ 0

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 969

4.2. Real-world optimization test case

Having analysed the performance of PC-MOTS on a series of widely adopted test functions (seeSection 4.1), the next step is to verify its effectiveness in a more challenging (and more closelyrepresentative of the type of optimization problem of interest to the authors) test case. At the sametime, the problem needs to be soluble within reasonable computational limits, in order to be ableto run the optimization sufficient times to achieve statistical significance.

A real-world problem that complies with all these requirements is the preliminary design ofcompression systems for gas turbine engines: starting from an existing geometry for a genericseven-stage core compressor (Jarrett et al. 2007), the optimization had the goal of seeking possibleimprovements in the performance without penalizing the operating margin and avoiding largechanges in the geometry of the other engine components.

The optimization problem is described in detail by Ghisu et al. (2006). Forty-seven designvariables allow the modification of annulus shape, number of blades, blade solidities, pressure ratiodistribution and flow angles; isentropic efficiency and surge margin represent the two objectivesof the optimization, while a number of constraints have been specified to limit the impact of themodified design on the preceding and following components and the load carried by the differentblade rows (for a total of two inequality and four equality constraints). The optimization problemis summarized in Table 4.

A proprietary code for multi-stage axial compressor mean-line performance prediction – widelydeveloped, tested and applied to the design of highly successful turbomachines in recent decades– was used to evaluate compressor performance.

The presence of design constraints can complicate significantly an optimization problem,dividing the design space into feasible and infeasible regions, which are not necessarily con-vex and possibly disjoint: a good choice of constraint handling approach can prove essential fora successful exploration of the design space (Bäck et al. 1997).

In general, constraints were treated by adding a penalty term to the objective function. Non-converged solutions (which often arise in aerodynamic problems) were treated with a barriermethod, assigning a very high value to the objective function(s) and thus advising the optimizerto move away from the corresponding design.

4.3. Performance analysis

The output of a single-objective optimization is a single objective function value – this allows theperformance of different optimizers to be compared directly on the basis of the obtained objectivevalues.

Table 4. Definition of the optimization problem.

minimize −η

−SM

subject to DHmin ≥ DH

SPRmax ≤ SPR

DFmax ≤ DF

Kochmax ≤ Koch

m = m

PR = PR

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

970 T. Ghisu et al.

In contrast, the output of a multi-objective optimization is a set of solutions (the approximationset) approximating the true Pareto-optimal set, making the comparison of different optimizersless straightforward. It is therefore essential to be able to assign a measure of quality to theapproximation set. Zitzler et al. (2003) give a detailed discussion of the performance assessmentof multi-objective optimizers. Following their advice, and that given by Fonseca et al. (2005),two performance indicators have been used in this study: the unary epsilon indicator and thehypervolume indicator (both described below).

Furthermore, as the optimization algorithms under evaluation are not deterministic, the resultsfrom multiple runs must be collated and assessed.A suitable statistical test for determining whethercertain algorithms tend to produce better approximation sets (using the performance indicatorschosen) is the Mann–Whitney test (Conover 1999), described below.

The Pareto fronts obtained by the two TS variants were also compared according to domination-related criteria. The percentage of solutions in a Pareto front found by one optimizer dominatedby the solutions in a Pareto front found by the other optimizer, and vice versa, were calculated,and the contribution of each optimizer to a ‘global’Pareto front, obtained by combining the resultsof the two runs, was also determined. As 100 runs were available for each test function with eachTS variant, 10,000 different pairs of runs could be compared, leading to a statistical distributionof these measures of comparison.

4.3.1. Unary epsilon indicator

The unary epsilon indicator was proposed by Zitzler et al. (2003) and makes direct use of theprinciple of Pareto-dominance, making it very intuitive. Consider two approximation sets A andB where A dominates B. The epsilon indicator is a measure of the smallest distance one wouldneed to translate every point in B so that it dominates A. If set A is chosen to be a reference set,such that it dominates sets B and C, then B and C can be directly compared on the basis of theepsilon indicator with respect to the reference set.

4.3.2. Hypervolume indicator

The hypervolume indicator was proposed by Zitzler and Thiele (1998) and its properties arediscussed in depth in Zitzler et al. (2003). Each point in the approximation set forms a hyper-rectangle of given hypervolume with respect to a reference point that lies beyond the boundsof the approximation set. The hypervolume indicator is the hyperspace of the union of all thesehyperrectangles.

4.3.3. Mann–Whitney test

The Mann–Whitney test (also known as the Wilcoxon test) is a non-parametric test designedto determine whether a certain population tends to produce greater observed values of a givenquantity than a second population. It is a rank-based method – that is, all observations from thepopulations are ranked together, and the ranks of individuals in a population are used to calculatethe test statistic. Its null hypothesis is that the two populations are identical (i.e. they come fromthe same distribution). Its alternate hypothesis is that one of the populations tends to yield greaterobserved values than the other. Conover (1999) gives a detailed explanation of the method.

In this work, the null hypothesis has been tested at a confidence level of 0.05 and, in cases wherethis could be discarded, the probability of PC-MOTS being better than MOTS evaluated. Thisprobability represents the complement to unity of the level of significance for the correspondingone-tailed test.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 971

4.4. Experimental details

4.4.1. Analytical test functions

Each algorithm was run 100 times on each test function and the non-dominated sets obtainedfor each optimization run – after 9000, 15,000 and 60,000 evaluations respectively for the 10variable, 30 variable and 60 variable optimization problems – were archived and used to generatethe performance measures. Similar analyses were also performed at earlier stages during theoptimization process: the results are not presented here for conciseness, but are available from theauthors on request. Every optimization run was initialized from the centre of the design space.This choice seems more appropriate for the type of application of interest to the authors, sinceobtaining a feasible design for high dimensionality, highly constrained design problems by meansof a random selection is often extremely difficult (Ghisu 2009). Real-world design optimizationproblems often start from a known datum design.

The choice of the number of optimization steps is in line with previous studies – Deb (2001),Jaeggi et al. (2008) and Asouti and Giannakoglou (2009) have assessed algorithm performanceon analytical test functions (from the ZDT family) at up to 25,000 function evaluations, whileKipouros et al. (2008) required approximately 40,000 evaluations in the solution of a real-worldbiojective optimization problem (the aerodynamic optimization of a compressor blade) in a26-variable design space.

Since the original MOTS algorithm (Jaeggi et al. 2008) was implemented in a C++ program, themodifications required by the PC-MOTS algorithm were implemented in C++ and FORTRAN 77.

The combined set of non-dominated designs from all 200 optimization runs (100 for eachMOTS variant) was used as the reference set for calculating the performance indicators. Theresults were normalized such that all objective function values fell in the range 0.0–1.0. The point{1.0, . . . , 1.0} was used as the nadir point for the hypervolume calculation, while the indicatorswere always expressed relative to the reference set. The calculation of the performance indicatorswas performed using the software provided in the PISA toolkit (Bleuler et al. 2003).

4.4.2. Real-world test case

An optimization run for the real-world test case presented in Section 4.2 required approximately3 hours on an AMD Athlon 2.0 GHz machine. Fifty optimization runs were performed for eachMOTS variant, for a total run time of about 12 days. The approximation sets were archived every20,000 evaluations, until 200,000 evaluations were reached. The same procedure described inSection 4.4.1 was used to generate the performance metrics.

4.5. Parameters settings

The parameters required by both MOTS variants are summarized in Table 1. The ones already usedby the original MOTS algorithm were kept fixed at the values specified by Jaeggi et al. (2008).The use of k to maximize the use of distributed computing resources has been already discussedin Section 3.2. The remaining parameters (npca, α and fred) were set by the authors based onexperience. A thorough study of the influence of these parameters on the algorithm’s performancewas not conducted, in part because of the time required for a single optimization (especially forreal-world optimization problems), and in part because of the likely dependence of these resultson the specific optimization problem under consideration. The main benefit of the algorithm isits ability to change the design space parametrization adaptively and to concentrate the search onthe most important PCs; the proposed parameters represent one possible way of achieving thesegoals and have been set to strike a suitable balance between a focus on the most important PCs

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

972 T. Ghisu et al.

and the need to allow the algorithm to explore directions corresponding to null eigenvalues. Thenumber of PCA transformations performed during an optimization depends both on the size andthe complexity of the design space (i.e. on how different the optimal basis and the approximatedones are). A simple study of the effect of npca (likely to be the most influential parameter) onthe algorithm’s performance was conducted by comparing the results obtained for three differentvalues of this parameter (50, 100 and 200) on the test functions introduced in Section 4.4.1.For conciseness, this comparison is presented only in terms of the domination-related criteriadescribed in Section 4.3.

4.6. Run-time performance

In addition to the computational cost of the original MOTS implementation, the PC-MOTSalgorithm requires the eigenvalues and eigenvectors of the covariance matrix C to be calcu-lated every npca optimization steps. The most efficient method is by means of a singular valuedecomposition (SVD), which allows the matrix X, which contains the approximation set andhence is a nvar × nmtm matrix, to be expressed as

X = U�VT (13)

where U is a nvar × nvar matrix the columns of which are the eigenvectors of the covariancematrix C = XXT, � is a nvar × nmtm diagonal matrix containing the eigenvalues of C and VT isthe transpose of V, a nmtm × nmtm matrix containing the eigenvectors of XTX. More details onthe SVD methodology are given by Press et al. (1994).

Detailed run-time figures were not calculated for two reasons. First, the additional complexityis minimal, especially in consideration of the fact that the SVD only needs to be performedevery npca optimization steps. Second, and more importantly, it is not uncommon for real-worldoptimization problems to require a design evaluation time of the order of minutes or even hours,making the portion of the total computational cost attributable to the optimization algorithm itselfnegligible.

As highlighted by Jaeggi et al. (2008), a specific feature of both TS algorithms is the use ofmultiple memories (some unbounded) which can impact the performance of the algorithm throughthe computer’s RAM usage. For instance, saving 25,000 design solutions with 100 design variablesand 5 objectives would require a total of 2,625,000 foating point numbers to be stored, or about10 MB. This represents a very low memory requirement for modern desktop or server computersystems.

5. Results

5.1. Analytical test functions

5.1.1. ZDT family

The means and standard deviations of both the unary epsilon and hypervolume performanceindicators are shown in Table 5 (a lower value indicates a better approximation set). The samestatistics for the size of the approximation sets are also shown in Table 5. For all five problemsand for both performance indicators the Mann–Whitney test shows that PC-MOTS exhibits betterperformance than MOTS with a probability of 1.

It is evident that re-directing the search towards the most important PC directions and reducingthe design space to include only the most important of these allows the optimizer to concentrate the

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 973

Table 5. Mean and standard deviations of the epsilon and hypervolume indicators, and of the number ofPareto-optimal solutions (PS size), for the ZDT family of test functions.

MOTS PC-MOTS

Functions Evaluations Indicator Mean SD Mean SD

ZDT1 15,000 eps 0.0215 0.0064 0.0030 0.0008hyp 0.0140 0.0077 0.0003 0.0002

PS size 98.57 9.67 1000.18 490.06

ZDT2 15,000 eps 0.0250 0.0134 0.0026 0.0006hyp 0.0243 0.0217 0.0002 0.0001

PS size 103.31 10.34 1000.19 508.67

ZDT3 15,000 eps 0.2286 0.1266 0.0068 0.0041hyp 0.2888 0.1688 0.0030 0.0062

PS size 42.86 5.39 303.65 67.22

ZDT4 9000 eps 0.0246 0.0179 0.0079 0.0069hyp 0.0091 0.0028 0.0015 0.0013

PS size 21.42 2.68 107.86 31.27

ZDT6 9000 eps 0.0251 0.0069 0.0053 0.0010hyp 0.0320 0.0198 0.0072 0.0015

PS size 275.02 16.50 2125.21 260.28

Figure 3. Pareto fronts comparison for ZDT3 after 10,000 evaluations (50% attainment surfaces).

search in the regions that are more likely to produce optimal designs, as shown by the significantincrease in the size of the approximation set (usually one order of magnitude larger for PC-MOTSthan for MOTS). The more effective search is reflected in the lower values of the performancemetrics (usually one order of magnitude, and often two orders, lower for PC-MOTS than forMOTS). Performance variability is also correspondingly decreased.

In order to understand better the differences in the evolution of the Pareto fronts, in Figures 3and 4 a comparison between the 50% attainment surfaces – representing the combination of designsolutions attained in at least half of the optimization runs – obtained by MOTS and PC-MOTSfor two test functions (ZDT3 and ZDT6), after two-thirds of all evaluations, is presented. ZDT3is a particularly challenging optimization problem, since the Pareto front is disjoint and formedby five separate curves: it is evident how, even after 10,000 evaluation, PC-MOTS has already

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

974 T. Ghisu et al.

Figure 4. Pareto fronts comparison for ZDT6 after 6000 evaluations (50% attainment surfaces).

been able to define all these separate regions of the Pareto front quite satisfactorily, while onlytwo of these have been adequately explored by MOTS. The situation is similar for ZDT6 (a lesschallenging test function): it is clear that 6000 evaluations are sufficient for PC-MOTS to exploreevery region of the Pareto front, while, after the same number of evaluations, only the lower partof the Pareto front has been adequately explored by MOTS.

The same results are illustrated using the domination-based criteria (described in Section 4.3):Figure 5 shows the distributions for the percentage of Pareto-optimal solutions found by oneMOTS variant which are dominated by solutions found by the other, while Figure 6 shows thecontribution of each variant to the global combined Pareto front, for three different values ofnpca. These figures clearly illustrate the superiority of the performance of PC-MOTS on the ZDTproblems and the limited sensitivity of the algorithm to the value of parameter npca. A slightdeterioration in performance is apparent for the largest npca value, due to the larger intervalsbetween successive PCAs which limit the adaptivity of the design space parametrization.

5.1.2. DTLZ family

Means and standard deviations for the unary epsilon and hypervolume indicators and for the sizeof the approximation sets are reported in Table 6, for each of the seven test functions from theDTLZ family. The results from the Mann–Whitney test show that PC-MOTS performs better thanMOTS according to both performance indicators with probability 1 on every DTLZ test functionexcept DTLZ7. In this latter case, there is no significant difference between the two algorithms.This can be explained by looking at the mathematical formulation of the problem (Table 3), whichhas a particular symmetry that makes every design variable equivalent from the point of view ofthe objective functions; in other words, there is no optimal basis and any rotation of the system ofcoordinates leads to a representation of the design space that is equivalent to the original one. Ina similar situation, while it is clearly impossible to produce any performance improvements withthe proposed algorithm, it is nevertheless important to verify that no penalties are introduced bythe adaptive change of coordinates.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 975

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

(a) MOTS.

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

0.6

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

npca

= 50

npca

= 100

npca

= 200

(b) PC-MOTS.

Figure 5. Percentages of dominated points (ZDT series).

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

CONTRIBUTED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S npca

= 50

npca

= 100

npca

= 200

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

CONTRIBUTED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

(a) MOTS.

(b) PC-MOTS.

Figure 6. Contributions to the global Pareto front (ZDT series).

Excluding the DTLZ7 function, not only does PC-MOTS outperform MOTS in all cases, butthere is also a significant reduction in both the performance indicators and a significantly largernumber of designs in the approximation sets found (Table 6). The variability in the performanceindicators is also reduced for most of the test functions.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

976 T. Ghisu et al.

Table 6. Mean and standard deviations of the epsilon and hypervolume indicators, and of the numberof Pareto-optimal solutions (PS size), for the DTLZ family of test functions.

MOTS PC-MOTS

Functions Evaluations Indicator Mean SD Mean SD

DTLZ1 60,000 eps 0.4355 0.1213 0.1910 0.0826hyp 0.0027 0.0014 0.0008 0.0006

PS size 697.18 36.18 5751.18 812.73

DTLZ2 60,000 eps 0.2141 0.0654 0.1079 0.0457hyp 0.0944 0.0164 0.0056 0.0063

PS size 687.20 38.15 8447.63 1066.12

DTLZ3 60,000 eps 0.1328 0.0494 0.0760 0.0200hyp 0.0103 0.0060 0.0046 0.0038

PS size 702.30 41.14 3918.78 620.33

DTLZ4 60,000 eps 0.4101 0.0914 0.2186 0.0384hyp 0.3636 0.1065 0.1091 0.0516

PS size 885.06 100.33 3295.28 632.80

DTLZ5 60,000 eps 0.1422 0.0131 0.1147 0.0085hyp 0.0974 0.0265 0.0683 0.0213

PS size 702.98 96.03 1000.62 148.04

DTLZ6 60,000 eps 0.7511 0.0009 0.4183 0.1348hyp 0.1169 0.0008 0.0484 0.0331

PS size 761.74 91.76 10650.44 1140.84

DTLZ7 60,000 eps 0.5518 0.1412 0.5748 0.1059hyp 0.1975 0.0496 0.2038 0.0272

PS size 74.90 65.38 78.38 67.95

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

npca

= 50

npca

= 100

npca

= 200

(a) MOTS.

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

(b) PC-MOTS.

Figure 7. Percentages of dominated points (DTLZ series).

The same results were analysed using the domination-based criteria. Figure 7 shows the dis-tributions for the percentage of Pareto-optimal solutions found by one MOTS variant which aredominated by solutions found by the other, for three values of npca. The contributions of eachvariant to the global combined Pareto front are very similar to those for the ZDT family (Figure 6).The DTLZ7 function was excluded from this analysis, for the reasons explained above. Again,

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 977

the superiority of the performance of PC-MOTS is readily apparent, for all three values of theparameter npca. A limited reduction in the algorithm’s effectiveness can be observed for largervalues of npca, due to the reduced adaptivity of the design space parametrization.

5.2. Real-world test case

Having verified the performance of the proposed improvement of the local search algorithm ona series of standard analytical test functions, the next step is to validate the approach on a morecomplex optimization problem. The preliminary design of a seven-stage intermediate pressurecompressor from a three-spool gas turbine engine, presented in Section 4.2, represents an ‘ideal’test case, as it presents a complexity typical of real-world optimization problems but with areasonable computational cost, making it possible to execute a number of optimization runssufficient to obtain statistically significant results.

The ratios between the average performance metrics for the two MOTS variants are presented inFigure 8: it is clear that, at the price of a small initial degradation in the performance indicators dueto the time required to locate a satisfactory approximation of the optimal basis, there is then a sig-nificant performance improvement even in the early stages of the optimization process (40,000 to60,000 evaluations, corresponding to approximately 800 to 1200 optimization steps for the MOTSalgorithm). The ratios between the performance metrics for the two variants continue to decrease,at least until 200,000 evaluations, although they seem to be tending towards asymptotic values.

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2x105

0

0.2

0.4

0.6

0.8

1

EVALUATIONS

RA

TIO

EPSILON INDICATORHYPERVOLUME INDICATOR

Figure 8. Variation of the ratios of average epsilon and hypervolume indicators – PC-MOTS ÷ MOTS.

Table 7. Mann–Whitney p-values, for the real-world test case. A = MOTS,B = PC-MOTS. H0 means that the null hypothesis is verified.

Evaluations Indicator p(B > A) Best algorithm

20,000 eps 0.0146 MOTShyp H0 –

40,000 eps 0.9999 PC-MOTShyp 1.0 PC-MOTS

60,000+ eps 1.0 PC-MOTShyp 1.0 PC-MOTS

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

978 T. Ghisu et al.

The results of the Mann–Whitney test are reported in Table 7. The p-values shown representthe probabilities that Algorithm B (PC-MOTS) exhibits better performance than Algorithm A(MOTS) for the specified performance indicator at the given stage in the optimization. PC-MOTSoutperforms MOTS at all evaluation levels, with the exception of the first one, the particularity ofwhich has already been highlighted. What is even more remarkable is the fact that the p-values

0 0.2 0.4 0.6 0.8 10

5

10

15

20

25

30

ISENTROPIC EFFICIENCY (PP IMPROVEMENT)

SU

RG

E M

AR

GIN

(%

IMP

RO

VE

ME

NT

)

PC−MOTSMOTS

(a) 20,000 evaluations.

0 0.2 0.4 0.6 0.8 10

5

10

15

20

25

30

ISENTROPIC EFFICIENCY (PP IMPROVEMENT)

SU

RG

E M

AR

GIN

(%

IMP

RO

VE

ME

NT

)

PC−MOTSMOTS

(b) 200,000 evaluations.

Figure 9. Pareto front evolution for the compressor test case (50% attainment surfaces).

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

(a) MOTS.

0 10 20 30 40 50 60 70 80 90 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

DOMINATED SOLUTIONS (PERCENTAGE)

FR

AC

TIO

N O

F R

UN

S

(b) PC-MOTS.

Figure 10. Percentages of dominated points (real-world test case).

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 979

are almost always 1. This means that the worst optimization run with PC-MOTS is better thanthe best MOTS optimization, if one is prepared to complete at least 40,000 evaluations, which isa relatively small number for the type of optimization problem under consideration (47 variablesand 2 objectives).

Figure 9 presents a comparison of the two approximation sets at the first and last evaluationlevels. The percentage point improvement in isentropic efficiency and percentage improvementin surge margin are reported here, to allow a quantification of the performance gains from thepoint of view of the compressor design. 50% attainment surfaces are shown. While the Paretofronts after 20,000 evaluations (Figure 9(a)) are similar, the second plot (for 200,000 evaluations)shows a consistent improvement in the search effectiveness, which translates into a more advancedapproximation set. The difference in achieved performance improvements is evident. These resultsdemonstrate how the adaptive design space parametrization and size reduction strategies facilitatenot only a faster search of the design space (leading to equal quality results in shorter times) butalso to better results, since concentrating the search more on the most important PCs allows themost promising areas of the design space to be searched thoroughly and noisy design variables,producing small or oscillatory variations in the objective functions, to be de-emphasized.

The same results were also analysed using the domination-based criteria. Figure 10 shows thedistributions for the percentage of Pareto-optimal solutions found by one MOTS variant whichare dominated by solutions found by the other. The contributions of each variant to the globalcombined Pareto front are again similar to those for the ZDT family (Figure 6). These analysesconfirm the superiority of the performance of PC-MOTS.

6. Conclusions

Real-world optimization problems are often characterized by large design spaces and a largenumber of objectives and constraints, generating a highly fragmented, multi-modal landscapethat can prove particularly challenging for any optimization algorithm. Tabu Search has provedparticularly efficient in the solution of such optimization problems, thanks to its reliance on alocal search algorithm (Hooke and Jeeves pattern search in the case of our implementation) that,by applying small changes to the design vector, is able to navigate the complex design spaceefficiently. In this work, a modification of the local search algorithm, based on an adaptive changeof the design space parametrization, allows a re-direction of the search towards the most importantdirections of the design space – those that maximize the sum of the square of the projection of theoptimal designs on the directions themselves, which are more likely to produce improvements inthe figures of merit.

The proposed modification, based on a Principal Components’ Analysis of the approximationset, was integrated into the multi-objective Tabu Search implementation of Jaeggi et al. (2008).This allows the most important directions of the design space to be found as the eigenvectorsof the covariance matrix constructed with the design vectors in the approximation set, whilethe eigenvalues of the same matrix give a measure of the relative importance of the directionsdefining the new parametrization. This represents an efficient and reliable criterion by which toreduce (at least temporarily) the dimensionality of the problem, leading to a greater effective-ness of the local search (not only because of the smaller number of designs to be evaluated ateach optimization step, but also because of the elimination of the noisy, less important designvariables).

The two algorithms have been compared on two sets of standard test functions (from the ZDTand DTLZ families) and on a real-world optimization problem (the optimization of a seven-stageaxial compressor preliminary design). The results demonstrate that the adoption of the adaptivedesign space parametrization and reduction of the problem dimensionality to include only the most

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

980 T. Ghisu et al.

significant search directions facilitates a substantial improvement in optimizer performance overa range of test functions, while performance variability is correspondingly reduced. The proposedmethodology relies on the existence of an optimal parametrization (see Section 3), different fromthe initial one. In complex design problems, generating an optimal parametrization by intuition isextremely difficult and, even if this can be done, it is extremely unlikely that all the design variableswill have the same impact on the objective functions. Thus, in most circumstances, adoption ofthe approach proposed in this paper will improve optimizer performance. Although the benefitsof this approach have been demonstrated for a multi-objective Tabu Search implementation, theapproach is not optimizer-specific and could be readily applied to other meta-heuristics.

References

Asouti, V.G. and Giannakoglou, K.C., 2009. Aerodynamic optimization using a parallel asynchronous evolutionaryalgorithm controlled by strongly interacting demes. Engineering Optimization, 41 (3), 241–257.

Bäck, T., Fogel, D.B., and Michalewicz, Z., 1997. Handbook of evolutionary computation. New York: Oxford UniversityPress.

Baykasoglu, A., Owen, S., and Gindy, 1999. A taboo search based approach to find the Pareto optimal set in multipleobjective optimization. Engineering Optimization, 31 (6), 731–748.

Bleuler, S., et al., 2003. PISA – a Platform and programming language Independent interface for Search Algorithms. In:C.M. Fonseca, et al., eds. Evolutionary multi-criterion optimization (EMO 2003). Lecture notes in computer scienceVol. 2632/2003. Heidelberg: Springer-Verlag, 494–508.

Caballero, R., Gandibleux, X., and Molina, J., 2004. MOAMP – a generic multi-objective metaheuristic using an adaptivememory. Technical report, University of Valenciennes.

Conover, W., 1999. Practical nonparametric statistics. 3rd ed. New York: Wiley.Deb, K., 2001. Multi-objective optimization using evolutionary algorithms. Chichester: Wiley.Deb, K., et al., 2002a. A fast and elitist multiobjective genetic algorithm: NSGA–II. IEEE Transactions on Evolutionary

Computation, 6 (2), 182–197.Deb, K., et al., 2002b. Scalable multi-objective optimization test problems. In: Congress on evolutionary computation

(CEC 2002), Honolulu, HI. 825–830.Duvigneau, R. and Visonneau, M., 2001. Single- and multi-objective optimisation for high fidelity CFD using Genetic

Algorithms. In: Evolutionary methods for design, optimization and control, Athens, Greece, 19–21 September 2001.Duvigneau, R. and Visonneau, M., 2003. Shape optimization strategies for complex applications in Computational Fluid

Dynamics. In: 2nd international conference on computer applications and information technology in the maritimeindustries, Hamburg, Germany, 14–17 May 2003.

Fonseca, C., et al., 2005. A tutorial on the performance assessment of stochastic multiobjective optimizers. Technicalreport – an invited talk presented by J. Knowles at EMO 2005, Guanajuato, Mexico.

Gaiddon, A. and Knight, D.D., 2003. Multicriteria design optimization of integrated three-dimensional supersonic inlets.Journal of Propulsion and Power, 19 (3), 456–463.

Ghisu, T., 2009. Robust aerodynamic design of compression systems. Thesis (PhD). University of Cambridge, UK.Ghisu, T., et al., 2006. Multi-objective optimisation of aero-engine compressors. In: 3rd international conference on the

future of gas turbine technology, Brussels, Belgium, 11–12 October 2006.Glover, F. and Laguna, M., 1997. Tabu search. Norwell, MA: Kluwer Academic.Harvey, S.A., 2002. The design optimisation of turbomachinery blade rows. Thesis (PhD). University of Cambridge, UK.Hooke, R. and Jeeves, T.A., 1961. ‘Direct search’solution of numerical and statistical problems. Journal of the Association

for Computing Machinery, 2 (8), 212–229.Jaeggi, D.M., et al., 2008. The development of a multi-objective Tabu Search algorithm for continuous optimisation

problems. European Journal of Operational Research, 185 (3), 1192–1212.Jameson, A., 2003. CFD for aerodynamic design and optimization: its evolution over the last three decades. In: 16th AIAA

computational fluid dynamics conference (AIAA 2003), Orlando, FL, 23–26 June 2003. 3438.Jarrett, J.P., Dawes, W.N., and Clarkson, P.J., 2007. An approach to integrated multi-disciplinary turbomachinery design.

ASME Journal of Turbomachinery, 129 (3), 488–494.Jones, D., Mirrazavi, S.K., and Tamiz, M., 2002. Multi-objective meta-heuristics: an overview of the current state-of-the-

art. European Journal of Operational Research, 137 (1), 1–9.Kellar, W.P., 2002. Geometry modeling in computational fluid dynamics and design optimisation. Thesis (PhD). University

of Cambridge, UK.Keskin, A. and Bestle, D., 2006. Application of multi-objective optimization to axial compressor preliminary design.

Aerospace Science and Technology, 10 (7), 581–589.Kipouros, T., et al., 2008. Biobjective design optimisation for axial compressors using Tabu Search. AIAA Journal,

46 (3), 701–711.Kirby, M., 2002. Geometric data analysis. An empirical approach to dimensionality reduction and the study of patterns.

New York: Wiley.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14

Engineering Optimization 981

Knowles, J.D. and Corne, D.W., 1999. Approximating the nondominated front using the Pareto archived evolutionstrategies. Evolutionary Computation, 3 (7), 1–26.

Kroo, I., 2004. Innovations in aeronautics – 2004 AIAA Dryden Lecture. In: 42nd AIAA aerospace sciences meeting andexhibit (AIAA 2004), Reno, NV, 5–8 January 2004. 0001.

Okabe, T., et al., 2004. On test functions for evolutionary multi-objective optimization. In: X. Yao, et al., eds. Parallelproblem solving from nature – PPSNVIII. Lecture notes in computer scienceVol. 3242/2004. Berlin: Springer-Verlag,792–802.

Oyama, A., Liou, M.S., and Obayashi, S., 2002. Transonic axial-flow blade shape optimization using evolutionaryalgorithm and three dimensional Navier–Stokes solver. In: 9thAIAA/ISSMO symposium on multidisciplinary analysisand optimization (AIAA 2002), Atlanta, GA, 4–6 September 2002. 5642.

Press, W.H., et al., 1994. Numerical recipes in Fortran: the art of scientific computing. Cambridge, UK: CambridgeUniversity Press.

Shahpar, S., 2004. Automatic aerodynamic design optimisation of turbomachinery components – an industrialperspective. In: H. Deconinck, J. Periaux, and K. Giannakoglou, eds. Optimization methods & tools formulticriteria/multidisciplinary design applications to aeronautics and turbomachinery. VKI lecture series 2004-07.1–40.

Suppapitnarm, A., et al., 2000. A simulated annealing algorithm for multiobjective optimization. Engineering Optimiza-tion, 33 (1), 59–85.

Zitzler, E., et al., 2003. Performance assessment of multiobjective optimizers: an analysis and review. IEEE Transactionson Evolutionary Computation, 7 (2), 117–132.

Zitzler, E. and Thiele, L., 1998. Multiobjective optimization using evolutionary algorithms – a comparative case study.In: W. Ebeling, et al., eds. Parallel problem solving from nature – PPSN V. Lecture notes in computer science Vol.2632/1998. Heidelberg: Springer-Verlag, 292–301.

Dow

nloa

ded

by [

The

Uni

vers

ity o

f B

ritis

h C

olum

bia]

at 1

7:39

09

Dec

embe

r 20

14