design optimization using hyper-reduced-order models

Struct Multidisc Optim (2015) 51:919–940DOI 10.1007/s00158-014-1183-y

RESEARCH PAPER

Design optimization using hyper-reduced-order models

David Amsallem ·Matthew Zahr ·Youngsoo Choi ·Charbel Farhat

Received: 12 November 2013 / Revised: 14 September 2014 / Accepted: 27 September 2014 / Published online: 1 November 2014© Springer-Verlag Berlin Heidelberg 2014

Abstract Solving large-scale PDE-constrained optimiza-tion problems presents computational challenges due to thelarge dimensional set of underlying equations that haveto be handled by the optimizer. Recently, projection-basednonlinear reduced-order models have been proposed to beused in place of high-dimensional models in a design opti-mization procedure. The dimensionality of the solutionspace is reduced using a reduced-order basis constructed byProper Orthogonal Decomposition. In the case of nonlin-ear equations, however, this is not sufficient to ensure thatthe cost associated with the optimization procedure doesnot scale with the high dimension. To achieve that goal,

A preliminary version of this work was presented at the WorldCongress on Structural and Multidisciplinary Optimization in2013.

D. Amsallem (�) · Y. Choi · C. FarhatDepartment of Aeronautics and Astronautics, Stanford University,Stanford, CA 94305, USAe-mail: [email protected]

Y. Choie-mail: [email protected]

M. Zahr · C. FarhatInstitute for Computational and Mathematical Engineering,Stanford University, Stanford, CA 94305, USA

M. Zahre-mail: [email protected]

C. FarhatDepartment of Mechanical Engineering, Stanford University,Stanford, CA 94305, USAe-mail: [email protected]

an additional reduction step, hyper-reduction is applied.Then, solving the resulting reduced set of equations onlyrequires a reduced dimensional domain and large speedupscan be achieved. In the case of design optimization, itis shown in this paper that an additional approximationof the objective function is required. This is achieved bythe construction of a surrogate objective using radial basisfunctions. The proposed method is illustrated with twoapplications: the shape optimization of a simplified noz-zle inlet model and the design optimization of a chemicalreaction.

Keywords PDE-constrained optimization · Surrogatemodeling · Parametric model reduction · Hyper-reduction ·Discrete empirical interpolation method · Shapeoptimization

1 Introduction

Designers of engineering systems have today access tohigh-fidelity physics-based computational codes that canprovide a detailed analysis for a particular design. Manyof these physics-based models rely on the discretizationof a set of partial differential equations (PDEs). Incor-porating these equations in the design optimization pro-cess as constraints that define the physical behavior ofthe system of interest has led to the development of thefield of PDE-constrained optimization (Gunzburger 2003;Biegler et al. 2003, 2007). In that context, enforcing thediscretized PDE constraint requires repetitive solutions ofthe PDE in a design optimization procedure. The com-putational burden associated with these repeated solutionscan be prohibitive as it dominates the computational costof the entire optimization procedure. Furthermore, in the

mailto:[email protected]




920 D. Amsallem et al.

case of Simultaneous Analysis and Design (SAND) (Haftka1985), the optimizer needs to handle both the design vari-able and the state variable associated with the PDE, aswell as the large PDE constraint. For large-scale PDE-basedmodels, this leads to many challenges including construct-ing appropriate pre-conditioners for solving the resultinglinear systems, developing globalization techniques andhandling the inexactness associated with large-scale linearsolvers (Biros and Ghattas 2005a, b). Another major lim-itation appears when the PDE is time-dependent, resultingin an optimization variable which dimension is the prod-uct of the number of variables in the large-scale systemtimes the number of time intervals considered in the timediscretization.

To alleviate the prohibitive cost associated with the PDEconstraint, surrogate-based optimization (Alexandrov et al.1998; Queipo et al. 2005; Forrester and Keane 2009) sub-stitutes the original design problem with an approximateproblem for which evaluating the objective and constraintsis inexpensive. In particular, physics-based approachesusing projection-based reduced-order models as surrogatesfor the large-scale high dimensional model (HDM) havebeen proposed recently, more specifically in the fields ofshape optimization (LeGresley and Alonso 2000; Weickumet al. 2009; Carlberg and Farhat 2010; Manzoni et al.2011; Xiao et al. 2012; Gogu and Passieux 2013; Zahret al. 2013; Zahr and Farhat 2014) and optimal control(Gunzburger 2003; Homberg and Volkwein 2003; Fahland Sachs 2003; Dihlmann and Haasdonk 2013; Yue andMeerbergen 2013). These approaches rely on the construc-tion of reduced dimensional state equations using solu-tions of the HDM for particular designs. These designscan be chosen a priori using for instance a designof experiments or progressively during the optimizationprocess, whenever the reduced-order model (ROM) isfound to be inaccurate for the current design (Fahl andSachs 2003; Zahr et al. 2013; Yue and Meerbergen 2013;Zahr and Farhat 2014). The number of degrees of free-dom (dof) associated with the ROM is in general ordersof magnitude smaller than the number of dofs in theHDM.

When the HDM of interest is nonlinear, however, reduc-ing the dimension of the equations does not necessarilyreduce the complexity associated with solving the equa-tions, as evaluating the underlying high dimensional non-linear function is necessary prior to its reduction. Thus,another level of approximation, hyper-reduction, is requiredto achieve important speedups (Everson and Sirovich 1995;Ryckelynck 2005; Astrid et al. 2008; Chaturantabut andSorensen 2010; Carlberg et al. 2011; Amsallem et al. 2012

Carlberg et al. 2013). In the context of hyper-reduction,most of the original large-scale computational domain isdiscarded and only a reduced portion of the state vec-tor is retained. This is called the reduced mesh in thecontext of computational fluid dynamics (CFD) (Carlberget al. 2013). In the context of optimization, however, thisis problematic as the objective function and additional con-straints may depend on degrees of freedom pertaining tothe discarded mesh. If the number of such dofs is small,these can be added to the reduced mesh without incurringexcessive additional cost. If the objective function requiresstoring and evaluating a large number of entries of thestate vector, however, the associated additional computa-tional burden can lead to a drastic reduction in speedup.Inverse problems for aerodynamic applications (Jameson1988) is a popular case for which a large number ofentries of the state vectors is required to evaluate theobjective. In this work, an approach based on the con-struction of a surrogate objective function is introducedto address this case. The cost associated with evaluat-ing this surrogate and its sensitivities becomes inexpensiveas it only scales with the dimensionality of the reduced-order model. A cross-validation procedure is applied toenable the construction of a robust surrogate function acrossparameters.

The contributions of this paper are therefore to iden-tify and propose a comprehensive procedure to addresscomputational bottlenecks arising in the use of projection-based model reduction in the context of optimizationof a system defined by a parameterized set of non-linear partial differential equations. The procedure isbased on two distinct approximation techniques devel-oped in recent years, namely hyper-reduction techniquesto reduce the complexity associated with solving the non-linear reduced-order model as well as a radial basis func-tion surrogate modeling to reduce the complexity associ-ated with evaluating the objective function and additionalconstraints.

This paper is organized as follows. The optimizationproblem of interest is presented in Section 2. Reduced-order modeling using hyper-reduction is presented inSection 3 and the development of an optimization frame-work using hyper-reduced models is proposed in Section 4.The capacity of the proposed framework to speeding updesign optimization procedures is illustrated with two aca-demic examples in Section 5 with the design of theshape of a simplified nozzle model and in Section 6with the design optimization of the reaction of a pre-mixed H2-air flame. Finally, conclusions are offered inSection 7.

Design optimization using hyper-reduced-order models 921

2 Problem of interest

2.1 Design optimization

The problem of interest is that of design optimization thatcan be abstracted as

(1)

p ∈ D ⊂ RNp is the vector of Np design variables defin-

ing the system to be optimized and D denotes the designspace. w ∈ R

Nw denotes the vector of Nw state variables.f (·, ·) ∈ R is the objective function and c(·, ·) ∈ R

Nw isthe nonlinear state equation constraint resulting from thediscretization of a steady-state PDE. Finally, k(·, ·) ∈ R

Ni

defines a small number (Ni � Nw) of additional constraintsassociated with the problem.1

Remark 1 In the optimization problem (1), variables han-dled by the optimizer are both the parameter vector p and thevector of state variables w. This problem therefore pertainsto the framework of Simultaneous Analysis and Design(SAND) (Haftka 1985).

2.2 Approach

The first-order optimality conditions for Problem (1) are

c(w,p) = 0,

μ ≥ 0,

k(w,p)T μ = 0,

k(w,p) ≤ 0, (2)

∂f

∂w+

(∂c∂w

)T

λ +(

∂k∂w

)T

μ = 0Nw,

∂f

∂p+

(∂c∂p

)T

λ +(

∂k∂p

)T

μ = 0Np .

There are several possible approaches of solving Problem(1): trust-region methods, Sequential Quadratic Program-ming (SQP) methods, and interior point methods (Nocedaland Wright 2006). Among these possibilities, an active setSQP method is used to solve Problem (1). A brief descrip-tion of the method is provided below. For more details,

1For simplicity, additional equality constraints are here embedded ink(·, ·) ≤ 0 as an equality constraint can always be written as twoinequality constraints

see Gill et al. (1981). A Lagrangian associated with (1) isdefined as

L(x, λ, μ) = f (x) + λT c(x) + μT k(x), (3)

where λ ∈ RNw and μ ∈ R

Ni are the vectors of Lagrangemultipliers corresponding to equality and inequality con-straints, respectively. The state variables w and the designvariables p are recasted as a single vector

x =[wp

]∈ R

Nx , (4)

with Nx = Nw + Np, The following quadratic model ofLagrangian (3) subjected to linearized constraints is solvedto obtain a search direction q:

minq∈RNx qT g + 12q

T Hqs.t. Acq + c = 0

Akq + k ≤ 0,(5)

where Ac ∈ RNw×Nx is the Jacobian of the state equations c

with respect to x

Ac(x) =[

∂c∂w

(x)∂c∂p

(x)]

= ∂c∂x

(x) (6)

and Ak ∈ RNi×Nx is the Jacobian of the inequality con-

straints k with respect to x

Ak(x) =[

∂k∂w

(x)∂k∂p

(x)]

= ∂k∂x

(x). (7)

H ∈ RNx×Nx denotes the Hessian of the Lagrangian L with

respect to x

H(x, λ, μ) = ∂2L∂x2

(x, λ). (8)

In practice, the Hessian H is replaced by a quasi-NewtonapproximationB ∈ R

Nx×Nx based on the Broyden-Fletcher-Goldfarb-Shanno (BFGS) update (Nocedal and Wright2006) by including only the active biding constraints in aworking set. A line search procedure is then conducted todetermine a step length. If there are binding constraints withnegative multipliers, the constraints are removed fromwork-ing set. If there are any new binding constraints, those con-straints are added to working set. The procedure is repeateduntil all the multipliers corresponding to constraints in theworking set are positive.

Once Subproblem (5) is solved and a search direction qis obtained, a merit function m(x, λ, μ) is used in the linesearch method to control the size of the step. If the resultingupdated vector x leads to a constraint violation c(x) �= 0 thatis too large, the convergence of the Newton’s method maybe affected. To remediate this issue, whenever the norm ofthe constraint violation ‖c(x)‖2 is above a certain threshold,


the state equation solver is called to update w with p heldconstant. To keep the paper self-contained, the resultingprocedure is detailed in Algorithm 1.

3 Hyper-reduced-order models

3.1 Model reduction

The constraint c(w,p) = 0 originates from the discretiza-tion of a partial differential equation. As such, the numberNw of equations in that constraint scales proportionally tothe dimension of the chosen discretization of the spatialdomain. This implies that Nw can be sufficiently large tomake optimization impractical. This large dimension incursexpensive computations for the solution of the discretizedPDE as well as the solution of the PDE-constrained opti-mization using Algorithm 1. The number Np of parametersis assumed to be moderate here, and as such, a reduction ofthe number of parameters is not necessary.

3.1.1 State approximation

Techniques pertaining to projection-based model reduc-tion reduce the cost associated with solving the discretizedPDE by searching for a solution w in a lower dimen-sional affine subspace of R

Nw . More specifically, this

affine subspace is uniquely determined by an intersectpoint w0 ∈ R

Nw and a reduced-order basis (ROB) V ∈R

Nw×nw having full column-rank. The solution w(p) tothe state equation constraint c(w(p),p) = 0 is thenapproximated as

w(p) ≈ w0 + Vwr (p). (9)

In the case of parametric model order reduction (pMOR),there are several possible strategies for constructing aROB V:

1. The ROB V can be held constant across parameterswhich corresponds to a global ROB. This is the case inthe remainder of the present work.

2. The ROB can be made parameter dependent V = V(p).This is the case of a local ROB (Amsallem and Farhat2008, 2011; Amsallem et al. 2012).

In both cases, substituting the approximation (9) into thestate equation leads to an overdetermined system of Nw

equations in terms of nw unknowns

c(w0 + Vwr (p),p) ≈ 0. (10)

3.1.2 Galerkin projection

In the case of Galerkin projection, the residual vector r =c(w0 + Vwr (p),p) is constrained to be orthogonal to theROB V, leading to a system of nw equations in terms of nw

unknowns.

VT c(w0 + Vwr (p),p) = 0. (11)

Galerkin projection is used in the remainder of this work.

Remark An alternative way to achieve a unique solutionin (10) is to solve for wr (p) that minimizes the ‖‖2-norm of the residual vector r (LeGresley and Alonso 2000;Bui-Thanh et al. 2008; Carlberg et al. 2011). This requiresthe solution of the following nonlinear least-squaresproblem

minwr∈Rnw

‖c(w0 + Vwr (p),p)‖2. (12)

This approach is recommended when the Jacobian∂c∂w

is

non-symmetric.

3.1.3 Data compression

The ROB V ∈ RNw×nw is here built using Proper

Orthogonal Decomposition (POD) by the method of snap-shots (Sirovich 1987). The approach proceeds by firstcomputing solutions w(pi ), i = 1, · · · , Ns of the dis-cretized PDE for Ns distinct values of the design variables


pi , i = 1, · · · , Ns . These snapshots are then stored in asnapshot matrix

W = [w(p1) · · · w(pNs )

] − w01T . (13)

The term −w01T is added to ensure that an optimalapproximation of the form (9) is eventually computed. 1 is avector of ones of dimension Ns . w0 can be chosen to be themean of the snapshots or an initial guess for the solution ofc(w,p) = 0.

A Singular Value Decomposition (SVD) of W is com-puted and V is constructed by retaining the first nw leftsingular vectors in the SVD. The procedure is summarizedin Algorithm 2. By construction, the ROB satisfies VT V =Inw .

It can be shown by the Eckert-Young-Minsky theorem(Antoulas 2005) that such a ROB V minimizes the projec-tion error of the snapshot set. In other words, V is a solutionto the problem

minY∈RNw×nw ‖W − YYT W‖F

s.t. YT Y = Inw .(14)

Remark 1 is possible (Weickum et al. 2009; Carlberg andFarhat 2010; Hay et al. 2010) to enrich the snapshot set byadding in W additional columns containing the sensitivityinformation

∂w∂pj

(pi ), j = 1, · · · , Np, i = 1, · · · , Ns. (15)

Adding those sensitivities results in a snapshot matrixW ofdimension Ns(Np + 1).

Remark 2 In practice, the dimension nw of the ROB canbe determined in Step 4 of Algorithm 2 by monitoring thesnapshot projection relative error resulting from the use of aROB Yn of dimension 1 ≤ n ≤ Ns . This error can be easilycalculated using the singular values σi, i = 1, · · · , Ns

computed in Algorithm 2 as

e(n) = ‖W − YnYTn W‖F

‖W‖F

=(∑Ns

i=n+1 σ 2i∑Ns

i=1 σ 2i

) 12

. (16)

Choosing a threshold ε for the error, nw is chosen as theminimizer of

min1≤n≤Ns n

s.t. e(n) ≤ ε(17)

and the ROB obtained as V = Ynw .

3.2 Hyper-reduction

The nonlinear system of equations (11) is typically solvedusing an iterative procedure such as Newton’s method. Thisrequires evaluating for a given iterate w(k)

r the reducedconstraint

cr (w(k)r ,p) = VT c

(w0 + Vw(k)

r ,p)

. (18)

The cost associated with this evaluation scales withthe dimension Nw of the high-dimensional problem as itrequires the following steps:

1. Form the large dimensional vector w0 + Vw(k)r ∈ R

Nw

2. Evaluate the residual vector c(w0 + Vw(k)

r ,p)

∈ RNw

3. Pre-multiply c(w0 + Vw(k)

r ,p)

∈ RNw by VT ∈

Rnw×Nw .

To alleviate the prohibitive cost associated with each ofthese three steps, an additional level of approximation,hyper-reduction can be introduced (Everson and Sirovich1995; Ryckelynck 2005; Astrid et al. 2008; Chaturantabutand Sorensen 2010; Carlberg et al. 2011; Amsallem et al.2012; Carlberg et al. 2013).

In the context of hyper-reduction, the nonlinear residualc(·, ·) is approximated as a linear combination of pre-computed basis vectors stored in a ROB Vc ∈ R

Nw×nc .The ROB Vc is constructed by applying the POD procedureoutlined in Algorithm 2 to the snapshot matrix

c = [c(w(p1),p1) · · · c(w(pNs ),pNs )

] ∈ RNw×Ns . (19)

Vc is then used to approximate the state equation as

c(w0 + Vwr ,p) ≈ Vczr (wr ,p). (20)

The small-dimensional vector zr (wr ,p) ∈ Rnc is com-

puted using a two-step gappy POD procedure (Everson andSirovich 1995).

1. Evaluate a few rows c(w0 + Vwr ,p) ∈ R|E | of c(w0 +

Vwr ,p), E ⊂ {1, · · · , Nw} denoting the set of sampledrows with |E | ≥ nc.

2. Compute zr (wr ,p) by least-squares reconstruction asthe solution to

minzr∈Rnw

‖c(w0 + Vwr ,p) − Vczr‖2, (21)


Vc denoting the restriction of the rows of Vc to E . Thesolution to (21) is

zr = V†c c(w0 + Vwr ,p), (22)

where M† denotes the Moore-Penrose pseudo-inverse of agiven matrixM.

When the Jacobian∂c∂w

is sparse, evaluating c(wr ,p)

only requires evaluating and storing a subset E ′ ⊂ RNw of

entries of the state vector w = w0 + Vwr . The portion ofthe state vector corresponding to this subset is subsequentlydenoted as w = w0 + Vwr and as a result

c(w0 + Vwr ,p) = c(w0 + Vwr ,p). (23)

The subset of the mesh corresponding to w is called thereduced mesh. In practice, only the reduced mesh is used inthe solution of the hyper-reduced-order model (HROM) andsince |E ′| � Nw, this ensures that large speedups can beachieved (Carlberg et al. 2013).

Then, the state equation is approximated as

c(w0 + Vwr ,p) ≈ VcV†c c(w0 + Vwr ,p). (24)

Substituting (24) into (11), Galerkin projection then leadsto the system of nw equations in terms of nw unknowns

chr (wr ,p) = VT VcV†c c(w0 + Vwr ,p) = 0. (25)

Z=(VT VcV

†c

)T ∈ R|E |×nw is of small dimension and can

be pre-computed and stored ahead of the HROM computa-tion. The hyper-reduced state equation is

chr (wr ,p) = ZT c(w0 + Vwr ,p) = 0. (26)

This is a reduced set of nonlinear equations that can besolved iteratively using Newton’s method.

Remark 1 Often, the residual c(w,p) can be explicitlywritten as the sum of a nonlinear term cNL(w,p) and asum cL(w,p) of parameter dependent terms that are affinein w

cL(w,p) =MA∑i=1

θ(i)A (p)A(i)

L w +Mb∑i=1

θ(i)b (p)b(i)

L , (27)

{θ

(i)A (·)

}MA

i=1and

{θ

(i)b (·)

}Mb

i=1are arbitrary functions of the

parameter vector p. In that case, the hyper-reduction tech-nique is only applied to the nonlinear term cNL(w,p). The

projection cL,r (wr ,p) = VT cL (w0 + Vwr ,p) of the lin-ear term can indeed be efficiently directly evaluated for anyvalue of wr and p with a cost that does not scale with Nw as

cL,r (wr ,p) = ∑MA

i=1 θ(i)(p)(A(i)L,rwr + w(i)

0,r

)

+ ∑Mb

i=1 θ(i)b (p)b(i)

L,r ,(28)

provided the following quantities are pre-computed offline:

w(i)0,r = VT A(i)

L w0, i = 1, · · · , MA,

A(i)L,r = VT A(i)

L V, i = 1, · · · , MA,

b(i)L,r = VT b(i)

L , i = 1, · · · , Mb.

(29)

Remark 2 When the number of sampled rows |E | is equalto the number of vectors nc, this hyper-reduction approachis identical to the DEIM algorithm of Chaturantabut andSorensen (2010).

Remark 3 The rows E are in practice chosen by applying thegreedy algorithm outlined by Chaturantabut and Sorensen(2010).

3.3 Design of experiments

In order to build a ROB V and an associated hyper-reducedorder model that is robust in the parameter space and cantherefore be used in the optimization process, two distinctapproaches are possible.

1. The snapshot locations {pi}Ns

i=1 can be determined apriori by sampling the design space D using spacefilling techniques such as uniform sampling, latinhypercube sampling (LHS) or orthogonal arrays (Sackset al. 1989; Queipo et al. 2005). This is the procedureselected in the application of Section 5.

2. Alternatively, the snapshot location can be determinediteratively by a greedy procedure (Veroy and Patera2005; Bui-Thanh et al. 2008; Paul-Dubois-Taine andAmsallem 2014) based on an a posteriori indicatorof the error associated with the hyper-reduced-ordermodel. The following error indicator is used in thepresent work

I(wr ,p) = ‖c(w0 + Vwr ,p)‖2. (30)

This indicator is the norm of the HDM residual asso-ciated with the solution of the HROM from (20). Thecorresponding greedy procedure is reported in Algo-rithm 3. This procedure is selected in the application ofSection 6.


3. After an initial sampling, the basis V can be recon-structed at multiple times during the design optimiza-tion process, whenever the associated ROM is foundto be inaccurate or cannot be trusted anymore (Zahret al. 2013; Yue and Meerbergen 2013; Zahr andFarhat 2014). Similarly as in the greedy procedure, thisclass of methods requires an efficient error indicator toassess the accuracy of the ROM.

The first two approaches are chosen here as theylead to a design optimization procedure having a com-putational cost that does not scale with Nw. However,the third approach could be preferred in cases wherethe initial design of experiments does not lead to aROM that is accurate in the entire parameter domainD. This is likely the case in high-dimensional parameterdomains.

4 Optimization using hyper-reduced-order models

4.1 Problem of interest

The hyper-reduced state equation (26) is substituted inthe optimization problem (1) leading to the reduced-sizeproblem

(31)

where

fr(wr ,p) = f (w0 + Vwr ,p),

kr (wr ,p) = k(w0 + Vwr ,p), (32)

and

chr (wr ,p) = ZT c(w0 + Vwr ,p) (33)

with Z =(VcV

†c

)T

V.

4.2 Solution of the optimization problem

The Lagrangian associated with (31) is defined as

Lhr (xr , λr , μ) = f (wr ,p) + λTr chr (wr ,p) + μT kr (wr ,p)

(34)

where λr ∈ Rnw is the vector of Lagrange multipliers of

small dimension nw � Nw.The optimization problem (31) can then be solved iter-

atively using an active set SQP approach similarly as inSection 2.

Defining a reduced vector

xr =[wr

p

]∈ R

nx , (35)

where nx = nw + Np, the gradient of the objective functionwith respect to reduced variable xr is then

gr (xr ) = dfr

dxr

(xr ) =[

∂fr

∂wr

(xr )∂fr

∂p(xr )

]∈ R

nx (36)

and the Jacobian of the hyper-reduced constraints chr (xr ) =0 is

Ahr (xr ) = dchr

dxr

(xr ) =[

∂chr

∂wr

(xr )∂chr

∂p(xr )

]∈ R

nw×nx .

(37)

The Jacobian of the inequality constraints is

Akr (xr ) = dkr

dxr

(xr ) =[

∂kr

∂wr

(xr )∂kr

∂p(xr )

]∈ R

Ni×nx .

(38)

At each iteration k, this results in the solution of aquadratic program of reduced dimension

minqr∈Rnx

qTr g

(k)r + 1

2qTr B

(k)hr qr

s.t. Ahr(k)qr + c(k)

hr = 0

A(k)kr qr + k(k)

r ≤ 0,

(39)

where the vector qr is of dimension nx and Bhr denotesa quasi-Newton approximation of the Hessian of Lhr withrespect to xr . The corresponding iterative procedure is sum-marized in Algorithm 4. The reader can check that the


structure of Algorithm 4 is identical to the structure of itshigh-dimensional counterpart Algorithm 1.

4.3 Sensitivity analysis

In Algorithm 4, sensitivities associated with the hyper-reduced-order model are required to evaluate gr , Ahr , andAkr . Their elements can be computed as follows.

1. Sensitivity of the objective function with respect to theparameters

∂fr

∂p(wr ,p) = ∂f

∂p(w0 + Vwr ,p). (40)

2. Sensitivity of the objective function with respect to thereduced variables

∂fr

∂wr

(wr ,p) = ∂f

∂w(w0 + Vwr ,p)V. (41)

3. Sensitivity of the hyper-reduced state equation withrespect to the parameters

∂chr

∂p(wr ,p) = ZT ∂c

∂p(w0 + Vwr ,p). (42)

4. Sensitivity of the hyper-reduced state equation withrespect to the reduced variables

∂chr

∂wr

(wr ,p) = ZT ∂c∂w

(w0 + Vwr ,p)V. (43)

5. Sensitivity of the inequality constraints with respect tothe parameters

∂kr

∂p(wr ,p) = ∂k

∂p(w0 + Vwr ,p). (44)

6. Sensitivity of the inequality constraints with respect tothe reduced variables∂kr

∂wr

(wr ,p) = ∂k∂wr

(w0 + Vwr ,p). (45)

The cost associated with computing (42) and (43) scaleswith the size of the hyper-reduced mesh instead of Nw.A more detailed computational cost analysis is offered inSection 4.5.

4.4 Surrogate objective function

In the case of hyper-reduction, it is not possible to evaluatein general an output function of the solution as this functionmay involve entries of w that are not in the reduced mesh:only the subset w ≈ w0 + Vwr ∈ R

|E ′| of entries in thestate vector is available. Therefore, the objective functionf , inequality constraints k and their sensitivities cannot bedirectly evaluated.

In the case of a single analysis, output functions can beefficiently reconstructed in a post-processing step as out-lined in Carlberg et al. (2013). However, in the contextof optimization, outputs such as the objective function andinequality constraints need to be repeatedly evaluated in theoptimization loop. As a result, their evaluation cannot bepostponed to a post-processing stage. There are thereforetwo cases that call for distinct approaches: (1) when theobjective function f is a polynomial function of the statew and the parameter p and (2) when it is not a polyno-mial function of these variables. As shown below, the firstcase does not present any issue and can be handled eas-ily whereas the second case requires a specific approach.The two cases described below focus on the objective func-tion f , but the analysis can be generalized to the inequalityfunctions k by considering each of the Ni inequalitiesindividually.

4.4.1 Polynomial function f

Consider as an illustrative example the case of a least-squares objective function with regularization

f (w,p) = 1

2‖w − w∗‖22 + ρ

2‖p‖22. (46)

Its sensitivities are∂f

∂w(w,p) = w − w∗, ∂f

∂p(w,p) = ρp. (47)


f is polynomial in w and p as it can be developed as

f (w,p) = 1

2wT w − wT w∗ + 1

2‖w∗‖22 + ρ

2pT p. (48)

In the case of model reduction w ≈ w0 + Vwr and theexpansion of fr(wr ,p) = f (w0 + Vwr ,p) in terms of thereduced variable wr is

fr(wr ,p) = 12w

Tr wr − wT

r VT (w∗ − w0)

+ 12‖w∗ − w0‖22 + ρ

2pT p,

(49)

which is also a polynomial function of wr and p. Further-more, the terms VT (w∗ −w0) ∈ R

nw and 12‖w∗ −w0‖22 ∈ R

are small and can be pre-computed offline and stored. Thesensitivities are also polynomial in wr as

∂fr

∂wr

(wr ,p) = wr −VT (w∗−w0),∂fr

∂p(wr ,p) = ρp. (50)

This observation generalizes to the case of any polynomialfunction f of w and p and as such, no special treatmentis required as fr and its sensitivities can be evaluated in anumber of operations that scales only as a polynomial of Np

and nw.As the degree d of the polynomial function f (·, ·)

becomes higher, the number of terms that needs to be pre-

computed increases asO(n

d1w N

d2p

)with d1+d2 = d , which

renders this approach infeasible for large values of d .

4.4.2 Non-polynomial function f

In the case of a general function f of w and p, evaluatingfr(wr ,p) = f (w0 +Vwr ,p) requires first constructing thelarge dimensional vector w = w0 + Vwr ∈ R

Nw and thenevaluating f (w,p). In a hyper-reduction setting, however,w cannot be entirely reconstructed as only its trace w on thereduced mesh is available online. If the number of entriesin w that are required to evaluate f is small, these entriescan be added to w for a moderate additional online cost.When a large number of entries in w is required, however,adding these entries to w jeopardizes the large speedupsassociated with the use of a hyper-reduced-order model.This observation calls for an alternative approach.

Since fr(wr ,p) cannot be evaluated exactly, it is replacedby an approximation that can be cheaply evaluated. A gappyPOD approach cannot be applied to fr , however, as fr

is a scalar valued function and hyper-reduction applies tolarge-scale vector valued functions. A different approach isdeveloped here. It proceeds by fitting (in an interpolativefashion) a surrogate function fr (wr ,p) to fr(wr ,p). Theoptimization problem (31) without inequality constraintsbecomes

(51)

In this work, radial basis functions (RBFs) are usedto determine the surrogate function fr . Its expressionbecomes

fr (wr ,p) =Nf∑i=1

aiφεi (wr ,p), (52)

where {φεi (·, ·)}

Nf

i=1 is a set of RBFs of the variables (wr ,p),

ε > 0 is an appropriate scaling factor and {ai}Nf

i=1 areappropriate weights.

When the RBFs are smooth, the sensitivities of thesurrogate function fr can be easily computed as

∂fr

∂wr

(wr ,p) =Nf∑i=1

ai

∂φεi

∂wr

(wr ,p) (53)

and

∂fr

∂p(wr ,p) =

Nf∑i=1

ai

∂φεi

∂p(wr ,p). (54)

In practice, fr is computed by sampling f and its deriva-

tives for many combinations{xrj

}Ns

j=1 ={[

wrj

pj

]}Ns

j=1of

the parameters and reduced variables. For simplicity, in thispaper, the sampling for determining the surrogate objectivefunction coincides with the state snapshot sampling for con-structing the ROB V. In that case, wrj is computed as theorthogonal projection of the snapshot w(pj ) onto the affinesubspace w0 + range(V):

wrj = w0 + VVT(w

(pj

) − w0). (55)

Several choices for the RBFs {φεi (·, ·)}

Nf

i=1 are possible.Two cases are considered here.

1. Lagrange interpolation:The surrogate fr matches the objective function fr atthe sample points

{xrj

}Ns

j=1:

fr

(xrj

) = fr

(xrj

), j = 1, · · · , Ns. (56)

In that case, Nf = Ns and the basis functions arechosen as

φεi (wr ,p) = φε(‖xr − xrj‖2), i = 1, · · · , Ns (57)

where φε(·) is a RBF.2 In the present work, GaussianRBFs are considered:

φε(r) = exp

(− r2

ε2

), r > 0, (58)

but a large choice of RBFs exists (Fasshauer 2007).The resulting Lagrange RBF-based surrogate will bedenoted as f L

r in the remainder of this paper.2. Hermite interpolation:


In that case, the surrogate fr satisfies (56) and alsomatches the first order derivative of fr at the samplepoints:

∂fr

∂xi

(xrj

) = ∂fr

∂xi

(xrj

),

i = 1 · · · , nx, j = 1, · · · , Ns,(59)

xi denoting the i-th entry in xr .In that case, Nf = Ns(nx + 1) and the basis functionsare

φεi (wr ,p)=

⎧⎪⎨⎪⎩

φε (‖xr − xri‖2) if 1 ≤ i ≤ Ns

∂φε

∂xj

(‖xr − xrl‖2) if i = jNs + l >Ns .

(60)

The Hermite RBF-based surrogate function will bedenoted as f H

r .

Determining the weights {ai}Nf

i=1 entails the solution of adense linear system of dimensionNf . Hence, Hermite inter-polation leads to the solution of a much larger system.However, usually, sensitivity information for the objectivefunction at the sample points can be inexpensively com-puted and it will be shown in Section 5 that includingthis information leads to a much more accurate surrogateobjective function at a very moderate additional cost. Thisobservation is in accordance with the work of Alexandrovet al. (1998) where it is proved that, in a trust-region frame-work, first-order consistency between a surrogate and theoriginal objective function is sufficient for convergence ofan optimization procedure using the surrogate objective tothe same local minimum as optimization using the originalfunction fr .

In order to build a robust surrogate function fr a cross-validation procedure is used to select ε. It is detailed inAppendix.

To guarantee that the surrogate fr is accurate, for thevalue of ε selected by the cross-validation, the reducedobjective function fr and its surrogate fr are evalu-ated for a large number NMC of arbitrary combinations{(

w(i)r ,p(i)

)}NMC

i=1and the associated relative errors calcu-

lated as

e(i)fr

=∣∣∣fr

(w(i)

r ,p(i))

− fr

(w(i)

r ,p(i))∣∣∣∣∣∣fr

(w(i)

r ,p(i))∣∣∣ , i = 1 · · · , NMC

(61)

2A weighted norm can be involved in place of the Euclidian normwhen the entries in wr and p are of different scales

AMonte-Carlo estimation of the average relative error asso-ciated with the surrogate model fr is then computed as

efr = 1

NMC

NMC∑i=1

e(i)fr

. (62)

In practice, at the stage of the surrogate model construc-tion, evaluating the reduced objective function frNMC timesis not cost prohibitive as each evaluation does not requiresolving a HDM but only evaluating the objective. However,because a given arbitrary combination (wr ,p) does not nec-essarily satisfy the hyper-reduced equation chr (wr ,p) =0 certain combinations may lead to reconstructed statesw0 + Vwr that may lead to unphysical steps in the objec-tive function evaluation, such as a negative pressures for afluid system for instance. To avoid such cases as much aspossible, wr is restricted to taking values in the convex hullof the values in the training set

{wrj

}Ns

j=1. If an unphysicalstate is still obtained while enforcing those bounds, the cor-responding sample is simply rejected from the Monte-Carloprocedure.

If efr meets a desired accuracy requirement, the sur-rogate model is accepted, otherwise an additional samplevalue for the parameter vector pNs+1 is chosen and thecorresponding HDM solved for w(pNs+1). The surrogatemodel is subsequently updated with the corresponding addi-tional information on fr .3

4.5 Computational complexity analysis

The approximate operation counts associated with com-puting the surrogate fr and its sensitivities is reported inTable 1. These operations counts depend on αφ and βφ ,the number of floating point operations required to eval-uate φε and its sensitivity with respect to r , respectively.These operations counts scale linearly with the number Nf

of radial basis functions considered in the approximation ofthe objective function.

The operation counts associated with computing thehyper-reduced state equations chr and its sensitivities arealso reported in the same table. In this table, ωc denotesthe average number of floating point operations requiredto compute one entry of c and γc and μc denote the aver-age number of operations required to compute one row of∂c∂w and ∂c

∂p , respectively. These number of operations areindependent of Nw when the state equation originates fromthe direct discretization of a PDE. ξc denotes the averagenumber of non-zero entries per row in ∂c

∂w .The operation counts associated with computing each of

the Ni individual surrogate inequality functions in kr based

3The ROB V can be also updated by including the additional informa-tion obtained at pNs+1


Table 1 Computational complexity for computing fr , chr , kr andtheir sensitivities

Evaluation Approximate operation count

fr

(αφ + 3

(nw + Np

))Nf + 2Nf

∂fr

∂wr

(βφ + 2

)nwNf + 2Nf

∂fr

∂p

(βφ + 2

)NpNf + 2Nf

chr ωc|E | + 2nw|E |∂chr

∂wr

γc|E | + 2ξcnw|E | + 2n2w|E |∂chr

∂pμc|E | + 2nwNp|E |

kr

((αφ + 3

(nw + Np

))Nk + 2Nk

)Ni

∂kr

∂wr

((βφ + 2

)nwNk + 2Nk

)Ni

∂kr

∂p

((βφ + 2

)NpNk + 2Nk

)Ni

on Nk RBFs are similar to the ones for fr and reported inTable 1.4

The online computational cost associated with solvingthe hyper-reduced optimization problem (51) is thereforeindependent of the dimension Nw of the underlying large-scale system of state equations.

5 Application to a rocket nozzle shape design

The PDE-constrained optimization problem associated withthis example is that of the inverse shape optimization of anacademic nozzle inlet. For that purpose a solution to theunderlying PDE is computed numerically for a given targetshape parameter p . This solution, denoted as w = w(p )

becomes the target solution. The objective is then to deter-mine the parameter p that leads to g(w(p)) = g(w ), whereg(·) is a function of the solution.

The nonlinear steady PDE of interest is Young et al.(2003)

∂

∂x(A(x)ρ(u)u) = 0, x ∈ [0, L], (63)

L denoting the length of the nozzle and A(x) denotes thenozzle local area at x ∈ [0, L]. u denotes the velocityand is computed as the gradient of a scalar potential φ as

4For simplicity, in this complexity analysis, it is assumed that the sameclass of RBFs φε can be used to interpolate the objective function andeach of the Ni inequality constraints in kr .

u(x) = ∂φ

∂x(x). The density ρ associated with u is computed

by the isentropic formula as

ρ(u) =(1 + γ + 1

2(1 − u2)

) 1γ−1

, (64)

where γ = 1.4 is the specific heat ratio. The pressure p can

be computed as p = ργ

γ, the speed of sound c =

(γpρ

)0.5and theMach number asM = u

c. Note that all flow variables

are non-dimensionalized.Equation (63) is discretized by finite difference approxi-

mation using mass flux biasing (Young et al. 2003), result-ing in a state vector w of dimension Nw = 2046. A veryfine discretization in space is chosen to illustrate the poten-tial speedup associated with the hyper-reduction approach.Each entry in the state vector corresponds to the potentialvalue at a point of the computational mesh. The resultingset of nonlinear equations is solved iteratively by Newton’smethod.

The function g(·) is here the vector of local Mach numberin the nozzlem(·) ∈ R

Nw . The objective function associatedwith the PDE-constrained problem is therefore

f (w) = 1

2‖m(w) − m(w )‖22. (65)

f is a non-polynomial function of w and, as a result, itsreduced-order counterpart fr will be approximated usingthe approach developed in Section 4.4.

In a first set of experiments, the nozzle shape is parame-terized as a quadratic polynomial defined by a two dimen-sional parameter vector p = [p1, p2]T : p1 defines the throatarea and p2 the throat location, as depicted in Fig. 1. Thenozzle area is then parameterized as A(x) = 0.6(x −p2)

2+p1.

The two parameters are bounded as 0.48 ≤ p1 ≤ 0.7 and0.8 ≤ p2 ≤ 1.1. A target configuration p = [0.52, 1] isthen chosen as well as an initial guess p(0) = [0.7, 1.1]T forthe inverse shape optimization problem.

Nine sample points are first randomly chosen in theparameter space D = [0.48, 0.7] × [0.8, 1.1] using LHS.

Fig. 1 Two dimensional parameterization of the nozzle geometry


For each of these sample points pj , j = 1, · · · , 9, the high-dimensional model is solved and the steady-state solutionw(pj ) of the PDE stored as a snapshot. Hence, Ns = 9snapshots are collected. A ROB V of dimension nw = 5is then constructed by SVD. It corresponds to a projectionerror e(5) = 0.018%, as defined by (16).

A hyper-reduced order model of dimension nc = 11is then constructed by applying the procedure outlined inSection 3. |E | = 11 rows are sampled from the originalresidual c(·, ·) of dimension Nw = 2046.

The Lagrange and Hermite RBF-based approaches out-lined in Section 4.4 are first compared by constructingtwo surrogate objective functions f L

r (wr ) and f Hr (wr ),

respectively. These interpolation-based functions are con-structed by first sampling fr at the Ns = 9 aforementionedsampled points and then applying the cross-validation pro-cedure outlined in Appendix. The two surrogate functionsare represented in Figs. 2 and 3 as a function of the firsttwo reduced variables wr1 and wr2. The original objec-tive function is depicted in Fig. 4. The reader can observethat f H

r (wr ) approximates fr much better than f Lr (wr ).

In particular, the minima of f Hr (wr ) and fr are very

close, which is not the case for f Lr (wr ). Quantitatively, the

Monte-Carlo estimators of the average relative error definedin (62) are respectively eL

fr= 12.6% for the Lagrange

RBF interpolation and eHfr

= 0.32% for the Hermite RBFinterpolation.

f Lr (wr ) and f H

r (wr ) are then successively used as surro-gates for fr in the shape optimization procedure. Matlab’sfmincon routine with an SQP solver is used as the opti-mization software. All the simulations in this work areperformed on a Mac Book Pro 2.9 GHz Intel Core i7,8GB 1600 MHz DDR3. The optimal shapes obtained byapplying each of the two procedures are then reportedin Fig. 5 together with the corresponding Mach num-ber distribution in the nozzle in Fig. 6. In the Lagrange

Fig. 2 f Lr as a function of wr1 and wr2. The samples are shown as

black squares

Fig. 3 f Hr as a function of wr1 and wr2. The samples are shown as

black squares

interpolation case, the optimal shape parameter obtainedis pL = [0.5048, 1.0065]T , which corresponds to a rela-tive error for A(x) equal to 1.1%, while for the Hermiteinterpolation case, it is pH = [0.5251, 0.998]T , whichlead to 0.8% relative error for A(x). One can observe thatthe constructed HROM together with f L

r leads to an opti-mized shape that is reasonably close to the target shapewhile the HROM together with f H

r very accurately pre-dicts the optimal nozzle shape. This illustrates the advantageof adding sensitivity information for building the surrogateobjective.

The effect of cross-validation (CV) on the accuracy of thesurrogate objective is then studied by varying the number oflatin hypercube samples Ns from 5 to 15. For each value ofthe sample size, 10 latin hypercube samples are randomlyselected and for each of those samplings, a HROM is con-structed and four surrogate objectives are built as follows: 1)using Lagrange RBFs without CV, 2) using Hermite RBFswithout CV, 3) using Lagrange RBFs with CV and 4) using

Fig. 4 fr as a function of wr1 and wr2. The samples are shown asblack squares


Fig. 5 Comparison of the optimized nozzle shapes obtained using aHROM with Lagrange and Hermite objective surrogates as well as thetarget optimal shape

Hermite RBFs with CV. When CV is not used to determinethe parameter ε, it is chosen a priori as

ε = dmin+dmax2

= minxri �=xrj ‖xri−xrj ‖2+maxxri �=xrj ‖xri−xrj ‖22 .

(66)

For each of those models, the shape optimization proce-dure is run and the corresponding relative errors determined.Figure 7 presents the relative errors averaged over the 10random samplings. The reader can observe, that, whenNs <

10, Lagrange surrogate functions lead to incorrect shapeson average, regardless of whether cross-validation is usedor not. On the other hand, Hermite interpolation-based sur-rogate functions lead to correct optimal shapes even for a

Fig. 6 Comparison of the Mach number distributions obtained usinga HROM with Lagrange and Hermite objective surrogates as well asthe target Mach distribution

Fig. 7 Comparison of the errors associated with the optimal shapesreturned using each of the four surrogate objective constructionprocedures

small number of samples Ns . Furthermore, for Hermite sur-rogate objectives, cross-validation leads to optimal shapesthat are even more accurate on average.

Finally, the CPU times associated with applying the opti-mization procedure using the HROM are compared to thatof using the HDM in Table 2. More specifically, the caseof Ns = 10 sample HDM solutions is considered and theoffline and online CPU times are compared for each ofthe four HROM procedures outlined above. The reader canobserve that very large online speedups, close to 250 areassociated with the HROMs when compared to the HDM.Speedups of 8 are still observed when adding the offline(sampling) CPU times to the online timings in the case ofHROMs. Furthermore, the cross-validation step does notadd a significant offline time in the procedure. Similarly,Hermite interpolation, which leads to the best surrogateobjectives, does not lead to a significant online computa-tional burden when compared to Lagrange interpolation.

In a second set of computer experiments, a five dimen-sional parameterization of the nozzle shape is consideredusing cubic splines. The five parameters correspond to thelocation of five equi-spaced points of the nozzle, as depictedin Fig. 8.

Table 2 CPU timings associated with each optimization procedure inthe case of the two dimensional parameterization

Model HDM HROM HROM HROM HROM

RBF type - Hermite Lagrange Hermite Lagrange

Cross-validation - no no yes yes

Ns - 10 10 10 10

Offline CPU time - 5.95 s 5.76 s 6.33 s 5.97 s

Online CPU time 47.8 s 0.24 s 0.18 s 0.21 s 0.19 s

Total CPU time 47.8 s 6.19 s 5.94 s 6.54 s 6.13 s


Fig. 8 Five dimensional parameterization of the nozzle geometry

Fifteen sampling points are randomly sampled usingLHS. The HDM is solved for each of those 15 points and 15steady-state solutions stored in a snapshot matrix as

W = [w(p1), · · · ,w(p15)]. (67)

A ROB of dimension nw = 8 is constructed by SVD. ThisROB corresponds to a projection error e(8) = 0.06%. Next,a HROM of dimension nc = 20 is constructed. This HROMis obtained by sampling |E | = 20 rows of c(·, ·).

The numerical experiments using the two-dimensionalshape parameterization case have shown that HermiteRBF surrogate objectives provide more accurate optimizedshapes for a very moderate additional cost. For that reason,a Hermite RBF-based surrogate objective is built using the15 samples determined in the design of experiments. Cross-validation is then applied to select an optimal parameterε . The Monte-Carlo estimator of the average relative errordefined in (62) is then eH

fr= 0.61%.

The resulting optimized shape is depicted in Fig. 9 andthe corresponding Mach number distribution in Fig. 10.The optimized nozzle shape follows closely the target shape:it corresponds to a relative error for A(x) equal to 2.1%.This is confirmed by comparing the original objective func-tion to the surrogate counterpart in Figs. 11 and 12: eventhough the samples are not located in the vicinity of thefunction minimum, the surrogate objective is very simi-lar to its original objective function. Finally, CPU timings

Fig. 9 Comparison of the optimized nozzle shapes obtained using aHROM with Hermite objective surrogates as well as the target optimalshape

Fig. 10 Comparison of the Mach number distributions obtained usinga HROM with Hermite objective surrogates as well as the target Machdistribution

are reported in Table 3. A speedup of 16 is observedfor the online optimization procedure, leading, after theoffline HROM construction cost is included, to an overallspeedup of 8.

6 Application to the optimization of a chemical reaction

The second application is that of the optimization of thereaction of a premixed H2-air flame in a two-dimensionalmodel. The reaction, described in Buffoni and Willcox(2010), is 2H2+O2 → 2H2O and is modeled by the follow-ing nonlinear steady advection-diffusion-reaction equation

U · ∇W − κ�W = S(W), X ∈ [0, Lx] × [0, Ly], (68)

where the state vector

W(x) = [T (X ), YH2(X ), YO2(X ), YH2O(X )]T ∈ R4 (69)

contains the temperature T and the mass fraction Yi of eachspecie i ∈ {H2, O2, H2O}. Lx and Ly are the length and thewidth of the domain, respectively. The nonlinear reactionsource term

Fig. 11 fr as a function of wr1 and wr2. The samples are shown asblack squares


Fig. 12 f Hr as a function of wr1 and wr2. The samples are shown as

black squares

S(W) = [ST (W),SH2(W),SO2(W),SH2O(W)] (70)

is of Arrhenius type and

Si (W) = −νiWi

ρ

(ρYH2WH2

)νH2(

ρYO2WO2

)νO2A exp

(− ERT

)i ∈ {H2, O2, H2O}

ST (W) = QSH2O(W)

(71)

with the stoichiometric coefficients νH2 = 2, νO2 = 1 andνH2O = −2. The molecular weights of the three speciesare WH2 = 2.016 g · mol−1, WO2 = 31.9 g · mol−1 andWH2O = 18 g · mol−1. The density of the mixture is ρ =1.39 × 10−3 g · cm−3. The universal gas constant is R =8.314 J · mol−1 · K−1 and the heat of the reaction is Q =9800 K. The diffusivity is κ = 2 cm2 · s−1.

The advection speed U (which is parallel to the x-axis),the pre-exponential factor A and the activation energy E

will be allowed to vary in the following study. Furthermore,the computational domain � = [0, Lx] × [0, Ly] is param-eterized by (Lx, Ly) as well as the length Ly,middle of theportion of the left boundary onto which a Dirichlet bound-ary condition T (X ) = Tleft is enforced. Everywhere else onthe left boundary, T (X ) = 300 K. Homogeneous Neumannboundary conditions are enforced at all other three bound-aries of the computational domain. The problem of interestis therefore defined by a vector of seven parameters

p = [A, E,U , Lx, Ly, Ly,middle, Tleft]T . (72)

Table 3 CPU timings associated with each optimization procedure inthe case of the five dimensional parameterization

Model HDM HROM

Offline CPU time - 5.09 s

Online CPU time 78.8 s 4.87 s

Total CPU time 78.8 s 9.96 s

Fig. 13 Parameterization of the reactive flow problem

Figure 13 summarizes the parameterization of the prob-lem. All seven parameters pi i ∈ {1, · · · , 7} are bounded aspmin

i ≤ pi ≤ pmaxi . The bounds considered in this work are

provided in Table 4.The PDE is discretized in space by the finite differences

method using a cartesian mesh of Nx × Ny nodes. Twomeshes are considered in this work: a coarse mesh for which(Nx, Ny) = (16, 16) and a fine mesh for which (Nx, Ny) =(76, 76). As a result, the state vector w has Nw = 960 dofsin the coarse mesh and Nw = 22, 800 dofs in the fine mesh.An upwinding scheme is used to approximate the advectionterm. The resulting set of nonlinear equations is then solvedby the Newton-Raphson method.

The optimization problem of interest at the continuouslevel is

maxW,p

∫�(p)

YH2O(X )dxdy

s.t. U(p) · ∇W − κ�W = S(W,p), X ∈ �(p)

maxX∈�(p),p

T (X ) ≤ Tmax. (73)

After discretization in space, the problem becomes of theform (1) with a linear objective

f (w,p) = a(p)T w + b(p) (74)

and a nonlinear constraint of the form

k(w,p) = max (Dw) − Tmax, (75)

where the matrix D restricts the state vector to its tempera-ture entries only.

Since the maximum operator is non-differentiable, it isapproximated by the soft-maximum defined for a vector y ∈R

N as

smax(y) =∑N

i=1 yieαyi∑N

i=1 eαyi

(76)

Table 4 Bounds for each of the seven parameters in the reactive flowapplication

pi A E U Lx Ly Ly,middle Tleft

unit - J · mol−1 m · s−1 m m m K

pmini 0.5 2 × 103 0.4 1.5 × 10−2 8 × 10−3 2 × 10−3 900

pmaxi 50 6 × 103 0.6 2.5 × 10−2 10−2 4 × 10−3 1000


with α = 0.1. This operator is differentiable and thecoefficient α is in practice chosen by trial-and-error sothat smax approximates the max operator very well for therange of temperature values considered in this section. Thenonlinear constraint is then

k(w,p) = smax (Dw) − Tmax. (77)

NTmax = 9 different values for Tmax are considered:

Tmax ∈{1000, 1250, 1500, 1750, 2000, 2250, 2500, 2750, 3000}K.

(78)

Each optimization problem is then solved by an interiorpoint method usingMatlab’s fmincon function together witha multi-start strategy to avoid local maxima. Nstart = 10

Fig. 15 Maximum temperature computed on the fine mesh for eachoptimal design

instances of the problem are created with different initialguesses. As a result, there is a total of NTmax × Nstart = 90optimization problems to be solved.

Fig. 14 Comparison of the optimal designs found for each value of Tmax by the HDM and HROM-based optimization procedures based on thecoarse


Fig. 16 Solution computed onthe coarse mesh for the optimaldesign returned by the HDM

Fig. 17 Solution computed onthe fine mesh for the optimaldesign returned by the HDMbased on the coarse mesh


Fig. 18 Evolution of the error during the greedy HROM trainingprocedure on the coarse mesh

Due to limitations in RAM, only the optimization prob-lem constrained by the PDE discretized on the coarse meshcan be solved using the HDM and fmincon. For the coarseproblem, there is a total of Nx = 960 + 7 = 967 variablesin the optimization problem.

The HDM is first used to solve the 90 optimization prob-lems of interest. The corresponding optimal designs that arefound are reported in Fig. 14. For several values of Tmax,multiple local minima are found by the optimization pro-cedure, validating the chosen multi-start strategy. For eachoptimal design that is found, the HDM solution is com-puted on the fine mesh and the maximum temperature in thedomain determined. The corresponding results are reportedin Fig. 15. One can observe that for most points the con-straint is violated by the optimal design. This is due tothe fact that the mesh is too coarse to accurately capturethe physical behavior. The solutions for the optimal designobtained for Tmax = 2750 K are shown in Figs. 16 and 17.The reader can observe that the solutions are qualitativelyand quantitatively different. In particular, the maximumtemperature for the solution computed on the coarse meshis 2742 K while it is 2913 K when the solution is computedon the fine mesh.

Fig. 19 Evolution of the error during the greedy HROM trainingprocedure on the fine mesh

Next, a HROM is constructed by the greedy procedureoutlined in Algorithm 3. For that purpose, Nc = 175 candi-dates are chosen by sampling the parameter domain using acombination of Latin Hypercube samples and full-factorialsampling. Nmax

iter = 20 greedy iterations are performed anda HROM with parameters nw = 50, nc = 50 and |E | = 50is constructed. Note that for this problem, the state equationcan be decomposed into an affine part in w and a nonlinearpart, as emphasized in Remark 1 of Section 3.2. Only thenonlinear component of the state equation is then approxi-mated by hyper-reduction. Nv = 50 validations points areselected by LHS in the parameter domain D. These pointsserve to monitor at each step of the greedy procedure theerror associated with the HROM. The maximum relativeerror and average relative error computed on the validationpoints are reported in Fig. 18. One can observe that theseerrors decrease as the procedure progresses. After 20 greedyiterations, the maximum error is 1.7% and the average erroris as low as 0.7%.

A surrogate function for the constraints is then computedusing RBFs. Due to the large number of training points andthe large number of reduced variables, it is not tractable tocompute a Hermite interpolant and as a result a Lagrange

Table 5 CPU timingsassociated with eachoptimization procedure for thereaction problem (*: thespeedup cannot be computed asthe optimization problem basedon the HDM is intractable)

Model HDM HROM HDM HROM

Mesh Coarse Coarse Fine Fine

Offline CPU time - 12 min 08 s - 8 h 5 min

Online CPU time 18 h 6 min 27 min 18 s intractable 26 min 59 s

Online Speedup - 39.8 - *

Total CPU time 18 h 6 min 39 min 26 s intractable 8 h 32 min

Total Speedup - 27.5 - *


interpolant is chosen. For that interpolant, the Monte-Carloestimator of the average relative error defined in (62) iseLfr

= 1.4%, validating the construction of the surrogateconstraint function.

The HROM is then used to solve the 90 optimizationproblems. The optimal designs that are returned are alsoreported in Fig. 14. For a couple of values of Tmax, the opti-mal designs are identical to the one found by the HDMprocedure. When a high-dimensional model is used to com-pute the solution at the optimized design, the maximumtemperature is always within less than 5.8% above Tmax.Interestingly, the designs obtained using the HROM forTmax = 2500K and 3000K are superior to their HDM-based counterparts. Both HROM-based designs satisfy themaximum temperature constraint and are associated witha better objective. The CPU timings associated with eachprocedure are reported in Table 5. An online speedup of 39.8

is obtained with the HROM. The HROM construction CPUtime is 12 minutes and when that timing is also taken intoaccount, an overall speedup of 27.5 is found.

As shown in Fig. 15, many designs returned by thecomputations involving the coarse mesh violate the con-straints when evaluated using the fine mesh. However, asemphasized above, the optimization problem involving anHDM on a finer mesh is not tractable any more, motivatingthe HROM-based approach.

For the problem defined on the fine mesh, there is atotal of Nx = 22,800+ 7= 22,807 variables in the opti-mization problem. The HROM construction is repeated withthe same parameters as for the coarse mesh. After 20 greedyiterations, the maximum error computed over 50 validationpoints is 3.6% and the average error is as low as 1.4%.These errors computed on the validation points at each itera-tion of the greedy procedure are reported in Fig. 19. Again,

Fig. 20 Comparison of the optimal designs found for each value of Tmax by the HROM-based optimization procedures based on the fine mesh


Fig. 21 Maximum temperature computed on the fine mesh for eachoptimal design

one observe the sharp decrease in error as the training pro-cedure progresses. As reported in Table 5, the constructionof the HROM takes about 8 hours.

A Lagrange RBF interpolant is constructed for which aMonte-Carlo estimator of the relative error is eL

fr= 5.1%.

The HROM is then used to solve the 90 problems ofinterest. The optimal designs are reported in Fig. 20. Foreach optimal design that is computed using the HROM, theHDM solution is computed and the associated maximumtemperature evaluated. The results are reported in Fig. 21.The optimal designs are always within 7% of the constrains,which is consistent with the value for eL

frthat was reported.

This underlines that a margin should be added to the con-strains when a surrogate model is used to approximate it.The Monte-Carlo estimator of the relative error appears tobe a promising indicator for this margin.

As indicated in Table 5, solving the 90 optimization prob-lems using the HROM only takes about 27 minutes, whichcorresponds to about 18 seconds per problem. This is tobe compared with the fact that the optimization problemsinvolving the HDM is intractable. It is interesting to noticethat the online cost associated with the HROMs is not sen-sitive on the dimension of the HDM mesh. This can beexplained by the fact that for both cases, the dimensionsof all the quantities involved in the reduced optimizationproblem are identical.

7 Conclusion

This paper presents an approach for the fast solution ofnonlinear PDE constrained optimization problems usingprojection-based model reduction. Because projecting thelarge set of equations onto a reduced-order basis does not

necessarily lead to a reduction of the complexity associatedwith solving the model, an additional reduction procedure,hyper-reduction, is applied. As hyper-reduction operates ona much reduced computational domain, the reduced mesh,the original computational domain is discarded and the com-plexity of solving the hyper-reduced-order model (HROM)does not scale with the large dimension any more, lead-ing to important computational speedups. However, becausethe original mesh is discarded, scalar functions of the entirestate vector need to be computed as a function of the reducedcoordinates only. This is the case of the objective func-tion in an optimization context. When the scalar function ispolynomial, this can be easily achieved by pre-computingthe coefficient terms in the polynomial expansion. When thefunction is not polynomial, an approach based on the con-struction of a surrogate function is proposed in the presentwork.

Applications of this framework to the inverse shape opti-mization of a nozzle inlet as well as the optimization ofa flame reaction show that applying HROM-based opti-mization has the capability to lead to large computationalspeedup. Furthermore, when tractable, constructing Her-mite interpolation-based surrogate functions leads to high-fidelity approximations and optimized designs that are veryclose to the target designs. A cross-validation step is usedto ensure that the surrogate is robust with respect to shapeparameter changes.

Acknowledgments The authors acknowledge partial support by theArmy Research Laboratory through the Army High PerformanceComputing Research Center under Cooperative Agreement W911NF-07-2-0027, and partial support by the Office of Naval Research undergrant no. N00014-11-1-0707. This document does not necessarilyreflect the position of these institutions, and no official endorsementshould be inferred.

Appendix: Cross-validation procedure for choosingthe radial basis function parameter

The cross-validation procedure determines the optimal RBFparameter ε > 0 as in the work of (Rippa 1999). Forthat purpose, the sample set

{xrj

}Ns

j=1 is partitioned into K

non-overlapping subsets {Sj }Kj=1, and a series of candidateparameters εi, i = 1, · · · , Nc is proposed. The generaliza-tion error associated with each candidate parameter is thenestimated by building surrogates using K − 1 subsets andtesting its accuracy on the remaining subset. The optimalparameter ε is then selected as being the one minimizingthe generalization error. The procedure is summarized inAlgorithm 5.


References

Alexandrov NM, Dennis Jr JE, Lewis RM (1998) A trust-regionframework for managing the use of approximation models inoptimization. Structural Optimization 15:16–23

Amsallem D, Farhat C (2008) Interpolation method for adaptingreduced-order models and application to aeroelasticity. AIAA J46(7):1803–1813

Amsallem D, Farhat C (2011) An online method for interpolatinglinear parametric reduced-order models. SIAM J Sci Comput33(5):2169–2198

Amsallem D, Zahr MJ, Farhat C (2012) Nonlinear model order reduc-tion based on local reduced-order bases. Int J Numer MethodsEng 92(10):891–916

Antoulas A (2005) Approximation of large-scale dynamical systems.SIAM

Astrid P, Weiland S, Willcox K (2008) Missing point estimationin models described by proper orthogonal decomposition. IEEETrans Autom Control 53(10):2237–2251

Biegler LT, Ghattas O, Heinkenschloss M, van Bloemen Waanders B(eds) (2003) Large-scale PDE-constrained optimization. Springer

Biegler LT, Ghattas O, Heinkenschloss M, Keyes D, van BloemenWaanders B (eds) (2007) Real-time PDE-constrained optimiza-tion. SIAM

Biros G, Ghattas O (2005a) Parallel Lagrange–Newton–Krylov–Schurmethods for PDE-constrained optimization. Part I: the Krylov–Schur solver. SIAM J Sci Comput 27(2):687–713

Biros G, Ghattas O (2005b) Parallel Lagrange–Newton–Krylov–Schurmethods for PDE-constrained optimization. Part II: the Lagrange–

Newton solver and its application to optimal control of steadyviscous flows. SIAM J Sci Comput 27(2):714–739

Buffoni M, Willcox K (2010) Projection-based model reduction forreacting flows. AIAA Paper 2010-5008, 40th Fluid DynamicsConference and Exhibit, 28 June - 1 July 2010, Chicago, Illinois

Bui-Thanh T, Willcox K, Ghattas O (2008) Parametric reduced-order models for probabilistic analysis of unsteady aerodynamicapplications. AIAA J 46(10):2520–2529

Carlberg K, Farhat C (2010) A low-cost, goal-oriented ‘compactproper orthogonal decomposition’ basis for model reduction ofstatic systems. Int J Numer Methods Eng 86(3):381–402

Carlberg K, Bou-Mosleh C, Farhat C (2011) Efficient non-linearmodel reduction via a least-squares Petrov–Galerkin projectionand compressive tensor approximations. Int J Numer MethodsEng 86(2):155–181

Carlberg K, Farhat C, Cortial J, Amsallem D (2013) The GNATmethod for nonlinear model reduction: effective implementationand application to computational fluid dynamics and turbulentflows. J Comput Phys 242(C):623–647

Chaturantabut S, Sorensen D (2010) Nonlinear model reduction viadiscrete empirical interpolation. SIAM J Sci Comput 32(5):2737–2764

Dihlmann M, Haasdonk B (2013) Certified PDE-constrained parame-ter optimization using reduced basis surrogate models for evolu-tion problems. Submitted to J Comput Optim Appl

Everson R, Sirovich L (1995) Karhunen–Loeve procedure for gappydata. J Opt Soc Am A 12(8):1657–1664

Fahl M, Sachs EW (2003) Reduced order modelling approachesto PDE-constrained optimization based on proper orthogonaldecomposition. In: Biegler LT, Ghattas O, Heinkenschloss M,van Bloemen Waanders B (eds) Large-scale PDE-constrainedoptimization. Springer, Berlin, pp 268–280

Fasshauer GE (2007) Meshfree approximation methods with MAT-LAB. World Scientific

Forrester A, Keane A (2009) Recent advances in surrogate-basedoptimization. Prog Aerosp Sci 45(1-3):50–79

Gill PE, Murray W, Wright MH (1981) Practical optimization. Aca-demic Press

Gogu C, Passieux JC (2013) Efficient surrogate construction by com-bining response surface methodology and reduced order modeling.Struct Multidiscip Optim 47(6):821–837

Gunzburger M (2003) Perspectives in Flow Control and Optimization.SIAM

Haftka RT (1985) Simultaneous analysis and design. AIAA J23(7):1099–1103

Hay A, Borggaard JT, Akhtar I, Pelletier D (2010) Reduced-ordermodels for parameter dependent geometries based on shape sensi-tivity analysis. J Comput Phys 229(4):1327–1352

Homberg D, Volkwein S (2003) Control of laser surface hardening bya reduced-order approach using proper orthogonal decomposition.Math Comput Model 38(10):1003–1028

Jameson A (1988) Aerodynamic design via control theory. J SciComput 3(3):233–260

LeGresley PA, Alonso JJ (2000) Airfoil design optimization usingreduced order models based on proper orthogonal decomposi-tion. AIAA Paper 2000-2545 Fluids 2000 Conference and Exhibit,Denver, CO

Manzoni A, Quarteroni A, Rozza G (2011) Shape optimization for vis-cous flows by reduced basis method and free form deformation.Int J Num Meth Eng

Nocedal J, Wright SJ (2006) Numerical optimization. SpringerPaul-Dubois-Taine A, Amsallem D (2014) An adaptive and efficient

greedy procedure for the optimal training of parametric reduced-order models. Int J Num Meth Eng


Queipo NV, Haftka RT, Shyy W, Goel T (2005) Surrogate-basedanalysis and optimization. Prog Aerosp Sci 41:1–28

Rippa S (1999) An algorithm for selecting a good value for the param-eter c in radial basis function interpolation. Adv Comput Math11(2/3):193–210

Ryckelynck D (2005) A priori hyperreduction method: an adaptiveapproach. J Comput Phys 202(1):346–366

Sacks J, Welch WJ, Mitchell TJ, Wynn HP (1989) Design and analysisof computer experiments. Stat Sci 4(4):409–423

Sirovich L (1987) Turbulence and the dynamics of coherent structures.Part I: coherent structures. Q Appl Math 45(3):561–571

Veroy K, Patera AT (2005) Certified real-time solution of theparametrized steady incompressible Navier-Stokes equations: rig-orous reduced-basis a posteriori error bounds. Int J Numer Meth-ods Fluids 47(8–9):773–788

Weickum G, Eldred MS, Maute K (2009) A multi-point reduced-ordermodeling approach of transient structural dynamics with appli-cation to robust design optimization. Struct Multidiscip Optim38(6):599–611

Xiao M, Breitkopf P, Filomeno Coelho R, Knopf-Lenoir C, Villon P(2012) Enhanced POD projection basis with application to shapeoptimization of car engine intake port. Struct Multidiscip Optim46(1):129–136

Young DP, Huffman WP, Melvin RG, Hilmes CL, Johnson FT (2003)Nonlinear elimination in aerodynamic analysis and design opti-mization. In: Large-scale PDE-constrained optimization. Springer,Berlin, Heidelberg, pp 17–43

Yue Y, Meerbergen K (2013) Accelerating optimization of paramet-ric linear systems by model order reduction. SIAM J Optim23(2):1344–1370

Zahr MJ, Farhat C (2014) Progressive construction of a parametricreduced-order model for PDE-constrained optimization. Acceptedfor publication, Int J Num Meth Eng

Zahr MJ, Amsallem D, Farhat C (2013) Construction ofparametrically-robust CFD-based reduced-order models for PDE-constrained optimization. In: AIAA Paper 2013–2845, 21st AIAAComputational Fluid Dynamics Conference, San Diego, CA, June26–29, 2013, American Institute of Aeronautics and Astronautics,Reston, Virginia

design optimization using hyper-reduced-order models

Documents