the new faces of nonlinear programming
DESCRIPTION
The New Faces of Nonlinear Programming. Jorge Nocedal Optimization Technology Center Argonne-Northwestern. New Problems, New Algorithms, New Software. Traditional Applications: solve larger problems, more robustness New classes of applications Advances in modeling languages: AMPL , … - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/1.jpg)
The New Faces of Nonlinear Programming
Jorge Nocedal Optimization Technology Center
Argonne-Northwestern
n
i
i
x
x
IixgEixh
xf
,0)( ,0)( s.t.
)( min
![Page 2: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/2.jpg)
04/22/23 McMaster 2
New Problems, New Algorithms, New Software
• Traditional Applications: solve larger problems, more robustness• New classes of applications• Advances in modeling languages: AMPL, …• Automatic differentiation• Interior Methods• Test problems (CUTE, COPS)• New packages: LOQO, KNITRO,.• Internet Optimization: NEOS server
![Page 3: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/3.jpg)
04/22/23 McMaster 3
NEOS Server
• Argonne • Northwestern
MINOS. SNOPT, FILTER, LANCELOTLOQO, KNITRO
USER
![Page 4: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/4.jpg)
04/22/23 McMaster 4
Semi-infinite Optimization Mixed Integer Nonlinearly Constrained Optimization Mixed Integer Linear Programming Nonlinearly Constrained Optimization Semidefinite & Second Order Cone Programming Linear Programming Unconstrained Optimization Linear Network Optimization Complementarity Problems Nondifferentiable Optimization Stochastic Linear Programming Global Optimization Application-specific Optimization
![Page 5: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/5.jpg)
04/22/23 McMaster 5
Part I:New Classes of Problems
Instead of new algorithms/software/ adapt existing techniques
• Equilibrium constraints: (T. Luo)o Bi-level programmingo Complementarity constraints
• Semi-definite programming (??)• PDE-constrained optimization
o Differential algebraic systems
![Page 6: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/6.jpg)
04/22/23 McMaster 6
Nonlinear Optimization Formulation
• Theme: constraints involve a difficult computation/simulation.
• Limitations of this formulation?• Logic constraints
n
i
i
x
x
IixgEixh
xf
,0)( ,0)( s.t.
)( min
![Page 7: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/7.jpg)
04/22/23 McMaster 7
Equilibrium Constraints
• Structurally difficult• No strictly feasible direction• Algorithmically…
0 0 ,0
ii yxyx
xy
10
01
x
y
f
c
0ii yx
![Page 8: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/8.jpg)
04/22/23 McMaster 8
• Optimization problem with equilibrium constraints not stable cannot apply NLP algorithms to it
• Confirmed by experimental evidence (??)
• Reality: software not capable of dealing with degeneracy, not sufficiently robust
• Theoretical mistake: lack of stability does not imply practical problems. Structural degeneracy.
• Active Set SQP (Leyffer et al)
• Interior Methods: solve perturbed problem
![Page 9: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/9.jpg)
04/22/23 McMaster 9
origin
destination
For each origin-destination pair (o,d) we have:
• qod: demand (in terms of flow) between o and d
• K : index set of paths from o to d
• fk : flow along path k, for each k in K
•ck(f): cost of travel along path k (usually time), for each path k in K
• λ = λ(qod) : minimum possible travel cost between o and d
Traffic Assignment
• Vector x of link flows, •Efficiency parameters (capacity, speed limit) given at link level• The path flows and costs are aggregated (based on x) through adjacency matrix A• Need constraints for demand satisfaction and conservation of flow• Many origin-destination pairs may exists in the network
![Page 10: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/10.jpg)
04/22/23 McMaster 10
origin
destination
•Improvements to network: Traffic Network Design•Discrete: add lanes, links•Continuous: link capacity expansion•Boyce (1979) ed, •Continuous capacity?
Network Design (Continuous Equilibrium)
•Complex interaction between System Optimal and User Equilibrium is recognized -> bilevel programming (e.g. Abdulaal-LeBlanc)
![Page 11: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/11.jpg)
04/22/23 McMaster 11
origin
destination
•Given network: G=(N,A)•Find additions yi to capacities ci of links i in A•So that:Cost of improvement and efficiency of network is minimized
Example of Continuous Equilibirium Network Design
iii
iiiii
iiii
yygitxyxT
xyxt
oft improvemencapacity ofcost )(link in time travel total ),(
flowfor ilink in time travel),(
xyts
ygyxT iiiii
0 ..
)](),([min
An equlibrium flow
![Page 12: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/12.jpg)
04/22/23 McMaster 12
Together with demand satisfaction and conservation of flow, we need to demand EQUILIBRIUM, which in this case looks like:
λ = λ(qod) : minimum possible travel cost between o and d
• If there is flow on path k (fk > 0): path k is a minimum cost path (ck = λ)
• If path k is relatively expensive (ck > λ): no one uses this path (fk = 0)
0)( and 0 0 ecf ecf
KKT Conditions
![Page 13: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/13.jpg)
04/22/23 McMaster 13
Partial Differential Equations and Optimization (Tsai, Byrd,N)
Mems flaps Desiredflow
Navier-Stokes equations
Determine position of mems flaps to optimizeQuality of exhaust flow
Phase II: boundary control
![Page 14: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/14.jpg)
04/22/23 McMaster 14
Partial Differential Equations (PDEs)
• Systems that evolve in space (several dimensions) and time are described by PDEs
• Solution: function u(x,t) – infinite-dimen prob• More space dimen.: great computational and
storage cost
c)(hyperboli equation wave)(parabolic equation heat
xxtt
xxt
cuucuu
![Page 15: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/15.jpg)
04/22/23 McMaster 15
Success of PDE simulations
3D Large Eddy Simulation around an airfoil
![Page 16: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/16.jpg)
04/22/23 McMaster 16
• Fluid flow described by Navier-Stokes
on 0in 0
in 0)()(1
uu
puuuur
T
•Solution of nonlinear PDEs•Newton-Krylov
•Sequence of meshes, Krylov (FGMRES)-(full)
multigrid (Krylov smoother).
Parallel computing to obtain high resolution
)()(' kkk xcdxc
![Page 17: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/17.jpg)
04/22/23 McMaster 17
Now optimize!
• Robustness of PDE solvers: millions of variables, hundreds of processors, multiple physical interactions
• Introduce free parameters• Finite-dimensional formulation
system of PDEs
avulvucts
vuf
),( 0),( ..),( min
![Page 18: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/18.jpg)
04/22/23 McMaster 18
State-of-the-art algorithms
• KKT system Newton-Lagrange
• Active set: SNOPT, FilterSQP factor subset of A, reduced Hessian
• Interior: LOQO, KNITRO factor
• Algorithms must accept iterative solution of constraint linearization. Av A’v
)(
)()()(xc
xcxfxFT
0
or
0
TT AAI
AAW
c
flp
AAW T
0
![Page 19: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/19.jpg)
04/22/23 McMaster 19
Unconstrained reformulation
• Linearize constraints• Eliminate state variables xs (basic)
• Minimize w.r.t. controls xd (non-bas) • New problem
Modern optimization SQP:
0),( . min
ds
ds
xxcts),xf(x
cdAdtsddq
cTcs
cs
TsA ..
),( min
cdAd cTcs T
sA
)(min dxf
![Page 20: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/20.jpg)
04/22/23 McMaster 20
Weather Forecasting - Oceanography
• = state of atmosphere,
• Observations: Time windows i: length = a few time steps
• Short integration: from initial condition Problem: unknown
2
0
)(min i
M
ii xy
ix 1ix
iy
0x
610nMxxx ...,, ,10
x
![Page 21: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/21.jpg)
04/22/23 McMaster 21
• Background field • Observations • Background covar• Obs covar• Time
Constraints eliminated, no bounds, inequalities3 Spaces: grid point, spectral, observation
Nonlinear Least Squares Problem
))(())((
)()()(min
1
0
01
00
iiiT
ii
M
ii
bb
xHyRxHy
xxBxxxJ
i
i
b
tRByx
![Page 22: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/22.jpg)
04/22/23 McMaster 22
Part III:New Algorithms
• New applications• New methods• New software• New tools (modeling languages, automatic
differentaition)
![Page 23: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/23.jpg)
04/22/23 McMaster 23
Part IIIAdvances in NLP Algorithms:Active Set SQP
Before 1998:• Active Set SQP software: highly complex• Many dense, substandard versions• Quasi-Newton (SNOPT, MINOS)
Present:• Filter, Second derivatives (FilterSQP)• SNOPT second derivatives in progress• Can SQP compete with Interior Methods?Future:• Linear Programming Based (Dundee, Northwestern.)
![Page 24: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/24.jpg)
04/22/23 McMaster 24
Interior Methods Terlaky
Newton’s method to KKT conditions of equal-problem:
Reformulate to avoid rational functions: primal-dualBacktrack (difficulties!)Update barrier parameterInitial point strategy-failures
0. 0)( s.t.
)ln()( min
sg(x)xh
sxf i
0. 0)( s.t.
)( min
g(x)xh
xf
),(),(' sxFdsxF
0, s
)(
)()()(xc
xcxfxFT
![Page 25: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/25.jpg)
04/22/23 McMaster 25
Nonlinear Interior Methods
Approach I: LOQO,OPINEL,BOEING,IPOPT,…• Modify W, • Merit Function/Filter
Approach II: (KNITRO)
0. 0)( s.t.
)ln()( min
sg(x)xh
sxf i
sdsAdxc
q
s
||d|| 0)( s.t.
(d) min
c
flp
AAW T
0
Eliminate constraints-null space approachApproximate solutionIterative solution by conjugate gradientsMerit function
enforcement
![Page 26: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/26.jpg)
04/22/23 McMaster 26
KNITRO 3.0
Interior Active
Iterative Direct
The Future (summer 2003)
![Page 27: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/27.jpg)
04/22/23 McMaster 27
LP-EQP based on a Penalty Approach
Equality constrained quadratic program
||d|| 00s.t
min
dggdhh
df
T
T
T
|||| 0
0 ..
dWidg
dhtsT
T
)( min dq L-1 penalty
LP
EQP
Working set W
![Page 28: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/28.jpg)
04/22/23 McMaster 28
Step computation
dc: min quadratic
model of merit function: Cauchy point
Dogleg approach
EQP by projected CG
dLP
dc
deqpd
xk
![Page 29: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/29.jpg)
04/22/23 McMaster 29
Rationale for Integration
• Active set approach: LP + EQP (Fletcher) shares EQP solution with Interior Algor.• preconditioning• Final active set identification• Warm starts• Share interfaces, stop tests, testing, object c.• Both trust region methods• New algorithms…
![Page 30: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/30.jpg)
04/22/23 McMaster 30
Remarks on:New Software—Interior Methods
– Change of barrier parametero Guiding principles (increase/decrease)o Scale invariance, initial pointo Global convergence
– When to attempt superlinear convergence– Seemingly superior to active set SQP codes
o Only choice for very large (reduced space) Versatile
o 1st derivs only; do/not factor Hessiano Iterative vs direct solverso Feasible/infeasible modes
LOQO, KNITRO
![Page 31: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/31.jpg)
04/22/23 McMaster 31
Tests Sets: CUTE (850 problems) COPS
![Page 32: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/32.jpg)
On many practical applications…
![Page 33: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/33.jpg)
04/22/23 McMaster 33
Final Remarks
• Extension of core NLP algorithms/sofware instead of special-purpose methods
• Investigation of limitations: degeneracies, multilevels
• Robustness (fundamental algorithmic)• More iterative options• How far can NLP methods scale up?
![Page 34: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/34.jpg)
04/22/23 McMaster 34
L1 Merit Function
L1 linear program and penalty parameter selection
• Try to find minimum s.t. residuals = zero• W= constraints with zero residuals• Can achieve robustness and efficiency
},0max{||)( ii ghxf
dtdgg
rqdhhtstrqdf
T
T
T
. .)(min
![Page 35: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/35.jpg)
04/22/23 McMaster 35
Interior Method- Direct
• Solve primal-dual system• Retain trust region• Revert to Interior Iterative
Step is too longInertia is not correctStep is rejected by merit functionAdaptive barrier parameter (Morales, Orban)
0AAW T
dn
Slack bound
![Page 36: The New Faces of Nonlinear Programming](https://reader030.vdocuments.mx/reader030/viewer/2022020117/568167ab550346895ddcf7fe/html5/thumbnails/36.jpg)
04/22/23 McMaster 36
Interior Method (Iterative, CG)
Projected CG:
0. 0)( s.t.
)ln()( min
sg(x)xh
sxf i
sdsdSd.r Adsg(x)
rAdxhWddd
ssx
g
h
TT
|||| ||||
)( s.t.min
1
Infeasibleslacks
= barrier function
Primal method, no multipliers