m15 part2 tier1 nonlinear - polytechnique montréal · 2006-10-23 · multivariable unconstrained...

Process Optimization

Click to edit Master title styleTier I: Mathematical Methods of

Optimization

Section 3:Nonlinear Programming

Click to edit Master title styleIntroduction to Nonlinear

Programming• We already talked about a basic aspect of

nonlinear programming (NLP) in the Introduction Chapter when we considered unconstrained optimization.

Click to edit Master title styleIntroduction to Nonlinear

Programming

• We optimized one-variable nonlinear functions using the 1st and 2nd derivatives.

• We will use the same concept here extended to functions with more than one variable.

Click to edit Master title styleMultivariable Unconstrained

Optimization

• For functions with one variable, we use the 1st

and 2nd derivatives.

• For functions with multiple variables, we use identical information that is the gradient and the Hessian.

• The gradient is the first derivative with respect to all variables whereas the Hessian is the equivalent of the second derivative

Click to edit Master title styleThe Gradient

• Review of the gradient (∇):For a function “f ”, of variables x1, x2, …, xn:

⎥⎦

⎤⎢⎣

⎡∂∂

∂∂

∂∂

=∇nx

fxf

xff K

21

Example: 231

321 )(3)(215 xxxxf −+=

[ ]312

22

3 6)(6)(315 xxxxf −−=∇

Click to edit Master title styleThe Hessian

• The Hessian (∇2) of f(x1, x2, …, xn) is:

⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

∂∂

∂∂∂

∂∂∂

∂∂∂

∂∂

∂∂∂

∂∂∂

∂∂∂

∂∂

=∇

2

2

2

2

1

2

2

2

22

2

12

21

2

21

2

21

2

2

nnn

n

n

xf

xxf

xxf

xxf

xf

xxf

xxf

xxf

xf

f

L

MOMM

K

K

Click to edit Master title styleHessian Example

• Example (from previously):2

313

21 )(3)(215 xxxxf −+=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−

−=∇

13

2

32

6060120600

xxx

xf

[ ]312

22

3 6)(6)(315 xxxxf −−=∇

Click to edit Master title styleUnconstrained Optimization

The optimization procedure for multivariable functions is:

1. Solve for the gradient of the function equal to zero to obtain candidate points.

2. Obtain the Hessian of the function and evaluate it at each of the candidate points

• If the result is “positive definite” (defined later) then the point is a local minimum.

• If the result is “negative definite” (defined later) then the point is a local maximum.

Click to edit Master title stylePositive/Negative Definite

• A matrix is “positive definite” if all of the eigenvalues of the matrix are positive (> 0)

• A matrix is “negative definite” if all of the eigenvalues of the matrix are negative (< 0)

Click to edit Master title stylePositive/Negative Semi-definite

• A matrix is “positive semi-definite” if all of the eigenvalues are non-negative (≥ 0)

• A matrix is “negative semi-definite” if all of the eigenvalues are non-positive (≤ 0)

Click to edit Master title styleExample Matrix

Given the matrix A:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−−−=211175

542A

The eigenvalues of A are:22 −=λ702.31 −=λ 702.23 =λ

This matrix is negative definite

Click to edit Master title styleUnconstrained NLP Example

Consider the problem:Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2

– x2x3 + (x3)2 + x3

First, we find the gradient with respect to xi:

1 2

1 2 3

2 3

2 122 1

x xf x x x

x x

+ −⎡ ⎤⎢ ⎥∇ = − + −⎢ ⎥⎢ ⎥− + +⎣ ⎦


Next, we set the gradient equal to zero:

0=∇f⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

++−−+−−+

000

12212

32

321

21

xxxx

xx⇒

So, we have a system of 3 equations and 3 unknowns. When we solve, we get:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

111

3

2

1

xxx

x


So we have only one candidate point to check.

Find the Hessian:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

−=∇

210121

0122 f


The eigenvalues of this matrix are:

414.31 =λ 586.02 =λ 23 =λ

All of the eigenvalues are > 0, so the Hessian is positive definite.

So, the point is a minimum⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

111

3

2

1

xxx

x


Unlike in Linear Programming, unless we know the shape of the function being minimized or can determine whether it is convex, we cannot tell whether this point is the global minimum or if there are function values smaller than it.

Click to edit Master title styleMethod of Solution

• In the previous example, when we set the gradient equal to zero, we had a system of 3 linear equations & 3 unknowns.

• For other problems, these equations could be nonlinear.

• Thus, the problem can become trying to solve a system of nonlinear equations, which can be very difficult.

Click to edit Master title styleMethod of Solution

• To avoid this difficulty, NLP problems are usually solved numerically.

• We will now look at examples of numerical methods used to find the optimum point for single-variable NLP problems. These and other methods may be found in any numerical methods reference.

Click to edit Master title styleNewton’s Method

When solving the equation f ′(x) = 0 to find a minimum or maximum, one can use the iteration step:

)()(

''

'1

k

kkk

xfxfxx −=+

where k is the current iteration. Iteration is continued until |xk+1 – xk| < ε where ε is some specified tolerance.

Click to edit Master title styleNewton’s Method Diagram

Newton’s Method approximates f′ (x) as a straight line at xk and obtains a new point (xk+1), which is used to approximate the function at the next iteration. This is carried on until the new point is sufficiently close to x*.

xx* xk+1 xk

Tangent of f′ (x) at xk

f′ (x)

Click to edit Master title styleNewton’s Method Comments

• One must ensure that f (xk+1) < f (xk) for finding a minimum and f (xk+1) > f (xk) for finding a maximum.

• Disadvantages:– Both the first and second derivatives must be

calculated– The initial guess is very important – if it is not

close enough to the solution, the method may not converge

Click to edit Master title styleRegula-Falsi Method

This method requires two points, xa & xb that bracket the solution to the equation f ′(x) = 0.

)()()()(

''

'

ab

abbbc

xfxfxxxfxx

−−⋅

−=

where xc will be between xa & xb. The next interval will be xc and either xa or xb, whichever has the sign opposite of xc.

Click to edit Master title styleRegula-Falsi Diagram

The Regula-Falsi method approximates the function f′ (x) as a straight line and interpolates to find the root.

xxa

xb

xc

x*

f′ (x)

Click to edit Master title styleRegula-Falsi Comments

• This method requires initial knowledge of two points bounding the solution

• However, it does not require the calculation of the second derivative

• The Regula-Falsi Method requires slightly more iterations to converge than the Newton’s Method

Click to edit Master title styleMultivariable Optimization

• Now we will consider unconstrained multivariable optimization

• Nearly all multivariable optimization methods do the following:

1. Choose a search direction dk

2. Minimize along that direction to find a new point:

where k is the current iteration number and αk

is a positive scalar called the step size.

kkkk dxx α+=+1

Click to edit Master title styleThe Step Size

• The step size, αk, is calculated in the following way:

• We want to minimize the function f(xk+1) = f(xk +αkdk) where the only variable is αk

because xk & dk are known.• We set and solve for αk

using a single-variable solution method such as the ones shown previously.

( ) 0d

d=

+k

kkkfαα dx

Click to edit Master title styleSteepest Descent Method

• This method is very simple – it uses the gradient (for maximization) or the negative gradient (for minimization) as the search direction:

)( kk f xd ∇⎭⎬⎫

⎩⎨⎧−+

= for⎭⎬⎫

⎩⎨⎧

minmax

So, )(1 kkkk f xxx ∇⎭⎬⎫

⎩⎨⎧−+

=+ α

Click to edit Master title styleSteepest Descent Method

• Because the gradient is the rate of change of the function at that point, using the gradient (or negative gradient) as the search direction helps reduce the number of iterations needed

x1

x2 f(x) = 5f(x) = 20

f(x) = 25

xk

∇f(xk)

-∇f(xk)

Click to edit Master title styleSteepest Descent Method Steps

So the steps of the Steepest Descent Method are:

1. Choose an initial point x0

2. Calculate the gradient ∇f(xk) where k is the iteration number

3. Calculate the search vector:4. Calculate the next x:

Use a single-variable optimization method to determine αk.

)( kk f xd ±∇=kkkk dxx α+=+1

Click to edit Master title styleSteepest Descent Method Steps

5. To determine convergence, either use some given tolerance ε1 and evaluate:

for convergenceOr, use another tolerance ε2 and evaluate:

for convergence

11 )()( ε<−+ kk ff xx

2)( ε<∇ kf x

Click to edit Master title styleConvergence

• These two criteria can be used for any of the multivariable optimization methods discussed here

Recall: The norm of a vector, ||x|| is given by:22

22

1 )()()( nT xxx +++=⋅= Lxxx

Click to edit Master title styleSteepest Descent Example

Let’s solve the earlier problem with the Steepest Descent Method:

Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2

– x2x3 + (x3)2 + x3

Let’s pick

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

000

0x


[ ]10000001)0(2)( 00 ++−−+−−+−=−∇= xd f

[ ]122)1(2)( 3232121 ++−−+−−+=∇ xxxxxxxf x

[ ] [ ]101101 −−=−=

[ ] [ ]101000 01 −−⋅+= αx

Now, we need to determine α0


2)(4)( 00

1

−= ααd

df x

)()(00)1)(()()( 0200201 αααα −++−+−+=xf

)(2)(2 020 αα −=

Now, set equal to zero and solve:

21

420 ==α2)(4 0 =α ⇒


So,[ ] [ ]101000 01 −−⋅+= αx

[ ] ⎥⎦⎤

⎢⎣⎡ −−+=

210

21000

⎥⎦⎤

⎢⎣⎡ −−=∴

210

211x


⎥⎦⎤

⎢⎣⎡ +−++++−−=−∇= 110

210

21011)( 11 xd f

[ ]0101 −=∴d

Take the negative gradient to find the next search direction:


[ ]010210

21 12 −⋅+⎥⎦

⎤⎢⎣⎡ −−= αx

⎥⎦⎤

⎢⎣⎡ −−−=

21

21 1α

Update the iteration formula:


1)(2)( 11

1

−= ααd

df x

( )21

41

21)()(1

21

41)( 12112 −+⎟

⎠⎞

⎜⎝⎛−++⎟

⎠⎞

⎜⎝⎛−+= αααxf

21)( 121 −−= αα

Insert into the original function & take the derivative so that we can find α1:


Now we can set the derivative equal to zero and solve for α1:

211 =α1)(2 1 =α ⇒


Now, calculate x2:

[ ]010210

21 12 −⋅+⎥⎦

⎤⎢⎣⎡ −−= αx

⎥⎦⎤

⎢⎣⎡ −+⎥⎦

⎤⎢⎣⎡ −−= 0

210

210

21

⎥⎦⎤

⎢⎣⎡ −−−=∴

21

21

212x


So,

⎥⎦⎤

⎢⎣⎡ +−+−++−−=−∇= 11

21

211

21

2111)( 22 xd f

⎥⎦⎤

⎢⎣⎡ −−=∴

210

212d

⎥⎦⎤

⎢⎣⎡ −−⋅+⎥⎦

⎤⎢⎣⎡ −−−=

210

21

21

21

21 23 αx

⎥⎦⎤

⎢⎣⎡ +−−+−= )1(

21

21)1(

21 22 αα


23)1()( 2

2

3

−+= ααd

df x

41)1(

23)1(

21)( 2223 ++−+= ααxf

Set the derivative equal to zero and solve:

212 =α

23)1( 2 =+α ⇒

Find α2:


Calculate x3:

⎥⎦⎤

⎢⎣⎡ −−⋅+⎥⎦

⎤⎢⎣⎡ −−−=

210

21

21

21

21 23 αx

⎥⎦⎤

⎢⎣⎡ −−+⎥⎦

⎤⎢⎣⎡ −−−=

410

41

21

21

21

⎥⎦⎤

⎢⎣⎡ −−−=∴

43

21

433x


⎥⎦⎤

⎢⎣⎡ −=⎥⎦

⎤⎢⎣⎡−=−∇= 0

2100

210)( 33 xd f

⎥⎦⎤

⎢⎣⎡ −⋅+⎥⎦

⎤⎢⎣⎡ −−−= 0

210

43

21

43 34 αx

⎥⎦⎤

⎢⎣⎡ −+−−=

43)1(

21

43 3α

Find the next search direction:


089)1(

21)( 3

3

4

=−+= ααd

df x

23)(

23)1(

41)( 3234 −−+= ααxfFind α3:

453 =⇒α


So, x4 becomes:

⎥⎦⎤

⎢⎣⎡ −+⎥⎦

⎤⎢⎣⎡ −−−= 0

850

43

21

434x

⎥⎦⎤

⎢⎣⎡ −−−=∴

43

89

434x


⎥⎦⎤

⎢⎣⎡ −−=⎥⎦

⎤⎢⎣⎡ −−=−∇=

85

43

85

85

43

85)( 44 xd f

⎥⎦⎤

⎢⎣⎡ −−⋅+⎥⎦

⎤⎢⎣⎡ −−−=

85

43

85

43

89

43 45 αx

⎥⎦⎤

⎢⎣⎡ +−−−+−= )

253(

41)

23(

43)

253(

41 444 ααα

The next search direction:


03243

1673)( 4

4

5

=−= ααd

df x

6451

3243)(

3273)( 4245 −−= ααxf

146434 =α⇒

Find α4:


Update x5:

⎥⎦⎤

⎢⎣⎡ −−⋅+⎥⎦

⎤⎢⎣⎡ −−−=

85

43

85

14643

43

89

435x

⎥⎦⎤

⎢⎣⎡ −−−=∴

11681091

7366

116810915x


Let’s check to see if the convergence criteria is satisfied

Evaluate ||∇f(x5)||:

⎥⎦⎤

⎢⎣⎡=∇

58421

58435

58421)( 5xf

( ) ( ) ( ) 0786.058421

58435

58421)(

2225 =++=∇ xf


So, ||∇f(x5)|| = 0.0786, which is very small and we can take it to be close enough to zero for our example

Notice that the answer of

⎥⎦⎤

⎢⎣⎡ −−−=

11681091

7366

11681091x

is very close to the value of that we obtained analytically

[ ]111* −−−=x

Click to edit Master title styleQuadratic Functions

• Quadratic functions are important for the next method we will look at

• A quadratic function can be written in the form: xTQx where x is the vector of variables and Q is a matrix of coefficients

Example:[ ] )2(x x2x )(x2

210121

0122

2212

1

3

2

1

321 +−=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

−

xxx

xxx

)2(x x2x– 2332 +

Click to edit Master title styleConjugate Gradient Method

• The Conjugate Gradient Method has the property that if f(x) is quadratic, it will take exactly n iterations to converge, where n is the number of variables in the x vector

• Although it works especially well with quadratic functions, this method will work with non-quadratic functions also

Click to edit Master title styleConjugate Gradient Steps

1. Choose a starting point x0 and calculate f(x0). Let d0 = -∇f(x0)

2. Calculate x1 using:Find α0 by performing a single-variable optimization on f(x0 +α0d0) using the methods discussed earlier. (See illustration after algorithm explanation)

0001 dxx α+=


3. Calculate f(x1) and ∇f(x1). The new search direction is calculated using the equation:

)()()()()( 00

11011

xxxxdxd

fffff T

T

∇⋅∇∇⋅∇

+−∇=

This can be generalized for the kth iteration:

)()()()()(

1111

kkT

kkTkkk

fffff

xxxxdxd

∇⋅∇∇⋅∇

+−∇=++

++


4. Use either of the two methods discussed earlier to determine tolerance:

11 )()( ε<−+ kk ff xx

2)( ε<∇ kf x

Or,

Click to edit Master title styleNumber of Iterations

• For quadratic functions, this method will converge in n iterations (k = n)

• For non-quadratic functions, after niterations, the algorithm cycles again with dn+1 becoming d0.

Click to edit Master title styleStep Size for Quadratic Functions

• When optimizing the step size, we can approximate the function to be optimized in the following manner:

)()()()()()( 221 kkTkkkTkkk ffff dxddxxdx αααα ⋅∇⋅+⋅∇+≈+

• For a quadratic function, this is not an approximation – it is exact


We take the derivative of that function with respect to α and set it equal to zero:

0)()()()( 2 =⋅∇⋅+⋅∇=+ kkTkkkT

kk

ffd

df dxddxdx ααα

The solution to this equation is:

kkTk

kkToptk

ff

dxddx⋅∇⋅

⋅∇−=

)()()(

2,α


• So, for the problem of optimizing a quadratic function,

kkTk

kkToptk

ff

dxddx⋅∇⋅

⋅∇−=

)()()(

2,α

is the optimum step size.• For a non-quadratic function, this is an

approximation of the optimum step size.

Click to edit Master title styleMultivariable Newton’s Method

We can approximate the gradient of f at a point x0 by:

)()()()( 0020 xxxxx −⋅∇+∇≈∇ fff

We can set the right-hand side equal to zero and rearrange to give:

[ ] )()( 01020 xxxx ff ∇⋅∇−=−

Click to edit Master title styleMultivariable Newton’s Method

We can generalize this equation to give an iterative expression for the Newton’s Method:

[ ] )()( 121 kkkk ff xxxx ∇⋅∇−=−+

where k is the iteration number

Click to edit Master title styleNewton’s Method Steps

1. Choose a starting point, x0

2. Calculate ∇f(xk) and ∇2f(xk)3. Calculate the next x using the equation

4. Use either of the convergence criteria discussed earlier to determine convergence. If it hasn’t converged, return to step 2.

[ ] )()( 121 kkkk ff xxxx ∇⋅∇−=−+

Click to edit Master title styleComments on Newton’s Method

• We can see that unlike the previous two methods, Newton’s Method uses both the gradient and the Hessian

• This usually reduces the number of iterations needed, but increases the computation needed for each iteration

• So, for very complex functions, a simpler method is usually faster

Click to edit Master title styleNewton’s Method Example

For an example, we will use the same problem as before:

Minimize f(x1, x2, x3) = (x1)2 + x1(1 – x2) + (x2)2

– x2x3 + (x3)2 + x3

[ ]12212)( 3232121 ++−−+−+−=∇ xxxxxxxf x


The Hessian is:

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

−=∇

210121

012)(2 xf

And we will need the inverse of the Hessian:

[ ]⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

−=∇

−

−

43

21

41

2112

14

12

14

3

210121

012)(

1

12 xf


So, pick⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡=

000

0x

Calculate the gradient for the 1st iteration:

[ ]100000100)( 0 ++−−+−+−=∇ xf

[ ]101)( 0 =∇⇒ xf


So, the new x is:11 0 2 0 0( ) ( )

3 1 14 2 40 1

1 1 0 1 02 20 131 1

4 2 4

f f−

⎡ ⎤= − ∇ ⋅∇⎣ ⎦⎡ ⎤

⎡ ⎤ ⎡ ⎤⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥= − ⋅⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎣ ⎦ ⎣ ⎦

⎢ ⎥⎣ ⎦

x x x x

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−−−

=∴111

1x


Now calculate the new gradient:

[ ] [ ]000121121112)( 1 =+−+−++−=∇ xf

Since the gradient is zero, the method has converged

Click to edit Master title styleComments on Example

• Because it uses the 2nd derivative, Newton’s Method models quadratic functions exactly and can find the optimum point in one iteration.

• If the function had been a higher order, the Hessian would not have been constant and it would have been much more work to calculate the Hessian and take the inverse for each iteration.

Click to edit Master title styleConstrained Nonlinear

Optimization• Previously in this chapter, we solved NLP

problems that only had objective functions, with no constraints.

• Now we will look at methods on how to solve problems that include constraints.

Click to edit Master title styleNLP with Equality Constraints

• First, we will look at problems that only contain equality constraints:

Minimize f(x) x = [x1 x2 … xn]Subject to: hi(x) = bi i = 1, 2, …, m

Click to edit Master title styleIllustration

Consider the problem:Minimize x1 + x2

Subject to: (x1)2 + (x2)2 – 1 = 0

The feasible region is a circle with a radius of one. The possible objective function curves are lines with a slope of -1. The minimum will be the point where the lowest line still touches the circle.

Click to edit Master title styleGraph of Illustration

f(x) = 1

f(x) = 0

f(x) = -1.414

Feasible region

⎥⎦

⎤⎢⎣

⎡=

707.0707.0*x

)(xf∇

The gradient of f points in the direction of

increasing f

Click to edit Master title styleMore on the Graph

• Since the objective function lines are straight parallel lines, the gradient of f is a straight line pointing toward the direction of increasing f, which is to the upper right

• The gradient of h will be pointing out from the circle and so its direction will depend on the point at which the gradient is evaluated.

Click to edit Master title styleFurther Details

f(x) = 1

f(x) = 0

f(x) = -1.414

Feasible region

⎥⎦

⎤⎢⎣

⎡=

707.0707.0*x

)( *xh∇

)( *xf∇

⎥⎦

⎤⎢⎣

⎡=

011x

)( 1xf∇

Tangent Plane

)( 1xh∇x1

x2

Click to edit Master title styleConclusions

• At the optimum point, ∇f(x) is perpendicular to ∇h(x)

• As we can see at point x1, ∇f(x) is not perpendicular to ∇h(x) and we can move (down) to improve the objective function

• We can say that at a max or min, ∇f(x)must be perpendicular to ∇h(x)– Otherwise, we could improve the objective

function by changing position

Click to edit Master title styleFirst Order Necessary Conditions

So, in order for a point to be a minimum (or maximum), it must satisfy the following equation:

This equation means that ∇f(x*) and ∇h(x*)must be in exactly opposite directions at a minimum or maximum point

0)()( *** =∇⋅+∇ xx hf λ

Click to edit Master title styleThe Lagrangian Function

To help in using this fact, we introduce the Lagrangian Function, L(x,λ):

)()(),( xxx hfL ⋅+= λλ

Review: The notation ∇x f(x,y) means the gradient of f with respect to x.

So, )()(),( xxx xxx hfL ∇⋅+∇=∇ λλ


• So, using the new notation to express the First Order Necessary Conditions (FONC), if x* is a minimum (or maximum) then

( ) 0),( ** ,=∇

λλ

xx xL

and to ensure feasibility.0)( * =xh


• Another way to think about it is that the one Lagrangian function includes all information about our problem

• So, we can treat the Lagrangian as an unconstrained optimization problem with variables x1, x2, …, xn and λ1, λ2, …, λm.

We can solve it by solving the equations

0x=

∂∂L

& 0λ=

∂∂L

Click to edit Master title styleUsing the FONC

Using the FONC for the previous example,

)()(),( xxx hfL ⋅+= λλ( ) ( )( )12

22

121 −+⋅++= xxxx λ

⎥⎦

⎤⎢⎣

⎡=

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

∂∂

∂∂

=∇00

),(2

1

xL

xL

Lx λx

And the first FONC equation is:

Click to edit Master title styleFONC Example

This becomes:

&

021 11

=+=∂∂ xxL λ

021 22

=+=∂∂ xxL λ

The feasibility equation is:

or,

( ) ( ) 0122

21 =−+ xx

( ) ( ) 0122

21 =−+=

∂∂ xxLλ

Click to edit Master title styleFONC Example

So, we have three equations and three unknowns.

When they are solved simultaneously, we obtain

707.021 ±== xx & 707.0m=λ

We can see from the graph that positive x1 & x2 corresponds to a maximum while negative x1 & x2 corresponds to the minimum.

Click to edit Master title styleFONC Observations

• If you go back to the LP Chapter and look at the mathematical definition of the KKT conditions, you may notice that they look just like our FONC that we just used

• This is because it is the same concept• We simply used a slightly different

derivation this time but obtained the same result

Click to edit Master title styleLimitations of FONC

• The FONC do not guarantee that the solution(s) will be minimums/maximums.

• As in the case of unconstrained optimization, they only provide us with candidate points that need to be verified by the second order conditions.

• Only if the problem is convex do the FONC guarantee the solutions will be extreme points.

Click to edit Master title styleSecond Order Necessary

Conditions (SONC)For where

and for y where

)()(),( 222 xxx hfLx ∇⋅+∇=∇ λλ),(2 λxLx∇

0y

x

xyxJ

x

=⋅

⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢

⎣

⎡

∂∂

∂∂

=⋅

)(

1

*

*

)(m

hh

h

M

If x* is a local minimum, then 0),( *2 ≥⋅∇⋅ yxy λLxT

Click to edit Master title styleSecond Order Sufficient Conditions

(SOSC)

• y can be thought of as being a tangent plane as in the graphical example shown previously– Jh is just the gradients of each h(x) equation

and we saw in the example that the tangent plane must be perpendicular to ∇h(x) and that is why

0),( *2 >⋅∇⋅ yxy λLxT

0yJ =⋅)(xh

Click to edit Master title styleThe y Vector

x1

x3

x2

Tangent Plane(all possible y vectors))(xh∇

0)( =xh

*x

The tangent plane is the location of all yvectors and intersects with x*

It must be orthogonal (perpendicular) to ∇h(x)

Click to edit Master title styleMaximization Problems

• The previous definitions of the SONC & SOSC were for minimization problems

• For maximization problems, the sense of the inequality sign will be reversed

For maximization problems:SONC:

SOSC:

0),( *2 ≤⋅∇⋅ yxy λLxT

0),( *2 <⋅∇⋅ yxy λLxT

Click to edit Master title styleNecessary & Sufficient

• The necessary conditions are required for a point to be an extremum but even if they are satisfied, they do not guarantee that the point is an extremum.

• If the sufficient conditions are true, then the point is guaranteed to be an extremum. But if they are not satisfied, this does not mean that the point is not an extremum.

Click to edit Master title styleProcedure

1. Solve the FONC to obtain candidate points.

2. Test the candidate points with the SONC– Eliminate any points that do not satisfy the

SONC3. Test the remaining points with the SOSC

– The points that satisfy them are min/max’s – For the points that do not satisfy, we cannot

say whether they are extreme points or not

Click to edit Master title styleProblems with Inequality

ConstraintsWe will consider problems such as:Minimize f(x)Subject to: hi(x) = 0 i = 1, …, m

& gj(x) ≤ 0 j = 1, …, pAn inequality constraint, gj(x) ≤ 0 is called “active” at x* if gj(x*) = 0.Let the set I(x*) contain all the indices of the active constraints at x*:

0)( * =xjg for all j in set I(x*)

Click to edit Master title styleLagrangian for Equality &

Inequality Constraint ProblemsThe Lagrangian is written:

We use λ’s for the equalities & μ’s for the inequalities.

∑∑==

⋅+⋅+=p

jjji

m

ii ghfL

11)()()(),,( xxxμλx μλ

Click to edit Master title styleFONC for Equality & Inequality

ConstraintsFor the general Lagrangian, the FONC

become

∑∑==

=∇⋅+∇⋅+∇=∇p

jjji

m

ii ghfL

1

*

1

* *)(*)(*)(*)*,*,( 0xxxμλx μλ

and the complementary slackness condition:

[ ] ,0)( ** =⋅ xjj gμ,0* ≥jμ pj K,1=

Click to edit Master title styleSONC for Equality & Inequality

ConstraintsThe SONC (for a minimization problem) are:

where as before.

0yxy x ≥⋅∇⋅ ),,( ***2 μλLT

0yxJ =⋅)( *

This time, J(x*) is the matrix of the gradients of all the equality constraints and only the inequality constraints that are active at x*.

Click to edit Master title styleSOSC for Equality & Inequality

Constraints• The SOSC for a minimization problem with

equality & inequality constraints are:

0yxy >⋅∇⋅ ),,( ***2 μλLxT

Click to edit Master title styleGeneralized Lagrangian Example

• Solve the problem:Minimize f(x) = (x1 – 1)2 + (x2)2

Subject to: h(x) = (x1)2 + (x2)2 + x1 + x2 = 0g(x) = x1 – (x2)2 ≤ 0

The Lagrangian for this problem is:

( ) ( ) ( ){ }2122

12

22

1 2)1(),,( xxxxxxL +++⋅++−= λμλx

( ){ }221 xx −⋅+ μ


• The first order necessary conditions:

( ) 0212 111

=++⋅⋅+−=∂∂ μλλ xxxL

0222 2222

=⋅⋅−+⋅⋅+=∂∂ xxxxL μλλ

( ) ( ) 0212

22

1 =+++=∂∂ xxxxLλ

( )( ) 0221 =−⋅ xxμ


Solving the 4 FONC equations, we get 2 solutions:

⎥⎦

⎤⎢⎣

⎡−

=4534.0

2056.0)1(x 9537.0=μ45.0=λ &

and ⎥⎦

⎤⎢⎣

⎡=

00)2(x 2−=μ0=λ &2)

1)


Now try the SONC at the 1st solution:Both h(x) & g(x) are active at this point (they both equal zero). So, the Jacobian is the gradient of both functions evaluated at x(1):

( )( ) ⎥⎦

⎤⎢⎣

⎡−−

=⎥⎦

⎤⎢⎣

⎡−

++=

9068.010932.0411.1

211212

)1(2

211

x

xJx

xx


The only solution to the equation:0yxJ =⋅)( )1(

is: ⎥⎦

⎤⎢⎣

⎡=

0000

y

And the Hessian of the Lagrangian is:

⎥⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡−+

+=∇

993.0009.2

2220022

)1(

2

x

x μλλ

L


So, the SONC equation is:

0≥⎥⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡⋅⎥⎦

⎤⎢⎣

⎡⋅⎥⎦

⎤⎢⎣

⎡0000

0000

993.0009.2

0000

This inequality is true, so the SONC is satisfied for x(1) and it is still a candidate point.


The SOSC equation is:

And we just calculated the left-hand side of the equation to be the zero matrix. So, in our case for x2:

0yxy x >⋅∇⋅ ),,( ***2 μλLT

0yyxx >⎥

⎦

⎤⎢⎣

⎡=⋅∇⋅

0000

)1(

2LT

So, the SOSC are not satisfied.


For the second solution:Again, both h(x) & g(x) are active at this point. So, the Jacobian is:

( ) ⎥⎦

⎤⎢⎣

⎡−

=⎥⎦

⎤⎢⎣

⎡−

++=

0111

211212

)2(2

21)2(

x

xJx

xx


The only solution to the equation:0yxJ =⋅)( )2(

is: ⎥⎦

⎤⎢⎣

⎡=

0000

y

And the Hessian of the Lagrangian is:

⎥⎦

⎤⎢⎣

⎡−

=⎥⎦

⎤⎢⎣

⎡−+

+=∇

2002

2220022

)2(

2

x

x μλλ

L


So, the SONC equation is:

0≥⎥⎦

⎤⎢⎣

⎡=⎥

⎦

⎤⎢⎣

⎡⋅⎥⎦

⎤⎢⎣

⎡−

⋅⎥⎦

⎤⎢⎣

⎡0000

0000

2002

0000

This inequality is true, so the SONC is satisfied for x(2) and it is still a candidate point


The SOSC equation is:

And we just calculated the left-hand side of the equation to be the zero matrix. So, in our case for x2:

0yxy x >⋅∇⋅ ),,( ***2 μλLT

0yyxx >⎥

⎦

⎤⎢⎣

⎡=⋅∇⋅

0000

)2(

2LT

So, the SOSC are not satisfied.

Click to edit Master title styleExample Conclusions

• So, we can say that both x(1) & x(2) may be local minimums, but we cannot be sure because the SOSC are not satisfied for either point.

Click to edit Master title styleNumerical Methods

• As you can see from this example, the most difficult step is to solve a system of nonlinear equations to obtain the candidate points.

• Instead of taking gradients of functions, automated NLP solvers use various methods to change a general NLP into an easier optimization problem.

Click to edit Master title styleExcel Example

Let’s solve the previous example with Excel:

Minimize f(x) = (x1 – 1)2 + (x2)2

Subject to: h(x) = (x1)2 + (x2)2 + x1 + x2 = 0g(x) = x1 – (x2)2 ≤ 0


We enter the objective function and constraint equations into the spreadsheet:

Variables:x1 x20 0

Value LimitObjective Function: =(A3-1)^2+B3^2

Constraint 1: =A3^2+B3^2+A3+B3 0Constraint 2: =A3-B3^2 0


Now, open the solver dialog box under the Tools menu and specify the objective function value as the target cell and choose the Min option. As it is written, A3 & B3 are the variable cells. And the constraints should be added – the equality constraint and the ≤ constraint.


The solver box should look like the following:


• This is a nonlinear model, so unlike the examples in the last chapter, we won’t choose the “Assume Linear Model” in the options menu

• Also, x1 & x2 are not specified to be positive, so we don’t check the “Assume Non-negative” box

• If desired, the tolerance may be decreased to 0.1%


• When we solve the problem, the spreadsheet doesn’t change because our initial guess of x1 = 0 & x2 = 0 is an optimum solution, as we found when we solved the problem analytically.


However, if we choose initial values of both x1 & x2 as -1, we get the following solution

Click to edit Master title styleConclusions

• So, by varying the initial values, we can get both of the candidate points we found previously

• However, the NLP solver tells us that they are both local minimum points

Click to edit Master title styleReferences

Material for this chapter has been taken from:

• Optimization of Chemical Processes 2nd

Ed.; Edgar, Thomas; David Himmelblau; & Leon Lasdon.

m15 part2 tier1 nonlinear - polytechnique montréal · 2006-10-23 · multivariable unconstrained...

Documents