12. concept of constrained optimum design (kuhn-tucker...

Hae-Jin ChoiSchool of Mechanical Engineering,

Chung-Ang University

12. Concept of Constrained Optimum Design

(Kuhn-Tucker necessary conditions)

SCHOOL OF MECHANICAL ENG.

CHUNG-ANG UNIVERSITY-1-

Optimization with equality constraints

Lagrange Multiplier

Necessary condition

Optimization with inequality constraints

Kuhn-Tucker (K-T) necessary condition

Global optimality condition

Sufficient condition for general optimization problem

DOE and Optimization

Constrained Optimization



Necessary condition for constrained minimization problem with only

equality constraints.

Let assume a optimum problem of

Minimization of an objective function, f(x1, x2).

With an equality constraint, h(x1, x2)=0.

The equality constraint, h(x1, x2)=0, may be reformatted as

The constrained problem may be converted to an unconstrained problem

Minimization of an converted objective function,


Optimization with Equality Constraints

2 1( )x x

1 1( , ( ))f x x



However, in many case, an explicit form, , of an equality

constraint is not available; therefore,

We introduce Lagrange Multipliers

Using the chain rule,

or

At an optimum point (x1*, x2*), the necessary condition is


Lagrange Multipliers

2 1( )x x

1 2 1 2 2

1 1 2 1

( , ) ( , )f x x f x x dxdf

dx x x dx

1 2 1 2

1 1 2 1

( , ) ( , )f x x f x xdf d

dx x x dx

* * * * * *

1 2 1 2 1 2

1 1 2 1

( , ) ( , ) ( , )0

df x x f x x f x x d

dx x x dx(1)



Since is unknown, we need to eliminate it from the equation, to

eliminate it, we differentiate the constraint equation, h(x1, x2)=0, at the

point (x1*, x2*)

Replacing in Eq. (1) with Eq. (2)



1( )x

* * * * * *

1 2 1 2 1 2

1 1 2 1

* *

1 2 1

* *

1 1 2 2

( , ) ( , ) ( , )0

( , ) /

( , ) /

dh x x h x x h x x d

dx x x dx

or

h x x xd

dx h x x x(2)

1

d

dx

* * * * * *

1 2 1 2 1 2 1

* *

1 2 1 2 2

( , ) ( , ) ( , ) /0

( , ) /

f x x f x x h x x x

x x h x x x(3)



Defining Lagrange Multiplier as

Eq.(3) becomes

Rearranging Eq. (4)

Solving Eq.(5),Eq. (6), and h(x1, x2)=0, simultaneously for x1, x2 and v

Here, v is an Lagrange Multiplier



* *

1 2 2

* *

1 2 2

( , ) /

( , ) /

f x x xv

h x x x(4)

* * * *

1 2 1 2

1 1

( , ) ( , )0

f x x h x xv

x x(5)

* * * *

1 2 1 2

2 2

( , ) ( , )0

f x x h x xv

x x(6)


CHUNG-ANG UNIVERSITY-6-DOE and Optimization


Problem definition

Find

Satisfy

( ) 0 ; 1

Minimize

( )

i

Given

h i to p

f

x

x

x

Constrained Problem Unconstrained Problem

1

Problem definition

Find

,

Satisfy

...

Minimize

( , ) ( ) ( ) ( )p

T

j j

j

Given

L f v h f

x v

x v x x x v h(x)



Objective function with Lagrange multiplier

Necessary condition in terms of a Lagrange function defined above is


Necessary Condition of Lagrange Function

* ** *

* **

( )( ) 0 or 0; 1

( )( ) 0 ; 1

i

j

j

LL i to n

x

and

Lh j to p

v

x , vx , v

x , vx

1

( , ) ( ) ( ) ( )p

T

j j

j

L f v h fx v x x x v h(x)



Example 1: Lagrange Multiplier

1 2

1 2 1 2

2 2

1 2 1 2

Problem statement

,

( , ) 2 0

( , ) ( 1.5) ( 1.5)

Given

Find

x x

Satisfy

h x x x x

Minimize

f x x x x

1 2 1 2 1 2

2 2

1 2 1 2

1

1

2

2

1 2

The Lagrange function with Lagrange multipler

( , , ) ( ) ( , )

( 1.5) ( 1.5) ( 2)

2( 1.5) 0 (1)

2( 1.5) 0 (2)

2 0 (3)

Solvin

L x x v f x x vh x x

x x v x x

Lx v

x

Lx v

x

Lx x

v

1 2

g Eqs. (1)-(3) simultaneously

1, 1 and 1.x x v



Geometrical meaning of Lagrange Multipliers

* *( ) 0

Rearraging

L v

v

x , v f h

f h

At candidate minimum point,

gradient of the cost and constraint

functions are along the same line

and proportional to each other.

The Lagrange multiplier v is the

proportionality constant



Cylindrical tank design. The problem is to find radius R and length l of

the cylinder to minimize surface area while achieving the storage volume

V.


Example 2: Cylinder Tank Design

2

1

Assumption of thin sheet metal

V = given volume to achieve

=Diameter of cylindrical storage (cm)

=length of cylindrical storage (cm)

g ( , ) 0

Given

Find

R

l

Satisfy

R l R l V

2 ( , )

Minimize

f D H R Rl

2 2

2

2

( )

2 2 0

0

0

L R Rl v R l V

LR l vRl

R

LR v R

l

Lh R l V

v

1/3

*

1/3

*

1/3

*

2

2

4

1 2

Solving

VR

Vl

vR V

or Newton-Raphson method



The design problems formulated in Lecture 9 module often included

inequality constraints of the form

We can transform an inequality constraint to an equality by adding a new

variable to it, called the slack variable, which must always be

nonnegative (i.e., positive or zero)

To avoid additional constraint ,


Necessary Condition for Inequality Constraints

( ) 0; 1 ig i to m x

( ) 0 where 0i i ig s s x

0is

2( ) 0i ig s x



The design problems formulated in Lecture 9 module often included

inequality constraints of the form

We can transform an inequality constraint to an equality by adding a new

variable to it, called the slack variable, which must always be

nonnegative (i.e., positive or zero)

To avoid additional constraint ,

Now, this form can be used for Lagrange multiplier to find necessary

condition



( ) 0; 1 ig i to m x

( ) 0 where 0i i ig s s x

0is 2( ) 0i ig s x



New equations with the necessary condition,

for the Lagrangian L to be stationary with respect to the slack variables

Lagrange multipliers, u, of inequality conditions must be non-negative.

If the constraint is inactive at the optimum, its associated Lagrange

multiplier is zero.

If it is active then associated multiplier must be non-negative.



/ 0iL s

* 0; 1 ju j to m



Resolving Example 1 with an inequality constraint


Example 3: Resolving Example 1

2 2 2

1 2 1 2

1

1

2

2

2

1 2

( 1.5) ( 1.5) ( 2 )

2( 1.5) 0

2( 1.5) 0

2 0

2 0

L x x u x x s

Lx u

x

Lx u

x

Lx x s

u

Lus

s

1 2

1 2 1 2

2 2

1 2 1 2

Problem statement

,

( , ) 2 0

( , ) ( 1.5) ( 1.5)

Given

Find

x x

Satisfy

h x x x x

Minimize

f x x x x



Example 3: Resolving Example 1

2 2 2

1 2 1 2

1

1

2

2

2

1 2

( 1.5) ( 1.5) ( 2 )

2( 1.5) 0

2( 1.5) 0

2 0

2 0

L x x u x x s

Lx u

x

Lx u

x

Lx x s

u

Lus

s

The equations are nonlinear and they can

have many roots

(Sol 1) Setting s=0 to satisfy 2us=0,

x1=x2=1, u=1 and s=0

(Sol 2) Setting u=0 to satisfy 2us=0,x1=x2=1.5, u=0 , and s2=-1

This is not a valid solution since g=- s2>0

Note: the necessary condition u≥0 insures that the gradients of the

cost and the constraint functions point in opposite directions.

This way f cannot be reduced any further by stepping in the

negative gradient direction without violating the constraints



The equality and inequality constraints can be summed up in what are

commonly known as the Kuhn-Tucker (K-T) necessary conditions


Kuhn-Tucker (K-T) Necessary Conditions

2

1 1

( , , , ) ( ) ( ) ( ( ) )

p m

i i j j j

i j

L x v u s f x v h x u g x s

T T 2f(x) + v h(x) +u (g(x) + s )

* *

1 1

2

*

*

0; 1

( ) 0; 1

( ) 0; 1

0; 1

0; 1

all derivatives ar

p mji

i j

i jk k k k

i

j j

j j

j

ghL fv u k to n

x x x x

h i to p

g s j to m

u s j to m

u j to m

*

*

x

x

e evaluated at point *x

(n+p+2m) unknowns and equations



Negative gradient direction (steepest descent direction) for the objective

function is a linear combination of the gradients of constraints with Lagrange

multipliers.

With m inequality constraints, the switching condition (ujsj=0) lead to 2m

distinct solution cases

Evolution of si essentially implies evaluation of the constraints function gi(x)

since si2=-gi(x).

If an inequality constraint gi(x)≤0 is inactive at the candidate minimum point x*, (i.e., gi(x*)<0 or

si2>0), then the corresponding Lagrange multiplier, ui

*=0 .

If it is active (i.e., gi(x*)=0 or si2=0), then ui

*≥0


Kuhn-Tucker (K-T) Necessary Conditions

* *

1 1

; 1 p m

jii j

i jk k k

ghfv u k to n

x x x



Example 4: K-T necessary conditions

3 2

0

0

1 1Minimize ( ) ( )

3 2

Subject to

where

0 , and are constants)

f x x b c x bcx f

a x d

a b c d fa b c d

1

2

0

0

g a x

g x d

Four cases

3 2 2 2

0 1 1 2 2

2

1 2

2

1

2

2

1 1 2 2

1 2

1 1( ) ( ) ( )

3 2

K-T necessary condition

( ) 0

( ) 0

( ) 0

0, 0

0, 0

L x b c x bcx f u a x s u x d s

Lx b c x bc u u

x

a x s

x d s

u s u s

u u




Satisfy sufficient

condition

c is the local

minimum

1 2

2 2

1 2

2 2

1 2

2

2

2

2

Case 1: u 0, u 0, which give

( )( ) 0

At

0, 0 (Valid)

At

0, 0 (Valid)

Checking sufficient condtion

2 ( )

0 at

The cost funct

x b x c

x b

s b a s d b

x c

s c a s d c

d fx b c

dx

d fx c

dx

ion is a local minimum at x=c




Satisfy K-T condition

Candidate minimum point

A feasible move from the point

increases the cost function

1 2

2

1 2

1

1 2

Case 2: 0, 0

(Point D)

( )( ) 0 (Not Valid)

Case 3: 0, 0

(Point A)

( )( ) 0 (Valid)

Case 4: s 0,s 0

and , which is not possible (Not Valid)

u s

x d

u d c d b

s u

x a

u a b a c

x a x d

a b c d




2 2

1 2 1 2

2 2

1 2

Minimize

( ) -3

Subject to

( ) 6 0

f x x x x

g x x

x

x

2 2 2 2 2

1 2 1 2 1 2

1 2 1

1

2 1 2

2

2 2 2

1 2

-3 ( 6 )

2 3 2 0

2 3 2 0

6 0

0

0

L x x x x u x x s

Lx x ux

x

Lx x ux

x

x x s

us

u

(Sol) Lagrange function is

Case 1: u=0

1 2

2 1

2

1 2

2 3 0

2 3 0

0, 0, 6 (valid point)

x x

x x

x x u s

Case 2: s=0

1 2

2 1

2 2

1 2

1 2

1 2

1 2

1 2

Solving three equations for , , and simultaneously

1 3 / 2

Substituting for u

13,

2

13,

2

53, (Not valid)

2

53, (Not valid)

2

x x u

u x x

x x

x x u

x x u

x x u

x x u




Case 3: u=s=0

All K-T conditions cannot be

satisfied (not valid)

Point A and B satisfy the sufficient condition for local minima.

Any feasible move from the points results in an increase in the cost, and any further

reduction in the cost results in violation of the constraint.

Point 0 is a saddle point (stationary point)

x1 x2 u f

0 0 0 0

1/2 -3

1/2 -3

3 3

3 3

Point 0

Point A

Point B



Example 6: K-T necessary condition

2 2

1 2 1 2 1 2

1 1 2

2 1 2

Minimize

( , ) 2 2 2

Subject to

2 4 0

2 4 0

f x x x x x x

g x x

g x x

1 1 2

1

2 1 2

2

2

1 1 2 1

2

2 1 2 2

1 1 2 2

1 2

K-T necessary condtions are

2 2 2 0

2 2 2 0

2 4 0

2 4 0

0, 0,

0, 0

Lx u u

x

Lx u u

x

g x x s

g x x s

u s u s

u u

2 2

1 2 1 2

2 2

1 1 2 1 2 1 2 2

2 2 2

( 2 4 ) ( 2 4 )

L x x x x

u x x s u x x s

1 2

1 2 2

1 1 2

1 1 2 2

Four cases

1. 0, 0

2. 0, 0 ( 0)

3. 0 ( 0), 0,

4. 0 ( 0), 0 ( 0)

u u

u s or g

s or g u

s or g s or g




1 2

2 2

1 2 1 2

Case 1: 0

This gives

1, 1, 1

(Not Valid)

u u

x x and s s

1 2

1 2

2 2

1 2

1 2 1 2

2

1

Case 2: u =0, s =0

2 - 2 - 0

2 - 2 - 2 0

- - 2 4 0

Solving for the unknowns

1.2; 1.4; 0; 0.4; 0.2

This will give s 0.2 0 (Not Valid)

x u

x u

x x

x x u u f

1 2

1 1

2 1

1 1

1 2 1 2

2

2

Case 3: s 0, 0

2 - 2 - 2 0

2 - 2 - 0

-2 - 4 0

Solving for the unknowns

1.4; 1.2; 0.4; 0; 0.2

This will give s 0.2 0 (Not Valid)

u

x u

x u

x x

x x u u f

1 2

1 1 2

2 1 2

1 2

1 2

1 2 1 2

Case 4: s 0,s 0

Solving the following equations for the unknowns

2 2 2 0

2 2 2 0

2 4 0

2 4 0

4 4 2 2, , , (Candidate of local minimum)

3 3 9 9

x u u

x u u

x x

x x

x x u u




Given

Problem definition

Find

x, y

Subject to

g1=x+y-12≤0,

g2=x-8 ≤0

Minimize

f(x,y)=(x-10)2+(y-8)2

Find the candidates of local minima applying K-T necessary conditions


HOMEWORK



Optimization with equality constraints

Lagrange Multiplier

Necessary condition

Optimization with inequality constraints

Kuhn-Tucker (K-T) necessary condition

Global optimality condition

Sufficient condition for general optimization problems


Constrained Optimization

12. concept of constrained optimum design (kuhn-tucker...

Documents