By Liyun ZhangAug.9th,2012

*Optimization for Nonlinear

Programming

*Topic

*Method for Single Variable Unconstraint Optimization :

1. Quadratic Interpolation method

2. Golden Section method

*Method for Multivariable Constraint Optimization Method

1. Lagrange Multiplier Method

2. Exterior-point Penalty Method

*Quadratic Interpolation

*Solve for: unconstraint optimization problems with single

variable and unimodal function

*Advantage and Disadvantage:

converges fast but UNRILIABLE

(you will see it in a later example)

*`

target function: f(x)

select three points x0, x1, x2

on f(x) such that:

x0<x1<x2,

f2=min(f0, f1, f2)

there must exist a quadratic

function:

q(x)=a0+a1x+a2x^2

which connects these points

such that:

q(x0)=a0+a1x0+a2x0^2=f(x0)

q(x1)=a0+a1x1+a2x1^2=f(x1)

q(x2)=a0+a1x2+a2x2^2=f(x2)

*Mathematical Background:

*Mathematical Background(cont.):

suppose (x3,f3) is the minimum point of q(x), then

q’(x)=0, or x3= -

we can solve that

a1=

a2= -

so

x3=

( x 1^2− x2^2) f 0+( x 2^2− x0^2) f 1+( x 0^2− x1^2) f 2( x 0−x 1)( x1− x2)( x2−x 0)

Initialization:Determine x0, x1 and x2 which is known to contain the minimum of the function f(x).Step 1If x3-x1<ε (a sufficiently small number), then the minimum occurs at xo=x3 and stop iteration, else go to Step 2.Step 2 Evaluate x3 and x1If x3<x1, go to Step 3, else go to Step 4. Step 3Evaluate f(x3) and f(x1)If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x0 x2=x1 x1=x3If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x3 x1=x1 x2=x2

*Algorithm:

*Algorithm(cont.):

Step 4

Evaluate f(x3) and f(x1)

If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5

x0=x1

x1=x3

x2=x2

If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5

x0=x0

x1=x1

x2=x3

Step 5

Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3. code

illustration

*Application and Comparison:

Example 1.

Find the minimum point and value of the function f(x)=(x^2-2)^2/2-1Solution:

f1001=inline('(x.*x-2).^2/2-1','x');

a=0;

b=5;

TolX=1e-5;

TolFun=1e-8;

MaxIter=100;

[xoq,foq]=Opt_Quadratic(f1001,[a,b],TolX,TolFun,MaxIter)

xoq =

1.414187140007763

foq =

-0.999999997207487

Fminbnd method attempts to solve the optimum of one variable function within a fixed interval.

The most common syntaxes are:

[x, f]=fminbnd (fun, x1, x2)

[x, f]=fminbnd (fun, x1, x2, options)

In Ex. 1, we have

x1=0

x2=5

and the result is:

x =

1.414214286263162

f =

-0.999999999997904

*Application and Comparison(cont.):

*Golden Section Method

*Solve for: unconstraint optimization problems with single variable and

unimodal function

*Advantages: 1. more reliable than quadratic interpolation method

2. still converges fast since:

a. rely on interval approximation instead of derivative

calculations.

b. the rate of the convergence is fixed to avoid the wider

interval being used many times which may slow down the rate

of convergence.

c. interior points are selected to reduce the bounds as quickly as

possible.

*Mathematical Background:

Select the intermediate points x3 and x4:

c/a=a/b

c/(b-c)=a/b

so

k=a/b=(1+ )/2x3=x1+1/(k+1)*(x2-x1)

x4=x1+k/(k+1)*(x2-x1)

*Algorithm:

Initialization:

Determine x1 and x2 which is known to contain the minimum of the function f(x).

Step 1

Determine two intermediate points x3 and x4 such that

x3=x1+(1-r)*h

x4=x1+r*h

where

r=k/(k+1)= ( -1)/2

h=x2-x1

Step 2

If x4-x3<ε (a sufficiently small number), then the minimum occurs at xo=x3 (f(x3) < f(x4)) or xo=x4 (f(x3) > f(x4)) and stop iterating, else go to Step3.

Step3

Evaluate f(x3) and f(x4)

If f(x3)<f(x4), then determine new x1 and x2 as shown following, and go to Step4.

x1=x1

x2=x4

If f(x3)>f(x4), then determine the new x1 and x2 as shown following, and go to Step4.

x1=x3

x2=x2

Step 4

Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3, x4.

code

illustration

Example 2.

Find the minimum point and value of the function x-(x^2-2)^3/2, x∈[0,4]Solution:f1002=inline('x-(x.*x-2).^3/2','x');

 a=0;

 b=4;

 TolX=1e-4;

TolFun=1e-4;

MaxIter=100;  

[xog,fog]=Opt_Golden(f1002,a,b,TolX,TolFun,MaxIter)

xog =

3.999999954235401

fog =

-1.367999892407431e+003

 [xoq,foq]=Opt_Quadratic(f1002,[a,b],TolX,TolFun,MaxIter)

xoq =

1.938216986786059

foq =

-0.772297597277559

*Application and Comparison

*Application and Comparison(cont.):

* Apply primarily on:

optimization problems with multivariable

functions and equality constraints

* Mathematical Backgrounds:

Consider the function z=f(x,y) which is subject to ϕ(x,y)=0 or

y=g(x), suppose x0 is the minimum point of z=f(x,g(x)), then

x0 is the solution of the following equation system:

f’x+ λ ϕ’x=0

f’y+ λ ϕ’y=0

ϕ(x,y)=0

More generally, we can introuduce a transformed function

L(x1,x2…xm; λ1,λ2…λn)=f(x1,x2…xm)+ ∑λn ϕn(x1,x2…xn)

and find all x satisfying the following equation system

f’xi+ ∑ λn ϕn’xi=0 (i=1,2…m)

ϕk(x1,x2…xm) =0 (k=1,2…n)

substitute {x} back into f(x1,x2…xm) and then compare their value

*Lagrange Multiplier Method

*Algorithm and comparisonExample 3.

Find the minimum point and value of the function f(x1,x2)=x1+x2, s.t x1^2+x2^2=2 Solution: code

>> [xo,yo,fo]=Lag()

xo =

-1

1

yo =

-1

1

fo =

-2

2

so minimum is -2 at (-1,-1)

*Exterior-point Penalty Method* Apply primarily on:

optimization problems with multivariable functions and equality or

inequality constraints, and the initial guess point is out of the feasible

domain.

* Advantage: work well when the initial guess point is out of the domain so that the

computation is subjected to less restrictions compared with the interior-

point penalty method.

Consider the function f(x) which is subject to a series of equality or inequality constraints Ci(x)>=0(i=1,2…m),to find the minimum point of f(x), we combine these conditions into a single function:

F(x)=f(x)+c*∑g(Ci(x)).

penalty coefficients: ci=c1*p^(i-1) where p>=2

penalty function: g(Ci(x))=min(0, Ci(x))^2

Note that the optimum point of the unconstrained function will

eventually converge to the original constrained one when

c*∑g(Ci(x))<=ɛ.

One way to search for the optimum point of the unconstrained

function is the Newton method, which resorts to the derivative method.

*Mathematical Background

*Algorithm:

Initialization

Set up the initial optimum guess point x0, c1,p(p>=2), accuracy ɛ>0, and let k=1.

Step 1

Combine the given conditions into a new function F(x)=f(x)+ci*(∑hp(x)^2+∑gs(x)^2),

hp(x)=0 (p=1,2…m)

gs(x)>=0 (s=1,2…n)

Step 2

Search for the minimum of F(x) from the point xk-1 with unconstraint optimization methods.

Step 3

Let S(xk)= ck* (∑hp(x)^2+∑gs(x)^2), if S(xk)<= ɛ, stop iteration, else let k=k+1 and go to step 2

code

* Application and ComparisonExample 4.

Find the minimum point and value of the function f(x1,x2)=x1^2-x1*x2+x2-x1+1, s.t 2x1+3x2-9=0x2^2+x1^2-6>=0x1>=0 x2>=0

Solution:

let c1=0.05, p=2, (x1,x2)=(2,2)

>> syms x1 x2;

>> f=x1^2-x1*x2+x2-x1+1;

>> g=[x1^2+x2^2-6;x1;x2]; h=[2*x1+3*x2-9];

>> [x,minf]=minEPF(f,[2 2],g,h,0.05,2,[x1 x2])

>> x= 1.4000

2.0668

minf= 0.7333

Fmincon attempts to solve constraint optimization problems of the form:

min f(x) subject to: A*x<=b, Aeq*x=beq (linear constraints)

x subject to: c(x)<=0, ceq(x)=0 (nonlinear constraints)

lb<=x<=ub (bounds)

The common syntax for fmincon is

[x,fval]=fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon)

where x0 is the initial guess.

In Ex. 4, we have

>> A=[-1 0; 0 -1];

>> b=[0; 0];

>> Aeq=[2 3];

>> beq=[9];

and the result is

x =

1.399999998247254 2.066666667835164

fval=

0.733333333333334