part 4 nonlinear programming 4.3 successive linear programming
TRANSCRIPT
Part 4 Nonlinear Programming
4.3 Successive Linear Programming
Basic Concept
0 0 0 0;f f f f
y y y y y y y
Approach 1: Direct Use of Linear Programs
The simplest and most direct use of the linearization construction is to replace the general nonlinear problem with a complete linearization of all problem functions at some selected estimate solution.
The linearized problem takes the form of a linear program and can be solved as such.
Case 1.1 The linearly constrained case
min
. .
Because ( ) is nonlinear, the optimal solution no
longer needs
f
s t
f
y
Ay b
y 0
y
to be confined to corner points of the
feasible region but can lie anywhere within the region.
Case 1.1The approximate problem
0min ;
. .
f
s t
y y
Ay b
y 0
*
*
0 0 * 0
0 0 0 * 0
0 * 0
0
How is , the solution of the approximate problem,
related to , the solution of the original problem?
Note,
; ;
or
0
Since is in the direction of steepest
f f
f f f
f
f
y
y
y y y y
y y y y y
y y y
y
* 0
ascent, the
vector is in a descent direction on the surface
of .f
y y
y
Bounded Line Search
*
0
0
Since is a corner point of the feasible region,
and since is feasible, all points on the line between
the two will be feasible.
The line search can be confined to the bounded line
segment,
y
y
y y
* 0 where 0 1 y y
Frank-Wolfe Algorithm
( )
( )
( )
Step 1: Calculate
Step 2: Solve the LP subproblem
min
. .
Let be the optimal solution.
k
k
k
f
f
s t
y
y z
Az b
z 0
z
( )
( ) ( ) ( )
0 1
( 1) ( ) ( ) ( ) ( )
( 1)
( 1) ( )
( 1)
( 1) ( )
( 1)
Step 3: Find which solves
min
Step 4: Calculate
Step 5: Convergence check.
Otherwise, go t( ) ( )
( )
k
k k k
k k k k k
k
k k
k
k k
k
f
f
f f
f
y z y
y y z y
y
y y
y
y y
y
o step 1.
Remark
( ) ( ) ( )
( ) ( ) ( )
( )
( ) ( ) ( )
If the objective function is convex, then
for all feasible points. This implies
min min ; ; ...lower bound
It is also clear that
min
Thus, ; g
k k k
k k k
k
k k k
f f f
f f f
f f
f f
y y y y y
y y y z y
y y
y z y
ives an estimate of the
improvement in objective function.
Case 1.2The general LP case
min
. .
0 1, 2, ,
0 1, 2, ,
1, 2, ,
j
k
i i i
f
s t
g j J
h k K
U x L i N
y
y
y
Direct Linear Approximation
( ) ( ) ( )
( ) ( ) ( )
( ) ( ) ( )
( )
min
. .
0 1, 2, ,
0 1, 2, ,
1, 2, ,
Note that even if is a feasible point of the original
nonlinear proble
t t t
t t tj j
t t tk k
i i i
t
f f
s t
g g j J
h h k K
U x L i N
y y y y
y y y y
y y y y
y
( 1)m, there is no assurance that will
be feasible!
ty
Remark
In order attain convergence to the true optimum, it is sufficient that at each iteration an improvement be made in both the objective function and constraint infeasibility.
This type of monotonic behavior will occur if the problem functions are mildly nonlinear.
Approach 2Separable Programming
The motivation for this technique stems from the observation that a good way of improving the linear approximation over a large interval is to partition the interval into subintervals and construct individual linear approximation over each subinterval, i.e., piecewise linear approximation.
Case 2.1Single-Variable Functions
Line Segment in Interval k
( 1) ( )
( ) ( )( 1) ( )
( ) ( 1)
The equation of line in interval is
k kk k
k k
k k
k
f ff x f x x
x x
x x x
Line Segment in Interval k
( ) ( ) ( 1) ( 1)
( ) ( 1) ( ) ( 1)
( 1) ( )( ) ( ) ( ) ( 1) ( 1) ( )
( 1) ( )
( 1) ( )( ) ( 1) ( 1) ( 1) ( )
( 1) ( )
( ) ( 1) ( 1) ( )
(
where
1 and , 0
( )
k k k k
k k k k
k kk k k k k k
k k
k kk k k k k
k k
k k k k
k
x x x
f ff x f x x x
x x
f ff x x
x x
f f f
) ( ) ( 1) ( 1)k k kf f
General Formula
(1) ( )
( ) ( )
1
( ) ( )
1
( ) ( )
1
( ) ( )
For any point in
where
1 and 0 1, 2, ,
0 if 1 and 1, 2, , 2
K
Kk k
k
Kk k
k
Kk k
k
i j
x x x
x x
f x f
k K
j i i K
Case 2.2Multivariable Separable Functions
1
A function is said to be separable if it can be
expressed as the sum of the single-variable functions
that each involve only one of the N variables, i.e.,
N
i ii
f f x
x
General Formula
1 2
(1) (2) ( )
1 2
( ) ( ) ( ) ( ) ( ) ( )1 1 2 2
1 1 1
( ) ( )
1 1
Let
where 1, 2, ,
Then
, , ,
N
i
Ki i i i i
N
KK Kk k k k k k
N Nk k k
KNk ki i
i k
L x x x U
i N
f x x x
f f f
f
General Formula
( ) ( ) ( ) ( )
1
( )
1
( )
( ) ( )
In the previous slide,
and
1
0 1, 2, ,
0 if 1 and 1,2, , 2
and
1,2, ,
i
i
Kk k k k
i i i i i ik
Kki
k
ki i
m ni i i
x x f f x
k K
n m m K
i N
Restricted Basis Entry
Prior to entering one lambda into the basis (which will make it nonzero), a check should be made to ensure that no more than one other lambda associated with the same x_i is in the basis.
If there is one such lambda in the basis, it has to be adjacent.
Example
41 2 1 2
21 2 1 2
1
2
max ,
. .
, 9 2 3 0
0
0
f x x x x
s t
g x x x x
x
x
1 2 1 1 2 2
41 1 1
2 2 2
1 2 1 1 2 2
21 1 1
2 2 2
,
,
2
9 3
f x x f x f x
f x x
f x x
g x x g x g x
g x x
g x x
1Construct the approximation over 0 3
and use 4 equidistant points
x
k
1 0 0 0
2 1 1 -2
3 2 16 -8
4 3 81 -18
( )1kx ( )
1kf ( )
1kg
(1) (2) (3) (4)1 1 1 1 1 1
(1) (2) (3) (4)1 1 1 1 1 1
(1) (2) (3) (4)1 1 1 1
Thus,
0 1 16 81
0 2 8 18
1
f x
g x
1 2 1 1 2 2
1 2 1 1 2 2
(2) (3) (4)1 1 1 2
(2) (3) (4)1 1 1 2 3
(1) (2) (3) (4)1 1 1 1
(1) (2) (3) (4)1 1 1 1 2 3
max ,
. .
,
2 8 18 3 9 0
( 2 8 18 3 9)
1
, , , , , 0
f x x f x f x
s t
g x x g x g x
x
x x
x x
Homework