lec 20 multivar cauchy method
TRANSCRIPT
-
8/9/2019 Lec 20 Multivar Cauchy Method
1/26
Multi-variable Unconstrained Optimization:
Cauchy method and Newton Method
Dr. Nasir M Mirza
Optimization Techniques Optimization Techniques
Email: [email protected]
-
8/9/2019 Lec 20 Multivar Cauchy Method
2/26
In this Lecture …
• In this lecture we will discuss two important methods
that deal with Multi-variable Unconstrained
Optimization:
• Cauchy’s Steepest Ascent Method
• Newton's Method
-
8/9/2019 Lec 20 Multivar Cauchy Method
3/26
Cauchy’s Steepest Ascent Method
• The search direction used in Cauchy’s method is the
negative of the gradient at any particular point x*.
• Since this direction gives maximum descent in
function values, it is also known as the steepest
descent method.
• At every iteration the derivative is computed at
current point and a unidirectional search is performed
in negative to this derivative direction to findmaximum.
• The maximum becomes the current point and search
is continued from this point.
-
8/9/2019 Lec 20 Multivar Cauchy Method
4/26
-
8/9/2019 Lec 20 Multivar Cauchy Method
5/26
Example: Suppose f ( x, y) = 2 xy + 2 x – x2 – 2 y2
Using the steepest ascent method to find the next point if we aremoving from point (-1, 1).
y x y
f x y
x
f 42222 −=
∂∂−+=
∂∂
⎥⎦⎤⎢
⎣⎡−
=⎥⎦⎤⎢
⎣⎡
−−−−+=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
−∂
∂
−∂
∂
=∇−6
6)1(4)1(2
)1(22)1(2
)1,1(
)1,1(
),1,1(At
y
f x
f
f
)6161()(Let hh,- f h g −+=
Next step is to find h that maximize g (h)
-
8/9/2019 Lec 20 Multivar Cauchy Method
6/26
2.0036072
yields0)('Setting
180727...)6161()(
222)(
2
22
=⇒=−
=
−+−==−+=
−−+=
hh
h g
hhhh,- f h g
y x x xy x f
If h = 0.2 maximizes g (h), then x = -1+6(0.2) = 0.2 and y = 1-6(0.2) = -0.2 would maximize f ( x, y).
So moving along the direction of gradient from point (-1,1), we would reach the optimum point (which is our nextpoint) at (0.2, -0.2).
-
8/9/2019 Lec 20 Multivar Cauchy Method
7/26
EXERCISE 3.4.2
Consider the Himmelblau
function:
Minimize:f(x, y) = (x2 + y – 11)2 + (x + y2
– 7)2
•Step 1: In order to ensure
proper convergence, a large value
of M (= 100) is usually chosen.
The choice of M also depends on the available time and computing
resource. Let us choose M = 100, an initial point x(0) = (0, 0)T, and
termination parameters ε1 = ε2 = 10-3
. We also set k = 0.
-
8/9/2019 Lec 20 Multivar Cauchy Method
8/26
EXERCISE 3.4.2
3D Graph
-
8/9/2019 Lec 20 Multivar Cauchy Method
9/26
Example
% Matlab program to
draw contour of function
[X,Y] = meshgrid(0:.1:5);
Z = (X.*X + Y -11.).^2. +(
X + Y.*Y - 7.).^2.contour(X, Y, Z, 150);
colormap(jet); Minimum point
Contour graph:
-
8/9/2019 Lec 20 Multivar Cauchy Method
10/26
EXERCISE 3.4.2
h
h x f h x f
x
f ii
i 2
)()( −−+=
∂
∂
Step 2:
The derivative at x(0) = (0, 0)T is
first calculated
and found to be (-14, -22)T, whichis identical to the exact derivative
at that point (Figure 3.12).
Step 3:The magnitude of the derivative vector is not small and k =a < M =
100. Thus, we do not terminate; rather we proceed to Step 4.
-
8/9/2019 Lec 20 Multivar Cauchy Method
11/26
EXERCISE 3.4.2
• Step 4: At this step, we need to perform a line search from x(0)
in the direction - f(x(0)) such that the function value is minimum.
• Along that direction, any point can be expressed by fixing a value orthe parameter a(0) in the equation:
x = x(0) - a(0) f(x(0)) = (14a(0), 22a(0)) T .
1.0
yields)1,0(0)('Setting
)714484)(14968(2)1122196)(22392(2)(
)714484()1122196()2214()(
)7()11()(
22
2
2222
2222
=
=
−+++−++=′
−++−+==
−++−+=
a
region for a g
aaaaaaa g
aaaaaa, f a g
y x y x x f
-
8/9/2019 Lec 20 Multivar Cauchy Method
12/26
EXERCISE 3.4.2
• Thus, the minimum point along the search direction isx(1) == (1.788, 2.810)T.
• Step 5: Since, x(1) and x(0) are quite different, we donot terminate; rather we move back to Step 2. Thiscompletes one iteration of Cauchy's method. The totalnumber of function evaluations required in this iterationis equal to 30.
• Step 2: The derivative vector at this point, computednumerically, is (-30.7, 18.8) T .
-
8/9/2019 Lec 20 Multivar Cauchy Method
13/26
EXERCISE 3.4.2
• Step 3: This magnitude of the derivative vector is not
smaller than ε1. Thus, we continue with Step 4.
• Step 4: Another unidirectional search along (30.70, -
18.80) T from the point x(1) = (1.788, 2.810)T using
the golden section search finds the new point:
• x(2) = (3.00, 1.99)T with a function value equal to 0.018.
-
8/9/2019 Lec 20 Multivar Cauchy Method
14/26
Newton’s Method
• This method uses second – order derivatives to createsearch directions.
• This allows faster convergence to the minimum point.• Consider the first three terms in Taylor’s series
expansion of a multivariable function, it can be shown
that the first order optimality condition will be satisfiedif following search direction is used:
[ ] )()( )(1
)(2)( k k k x f x f s ∇∇−= −
-
8/9/2019 Lec 20 Multivar Cauchy Method
15/26
Newton's Method
One-dimensionalOptimization
Multi-dimensionalOptimization
At theoptimal
Newton's
Method
0)( =∇ x f 0)(' =i x f
)(1
1 iiii f xHxx ∇−= −
+
)(")('
1
i
iii
x f x f x x −=+
Hi is the Hessian matrix (or
matrix of 2nd
partialderivatives) of f evaluated
at xi.)(')(" 1 ii x f x f
−
-
8/9/2019 Lec 20 Multivar Cauchy Method
16/26
Newton's Method
• Converge quadratic fashion.
• May diverge if the starting point is not close
enough to the optimum point.
• Costly to evaluate H-1.
)(11 iiii f xHxx ∇−= −
+
-
8/9/2019 Lec 20 Multivar Cauchy Method
17/26
-
8/9/2019 Lec 20 Multivar Cauchy Method
18/26
EXERCISE 3.4.3
3D Graph
-
8/9/2019 Lec 20 Multivar Cauchy Method
19/26
Example on Newton Method
% Matlab program todraw contour of function
[X,Y] = meshgrid(0:.1:5);
Z = (X.*X + Y -11.).^2. +(
X + Y.*Y - 7.).^2.
contour(X, Y, Z, 150);colormap(jet);
Minimum point
Contour graph:
-
8/9/2019 Lec 20 Multivar Cauchy Method
20/26
Example on Newton Method
( ) ( )
( ) ( )( ) ( )
[ ] .22000.42 and2214),0,0(At
621624
2442412
1422222272112
14424472114
711),(
T
2
2
3222
2322
2222
⎥⎦
⎤⎢⎣
⎡
−
−=−−=∇
⎥⎦
⎤⎢⎣
⎡
++−+
+−+=
−++−−=−++−+=∂
∂
−+−+=−++−+=
∂
∂
−++−+=
H
H
f
y x y x
y x y x
y y xy y x y x y y x y
f
y x xy x y x y x x x
f
y x y x y x f
-
8/9/2019 Lec 20 Multivar Cauchy Method
21/26
Example on Newton Method
( ) ( )
[ ]
⎟⎟ ⎠
⎞⎜⎜⎝
⎛
−
−=
⎟⎟ ⎠
⎞⎜⎜⎝
⎛
−
−⎟⎟ ⎠
⎞⎜⎜⎝
⎛
−
−−⎟⎟
⎠
⎞⎜⎜⎝
⎛ =
∇−=
⎥⎦⎤⎢
⎣⎡ −
−=−−=∇
−++−+=
−
a
a
a f a x x
f
y x y x y x f
333.0
22
14
420
022
9240
0)0()1(
.22000.42 and2214),0,0(At
711),(
1
T
2222
H
H
-
8/9/2019 Lec 20 Multivar Cauchy Method
22/26
Example on Newton Method
• Performing a unidirectional search along thisdirection, we obtain value of a = -3.349.
• Since this quantity is negative, the functionvalue does not reduce in the given searchdirection.
• Instead, function value reduces in the oppositedirection.
• This shows that search direction in Newtonmethod may not always be descent.
• When this happens we restart with a new point.
-
8/9/2019 Lec 20 Multivar Cauchy Method
23/26
Example on Newton Method
Now again select a new initial point x(0) = (2, 1)T,
and function value at this point is f(x(0)) = 52.
Step 2: The derivative at this point is calculated as (-57, -24)T.
Step 3: Since the termination criteria are not met we go to the step 4.Step 4: At this point the Hessian is given as:
( ) ( )
( ) ( )
( ) ( )
[ ] .510
1010 and2457),1,2(At
32224
2442412
1422222272112
14424472114
711),(
T
2
2
3222
2322
2222
⎥⎦⎤⎢
⎣⎡=−−=∇
⎥⎦
⎤⎢⎣
⎡
++−+
+−+=
−++−−=−++−+=∂
∂
−+−+=−++−+=∂∂
−++−+=
H
H
f
y x y x
y x y x
y y xy y x y x y y x y
f
y x xy x y x y x x x f
y x y x y x f
-
8/9/2019 Lec 20 Multivar Cauchy Method
24/26
Example on Newton Method
( ) ( )
[ ]
⎟⎟
⎠
⎞⎜⎜
⎝
⎛
+
−=
⎟⎟
⎠
⎞⎜⎜
⎝
⎛
−
−⎟⎟
⎠
⎞⎜⎜
⎝
⎛
−
−
−−⎟⎟
⎠
⎞⎜⎜
⎝
⎛ =
∇−=
⎥⎦⎤⎢
⎣⎡=−−=∇
−++−+=
−
a
a
a f a x x
f
y x y x y x f
6.61
9.0224
57
1010
105
501
2)0()1(
.510
1010 and2457),0,0(At
711),(
1
T
2222
H
H
-
8/9/2019 Lec 20 Multivar Cauchy Method
25/26
Example on Newton Method
• A unidirectional search along this directionreveals that the minimum occurs at a = 0.34
• Since this quantity is positive, we accept this.• The new point is then (1.694, 3.244) and
function value is 44.335 which is smaller than
the previous point value of 52.• It means we are going in right direction.
• When we continue for next two iterations we
reach at point (3.021, 2.003)T with f = 0.0178;• Hence answer is point (3, 2)T for this case when
we go further.
-
8/9/2019 Lec 20 Multivar Cauchy Method
26/26
EXERCISE 3.4.3
• This method is effective for initial points close to the
optimum.
• This demands some knowledge of the optimum point.
• Computation of the Hessian matrix and its inverse is
computationally expensive.