method of hooke and jeeves. multidimensional search without using derivatives in this section we...

METHOD OF HOOKE AND JEEVES

Multidimensional Search Without Using Derivatives

In this section we consider the problem of minimizing a function f of several variables without using derivatives. The methods described here proceed in the following manner.

Multidimensional Search Without Using Derivatives

Given a vector x, a suitable direction d is first determined, and then f is minimized from x in the direction d by one of the techniques discussed earlier.

X = ( x1 , x2 , … , xn )

Direction = dx

Minimizing F(x) in dx direction

What is the problem?

We are required to solve a line search problem of the form to:

minimize f(x + d) λsubject to: L λ

where L is typically of the form:

L = { : ≥ 0 } or λ λ L = R L = { : a ≤ ≤ b }λ λ

What is the problem?

In the statements of the algorithms, for the purpose of simplicity we have assumed that a minimizing point exists. However, this may not be the case. Here, the optimal objective value of the line search problem may be: A) Unbounded: Then the original problem is

unbounded and we may stop.B) Finite but not achieved at any particular : λ

Then could be chosen as such that λf(x+ d) is sufficiently close to the value:

inf { f(x+ d): L }.λ λ

Method of Hooke and Jeeves

The method of Hooke and Jeeves performs two types of search:

Exploratory search Pattern search

Method of Hooke and Jeeves

Given x1, an exploratory search along the coordinate directions produces the point x2. Now a pattern search along the direction x2- x1 leads to the point y.

Patte

rn Search

Explorato

ry se

arch along

the co

ordinate

axes

x 1

x 2

y

Method of Hooke and Jeeves

Another exploratory search starting from y gives the point x3. The next pattern search is conducted along the direction x3 –x2, yielding y'. The process is then repeated.

Patte

rn Search

Exploratory search along the coordinate axes

x 1

x 2

y

x 3

y'

Summary of the Method of Hooke and Jeeves Using Line Searches

As originally proposed by Hooke and Jeeves, the method does not perform any line search but rather takes discrete steps along the search directions, as we discuss later. Here we present a continuous version of the method using line searches along the coordinate directions d1 ,..., dn and the pattern direction.

Initialization Step

I. Choose a scalar ε > 0 to be used in terminating the algorithm.

II. Choose a starting point x1

III. let y1 = x1

IV. let k = j = 1V. Go to the Main Step.

Main Step

I. Step 1:1. Let λj be an optimal solution to the problem to:

minimize f(yj + dλ j)

subject to: Rλ yj+1 = yj +λj dj

2. If j < n, replace j by j + 1, and repeat Step 1. Otherwise, if j = n, let xk+1=yn+1 .

3. If ||xk+1- xk|| < , stop; ε

otherwise, go to Step2.

Main Step

II. Step 2:1. Let d = xk+1 – xk

2. Let be an optimal solution to the problem to:

minimize f(xk+1+ d)λ

Subject to: Rλ3. Let y1 = xk+1+ d

4. Let j = 15. replace k by k+16. Go to Step 1.

Example IConsider the following problem:

Minimize (x1 – 2)4 +(x1 – 2x2)2

Note that the optimal solution is (2, 1) with objective value equal to zero.

At each iteration:• An exploratory search along the coordinate directions gives the points y2 and

y3

• A pattern search along the direction d = xk+1 – xk gives the point y1

• Except at iteration k = 1, where y1 = x1

• Four iterations were required to move from the initial point to the optimal point (2, 1) whose objective value is zero.

• At this point, ||x5 – x4|| = 0.045

• and the procedure is terminated.

Summary of Computations for the Method of Hooke and Jeeves Using Line Searches

Method of Hooke and Jeeves using line searches.

Point

The Pattern Search has substantially improved the Convergence behavior by moving along a direction that is almost parallel to the valley shown by dashed lines.

Convergence of the Method of Hooke and Jeeves

Suppose that:I. f differentiable II. Let the solution set = Ω { : f ( )= 0).

Note: Each iteration of the method of Hooke and Jeeves consists of an application of the cyclic coordinate method, in addition to a pattern search.

III. Let the cyclic coordinate search be denoted by the map B and the pattern search be denoted by the map C.

IV. If the minimum of f along any line is unique and letting = α f

then (y) < ( x ) for x . α α ΩV. By the definition of C, (z) ≤ (y) for z C(y).α α

Assuming that = { x : f ( x ) < f ( xΛ 1 ) } , where x1 is the starting point, is compact, convergence of the procedure is established.

Method of Hooke and Jeeves with Discrete Steps

The method of Hooke and Jeeves, as originally proposed, does not perform line searches but, instead, adopts a simple scheme involving functional evaluations. A summary of the method is given below:

Initialization Step

I. Let d1,...,dn be the coordinate directions.

II. Choose a scalar > 0 to be used for εterminating the algorithm.

III. choose an initial step size, ≥ .Δ εIV. Choose an acceleration factor, >0.αV. Choose a starting point x1, let y1= x1, let

k=j=1VI. Go to the Main Step.

Main Step

I. Comparing f(yj+ dΔ j) and f(yj):

1) If f(yj+ dΔ j) < f(yj), the trial is termed a success; let yj+1= yj+ dΔ j , and go to Step2.

2) If f(yj+ dΔ j) ≥ f(yj), the trial is deemed a failure. In this case:

1) If f(yj - dΔ j) < f(yj), let yj+1= yj - dΔ j , and go to Step 2;

2) if f(yj - dΔ j) ≥ f(yj), let yj+1= yj, and go to Step 2 .

Main StepII. Comparing j and n

1. If j < n, replace j by j+1, and repeat Step 1.2. Otherwise:

a. if f(yn+1) < f(xk) go to Step 3.

b. if f(yn+1) ≥ f(xk) go to Step 4.

III. Do as follows:1. Let xk+1 = yn+1

2. Let y1 = xk+1 + (xα k+1 – xk)

3. Replace k by k+ l 4. Let j = 15. Go to Step 1.

Main Step

IV. Compare and Δ ε1. If ≤ , stop ; xΔ ε k is the prescribed solution.

2. Otherwise, replace by /2.Δ Δ3. Let y1 = xk

4. Let xk+1=xk

5. Replace k by k+ l , 6. Let j = 17. Repeat Step 1.

Note Steps I and II describe an exploratory search. Step III is an acceleration step along the

direction xk+1- xk. A decision whether to accept or reject the

acceleration step is not made until after an exploratory search is performed.

In Step IV, the step size Δ is reduced. The procedure could easily be modified so that different step sizes are used along the different directions. This is sometimes adopted for the purpose of scaling.

Example II: Solve the problem using the method of Hooke and Jeeves with discrete stepsConsider the following problem:

Minimize (x1 – 2)4 + (x1- 2 x2 )2

In which the parameters and are chosen α Δas 1.0 and 0.2, respectively. The algorithm start from (0.0, 3.0). The points generated are numbered equentially.The acceleration step that is rejected is shown

by the dashed lines.

Method of Hooke and Jeeves using discrete steps starting from (0.0,3.0). The numbers denote the order in which points are generated.

The following table give a more comprehensive illustration, It summarizes the computations starting from the new initial point (2.0, 3.0). Here:

(S) denotes that the trial is a Success (F) denotes that the trial is a Failure At the iteration, and at subsequent iterations whenever:

If f(y3) ≥ f (xk) then the vector y1 is taken as xk .

Otherwise, y1 = 2xk+1- xk .

Note: At the end iteration k =10, the point (1.70, 0.80) is reached having an objective value 0.02.

The procedure is stopped here with the termination parameter ε = 0.1.

If a greater degree of accuracy is required, should be Δreduced to 0.05.

The Summary Table of Computations for the Method of Hooke and Jeeves with Discrete Steps

Method of Hooke and Jeeves using line searches. The numbers denote the order in which Donets are generated.

THANK YOU ALL FOR YOUR ATTENTION