d nagesh kumar, iiscoptimization methods: m8l4 1 advanced topics in optimization direct and indirect...
TRANSCRIPT
![Page 1: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/1.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L41
Advanced Topics in Optimization
Direct and Indirect Search Methods
![Page 2: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/2.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L42
Introduction
Real World Problems
Nonlinear Optimization
Complicated Objective Function
Analytical Solution
Possible Solution: Search Algorithm
![Page 3: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/3.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L43
Flowchart for Search Algorithm
Start with Trial Solution Xi, Set i=1
Compute Objective function f(Xi)
Generate new solution Xi+1
Compute Objective function f(Xi+1)
Convergence Check
Optimal Solution, Xopt=Xi
Yes
Set i=i+1No
![Page 4: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/4.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L44
Classification of Search Algorithm
Search Algorithm
Direct Search Algorithm
•Considers only objective function
•Partial derivatives of objective function: not considered
•Called nongradient or zeroth order method
Indirect Search Algorithm
•Partial derivatives of objective function are considered
•Also called descent method
![Page 5: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/5.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L45
Direct Search Algorithm
Direct Search Algorithm
Random Search Method
Grid Search Method
Pattern Directions
Univariate Method
Rosen Brock’s Method
Simplex Method
![Page 6: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/6.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L46
Direct Search Algorithm (Contd..)
Random Search Method: Generates trial solution for the decision variables. Classification: Random jump, random walk, and random walk
with direction exploitation. Random Jump: generates huge number of data points
assuming uniform distribution of decision variables and selects the best one
Random Walk: generates trial solution with sequential improvements using scalar step length and unit random vector.
Random Walk with Direction Exploitation: Improved version of random walk search, successful direction of generating trial solution is found out and steps are taken along this direction.
![Page 7: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/7.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L47
Direct Search Algorithm (Contd..)
Grid Search Method Methodology involves setting up grids in the decision
space. Objective function values are evaluated at the grid
points. The point corresponding to the best objective
function value considered as optimum solution. Major Drawback: number of grid points increases
exponentially with the number of decision variables.
![Page 8: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/8.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L48
Direct Search Algorithm (Contd..)
Univariate Method:
Generates trial solution for one decision variable keeping all others fixed.
Best solution for each of the decision variables keeping others constant are obtained.
The whole process is repeated iteratively till convergence.
![Page 9: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/9.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L49
Pattern Directions: In Univariate method, search direction is same as coordinate
axes directions. Makes the convergence process slow. In pattern directions, search is performed not along the
coordinate directions, but along the direction towards the best solution.
Popular Methods: Hooke and Jeeves’ Method, and Powells method.
Hooke and Jeeves’ Method: uses exploratory move and pattern move.
Powells method: uses conjugate gradient
Direct Search Algorithm (Contd..)
![Page 10: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/10.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L410
Direct Search Algorithm (Contd..)
Rosen Brock’s Method of Rotating Coordinates:Modified version of Hooke and Jeeves’ Method.Coordinate system is rotated in such a way that the first axis always orients to the locally estimated direction of the best solution and all the axes are made mutually orthogonal and normal to the first one.Simplex MethodConventional direct search algorithm. The best solution lies on the vertices of a geometric figure in N-dimensional space made of a set of N+1 points.The method compares the objective function values at the N+1 vertices and moves towards the optimum point iteratively. Movement of the simplex algorithm: reflection, contraction and expansion.
![Page 11: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/11.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L411
Indirect Search Algorithm
Indirect Search Algorithm
Steepest Descent Method
Conjugate Gradient Method
Newton’s Method
Quasi-Newton Method
Marquardt Method
![Page 12: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/12.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L412
Indirect Search Algorithm (Contd..)
Steepest Descent Method: Search starts from an initial trial point X1
Iteratively moves along the steepest descent direction until optimum point is found.
Drawback: Solution may get stuck to a local optima.Conjugate Gradient Method Convergence technique uses the concept of
conjugate gradient Uses the properties of quadratic convergence
![Page 13: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/13.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L413
Indirect Search Algorithm (Contd..)
Newton’s Method: Based on Taylor series expansion.
[Ji]=[J]|xi is the Hessian Matrix Setting the partial derivatives of the objective function to zero, minimum
f(x) can be obtained
From both the Eq.
The improved solution is given by
Process continues till convergence
iiT
iiTii XXJXXXXfXfXf
2
1
Nj
x
Xf
j
,...,2,1,0
0 iii XXJff
iiii fJXX
11
![Page 14: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/14.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L414
Indirect Search Algorithm (Contd..)
Marquardt Method: Combination of both steepest descent algorithm and Newton’s
method By modifying the diagonal elements of Hessian matrix
iteratively optimum solution is obtainedQuasi-Newton Method Based on Newton’s method They approximate the Hessian matrix, or its inverse, in order to
reduce the amount of computation per iteration. The Hessian matrix is updated using the secant equation, a
generalization of the secant method for multidimensional problems
![Page 15: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/15.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L415
Constrained Optimization
Search algorithms cannot be directly applied to constrained optimization.
Constrained optimization can be converted into unconstrained optimization using penalty function when there is a constraint violation.
=1( for minimization problem) and -1 ( for maximization problem), M=dummy variable with a very high value
2 MXfXf iiPenalty Function
![Page 16: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/16.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L416
Summary
Nonlinear optimization with complicated objective functions or constraints: Can be solved by search algorithm.
Care should be taken so that the search should not get stuck at local optima.
Search algorithms are not directly applicable to constrained optimization
Penalty functions can be used to convert constrained optimization into unconstrained optimization
![Page 17: D Nagesh Kumar, IIScOptimization Methods: M8L4 1 Advanced Topics in Optimization Direct and Indirect Search Methods](https://reader035.vdocuments.mx/reader035/viewer/2022072008/56649d6e5503460f94a4f452/html5/thumbnails/17.jpg)
D Nagesh Kumar, IISc Optimization Methods: M8L417
Thank You