# MAE 552 – Heuristic Optimization Lecture 3 January 28, 2002

Post on 20-Dec-2015

212 views

Embed Size (px)

TRANSCRIPT

<ul><li> Slide 1 </li> <li> MAE 552 Heuristic Optimization Lecture 3 January 28, 2002 </li> <li> Slide 2 </li> <li> Types of Optimization and Optimization Algorithms There are a many different types of optimization and even more algorithms to perform it. Why are there so many algorithms? None of the traditional algorithms are robust. For every different type of problem there is a different solution method. This can become a problem when a user only knows a limited number of algorithms and tries to apply them to problems they were not intended for. </li> <li> Slide 3 </li> <li> Types of Optimization </li> <li> Slide 4 </li> <li> Important Categories Methods can be classified according to: 1.The type of problem being addressed Convex vs. Multi-Modal Continuous vs. Discrete Constrained vs. Unconstrained Linear, Quadratic and Nonlinear Programming Analytical or Numerical </li> <li> Slide 5 </li> <li> Important Categories Methods can be classified according to: 2.The type of technique being used Deterministic vs. Random Derivative-based vs. Derivative free Global vs. Local </li> <li> Slide 6 </li> <li> Types of Problems: Convex versus Multimodal 1.Strictly Convex Problems have a single optima Local Optima = Global Optima Hessian matrix has positive eigenvalues F(x) x*x* </li> <li> Slide 7 </li> <li> Types of Problems: Convex versus Multimodal 1.Non Convex Problems may have multiple local optima x F(x) </li> <li> Slide 8 </li> <li> Types of Problems: Continuous vs. Discrete 1.In continuous problems x is defined over a range x l x x u Examples: length, weight, temperature etc. F(x) x </li> <li> Slide 9 </li> <li> Types of Problems: Continuous vs. Discrete 1.In discrete problems x is defined only for specific values x l x x u where x can take on only specific values Examples: Material Type, routes taken, standardized sizing F(x) x </li> <li> Slide 10 </li> <li> Types of Problems: Linear, Quadratic and Nonlinear F 1.Problems with Linear or Quadratic Objective Functions and constraints are much easier to solve than those with general nonlinear terms. Examples: Linear F=ax 1 + bx 2 + c Quadratic F=F=ax 1 + bx 2 + cx 2 2 +d Nonlinear F=ax 1 + bx 2 2 + cx 2 5 +x 1 x 2 1/2 </li> <li> Slide 11 </li> <li> Types of Problems:Analytical or Numerical Analytical Problems: Equations available to calculate objective function and constraints F(x)=ax 1 + bx 2 + c g 1 (x)=x 1 +x 2 0 Numerical Problems: Analysis codes used to determine objective function and constraints ANALYSIS x F(x) g(x) </li> <li> Slide 12 </li> <li> Type of Technique : Deterministic vs. Random Deterministic Methods: With a given set of initial conditions these algorithms always follow the same path. XoXo Examples: Enumerative Search, Steepest Descent, GRG, Conjugate Gradient, Powells etc. </li> <li> Slide 13 </li> <li> Type of Technique : Deterministic vs. Random Random Methods: With a given set of initial conditions these algorithms follow different paths with each run XoXo Examples: Scatter Search, Genetic Algorithms, Simulated Annealing </li> <li> Slide 14 </li> <li> Type of Technique : Derivative-based/Derivative free Derivative Based Methods: First and Second Derivatives of the objective Function are calculated either numerically of analytically Examples: Steepest Descent, Newtons Method, Conjugate Gradient, Varible Metric Method etc. Derivative Free Methods: Only F(X) information is used to guide the search Examples: Powells Search, Random Search, SA, GA, etc. </li> <li> Slide 15 </li> <li> Type of Technique :Local or Global Optimizer Local Search Methods: Find best design within a portion of the design space usually a convex space (neighborhood). These methods include almost all of the traditional numerical techniques zero-order and derivative-based. Tend to get stuck in valleys of the design space F(x) x*x* Global Opt. Local Opt. </li> <li> Slide 16 </li> <li> Type of Technique :Local or Global Optimizer Global Search Methods: Find best design within entire design space. Tend to be robust but inefficient and expensive to run. F(x) x*x* Global Opt. Local Opt. </li> <li> Slide 17 </li> <li> Choosing the Right Method 1.It is very important to match the optimization method to the problem that you are solving. 2.Know the assumptions involved with each method. Example: LP requires linear objective and constraints. 3.Know the relative efficiency of each proposed method. Example: Using a Enumerative Search to solve a convex problem will be very inefficient. 4.Some problems are intractable!!! There are no methods that can guarantee a global optima in finite time. Use methods that find good solutions fast. </li> <li> Slide 18 </li> <li> Basics of problem Solving Which algorithm is right for a problem is strongly dependent on how the problem is mathematically described. Three basic concepts common to every algorithmic approach to problem solving 1.A representation of the problem 2. The objective 3. The evaluation function </li> <li> Slide 19 </li> <li> Basics of problem Solving- Representation The representation encodes alternative candidate solutions for manipulation and defines the size of the search space. For example in NLP typically the design variables are represented as real numbers in n dimensions. Is this the only possibility? In the TSP the representation we have used is a permutation of the natural numbers 1,2,3,.,n. The size of the search space is not determined by the problem, it is determined by your representation and the way that you handle this encoding. </li> <li> Slide 20 </li> <li> Basics of problem Solving- Representation Your choice of representation will determine how you will be able to manipulate the potential solutions Alternative Representations for a NLP {x 1,x 2 }={5,6} {x 1,x 2 }={101,110} 0x 1 10 10 X 2 0 </li> <li> Slide 21 </li> <li> Basics of problem Solving-Objective The objective represents mathematically the task to be achieved For the TSP the objective is typically to minimize the distance covered by the salesman subject to the constraint of visiting each city once and only once and returning the starting city. min dist (legal trip) For NLP an example might be: min F=x 1 +x 2 +x 3 </li> <li> Slide 22 </li> <li> Basics of problem Solving-Evaluation Function The evaluation function is typically a mapping from the space of possible candidate solutions to a set of numbers, where each element from the space of possible solutions is assigned a number to indicate its quality. The EF allows you to compare solutions in terms of quality. Ordinal EFs allow you to rank the candidate solutions. Current Candidate is the tenth best Numerical evaluations allow you to rank the solutions and also their degree of quality. Current Candidate is 1.3 units better that the second best candidate </li> <li> Slide 23 </li> <li> Basics of problem Solving-Evaluation Function Since it could be computationally expensive to determine a numerical value for how good or bad a candidate it might be good enough to know approximately how good a solution is. Eval. Function Candidate 1 Candidate 2 Superior Candidate </li> <li> Slide 24 </li> <li> Basics of problem Solving-Evaluation Function For every real world problem the evaluation problem is chosen by the designer. It should of course indicate for instance that a solution that meets the objective is better than one that does not. It should also depend on factors such as the computational complexity of the problem. Often the objective function indicates a good evaluation function. Objective - Minimize Stress -> Evaluation Function -Stress </li> <li> Slide 25 </li> <li> Basics of problem Solving-Evaluation Function Other times you cannot derive a useful Evaluation Function from the objective: In the SAT problem the objective is to find a set of boolean (TRUE,FALSE) variables that satisfies a logical statement (makes it TRUE). All wrong candidate solutions return FALSE which does not tell you how to improve the solution. </li> <li> Slide 26 </li> <li> Basics of problem Solving-Defining a Search Problem When you design an evaluation function you need to consider that for many problems the only solutions of interest are the subset that are Feasible (satisfy the constraints). The feasible space can be defined as F where F S. A search problem can then be defined as: Given a search space S and its feasible part F S find x F such that eval(x) eval(y) for all y F Note that the objective does not appear at all in the formulation!! If your EF does not correspond with the objective you will searching for the answer to the wrong problem. </li> <li> Slide 27 </li> <li> Basics of problem Solving-Defining a Search Problem A point x that satisfies the condition is called a global solution. Finding a global solution can be difficult and impossible to prove in some cases. It would be easier if we could limit the search to a smaller area of S. This fact underlies many search techniques. </li> <li> Slide 28 </li> <li> Basics of problem Solving-Neighborhood Search If we concentrate on the area of S near to some point in the search space we can more easily look in this neighborhood. S x N(x) N(x) of x is a set of all points in the search space that close to the given point x. N(x) ={y S: dist(x,y) } </li> <li> Slide 29 </li> <li> Basics of problem Solving-Neighborhood Search For a continuous NLP the Euclidean distance can be used to define a neighborhood. For the TSP a 2-swap neighborhood can be defined as all of the candidates that would result from swapping two cities in a given tour. A solution x (a permutation of n=5 cities) 1-2-3-4-5 has n(n-1)/2 neighbors including 1-3-2-4-5 (swapping cities 2 and 3) 5-2-3-4-1 (swapping cities 1 and 5) etc. </li> <li> Slide 30 </li> <li> Basics of problem Solving-Neighborhood Search F=x 2 +3 xcxc Example: Quadratic Objective with no Constraints </li> <li> Slide 31 </li> <li> Basics of problem Solving-Neighborhood Search Step 1: Define a neighborhood around point x c. N(x): x c - x x c + Min F=x 2 +3 xcxc </li> <li> Slide 32 </li> <li> Basics of problem Solving-Neighborhood Search Step 2: Sample a candidate solution from the neighborhood and evaluate it. if F(x 1 )>F(x c ) reject point and choose another. F=x 2 +3 xcxc x1x1 </li> <li> Slide 33 </li> <li> Basics of problem Solving-Neighborhood Search if F(x 1 )</li></ul>