Transcript
Page 1: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

1© 2016 The MathWorks, Inc.

Solving Optimization Problems with MATLAB

Loren Shure

Page 2: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

2

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

Page 3: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

3

Optimization Problems

Minimize Risk

Maximize Profits

Maximize Fuel Efficiency

Page 4: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

4

Design Process

Initial

Design

Variables

System

Modify

Design

Variables

Optimal

DesignObjectives

met?

No

Yes

Page 5: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

5

Manually (trial-and-error or iteratively)

Why use Optimization?

Initial

Guess

Page 6: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

6

Automatically (using optimization techniques)

Initial

Guess

Why use Optimization?

Page 7: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

7

Why use Optimization?

Finding better (optimal) designs and decisions

Faster design and decision evaluations

Automate routine decisions

Useful for trade-off analysis

Non-intuitive designs may be found

Antenna Design Using Genetic Algorithmhttp://ic.arc.nasa.gov/projects/esg/research/antenna.htm

Page 8: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

8

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

Page 9: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

9

Curve Fitting Demo

Given some data:

Fit a curve of the form:

teccty 21

)(

t = [0 .3 .8 1.1 1.6 2.3];

y = [.82 .72 .63 .60 .55 .50];

Page 10: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

10

How to solve?

Ecc

cey

eccty

t

t

2

1

21

1

)(

As a linear system of equations:

2

2min yEc

Ecy

c

Can’t solve this exactly

(6 eqns, 2 unknowns)

An optimization problem!

2

1

3.2

6.1

1.1

8.0

3.0

0

1

1

1

1

1

1

50.0

55.0

60.0

63.0

72.0

82.0

c

c

e

e

e

e

e

e

Page 11: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

11

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

Page 12: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

12

Nonlinear Optimization

min𝑥

log 1 + 𝑥1 −4

3

2

+ 3 𝑥1 + 𝑥2 − 𝑥13 2

Page 13: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

13

Nonlinear Optimization - Modeling Gantry Crane

Determine acceleration profile that

minimizes payload swing

s25s4

s20s1

s20s1

:sConstraint

2

1

21

f

p

p

ppf

t

t

t

ttt

Page 14: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

14

Symbolic Math ToolboxFunctions for analytical computations

Conveniently manage & document symbolic

computations in Live Editor

• Math notation, embedded text, graphics

• Share work as pdf or html

Perform exact computations using familiar

MATLAB syntax in MATLAB

Integration

Differentiation

Solving equations

Transforms

Simplification

Integrate with numeric computing – MATLAB,

Simulink and Simscape language

Perform Variable-precision arithmetic

Page 15: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

15

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

Page 16: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

16

Mixed-Integer Programming

Many things exist in discrete amounts:

– Shares of stock

– Number of cars a factory produces

– Number of cows on a farm

Often have binary decisions:

– On/off

– Buy/don’t buy

Mixed-integer linear programming:

– Solve optimization problem while enforcing that certain variables need to be integer

Page 17: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

17

Continuous and integer variables

𝑥1 ∈ 0, 100 𝑥2 ∈ 1,2,3,4,5

Linear objective and constraints

min𝑥

−𝑥1 − 2𝑥2

𝑥1 + 4𝑥2 ≤ 20𝑥1 + 𝑥2 = 10

such that

Mixed-Integer Linear Programming

Page 18: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

18

Traveling Salesman Problem

Problem

How to find the shortest path through a series of points?

Solution

Calculate distances between all combinations of points

Solve an optimization problem where variables correspond to trips between two points

1

11

0

1

1

0

0

00

Page 19: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

19

Introduction

Least-squares minimization

Nonlinear optimization

Mixed-integer programming

Global optimization

Topics

Page 20: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

20

Example Global Optimization Problems

Why does fmincon have a hard time finding the

function minimum?

0 5 10

-10

-5

0

5

10

x

Starting at 10

0 5 10

-10

-5

0

5

10

x

Starting at 8

0 5 10

-10

-5

0

5

10

x

Starting at 6

x s

in(x

) +

x c

os(2

x)

0 5 10

-10

-5

0

5

10

x

Starting at 3

0 5 10

-10

-5

0

5

10

x

Starting at 1

0 5 10

-10

-5

0

5

10

x

Starting at 0

x s

in(x

) +

x c

os(2

x)

Page 21: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

21

Initial

Guess

Example Global Optimization Problems

Why didn’t fminunc find the maximum efficiency?

Page 22: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

22

Example Global Optimization Problems

Why didn’t nonlinear regression find a good fit?

0 20 40 60 80 100 1200

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

t

c

c=b1e-b

4t+b

2e-b

5t+b

3e-b

6t

Page 23: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

23

Global Optimization

Goal:

Want to find the lowest/largest value of

the nonlinear function that has many local

minima/maxima

Problem:

Traditional solvers often return one of the

local minima (not the global)

Solution:

A solver that locates globally optimal

solutions

Global Minimum at [0 0]

Rastrigin’s Function

Page 24: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

24

Global Optimization Solvers Covered Today

Multi Start

Global Search

Simulated Annealing

Pattern Search

Particle Swarm

Genetic Algorithm

Page 25: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

25

MultiStart Demo – Nonlinear Regression

lsqcurvefit solution MultiStart solution

Page 26: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

26

MULTISTART

Page 27: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

27

What is MultiStart?

Run a local solver from each set

of start points

Option to filter starting points

based on feasibility

Supports parallel computing

Page 28: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

28

MultiStart Demo – Peaks Function

Page 29: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

29

GLOBAL SEARCH

Page 30: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

30

What is GlobalSearch?

Multistart heuristic algorithm

Calls fmincon from multiple

start points to try and find a

global minimum

Filters/removes non-promising

start points

Page 31: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

31

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview Schematic Problem

Peaks function

Three minima

Green, z = -0.065

Red, z= -3.05

Blue, z = -6.55

x

y

Page 32: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

32

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 0Run from specified x0

x

y

Page 33: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

33

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

00

4

0

-2

x

y

Page 34: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

34

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

3

6

0

00

4

0

-2

x

y

Page 35: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

35

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 1

x

y

Page 36: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

36

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

Page 37: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

37

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

Page 38: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

38

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

Page 39: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

39

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

Page 40: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

40

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

6

Current penalty

threshold value : 4

x

y

Page 41: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

41

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

x

y

Page 42: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

42

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2

Current penalty

threshold value : 4

-3

x

y

Page 43: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

43

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

GlobalSearch Overview – Stage 2Expand basin of attraction if minimum already found

Current penalty threshold value : 2

-0.1

x

y Basins can overlap

Page 44: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

44

GlobalSearch Demo – Peaks Function

Page 45: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

45

SIMULATED ANNEALING

Page 46: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

46

What is Simulated Annealing?

A probabilistic metaheuristic

approach based upon the physical

process of annealing in

metallurgy.

Controlled cooling of a metal

allows atoms to realign from a

random higher energy state to an

ordered crystalline (globally) lower

energy state

Page 47: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

47

Simulated Annealing Overview – Iteration 1Run from specified x0

x

y

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

0.9

Page 48: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

48

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Possible New Points:

Standard Normal N(0,1) * Temperature

Temperature = 1

Page 49: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

49

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

11.01

1/)9.03(

Taccepte

P

Page 50: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

50

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

0.3

Page 51: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

51

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 1

3

x

y 0.9

Temperature = 1

0.3

Page 52: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

52

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2

3

x

y 0.9

Temperature = 1

0.3

Page 53: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

53

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration 2

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

Page 54: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

54

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N-1

3

x

y 0.9

Temperature = 0.1

0.3

-1.2

-3

Page 55: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

55

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

Page 56: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

56

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

27.01

1/))3(2(

Taccepte

P

Page 57: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

57

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration NReannealing

3

x

y 0.9

Temperature = 1

0.3

-1.2

-3

-2

Page 58: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

58

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

Page 59: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

59

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration N+1

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

Page 60: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

60

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Simulated Annealing Overview – Iteration …

3

x

y 0.9

Temperature = 0.75

0.3

-1.2

-3

-2

-3

-6.5

Page 61: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

61

Simulated Annealing – Peaks Function

Page 62: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

62

PATTERN SEARCH

(DIRECT SEARCH)

Page 63: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

63

What is Pattern Search?

An approach that uses a pattern

of search directions around the

existing points

Expands/contracts around the

current point when a solution is

not found

Does not rely on gradients: works

on smooth and nonsmooth

problems

Page 64: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

64

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Run from specified x0

x

y

3

Page 65: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

65

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 1Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

Page 66: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

66

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 2

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Complete Poll

Page 67: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

67

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 3

x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Page 68: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

68

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration 4

x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Page 69: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

69

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search Overview – Iteration NContinue expansion/contraction until convergence…

x

y

31.6

0.4

4.6

2.8

-4

0.3-2.8

-6.5

Page 70: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

70

Pattern Search – Peaks Function

Page 71: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

71

Pattern Search Climbs Mount Washington

Page 72: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

72

PARTICLE SWARM

Page 73: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

73

What is Particle Swarm Optimization?

A collection of particles move

throughout the region

Particles have velocity and are

affected by the other particles in

the swarm

Does not rely on gradients: works

on smooth and nonsmooth

problems

Page 74: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

74

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration 1Initialize particle locations and velocities, evaluate all

locations

x

y

Page 75: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

75

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

yPrevious

Velocity

Best Location

for this Particle

Best Location for

Neighbor Particles

Page 76: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

76

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NUpdate velocities for each particle

x

y

New

Velocity

Page 77: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

77

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NMove particles based on new velocities

x

y

Page 78: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

78

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Particle Swarm Overview – Iteration NContinue swarming until convergence

x

y

Page 79: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

79

Particle Swarm – Peaks Function

Page 80: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

80

GENETIC ALGORITHM

Page 81: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

81

What is a Genetic Algorithm?

Uses concepts from evolutionary

biology

Start with an initial generation of

candidate solutions that are tested

against the objective function

Subsequent generations evolve

from the 1st through selection,

crossover and mutation

Page 82: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

82

How Evolution Works – Binary Case

Selection

– Retain the best performing bit strings from one generation to the next. Favor these for

reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

Page 83: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

83

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Evaluate initial population

x

y

Page 84: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

84

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Select a few good solutions for reproduction

x

y

Page 85: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

85

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2Generate new population and evaluate

x

y

Page 86: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

86

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2

x

y

Page 87: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

87

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

Page 88: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

88

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

Page 89: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

89

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration NContinue process until stopping criteria are met

x

y

Solution found

Page 90: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

90

Genetic Algorithm – Peaks Function

Page 91: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

91

Genetic Algorithm – Integer Constraints

Mixed Integer Optimization

s.t. some x are integers

Examples

Only certain sizes of components

available

Can only purchase whole shares of stock

)(min xfx

Page 92: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

92

Application: Circuit Component Selection

6 components to size

Only certain sizes available

Objective:

– Match Voltage vs. Temperature

curve

Thermistor Circuit

300 Ω

330 Ω

360 Ω

180k Ω

200k Ω

220k Ω

Thermistors:

Resistance varies

nonlinearly with

temperature

?

𝑅𝑇𝐻 =𝑅𝑇𝐻,𝑁𝑜𝑚

𝑒𝛽𝑇−𝑇𝑁𝑜𝑚𝑇∗𝑇𝑁𝑜𝑚

Page 93: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

93

Global Optimization Toolbox Solvers

GlobalSearch, MultiStart

– Well suited for smooth objective and constraints

– Return the location of local and global minima

ga, gamultiobj, simulannealbnd, particleswarm

– Many function evaluations to sample the search space

– Work on both smooth and nonsmooth problems

patternsearch

– Fewer function evaluations than ga, simulannealbnd, particlewarm

– Does not rely on gradient calculation like GlobalSearch and MultiStart

– Works on both smooth and nonsmooth problems

Page 94: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

94

Optimization Toolbox Solvers

fmincon, fminbnd, fminunc, fgoalattain, fminimax

– Nonlinear constraints and objectives

– Gradient-based methods for smooth objectives and constraints

quadprog, linprog

– Linear constraints and quadratic or linear objective, respectively

intlinprog

– Linear constraints and objective and integer variables

lsqlin, lsqnonneg

– Constrained linear least squares

lsqnonlin, lsqcurvefit

– Nonlinear least squares

fsolve

– Nonlinear equations

Page 95: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

95

Speeding-up with Parallel Computing

Global Optimization solvers that support Parallel

Computing:– ga, gamultiobj: Members of population evaluated in parallel

at each iteration

– patternsearch: Poll points evaluated in parallel at each

iteration

– particleswarm: Population evaluated in parallel at each

iteration

– MultiStart: Start points evaluated in parallel

Optimization solvers that support Parallel Computing:– fmincon: parallel evaluation of objective function for finite

differences

– fminunc, fminimax, fgoalattain,fsolve,

lsqcurvefit, lsqnonlin: same as fmincon

Parallel Computing can also be used in the Objective

Function– parfor

Page 96: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

96

Speed up parallel applications

Take advantage of GPUs

Prototype code for your cluster

Parallel Computing Toolbox for the Desktop

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

Page 97: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

97

Scale Up to Clusters and Clouds

Cluster

Scheduler

Computer Cluster

… … …

Simulink, Blocksets,

and Other Toolboxes

Local

Desktop Computer

MATLAB

……

Page 98: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

98

Learn More about Optimization with MATLAB

Recorded webinar: Optimization in

MATLAB: An Introduction to Quadratic

Programming

Optimization Toolbox Web demo:

Finding an Optimal Path using MATLAB

and Optimization Toolbox

MATLAB Digest: Improving

Optimization Performance with Parallel

Computing

MATLAB Digest: Using Symbolic

Gradients for OptimizationRecorded webinar: Mixed Integer

Linear Programming in MATLAB

Recorded webinar: Optimization in

MATLAB for Financial Applications

1 2 3 4 50

500

1000

1500

2000

2500

3000

Bonds

# P

urc

ha

se

d

Cash Flow Matching Example

Page 99: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

99

Key Takeaways

Solve a wide variety of optimization problems in MATLAB

– Linear and Nonlinear

– Continuous and mixed-integer

– Smooth and Nonsmooth

Find better solutions to multiple minima and non-smooth problems using global

optimization

Use symbolic math for setting up problems and automatically calculating gradients

Using parallel computing to speed up optimization problems

Page 100: Solving Optimization Problems with MATLAB - … · Symbolic Math Toolbox Functions for analytical computations ... Global Search ... problems. 64 -3 -2 -1 0 1 2 3-3-2

100© 2016 The MathWorks, Inc.

Questions?


Top Related