optimization in matlab - mathworks.com · optimization in matlab seth deland. 2 topics ......

1© 2014 The MathWorks, Inc.

Optimization in MATLAB

Seth DeLand

2

Topics

Intro

Using gradient-based solvers

Optimization in Comp. Finance toolboxes

Global optimization

Speeding up your optimizations

3

Optimization workflow

Initial

Design

Variables

Model

Modify

Design

Variables

Optimal

DesignObjectives

met?

No

Yes

4

Optimization Problems

Minimize Risk

Maximize Profits

Maximize Fuel Efficiency

5

Demo: Solving an optimization problem

Simple objective function of 2 variables:

𝑓 𝑥 = log 1 + 3 𝑥2 − 𝑥13− 𝑥1

2

+ 𝑥1 −4

3

2

Bound constraints:

−2.5 ≤ 𝑥1 ≤ 2.5

−2.5 ≤ 𝑥2 ≤ 2.5

-2

-1

0

1

2

-2

-1

0

1

2

0

2

4

6

6

Portfolio Optimization – Quadratic Constraints

Maximize returns, constraint is risk

minimize − 𝑓𝑇𝑥 𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜1

2𝑥𝑇𝐶𝑥 ≤ 𝐶𝑚𝑎𝑥

Use fmincon interior-point algorithm

– Specify analytic gradients of objective function and constraints

𝑔 = −𝑓 (gradient of objective function)

𝑔𝑐 = 𝐶𝑥 (gradient of constraint)

– Hessian (H) of Lagrangian L = −𝑓𝑇𝑥 + λ(1

2𝑥𝑇𝐶𝑥 − 𝐶𝑚𝑎𝑥)

H = ∇2 L = λC

This is a second order conic programming problem

7

Demo: Cash-flow matching

Idea: Buy bonds to cover pension

fund obligations

Variables: How many of each

bond to buy?

Constraints: Payments from

bonds must be greater than or

equal to pension fund obligations

Objective: Minimize the size of

the investment you make 1 2 3 4 5 6 7 80

20

40

60

80

100

120

Time Period

Ca

sh

Flo

w (

$)

1 2 3 4 5 6 7 80

5

10

15x 10

5

Time Period

Pa

ym

en

ts (

$)

8

𝑛1⋮𝑛5

Extending the problem

−𝑐𝑎𝑠ℎ𝐹𝑙𝑜𝑤𝑠

𝑛1⋮𝑛5

≤ −𝑜𝑏𝑙𝑖𝑔𝑎𝑡𝑖𝑜𝑛𝑠

𝑛1⋮𝑛5𝑦1⋮𝑦5

1

0⋱

0

1

−1000

0⋱

0

−1000

𝑛1⋮𝑛5𝑦1⋮𝑦5

=0⋮0

min𝑥

𝑓𝑇𝑥 𝐴𝑥 ≤ 𝑏

𝐴𝑒𝑞𝑥 = 𝑏𝑒𝑞𝑙𝑏 ≤ 𝑥 ≤ 𝑢𝑏

s.t.

𝑥(intcon) must be integers𝑥:

𝑛1 = 1000𝑦1⋮

𝑛5 = 1000𝑦5

−𝑐𝑎𝑠ℎ𝐹𝑙𝑜𝑤𝑠 𝑧𝑒𝑟𝑜𝑠(8,5)

𝑛1⋮

𝑛5𝑦1⋮

𝑦5

≤ −𝑜𝑏𝑙𝑖𝑔𝑎𝑡𝑖𝑜𝑛𝑠

9

Optimization Toolbox solvers

Linear and Mixed-Integer

• LINPROG

• INTLINPROG

Quadratic

• QUADPROG

Nonlinear

• FMINCON

• FMINUNC

• FMINBND

• FMINSEARCH

• FSEMINF

Least Squares

• LSQLIN

• LSQNONNEG

• LSQCURVEFIT

• LSQNONLIN

Nonlinear Equation Solving

• FSOLVE

• FZERO

Multiobjective

• FGOALATTAIN

• FMINIMAX

10

Key optimization problems addressed by

financial toolboxes

• Mean-variance portfolio optimization

• Conditional Value at Risk (CVaR) portfolio optimization

Financial Toolbox

• Parameter estimation of conditional mean/variance models

Econometrics Toolbox

• Hedging portfolios

• Fitting interest rate curves

• Bootstrapping

Financial Instruments

Toolbox

11

-2

0

2

-3-2

-10

12

3

-6

-4

-2

0

2

4

6

8

x

Peaks

y

Global optimization algorithms

MultiStart

GlobalSearch

Patternsearch

Genetic Algorithm

Simulated Annealing

Local minimaGlobal

minima

12

What is MultiStart?

Run a local solver from

each set of start points

Option to filter starting

points based on feasibility

Supports parallel

computing

13

MultiStart demo – nonlinear regression

lsqcurvefit solution MultiStart solution

14

What is Pattern Search?

An approach that uses a

pattern of search directions

around the existing points

Expands/contracts around

the current point when a

solution is not found

Does not rely on gradients:

works on smooth and

nonsmooth problems

15

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration 1Run from specified x0

x

y

3

16

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration 1Apply pattern vector, poll new points for improvement

x

y

3

Mesh size = 1

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

0_*_ xvectorpatternsizemeshPnew

0]0,1[*1 x

1.6

0.4

4.6

2.8

First poll successful

Complete Poll (not default)

17

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration 2

x

y

3

Mesh size = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

Complete Poll

18

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration 3

x

y

3

Mesh size = 4

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

19

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration 4

x

y

3

Mesh size = 4*0.5 = 2

Pattern vectors = [1,0], [0,1], [-1,0], [0,-1]

1.6

0.4

4.6

2.8

-4

0.3-2.8

20

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Pattern Search overview – Iteration NContinue expansion/contraction until convergence…

x

y

31.6

0.4

4.6

2.8

-4

0.3-2.8

-6.5

21

Patternsearch demo – stochastic function

Stochastic objective function

22

What is a Genetic Algorithm?

Uses concepts from

evolutionary biology

Start with an initial generation

of candidate solutions that are

tested against the objective

function

Subsequent generations

evolve from the 1st through

selection, crossover and

mutation

23

How evolution works – binary case

Selection

– Retain the best performing bit strings from one generation to the next.

Favor these for reproduction

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

Crossover

– parent1 = [ 1 0 1 0 0 1 1 0 0 0 ]

– parent2 = [ 1 0 0 1 0 0 1 0 1 0 ]

– child = [ 1 0 0 0 0 1 1 0 1 0 ]

Mutation

– parent = [ 1 0 1 0 0 1 1 0 0 0 ]

– child = [ 0 1 0 1 0 1 0 0 0 1 ]

24

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Evaluate initial population

x

y

25

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 1Select a few good solutions for reproduction

x

y

26

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2Generate new population and evaluate

x

y

27

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 2

x

y

28

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

29

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration 3

x

y

30

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

Genetic Algorithm – Iteration NContinue process until stopping criteria are met

x

y

Solution found

31

Genetic Algorithm demo - Multiobjective

Use gamultiobj to find Pareto front

Two competing objectives:

Second objective has sinusoidal

component – results in

discontinuous front

32

Global Optimization

Solvers designed to explore the solution space and find

global solutions

Problems can be stochastic, or nonsmooth

Solvers in Global Optimization Toolbox

– MultiStart

– GlobalSearch

– simulannealbnd

– patternsearch

– ga (single- and multi-objective)

Parallel Computing with ga, patternsearch, MultiStart

33

Speeding up with Parallel Computing Toolbox

Global Optimization Toolbox:– ga: Members of population evaluated in parallel at each generation

– patternsearch: Pattern evaluated in parallel (CompletePoll == on)

– MultiStart: Start points evaluated in parallel

Optimization Toolbox:– fmincon: parallel evaluation of objective function for finite differences

– fminimax, fgoalattain: same as fmincon

In the objective function– parfor

34

Key takeaways

Solvers for a wide variety of problems

– New Mixed-Integer Linear Programming Solver

Supply gradient and Hessian if possible

Global solvers for multiple minimum

and nonsmooth problems

Speed up with parallel computing

35

Learn more about optimization with MATLAB

MATLAB Digest: Using Symbolic

Gradients for Optimization

Recorded webinar: Tips and

Tricks – Getting Started Using

Optimization with MATLAB

Recorded webinar: Global

Optimization with MATLAB

Products

MATLAB Digest: Improving

Optimization Performance with

Parallel Computing