optimal control slide

Optimal Control Problems with Control and State Constraints Numrical Method: Discretize and Optimize Theory of Optimal Control Problems with Mixed Control-State Constraints Example: Rayleigh Problem with Different Constraints and Objectives Example: Optimal Exploitation of Renewable Resources

Tutorial on Control and State Constrained OptimalControl Problems

Part 2 : Mixed Control-State Constraints

Helmut Maurer

University of Munster, GermanyInstitute of Computational and Applied Mathematics

SADCO Summer School

Imperial College London, September 5, 2011

Helmut Maurer University of Munster, Germany Institute of Computational and Applied MathematicsTutorial on Control and State Constrained Optimal Control Problems [0.5mm] Part 2 : Mixed Control-State Constraints

inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Outline

1 Optimal Control Problems with Control and State Constraints

2 Numrical Method: Discretize and Optimize

3 Theory of Optimal Control Problems with Mixed Control-StateConstraints

4 Example: Rayleigh Problem with Different Constraints andObjectives

5 Example: Optimal Exploitation of Renewable Resources


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Control Problem

state x(t) Rn, control u(t) Rm .Dynamics and Boundary Conditions

x(t) = f (t, x(t), u(t)), a.e. t [0, tf ],x(t) = x0 , (x(tf )) = 0 ( : Rn Rr ).

Control and State Constraints

c(x(t), u(t)) 0 , 0 t tf , ( c : Rn Rm Rk )s(x(t)) 0 , 0 t tf , ( s : Rn Rl )

Minimize

J(u, x) = g(x(tf )) +

tf0

f0(t, x(t), u(t)) dt


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Discretization

For simplicity consider a MAYERtype problem with cost functional

J(u, x) = g(x(tf )) .

This can achieved by considering the additional state variable x0with

x0 = f0(x , u) , x0(0) = 0 .

Then we have

x0(tf ) =

tf0

f0(t, x(t), u(t)) .

Choose an integer N N, a stepsize h and grid points ti :h = tf /N , ti : = ih , (i = 0, 1, . . . ,N) .

Approximation of control and state at grid points:

u(ti ) ui Rm , x(ti ) xi Rn (i = 0, . . . ,N)Helmut Maurer University of Munster, Germany Institute of Computational and Applied MathematicsTutorial on Control and State Constrained Optimal Control Problems [0.5mm] Part 2 : Mixed Control-State Constraints

inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Large-scale NLP using EULERs method

Minimize

J(u, x) = g(xN)

subject to

xi+1 = xi + h f (ti , xi , ui ), i = 0, ..,N 1,x0 = x0 , (xN) = 0,

c(xi , ui ) 0 , i = 0, ..,N,s(xi ) 0 , i = 0, ..,N,

Optimization variable for full discretization:

z := (u0, x1, u1, x2, ..., uN1, xN , uN) RN(m+n)+m


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


NLP Solvers

AMPL : Programming language (Fourer, Gay, Kernighan)

IPOPT : Interior point method (Andreas Wachter)

LOQO : Interior point method (Vanderbei et al.)

Other NLP solvers embedded in AMPL : cf. NEOS server

NUDOCCCS : optimal control package (Christof Buskens)

WORHP : SQP solver (Christof Buskens, Matthias Gerdts)

Special feature: solvers provide LAGRANGE-multipliers asapproximations of the adjoint variables.


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Control Problem with Control-State Constraints

State x(t) Rn, Control u(t) Rm.All functions are assumed to be suffciently smooth

Dynamics and Boundary Conditions

x(t) = f (x(t), u(t)), a.e. t [0, tf ],x(0) = x0 Rn, (x(tf )) = 0 Rk ,

( 0 = (x(0), x(tf )) mixed boundary conditions )

Mixed Control-State Constraints

c(x(t), u(t)) , t [0, tf ], c : Rn Rm RControl bounds u(t) are included by c(x , u) = u.

Minimize

J(u, x) = g(x(tf )) +

tf0

f0(x(t), u) dt


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Hamiltonian

Hamiltonian

H(x , , u) = 0 f (x , u) + f (x , u) Rn (row vector)Augmented Hamiltonian

H(x , , , u) = H(x , , u) + c(x , u)= 0 f (x , u) + f (x , u) + c(x , u), R .

Let (u, x) L([0,T ],Rm)W1,([0,T ],Rn) be alocally optimal pair of functions.

Regularity assumption

cu(x(t), u(t)) 6= 0 t JaJa := { t [0, tf ] | c(x(t), u(t)) = or = }


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Minimum Principle of Pontryagin et al. and Hestenes

Let (u, x) L([0, tf ],Rm)W1,([0, tf ],Rn) be a locallyoptimal pair of functions that satisfies the regularityassumption. Then there exist

an adjoint (costate) function W1,([0, tf ],Rn) and ascalar 0 0 ,a multiplier function L([0, tf ],R),and a multiplier Rr associated to the boundarycondition (x(tf )) = 0

that satisfy the following conditions for a.a. t [0, tf ], wherethe argument (t) denotes evaluation along the trajectory(x(t), u(t), (t)) :


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Minimum Principle of Pontraygin et al. and Hestenes

(i) Adjoint ODE and transversality condition:

(t) = Hx(t) = (0 f0 + f )x(t) (t) cx(t) ,(tf ) = (0 g + )x(x(tf )) ,

(iia) Minimum Condition for Hamiltonian:

H(x(t), (t), u(t)) = min {H(x(t), (t), u) | c(x(t), u) }(iib) Local Minimum Condition for Augmented Hamiltonian:

0 = Hu(t) = (0 f0 + f )u(t) + (t) cu(t)(iii) Sign of multiplier and complementarity condition:

(t) 0, if c(x(t), u(t)) = ; (t) 0, if c(x(t), u(t)) = ,(t) = 0 , if < c(x(t), u(t)) < .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Evaluation of the Minimum Principle: boundary arc

Boundary arc: Let [t1, t2] , 0 t1 < t2 < tf , be an interval with

c(x(t), u(t)) = or c(x(t), u(t)) = t1 t t2 .

For simplicity assume a scalar control, i.e., m = 1 .Due to the regularity condition cu(x(t), u(t)) 6= 0 there exists asmooth function ub(x) satisfying

c(x , ub(x)) ( ) x in a neighborhood of the trajectory.

The control ub(x) is called the boundary control and yields the

optimal control by the relation u(t) = ub(x(t)) .

It follows from the local minimum condition 0 = Hu = Hu + cuthat the multiplier is given by

= (x , ) = Hu(x , , ub(x)) / cu(x , ub(x)) .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I : Regular Hamiltonian, u is continuous

CASE I : Consider optimal control problems which satisfy the

Assumption: The Hamiltonian H(x , , u) is regular, i.e., it admits aunique minimum u. The strict Legendre condition holds:

Huu(t) > 0 t [0, tf ] .(a) Then there exists a free control u = u free(x , ) satisfying

Hu(x , , u free(x , )) 0 .(b) The optimal control u(t) is continuous in [0, tf ] .

Claim (b) follows from the continuity and regularity of H.

The continuity of the control implies junctions conditions atjunction points tk (k = 1, 2) with the boundary:

u free(x(tk), (tk)) = ub(x(tk)) , (tk) = 0 (k = 1, 2).


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Rayleigh Problem with Quadratic Control

The Rayleigh problem is a variant of the van der Pol Oszillator,where x1 denotes the electric current.

Control problem for the Rayleigh Equation

Minimize J(x , u) =tf0

(u2 + x21 ) dt (tf = 4.5)

subject to

x1 = x2, x1(0) = 5,x2 = x1 + x2(1.4 0.14x22 ) + 4u, x2(0) = 5.

Three types of constraints:

Case (a) : no control constraints.Case (b) : control constraint 1 u(t) 1 .Case (c) : mixed control-state constraint

u(t) + x1(t)/6 0, = 1,2Helmut Maurer University of Munster, Germany Institute of Computational and Applied MathematicsTutorial on Control and State Constrained Optimal Control Problems [0.5mm] Part 2 : Mixed Control-State Constraints

inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I (a) : Rayleigh problem, no constraint

Normal Hamiltonian:

H(x , , u) = u2 + x21 + 1 x2 + 2 (x1 + x2(1.4 0.14x22 ) + 4u)

Adjoint Equations:

1 = Hx1 = 2x1 + 2 1(tf ) = 0,2 = Hx2 = 1 2(1.4 0.42x22 ) 2(tf ) = 0,

Minimum condition:

0 = Hu = 2u + 42 u = u free(x , ) = 22 .

Shooting method for solving the boundary value problem for (x , ):Determine unknown shooting vector s = (0) R2 that satisfiesthe terminal condition (tf ) = 0 : use Newtons method


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I : Rayleigh problem without constraints

-6-5-4-3-2-1 0 1 2 3 4 5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 1

, x 2

state variables x1 , x2

-5-4-3-2-1 0 1 2 3 4 5

-6 -5 -4 -3 -2 -1 0 1

( x 1, x

2)

phaseportrait (x1,x2)

-2-1 0 1 2 3 4 5 6 7

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-9-8-7-6-5-4-3-2-1 0 1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

l am

b da 1

,

l a

mb d

a 2

adjoint variables lambda1 , lambda2

Note: Hamiltionian is regular, control u(t) is continuous (analytic).


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I (b) : Rayleigh problem, constraint 1 u(t) 1Hamiltonian H and adjoint equations are as in Case (a).The free control is given by u free(x , ) = 22 .Structure of optimal control:

u(t) =

1 for 0 t t1

22(t) for t1 t t21 for t2 t t3

22(t) for t3 t tf

Junction conditions: Continuity of the control implies

u(tk) = 22(tk) = 1 | 1 | 1 , k = 1, 2, 3 .

Shooting method for solving the boundary value problem for (x , ):Determine shooting vector s = ((0), t1, t2, t3) R2+3 thatsatisfies 2 terminal conditions (tf ) = 0 and 3 junction conditions.


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Rayleigh problem with control constraint | u(t) | 1

-8

-6

-4

-2

0

2

4

6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 1

, x 2


-5-4-3-2-1 0 1 2 3 4 5

-7 -6 -5 -4 -3 -2 -1 0 1

( x 1, x

2)

phaseportrait (x1,x2)

-1

-0.5

0

0.5

1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-14-12-10

-8-6-4-2 0 2

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

l am

b da 1

,

l a

mb d

a 2


Note: Hamiltonian is regular, control u(t) is continuous.Junction conditions: 22(tk) = 1 | 1 | 1 , k = 1, 2, 3


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I (c) : Rayleigh problem, 1 u + x1/6 0Augmented (normal) Hamiltonian:

H(x , , , u) = u2 + x21 + 1 x2+2 (x1 + x2(1.4 0.14x22 ) + 4u) + (u + x1/6)

Adjoint Equations:

1 = Hx1 = 2x1 + 2 /6 , 1(tf ) = 0,2 = Hx2 = 1 2(1.4 0.42x22 ) 2(tf ) = 0,

Free control : u free(x , ) = 22 .Boundary control : ub(x) = x1/6 for {1, 0} .

Multiplier :

= (x , ) = Hu(x , , ub(x)) / cu(x , ub(x)) = 2ub(x) + 42 .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case I (c) : Rayleigh problem, 1 u + x1/6 0

-1.5

-1

-0.5

0

0.5

1

1.5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-1

-0.8

-0.6

-0.4

-0.2

0

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

C ( x, u )

= u +

x 1/ 6

mixed constraint -1


Case I (c) : Rayleigh problem, structure of optimal controlfor mixed constraint 1 u + x1/6 0

-1

-0.8

-0.6

-0.4

-0.2

0

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

C ( x, u )

= u + x

1/ 6

mixed constraint -1


Case II : control u appears linearly

CASE II : Control appears linearly in the cost functional, dynamicsand mixed control-state constraint. Let u be scalar.

Dynamics and Boundary Conditions

x(t) = f1(x(t)) + f2(x(t)) u(t), a.e. t [0, tf ],x(0) = x0 Rn, (x(tf )) = 0 Rk ,

Mixed Control-State Constraints

c1(x(t)) + c2(x(t)) u(t) t [0, tf ]. c1, c2 : Rn R

Minimize

J(u, x) = g(x(tf )) +

tf0

( f01(x(t)) + f02(x(t) ) u(t) dt


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Hamiltonian and switching function

Normal Hamiltonian

H(x , , u) = f01(x) + f1(x) + [ f02(x) + f2(x) ] u .

Augmented Hamiltonian

H(x , , , u) = H(x , , u) + (c1(x) + c2(x) u)

The optimal control u(t) solves the minimization problem

min {H(x(t), (t), u) | c1(x(t)) + c2(x(t)) u }

Define the switching function

(x , ) = Hu(x , , u) = f02(x) + f2(x) , (t) = (x(t), (t)) .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Hamiltonian

The minimum condition is equivalent to the minimization problem

min {(t) u) | c1(x(t)) + c2(x(t)) u }We deduce the control law

c1(x(t))+c2(x(t))u(t) =

, if (t) c2(x(t)) > 0 , if (t) c2(x(t)) < 0

undetermined if (t) 0

The control u is called bang-bang in an interval I [0, tf ], if(t) c2(x(t)) 6= 0 for all t I . The control u is called singular inan interval Ising [0, tf ], if (t) c2(x(t)) 0 for all t Ising.For the control constraint u(t) with c1(x) = 0, c2(x) = 1we get the classical control law

u(t) =

, if (t) > 0 , if (t) < 0

undetermined , if (t) 0


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Bang-Bang and Singular Controls


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Rayleigh problem with 1 u(t) 1Rayleigh problem with control appearing linearly


(x21 + x22 ) dt (tf = 4.5)

subject to

x1 = x2, x1(0) = 5 ,x2 = x1 + x2(1.4 0.14x22 ) + 4u , x2(0) = 5,1 u(t) 1 .

Adjoint Equations:

1 = Hx1 = 2x1 + 2 , 1(tf ) = 0,2 = Hx2 = 2x2 1 2(1.4 0.42x22 ) , 2(tf ) = 0,

The switching function (t) = Hu(t) = 42(t) gives thecontrol law

u(t) = sign (2(t)).


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Rayleigh problem, 1 u(t) 1

-1

-0.5

0

0.5

1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-8

-6

-4

-2

0

2

4

6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 1

, x 2


-1

-0.5

0

0.5

1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u , l

a mb d

a 2/ 1

0

control u and (scaled) switching function

-18-16-14-12-10

-8-6-4-2 0 2 4

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

l am

b da 1

,

l a

mb d

a 2


Control u(t) is bang-bang-singular.Switching conditions: (t1) = 0 , 2(t) 0 t [t2, tf ].


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Rayleigh problem, u + x1/6 0


(x21 + x22 ) dt (tf = 4.5)

subject to

x1 = x2, x1(0) = 5 ,x2 = x1 + x2(1.4 0.14x22 ) + 4u , x2(0) = 5,

and the mixed control-state constraint

u(t) + x1(t)/6 0 0 t tf .

Adjoint Equations:

1 = Hx1 = 2x1 + 2 /6 , 1(tf ) = 0,2 = Hx2 = 2x2 1 2(1.4 0.42x22 ) , 2(tf ) = 0,


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Control law for u + x1/6 0

The switching function is (t) = Hu(t) = 42(t) .In view of c2(x) 1 we have the control law

u + x1/6 =

< 0 , if 2(t) > 00 , if 2(t) < 0undetermined , if 2(t) 0


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Case II : Rayleigh problem, 1 u + x1/6 0

-1.5

-1

-0.5

0

0.5

1

1.5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-8

-6

-4

-2

0

2

4

6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 1

, x 2


-2

-1.5

-1

-0.5

0

0.5

1

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u , l

a mb d

a 2/ 1

0

C(x,u) and (scaled) switching function

-40

-30

-20

-10

0

10

20

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

mu

multiplier for constraint -1


Case II : Rayleigh problem, 2 u + x1/6 0

-2

-1.5

-1

-0.5

0

0.5

1

1.5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

u

optimal control u

-8

-6

-4

-2

0

2

4

6

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

x 1

, x 2


-2

-1.5

-1

-0.5

0

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

c ( x, u )

, la m

b da 2

/ 10

c(x,u) and (scaled) switching function

-40-35-30-25-20-15-10

-5 0 5

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5

mu

multiplier for constraint -1


Optimal Fishing, Clark, Clarke, Munro

Colin W. Clark, Frank H. Clarke, Gordon R. Munro:The optimal exploutation of renewable resource stock: problem ofirreversible investment, Econometric 47, pp. 2547 (1979).

State variables and control variables:

x(t) : population biomass at time t [0, tf ] ,renewable resource, e.g., fish,

K (t) : amount of capital invested in the fishery,e.g., number of standardized fishing vessels available,

E (t) : fishing effort (control), h(t) = E (t)x(t) is harvest rate ,

I (t) : investment rate (control),


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Fishing: optimal control model

Dynamics in [0, tf ] ( here: a = 1, b = 5, = 0 )

x(t) = a x(t) (1 x(t)/b) E (t) x(t) , x(0) = x0 ,K (t) = I (t) K (t) , K (0) = K0 .

Mixed Control-State Constraint and Control Constraint

0 E (t) K (t) , 0 I (t) Imax , t [0, tf ],

Maximize benefit ( parameters: r = 0.05, cE = 2, cI = 1.1 )

J(u, x) =

tf0

exp(r t)( p E (t) x(t) cE E (t) cI I (t) ) dt


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Fishing: x0 = 0.5, K0 = 0.2, Imax = 0.5

0

0.5

1

1.5

2

0 2 4 6 8 10

controls E, I and state variables 0.5*x , K

"E.dat""I.dat"

"K.dat""x-scal.dat"


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Fishing: x0 = 0.5,K0 = 0.6, Imax = 0.5

0

0.5

1

1.5

2

0 2 4 6 8 10


"E.dat""I.dat"

"K.dat""x-scal.dat"

Fishing rate : E (t) = 0, E (t) = singular, E (t) = K (t).

Investment rate : I (t) = 0, I (t) = Imax , I (t) = 0.


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Fishing: x0 = 0.2,K0 = 0.1, Imax = 0.1

0

0.5

1

1.5

2

0 2 4 6 8 10


"E.dat""I.dat"

"K.dat""x-scal.dat"

Fishing rate : E (t) = 0, E (t) = K (t), E (t) singular, E (t) = K (t).

Investment rate : 2 arcs with I (t) = Imax .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011


Optimal Fishing: x0 = 1.0, K0 = 0.5, Imax = 3

0

0.5

1

1.5

2

2.5

3

3.5

4

0 2 4 6 8 10


"E.dat""I.dat"

"K.dat""x-scal.dat"

Fishing rate : E (t) = 0, E (t) singular E (t) = K (t),

Investment rate : 1 impulse with I (t) = Imax .


inria

-006

2951

8, v

ersio

n 1

- 6 O

ct 2

011

Optimal Control Problems with Control and State ConstraintsNumrical Method: Discretize and OptimizeTheory of Optimal Control Problems with Mixed Control-State ConstraintsExample: Rayleigh Problem with Different Constraints and ObjectivesExample: Optimal Exploitation of Renewable Resources

optimal control slide

Documents

approximation of control

control ut rm

state constraintscxt

objectives example

rayleigh problem

mayertype problem

applied mathematicstutorial

integer n n