chapter 4 the maximum principle and economic … maximum principle and economic interpretations ......

Chapter 4

The Maximum Principle and Economic Interpretations

Until this juncture our development of the necessary and sufficient conditions of optimal control

theory was essentially a reformulation of those from the classical calculus of variations. As a re-

sult, the power and reach of optimal control theory has not been fully exposed nor exploited.

The goal of this chapter, therefore, is to state and prove some necessary and sufficient conditions

for a class of control problems that permit the full capability of optimal control theory to be re-

alized. The theorems are not the most general we will encounter, but they do highlight the moti-

vation for the name Maximum Principle. Moreover, our proof of the necessary conditions will

employ some continuity assumptions that are, strictly speaking, not needed for a rigorous proof,

but will none the less be employed so as to ease the technical burden and bring in some connec-

tions with the Principle of Optimality and dynamic programming. We will then use the neces-

sary conditions to explicitly solve for the optimal paths of some examples, some of which will be

devoid of any economic content so as to emphasize how to arrive at a solution in practice. The

reader is encouraged to work through the proof of the necessary and sufficient conditions, as it

provides the reader with a better understanding of how they differ from those presented earlier.

Note that the theorems are stated and proven for a class of control problems with many state and

control variables.

The optimal control problem under consideration is to find a piecewise continuous con-

trol vector function u( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )defu u uM1 2 and its associated piecewise smooth state vector

function x( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )defx x xN1 2 , defined on the fixed time interval [ , ]t t0 1 that will solve the

ensuing optimal control problem

V t t J f t t t dtt

t

( , , , ) max [ ( ), ( )] , ( ), ( )( )

0 0 1 1

0

1

x x x u x uu

= ◊ ◊ = ( )ÏÌÔ

ÓÔ

¸˝ÔÔ◊ Údef def

s.t. ˙ ( ) , ( ), ( )x g x ut t t t= ( ) (1)

u x x x x( ) , ( ) , ( )t U t tMŒ Õ ¬ = =0 0 1 1,

Chapter 4: The Maximum Principle and Economic Interpretations

100

where g( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )defg g gN1 2 and ˙ ( ) ˙ ( ), ˙ ( ), , ˙ ( )x ◊ = ◊ ◊ º ◊( )def

x x xN1 2 . Notice that the terminal

boundary condition x x( )t1 1= specifies that the value of the state variable must be equal to some

given (i.e., fixed) value at the terminal date of the planning horizon, since x1 is not specified to

be a decision variable in the problem statement. Thus problem (1) is a fixed endpoints problem,

unlike that encountered in Chapters 2 and 3, where x x( )t1 1= was a decision variable. Also re-

call that the control set U specifies the region in which the values of the control variable must lie,

and that it is a fixed subset of ¬M which is assumed to be independent of the state variables.

We will relax this assumption in a later chapter.

In typical economic problems V t t( , , , )0 0 1 1x x represents the maximum present discounted

profit of a firm that begins its operations with the capital stock x0 at time t0 , and ends its opera-

tions with the capital stock x1 at time t1. Alternatively, V t t( , , , )0 0 1 1x x is the maximum total value

of the asset x0 at time t0 , and thus is the purchase price of the asset x0 at time t0 , since the

maximum present value of the stream of net benefits from holding x0 at time t0 and using the

optimal plan is V t t( , , , )0 0 1 1x x . The function V ( )◊ is called the optimal value function.

Because we are allowing for a more general class of control variables than we did earlier,

we must amend our definition of an admissible control. To this end we have the following defi-

nition, which should be contrasted with that given earlier in Definition 2.1.

Definition 4.1: We call x u( ), ( )t t( ) an admissible pair if u( )◊ is any piecewise continuous con-

trol vector function such that u( ) [ , ]t U t t tŒ " Œ 0 1 and x( )◊ is a piecewise smooth state vector

function such that ˙ ( ) , ( ), ( )x g x ut t t t= ( ) , x x( )t0 0= , and x x( )t1 1= .

In developing the necessary conditions in Chapter 2, we found it convenient to define a

function called the Hamiltonian, and will find it convenient to do so here too. In the present case

there are N state variables and N ordinary differential equations describing their rates of changes

with respect to time. Consequently, we associate with each of the differential equations a costate

function, say ln ( )◊ , n N= º1 2, , , , and form the Hamiltonian for problem (1), namely


101

H t f t g t

f t t

nn

n

N

( , , , ) ( , , ) ( , , )

( , , ) ( , , ),

x u x u x u

x u g x u

ll

ll

= +

= + ¢

=Âdef

def

l1 (2)

where ll = ºdef( , , , )l l l1 2 N and ¢ =

=Âll g x u x u( , , ) ( , , )t g tnn

n

Ndef l1

is the scalar or inner product of the

vectors ll and g x u( , , )t .

The ensuing proof of the necessary conditions for problem (1) will rely on the following

three assumptions, in addition to the fundamental ones stated in Chapter 1:

(A.1) Let S NŒ ¬ be a nonempty set of points from which one can bring the system to the ter-

minal point x1 at time t1 by the use of an optimal control, where x0 ŒS .

(A.2) S is an open set.

(A.3) V C t t t S( ) ( , ) ( , )( )◊ Œ " Œ ¥20 1x , where V ( )◊ is the optimal value function for problem (1).

Assumption (A.1) means that there exists values of the state vector such that when we take the

initial state as one of these values, we can arrive at the given terminal state by the terminal date

of the planning horizon using an optimal control. Assumption (A.2) says that the boundary of

the set S is not part of S, that is to say, S consists only of interior points. The C( )2 nature of the

optimal value function in assumption (A.3) is adopted so as to avoid getting caught up in techni-

cal details. It will be pointed out during the proof of the necessary conditions where these as-

sumptions are used. The approach we take below to prove the necessary conditions of problem

(1) is often referred to as the “dynamic programming proof” of the necessary conditions. As you

will shortly see, this method of proof is quite different from the variational proof of the necessary

conditions given in Chapter 2.

Before we state and prove the necessary conditions, it is preferable to introduce the key

principle behind the forgoing proof, videlicet, the principle of optimality. The principle of opti-

mality is illustrated in Figure 4.1 for an optimal control problem with one control variable. Let

the curve v s( ) be the optimal path of the control variable for the entire planning horizon [ , ]t t0 1 ,

beginning with stock x0 at time t0 and ending with stock x1 at time t1. In Figure 4.1 this path is

divided into two parts, A and B, with time s t= being the break point in the time horizon. The


102

principle of optimality asserts that trajectory B, defined for the interval [ , ]t t1 , must, in its own

right, be the optimal path for this interval given that at time s t= , t t tŒ( , )0 1 , the initial state is

taken as z t( ) , the terminal state from the interval [ , ]t t0 . Notice that we have not drawn the opti-

mal path of the control variable as a differentiable function of time so as to emphasize the more

general character of the optimal control problem we are studying in this chapter relative to that

studied in the prior two chapters.

s

v(s)

v(t1)

v(t0)

v(t)

t0 t1t

A

B

Figure 4.1

An alternative view of the principle of optimality is given in Figure 4.2, where the prob-

lem is to find the least cost path from the initial state x0 to the terminal state xT . The numbers

along each arc indicate the cost of moving between the adjacent points. First note that the opti-

mal path is not unique and is given by the two thick paths. They represent the least cost paths of


103

reaching the terminal state xT from the given initial state x0 . In this context the principle of op-

timality asserts that, roughly speaking, if you chop off the first arc (or any beginning set of arcs)

from an optimal sequence of arcs, the remaining abridged sequence of arcs must still be optimal

in its own right. In other words, the principle of optimality asserts that any portion of an optimal

arc is optimal.

••

•

••

•

•

•

•

•

•

CG

D

E

F

N

M

L

B

A

H

I

3

3

3

3

3

3

t = 3 t = 2 t = 1 t = 0 t = T

6

4

14

6

Figure 4.2

x0

xT

2

1

21

2

24

5

3

3

4

J

K

For example, in Figure 4.2, the path x ADIxT0 is an optimal path from x0 to xT . Thus by

the principle of optimality the least cost path from A to xT must be ADIxT , which is precisely

the remaining part of the overall optimal path when taking A as the initial state, as inspection of

Figure 4.2 confirms. Conversely, since the path ADIxT is the least cost path from state A to state


104

xT , any longer optimal path, say from x0 to xT , that passes through A must use the path ADIxT

to reach state xT . Again, inspection of Figure 4.2 confirms this too. The same results would

also hold if we took, say, state D as the initial state, as you should confirm. Likewise, similar re-

sults hold if we use the other optimal arc x BELxT0 , as you should also confirm.

Finally, note that the myopic optimal path is given by x BFMxT0 . This is the path chosen

by an individual who looks only at the cost of moving from one state to the next, and chooses the

path that minimizes the period by period cost of moving from x0 to xT . Observe that the total

cost along the path x BFMxT0 is greater than that along paths x ADIxT0 and x BELxT0 . This illus-

trates the general result that a myopic one-stage-at-a-time optimization procedure will not, in

general, yield the least cost path in a dynamic environment.

The principle of optimality can be proven in a straightforward way by a contrapositive

argument. It is important to prove this principle and that you follow it carefully, since, as noted

above, the principle of optimality forms the basis of the ensuing proof of the necessary condi-

tions of optimal control theory.

Theorem 4.1 (The Principle of Optimality): An optimal policy has the property that whatever

the initial state and initial decision are, the remaining decisions must constitute an optimal pol-

icy with regard to the state resulting from the first decision.

Proof: Let v( )s be the optimal control path " Œs t t[ , ]0 1 , beginning at time t0 in state x0 and

ending at time t1 in state x1, and let z( )s be the corresponding state path. A contrapositive proof

will be employed. Assume, therefore, that v( )s is the optimal control path " Œs t t[ , ]0 , t t tŒ( , )0 1 ,

beginning at time t0 in state x0 and ending at time t with state z( )t . Also assume, contrary to the

conclusion of the theorem, that u v( ) ( )s sπ is the optimal control path " Œs t t( , ]1 , beginning with

state z( )t at time t and ending with state x1 at time t1. Then v( )s could not be the optimal con-

trol path " Œs t t[ , ]0 1 since one could use v( )s " Œs t t[ , ]0 and switch to u( )s " Œs t t( , ]1 , thereby

giving the objective functional a larger value than using v( )s " Œs t t[ , ]0 1 . This contradicts the


105

assumed optimality (the hypothesis) of v( )s " Œs t t[ , ]0 1 thereby proving the principle of opti-

mality. Q.E.D.

We are now ready to state and prove the necessary conditions for problem (1).

Theorem 4.2 (The Maximum Principle): Let z v( ), ( )t t( ) be an admissible pair for problem (1).

Then if z v( ), ( )t t( ) yields the absolute maximum of J[ ]◊ , it is necessary that there exist a piece-

wise smooth vector-valued function ll( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def l l l1 2 N such that for all t t tŒ[ , ]0 1 ,

v z uu

( ) argmax , ( ), , ( )t H t t tU

= ( )Œ

ll ,

i.e., if

M t t t H t t tU

, ( ), ( ) max , ( ), , ( )z z uu

ll ll( ) = ( )Œ

def,

then

M t t t H t t t t, ( ), ( ) , ( ), ( ), ( )z z vll ll( ) ∫ ( ),

or equivalently

H t t t t H t t t U, ( ), ( ), ( ) , ( ), , ( )z v z u ull ll( ) ≥ ( ) " Œ .

That is, for each t t tŒ[ , ]0 1 , H ( )◊ attains at v( )t its maximum with respect to u, " Œu U and

t t t, ( ), ( )z ll( ) fixed. Furthermore, except for the points of discontinuities of v( )t ,

˙ ( ) , ( ), ( ), ( ) , ( ), ( )z tH

t t t t g t t tnn

n= ( ) = ( )∂∂l

z v z vll , n N= º1 2, , , ,

˙ ( ) , ( ), ( ), ( )lnn

tH

xt t t t= - ( )∂

∂z v ll , n N= º1 2, , , ,

where the above notation means that the functions are first differentiated with respect to the

particular variable and then evaluated at t t t t, ( ), ( ), ( )z v ll( ) .

Proof: To begin the proof of the necessary conditions recall the definition of the optimal value

function V ( )◊ from problem (1):

V t f s s s ds s s s s s U t tt

t

( , ) max , ( ), ( ) ˙ ( ) , ( ), ( ) , ( ) , ( ) , ( )( )

0 0 0 0 1 1

0

1

x x u x g x u u x x x xu

= ( ) = ( ) Œ = =ÏÌÔ

ÓÔ

¸˝ÔÔ◊ Údef

s.t. . (3)


106

Note that we now use s as the dummy variable of integration since we intend to let t denote a

particular date in the planning horizon. Since the optimal pair z v( ), ( )◊ ◊( ) is assumed to exist, this

definition is equivalent to

V t f s s s dst

t

( , ) , ( ), ( )0 0

0

1

x z v∫ ( )Ú , (4)

Since ( , , , )t t0 0 1 1x x are the parameters of the problem (1), the optimal value function V ( )◊ depends

on all four parameters in general, as well as any other parameter that might appear in problem (1)

. The notation in Eqs. (3) and (4), namely V t( , )0 0x , is adopted to streamline the exposition since

the parameters ( , )t1 1x play no essential role in what follows.

By the principle of optimality it follows that Eq. (4) can be written as

V t f s s s ds V t tt

t

( , ) , ( ), ( ) , ( )0 0

0

x z v z∫ ( ) + ( )Ú , (5)

which holds for every t t tŒ[ , ]0 1 . From the definitions of V ( )◊ in Eqs. (3) and (4) it should be

clear that

V t t f s s s dst

t

, ( ) , ( ), ( )z z v( ) ∫ ( )Ú1

, (6)

in which case the correctness of Eq. (5) should now seem more evident given the additive prop-

erty of integrals with respect to the limits of integration. Equation (6) asserts that V t t, ( )z( ) is the

maximum total value of the asset z( )t at time t, and thus is the purchase price of the asset z( )t at

time t, since the maximum present value of the stream of net benefits from holding z( )t at time t

and using the optimal plan is V t t, ( )z( ) . Equation (5) thus asserts that the total value of the asset

x0 at time t0 can be thought of as two separate additive components: (i) the return from begin-

ning with stock x0 at time t0 and continuing the optimal plan until time t, and (ii) the return from

beginning with stock z( )t at time t and continuing on with the optimal plan, where z( )t is the

stock at time t which resulted from the optimal plan in (i). This assertion is, in its essence, the

principle of optimality.

Now differentiate Eq. (5) with respect to t using the chain rule and Leibniz’s rule to get


107

00

∫ ( ) + ( )È

ÎÍÍ

˘

˚˙˙Úd

dtf s s s ds V t t

t

t

, ( ), ( ) , ( )z v z ,

or

01

∫ ( ) + ( ) + ( )=

Âf t t t V t t V t t z tt x nn

N

n, ( ), ( ) , ( ) , ( ) ˙ ( )z v z z , (7)

which also holds for every t t tŒ[ , ]0 1 . Since z v( ), ( )◊ ◊( ) is the optimal pair in problem (1) it must

satisfy the state equation identically, that is, ˙( ) , ( ), ( )z g z vt t t t∫ ( ) for all t t tŒ[ , ]0 1 . Substituting

˙( ) , ( ), ( )z g z vt t t t∫ ( ) into Eq. (7) therefore yields

V t t V t t g t t t f t t t t t tt xn

n

N

n, ( ) , ( ) , ( ), ( ) , ( ), ( ) [ , ]z z z v z v( ) + ( ) ( ) + ( ) ∫ " Œ

=Â

10 10 . (8)

This is the fundamental partial differential equation obeyed by the optimal value function V ( )◊ ,

known as the Hamilton-Jacobi-Bellman equation. The Hamilton-Jacobi-Bellman equation is

commonly written in the alternative form - ( ) ∫ ( ) + ( ) ( )V t t f t t t V t t t t tt , ( ) , ( ), ( ) , ( ) , ( ), ( )z z v z g z vx .

We will return to this fundamental result in later chapters when we discuss continuous time dy-

namic programming and intertemporal duality theory. Equation (8) is an important result in this

proof. We will return to it shortly.

Now consider the control vector ee ŒU and define the perturbed control as

uu

( )[ , )

( ) [ , ],s

s t t t

s s t t t=

" Œ +

" Œ +ÏÌÓ

def eeee

0 0

0 1

D

D

where Dt > 0, uee ( )s is the time path of the optimal control corresponding to the initial point

t t t t0 0+ +( )D D, ( )xee , and xee ( )s is the time path of the state vector corresponding to the perturbed

control path u( )s . By assumption, v( )◊ is the optimal control, thereby implying that u( )◊ is a

suboptimal control. As a result, it follows that

V t J J f s s ds V t t t tt

t t

( , ) [ ( ), ( )] [ ( ), ( )] , ( ), , ( )0 0 0 0

0

0

x z v x u x x∫ ◊ ◊ ≥ ◊ ◊ = ( ) + + +( )+

Úee ee eeeeD

D D . (9)

The pair x uee ( ), ( )◊ ◊( ) must be admissible if we are to compare the value of problem (1) using the

controls u( )◊ and v( )◊ . This implies that x z xee ( ) ( )t t0 0 0= = and x z xee ( ) ( )t t1 1 1= = . Upon substi-

tuting x xee ( )t0 0= in Eq. (9) and rearranging it we get


108

V t t t t V t t f s s dst

t t

0 0 0 0

0

0

+ +( ) - ( ) £ - ( )+

ÚD DD

, ( ) , ( ) , ( ),x x xee ee ee ee . (10)

Now divide Eq. (10) by Dt > 0 and let Dt Æ 0 to get

lim, ( ) , ( )

lim

, ( ),

D D

D

D DD Dt t

t

t t

V t t t t V t t

t

f s s ds

tÆ Æ

+

+ +( ) - ( )£ -

( )Ú0

0 0 0 0

0

0

0

x xx

ee eeee ee

. (11)

The inequality in Eq. (10) is preserved as Dt Æ 0 because of the assumed differentiability of

V ( )◊ from assumption (A.3). Our goal now is to evaluate the limits in Eq. (11) so that it can be

put in the form of Eq. (8).

We begin with the left-hand-side of Eq. (11). To this end, define the function G( )◊ by the

formula G t V t t t t( ) , ( )D D D= + +( )def0 0xee , which implies that G V t t( ) , ( )0 0 0= ( )xee . By the chain

rule we also have that ¢ = + +( ) + + +( ) +=ÂG t V t t t t V t t t t x t tt x nn

N

n( ) , ( ) , ( ) ˙ ( )D D D D D D0 0 0 0 01

x xee ee ee .

Using these results and the definition of the derivative, the left-hand-side of Eq. (11) becomes

lim, ( ) , ( )

lim( ) ( )

( ) ( )

, ( ) , ( ) ˙ ( )

D D

D

D DD

DD

D

t t

t

t x nn

N

V t t t t V t t

t

G t G

t

G t G

V t t V t t x tn

Æ Æ

=

=

+ +( ) - ( )=

+ -

= ¢ = ¢

= ( ) + ( )

0

0 0 0 0

0

0

0 0 0 0 01

0 0

0

x x

x x

ee ee

ee ee ee

def

ÂÂ .

(12)

We will return to this result shortly.

To evaluate the right-hand-side of Eq. (11) define the function F( )◊ by the formula

F t f s s dst

t t

( ) , ( ),DD

= ( )+

Údef xee ee0

0

,

and then recognize that F f s s dst

t( ) , ( ),0 0

0

0= ( ) =Ú xee ee and that ¢ = + +( )F t f t t t t( ) , ( ),D D D0 0xee ee

by Leibniz’s rule. Using these results and the definition of the derivative, the right-hand-side of

Eq. (11) becomes


109

-( )

= - = -+ -

= - ¢ = - ¢ = - + +( )= -

Æ

+

Æ Æ

= =

Úlim

, ( ),

lim( )

lim( ) ( )

( ) ( ) , ( ),

, (

D

D

D D

D D

DD

DD

DD D D

t

t

t t

t t

t t

f s s ds

t

F t

t

F t F

t

F t F f t t t t

f t t

0 0 0

0 0 0 0

0

0

0

0 0

0

x

x

x

ee

ee

ee

ee

ee

00), .ee( )(13)

Substituting Eqs. (12) and (13) in Eq. (11) gives

V t V t x t f tt x nn

N

n( , ) ( , ) ˙ ( ) ( , , )0 0 0 0 0

10 0x x x+ £ -

=Â ee ee , (14)

where x xee ( )t0 0= was used since the pair x uee ( ), ( )◊ ◊( ) is admissible by assumption. Recall also

that admissibility requires that xee ( )◊ satisfy the state equation, hence ˙ ( ) , ( ),x g xee ee eet t t0 0 0= ( ) .

Substituting ˙ ( ) , ( ),x g xee ee eet t t0 0 0= ( ) into Eq. (14) and again using x xee ( )t0 0= gives

V t V t g t f tt xn

n

N

n( , ) ( , ) , , ( , , )0 0 0 0 0 0

10 0 0x x x x+ ( ) + £

=Â ee ee . (15)

Now replace the initial point ( , )t0 0x in Eq. (15) with an arbitrary point ( , ) [ , ]t t t Sx Œ ¥0 1 to get

the inequality

V t V t g t f tt xn

n

N

n( , ) ( , ) , , ( , , )x x x x+ ( ) + £

=Â ee ee

1

0 , (16)

which is valid " Œx S , " Œt t t[ , ]0 1 , and " Œee U . This is where assumption (A.1) is used.

By defining the function f( )◊ as

f( , , ) ( , ) ( , ) , , ( , , )t V t V t g t f tt xn

n

N

nx x x x xee ee ee= + ( ) +

=Âdef

1

, (17)

and then referring to Eq. (8) and Eq. (16) respectively, it should be clear that we have shown that

f t t t t t t, ( ), ( ) [ , ]z v( ) ∫ " Œ0 0 1 , (18)

as well as

f( , , )t x ee £ 0 " Œx S , " Œt t t[ , ]0 1 , and " Œee U . (19)

The Maximum Principle and the costate equation follow from Eqs. (18) and (19), as we now

proceed to demonstrate.

To see how the Maximum Principle follows from Eqs. (18) and (19), first set x z= ( )t and

hold it fixed. Then Eqs. (18) and (19) imply that


110

f ft t t t tU

, ( ), ( ) max , ( ),z v z( ) = ( )Œee

ee . (20)

Next define

l j xt V t tj

( ) , ( )= ( )def z , j N= º1 2, , , , (21)

l0 1=def, (22)

H t f t t f t g tnn

n

N

( , , , ) ( , , ) ( , , ) ( , , ) ( , , )x u x u g x u x u x ull ll= + ¢ = +=

Âdef l l l0 01

. (23)

Upon using Eq. (17), that is, the definition of the function f( )◊ , Eq. (20) takes the form

V t t V t t g t t t f t t t

V t t V t t g t t f t t

t xn

n

N

Ut x

n

n

N

n

n

, ( ) , ( ) , ( ), ( ) , ( ), ( )

max , ( ) , ( ) , ( ), , ( ), .

z z z v z v

z z z z

( ) + ( ) ( ) + ( )

= ( ) + ( ) ( ) + ( )ÏÌÓ

¸˝˛

=

Œ =

Â

Â1

1ee

ee ee

Since V t tt , ( )z( ) is independent of eeee it can be canceled from both sides of the above equality. On

using the definitions (21), (22), and (23), the above equation becomes

H t t t t H t t tU

, ( ), ( ), ( ) max , ( ), , ( )z v zll ee llee

( ) = ( )Œ

,

which is the Maximum Principle.

To derive the adjoint or costate equation, set ee = v( )t and hold it fixed in Eqs. (18) and

(19). Then Eqs. (18) and (19) imply that f t t, , ( )x v( ) attains its maximum value at x z= ( )t . Be-

cause x ŒS and S is an open set by assumption (A.2), the fact that f t t, , ( )x v( ) attains its maxi-

mum value at x z= ( )t implies the first-order necessary condition

∂∂

fx

t tj t

, , ( )( )

x vx z

( ) ==

0, j N= º1 2, , , . (24)

Using the definition of f( )◊ in Eq. (17) and carrying out the differentiation in Eq. (24) thus yields

V t t V t t g t t t V t t g t t t f t t ttx xn

N

xn

x xn

Nn

xj n j n j j, ( ) , ( ) , ( ), ( ) , ( ) , ( ), ( ) , ( ), ( )z z z v z z v z v( ) + ( ) ( ) + ( ) ( ) + ( ) =

= =Â Â

1 1

0 ,

(25)

for j N= º1 2, , , . Note that assumption (A.3), namely V C t t t S( ) ( , ) ( , )( )◊ Œ " Œ ¥20 1x has been

used in deriving Eq. (25). Now differentiate Eq. (21) with respect to t and use the fact that

˙( ) , ( ), ( )z g z vt t t t∫ ( ) to get


111

˙ ( ) , ( ) , ( ) , ( ), ( )l j x t x xn

n

N

t V t t V t t g t t tj j n

= ( ) + ( ) ( )=

Âz z z v1

, j N= º1 2, , , . (26)

Using Young’s Theorem on Eq. (26) and then substituting Eq. (26) into Eq. (25) and rearranging

the resulting expression yields

˙ ( ) , ( ), ( ) , ( ) , ( ), ( )l j x x xn

n

N

t f t t t V t t g t t tj n j

= - ( ) - ( ) ( )=

Âz v z z v1

, j N= º1 2, , , . (27)

Finally, using the definitions (21), (22), and (23) in Eq. (27) gives

˙ ( ) , ( ), ( ), ( )l j xt H t t t tj

= - ( )z v ll , j N= º1 2, , , ,

which is the costate or adjoint equation. This completes the proof of the necessary conditions.

The reader should note that given our three assumptions this proof is rigorous. Q.E.D.

Notice that in the definition of the Hamiltonian in Eq. (23) a constant costate variable l0

multiplies the integrand f t( , , )x u . This is the correct way to write the Hamiltonian in general. In

this case the necessary conditions would then include the additional requirement that

l0 1 0 1, ( ) [ , ]ll t t t tN( )¢ π " Œ+0 ,

where 0N +1 is the null (column) vector in ¬ +N 1, and furthermore that

l0 0= or l0 1= .

In most economic applications of optimal control theory one can safely set l0 1= . In essence,

assuming that l0 1= means that the integrand matters in determining the optimal solution to the

control problem. Alternatively, if l0 0= , then the integrand does not matter for determining the

optimal solution to the control problem in that the necessary conditions of Theorem 4.2 do not

change if we replace the integrand function f ( )◊ with any other function. The case l0 0= is thus

appropriately referred to as abnormal. We will typically assume that l0 1= for the remainder of

the book, even though, strictly speaking, it is possible for l0 0= to hold, as the ensuing example

illustrates. The mental exercises present some other control problems for which l0 0= in an op-

timal solution.


112

Example 4.1: Consider the fixed endpoints optimal control problem

max [ ( ), ( )] ( )( )u

T

J x u u t dt◊

◊ ◊ = Údef

0

s.t. ˙( ) [ ( )]x t u t= 2

x x T( ) , ( )0 0 0= = .

Notice that the only admissible pair for this problem is x t u t( ), ( ) ( , )( ) = 0 0 for all t TŒ[ , ]0 , as in-

spection of the state equation and boundary conditions will verify. Defining the Hamiltonian by

setting l0 1= yields H t x u u u( , , , )l l= +def 2 . Since there are no constraints on the control variable

the necessary condition max ( , , , )u H t x u l of Theorem 4.2 reduces to the familiar condition

H t x u uu( , , , )l l= + =1 2 0. Note, however, that the only admissible value of the control variable,

namely u t( ) = 0, is not a solution to this necessary condition. If we instead define the Hamilto-

nian with a constant costate variable l0 attached to the integrand function and assume it is not

equal to unity, then we have H t x u u u( , , , )l l l= +def0

2 . The necessary condition then reads

H t x u uu( , , , )l l l= + =0 2 0 . If we choose l0 0= , then u t( ) = 0 does indeed satisfy this neces-

sary condition.

Let’s now provide an economic interpretation to the necessary conditions of Theorem 4.2

and the functions used in their derivation. First consider a typical costate variable l j t( ), and re-

call that it is defined in Eq. (21) as l j xt V t tj

( ) , ( )= ( )def z , j N= º1 2, , , . Also recall that V t t, ( )z( ) is

the total value of the stock z( )t in an optimal plan that begins at time t. This interpretation there-

fore implies that l j xt V t tj

( ) , ( )= ( )def z , j N= º1 2, , , , is the marginal value or shadow value of the

jth stock z tj ( ) at time t in an optimal plan. That is, l j t( ) is the dollar amount by which the value

of the optimal program V t t, ( )z( ) would increase due to a marginal increase in the jth stock at

time t, the initial date of the optimal plan. Note that we will further refine and expand on the

economic interpretation of ll( )t when we prove the dynamic envelope theorem. For now this

interpretation and intuition will suffice.


113

Next we turn to an economic interpretation of the Hamiltonian defined in Eq. (2), scilicet

H t f t t( , , , ) ( , , ) ( , , )x u x u g x ull ll= + ¢def. The integrand f t( , , )x u can be thought of as the flow of

profits over a short time interval to the value of the objective functional J[ ( ), ( )]x u◊ ◊ . The state

equation ˙ ( , , )x g x u= t represents the flow of investment in the capital stock in physical terms

over this time interval, since x is the capital stock vector and x is the time rate of change of, or

investment in, the vector of stocks. Taking the inner product of g x u( , , )t with llll converts the

physical flow g x u( , , )t into dollar terms, since llll is the marginal value of the capital stock vector

at time t. The sum ¢ll g x u( , , )t therefore represents the value of the capital accumulated in a short

interval of time. Thus the Hamiltonian H t f t t( , , , ) ( , , ) ( , , )x u x u g x ull ll= + ¢def measures the total

contribution of the activities that take place in a short interval of time, which includes the imme-

diate effect f t( , , )x u of profit flow, and the future effect ll ◊g x u( , , )t of the value of the capital

accumulated. The Maximum Principle, namely H t t t t H t t t, ( ), ( ), ( ) , ( ), , ( )z v z ull ll( ) ≥ ( ) " Œu U ,

thus directs us to maximize the sum of the current and future value flow at each point in time

with our choice of the control variable u. That is, it is simply an assertion that for every

t t tŒ[ , ]0 1 , the value v( )t of the optimal control vector must maximize the value of the Hamilto-

nian over all admissible values of the control vector u( )t UŒ .

Finally, we come to the adjoint or costate equation - = +=Â˙ ( , , ) ( , , )l lj x n x

n

n

Nf t g t

j jx u x u

1,

j N= º1 2, , , . Because l j is the shadow value of the jth stock, l j can be thought of as the rate at

which the shadow value of the jth stock is changing or appreciating through time. Thus -l j is

the rate at which the marginal value of the jth stock is depreciating through time. The costate

equation therefore asserts that along the optimal path, the decrease in the marginal value of the

jth capital stock at any point in time is the sum of jth capital stock’s marginal contribution to

profit, videlicet f tx j( , , )x u , and its marginal contribution to the enhancement of the value of all

the capital stocks, to wit ln xn

n

Ng t

j( , , )x u

=Â 1.

With the economic interpretation of Theorem 4.2 now complete, let’s consider a corollary

to Theorem 4.2 that applies to a large class of optimal control problems that is often encountered

in economic theory. Consider, therefore, a situation in which v( ) int [ , ]t U t t tŒ " Œ 0 1 , that is, the


114

optimal solution for the control variables (assuming one exists) is in the interior of the control

region for the entire planning horizon. Equivalently, assume that the constraints on the control

variables are not binding on the optimal control for the entire planning horizon. Alternatively,

one could more strongly assume that U M= ¬ , thereby implying that there are no constraints on

the control variables. Under either of these two assumptions, the Maximum Principle of Theo-

rem 4.2, namely H t t t t H t t t, ( ), ( ), ( ) , ( ), , ( )z v z ull ll( ) ≥ ( ) " Œu U , implies the first-order necessary

condition

H t t t t t t tMu z v 0, ( ), ( ), ( ) [ , ]ll( )¢ = " Œ 0 1 ,

as well as the second-order necessary condition

¢ ( ) £ " Œ " Œ ¬h z v h huuH t t t t t t t M, ( ), ( ), ( ) [ , ]ll 0 0 1 and ,

assuming, of course, that H C( ) ( )◊ Œ 2 in u. The proof of these necessary conditions follows im-

mediately from the first-order necessary and second-order necessary conditions from static opti-

mization theory since (i) we are choosing u ŒU at each t t tŒ[ , ]0 1 to maximize H ( )◊ , and (ii) the

constraints are either not binding on the optimal control or simply do not exist under the above

two scenarios. We have therefore proven the following economically important corollary to

Theorem 4.2.

Corollary 4.2 (Simplified Maximum Principle): Let z v( ), ( )t t( ) be an admissible pair for

problem (1). Then if v( ) int [ , ]t U t t tŒ " Œ 0 1 , z v( ), ( )t t( ) yields the absolute maximum of J[ ]◊ , and

H C( ) ( )◊ Œ 2 in u, it is necessary that there exist a piecewise smooth vector-valued function

ll( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def l l l1 2 N such that for all t t tŒ[ , ]0 1 ,

H t t t tum, ( ), ( ), ( )z v ll( ) = 0 , m M= º1 2, , , ,

and

H t t t t h hu u m

m

MMM

ml ll

, ( ), ( ), ( )z v hll( ) £ " Œ ¬==

ÂÂ11

0 .

Furthermore, except for the points of discontinuities of v( )t ,


115

˙ ( ) , ( ), ( ), ( ) , ( ), ( )z tH

t t t t g t t tnn

n= ( ) = ( )∂∂l

z v z vll , n N= º1 2, , , ,

˙ ( ) , ( ), ( ), ( )lnn

tH

xt t t t= - ( )∂

∂z v ll , n N= º1 2, , , .

We now pause to illustrate how to use Corollary 4.2 with a simple mathematical example.

Example 4.2: Consider the optimal control problem

max ( ) [ ( )]( )u

x t u t dt◊

- -[ ]Ú 12

2

0

1

a

s.t. ˙( ) ( )x t u t= , x x x x( ) , ( )0 10 1= = ,

where a > 0 is a parameter. Define the Hamiltonian as H x u x u u( , , )l a l= - - +def 12

2 . Since

there are no constraints on the control variable the necessary conditions are given by Corollary

4.2, hence

H x u u H x uu uu( , , ) , ( , , )l a l l a= - + = = - £0 0.

Since a > 0 it follows that H x uuu( , , )l a= - < 0. Solving H x u uu( , , )l a l= - + = 0 for the con-

trol variable gives u = -a l1 . The other necessary conditions are thus

˙ ( , , ) , ( ) , ( )x H x u u x x x x= = = =l l 0 10 1,

˙ ( , ,, )l l= - =H x ux 1.

Substituting u = -a l1 into these differential equations yields

˙ , ( ) , ( )x x x x x= = =-a l10 10 1 ,

l = 1.

Separating the variables in l = 1 and integrating gives the general solution l( )t t c= + 1, where c1

is a constant of integration. Substituting l( )t t c= + 1 into x = -a l1 gives ˙ [ ]x t c= +-a 11 , and

separating the variables and integrating yields the general solution x t t c t c( ) = + +- -12

1 2 11 2a a ,

where c2 is another constant of integration. The constants of integration c1 and c2 are deter-

mined by using the initial and terminal conditions x x( )0 0= and x x( )1 1= :

x x c x( )0 0 2 0= fi = ,

x x c x x( ) [ ]1 1 1 1 012= fi = - -a .


116

These constants of integration are then substituted into the general solution for the state and co-

state variables found above to get the specific solutions

z t x x t x x t x( ; , , ) [ ]a a a0 112

1 21 0

12

10= + - - +- - ,

l a a( ; , , ) [ ]t x x t x x0 1 1 012= + - - .

Finally, substitute l a a( ; , , ) [ ]t x x t x x0 1 1 012= + - - into u = -a l1 to find the path of the control

variable that satisfies the necessary conditions:

v t x x t x x( ; , , ) [ ]a a0 11 1

2 1 0= - + -- .

Notice that the solution to the necessary conditions depends explicitly on the parameters

( , , )a x x0 1 of the control problem. The above solution to the necessary conditions should be

compared to that of Example 2.1, since the control problems are identical in the two examples,

save for a fixed terminal state in the present example.

The ensuing example has a closed and bounded (i.e., compact) control region. As a re-

sult, we must use Theorem 4.2 in order to derive the correct necessary conditions. Moreover, the

optimal control problem is linear in the control variable and thus must be handled differently if

one is to solve the necessary conditions.

Example 4.3: Consider the following optimal control problem:

max [ ( ), ( )] ( )( ),u x

J x u x t dt◊

◊ ◊ = Ú1

0

1def

s.t. ˙( ) ( ) ( ), ( ) , ( )x t x t u t x x x= + = =0 0 1 1,

u t U u u t( ) ( ) : ( )Œ = ◊ - £ £{ }def1 1 .

Here we have an explicit representation of the control set U, a closed interval on the real line.

The fact that U is a closed set (i.e., it includes its boundary as part of the set) is important, for

this means the optimal value of the control may be at the boundary, and thus the derivative form

of the Maximum Principle, namely H t x uu( , , , )l = 0 , is not in general valid. Hence, by Theorem

4.2, the necessary condition is to choose u to maximize the Hamiltonian subject to u UŒ .


117

The Hamiltonian is defined as H x u x x u x u( , , ) [ ] [ ]l l l l= + + = + +def1 . The optimal

path of the costate variable must necessarily satisfy ˙ ( , , )l l l= - = - -H x ux 1 and l( )1 0= , the

latter of which results from the fact that x1 is a decision variable. The integrating factor for this

linear ordinary differential equation with constant coefficients is m( ) expt ds et t= Ú[ ] =def

, so upon

multiplying both sides of l l+ = -1 by et it follows that the resulting differential equation is

equivalent to dd t

t te e[ ]l = - . Upon integrating we find that l( )t ce t= -- 1, where c is a constant

of integration. The free boundary condition l( )1 0= implies that c e= . Thus the specific solu-

tion to the costate equation is l( )t e t= --1 1, so the costate path is fully determined. Finally, be-

cause ˙( ) [ , ]l t e tt= - < " Œ-1 0 0 1 and l( )1 0= , we conclude that l( ) [ , )t t> " Œ0 0 1 . This fact is

important in what follows.

Now let’s examine the Hamiltonian H x u x u( , , ) [ ]l l l= + +def1 . Since the term l u in the

Hamiltonian is the only one that depends on u, maximizing the Hamiltonian with respect to u

subject to u Œ -[ , ]11 is equivalent to maximizing l u with respect to u subject to u Œ -[ , ]11 . Be-

cause l( ) [ , )t t> " Œ0 0 1 the maximum of l u (and thus H x u( , , )l ) with respect to u subject to

u Œ -[ , ]11 is attained at u v t= =( ) 1. From the transversality condition l( )1 0= it follows that the

Hamiltonian is independent of the control variable at t = 1, thereby implying that maximizing

H ( )◊ with respect to u subject to u Œ -[ , ]11 does not determine the value of the control at t = 1.

As a result, the choice of the control at t = 1 is arbitrary. We therefore set v( )1 1= so that the

control function v( )◊ is continuous on [ , ]-11 . In sum, the control that maximizes the Hamiltonian

is v t t( ) [ , ]= " Œ1 0 1 .

Given this control, the state variable must satisfy the differential equation x x- = 1 and

the initial condition x( )0 0= . The integrating factor of this linear first-order differential equation

is g ( ) expt ds et t= - Ú[ ] = -def

, so that upon multiplying both sides of x x- = 1 by e t- it follows that

the resulting differential equation is equivalent to dd t

t txe e[ ]- -= . Upon integrating we have the

general solution x t ket( ) = - 1, where k is a constant of integration. The initial condition

x( )0 0= implies that k = 1, hence z t et( ) = - 1 is the specific solution of the state equation. Note

also that the value of the objective functional is


118

J z v e dt e t et t

t

t[ ( ), ( )] [ ] [ ]◊ ◊ = - = - = - >Ú =

=1 2 0

0

1

0

1.

Because we don’t have as of yet any sufficiency theorems for this class of control problems we

will have to wait to conclude that our solution of the necessary conditions actually is the solution

of the optimal control problem.

It is natural at this juncture to complement the necessary conditions of Theorem 4.2 with

a set of sufficient conditions that also apply directly to optimal control problem (1). The proof of

the following sufficient conditions differs from our earlier proof of sufficient conditions in

Chapter 3, however, because now we have to be careful to properly account for the presence of

the control region U in our analysis, just as we did in the proof of Theorem 4.2. We begin, there-

fore, with a preliminary technical result that is of value in the proof of the sufficiency theorem.

Lemma 4.1: Let F A( ):◊ Æ ¬, where A NÕ ¬ , and let F C( ) ( )◊ Œ 1 on A. Also let S AŒint be a

convex set and x ŒS .

(i) F F S F S( ) ( ) ( )[ ]x x x x x x xx≥ " Œ fi - £ " Œ0 .

(ii) If F( )◊ is concave " Œx S, then F S F F Sx x x x x x x x( )[ ] ( ) ( )- £ " Œ fi ≥ " Œ0 .

Proof: To prove part (i), first define G F F( ) ( ) ( )a a a= - + -( )def x x x1 for a Œ[ , ]0 1 . Since S is a

convex set, a ax x+ - Œ( )1 S " Œx S and a Œ[ , ]0 1 . Thus F F S( ) ( )x x x≥ " Œ is equivalent to

the statement G F F( ) ( ) ( )a a a= - + -( ) ≥def x x x1 0 for a Œ[ , ]0 1 . Because G( )a ≥ 0 for all

a Œ[ , ]0 1 and G F F( ) ( ) ( )0 0= - =x x , it follows that ¢ ≥G ( )0 0 must hold, hence

¢ = - - ≥G F( ) ( )[ ]0 0x x x x .

Multiplying through by minus unity completes the proof of part (i).

To prove part (ii), first note that by Theorem 21.3 of Simon and Blume (1994) we have

the familiar inequality fundamental to concave functions, namely

F F F S( ) ( ) ( )[ ]x x x x x xx£ + - " Œ .


119

Since F Sx x x x x( )[ ]- £ " Œ0 by hypothesis, the above inequality immediately gives the desired

conclusion, to wit F F S( ) ( )x x x≥ " Œ . Q.E.D.

With Lemma 4.1 in hand, we may proceed to prove the ensuing sufficiency theorem for

the class of control problems we have been considering in this chapter.

Theorem 4.3 (Mangasarian Sufficient Conditions): Let z v( ), ( )t t( ) be an admissible pair for

problem (1). Assume that U is a convex set and that ∂ ∂f ( )◊ u and ∂ ∂g u( )◊ exist and are con-

tinuous, in addition to our basic assumptions on f ( )◊ and g( )◊ . Suppose that z v( ), ( )t t( ) satisfy

the necessary conditions of Theorem 4.2 for problem (1) with costate vector ll( )t , and let

H t f t t( , , , ) ( , , ) ( , , )x u x u g x ull ll= + ¢def be the value of the Hamiltonian function. If H ( )◊ is a con-

cave function of ( , ) [ , ]x u " Œt t t0 1 over an open convex set containing all the admissible values of

x u( ), ( )◊ ◊( ) when the costate vector is ll( )t , then v( )t is an optimal control and z v( ), ( )t t( ) yields

the global maximum of J[ ]◊ . If H ( )◊ is a strictly concave function under the same conditions,

then z v( ), ( )t t( ) yields the unique global maximum of J[ ]◊ .

Proof: Let x u( ), ( )t t( ) be any admissible pair. By hypothesis, H ( )◊ is a C( )1 concave function of

( , ) [ , ]x u " Œt t t0 1 . It therefore follows from Theorem 21.3 of Simon and Blume (1994) that

H t t t t H t t t t H t t t t t t

H t t t t t t

, ( ), ( ), ( ) , ( ), ( ), ( ) , ( ), ( ), ( ) [ ( ) ( )]

, ( ), ( ), ( ) [ ( ) ( )],

x u z v z v x z

z v u vx

u

ll ll llll

( ) £ ( ) + ( ) -+ ( ) -

(28)

for every t t tŒ[ , ]0 1 . Integrating both sides of the above inequality over the interval [ , ]t t0 1 and

using the definitions of H ( )◊ and J[ ]◊ yields

J J t t t t t t t dt

H t t t t t t dt

H t t t t

t

t

t

t

[ ( ), ( )] [ ( ), ( )] ( ) , ( ), ( ) , ( ), ( )

, ( ), ( ), ( ) [ ( ) ( )]

, ( ), ( ), ( )

x u z v g z v g x u

z v x z

z v

x

u

◊ ◊ £ ◊ ◊ + ¢ ( ) - ( )[ ]

+ ( ) -

+ ( )

Ú

Ú

ll

ll

ll

0

1

0

1

[[ ( ) ( )] .u vt t dtt

t

-Ú0

1

(29)


120

By admissibility ˙( ) , ( ), ( )z g z vt t t t∫ ( ) and ˙ ( ) , ( ), ( )x g x ut t t t∫ ( ) for every t t tŒ[ , ]0 1 , while Theorem

4.2 implies that ˙ ( ) , ( ), ( ), ( )ll llt H t t t t¢ ∫ - ( )x z v for every t t tŒ[ , ]0 1 . Substituting these three results

in Eq. (29) gives

J J t t t t t t dt

H t t t t t t dt

t

t

t

t

[ ( ), ( )] [ ( ), ( )] ( ) ˙( ) ˙ ( ) ˙ ( ) ( ) ( )

, ( ), ( ), ( ) [ ( ) ( )] .

x u z v z x z x

z v u vu

◊ ◊ £ ◊ ◊ + ¢ -[ ] + ¢ -[ ][ ]

+ ( ) -

Ú

Ú

ll ll

ll

0

1

0

1(30)

Because H ( )◊ is concave in ( , ) [ , ]x u " Œt t t0 1 it is also concave in u alone for each t t tŒ[ , ]0 1 .

Thus, by Lemma 4.1, we have that

H t t t t t t U H t t t t H t t t Uu z v u v u z v z u u, ( ), ( ), ( ) [ ( ) ( )] , ( ), ( ), ( ) , ( ), , ( )ll ll ll( ) - £ " Œ ¤ ( ) ≥ ( ) " Œ0 .

Using this fact in Eq. (30) permits the conclusion

J J t t t t t t dtt

t

[ ( ), ( )] [ ( ), ( )] ( ) ˙( ) ˙ ( ) ˙ ( ) ( ) ( )x u z v z x z x◊ ◊ £ ◊ ◊ + ¢ -[ ] + ¢ -[ ][ ]Ú ll ll0

1

. (31)

To wrap up the proof simply note that

d

dtt t t t t t t t t t t t t t tll ll ll ll ll( ) ( ) ( ) ( ) ˙( ) ˙ ( ) ( ) ( ) ˙ ( ) ( ) ˙( ) ˙ ( ) ˙ ( ) ( ) ( )¢ -[ ][ ] = ¢ -[ ] + -[ ]¢ = ¢ -[ ] + ¢ -[ ]z x z x z x z x z x ,

and substitute this result into Eq. (31) to get

J Jd

dtt t t dt

J t t t

J

t

t

t t

t t

[ ( ), ( )] [ ( ), ( )] ( ) ( ) ( )

[ ( ), ( )] ( ) ( ) ( )

[ ( ), ( )],

x u z v z x

z v z x

z v

◊ ◊ £ ◊ ◊ + ¢ -[ ][ ]

= ◊ ◊ + ¢ -[ ]= ◊ ◊

Ú

=

=

ll

ll0

1

0

1

since by admissibility we have x x( )t0 0= , z x( )t0 0= , x x( )t1 1= , and z x( )t1 1= . We have thus

shown that J J[ ( ), ( )] [ ( ), ( )]x u z v◊ ◊ £ ◊ ◊ for all admissible functions x u( ), ( )◊ ◊( ) , just as we wished to.

If H ( )◊ is strictly concave in ( , ) [ , ]x u " Œt t t0 1 , then the inequality in Eq. (28) becomes strict if

either x z( ) ( )t tπ or u v( ) ( )t tπ for some t t tŒ[ , ]0 1 . Then J J[ ( ), ( )] [ ( ), ( )]x u z v◊ ◊ < ◊ ◊ follows. This

shows that any admissible pair of functions x u( ), ( )◊ ◊( ) which are not identically equal to

z v( ), ( )◊ ◊( ) are suboptimal. Q.E.D.


121

Let’s pause briefly so we can see this theorem in action.

Example 4.4: We begin by examining Example 4.2. Recall that this control problem does not

have any constraints on the control variable, i.e., U = ¬ , a convex set. Also recall that

z t x x t x x t x( ; , , ) [ ]a a a0 112

1 21 0

12

10= + - - +- - ,

v t x x t x x( ; , , ) [ ]a a0 11 1

2 1 0= - + -- ,

l a a( ; , , ) [ ]t x x t x x0 1 1 012= + - - ,

is the only solution of the necessary conditions. We will now show that the Hamiltonian for this

control problem, scilicet H x u x u u( , , )l a l= - - +def 12

2 , is a concave function of ( , )x u for all

t Œ[ , ]0 1 . To check concavity we compute the Hessian matrix of H ( )◊ :

H H

H Huu ux

xu xx

È

ÎÍ

˘

˚˙ =

-È

ÎÍ

˘

˚˙

a 0

0 0.

Because the eigenvalues are - <a 0 and 0, the Hessian matrix is negative semidefinite and there-

fore H ( )◊ is concave in ( , )x u for all t Œ[ , ]0 1 . Thus the above triplet is the unique solution to the

posed control problem. Note that since H ( )◊ is a polynomial the continuity conditions of Theo-

rem 4.3 are satisfied.

Now turn to Example 4.3. In this example the control region U = -[ , ]11 is a closed and

bounded convex set. Since the Hamiltonian H x u x x u( , , ) [ ]l l= + +def is linear in ( , )x u , it is

concave in ( , )x u for all t Œ[ , ]0 1 . The continuity conditions of Theorem 4.3 are satisfied due to

the aforementioned linearity. Since v t( ) = 1, z t et( ) = - 1, and l( )t e t= --1 1 satisfy all the neces-

sary conditions of Theorem 4.2 and are unique except at t = 1 when any admissible value of the

control variable is optimal, Theorem 4.3 implies that this triplet is indeed the unique solution of

the control problem.

With the mechanics of Theorems 4.2 and 4.3 down, let’s now analyze the inventory ac-

cumulation problem presented in Example 1.4 using these theorems. If your memory of this op-

timal control model is a bit fuzzy, you should return to Example 1.4 right now and refresh it.


122

Example 4.5: Referring to Example 1.4, the optimal control problem that is to be solved to de-

termine the optimal inventory accumulation policy is given by

min [ ( ), ( )] [ ( )] ( )( )u

T

J x u c u t c x t dt◊

◊ ◊ = +[ ]Údef1

22

0

s.t. ˙( ) ( )x t u t= ,

x( )0 0= , x T xT( ) = ,

where x t( ) is the stock of inventory at time t, u t( ) is the production rate at time t, and xT > 0 is

the contracted (and thus fixed) amount of the good that must be delivered by the terminal date of

the planning horizon T > 0. Recall that c1 0> is a constant that shifts the average and marginal

cost of production and c2 0> is the constant unit cost of holding inventory. The fundamental

economic tradeoff in this control problem is between the cost of producing the good and the cost

of storing it. For example, if the firm’s manager produces a relatively large quantity of the good

early in the planning horizon, then those goods must be held in inventory for a relatively long pe-

riod of time, thereby driving up the inventory holding cost. On the other hand, if the manager

waited until later in the planning horizon to produce a large quantity of the good, then inventory

cost would be relatively low but production cost would be relatively high due to the “rush” at the

end of the planning period to produce the contracted amount xT > 0. The optimal production

plan balances these two costs. Note that the production rate is not constrained to be nonnegative

in the above statement of the control problem. We thus assume that it is not binding in the opti-

mal plan. Later in this example we will reexamine this assumption. Furthermore, after we intro-

duce two additional theorems on necessary and sufficient conditions, we will explicitly account

for the nonnegativity constraint on the production rate and examine how it affects the optimal

solution.

The Hamiltonian for the control problem is defined as H x u c u c x u( , , )l l= + +def1

22 ,

where l is the shadow cost of the inventory, for we are dealing with a cost minimization prob-


123

lem. Because we have assumed that the nonnegativity constraint on the production rate is not

binding, the necessary conditions are given by Corollary 4.2, and therefore read:

H x u c uu( , , )l l= + =2 01 , (32)

H x u cuu( , , )l = ≥2 01 , (33)

˙ ( , , )l l= - = -H x u cx 2, (34)

˙ ( , , )x H x u u= =l l , x( )0 0= , x T xT( ) = . (35)

Since c2 0> , it follows that Eq. (33) is satisfied with a strict inequality, that is to say,

H x u cuu( , , )l = >2 01 (remember that we are dealing with a minimization problem here).

Without even solving the necessary conditions we can glean some economic information

about the pattern of production. To see this, first solve Eq. (32) for the production rate to get

u c= -l ( )2 1 , and then differentiate this expression with respect to t and use Eq. (34) to arrive at

˙ ( )u c c= >2 12 0. This shows that the production rate rises over the planning horizon. Thus the

firm produces the fewest units of the good per unit of time at the beginning of the planning hori-

zon, and the most units of the good per unit of time at the end of the planning horizon. Figure

4.3 displays this qualitative conclusion graphically. Note that the slope of the curve in Figure 4.3

is given by the right-hand-side of ˙ ( )u c c= 2 12 , which is a positive constant.


124

u t( )

tT

Figure 4.3

u T( )

u( )0

1123

} -12 2 1

1c c

area under curve =

required production

Ú = - ==

0 0T

Tu s ds x T x x( ) ( ) ( )

Examination of Eq. (34) similarly reveals qualitative information. In particular, we see

that the shadow cost of inventory falls over the planning horizon. This is not at all surprising, for

an additional unit of stock received at the beginning of the planning period must be held in in-

ventory longer (and is thus more costly) than one received near the end of the planning horizon.

The declining shadow cost of the stock is essentially dual to the rising production rate, for one

can interpret the former as driving the latter, or the latter as driving the former. Also noteworthy

is the fact that since we assumed that the nonnegativity constraint on the production rate is not

binding, the shadow cost of inventory is negative. That this is true can be seen from Eq. (32), or

equivalently, from u c= -l ( )2 1 . This observation means that one more unit of stock in inven-

tory lowers the total cost of production, which is not at all surprising since one more unit of the

good in inventory implies that one less unit of the good must be produced in order to reach the

terminal stock of xT .

Let’s now solve the necessary conditions. Separating the variables and integrating the

costate equation (34) gives the general solution l( )t k c t= -1 2 , where k1 is a constant of integra-


125

tion. Substituting this expression in Eq. (32) yields u c t k c= -( ) ( )2 1 12 , which when substituted

in the state equation (35) produces ˙ ( ) ( )x c t k c= -2 1 12 . Separating the variables and integrating

the state equation gives the general solution for the inventory stock x t c c t c k t k( ) = - +- -14 2 1

1 2 12 1

11 2 ,

where k2 is another constant of integration. Now use the two boundary conditions to find the

two constants of integration:

x k( )0 0 02= fi =

x T x k c T c x TT T( ) = fi = - -1

12 2 1

12 .

Next substitute the values of k1 and k2 into the general solution for the inventory stock to find

the specific or definite solution that satisfies the boundary conditions:

z t c c T x c c t t T x T tT T( ; , , , ) [ ]1 214 2 1

1 1= - +- - . (36)

The corresponding production rate may be found by using u c t k c= -( ) ( )2 1 12 or differentiating

Eq. (36) with respect to t since x u= :

v t c c T x c c t T x TT T( ; , , , ) [ ]1 214 2 1

1 12= - +- - . (37)

Finally, the time path of the shadow cost of the inventory stock is found by substituting the value

of k1 in l( )t k c t= -1 2 to arrive at

l( ; , , , ) [ ]t c c T x c T t c x TT T1 212 2 1

12 2= - - - . (38)

It is important to emphasize that the solution to the necessary conditions depends explicitly on

the parameters aa =def( , , , )c c T xT1 2 of the inventory accumulation problem, just like the solution to

a static optimization problem does. Unlike the solution to a static optimization problem, how-

ever, the solution to the necessary conditions in general depends on the independent variable

time as well, thus indicating that even if the parameters are constant over the planning horizon

the inventory level and the production rate will in general vary over the planning period.

Before studying the comparative dynamics properties of the above solution let’s confirm

that it is in fact the solution to the posed optimal control problem. Since we are dealing with a

minimization problem and we have assumed that the nonnegativity constraint on the production

rate is not binding, all we have to do is verify that the Hamiltonian is a convex function of the

state and control variables. Inspection of the optimal control problem shows that the integrand


126

function is convex in ( , )x u and the transition function is linear in ( , )x u , hence by the analogue

of Theorem 3.2 for convex functions, we may conclude that the Hamiltonian is a convex func-

tion of ( , )x u . As a result, by the analogue of Theorem 4.3 for minimization problems, we know

that the above solution to the necessary conditions is a solution of the optimal control problem.

Actually, we can claim a bit more. Since the above solution of the necessary conditions is the

only one, it is therefore the unique solution of the control problem. Recall, however, that this

conclusion is true only if v t t T( ; ) [ , ]aa ≥ " Œ0 0 .

Turning to the comparative dynamics, observe that since we have an explicit solution to

the inventory accumulation problem, we may proceed to differentiate it with respect to the pa-

rameters of interest to conduct the comparative dynamics analysis. Let’s therefore proceed to

uncover and economically interpret some of the comparative dynamics properties of the unique

solution, leaving the rest of them for a mental exercises.

The production rate v t( ; )aa and the shadow cost of the inventory stock l( ; )t aa are the

more interesting economic variable, so we will focus our attention on them rather than on the in-

ventory stock. To begin the comparative dynamics analysis, partially differentiate the production

rate given in Eq. (37), and the shadow cost of the inventory given in Eq. (38), with respect to the

unit cost of holding inventory c2 to find that

∂v

ct c t T

t T

t T

t T T∂ 2

14 1

1

12

12

12

2

0 0

0

0

( ; ) [ ]

[ , )

( , ]

aa = -< " Œ= => " Œ

ÏÌÔ

ÓÔ

- for , (39)

∂l∂c

t T t

t T

t T

t T T2

12

12

12

12

2

0 0

0

0

( ; ) [ ]

[ , )

( , ]

aa = -> " Œ= =< " Œ

ÏÌÔ

ÓÔ

for . (40)

The first thing to notice about Eqs. (39) and (40) is that unlike a comparative statics result, a

comparative dynamics result can change sign with the passage of time. The economic interpre-

tation of Eq. (39) is straightforward and makes economic sense. Since inventory is now more

costly to hold, the production rate is decreased in the first half of the planning horizon so that

less inventory accumulates in this period. Because the firm must still have xT > 0 units in in-


127

ventory at the terminal time t T= , and T is fixed (i.e., the planning horizon is fixed), the produc-

tion rate must increase in the last half of the planning horizon in order to make up for the lower

production rate in the first half of the planning horizon. This way fewer units are held in inven-

tory for a longer period of time, thereby partially offsetting the higher cost of holding inventory.

Figure 4.4 depicts the effect of the increase in the inventory holding cost on the production rate.

Notice that the production rate path pivots counterclockwise about the point t T= 12 as a result of

the higher inventory holding costs. Observe, however, that because the firm is required to al-

ways produce the same number of units of the good over the same period of time, the area under

each time path, which is the required production, is the same in each case. In other words, the

area between the two curves in Figure 4.4 must be the same, implying that Area 1 equals Area 2.

The effect of the increase in inventory holding costs on the shadow cost of inventory,

given in Eq. (40), also has a straightforward economic interpretation. In this case we see that an

increase in the cost of holding inventory increases the shadow cost of inventory in the first half

of the planning horizon and decreases it in the second half, just the opposite of its effect on the

production rate. The economic explanation for this feature is essentially the same as that behind

the declining value of the shadow cost of inventory over the planning horizon, namely , an addi-

tional unit of stock received at the beginning of the planning period must be held in inventory

longer (and is thus more costly) than one received near the end of the planning horizon.


128

u t( )

tT

Figure 4.4

12 T

v t c c( ; )aa2 2= ¢

v t c c c( ; )aa2 2 2= ¢¢ > ¢

Area 1

Area 2

Now let’s reexamine our assumption that the nonnegativity constraint on the production

rate is not binding. We begin by checking to see if in fact the production rate v t( ; )aa satisfies the

constraint u t t T( ) [ , ]≥ " Œ0 0 . Because ˙( ; )v t c caa = >-12 2 1

1 0, the production rate v t( ; )aa is an in-

creasing function of time. Therefore, if the production rate at the initial date of the planning pe-

riod is nonnegative, then the production rate is nonnegative throughout the planning period since

it is a strictly increasing function of time. Conversely, if the production rate is nonnegative

throughout the planning horizon, then it is surely nonnegative at the initial date of the planning

period. Stated more formally, we’ve shown that v( ; )0 0aa ≥ if and only if v t t T( ; ) [ , ]aa ≥ " Œ0 0 .

Therefore, the nonnegativity constraint u t( ) ≥ 0 will be satisfied for all t TŒ[ , ]0 if and only if

v x T c c TT( ; )0 01 14 2 1

1aa = - ≥- - , or equivalently, if and only if

x c c TT ≥-1

4 2 11 2. (41)

Thus the pair of curves z t v t( ; ), ( ; )aa aa( ) is the solution of the inventory accumulation problem if

and only if Eq. (41) holds. The economic interpretation of the inequality in Eq. (41) is that the


129

total production xT is large relative to the delivery date T, and the unit storage cost c2 is suffi-

ciently small relative to the unit production cost coefficient c1, if and only if the production rate

is nonnegative throughout the planning period. If Eq. (41) does not hold then the start of pro-

duction is postponed in the optimal plan. We will investigate this situation in the subsequent ex-

ample, where we will explicitly take the nonnegativity constraint into account.

Recall that Theorem 4.2 was developed under the assumption that U is a fixed set, that is,

U is not a function of the state variables or time. We will treat the case where U is a function of

the state variables in a later chapter. It turns out, however, that Theorem 4.2 actually covers the

case where U is a function of t, say U t( ) , provided a constraint qualification (to be detailed be-

low) is satisfied. Let’s endeavor to see why this is so.

At one level it is easy to see why Theorem 4.2 applies when U t( ) is the control region:

nothing in the proof of Theorem 4.2 depended on the fact that the control region U was inde-

pendent of t. In actual applications of optimal control theory to economic problems, however,

the set U t( ) is almost always defined by a system of inequality constraints. That is,

U t h t t k Kk( ) ( ) : , ( ) , , , ,= ◊ ( ) ≥ = º{ }def u u 0 1 2 . (42)

For instance, in Example 4.3 we had that u t U u t u t( ) ( ) : ( )Œ = - £ £{ }def1 1 . This constraint can be

rewritten to conform to Eq. (42) as follows:

h t u t u t1 1 0, ( ) ( )( ) = + ≥def,

h t u t u t2 1 0, ( ) ( )( ) = - ≥def.

For our purposes, therefore, there is no loss in generality in defining the control region by a sys-

tem of inequality constraints. We will thus do so for the remainder of the textbook when con-

straints are considered. Because of this convention, we must therefore introduce a constraint

qualification on the constraint functions hk ( )◊ , k K= º1 2, , , , just as is done when studying static

optimization problems with constraints.


130

The optimal control problem we now seek to develop necessary and sufficient conditions

for can be stated as follows:

max [ ( ), ( )] , ( ), ( )( )u

x u x u◊

◊ ◊ = ( )ÚJ f t t t dtt

tdef

0

1

s.t. ˙ ( ) , ( ), ( )x g x ut t t t= ( ) , x x x x( ) , ( )t t0 0 1 1= = , (43)

h u 0t t K, ( )( ) ≥ ,

where u( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )defu u uM1 2 , g( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def

g g gN1 2 , ˙ ( ) ˙ ( ), ˙ ( ), , ˙ ( )x ◊ = ◊ ◊ º ◊( )defx x xN1 2 ,

h( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )defh h hK1 2 , and 0K is the K-element null vector in ¬K . It is important to ob-

serve that the vector of constraints h u 0t t K, ( )( ) ≥ in problem (43) is equivalent to the requirement

u u u( ) ( ) ( ) : , ( ) , , , ,t U t h t t k KkŒ = ◊ ( ) ≥ = º{ }def0 1 2 from Eq. (42). Before stating the theorems we

first introduce a definition that is a prerequisite for the constraint qualification we shall adopt.

Definition 4.2: Let X be a set. Then card X( ), the cardinal number of X , is the number of

elements in X .

For finite sets, the only type that we will employ when using Definition 4.2, this is an

elementary and straightforward concept. As usual, we let z v( ), ( )◊ ◊( ) be an optimal pair of func-

tions for problem (43). We may now state the constraint qualification of interest to us.

Rank Constraint Qualification: Define i t t k h t t k Kk, ( ) : , ( ) , , , ,v v( ) = ( ) = = º{ }def0 1 2 as the index

set of the binding constraints along the optimal path. For all t t tŒ[ , ]0 1 , if i t t, ( )v( ) π ∆ , i.e.,

i t t, ( )v( ) is non-empty, then the card i t t M, ( )v( )( ) ¥ Jacobian matrix

∂∂

h

ut t

k

m k t tm M

, ( ), ( )

, , ,

vv

( )È

ÎÍ

˘

˚˙

Œ ( )= ºi1 2

(44)

has a rank equal to card i t t, ( )v( )( ) . That is, the rank of the above Jacobian matrix is equal to the

number of its rows.


131

For example, if ¢k of the inequality constraints bind at a given point t in the planning ho-

rizon, 0 < ¢ <k K , then the above Jacobian matrix will be of order ¢ ¥k M and the constraint

qualification will be satisfied if the rank of the Jacobian equals ¢k . This rank condition on the

Jacobian has two important consequences. First, it implies that at least one control variable must

be present in each of the binding constraints. To see this, assume that the constraint qualification

holds but that one of the ¢k binding constraints, say the first (without loss of generality), does not

have any control variables in it. In this case the first row of the above Jacobian would be identi-

cally zero, thereby implying that the rows of the Jacobian are linearly dependent. This, in turn,

implies that the Jacobian is of rank less than ¢k , thus violating the constraint qualification. Sec-

ond, the constraint qualification implies that the number of binding constraints ¢k cannot be

greater than the number of control variables M. To see this, assume that the constraint qualifica-

tion holds but that ¢ >k M . Then there are more rows ( ¢k ) in the Jacobian than there are col-

umns (M). But the rank of a matrix cannot exceed the minimum of the number of its rows or

columns. This, in turn, implies that the rank of the Jacobian is min( , )¢ =k M M , which is less

than ¢k , thereby violating the constraint qualification.

With the constraint qualification now dealt with, we can turn to the necessary conditions

for problem (43). For ease of exposition we first remind the reader of the definition of the Ham-

iltonian: H t f t t f t g tnn

n

N( , , , ) ( , , ) ( , , ) ( , , ) ( , , )x u x u g x u x u x ull ll= + ¢ = +

=Âdef l1

. We should also note

that in addition to the standard assumptions on the functions f ( )◊ and g( )◊ given in Chapter 1, we

now make the following additional ones:

(A.1) ∂ ∂f u Cm( ) ( )◊ Œ 0 with respect to the 1 + +N M variables ( , , )t x u for m M= º1 2, , , .

(A.2) ∂ ∂g u Cnm( ) ( )◊ Œ 0 with respect to the 1 + +N M variables ( , , )t x u for m M= º1 2, , , and

n N= º1 2, , , .

(A.3) h Ck ( ) ( )◊ Œ 0 with respect to the 1 + M variables ( , )t u for k K= º1 2, , , .

(A.4) ∂ ∂h u Ckm( ) ( )◊ Œ 0 with respect to the 1 + M variables ( , )t u for k K= º1 2, , , and

m M= º1 2, , , .

We may now state the necessary conditions pertaining to problem (43).


132

Theorem 4.4 (Necessary Conditions, Inequality Constraints): Let z v( ), ( )t t( ) be an admissi-

ble pair for problem (43), and assume that the rank constraint qualification is satisfied. Then if

z v( ), ( )t t( ) yields the absolute maximum of J[ ]◊ , it is necessary that there exist a piecewise

smooth vector-valued function ll( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def l l l1 2 N and a piecewise continuous vector-

valued Lagrange multiplier function mm( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def m m m1 2 K , such that for all t t tŒ[ , ]0 1 ,

v z u h u 0u

( ) argmax , ( ), , ( ) , ( )t H t t t t t K= ( ) ( ) ≥{ }ll s.t. ,

i.e., if

M t t t H t t t t t K, ( ), ( ) max , ( ), , ( ) , ( )z z u h u 0u

ll ll( ) = ( ) ( ) ≥{ }defs.t. ,

then

M t t t H t t t t, ( ), ( ) , ( ), ( ), ( )z z vll ll( ) ∫ ( ),

or equivalently

H t t t t H t t t U t, ( ), ( ), ( ) , ( ), , ( ) ( )z v z u ull ll( ) ≥ ( ) " Œ .

where U t h t t k Kk( ) ( ) : , ( ) , , , ,= ◊ ( ) ≥ = º{ }def u u 0 1 2 . Because the rank constraint qualification is

assumed to hold, the above necessary condition implies that

∂∂

L

ut t t t t m M

m

, ( ), ( ), ( ), ( ) , , , ,z v ll mm( ) = = º0 1 2 ,

∂∂

∂∂

Lt t t t t t t

Lt t t t t k K

kk k

kmm m

m, ( ), ( ), ( ), ( ) , ( ) , ( ) , ( ), ( ), ( ), ( ) , , , ,z v z vll mm ll mm( ) ≥ ≥ ( ) = = º0 0 0 1 2 ,

where

L t H t t f t g t h tnn

n

N

kk

k

K

( , , , , ) ( , , , ) ( , ) ( , , ) ( , , ) ( , )x u x u h u x u x u ull mm ll mm= + ¢ = + += =

Â Âdef l m1 1

is the Lagrangian function. Furthermore, except for the points of discontinuities of v( )t ,

˙ ( ) , ( ), ( ), ( ) , ( ), ( )z tH

t t t t g t t tnn

n= ( ) = ( )∂∂l

z v z vll , n N= º1 2, , , ,

˙ ( ) , ( ), ( ), ( )lnn

tH

xt t t t= - ( )∂

∂z v ll , n N= º1 2, , , ,


133

where the above notation means that the functions are first differentiated with respect to the

particular variable and then evaluated at t t t t t, ( ), ( ), ( ), ( )z v ll mm( ) .

The proof of this theorem is a straightforward consequence of Theorem 4.2 and the Ka-

rush-Kuhn-Tucker Theorem, stated as Theorem 18.4 in Simon and Blume (1994). It is therefore

left for a mental exercise. Before stating a corresponding sufficiency theorem, we note several

important consequences of Theorem 4.4. First, in general, the Lagrange multiplier function

mm( ) ( ), ( ), , ( )◊ = ◊ ◊ º ◊( )def m m m1 2 K is piecewise continuous on [ , ]t t0 1 . It turns out, however, that mm( )◊ is

continuous whenever the optimal control function v( )◊ is continuous. Thus the discontinuities in

the Lagrange multipliers can only occur at points where the optimal control is discontinuous.

Second, the Lagrangian function evaluated along the optimal solution is a continuous function of

t, that is, L t t t t t, ( ), ( ), ( ), ( )z v ll mm( ) is a continuous function of t. Third, the total derivative of

L t t t t t, ( ), ( ), ( ), ( )z v ll mm( ) with respect to t, scilicet

˙ , ( ), ( ), ( ), ( ) , ( ), ( ), ( ), ( )L t t t t td

dtL t t t t tz v z vll mm ll mm( ) = ( )def

,

is equal to the partial derivative of L t t t t t, ( ), ( ), ( ), ( )z v ll mm( ) with respect to t, namely

L t t t t ttL t t t t tt , ( ), ( ), ( ), ( ) , ( ), ( ), ( ), ( )z v z vll mm ll mm( ) = ( )def ∂

∂,

at all continuity points of the optimal control function v( )◊ , assuming that f ( )◊ , g( )◊ , and h( )◊ are

C( )1 functions of ( , , )t x u . Other results similar to those derived in Mental Exercise 2.31 also

hold, as you will discover in a mental exercise in this chapter.

It is also worthwhile at this juncture to provide a rather thorough set of remarks on the

role that the rank constraint qualification plays in solving an optimal control problem. Following

Seierstad and Sydsæter (1987, pp. 278–279), the crucial observation is that the set of candidates

for optimality are contained in the union of the following two sets:

A = { }defadmissible pairs satisfying the necessary conditions ,

B = { }defadmissible pairs which fail to satisfy the rank constraint qualification .


134

In other words, the set A B» contains all the candidates for optimality. Because we have the

identity A B B A B» ∫ » -( ), a valid two-step approach for obtaining all of the candidates for

optimality is:

(i) First find all the admissible pairs which fail to satisfy the rank constraint qualifi-

cation for some t t tŒ[ , ]0 1 .

(ii) Then find all the admissible pairs which satisfy the necessary conditions and the

rank constraint qualification.

This is the approach applied in the book to find the candidates for optimality, and is also the pro-

cedure recommended because it is often the most successful. Note, however, that the identity

A B A B A» ∫ » -( ) is equally valid and suggests a different two-step procedure for obtaining

all of the candidates for optimality, to wit:

(i¢) First find all the admissible pairs which satisfy the necessary conditions, disre-

garding the rank constraint qualification.

(ii¢) Then find all the admissible pairs which fail to satisfy the rank constraint, ex-

cluding those obtained in (i¢).

Remember that either two step-procedure will yield all the candidates for optimality.

The following sufficiency theorem is not the most general known, but it will suffice for

our present purposes. Its proof is similar to that of Theorem 4.3 and so is left for a mental exer-

cise. It is a Mangasarian-type sufficiency result.

Theorem 4.5 (Sufficient Conditions, Inequality Constraints): Let z v( ), ( )t t( ) be an admissible

pair for problem (43). Suppose that z v( ), ( )t t( ) satisfy the necessary conditions of Theorem 4.4

for problem (43) with costate vector ll( )t and Lagrange multiplier vector mm( )t , and let

L t f t t t( , , , , ) ( , , ) ( , , ) ( , )x u x u g x u h ull mm ll mm= + ¢ + ¢def be the value of the Lagrangian function. If L( )◊

is a concave function of ( , ) [ , ]x u " Œt t t0 1 over an open convex set containing all the admissible

values of x u( ), ( )◊ ◊( ) when the costate vector is ll( )t and Lagrange multiplier vector is mm( )t , then

v( )t is an optimal control and z v( ), ( )t t( ) yields the global maximum of J[ ]◊ . If L( )◊ is a strictly


135

concave function under the same conditions, then z v( ), ( )t t( ) yields the unique global maximum

of J[ ]◊ .

A very important observation about this sufficiency theorem is that the rank constraint

qualification is not required, in sharp contrast to the necessary conditions of Theorem 4.4. It is

also noteworthy that this theorem can be strengthened to give an Arrow-type sufficiency result.

We will pursue this in a later chapter, however, when we study a more general class of optimal

control problems. Let’s now demonstrate Theorems 4.4 and 4.5 by returning to the inventory ac-

cumulation problem of Example 4.5, but this time we explicitly take into account the nonnega-

tivity restriction on the production rate.

Example 4.6: Recall the inventory accumulation problem from Example 4.5:

max [ ( ), ( )] [ ( )] ( )( )u

T

J x u c u t c x t dt◊

◊ ◊ = - +[ ]Údef1

22

0

s.t. ˙( ) ( )x t u t= , x( )0 0= , x T xT( ) = ,

h t u t u t, ( ) ( )( ) = ≥def0.

Notice that in this instance we have multiplied the objective functional by minus unity in order to

convert it into a maximization problem, the form required by Theorems 4.4 and 4.5. Recall that

in Example 4.5 we showed that the production rate constraint u t t T( ) [ , ]≥ " Œ0 0 is satisfied if

and only if x c c TT ≥-1

4 2 11 2 [recall Eq. (41)]. In the present situation, let’s therefore assume that

x c c TT < -14 2 1

1 2, so that the nonnegativity constraint on the production rate will be binding for

some period of time in the planning horizon. This is clearly the only case in which the nonnega-

tivity constraint is interesting.

To solve this version of the problem, we begin by defining the Hamiltonian and Lagran-

gian functions:

H x u c u c x u( , , )l l= - - +def1

22 ,

L x u c u c x u u( , , , )l m l m= - - + +def1

22 .


136

Since h t uu( , ) = 1 the rank constraint qualification is satisfied. Thus by Theorem 4.4 the neces-

sary conditions are

L x u c uu( , , , )l m l m= - + + =2 01 , (45)

L x u um l m( , , , ) = ≥ 0 , m ≥ 0, mu = 0, (46)

˙ ( , , )l l= - =H x u cx 2, (47)

˙ ( , , )x H x u u= =l l , x( )0 0= , x T xT( ) = . (48)

The Hessian matrix of the Lagrangian function is given by

L x u L x u

L x u L x u

cuu ux

xu xx

( , , , ) ( , , , )

( , , , ) ( , , , )

l m l ml m l m

È

ÎÍ

˘

˚˙ =

-È

ÎÍ

˘

˚˙

2 0

0 01 ,

which has eigenvalues - <2 01c and zero. Thus the Hessian matrix of the Lagrangian function is

negative semidefinite, which, as you know by now, is equivalent to the concavity of L( )◊ in

( , )x u . By Theorem 4.5, therefore, any solution of the necessary conditions is a solution of the

optimal control problem. Note, however, that since L( )◊ is not strictly concave in ( , )x u we can-

not appeal to Theorem 4.5 to claim uniqueness.

Since we have assumed that x c c TT < -14 2 1

1 2, we know that the length of the planning hori-

zon T is long relative to the amount of the product required to be produced xT . Moreover, we

also know that inventory is costly to hold, i.e., c2 0> . In light of these two observations, a rea-

sonably hypothesis as to the structure of the solution for the production rate is that there is an

initial period of the planning horizon, say 0 £ £t s, s TŒ( , )0 , in which the production rate is

zero, that is, u t( ) = 0 for all t sŒ[ , ]0 , after which the production rate is positive, i.e., u t( ) > 0 for

all t s TŒ( , ]. The time after which the production rate turns positive, namely s TŒ( , )0 , is called

the switching time . It is an endogenous variable, determined by the condition that the cumulative

amount produced by the end of the planning horizon equals the amount required by the contract.

We thus seek a solution of the necessary conditions with these characteristics. Such a solution

will be the solution of the control problem due to the aforementioned concavity of L( )◊ in ( , )x u .

To begin, first solve Eq. (45) for the control variable to get

uc

=+l m

2 1

. (49)


137

Our conjecture is that u t( ) = 0 for all t sŒ[ , ]0 , s TŒ( , )0 . Clearly, this is equivalent to

l m( ) ( )t t+ = 0 for all t sŒ[ , ]0 . Since l = >c2 0 by Eq. (47) (the costate equation), l( )t π 0 for

all t TŒ[ , ]0 , with the possible exception of an instant. This implies that the Lagrange multiplier

m( )t π 0 for all t sŒ[ , ]0 , with the possible exception of an instant, since l m( ) ( )t t+ = 0 for all

t sŒ[ , ]0 . Integrating l = >c2 0 yields l( )t c t a= +2 1 for all t TŒ[ , ]0 , where a1 is a constant of

integration. Because l m( ) ( )t t+ = 0 for all t sŒ[ , ]0 , m( )t c t a= - -2 1 for all t sŒ[ , ]0 . To find the

value of the constant of integration, recall our conjecture that u t( ) > 0 for all t s TŒ( , ], which

implies, by way of Eq. (46), that m( )t = 0 for all t s TŒ( , ]. Since the control function is a con-

tinuous (but not a differentiable) function of t under our hypothesis, the Lagrange multiplier

function is too, confirming a remark made after Theorem 4.4. We thus use the fact that m( )◊ is

continuous at t s= to get m( )s c s a= - - =2 1 0, or a c s1 2= - . Using this value of the constant of

integration yields the specific solution for the costate variable and Lagrange multiplier:

l( ; , ) [ ] [ , ]t s c c t s t T2 2 0= - " Œ , (50)

m( ; , )[ ] [ , ],

( , ].t s c

c s t t s

t s T22 0

0=

- " Œ" Œ

ÏÌÓ

(51)

Note that on the closed interval [ , ]0 s , l( ; , )t s c2 and m( ; , )t s c2 are only equal to zero at t s= , an

instant, just as claimed above. Substituting Eqs. (50) and (51) into Eq. (49) yields our specific

solution for the production rate:

v t s c c

t s

c t s

ct s T

( ; , , )

[ , ],

[ ]( , ].1 2 2

1

0 0

2

=" Œ

-" Œ

ÏÌÔ

ÓÔ(52)

Observe that m( )◊ and v( )◊ are continuous functions of t, just as asserted above. More precisely,

m( )◊ and v( )◊ are piecewise smooth functions of t.

To finish up the solution, we must find the time path of the stock of inventory and the

switching time s. Using Eq. (52) the state equation x u= becomes

˙

[ , ],

[ ]( , ].

x

t s

c t s

ct s T

=" Œ

-" Œ

ÏÌÔ

ÓÔ

0 0

22

1


138

Integrating this differential equation separately over each time interval yields the general solution

x t

a t s

c t

c

c st

ca t s T

( )

[ , ],

( , ],=

" Œ

- + " Œ

ÏÌÔ

ÓÔ

2

22

1

2

13

0

4 2

(53)

where a2 and a3 are constants of integration. Using the initial condition x( )0 0= gives a2 0= .

Because x( )◊ is a continuous function of t by Theorem 4.4, we can solve for a3 by equating both

portions of the solution in Eq. (53) at t s= using a2 0= . This yields the equation

c s

c

c s

ca2

2

1

22

134 2

0- + = ,

which when solved for a3 gives

ac s

c32

2

14= . (54)

Substituting Eq. (54) into Eq. (53) yields the specific solution of the state equation

z t s c c

t s

c

ct s t s T

( ; , , )

[ , ],

[ ] ( , ].1 2 2

1

2

0 0

4

=" Œ

- " Œ

ÏÌÔ

ÓÔ(55)

As expected, z( )◊ is a piecewise smooth function of t.

Finally, we seek to determine the switching time s. Since the only unused boundary con-

dition is the terminal one, namely x T xT( ) = , it can be used to determine s. Using this boundary

condition in Eq. (55) gives cc TT s x2

142[ ]- = , which yields

s Tc x

cT= - 2 1

2

(56)

as the solution for the switching time, as you are asked to show in a mental exercise. In sum,

Eqs. (50), (51), (52), (55), and (56) represent the unique solution to the problem since they are

the only solution to the necessary and sufficient conditions.

An interesting feature of the solution for the switching time is that the length of the sub-

interval in which production occurs, videlicet

T sc x

cT- = 2 1

2

,


139

is identical to the length of the planning horizon the firm would choose if T was a decision vari-

able, as we will see in Chapter 8. This observation means that the length of the subinterval in

which production occurs is unaffected by how distant the terminal period T is, as long as our

original assumption on the length of the planning horizon holds, to wit x c c TT < -14 2 1

1 2. This, in

turn, implies that the production rate in the interval [ , ]s T in also unaffected by an increase in T.

As can be seen from Eq. (56), ∂ ∂s T = 1, thus an increase in T lengthens the initial period when

no production occurs, but leaves the length of the period in which positive production occurs un-

changed, thereby implying that the positive production rate in the interval [ , ]s T is unaffected too.

We leave the remaining comparative dynamics results for a mental exercise.

In the next chapter we take up the study of optimal control problems that often exhibit

piecewise continuous optimal control functions. In order to solve such problems, we often must

approach them somewhat differently since the differential calculus has a more limited role in the

characterization of the optimal control. So as to convince you that this generality is not simply

of mathematical interest, we will present and solve three economic problems that exhibit piece-

wise continuous optimal control functions.


140

Mental Exercises

4.1 Consider the fixed endpoints optimal control problem

min [ ( ), ( )] [ ( )]( )u

J x u u t dt◊

◊ ◊ = Údef 2

0

1

s.t. ˙( ) ( ) ( ), ( ) , ( )x t x t u t x x= + = =0 1 1 0.

(a) Find the solution to the necessary conditions.

(b) Prove that the solution found in (a) solves the optimal control problem.


max [ ( ), ( )] ( ) [ ( )]( )u

J x u tu t u t dt◊

◊ ◊ = -[ ]Údef a 12

2

0

1

s.t. ˙( ) ( ) ( )x t u t x t= - , x x x( ) , ( )0 1 00= = .



4.3 The following questions pertain to Example 4.3.

(a) Show that u t t1( ) sin= is an admissible control.

(b) Show that u t212( ) = is an admissible control.

(c) Compute J x u[ ( ), ( )]1 1◊ ◊ and J x u[ ( ), ( )]2 2◊ ◊ .

(d) Compare J x u[ ( ), ( )]1 1◊ ◊ and J x u[ ( ), ( )]2 2◊ ◊ with the value of J[ ]◊ computed in the ex-

ample.

4.4 Consider the optimal control problem

max [ ( ), ( )] ( ) ( )( ),u x

J x u x t u t dt◊

◊ ◊ = +[ ]Ú1

0

1def

s.t. ˙( ) ( ) ( ) , ( ) , ( )x t x t u t t x x x= - + + = =0 1 1 1,

u t U u u t( ) ( ) : ( )Œ = ◊ £ £{ }def0 1 .

(a) Write down all the necessary conditions for this problem.

(b) Find the unique solution of the costate equation.

(c) Find the decision rule governing the optimal control, and find v t( ).

(d) Determine the associated path of the state variable z t( ) .


141

(e) Prove that the solution to the necessary conditions is the optimal solution of the

control problem.

(f) Solve the control problem directly, that is, without the aid of the Maximum Prin-

ciple. Hint: Solve the state equation.


max [ ( ), ( )] ( )( )u

J x u u t dt◊

◊ ◊ = Údef

0

1

s.t. ˙( ) ( ) [ ( )] , ( ) , ( )x t x t u t x x= + = =2 0 1 1 a .

where a is some parameter.

(a) Show that there is no solution to the optimal control problem if a £ 1.

(b) Would your answer to part a change if “max” was replaced by “min”? Why or

why not?

4.6 Consider the optimal control problem:

max ( )( )u

u t dt◊ Ú

0

1

s.t. ˙( ) ( ) ( ) , ( ) , ( )x t u t u t x x= - ( )[ ] = =2 20 0 1 0,

u t( ) [ , ]Œ 0 2 .

Prove that v t t( ) [ , ]= " Œ1 0 1 is the optimal control and satisfies the Maximum Principle

only for l0 0= .

4.7 Consider the two state variable optimal control problem:

max ( )( ),u x

t u t dt◊

-[ ]Ú1

12

0

1

s.t. ˙( ) ( ), ( ) , ( )x t u t x x x= = =0 0 1 1,

˙( ) ( ) ( ) , ( ) , ( )y t x t tu t y y= -[ ] = =20 0 1 0.

Prove that any constant control is optimal and satisfies the Maximum Principle only for

l0 0= . Hint: Note that x t tu t x t tu t u t tu t( ) ( ) ˙( ) ˙( ) ( ) ˙( )- ∫ fi - - ∫ - ∫0 0.


142

4.8 Consider the two control variable optimal control problem:

max [ ( ) ( )]( ), ( )u u

u t u t dt1 2

1 20

1

2◊ ◊

-Ú

s.t. ˙( ) ( ) ( ) , ( ) , ( )x t u t u t x x= -[ ] = =1 2

20 0 1 0 ,

u t u t1 211 11( ) [ , ], ( ) [ , ]Œ - Œ - .

Prove that v t t1 1 0 1( ) [ , ]= " Œ and v t t2 1 0 1( ) [ , ]= " Œ are optimal and satisfy the Maximum

Principle only for l0 0= .

4.9 Find necessary conditions for a solution of the fixed endpoints optimal control problem

with a bounded control variable, namely

max , ( ), ( )( )u

t

t

f t x t u t dt◊

( )Ú0

1

s.t. ˙( ) , ( ), ( )x t g t x t u t= ( ), x t x x t x( ) , ( )0 0 1 1= = ,

a t u t b t( ) ( ) ( )£ £ .


max [ ( ), ( )] ( ) ( ) [ ( )]( ),u x

J x u x t u t u t dt◊

◊ ◊ = - -[ ]Ú2

2 3 2

0

2def

s.t. ˙( ) ( ) ( ), ( ) , ( )x t x t u t x x x= + = =0 4 2 2,

u t U u u t( ) ( ) : ( )Œ = ◊ £ £{ }def0 2 .

Note that this problem is probably best tackled after Example 5.1 is understood.




max [ ( ), ( )] ( )( ),u x

J x u x t dt◊

◊ ◊ = Ú4

30

4def

s.t. ˙( ) ( ) ( ), ( ) , ( )x t x t u t x x x= + = =0 5 4 4 ,

u t U u u t( ) ( ) : ( )Œ = ◊ £ £{ }def0 2 .




143

4.12 Prove Theorem 4.4.

4.13 Consider the optimal control problem (43):

max [ ( ), ( )] , ( ), ( )( )u

x u x u◊

◊ ◊ = ( )ÚJ f t t t dtt

tdef

0

1

s.t. ˙ ( ) , ( ), ( )x g x ut t t t= ( ) , x x x x( ) , ( )t t0 0 1 1= = ,

h u 0t t K, ( )( ) ≥ ,

Let z v( ), ( )t t( ) be the optimal pair, ll( )t be corresponding time path of the costate vector,

and mm( )t the time path of the corresponding Lagrange multiplier vector. Define the

Hamiltonian as

H t f t t f t g tnn

n

N( , , , ) ( , , ) ( , , ) ( , , ) ( , , )x u x u g x u x u x ull ll= + ¢ = +

=Âdef l1

,

and the Lagrangian as

L t H t t f t g t h tnn

n

N

kk

k

K

( , , , , ) ( , , , ) ( , ) ( , , ) ( , , ) ( , )x u x u h u x u x u ull mm ll mm= + ¢ = + += =

Â Âdef l m1 1

.

Assume that f C( ) ( )◊ Œ 1 , g( ) ( )◊ ŒC 1 , and h( ) ( )◊ ŒC 1 in ( , , )t x u .

(a) Prove that

d

dtL t t t t t

tL t t t t t, ( ), ( ), ( ), ( ) , ( ), ( ), ( ), ( )z v z vll mm ll mm( ) = ( )∂

∂.

(b) Prove that if the optimal control problem is autonomous, that is, the independent

variable t doesn’t enter the functions f ( )◊ , g( )◊ , or h( )◊ explicitly, that is,

f tt ( , , )x u ∫ 0, g x u 0t Nt( , , ) ∫ , and h u 0t Kt( , ) ∫ , then L( )◊ is constant along the op-

timal path.

(c) Prove that

d

dtH t t t t

tH t t t t, ( ), ( ), ( ) , ( ), ( ), ( )z v z vll ll( ) = ( )∂

∂.

Note that this result is not the same as that in Mental Exercise 2.31.

(d) Prove that if the optimal control problem is autonomous then H ( )◊ is constant

along the optimal path too.

4.14 Prove Theorem 4.5.


144

4.15 This mental exercise asks to compute and economically interpret the remaining compara-

tive dynamics of the optimal inventory accumulation problem in Example 4.5.

(a) Derive the comparative dynamics result for an increase in total required produc-

tion xT on the production rate and the shadow cost of inventory. Provide an eco-

nomic interpretation.

(b) Derive the comparative dynamics result for an increase in the unit cost of produc-

tion coefficient c1 on the production rate and the shadow cost of inventory. Pro-

vide an economic interpretation.

(c) Derive the comparative dynamics result for an increase in the length of the pro-

duction run T on the production rate and the shadow cost of inventory. Provide

an economic interpretation.

4.16 With reference to Example 4.6, the inventory accumulation problem with a nonnegativity

constraint on the production rate:

(a) Explain why s TŒ( , )0 must hold.

(b) Provide an economic interpretation of l( ; )s bb = 0.

(c) Derive Eq. (56) for the switching time and explain your steps fully and carefully.

(d) Derive the comparative dynamics for the switching time and production rate for

an increase in c1. Provide an economic interpretation.

(e) Derive the comparative dynamics for the switching time and production rate for

an increase in c2 . Provide an economic interpretation.

(f) Derive the comparative dynamics for the switching time and production rate for

an increase in xT . Provide an economic interpretation.


min [ ( ), ( )] ( ) [ ( )]( )u

J x u tu t u t dt◊

◊ ◊ = +[ ]Údef 2

0

1

s.t. ˙( ) ( )x t u t= , x x( )0 0= , x x( )1 1= .


145

(a) Find the solution of the necessary conditions, and denote the solution triplet by

z t x x v t x x t x x( ; , ), ( ; , ), ( ; , )0 1 0 1 0 1l( ).

(b) Prove that the solution found in part (a) is the unique solution of the optimal con-

trol problem.

(c) Define the optimal value function by

V x x tv t x x v t x x dt( , ) ( ; , ) [ ( ; , )]0 1 0 1 0 12

0

1

= +[ ]Údef.

The optimal value function is thus the value of the objective functional J x u[ ( ), ( )]◊ ◊

evaluated along the solution curve. Find an explicit expression for V x x( , )0 1 .

(d) Compute ∂∂ x V x x

0 0 1( , ) and ∂∂ x V x x

1 0 1( , ).

(e) Show that

∂∂ x

V x x x x0

0 1 0 10( , ) ( ; , )= l

and

∂∂ x

V x x x x1

0 1 0 11( , ) ( ; , )= -l .

This is a special case of the Dynamic Envelope Theorem, which will be covered in

a general fashion in Chapter 9. Note also that the first result is equivalent to the

definition given in Eq. (21).

4.18 Consider a simple generalization of the optimal inventory accumulation problem in Ex-

ample 4.5. Assume now that the firm discounts its costs at the rate r > 0, but the problem

is otherwise unchanged. Thus the firm is asserted to solve

min [ ( ), ( )] [ ( )] ( )( )u

rtT

J x u c u t c x t e dt◊

-◊ ◊ = +[ ]Údef1

22

0

s.t. ˙( ) ( )x t u t= , x( )0 0= , x T xT( ) = .

Assume that the nonnegativity constraint u t t T( ) [ , ]≥ " Œ0 0 is not binding. This assump-

tion will be addressed later in this question.

(a) Derive the necessary conditions. Show that


146

2 21 2 1e c u t e c ds e c u trt rs

t

tr t- -

+- ++ = +Ú( ) ( )[ ]

ee e

along an optimal path, where e > 0 is sufficiently small. Provide an economic

interpretation of this integral equation.

(b) Find the solution of the necessary conditions, and denote the solution triplet by

z t v t t( ; ), ( ; ), ( ; )gg gg ggl( ) , where gg =def( , , , , )c c r T xT1 2 .

(c) Prove that the solution of the necessary conditions is the unique solution of the

optimal control problem.

(d) Show that ˙( ; )v t gg > 0 holds in the optimal plan. Provide an economic interpreta-

tion of this result.

(e) Derive a necessary and sufficient condition for the nonnegativity constraint

u t t T( ) [ , ]≥ " Œ0 0 to be nonbinding in an optimal production plan.

(f) Show that ∂l ∂( ; ) [ , ]t x t TTgg < " Œ0 0 and ∂ ∂˙( ; ) [ , ]z t x t TTgg > " Œ0 0 . Provide an

economic interpretation of these comparative dynamics results. Note that there

are no simple refutable comparative dynamics results for the shadow cost of the

inventory or the production rate for the remaining parameters, in sharp contrast to

the basic inventory accumulation model studied in Example 4.5 and in Mental

Exercise 4.15, which had the discount rate set equal to zero.

4.19 Consider another generalization of the optimal inventory accumulation problem in Ex-

ample 4.5, where now the production cost function is given by g( )◊ , and is assumed to

have the following properties:

g C g g u g u u( ) , ( ) , ( ) ( ) .( )◊ Œ = ¢ ≥ ¢¢ > ≥2 0 0 0 0 0 and for

Thus production costs are a nondecreasing strongly convex function of the production

rate. Also assume that costs are discounted at the rate r > 0. Under these assumptions,

the optimal inventory accumulation problem is given by

min [ ( ), ( )] ( )( )u

rtT

J x u e g u t c x dt◊

-◊ ◊ = ( ) +[ ]Údef2

0

s.t. ˙( ) ( )x t u t= , x( )0 0= , x T xT( ) = .


147

(a) Derive the necessary conditions.

(b) Show that ˙( )u t > 0 holds in an optimal plan. Provide an economic interpretation

of this result.

(c) Show that

e g u t e c ds e g u trt rs

t

tr t- -

+- +¢( ) + = ¢ +( )Ú( ) ( )[ ]

2

ee e

in an optimal plan, where e > 0 is sufficiently small. Provide an economic inter-

pretation of this integral equation.

(d) Now set the discount rate to zero, i.e., set r ∫ 0, and derive the necessary condi-

tions.

(e) Show that for a given u t( ) , ˙( )u t is larger for r > 0 than for r ∫ 0. Explain, in

economic terms, why this makes sense. Draw a graph in tu-space to show your

reasoning geometrically.

(f) For the r > 0 and r ∫ 0 cases, show that for a given u t( ) , ˙( )u t increases with an

increase in the per unit holding cost of the inventory c2 . Explain, in economic

terms, why this makes sense. Draw a graph in tu-space to show your reasoning

geometrically.

4.20 Recall the intertemporal utility maximization problem developed in Example 1.3:

max [ ( ), ( )] ( )( )c

rtT

J k c e U c t dt◊

-◊ ◊ = ( )Údef

0

s.t. ˙( ) ( ) ( )k t w ik t c t= + - , k k( )0 0= , k T kT( ) = .

Please refer back to Example 1.3 for the economic interpretation of the model and its

variables and parameters if you memory of it is vague. Also recall that U C( ) ( )◊ Œ 2 , and

that ¢ >U c( ) 0 and ¢¢ <U c( ) 0.

(a) Derive the necessary conditions and provide an economic interpretation of the

Maximum Principle equation.


148

(b) Prove that a solution of the necessary conditions, assuming that one exists, is a

solution of the intertemporal utility maximization problem.

(c) Prove that

e U c t i e U c s ds e U c trt rs

t

tr t- -

+- +¢( ) = ¢( ) + ¢ +( )Ú( ) ( ) ( )( )

ee e

along an optimal path, where e > 0 is sufficiently small. Provide an economic

interpretation of this integral equation.

(d) Prove that

˙( )( )

[ ]cU c

U ci r=

- ¢¢¢

ÈÎÍ

˘˚

-

along an optimal path. Provide an economic interpretation.

In order to tease out some more qualitative results and come up with a closed form solu-

tion of the problem, assume that U c c( ) ln=def, kT = 0, and w = 0.

(e) Derive the unique solution of the control problem under the above simplifying as-

sumptions. Denote it by k t c t t* *( )( ; ), ( ; ), ( ; )bb bb bbl , where bb =def( , , , )i r k T0 .

(f) Obtain the comparative dynamics of the optimal triplet with respect to i and pro-


(g) Obtain the comparative dynamics of the optimal triplet with respect to r and pro-


(h) Obtain the comparative dynamics of the optimal triplet with respect to k0 and

provide an economic interpretation.

(i) Obtain the comparative dynamics of the optimal triplet with respect to T and pro-


4.21 Suppose that at time t = 0 (the present) a mine contains an amount x0 0> of a nonrenew-

able resource stock, say, e.g., coal. The profit flow that results from extracting and sell-

ing the resource at the rate q t( ) is given by p ( ) lnq q=def. The firm lives over the closed

interval [ , ]0 T and discounts its profit at the rate r > 0. Moreover, the firm is required to


149

extract all the asset by time t T= , implying that cumulative extraction over the planning

horizon [ , ]0 T must equal the initial size of the deposit x0 0> . Stated in mathematical

terms, this constraint is given by

q s ds xT

( ) =Ú 00

.

The firm is asserted to maximize its present discounted value of profit. The optimal con-

trol problem can therefore be posed as

max ( )( )q

rtT

q t e dt◊

-( )Ú p0

s.t. q t dt xT

( ) =Ú 00

.

Unfortunately, at this juncture, we are not able to solve this isoperimetric control prob-

lem. We will have to wait until Chapter 7 to solve it in this form. We can, however,

transform it to a form readily solvable by our current methods. To this end, define x t( )

as the cumulative amount of the asset extracted by time t, that is

x t q s dst

( ) ( )= Údef

0

.

Applying Leibniz’s Rule to this definition and using the integral constraint on the extrac-

tion rate yields the associated differential equation and boundary conditions, namely

˙( ) ( ), ( ) , ( )x t q t x x T x= = =0 0 0 .

Hence, the equivalent optimal control problem in a form we can handle via the theorems

developed so far, is given by:

max [ ( ), ( )] [ln ( )]( )q

rtT

J x q q t e dt◊

-◊ ◊ = Údef

0

s.t. ˙( ) ( )x t q t= , x x T x( ) , ( )0 0 0= = .

(a) Derive the solution of the necessary conditions and denote the resulting triplet by

x t r x T q t r x T t r x T* *( )( ; , , ), ( ; , , ), ( ; , , )0 0 0l .


150

(b) Prove that the solution of the necessary conditions is the unique solution of the

control problem.

(c) Compute the comparative dynamics of an increase in the initial stock on the pre-

sent value shadow price of the stock and the extraction rate. That is, derive

∂∂

∂∂

lx

t r x Tq

xt r x T

00

00( ; , , ) ( ; , , ) and

*

.

Explain, in economic jargon, your results.

(d) Compute the comparative dynamics of an increase in the discount rate on the pre-

sent value shadow price of the stock and the extraction rate. Since you will not be

able to sign this result " Œt T[ , ]0 , evaluate your comparative dynamics results at

t = 0 and t T= . Explain, in economic jargon, your results. Hint: Recall that the

Taylor series representation for ey about the point y = 0 is given by ey ykk

k

==

+•Â !0.

(e) Compute the comparative dynamics of an increase in the planning horizon on the

cumulative stock of the resource and the extraction rate. Explain, in economic

jargon, your results.

Now let the profit flow be a general C( )2 function of the extraction rate, say p ( )q , with

¢ >p ( )q 0 and ¢¢ < ≥p ( ) .q q0 0 for Answer the remaining three parts of the exercise based

on the general profit function.

(f) Prove that the present value of marginal profit is constant over the entire horizon.

(g) Prove that marginal profit grows exponentially at the discount rate r.

(h) Prove that the optimal extraction rate declines through time.

4.22 Consider the so-called simplest problem in the calculus of variations, defined by

max [ ( )] , ( ), ˙( )( )x

t

t

J x f t x t x t dt◊

◊ = ( )Údef

0

1

s.t. x t x( )0 0= , x t x( )1 1= .

Observe that there are no constraints in the problem and that both of the endpoints are

fixed. Assume that f C( ) ( )◊ Œ 2 over an open set and that admissible functions are C( )2 . It


151

is well known that a necessary condition obeyed by the optimizing curve z t( ) is given by

the differential equation

d

dtf t x t x t f t x t x tx x˙ , ( ), ˙( ) , ( ), ˙( )( ) = ( ),

known as the Euler equation. This question asks you to prove that the necessary condi-

tions of the optimal control problem corresponding to the simplest problem in the calcu-

lus of variations are equivalent to the Euler equation. One benefit of answering this

question is that it provides some background material for Chapter 7.

(a) Transform the above calculus of variations problem into an equivalent optimal

control problem by letting ˙( ) ( )x t u t=def be the state equation.

(b) Write down the necessary conditions for the optimal control problem in (a).

(c) Show that if z v z z( ), ( ) ( ), ˙( )◊ ◊( ) = ◊ ◊( ) is a solution of the necessary conditions of the

optimal control problem in (a), then it is also a solution of the Euler equation.

(d) Show that if z z( ), ˙( )◊ ◊( ) is a solution of the Euler equation, then it is also a solution

of the necessary conditions of the optimal control problem in (a). Hint: Define

the variable l( ) , ( ), ˙( )˙t f t z t z tx= - ( )def.


152

Further Reading

The original statement of the Principle of Optimality is contained in Bellman (1957). Bellman

and Dreyfus (1962) is another useful reference on the principle. Both of these books still provide

readable and informative introductions to dynamic programming. Our use of the Principle of

Optimality in this chapter was very specific to the purpose at hand, namely, to prove the neces-

sary conditions for a rather general optimal control problem. As we will see in later chapters, the

Principle of Optimality will lead to the derivation of a useful partial differential equation called

the Hamilton-Jacobi-Bellman equation. This equation will become the centerpiece of our foray

into intertemporal duality theory. Chiang (1992), Leonard¢ and Van Long (1992), and Kamien

and Schwartz (1991) all provide at least some discussion of the Principle of Optimality and dy-

namic programming, with the latter providing a bit more than the first two.


153

References

Bellman, R. (1957), Dynamic Programming (New Jersey: Princeton University Press).

Bellman, R. and S. Dreyfus (1962), Applied Dynamic Programming (New Jersey: Princeton

University Press).

Chiang, A.C. (1992), Elements of Dynamic Optimization (New York: McGraw-Hill, Inc.).

Kamien, M.I. and Schwartz, N.L. (1991, 2nd Ed.), Dynamic Optimization: The Calculus of

Variations and Optimal Control in Economics and Management (New York: Elsevier

Science Publishing Co., Inc.).

Leonard¢ , D. and Van Long, N. (1992), Optimal Control Theory and Static Optimization in Eco-

nomics (New York: Cambridge University Press).

Seierstad, A. and Sydsæter, K. (1987), Optimal Control Theory with Economic Applications

(New York: Elsevier Science Publishing Co., Inc.).

Simon, C.P. and Blume, L. (1994), Mathematics for Economists (New York: W.W. Norton &

Company, Inc.).

chapter 4 the maximum principle and economic … maximum principle and economic interpretations ......

Documents