dynamic programming and its applications to economic...

Dynamic Programming

and its

Applications to Economic Theory

Juan Carlos Suarez Serrato

May 18, 2006

Contents

Contents i

Dedication ii

Acknowledgements iii

1 Introduction 1

1.1 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Historical Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

1.3 Applications and Extensions . . . . . . . . . . . . . . . . . . . . . . 5

2 Continuities 7

2.1 Functional Continuity . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2.2 Continuities of Correspondences . . . . . . . . . . . . . . . . . . . . . 10

3 Dynamic Programming 15

3.1 Contraction Mappings and Fixed Points . . . . . . . . . . . . . . . . 15

3.2 Theorem of the Maximum . . . . . . . . . . . . . . . . . . . . . . . . 18

3.3 Dynamic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . 21

3.4 Euler Equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

4 Economic Growth 34

Appendices 39

A Notation Index 39

Bibliography 40

i

Dedication

In the past year, two of my guiding lights faded. From my grandfathers I learnedthe meaning of hard work, the value of being honest with one’s self, and the deter-mination to strive against adversity. They gave me the motivation to fulfill theirhigh expectations and, through their example, the hope to follow in their footsteps.

ii

Acknowledgements

Throughout my college years I have enjoyed the support and encouragementof many people to whom I owe a great deal of respect and admiration. I wouldespecially like to thank my adviser, Dr. Allen Holder, for the time he devoted tothis project, and for being the most inspiring mathematician I know. His challengeshave always been stimulating, and his insights invaluable.

I wish to acknowledge other mathematicians and economists who did not in-fluence this project directly, but whose help and kindness shaped my experiencethe last four years. For his friendship and for nailing my feet to the ground whenI needed it done, I thank Dr. Eugenio Suarez. I would like to thank Dr. JohnHuston, my economics adviser, for providing valuable counsel I repeatedly failed totake. For their perennial support and inspiration, I would like to thank Dr. RobertoHasfura, Dr. Jorge Gonzalez, Dr. Barry Hirsch, Dr. Julie DeCourcy, and Dr. ScottChapman. I want to thank Dr. Sean Knabb for introducing me to this material,and to Dr. Charles Becker for allowing this to happen.

It is hard to overestimate the value of the Mathematics Department at TrinityUniversity; one can not think of a better place to be inspired. The past four yearswould not have been possible were it not for the students of Mathematics at Trinity,whose friendship and encouragement made our struggles fun.

Finally, commencement nor completion of this project would have been possi-ble without the understanding and support of my friends and family to whom I amenormously indebted.

iii

Chapter 1

Introduction

The use of Dynamic Programming in economic modelling has revolutionizedeconomic thought and has allowed the field to tackle interesting problems such asthose approached by Edward Prescott and Finn Kydland, the 2004 Nobel Laureatesin Economic Science. The award was presented “for their contributions to dynamicmacroeconomics: the time consistency of economic policy and the driving forces be-hind business cycles” (Nobel, 2004). The models they pioneered are now ubiquitousand are an indispensable part of every economist’s toolbox. Moreover, such topicsare given high priority in most doctoral programs in Economics. It is an apparentcontradiction that these models rarely manage to find their way into undergraduatecurricula in Economics programs.

The mathematical tools available to undergraduates majoring in Economicsis, in most cases, limited by the course offerings of the Mathematics department.In order to include dynamic models in undergraduate Economics programs, sometreatment of dynamic programming must be introduced in the course offerings ofMathematics departments. Although the author’s main interest is Economics, dy-namic programming spans several disciplines in application including Astronomy,Physics, and Engineering. Students of these disciplines would benefit from under-standing the applications of these methods to Economics as much as students ofother disciplines benefit from examples drawn from Physics and Engineering thatare presented in most Calculus courses.

This guide is an introduction to the field of dynamic programming that bal-ances rigor and applications. Although it is not a comprehensive survey of the field,it encompasses a rigorous development of the theory that is sufficient to tackle in-teresting problems. The attempt is to make this material accessible to advancedundergraduate students who have knowledge of mathematical optimization or realanalysis. The author shares the point of view of McShane (1989), who opines “or-

1

Chapter 1 – Introduction Suarez Serrato – Page 2

dinary undergraduate students of mathematics should be taught a form of controltheory simple enough to be understood and general enough to be applicable to manyproblems.”

The first objective is to contextualize some of the concepts used in the devel-opment of the theory behind dynamic programming in terms familiar to the targetaudience. These ideas include the use of correspondences as a generalized conceptof a function and the characterization of continuity to such correspondences. Thesecond objective is to present a rigorous development of the results that validate theuse of these methods. The motivation to present this development is twofold. First,it is desirable to establish the results. Second, the approach followed is attractivein concept and provides some simple yet interesting ways to attack problems thatmight arise elsewhere. The third and final objective is to understand, by consideringan application, the usefulness of the theory developed. Through this example weaim to understand why the development follows the path it does. We hope thisguide serves as an introductory overview that will encourage the reader to furtherpursue the study of dynamic programming.

The remainder of the chapter is organized as follows. Section 1.1 gives a briefintroduction to some of the main references associated with dynamic programming,outlines the texts that are central in our development, and suggests some preparatoryreading. Section 1.2 presents a brief historical account of the growth of the field.Section 1.3 presents an outline of some applications of the methods of dynamicprogramming and gives some suggestions as to how the reader might delve into thefield after understanding what is developed in this treatment.

1.1 Literature Review

Several approaches to the study of dynamic programming exist. These includethe Calculus of Variations, which gave birth to Optimal Control Theory, as well asother recursive methods of Dynamic Programming. These methods are all relatedand coincide in the idea of “thinking in states,” as Ljungqvist and Sargent (2004)note. By the concept of states, we mean that a problem of inter-temporal optimiza-tion will be affected, or have a different state, in a future period depending on thesolution achieved in this period. The recursiveness of states links these problemsthrough time making these methods a natural approach to problems in the fieldsof Physics and Economics. For our purposes we refer to dynamic programming,control theory, and recursive methods as the same concept giving preference to theterm dynamic programming. One of the goals of this project is to contextualizethese approaches both historically and topically and find a suitable approach to thematerial.


In exploring the field of dynamic programming from an applications pointof view, the methods presented in most books like Hoy et al. (2001), Ljungqvistand Sargent (2004), and Shone (2002) are “naıve,” as McShane (1989) notes, whereapplications are discussed with no existence theorems being established. Conversely,the books that develop the theory are overly dense and sometimes too involved tobe approachable by an undergraduate student. Finally, recent developments inStochastic Control Theory, such as those presented in Yong and Zhou (1999) andYoung (2000), make an introduction to this field even more daunting because theyrely on Measure Theory.

As an undergraduate seeking to understand the results that prove the methodsof dynamic programming, the author would have liked to have found a treatmentthat combines both rigor and applications. Given that the field is young, the originaldevelopments of the material are still accessible and relevant. Richard Bellman firstcoined the title of dynamic programming to the study of these methods in his 1957monograph (Bellman, 1957). It is a consensus between several authors writing inthis field that his original piece still provides much insight and is an enjoyableread. Pontryagin et al. (1964) present the other classic work in the field, which waspublished shortly afterwards and uses an approach that relies on partial differentialequations. The approaches used by the two classic treatments above view time as acontinuous process and employ several methods such as those of partial differentialequations to develop the material. These characteristics made these developmentsunappealing since most data and applications in Economics view time as a discreteprocess.

The treatment presented in Stokey et al. (1989) serves as main reference. Wefollow this development since it has several desirable attributes with respect to ap-plications and development of the material. Stokey et al. (1989) first present theresults of deterministic dynamic programming. Several examples of applications arethen presented in abstract form. A basic treatment of Measure Theory is then intro-duced and is sufficient to develop the results of stochastic dynamic programming,followed by some applications of these methods to more realistic examples. Thisdevelopment serves the authors’ purpose of having this book be a reference guidefor researches in the area. As such, it is inadequate to provide an introduction toundergraduates with an interest in the field, but is a great source of results andinsight that will drive this project.

The approach followed in Stokey et al. (1989) is useful since the reader may befamiliarized with the idea of thinking in “states” as well as develop an appreciationfor the field. In economics, the concept of states is analogue to the an agent’sdecision-set. At every time period, the agent being modelled chooses one decisionfrom its decision-set. This decision affects the possibilities available to him in thenext period. Following this process recursively, we have that the choices available


to an agent at any point in time depend on the original state, and the choices madeat every preceding time period. By developing the theory of deterministic dynamicprogramming, Stokey et al. (1989) habituate the reader to thinking in states throughthe use of correspondences.

In order to contextualize the material in Stokey et al. (1989), we rely on resultspresented in a typical undergraduate Mathematics program. Suggested prerequisitesinclude the sections on mathematical optimization, metric spaces, and real analysisas presented in Holder (2005), Bryant (1985), Rudin (1976), and Royden (1988)respectively. Finally, other results more particular to economics are drawn fromMas-Colell et al. (1995) and Chiang (1984).

1.2 Historical Motivation

As mentioned above, the contributions of Bellman (1957) and Pontryagin et al.(1964) formally described the results that solidified the study of these methods.Although these contribution were made recently, the problems they address havebeen attacked by some of the first mathematicians in history. The contributionsof these mathematicians provided some important results and influenced the workof their modern counterparts, although they did not establish the existence resultsformally of Pontryagin et al. (1964) or the insight of separating the infinite processinto a one-stage control problem as developed by Bellman (1957).

A comprehensive treatment of the history of dynamic programming and itsprecursors is beyond the scope of this guide. However, a short review of the accom-plishments is warranted. Ferguson (2004) traces the “development of the theory ofthe calculus of variations, from its roots in the work of Greek thinkers and continuingthrough to the Renaissance.” This study commences by setting up some problemsstudies in antiquity that seem to be direct precedents of the study of the calculusof variations. Some of these include Hero’s principle of least time and the isopara-metric problems of Pappus. Ferguson (2004) then follows the contributions of greatmathematicians such as Fermat, Newton, as well as Bernoulli’s postulation of thebrachistochrone problem, “the problem concerning the shortest distance betweentwo [accelerating] points,” (Fomin and Gelfand, 2000) in an infamous competition.The first real problem of this kind to be formulated and solved was posed by New-ton in his “famous work on mechanics methods, Philosophiae naturalis principiamathematica (1685) [· · · ], thus marking the birth of the theory of the calculus ofvariations”(Ferguson, 2004). Several other advances would facilitate the develop-ment of the current theory of dynamic programming. One example of these is theconception of the first recursive function by Kurt Godel (Goldstein, 2005).


Ferguson (2004) proceeds to tell the story of the correspondence between Eulerand Lagrange and how their interaction fomented the derivation of the Euler equa-tions, which is the goal of the final section in Chapter 3. The last topic in this studyis the emanation of optimal control theory from the calculus of variations. McShane(1989) presents a different perspective regarding the historical development of dy-namic programming. This study is centered on the history of optimal control theory,and has a less forgiving perspective.

McShane (1989) recognizes, as does Ferguson (2004), the lack of philosophicalrigor in the calculus of variations and criticizes it more pungently. When commentingon the treatment of the calculus of variations in Fomin and Gelfand (2000) he notesthat, “if the calculus of variations is mathematics, our conclusions must be deduciblelogically from the hypothesis, with no use of anything that is ‘clear from the physicalmeaning’ ”(McShane, 1989). In the same way McShane (1989) chastises some of theearly developments in the field of calculus of variations, he argues later developmentswere mislead. He states:

The whole subject was introverted. We who were working in it were strivingto advance the theory of the calculus of variations as an end in itself, withoutattention to its relation with other fields of activity (McShane, 1989).

In the perspective of McShane (1989), the contributions of Pontryagin et al. (1964)changed the focus of the calculus of variations by focusing on problems in Engineer-ing and Economics. “In the process, they incidentally introduced new and importantideas into the calculus of variations” (McShane, 1989).

Earlier contributions by Bellman (1957) and others made these methods ripefor the application in Economics. As Edward Prescott noted in his address whenreceiving the Nobel Prize, it took several years before he and others would rein-vent the existing fields of optimal control theory and dynamic optimization with anEconomics flavor (Nobel, 2004).

1.3 Applications and Extensions

It is desirable to demonstrate the power of these methods by considering ap-plications. Given the nature of this project, we examine in detail one problem, thatof deterministic economic growth. The approach taken in the final chapter of thisguide is fairly abstract, with the goal of this chapter being to elucidate some intu-ition into why the assumptions made to obtain the desired results are consistent witheconomic theory. However, there are numerous applications that are more concreteand that might help the reader establish a better understanding of the material.


Ljungqvist and Sargent (2004) present a plethora of applications in economicsthat might quench the reader’s thirst for a less rigorous development. Fischer andBlanchard (1989) present a thorough treatment of modern macroeconomics, in whichmany of the models presented rely on recursive methods to provide insight. Simi-larly, Sala-i-Martin (2000) presents notes from graduate courses in Economic Growththat assumes that students have the ability to solve problems of recursive nature.Barro and Sala-i-Martin (2003) reach a high point in the study of topics related toEconomic Growth and present several models that assume the student is familiarwith the concepts of dynamic programming. Aghion and Griffith (2005) extendcertain growth topics to include the idea of endogenous growth and technologicaldevelopment.

Other applications outside economics were considered for this guide. Themost entertaining applies the Bellman equation to the study of optimal policies insports. Romer (2002) applies the Bellman equation and finds an optimal policy forconversion plays in football. Several other applications in the fields of Engineeringand Physics exist but were not considered.

The nature of this project makes a comprehensive treatment of the field ofdynamic programming almost impossible. In this treatment, several interesting ex-tensions of the material will not be included but are worth mentioning. First, theuse of computational methods to find numerical solutions to stochastic optimal con-trol problems in economics has become ubiquitous in recent years. Diaz-Gimenez(2001) presents the method of linear quadratic approximations to evaluate dynamicprograms with macroeconomic data. Another topic of interest is the stability prop-erties of policy functions derived using the methods presented in this guide. Stokeyet al. (1989) and Vohra (2005) present treatments of the study of dynamic stabilityand interesting applications of these results.

The following Chapters are organized as follows. Chapter 2 introduces themathematical background necessary for the development of the theory of dynamicprogramming in Chapter 3, and it contextualizes it within other concepts advancedundergraduates should be familiar with. Chapter 3 develops the theory of determin-istic dynamic programming following the development to the derivation of the Eulerequations. Finally, Chapter 4 presents some insight into the necessity of the assump-tions established in Chapter 3 through the application of dynamic programming tothe analysis of deterministic economics growth in an abstract form.

Chapter 2

Continuities

In this Chapter we consider some material that is preliminary to the resultsthat follow in Chapter 3, and that is requisite to the development of dynamic pro-gramming. The main focus of this Chapter is the study of several definitions ofcontinuities for functions followed by the characterization of continuity for point-to-set mappings, or what we call correspondences. Much of the development thatfollows rests on a firm understanding of the concepts of upper and lower-hemi con-tinuities.

We begin section 2.1 by considering a general definition of functional continu-ity. We then relate this classic definition of continuity to the stronger property ofuniform continuity and upper and lower semi continuities for functions. Section 2.2introduces the concept of correspondences and relates several versions of continuityof correspondences to the analog properties of functions. The section concludes withtwo theorems that are useful in the proceeding section.

2.1 Functional Continuity

The following is a classical definition of functional continuity and is typicallyintroduced in a Real Analysis course.

Definition 1. (Continuous Functions.) Let (X, dX) and (Y, dY) be metric spaces;suppose E ⊂ X, and f maps E into Y . Then f is said to be continuous at p if forevery ε > 0 there exists δ > 0 such that

dX(x, p) < δ implies dY(f(x), f(p)) < ε.

As presented in Rudin (1976), Royden (1988), and other sources, continuity is

7

Chapter 2 – Continuities Suarez Serrato – Page 8

more generally defined as a topological concept. For our purposes there is little gainfrom such a general development, and we restrict the generality of our development.A property that is stronger than continuity is that of uniform continuity.

Definition 2. (Uniform Continuity.) Let f be a mapping of a metric space (X, dX)into a metric space (Y, dY). We say that f is uniformly continuous on X if for everyε > 0there exists δ > 0 such that

dX(q, p) < δ implies dY(f(q), f(p)) < ε.

Two differences exists between Definitions 1 and 2. First, the δ used in De-finition 1 depend on the point p. Secondly, the δ used in Definition 2 can be usedfor every pair of points such that dX(p, q) < δ. We say that uniform continuity is astronger property since every function with this property also has the property offunctional continuity. Theorem 1 below relates the concepts of functional continuitywith uniform continuity through the requirement that the space X be compact.

Theorem 1. Let f be a continuous mapping of a compact metric space (X, dX) intoa metric space (Y, dY). Then f is uniformly continuous in X .

Proof. Let ε > 0. Then ∀ x ∈ E, ∃ δx > 0 3 f(Nδx/2(x)) ⊆ Nε/2(f(x)). Since{Nδx/2(x) : x ∈ E} is an open cover of E, and since A is compact, there exists a finitesubcover, say {Nδxi/2(xi) : i = 1, 2, · · · , n}. Define δ = min{δxi

/2 : i = 1, 2, · · · , n}.Let p, q ∈ E be 3 d(p, q) < δ. Then p ∈ Nδ(xi) for some 1 ≤ j ≤ n. Also,

d(q, xj) ≤ d(p, q) + d(p, xj)

< δ + δxj/2

< δxj.

So,d(f(p), f(q)) ≤ d(f(p), f(xj)) + d(f(xj), f(q)) < ε.

¡

We proceed by introducing the concept of Lipschitz Continuity. This continuityessentially states that the difference in function values is bounded proportionally tothe distance between the arguments.

Definition 3. (Lipschitz Continuity.) Let f be a mapping of a metric space (X, dX)into a metric space (Y, dY). We say that f is Lipschitz continuous on X if ∃ λ > 0such that for every x, y ∈ X,

dY(f(x), f(y)) < λdX(x, y).


Theorem 2 relates the concepts introduced in Definition 3 to Definitions 1and 2. Namely, it states that a function that is Lipschitz continuous is uniformlycontinuous and thus continuous.

Theorem 2. Let f be a mapping of a metric space (X, dX) into a metric space Y withthe property of Lipschitz continuity. Then f is uniformly continuous.

Proof. Let f be as stated and define λ > 0 to be 3

dY(f(x), f(y)) < λdX(x, y).

Let ε > 0 and define δ = ελ. It follows that for q ∈ Nδ(p),

dY(f(x), f(y)) < λdX(x, y)

< λδ = ε.

Since δ was chosen independently of p, we have the desired result. ¡

The following definitions of continuities are not directly related to Lipschitzcontinuity. Instead, we view these as relaxed instances of Definition 1. Moreover, bythe definition of the lim sup and lim inf, these have the advantage of always existingin the extended real numbers.

Definition 4. (Functional Upper Semi-Continuity.) The function f is said to beupper semi-continuous at x0 if

lim supx→x0

f(x) ≤ f(x0).

Definition 5. (Functional Lower Semi-Continuity.) The function f is said to belower semi-continuous at x0 if

lim infx→x0

f(x) ≥ f(x0).

The claim above that the functional definitions of upper and lower semi-continuities are weaker than Definition 1 can be substantiated by verifying that bothlimitsinequalities in the definitions above hold when a function has the property ofbeing continuous. The next theorem provides a way to appreciate this relationship.

Theorem 3. Let f be a mapping of a metric space (X, dX) into a metric space (Y, dY)be such that it is both upper and lower semi-continuous. Then, f is continuous.


Proof. Let x0 ∈ X, and f be as stated. From the definitions above, it follows that

lim infx→x0

f(x) ≥ f(x0) ≥ lim supx→x0

f(x).

From the definitions of the lim inf and lim sup we always have the opposite rela-tionship. We thus have that lim inf

x→x0

f(x) = lim supx→x0

f(x), and the function f is

continuous at x0. ¡

2.2 Continuities of Correspondences

Our focus now shifts to the study of correspondences, or what is commonlyknown as point-to-set maps. In the same way we approached functional continuities,we study several definitions of continuities for correspondences. We further provideinsight as to how these properties relate to those introduced in the previous section.We first provide a formal definition of correspondences.

Definition 6. (Correspondence) A correspondence Γ : X → Y is a relation thatassigns a set Γ(x) ⊆ Y to each x ∈ X.

Correspondences prove to be valuable in the study of dynamic programmingsince they provide a natural way to relate the state in the present period to thestate in the future period. To acclimate the reader to the ideas of lower and upperhemi-continuity of a correspondence, we first introduce the concepts of lower andupper semi-continuity.

Definition 7. (Upper Semi-Continuity for Correspondences) The correspondence Γis upper semi-continuous at x0 if ∀ ε > 0, ∃ δ > 0 3

x ∈ Nδ(x) implies Γ(x) ⊆⋃

y∈Γ(x0)

Nε(y).

The word upper in Definition 7 makes sense since the ε neighborhood of thetarget set is an upper-approximation of the local images.

Definition 8. (Lower Semi-Continuity for Correspondences) The correspondence Γis lower semi-continuous if ∀ ε > 0,∃ δ > 0 3

x ∈ Nδ(x) implies Γ(x0) ⊆⋃

y∈Γ(x)

Nε(y).


Conversely, Definition 8 says that the target set is within an ε neighborhood ofthe local images. The following theorem relates upper and lower semi-continuity ofa function and correspondences is found in Holder (2005). The proof of this theoremis omitted for brevity.

Theorem 4. The function f : X → IR is upper or lower semi-continuous if andonly if the correspondence Γ : X→ IR defined by Γ(x) = {y : y ≤ f(x)} is upper orlower semi-continuous, respectively.

Next, we introduce the concept of a graph of a correspondence. The graph ofa correspondence is often used to establish related properties of the correspondence.

Definition 9. (Graph) The graph of a correspondence Γ is the set

A = {(x, y) ∈ X× Y : y ∈ Γ(x)}.

Having introduced several continuities, we now introduce the concept of up-per hemi-continuity. We establish the desired definition only for compact-valuedcorrespondences. By compact valued we mean that for any x ∈ X, the set Γ(x) iscompact. As Stokey et al. (1989) note, “a general definition of u.h.c. for all corre-spondences is available, but [· · · ] its wider scope is never useful” for our purposes.

Definition 10. (Upper Hemi-Continuity.) A compact-valued correspondence Γ isupper hemi-continuous (u.h.c.) at x if Γ(x) is nonempty and if, for every se-quence xn → x and every sequence {yn} 3 yn ∈ Γ(xn) ∀ n, ∃ a convergent subse-quence of {yn} whose limit point y is in Γ(x).

The following theorem establishes a relationship between the concept of u.h.c.and functional continuity. We attribute the following theorem to Mas-Colell et al.(1995), who note that u.h.c. “ can be thought of as a natural generalization of thenotion of continuity for functions.”

Theorem 5. A correspondence Γ is single-valued and u.h.c. if and only if it is alsocontinuous in the functional sense.

Proof. If Γ is continuous as a function, we have that its graph is closed. Moreover,since the continuous mappings of compact sets is compact, and therefore boundedas well. Thus, Γ is upper hemi-continuous as a correspondence.

Suppose now that Γ is upper hemi-continuous as a correspondence and considerany sequence xn → x ∈ A with xn ∈ A ∀ n. Define S = {xn : n = 1, 2, · · · } ∪ {x}.Now, let ε > 0 and set N ∈ IN to be such that dX(x, xn) < ε ∀ n ≥ N . Define r tobe such that

r > max{||x1||, ||x2||, · · · , ||xN ||, ||x||+ ε}.


We have that ||x|| < r, ∀ x ∈ S, so that S is bounded. Because S is also closed,it follows that S is compact, by the Heine-Borel theorem. By hypothesis, Γ(S) isbounded, and so Γ(S) is compact. Assume for the sake of obtaining a contradictionthat the sequence {Γ(xn)} ⊂ Γ(S) does not converge to Γ(x). Extract a subsequencexnk

→ x such that Γ(xnk) → y for some y ∈ Γ(S) 3 y 6= Γ(x). Then the graph of Γ

is not closed, which contradicts the property of u.h.c. as a correspondence. ¡

The use of single valued correspondences would destroy the motivation tointroduce the concept of correspondences and in general we do no consider them assingle valued. Nonetheless, the theorem above helps achieve the concept of u.h.c.The second type of continuity, lower hemi-continuity for correspondences, presentsanother generalization of the concept of functional continuity for correspondences.

Definition 11. (Lower Hemi-Continuity.) A correspondence Γ is lower hemi-continuous (l.h.c.) at x if Γ(x) is nonempty and if, for every y ∈ Γ(x) and everyxn → x, ∃ N ≥ 1 and a sequence {yn}∞n=N such that yn → y and yn ∈ Γ(x), ∀ n ≥N .

The theorem below establishes a result parallel to that relating u.h.c. to func-tional continuity presented in Theorem 5. For the purpose of brevity, a proof isomitted.

Theorem 6. If a correspondence Γ is single valued and l.h.c. is also continuous inthe functional sense.

The study of l.h.c. and u.h.c. on their own is required to address some of thetopics covered in the following section. Figure 2.1 shows in panel (a) a lower hemi-correspondence that is not upper hemi-continuous, and in panel (b) an upper hemi-continuous correspondence that is not lower hemo-continuous. The images provideintuition about the corresponding definitions. A general concept of continuity of acorrespondence is now introduced.

Definition 12. (Continuity of Correspondences) A correspondence Γ : X → Y iscontinuous at x ∈ X if it is both l.h.c. and u.h.c.

The proof of the following theorem uses the fact that if the graph of thecorrespondence is closed, then the graph is close-valued. That is, for every x ∈ X,the set Γ(x) is a closed set.


Figure 2.1: Upper and lower hemi-continuity Mas-Colell et al. (1995).

Theorem 7. Let Γ be a nonempty-valued correspondence, and let A be the graphof Γ. Suppose that A is closed, and that for a bounded set X ⊆ X, the set Γ(X) isbounded. Then Γ is compact-valued and u.h.c.

Proof. For any x ∈ X we have that since A is closed, it follows that Γ(x) is closed aswell. Moreover, since Γ is compact-valued by hypothesis, the Heine-Borel theoremyields compactness when restricting X to subsets of IRk.

Let x ∈ X, and let {xn} ⊂ X be such that xn → x. Since Γ(xn) 6= ∅, chooseyn ∈ Γ(xn) ∀ n. Since xn → x, there is a bounded set X ⊆ X containing {xn} and x.By hypothesis, Γ(X) is bounded. Hence, {yn} ⊆ Γ(X) has a convergent subsequence,call it {yn}; let y be the limits point of the subsequence. Then, {(xn, yn)} is aconvergent subsequence in A converging to (x, y); since A is closed, it follows that(x, y) ∈ A. Hence y ∈ Γ(x), so Γ is u.h.c. at x. Since x was chosen arbitrarily, weobtain the result. ¡

The last theorem in this section establishes another relationship between acorrespondence and its graph. In this case, the convexity of the graph A of thecorrespondence Γ is used to show Γ is u.h.c. under a set of circumstances.

Theorem 8. Let Γ be a nonempty-valued correspondence, and let A be the graph ofΓ. Suppose that A is convex, and that for a bounded set X ⊆ X, there is a boundedset Y ⊆ Y 3 Γ(x) ∩ Y 6= ∅, ∀ x ∈ X. Then Γ is l.h.c. at every interior point of X .

Proof. Choose x ∈ X\X′; y ∈ Γ(x); and {xn} ⊂ X with xn → x. Let ε > 0 be suchthat X = Nε(x). Note that for some N ≥ 1, xn ∈ X, ∀ n; without loss of generalitywe take N = 1.


Let D denote the boundary of the set X. Every point xn has at least onerepresentation as a convex combination of x and a point in D. For each n, chooseαn ∈ [0, 1] and dn ∈ D such that

xn = αndn + (1− αn)x.

D is a bounded set and xn → x, so αn → 0. Choose Y such that Γ(x)∩ Y 6= ∅,∀ x ∈X. Then for each n, choose yn ∈ Γ(dn) ∩ Y, and define

yn = αnyn + (1− αn)y.

Since (dn, yn) ∈ A, ∀ n, (x, y) ∈ A, and A is convex, it follows that (xn, yn) ∈A, ∀ n. Moreover, since αn → 0 and all of the yn’s lie in the bounded set Y, itfollows that yn → y. Hence {(xn, yn)} lies in A and converges to (x, y). ¡

This chapter provided the reader with the concepts of upper and lower hemi-continuities for correspondences as well presenting some results that will prove valu-able in the next section. We have studied correspondences and different types ofcontinuities. The following section uses the concepts introduced so far to establishthe foundations of dynamic programming.

Chapter 3

Dynamic Programming

In this Chapter, we establish the foundations of dynamic programming. InSection 3.1 we first consider topics of analytical nature like that of a contractionmapping and a fixed point. These topics, as well as those developed in prior sec-tions, lead to the establishment of the Theorem of the Maximum in section 3.2. InSection 3.3 we define the sequence problem and the functional equation problem.We then establish the congruency of their solutions under the assumption that thereturn function F is bounded. Finally, Section 3.4 explores the classical variationalapproach to dynamic problems and established the sufficiency of the Euler equationstogether with the transversality condition.

3.1 Contraction Mappings and Fixed Points

This section is devoted to the study of fixed points. Although fixed points can beguaranteed for many function types, we focus on those obtained from a contractionmapping. The formal definition of a fixed point and contraction follow.

Definition 13. (Fixed Point) A function mapping T : X → X has a fixed point iffor some x ∈ X, T (x) = x. Then, x is called a fixed point of T .

Definition 14. (Contraction) Let (X, dX) be a metric space and T : X → X be afunction mapping of X into itself. T is a contraction mapping (with modulus β)if for some β ∈ (0, 1),

dX(T (x), T (y)) ≤ βdX(x, y), ∀ x, y ∈ X.

The following theorem relates the two concepts introduced above by estab-lishing the fact that contraction mappings always have fixed points. The theorem

15

Chapter 3 – Dynamic Programming Suarez Serrato – Page 16

below examines complete spaces. That is, spaces in which every Cauchy sequenceis a convergent sequence.

Theorem 9. (Contraction Mapping Theorem) If (X, dX) is a complete metric spaceand T : X→ X is a contraction with modulus β, then

1. T has exactly one fixed point x ∈ X, and

2. for any x0 ∈ X, dX(Tn(x0), x) ≤ βndX(x0, x), n = 1, 2, · · · ,

where we define the iterates of T, the mappings {T n} by T 0(x) = x, andT n = T (T n−1(x)), n = 1, 2, · · · , n.

Proof. 1. We first prove the existence of a fixed point. Choose v0 ∈ X, and define{vn}∞n=0 by vn+1 = T (vn), so that vn = T n(v). By the contraction mappingproperty of T,

dX(v2, v1) = dX(T (v1), T (v0)) ≤ βdX(v1, v0).

By induction and the triangle inequality, we have that for any m > n

dX(vm, vn) ≤ dX(vm, vm−1) + · · ·+ dX(vn+1, vn)

≤ [βm−1 + · · ·+ βn]dX(v1, v0)

= βn[βm−n−1 + · · ·+ β + 1]dX(v1, v0)

≤ βn

1− βdX(v1, v0),

where the fourth inequality holds from

βm−n−1 + · · ·+ β + 1 =m−n−1∑

n=1

βn ≤∞∑

n=1

βn =1

1− β.

From the equation above, we gather {vn}∞n=0 is a Cauchy sequence. Since X isa complete metric space, it follows that vn → v ∈ X.

We now show uniqueness of the fixed point by showing T (v) = v. Note firstthat ∀ n, v0 ∈ X,

dX(T (v), v) ≤ dX(T (v), T n(v0)) + dX(Tn(v0), v)

≤ βdX(v, T n−1(v0)) + dX(Tn(v0), v).

Since both terms above converge to zero, we have that dX(Tv, v) → 0 asn →∞.


Finally, assume for the sake of obtaining a contradiction that v 6= v is suchthat T (v) = v. Then,

0 < a = dX(v, v) = dX(T (v), T (v)) ≤ βdX(v, v) = βa,

which cannot hold since a > 0, β < 1. Thus, there is not another v such thatT (v) = v and we have the desired uniqueness property.

2. Observe that for n ≥ 1,

dX(Tn(v0), v) = dX[T (T n−1(v0)), Tv] ≤ βdX(T

n−1(v0), v),

so that the result follows by induction.

¡

For our purposes, contraction mappings are restricted to metric spaces withdesirable properties such as completeness. The following corollaries establish resultsabout contraction mappings defined over complete metric spaces.

Corollary 1. Let (X, dX) be a complete metric space and T : X→ X be a contractionmapping with fixed point x ∈ X. If X′ is a closed subset of X and T (X′) ⊆ X′, thenx ∈ X′. If in addition T (X′) ⊆ X′′ ⊆ X′, then x ∈ X′′.

Proof. Choose v0 ∈ X′ and note that {T n(v0)} is a sequence in X′ converging tov. Since X′ is a closed subset of a compact space X it is itself closed. Thus, wehave that v ∈ X′. If in addition we have that T (X′) ⊆ X′′, then it follows thatv = Tv ∈ X′′. ¡

Corollary 2. Let (X, dX) be a complete metric space and T : X → X, and supposethat for some N ∈ IN, TN : X→ X is a contraction mapping with modulus β. Then

1. T has exactly one fixed point in X , and

2. for any x0 ∈ X, dX(TkN(x0), x) ≤ βkdX(x0, x), k = 1, 2, · · · .

Proof. We show that a unique fixed point v of TN is also the unique fixed point ofT . We have that

dX(T (v), v) = dX[T (TN(v)), TN(v)] = dX[TN(T (v)), TN(v)] ≤ βdX(T (v), v).

Since β ∈ (0, 1), this implies that d(T (v), v) = 0 and v is a fixed point of T .Uniqueness follows since any fixed point of T is also a fixed point of TN . The prooffor part 2 is the same as that of Theorem 9 and is thus omitted. ¡


The theorem below establishes the main result of this section. We understandcontraction mappings as functions that have desirable properties, especially regard-ing fixed points. The following theorem establishes the conditions that are sufficientfor a function mapping to be a contraction.

Theorem 10. (Blackwell’s sufficient conditions for a contraction) Let X ⊆ IRk, andlet B(X) be the space of bounded functions f : X→ IR, with respect to the sup norm.Let T : B(X) → B(X) be an operator satisfying

1. (monotonicity) If f, g ∈ B(X) and f(x) ≤ g(x), ∀ x ∈ X, then (Tf)(x) ≤(Tg)(x), ∀x ∈ X.

2. (discounting) there exists some β ∈ (0, 1) such that

T (f(x) + a) ≤ (Tf)(x) + βa, ∀f ∈ B(X), a ≥ 0, x ∈ X.

Proof. If f(x) ≤ g(x), ∀ x ∈ X, we write f ≤ g. For any f, g ∈ B(X), f ≤g + ||f − g||. Then properties 1 and 2 imply

Tf ≤ T (g + ||f − g||) ≤ Tg + β||f − g||.

Reversing the roles of f, g gives the same logic. So Tg ≤ Tf +β||f − g||. Combiningthe two inequalities we have that, ||Tg − Tf || ≤ β||f − g||. ¡

3.2 Theorem of the Maximum

In this section, we apply the concepts of hemi-continuity of correspondencesand establish the result that allows the study dynamic programming. The followingtheorem was first proposed by Pontryagin et al. (1964) and establishes that, undera set of conditions, dynamic problems are well-defined.

Theorem 11. (Theorem of the Maximum) Let X ⊂ IRk and Y ⊂ IRl, let f : X×Y→IR be a continuous function, and let Γ : X→ Y be a compact-valued and continuouscorrespondence. Then, the function h : X→ IR defined as

h(x) = maxy∈Γ(x)

f(x, y)

is continuous, and the correspondence G : X→ Y defined as

G(x) = {y ∈ Γ(x) : f(x, y) = h(x)}

is nonempty, compact-valued, and u.h.c.


Proof. Let x ∈ X. The set Γ(x) is nonempty and compact, and f(x, ·) is continuous;hence it attains its maximum by Wierstrauss’s Theorem and the set of maximizersG(x) is nonempty. Moreover, since G(x) ⊆ Γ(x) and by the compactness of Γ(x)we have that G(x) is bounded. Suppose yn → y, and yn ∈ G(x), ∀ n. Since Γ(x)is closed, y ∈ Γ(x). Also, since h(x) = f(x, yn);∀ n, and f is continuous, it followsthat f(x, y) = h(x). Hence y ∈ G(x); so G(x) is closed. Thus, G(x) is nonemptyand compact for each x.

Next, we show that G(x) is u.h.c. Fix x, and let {xn} be any sequence converg-ing to x. Choose yn ∈ G(xn), ∀ n. Since Γ is u.h.c. , there exists a subsequence {yn}converging to y ∈ Γ(x). Let z ∈ Γ(x). Since Γ is u.h.c., there exists a subsequence{ynk

} converging to y ∈ Γ(x). Hence G is u.h.c.

Finally, we show that h is continuous. Fix x, and let {xn} be any sequence con-verging to x. Choose yn ∈ Γ(xn), ∀ n. Let h = lim sup h(xn) and h = lim inf h(xn).Then there exists a subsequence {xnk

} such that h = lim f(xnk, ynk

). But since Gis u.h.c., there exists a subsequence of {ynk

}, call it {y′j}, converging to y ∈ G(x).Hence h = lim f(xj, y

′j) = f(x, y) = h(x). An analogous result establishes that

h(x) = h. Hence {h(xn)} converges to its limit of h(x). ¡

The following two results study how the results of the Theorem of the Maxi-mum change when stronger constraints are imposed on f and Γ.

Lemma 1. Let X ⊆ IRk and Y ⊆ IRl. Assume that the correspondence Γ : X → Yis nonempty, compact, convex valued, and continuous, and let A be the graph Γ.Assume that the function f : A → IR is continuous and that f(x, ·) is strictlyconcave, for each x ∈ X. Define g : X→ Y by

g(x) = arg maxy∈Γ(x)

f(x, y).

Then for ε > 0and x ∈ X, there exists δx > 0 3

y ∈ Γ(x) and |f(x, g(x))− f(x, y)| < δx implies ||g(x)− y|| < ε.

If X is compact, then δ > 0 can be chosen independently of x.

Proof. Note that under the stated assumptions g is a well-defined, continuous, andsingle valued. We first prove the claim for the case where X is compact. For eachε > 0 , define

Aε = {(x, y) ∈ A : ||g(x)− y|| ≥ ε}.If Aε = ∅, ∀ ε > 0, then Γ is single valued and the result is trivial. Otherwise thereexists ε > 0 sufficiently small such that for all 0 < ε < ε, the set Aε is nonempty


and compact. For any such ε define,

δ = min(x,y)∈Aε

|f(x, g(x))− f(x, y)|.

Since the function being minimized is continuous and Aε is compact, the minimumis attained. Moreover, since [x, g(x)] /∈ Aε ∀ x ∈ X, it follows that δ > 0. Then,

y ∈ Γ(x) and ||g(x)− y|| ≥ ε implies |f(x, g(x))− f(x, y)| ≥ δ.

If X is not compact, the argument above can be applied separately for eachfixed x ∈ X. ¡

Theorem 12. Let X,Y, Γ, A be defined as in the Lemma above. Let {fn} be a se-quence of continuous functions on A; assume that for each n and each x ∈ X, fn(x, ·)is strictly concave in its second argument. Assume that f has the same propertiesand that fn → f uniformly (in the sup norm). Define the functions gn and g by

gn(x) = arg maxy∈Γ(x)

fn(x, y), n = 1, 2 · · · , and

g(x) = arg maxy∈Γ(x)

f(x, y).

Then, gn → g pointwise.

Proof. First note that since gn(x) is the unique maximizer of fn(x, ·) on Γ(x), andg(x) is the unique maximizer of f(x, ·) on Γ(x), it follows that

0 ≤ f(x, g(x))− f(x, gn(x))

≤ f(x, g(x))− fn(x, g(x)) + fn(x, gn(x))− f(x, gn(x))

≤ 2||f − fn||, ∀ x ∈ X.

Since fn → f uniformly, it follows immediately that for any δ > 0, there existsMδ ≥ 1 such that

0 ≤ f(x, g(x))− f(x, gn(x)) ≤ 2||f − fn|| < δ, ∀ x ∈ X, ∀ n ≥ Mδ.

To show that gn → g pointwise, we must establish that for each ε > 0 andx ∈ X, there exists Nx ≥ 1 such that

||g(x)− gn(x)|| < ε, ∀ n ≥ Nx.

By the lemma above, it suffices to show that for any δx > 0 and x ∈ X there exists


Nx ≥ 1 such that

|f(x, g(x))− f(x, gn(x))| < δx, ∀n ≥ Nx.

It now follows that any Nx ≥ Mδx has the required property. ¡

3.3 Dynamic Programming

The dynamic programs to be studied in this section and thereafter are repre-sented in two different manners. The sequence problem (SP) is

max{xt+1}∞t=0

∞∑t=0

βtF (xt, xt+1)

s.t. xt+1 ∈ Γ(xt), t = 0, 1, 2, · · · ,

x0 ∈ X given.

The alternative functional equation,(FE) , v is

v(x) = maxy∈Γ(x)

[F (x, y) + βv(y)], ∀ x ∈ X.

Our focus is to characterize the properties of F,X,Y, Γ, and A that are nec-essary and sufficient for the solution of SP to satisfy FE and conversely. This isdesirable since there exist methods to solve the FE that might elucidate solutionsto particular SPs. This result might seem surprising when one takes into accounthow each problem relates the states at each time period. The SP relates the statesin term of sequences. At every point in time the SP might be solved with a differ-ent policy. On the other hand, the FE examines the problem one stage at a time.Moreover, the choice in the FE follows the same policy every stage.

Definition 15. (Feasible Plans) A sequence {xt}∞t=0 in X is called a plan. Givenx0 ∈ X, define

Π(x0) = {{xt}∞t=0 : xt+1 ∈ Γ(xt), t = 0, 1, 2, · · · }

as the set of feasible plans from x0.

The definition above provides a framework to relate the solutions of the FEto that of the SP. An element x ∈ Π(x0) is an infinite sequence, so that Π(x0) isa collection of sequences as defined by Γ. We proceed by establishing assumptionsthat drive our results. In Chapter 4, we reconcile these assumptions with those


imposed by the neo-classical theory of economic growth on the economic agentsbeing modelled.

Assumption 1. Γ(x) 6= ∅, ∀ x ∈ X.

Assumption 2. For all x0 ∈ X and x ∈ Π(x0),

limn→∞

n∑t=0

βtF (xt, xt+1) ∈ IR∗.

The following definition relates the return obtained at each period through thereturn function F and the discount rate β into the concept of utility.

Definition 16. (Utility Function) For each n = 0, 1, · · · , define un : Π(x0) → IR by

un(x) =n∑

t=0

βtF (xt, xt+1),

so that un(x) is the partial sum of discounted returns through the nth period horizon.If assumption 2 holds, the infinite sum exists and we define

u(x) = limn→∞

un(x).

Definition 17. (Supremum Function) If assumptions 1 and 2 hold, then Π(x0) 6=∅∀x0 ∈ X, and F is well defined ∀ x ∈ Π(x0). We define the supremum function as

v∗(x0) = supx∈Π(x0)

u(x).

Being that v∗ is the supremum for the SP with a known x0, it follows that:

1. If |v∗(x0)| < ∞,v∗(x0) ≥ u(x)∀ x ∈ Π(x0), (3.1)

and for any ε > 0,

v∗(x0) ≤ u(x) + ε for some x ∈ Π(x0). (3.2)

2. If v∗(x0) = ∞, then there exists a sequence of feasible plans {xn} such that

limn→∞

u(xn) = ∞.


3. If v∗(x0) = −∞, then

u(x) = −∞, ∀ x ∈ Π(x0).

The goal of this section is to find connections between the functions v∗ andv. We have that if assumptions 1 and 2 hold, v∗ is defined uniquely. Theorem 14presents conditions guaranteeing v∗ = v. In order to arrive at this result we firstestablish that the unique solution of the SP problem, v∗, satisfies the FE. This isestablished if the following three conditions hold:

1. If |v∗(x0)| < ∞,

v∗(x0) ≥ F (x0, y) + βv∗(y),∀ y ∈ Γ(x0), (3.3)

and for any ε > 0,

v∗(x0) ≤ F (x0, y) + βv∗(y) + ε for some y ∈ Γ(x0). (3.4)

2. If v∗(x0) = ∞, then there exists a sequence {yn} ⊆ Γ(x0) such that

limn→∞

[F (x0, y) + βv∗(yn)] = ∞. (3.5)

3. If v∗(x0) = −∞, then

F (x0, y) + βv∗(yk) = −∞ ∀y ∈ Γ(x0). (3.6)

The properties of v∗ that follow as a supremum-valued function and thoseoutlined above have a striking resemblance. In fact, the only difference is that theconditions to be shown consider a single period and postpone the evaluation of theremaining states by relying on v∗ starting from the next period. The followingLemma is useful in establishing the fact that the solution to the SP satisfies the FE.

Lemma 2. Let X, Γ, F, and β satisfy Assumption 2. Then, for any x0 ∈ X and any(x0, x1, x2, · · · ) = x ∈ Π(x0),

u(x) = F (x0, x1) + u(x′)

where x′ = (x1, x2, · · · ).


Proof. Under Assumption 2, for any x0 ∈ X and any x ∈ Π(x0),

u(x) = limn→∞

n∑t=0

βtF (xt, xt+1)

= F (x0, x1) + β limn→∞

n∑t=0

βtF (xt+1, xt+2)

= F (x0, x1) + βu(x).

¡

Theorem 13. Let X, Γ, F, and β satisfy Assumptions 1 and 2. Then v∗ satisfies(FE).

Proof. If β = 0, the result is trivial. Suppose that β > 0 and choose x0 ∈ X. Supposev∗(x0) is finite. Then (3.1) and (3.2) above hold. Let x1 ∈ Γ(x0) and ε > 0. By(3.2), there exists x′ = (x1, x2, · · · ) ∈ Π(x1) such that u(x′) ≥ v∗(x1)− ε. Note thatx ∈ Π(x0). From the 3.1 and the Lemma above, we have that

v∗(x0) ≥ U(x) = F (x0, x1) + βu(x′) ≥ F (x0, x1) + βv∗(x1)− βε.

Since ε was chosen arbitrarily, condition (3.3) is established.

Now let X0 ∈ X, and ε > 0. From (3.2) and the Lemma above, we can choosex = (x0, x1, · · · ) ∈ Π(x0), so that

v∗(x0) ≤ u(x) + ε = F (x0, x1) + βu(x′) + ε,

where x′ = (x1, x2, · · · ). It then follows from (3.1) that

v∗(x0) ≤ F (x0) + βv∗(x1) + ε,

since x1 ∈ Γ(x0), this established (3.4).

If v∗(x0) = +∞, then there exists a sequence there exists a sequence {xn} ⊆Π(x0) such that lim

n→∞u(xn) = +∞. Since x1,n ∈ Γ(x0), ∀n, and

u(xn) = F (x0, x1,n) + βu(x′n) ≤ F (x0, x1,n) + βv∗(x1,n), ∀ n,

it follows that (3.5) is established.

If v∗(x0) = −∞, then

u(x) = F (x0, x1) + βu(x′) = −∞,∀ (x0, x1, · · · ) = x ∈ Π(x0),


where x′ = (x1, x2, · · · ). Since F is real-valued, it follows that

u(x′) = −∞,∀ x1 ∈ Γ(x0),∀ x′ ∈ Π(x0).

Hence v∗(x0) = −∞, ∀ x1 ∈ Γ(x0). Since F is real-valued and β > 0, thecondition in (3.6) follows immediately. ¡

Insight regarding the result of the previous theorem is exposed when consid-ering the sets of solution of the SP and the set of solutions of the FE. The previoustheorem sates that the set of solutions to the FE contains the set of solutions of theSP. The following theorem provides a partial converse. That is, under an additionalrestriction, the set of solutions to the SP contains the set of solutions to the FE.

Theorem 14. Let X, Γ, F, and β satisfy Assumptions 1 and 2. If v is a solution to(FE) and satisfies

limn→∞

βnv(xn) = 0,∀ x ∈ Π(x0),∀ x0 ∈ X, (3.7)

then v = v∗.

Proof. If v(x0) is finite, then condition (3.3) and (3.4) hold. We show this implies(3.1) and (3.2) also hold. From (3.3), we have that for all x ∈ Π(x0),

v(x0) ≥ F (x0, x1) + βv(x1)

≥ F (x0, x1) + F (x1, x2) + β2v(x2)...

≥ un(x) + βn+1v(nn+1), n = 1, 2, · · ·

Taking the limit at ε →∞ and using (3.7) above, we get that (3.1) holds.

Next, fix ε > 0 and choose {δt}∞t=1 ⊂ IR+ such that∞∑

t=1

βt−1δt ≤ ε2. Since (3.2)

holds, choose x1 ∈ Γ(x0), x2 ∈ Γ(x1), · · · so that

v(xt) ≤ F (xt, xt1) + βv(xt+1) + δt+1, t = 1, 2, · · ·


Then x ∈ Π(x0), and

v(x0) ≤n∑

t=0

βtF (xt, xt+1) + βn+1v(xn+1) + (δ1 + · · ·+ βnδn+1)

≤ un(x) + βn+1v(xn+1) +ε

2, n = 1, 2, · · ·

Therefore, (3.7) implies that for n sufficiently large, v(x0) ≤ un(x) + ε. Sinceε > 0 was chosen arbitrarily , we have that (3.2) holds.

If (3.7) holds, then (3.6) implies that v cannot take the value −∞. If v(x0) =+∞, choose n ≥ 0 and (x0, x1, · · · ) such that xt ∈ Γ(xt−1) and v(xt) = +∞ fort = 1, 2, · · · , n, and v(xn+1) ≤ +∞ for all xn+1 ∈ Γ(xn). Clearly (3.7) impliesthat n is finite. Fix A > 0. Since v(xn) = +∞, (3.5) implies that we can choosexn+1,A ∈ Γ(xn) such that

F (xn, xn+1,A) + βv(xn+1,A) ≥ β−n

[A + 1−

n−1∑t=0

βtF (xt, xt+1)

].

Then choose xn+1,A ∈ Π(xn+1,A) such that u(xn+1,A) ≥ v(xn+1,A) − β−(n+1). Sincev(xn+1,A) is finite, the argument above shows that this is possible. Then xA =(x0, x1, · · · , xn, xn+1,A) ∈ Π(x0) and

u(xA) =n−1∑t=0

βtF (xt, xt+1) + βnF (xn, xn+1,A) + βn+1u(xn+1,A) ≥ A.

Since A > 0 was chosen arbitrarily, it follows that v∗(x0) = +∞. ¡

Theorem 15 relates the solutions to FE to those of SP by combining the ap-proach in both problems. The theorem statement essentially prolongs the use of theFE approach one period and uses the SP to maximize the first period.

Theorem 15. Let X, Γ, F, and β satisfy assumptions 1 and 2. Let x∗ ∈ Π(x0) be 3it attains the supremum in the SP for a given x0 ∈ X. Then,

v∗(x∗t ) = F (x∗t , x∗t+1) + βv∗t+1, t = 0, 1, 2, · · · . (3.8)

Proof. Since x∗ attains the supremum,

v∗(x∗t ) = u(x∗) = F (x0, x∗1) + βu(x∗′) (3.9)

≥ u(x) = F (x0, x1) + βu(x′), ∀ x ∈ Π(x0).


In particular, the inequality holds for all plans with xt = x∗t . Since (x∗1, x2, x3, · · · ) ∈Π(x∗t ) implies that (x0, x

∗1, x2, x3, · · · ) ∈ Π(x0), it follows that

u(x∗′) ≥ u(x′), ∀x ∈ Π(x∗1).

hence u(x∗′) = v(x∗1). Substituting this into (3.9), gives (3.8) for x0. Continuing byinduction establishes (3.8) for all t. ¡

The following theorem establishes a boundedness condition on the sequence xt

and shows that any sequence satisfying this condition is an optimal plan.

Theorem 16. Let X, Γ, F, and β satisfy Assumptions 1 and 2. Let x∗ ∈ Π(x0) be3 satisfies equation (3.8), and with

lim supt→∞

βtv∗(x∗t ) ≤ 0. (3.10)

Then x∗ attains the supremum in SP for a given x0.

Proof. Suppose that x∗ ∈ Π(x0) satisfies (3.8) and (3.10). Then, it follows byinduction on (3.8) that

v∗(x0) = un(x∗) + βn+1v∗(x∗n+1), n = 1, 2, · · ·

Then, using (3.10), we find that v∗(x0) ≤ u(x∗). Since x∗ ∈ Π(x0), the reverseinequality holds, establishing the result . ¡

As was mentioned above, the FE problems weighs the choice at every periodusing the same policy. In order to characterize this more formally, we introduce theconcept of a policy correspondence. We study a special class of policy correspon-dences, that is those that are members of the set C(X) of bounded and continuousfunctions.

Definition 18. (Policy Correspondence) Given a solution v ∈ C(X) to

v(x) = maxy∈Γ(x)

[F (x, y) + βv(y)],

we can define the policy correspondence G : X→ X by

G(x) = {y ∈ Γ(x) : v(x) = F (x, y) + βv(y)}.

Our focus now shifts to the study of policy correspondences. It is desirableto find conditions under which the policy correspondences have the properties nec-essary for the Theorem of the Maximum to describe the optimal solutions. Since


policy correspondences are defined in terms of the return function F and the corre-spondence Γ, G obtains the desired properties by placing the following restrictionson these.

Assumption 3. X is a convex subset of IRk, and the correspondence Γ : X→ X isnonempty, compact-valued, and continuous.

Assumption 4. The real valued function of the graph of f , F : A → IR, is boundedand continuous, and 0 < β < 1.

It is of importance to note the relationship between the return function andthe discount rate β. Although these are defined independently, their use in Theorem17 further assumes they satisfy (3.7). In that context, the sequence βn is requiredto decrease at a sufficient pace to fulfill the condition established. The claim thatAssumptions 3 and 4 modify the policy correspondence G in exactly the appropriateway is substantiated in Theorem 17.

Theorem 17. Let X, Γ, F, and β satisfy Assumptions 3 and 4, and let C(X) be thespace of bounded continuous functions f : X → IR, with the sup norm. Then theunique operator T maps C(X) into itself, T : C(X) → C(X); T has a unique fixedpoint v ∈ C(X); and for all v0 ∈ C(X),

||T n(v0)− v|| ≤ βn||v0 − v||, n = 0, 1, 2, · · · .

Moreover, given v, the optimal policy correspondence G : X → X defined in 11 iscompact-valued and u.h.c.

Proof. Under Assumption 3 and 4, for each f ∈ C(X) and x ∈ X, the problem in(3.2) is to maximize the continuous function [F (x, ·) + βf(·)] over the compact setΓ(x). Hence the maximum is attained. Since both F , and f are bounded, clearlyTf is also bounded; and since F and f are continuous, and Γ is compact-valued andcontinuous, it follows from the Theorem of the Maximum that Tf is continuous.Hence T : C(X) → C(X).

It is them immediate that T satisfies the condition’s of Blackwell’s sufficiencyconditions for a contraction. Since C(X) is a Banach space, it follows from thecontraction mapping theorem, that T has a unique fixed point v ∈ C(X), andso (3.3) holds. The stated properties of G then follow from the Theorem of theMaximum. ¡

In order to characterize the value function v and the policy correspondenceG, we require further assumptions on the behavior of Γ and F . The followingassumptions establish properties of F and Γ that allow us to use previous resultsabout fixed points and contraction mappings.


Assumption 5. For each y, F (·, y) is strictly increasing in each of its first k argu-ments.

Assumption 6. Γ is monotone in the sense that x ≤ x′ implies Γ(x) ⊆ Γ(x′).

To prove Theorem 18, we seek to show that Γ is u.h.c.. Assumption 6 providesthe required property to show this. If one imagines a sequence {xn} converging toits limit point, it follows that the sequence {Γ(xn)} is nested. This imagery helpsvisualize the over approximation of the target set as required by u.h.c. correspon-dences. Assumption 5 alone does not provide the required information about thebehavior of F . However, in combination with Assumptions 4 and 2, we have enoughinformation to present the following theorem.

Theorem 18. Let X, Γ, F, and β satisfy Assumptions 3–6, and let v satisfy

v(x) = maxy∈Γ(x)

[F (x, y) + βv(y)].

Then v is strictly increasing.

Proof. Let C ′(X) ⊂ C(X) be the set of bounded, continuous, and nondecreasingfunctions on X , and let C ′′(X) ⊂ C ′(X) be the set of strictly increasing functions.Since C ′(X) is a closed subset of the complete metric space C(X), it then followsthat if T [C ′(X)] ⊆ T [C ′′(X)]. Assumptions 5 and 6 ensure this is so. ¡

Assumption 7. F is strictly concave; that is,

F [θ(x, y) + (1− θ)(x′, y′)] ≥ θF (x, y) + (1− θ)F (x′, y′),

∀ (x, y), (x′, y′) ∈ A, and all θ ∈ (0, 1), and the inequality is strict if x 6= x′.

Assumption 8. Γ is convex in the sense that for any 0 ≤ θ ≤ 1, and x, x′ ∈ X,

y ∈ Γ(x) and y′ ∈ Γ(x′) implies

θy + (1− θ)y′ ∈ Γ[θx + (1− θ)x′].

It is worth noting that the convexity assumption stated in Assumption 8 isestablished within the same period. That is, it is assumed that x and x′ belong to agiven Γ(x) in the same time period. Convexity is not established inter-temporally.


Theorem 19. Let X, Γ, F and β satisfy Assumptions 3–4 and 7–8; let v satisfy

v(x) = maxy∈Γ(x)

[F (x, y) + βv(y)],

and let G satisfy

G(x) = {y ∈ Γ(x) : v(x) = F (x, y) + βv(y)}. (3.11)

Then, v is strictly concave and G is a continuous, single-valued function.

Proof. Let C ′(X) ⊂ C(X) be the set of bounded, continuous, weakly concave func-tions on X , and let C ′′(X) ⊂ C ′(X) be the set of strictly concave functions. SinceC ′(X) is a closed subset of a complete metric space C(X), by Theorem 17 andCorollary 1 to the Contraction Mapping Theorem, it is sufficient to show thatT [C ′(X)] ⊆ C ′′(X).

To verify that this is so, let f ∈ C ′(X) and let

x0 6= x1, θ ∈ (0, 1), and xθ = θx0 + (1− θ)x1.

Let yi ∈ Γ(xi) attain (Tf)xi, for i = 0, 1. It follows that

(Tf) ≥ F (xθ, yθ) + βf(yθ)

≥ θ[F (x0, y0) + βf(y0)] + (1− θ)F (x1, y1) + βf(y1)

= θ(Tf)x0 + (1− θ)(Tf)x1,

where the first line uses (3.2) and the fact that yθ ∈ Γ(xθ); the second uses thehypothesis that f is concave and the concavity restriction of F in Assumption 7; andthe last follows from the way y0, y1 were selected. Since x0 and x1 were arbitrary,it follows that Tf is strictly concave, and since f was arbitrary, it follows thatT [C ′(X)] ⊆ C ′′(X).

Hence, the unique fixed point v is strictly concave. Since F is also concave(Assumption 7) and, for each x ∈ X, Γ(x) is concave (Assumption 8), it follows thatthe maximum in (3.2) is attained. ¡

Theorems 19 and 20 characterize v by using the fact that the operator Tpreserves certain properties (Stokey et al., 1989). In many cases, it is difficult to solvefor the policy correspondences. In these cases, it is desirable to use approximationsof the policy function instead. Theorem 20 outlines the conditions under which thisis possible.


Theorem 20. (Convergence of the Policy Functions) Let X, Γ, F, and β satisfyAssumptions 3–4 and 7-8, and let v and g satisfy

v(x) = maxy∈Γ(x)

[F (x, y) + βv(y)], and g(x) = {y ∈ Γ(x) : v(x) = F (x, y) + βv(y)}.

Let C ′(X) be the set of bounded, continuous, concave functions f : X → IR, and letv0 ∈ C ′(X). Let {(vn, gn)} be defined by

vn+1 = Tvn, n = 0, 1, 2, · · · , and

gn(x) = arg maxy∈Γ(x)

[F (x, y) + βvn(y)], n = 0, 1, 2, · · ·

Then gn → g pointwise. If X is compact, then the convergence is uniform.

Proof. Let C ′′(X) ⊂ C ′(X) be the set of strictly concave functions f : X → IR. Asshown in Theorem 19, v ∈ C ′′(X). Moreover, as shown in the proof of that Theorem,T [C ′(X)] ⊂ C ′′(X). Since v0 ∈ C ′(X), it follows that every function vn, n = 1, 2, · · · ,is strictly concave. Define the functions {fn} and f by

fn(x, y) = F (x, y) + βvn(y), n = 1, 2, · · · , and

f(x, y) = F (x, y) + βv(y).

Since F satisfies Assumption 7, it follows that f and each function fn, n =1, 2, · · · , is strictly concave. Hence Theorem 19 applies and the desired results areproved. ¡

Assumption 9. F is differentiable on the interior of A.

It is a logical consequence of Assumption 9 that the function v in the FE isdifferentiable. The proof of this result assumes knowledge of results in the fieldof convex analysis that are considered to be overly esoteric for the target audience,and is thus omitted. Readers interested in a formal development of this result wouldbenefit from the treatment of sub-gradients in Rockafellar (1970) and the statementof the result in Stokey et al. (1989). In the final section of this chapter we rely onthis result to establish the classical variational result first introduced by Euler andthen proven sound by Lagrange (McShane, 1989).


3.4 Euler Equations

In this section, we present a solution mechanism to recursive problems. The cal-culus of variations approach to dynamic problems such as the brachiostochrone andisoparametric problems motivated mathematicians to establish the results presentedin this section. The isoparametric problem, first solved by Lagrange, provided theinsight to establish the sufficiency of the first-order conditions proposed by Euler(Ferguson, 2004). Although, the version of the results presented here follow the SPapproach to dynamic problems, the same results may be obtained following the FEapproach.

Definition 19. (Euler Equations) Let F satisfy assumptions 3-5,7, and 9; let Fx

denote the l-vector of partial derivatives (F1, · · · , Fl)in its first l arguments, Fy de-note the vector (Fl+1, · · · , F2l). Since F is continuously differentiable and strictlyconcave, if x∗t+1 is in the interior of Γ(x∗t ) for all t, the first-order conditions for

maxy

[F (x∗t , y) + βF (y, x∗t+2)]

s.t. y ∈ Γ(x∗t ) and x∗t+2 ∈ Γ(y)

are0 = Fy(x

∗t , x

∗t+1) + βFx(x

∗t+1, x

∗t+2), t = 0, 1, 2, · · · .

The Euler equations described above comprise a “system of l second-orderdifferential equations in the vector of state variables xt” Stokey et al. (1989). Inorder to solve for the optimal solution, we require an l-vector of boundary conditions.The condition we refer to as the transversality condition provides these conditions.

Definition 20. (Transversality Condition) The transversality condition states that

limt→∞

βtFx(xt∗, x∗t+1) · x∗t = 0.

We are now ready to establish the final result of this guide, that Definitions19 and 20 are sufficient conditions to obtain the maximizing sequence of the SP.

Theorem 21. (Sufficiency of the Euler and transversality conditions) Let X ⊆ IRl+,

and let F satisfy Assumptions 3-5,7, and 9. Then the sequence {x∗t+1}∞t=0 with x∗t+1 ∈Γ(x∗t )\Γ(x∗t )

′, t = 0, 1, 2, · · · , is optimal for the SP problem, given x0, if it satisfiesthe Euler and transversality conditions stated in Definitions 19 and 20.

Proof. Let x0 be given; let {x∗t} ∈ Π(x0) satisfy the conditions imposed in Definitions19 and 20; and let {xt} ∈ Π(x0) be any feasible sequence. It is sufficient to show


that the difference, call it D, between the objective function in SP evaluated at {x∗t}and at {xt} is nonnegative.

Since F is continuous, concave, and differentiable as according to Assumptions4, 7 and 9,

D = limT→∞

T∑t=0

βt[F (x∗t , x∗t+1) + F (xt, xt+1)]

≥ limT→∞

T∑t=0

βt[Fy(x∗t , x

∗t+1) · (x∗t − xt)− Fx(x

∗t , x

∗t+1) · (x∗t+1 − xt+1)],

where the second inequality follows from the fact that the linear approximation of aconcave function lies above the function itself. By the definition of Π(x0), we havethat x∗0 − x0 = 0. Then, rearranging terms gives

D ≥ limT→∞

{T−1∑t=0

βt[Fy(x∗t , x

∗t+1) + βFx(x

∗t+1, x

∗t+2)] · (x∗t+1 − xt+1)

+ βT Fy(x∗T , x∗T+1) · (x∗T − xT )

}.

Since {x∗t} satisfies Definition 19, the summation of the first T − 1 terms is zero.Therefore, substituting the equation in Definition 19 into the last term as well andusing Definition 20 yields

D ≥ − limT→∞

βT Fx(x∗T , x∗T+1) · (x∗T − xT )

≥ − limT→∞

βT Fx(x∗T , x∗T+1) · x∗T ,

where the last line uses the fact that Fx ≥ 0 by Assumption 4, and xt ≥ 0, ∀ t. Itthen follows from Definition 20 that D ≥ 0, establishing the desired result. ¡

Definitions 19 and 20 establish a straightforward procedure to solve recursiveproblems. Moreover, Theorem 21 proves their sufficiency so that the optimal se-quence is obtained. The objective in Chapter 3 is fulfilled with the development ofthe rigorous set of results that prove the existence of a maximum, relate the solu-tions of the SP and the FE, and outline a solution mechanism. Chapter 4 considersthe application of dynamic programming to the problem of Economic Growth andprovides economic intuition behind the restrictions placed on Γ, F,X,Y and v inChapter 3.

Chapter 4

Deterministic Economic Growth

In the final chapter of this guide we seek to elucidate some intuition aboutthe results of the previous chapter. It is easy to get lost in the technicality of thefield and forget that we are developing these methods not only for the sake of theirbeauty but also to apply them to economic theory. In this chapter, we presentthe one-sector model of optimal economic growth. The SP problem is presented inan abstract manner to indicate that any function satisfying the conditions outlinedcould be modelled in this way.

The neo-classical theory of economic growth associates behavioral assumptionsof economic agents, that is households and firms, with many of the properties wegive the utility and production functions in this example. We owe the formulationof this problem to Stokey et al. (1989), and the solutions to the author of this guide.

Consider the abstract notion of an economy with one good at every time period,the amount of which is yt. The good is produced with the technology describedby the gross production function G(kt, nt) = yt, where kt represents capital and nt

represents labor at time t. Neo-classical growth theory has that the gross productionfunction G has the following properties. First, since it relates current inputs tooutputs it is defined over the graph of the correspondence Γ. That is, G : A →IR+. Moreover, G is continuously differentiable, strictly increasing, homogeneous ofdegree one, and strictly quasi-concave. A function is homogenous of degree one ifG(tk, tn) = t1G(k, n). Moreover, a function is quasi-concave if the upper contour setof the function is concave. Further, G has the properties that

G(0, n) = 0, Gk(k, n) > 0, Gn(k, n) > 0, ∀ k, n > 0

limk→0

Gk(k, 1) = ∞, limk→∞

Gk(k, 1) = 0,

where Gk represents the partial derivative of the function with respect to its argu-

34

Chapter 4 – Deterministic Economic Growth Suarez Serrato – Page 35

ment k. The two limits above imply conditions on the marginal rate of productivity.

At each period, households face the choice of allocating the output betweeninvestment, denoted it, and consumption, denoted ct, so that we may relate

G(kt, nt) = yt ≥ ct + it. (4.1)

The budget constraint in equation (4.1) states that a household cannot con-sume or invest more of the good than what is available at a given time period.Capital is assumed to depreciate at rate δ ∈ [0, 1] every period. So that at everytime period the following equation must hold

kt+1 = (1− δ)kt + it. (4.2)

Combining the expressions in equations 4.1 and 4.2, we get that

G(kt, nt) ≥ ct + kt+1 − (1− δ)kt. (4.3)

In this model, we do not consider the household’s choice of providing laborand assume nt = 1 ∀ t. In order to express production in terms of net depreciation,and given that nt has been normalized to unity, we define

g(kt) = G(kt, 1) + (1− δ)kt, (4.4)

and note that g(kt) ≥ ct +kt+1. We assume non-satiability, which is defined to meanthat we get equality in the previous equation. We now study the properties of thefunction g(kt). From the definition of g as the sum of two continuously differentiableand strictly increasing functions we have that it inherits these properties. We alsohave that the sum of a strictly concave function and a linear function maintains theproperty of strict concavity. Moreover, we have that

g(0) = G(0, 1) = 0,

g′(k) = Gk(k, 1) + (1− δ) > 0,

limk→0

g′(k) = limk→0

[Gk(k, 1) + (1− δ)] = ∞, and

limk→∞

g′(k) = limk→∞

[Gk(k, 1) + (1− δ)] = (1− δ),

so that it preserves the properties of G modifying the last one by a horizontal shift.

The household’s role is to find a sequence of consumptions that maximizes its


utility. That is, it faces the SP described by

max{ct}∞t=0

∞∑t=0

βtU(ct), (4.5)

where β ∈ (0, 1) is the rate of time preference or discounting factor, and U isthe utility function of the household. The assumptions placed on U are that itis a bounded function from IR+ into IR that is continuously differentiable, strictlyincreasing, strictly concave, and lim

c→0U ′(c) = ∞.

It is implicit in the formulation of this problem that household utility is ad-ditively separable through time and is time-invariant. That is, the utility functiondoes not express an intrinsic trade off between consumption between periods. Thistrade-off is introduced through the state variable of capital. The utility function istime invariant since U(ct) = U(ct+1) if and only if ct = ct+1. These are standardassumptions in the literature of neo-classical economic theory, however, they shouldbe kept in mind when interpreting the results.

Using our previous derivation of the net production function we recast themaximization problem in terms of choosing a sequence of capital that maximizesthe household’s utility in this way:

max{kt+1}∞t=0

∞∑t=0

βtU(g(kt)− kt+1)

s.t. kt+1 ∈ Γ(kt), t = 0, 1, 2, · · · ,

k0 ∈ X given.

We proceed to show that the assumptions made in Chapter 3 to develop thedesired results follow from the behavioral assumptions imposed by neo-classical eco-nomics on the agents. That is, we wish to show that the functional restrictionsimposed in the previous Chapter are consistent with economic theory and that themethods of dynamic programming developed so far are a result of the need to modeleconomic behavior. Before we proceed, we summarize for conciseness the propertiesof the utility function U and the net production function g.

U1. 0 < β < 1. T1. g is continuous.U2. U is bounded. T2. g(0) = 0.U3. U is strictly increasing. T3. g is strictly increasing.U4. U is strictly concave. T4. g is weakly concave.U5. U is continuously differentiable. T5. g is continuously differentiable.

We now proceed to evaluate the set of assumptions made in previous chapters.


Henceforth, we refer to the function F as the inter-temporal function used in pre-vious chapters. In this example, the function F (kt, kt+1) = U(g(kt)− kt+1) and thecorrespondence Γ(kt) = [0, g(kt−1)].

Assumption 1. Γ(k) 6= ∅,∀ k ∈ X.Since Γ(kt) was defined to be Γ(kt) = [0, g(kt−1)], and g(0) = 0 we have that0 ∈ Γ(kt) ∀ t and hence Γ(kt) is always nonempty.

Assumption 2. For all k0 ∈ X and k ∈ Π(k0),

limn→∞

n∑t=0

βtU [g(kt)− kt+1] ∈ IR∗.

Since U is bounded, we have that ∃ M ∈ IN 3 |U(·)| ≤ M .

So, limn→∞

n∑t=0

βtU [g(kt)− kt+1] ≤ M limn→∞

n∑t=0

βt = M1−β

.

Assumption 3. X is a convex subset of IRk, and the correspondence Γ : X→ X isnonempty, compact-valued, and continuous.

Assumption 4. The function U [g(kt)− kt+1] : A → IR is bounded and continuous,and 0 < β < 1.We showed U is bounded in Assumption 2. We have that both U and g(kt)−kt+1 arecontinuous functions. Since the composition of continuous functions is continuousas well we have the second result. The third follows from hypothesis.

Assumption 5. For each kt+1, U [g(kt)− kt+1] is strictly increasing in kt and kt+1.From T3 and U3 we have that the function is strictly increasing.

Assumption 6. Γ is monotone in the sense that k ≤ k′ implies Γ(k) ⊆ Γ(k′).It follows from Γ(k) = [0, g(k)], that k ≤ k′ implies

Γ(k) = [0, g(k)] ⊆ [0, g(k′)] = Γ(k′).

Assumption 7. G is strictly concave; that is,

G[θ(kt, kt−1) + (1− θ)(k′t, k′t+1)] ≥ θG(kt, kt+1) + (1− θ)G(k′t, k

′t+1),

∀ (kt, kt+1), (k′t, k

′t+1) ∈ A, and all θ ∈ (0, 1), and the inequality is strict if kt 6= k′t.

This property is a direct consequence of U4.

Assumption 8. Γ is convex. This property follow from the characterization of Γin term of intervals in this problem.


The application of dynamic programming to economic modelling has beenshown to be consistent with the assumptions of economic theory. The problempresented in this chapter has been shown to suffice the conditions developed inChapter 3, and is thus well defined. Using Assumption 9 and the methods presentedin the last section of Chapter 3, one can solve the problem for specific functionalcharacterizations of utility and production.

As was mentioned, several extensions including stability dynamics, stochasticmodels, and numerical solutions make the application of dynamic programminguseful to problems in several disciplines. We hope the reader might continue thestudy of this field as much as the author has enjoyed this project and will continuethe study of this field in the Economics Ph.D. program at the University of Californiaat Berkeley.

Appendix A

Notation Index

¡ quod erat demonstrandum(QED)x ∈ X membershipX ⊂ Y,X ⊆ Y strict subset, subset⋂

,⋃

intersection, union∅ empty set\ set deletionXc complementX′ derived setX closureX× Y cartesian productIN natural numbersIR, IR∗ real numbers, extended real numbersIRk kth dimension Euclidean spaceIRk

+ non-negative subspace of IRk

IRk++ positive subspace of IRk

dX(x, y) distance function in the metric space (X, d)(X, dX) metric spaceNδ(p) δ-neighborhood around the point p| · | absolute value|| · || normB(X) space of bounded functions on XC(X) space of bounded and continuous functions on Xfn(X) iterates of the function f on XT n(B(X)) iterates of the function T on the function space B(X)Π(x0) the set of feasible plans from x0

39

Bibliography

Aghion, P. and R. Griffith (2005). Competition and Growth : Reconciling Theoryand Evidence. The MIT Press.

Barro, R. and X. Sala-i-Martin (2003). Economic Growth (Second ed.). Boston,MA: The MIT Press.

Bellman, R. (1957). Dynamic Programming. Princeton, New Jersey: PrincetonUniversity Press.

Bryant, V. (1985). Metric Spaces. Cambridge, UK: Cambridge University Press.

Chiang, A. C. (1984). Fundamental Methods of Mathematical Economics (Thirded.). McGraw-Hill.

Diaz-Gimenez, J. (2001). Computational Methods for the Study of DynamicEconomies, Chapter 1–Linear quadratic approximations: An introduction, pp.13–30. USA: Oxford University Press.

Ferguson, J. (2004). A brief survey of the history of the calculus of variations and itsapplications. University of Victoria HO/0402357, Arxiv Preprint Math. availableat http://arxiv.org/pdf/math/0402357.

Fischer, S. and O. Blanchard (1989). Lectures on Macroeconomics. Boston, MA:The MIT Press.

Fomin, S. V. and I. M. Gelfand (2000). Calculus of Variations. Dover Publications.

Goldstein, R. (2005). Incompleteness: the proof and paradox of Kurt Godel. NewYork: Norton & Company.

Holder, A. (2005). An Introduction to Mathematical Optimization (First ed.). SanAntonio, TX, 2005.

Hoy, M., J. Livernois, C. McKenna, R. Rees, and T. Stengos (2001). Mathematicsfor Economics (Second ed.). Boston, MA: The MIT Press.

40

Bibliography Suarez Serrato – Page 41

Lewis Carroll (1999). The Annotated Alice: The Definitive Edition. W. W. Norton& Company.

Ljungqvist, L. and T. Sargent (2004). Recursive Macroeconomic Theory (Seconded.). Boston, MA: The MIT Press.

Mas-Colell, A., M. D. Whinston, and J. R. Green (1995). Microeconomic Theory.USA: Oxford University Press.

McShane, E. (1989, September). The calculus of variations from the begin-ning through optimal control theory. SIAM Journal on Control and Optimiza-tion 27 (5), 915–940.

Niehans, J. (1994). A History of Economic Theory. Baltimore, MA: The JohnsHopkins University Press.

Nobel, F. (2004). The bank of sweden prize in economicsciences in memory of alfred nobel 2004. [web page]http://nobelprize.org/economics/laureates/2004/index.html. [Accessedon April 26, 2006].

Pontryagin, L. S. et al. (1964). The Mathematical Theory of Optimal Processes. NewYork: The Macmillan Company.

Rockafellar, T. (1970). Convex analysis. Princeton, N.J.: Princeton UniversityPress.

Romer, D. (2002, June). It’s fourth down and what does the bellman equa-tion say? a dynamic programming analysis of football strategy. NBER Work-ing Papers 9024, National Bureau of Economic Research, Inc. available athttp://ideas.repec.org/p/nbr/nberwo/9024.html.

Royden, H. L. (1988). Real Analysis (Third ed.). Princeton, New Jersey: PrenticeHall.

Rudin, W. (1976). Principles of Mathematical Analysis (Third ed.). McGraw-Hill.

Sala-i-Martin, X. (2000). Apuntes de Crecimiento Economico (Second ed.).Barcelona: Antoni Bosch.

Shone, R. (2002). Economic Dynamics : Phase Diagrams and their Economic Ap-plication (Second ed.). Cambridge, UK: Cambridge University Press.

Spiegel, H. W. (1983). The Growth of Economic Tought (Third ed.). Durham, NC:Duke University Press.

Bibliography Suarez Serrato – Page 42

Stokey, N., R. Lucas, and E. Prescott (1989). Recursive Methods in EconomicDynamics. Boston, MA: Harvard University Press.

Vohra, R. (2005). Advanced Mathematical Economics. New York: Routledge.

Yong, J. and X. Y. Zhou (1999). Stochastic Controls : Hamiltonian Systems andHJB Equations. Springer.

Young, L. C. (2000). Lecture on the Calculus of Variations and Optimal ControlTheory. Providence, Rhode Island: AMSChelsea Publishing.

dynamic programming and its applications to economic...

Documents