finite horizon lqg density regulator with wasserstein ... · finite horizon density regulator:...

Finite Horizon LQG Density Regulatorwith Wasserstein Terminal Cost

Abhishek Halder

Department of Electrical and Computer EngineeringTexas A&M University

College Station, TX 77843

Joint work with Eric D.B. Wendel (Draper Laboratory)

Finite Horizon Density Regulator: Outlook

AMS/IP Studies in Advanced Mathematics, vol 39, 2007, pp 23-35.

Optimal control of the Liouville equation

R. W. Brockett

Abstract. We explore the role of a special class of optimization problems

involving controlled Liouville equations. We present some new results on the

controllability of the Liouville equation and discuss the optimal control of itsmoments for some important special cases. The problems treated suggest a new

domain of applicability for optimal control, one which can include constraintsmore appropriate for the synthesis of control systems for a large class of real

world systems.

1. Introduction

In this paper we argue that some of the important limitations standing in theway of the wider use of optimal control can be circumvented by explicitly acknowl-edging that in most situations the apparatus implementing the control policy willjudged on its ability to cope with a distribution of initial states, rather than a singlestate. This makes it appropriate to represent the system dynamics in terms of itsLiouville equation and to formulate performance measures in that setting. That is,we argue in favor of replacing the usual control model x = f(x, u) by the first orderpartial di↵erential equation

@⇢(t, x)

@t= �

@x, f(x, u)⇢(t, x)

and for the consideration of performance measures which include non trajectorydependent terms such as the second and third terms on the right-hand side of

⇢(t, x)L(x, u)dxdt +

Later on we give a number of examples to support our contention but, in brief,we may say that by forcing the specification of a range of initial conditions thisformulation allows one to include some important types of performance measuresnot expressible in standard deterministic optimal control theory. In specific casesthis allows the designer to express, in mathematical terms, issues related to therole of feedback, robustness and cost of implementation. This approach has someaspects in common with stochastic control but occupies a position of intermediatecomplexity, lying between deterministic calculus of variations on finite dimensionalmanifolds and, the usually much less tractable, control of finite dimensional di↵u-sion processes.

In almost all applications of automatic control there is a trade o↵ between thegains to be had from trajectory optimization and the cost of the apparatus requiredto implement the optimal policy. A simpler apparatus that generates commands

This work was supported in part by the US Army Research O�ce under Boston Universityprime GC169369 NGD, the National Science Foundation grant number ECS 0424342, and by

DARPA under Stanford University Prime PY-1606.

Chapter 2Notes on the Control of the Liouville Equation

Roger Brockett

Abstract In these notes we motive the study of Liouville equations havingcontrol terms using examples from problem areas as diverse as atomic physics(NMR), biological motion control and minimum attention control. On one hand,the Liouville model is interpreted as applying to multiple trials involving a singlesystem and on the other, as applying to the control of many identical copiesof a single system; e.g., control of a flock. We illustrate the important role theLiouville formulation has in distinguishing between open loop and feedback control.Mathematical results involving controllability and optimization are discussed alongwith a theorem establishing the controllability of multiple moments associated withlinear models. The methods used succeed by relating the behavior of the solutionsof the Liouville equation to the behavior of the underlying ordinary differentialequation, the related stochastic differential equation, and the consideration of therelated moment equations.

2.1 Introduction

In these notes we describe a number of problems in automatic control related to theLiouville equation and various approximations of it. Some of these problems canbe cast either in terms of designing a single feedback controller which effectivelycontrols a particular system over repeated trials corresponding to different initialconditions or, alternatively, using a broadcast signal to simultaneously control manycopies of a particular system. Sometimes these different points of view lead toproblems that are identical from the mathematical point of view. In many cases acertain continuum limit can be formulated, either by considering an infinity of trials

R. Brockett (!)Division of Engineering and Applied Sciences, Harvard University, 345 Maxwell Dworkin,33 Oxford Street, Cambridge, MA 02138, USAe-mail: [email protected]

F. Alabau-Boussouira et al., Control of Partial Differential Equations,Lecture Notes in Mathematics 2048, DOI 10.1007/978-3-642-27893-8 2,© Springer-Verlag Berlin Heidelberg 2012

1158 IEEE TRANSACTIONS ON AUTOMATIC CONTROL, VOL. 61, NO. 5, MAY 2016

Optimal Steering of a Linear Stochastic System to aFinal Probability Distribution, Part I

Yongxin Chen, Student Member, IEEE, Tryphon T. Georgiou, Fellow, IEEE, and Michele Pavon

Abstract—We consider the problem of steering a linear dy-namical system with complete state observation from an initialGaussian distribution in state-space to a final one with minimumenergy control. The system is stochastically driven through thecontrol channels; an example for such a system is that of an inertialparticle experiencing random “white noise” forcing. We show thata target probability distribution can always be achieved in finitetime. The optimal control is given in state-feedback form and iscomputed explicitly by solving a pair of differential Lyapunovequations that are nonlinearly coupled through their boundaryvalues. This result, given its attractive algorithmic nature, appearsto have several potential applications such as to quality control,control of industrial processes, as well as to active control ofnanomechanical systems and molecular cooling. The problem tosteer a diffusion process between end-point marginals has a longhistory (Schrödinger bridges) and the present case of steering alinear stochastic system constitutes such a Schrödinger bridgefor possibly degenerate diffusions. Our results provide the firstimplementable form of the optimal control for a general Gauss–Markov process. Illustrative examples are provided for steeringinertial particles and for “cooling” a stochastic oscillator. A finalresult establishes directly the property of Schrödinger bridges asthe most likely random evolution between given marginals to thepresent context of linear stochastic systems. A second part to thiswork, that is to appear as part II, addresses the general situationwhere the stochastic excitation enters through channels that maydiffer from those used to control.

Index Terms—Linear stochastic system, Schrödinger bridge,stochastic control.

I. INTRODUCTION

THE most basic paradigm in optimal control deals with thesteering of a dynamical system between two end-points in

time, while minimizing a suitable cost functional—here, the ex-pected quadratic integral of the control input. The specificationsfor the marginal conditions are either explicit, requiring that thevalue of the state vector belongs to a specified set, or implicit,penalizing the distance of the state vector from a target location.

Manuscript received August 10, 2014; revised March 10, 2015; acceptedJune 23, 2015. Date of publication July 16, 2015; date of current versionApril 22, 2016. This work was supported in part by the National Science Foun-dation (NSF) under Grants ECCS-1027696 and ECCS-1509387, the AFOSRunder Grants FA9550-12-1-0319 and FA9550-15-1-0045 and by the Universityof Padova Research Project CPDA 140897. Recommended by Associate EditorC. Belta.

Y. Chen and T. T. Georgiou are with the Department of Electrical andComputer Engineering, University of Minnesota, Minneapolis, MN 55455 USA(e-mail: [email protected]; [email protected]).

M. Pavon is with the Dipartimento di Matematica, Università di Padova,Padova 35121, Italy (e-mail: [email protected]).

Color versions of one or more of the figures in this paper are available onlineat http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TAC.2015.2457784

In the presence of a stochastic disturbance, in both cases,the end-point conditions may be further modified accordingly.For instance, a bound on the probability of failing to meetexplicit constraints may be specified or, in the second case, thepenalty on the distance to a target location averaged. Herein,we consider a natural third alternative where the marginal end-point probability densities for the state vector are explicitlyspecified. This type of “soft conditioning” may be thought ofas a relaxed constraint with respect to the requirement that thestate vector belongs to a specified set. It leads to a problemwhich is in a way similar, but also sharply different fromthe classical LQG problem [1] as well as some of its morerecent variants such as the one with chance constraints, see,e.g., [2], [3].

This problem is relevant in a wide spectrum of classicalas well as emerging control applications. First and foremost,since it represents soft conditioning, it is of importance inapplications where a distribution rather than a set of values forthe state vector is a natural specification, e.g., in quality control,industrial and manufacturing processes. Applications may alsobe envisaged to control of aircrafts, UAVs, and autonomouscars. It is also relevant in a host of other applications at theforefront of modern technological developments such as in thecontrol at the molecular and even atomic scale, the shaping ofNMR pulse sequences, laser driven molecular reactions, quan-tum metrology, atomic force microscopy (AFM), dynamic forcemicroscopy (DFM) and many others; see [4]–[9]. For example,in surface topography using AFM, feedback control is used tolimit the fluctuations of the tip of the cantilever. Similarly, inother applications, the distribution of particles in phase space isshaped using a time-varying control potential, and the energyprofile of molecules and polymers is shaped using suitableenergy sources. Steering a thermodynamic system to a desiredsteady state corresponding to a lower effective temperatureis referred to as “cooling.” Cooling via feedback control isof great interest in both, microscopic as well as macroscopicelectro-mechanical systems. For instance, cooling to ultra-lowtemperatures is indispensable to investigate decoherence—see[10], [11] for a feedback cooling technique of a ton-scaleresonant-bar gravitational wave detector, and [12] for a surveyof cooling techniques for both meter-sized detectors as wellas nano-mechanical systems. For these diffusion-mediated de-vices, which are often called Brownian motors since work canbe extracted from them [13], motor efficiency can be cast asthe optimal control problem of steering the distribution of adiffusion process [14].

Interestingly, the special case of steering a Brownian diffu-sion between an initial and a final distribution relates to a seem-ingly disparate problem that was posed by Erwin Schrödingerin 1931/1932 [15]. In his quest for a more classical formulation

0018-9286 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

Optimal Steering of a Linear Stochastic Systemto a Final Probability Distribution, Part II

Abstract—We address the problem of steering the state of alinear stochastic system to a prescribed distribution over a finitehorizon with minimum energy, and the problem to maintain thestate at a stationary distribution over an infinite horizon withminimum power. For both problems the control and Gaussiannoise channels are allowed to be distinct, thereby, placing theresults of this paper outside of the scope of previous work both inprobability and in control. The special case where the disturbanceand control enter through the same channels has been addressedin the first part of this work that was presented as Part I. Herein,we present sufficient conditions for optimality in terms of a systemof dynamically coupled Riccati equations in the finite horizon caseand in terms of algebraic conditions for the stationary case. Wethen address the question of feasibility for both problems. For thefinite-horizon case, provided the system is controllable, we provethat without any restriction on the directionality of the stochasticdisturbance it is always possible to steer the state to any arbitraryGaussian distribution over any specified finite time-interval. Forthe stationary infinite horizon case, it is not always possible tomaintain the state at an arbitrary Gaussian distribution throughconstant state-feedback. It is shown that covariances of admissiblestationary Gaussian distributions are characterized by a certainLyapunov-like equation and, in fact, they coincide with the classof stationary state covariances that can be attained by a suitablestationary colored noise as input. We finally address the questionof how to compute suitable controls numerically. We present analternative to solving the system of coupled Riccati equations, byexpressing the optimal controls in the form of solutions to (con-vex) semi-definite programs for both cases. We conclude with anexample to steer the state covariance of the distribution of inertialparticles to an admissible stationary Gaussian distribution overa finite interval, to be maintained at that stationary distributionthereafter by constant-gain state-feedback control.

Index Terms—Covariance control, linear stochastic systems,Schrödinger bridges, stationary distributions, stochastic optimalcontrol.

I. INTRODUCTION

CONSIDER a linear system

x(t) = Ax(t) + Bu(t), t ∈ [0, ∞) (1)

Manuscript received October 13, 2014; revised March 10, 2015 and May 20,2015; accepted June 23, 2015. Date of publication July 16, 2015; date ofcurrent version April 22, 2016. This work was supported in part by theNational Science Foundation (NSF) under Grants ECCS-1027696 and ECCS-1509387 and the AFOSR under Grants FA9550-12-1-0319 and FA9550-15-1-0045, and by the University of Padova Research Project CPDA 140897.Recommended by Associate Editor C. Belta.

Y. Chen and T. T. Georgiou are with the Department of Electrical andComputer Engineering, University of Minnesota, Minneapolis, MN 55455USA (e-mail: [email protected]; [email protected]).

with A ∈ Rn×n, B ∈ Rn×m, x(t) ∈ Rn, and u(t) ∈ Rm, andthe problem to steer (1) from the origin to a given point x(T ) =ξ ∈ Rn. This of course is possible for any arbitrary ξ ∈ Rn

iff the system is controllable, i.e., the rank of [B,AB, . . . ,An−1B] is n, that is, when (A, B) is a controllable pair. Inthis case it is well known that the steering can be effected ina variety of ways, including “minimum-energy” control, overany prespecified interval [0, T ]. On the other hand, the problemto achieve and maintain a fixed value ξ for the state vector in astable manner is not always possible. For this to be the case fora given ξ, using feedback and feedforward control

0 = (A − BK)ξ + Bu (2)

must have a solution (u, K) for a constant value for the input uand a suitable value of K so that A − BK is Hurwitz (i.e., thefeedback system be asymptotically stable). It is easy to see thatthis reduces simply to the requirement that ξ satisfies

0 = Aξ + Bv

for some v; if there is such a v, we can always choose a suitableK so that A − BK is Hurwitz and then, from v and K, we cancompute the constant value u. Conversely, from u and K wecan obtain v = u − Kξ.

In the present paper, we discuss an analogous and quitesimilar dichotomy between our ability to assign the state-covariance of a linear stochastically driven system by steeringthe system over an interval [0, T ], and our ability to assignthe state-covariance of the ensuing stationary state processthrough constant state-feedback. It will be shown that the state-covariance can be assigned at the end of an interval throughsuitable feedback control if and only if the system is con-trollable. On the other hand, a positive semidefinite matrixis an admissible stationary state-covariance attained throughconstant feedback if and only if it satisfies a certain Lyapunov-like algebraic equation. Interestingly, the algebraic equationthat specifies which matrices are admissible stationary state-covariances through constant feedback is the same equationthat characterizes stationary state-covariances attained throughcolored stationary input noise in open loop.

Both of these problems, to steer and possibly maintain thestate statistics of a stochastically driven system, represent gen-eralizations of the classical regulator problem which is at theheart of many control applications and entails efficient andaccurate steering to a target location. Prime examples, thatbrought the subject of control and estimation to prominencesince the sixties include soft moon-landing, docking, and theguidance of space vehicles, aircraft navigation, robotics, andthe steering of quantum mechanical systems, to name a few[1]–[4]. The paradigm that is being considered in this paper