state constrained reachability for stochastic hybrid systems

Nonlinear Analysis: Hybrid Systems 5 (2011) 320–342

Contents lists available at ScienceDirect

Nonlinear Analysis: Hybrid Systems

journal homepage: www.elsevier.com/locate/nahs

State constrained reachability for stochastic hybrid systemsManuela L. Bujorianu ∗, Marius C. BujorianuUniversity of Manchester, Centre for Interdisciplinary Computational and Dynamic Analysis, Alan Turing Building, Oxford Road, Manchester M13 9PL, UK

a r t i c l e i n f o

Keywords:Stochastic hybrid systemsState constrained reachability analysisViewpointsMultilayer modelsTrajectory design

a b s t r a c t

Many control problems can be formulated as driving a system to reach some targetstates while avoiding some unwanted states. We study this problem for systems withregime change operating in uncertain environments. Nowadays, it is a common practiceto model such systems in the framework of stochastic hybrid system models. In thiscasting, the problem is formalized as a mathematical problem named state constrainedstochastic reachability analysis. In the state constrained stochastic reachability analysis,this probability is computed by imposing a constraint on the system to avoid the unwantedstates. The scope of this paper is twofold. First we define and investigate the stateconstrained reachability analysis in an abstract mathematical setting. We define theproblem for a general model of stochastic hybrid systems, and we show that the reachprobabilities can be computed as solutions of an elliptic integro-differential equation.Moreover, we extend the problem by considering randomized targets. We approachthis extension using stochastic dynamic programming. The second scope is to definea developmental setting in which the state constrained reachability analysis becomesmore tractable. This framework is based on multilayer modelling of a stochastic systemusing hierarchical viewpoints. Viewpoints represent a method originated from softwareengineering, where a system is described by multiple models created from differentperspectives. Using viewpoints, the reach probabilities can be easily computed, or evensymbolically calculated. The reach probabilities computed in one viewpoint can be usedin another viewpoint for improving the system control. We illustrate this technique fortrajectory design.

© 2010 Elsevier Ltd. All rights reserved.

1. Introduction

Hybrid systems form a class of systems whose behaviors are characterized by a non-trivial interaction between discreteand continuous dynamics. These systems accurately model technical systems from automotive industry, aeronautics, airtraffic control, robotics, and nanotechnology. Hybrid models are also used frequently in system biology and medicine,where their featuresmake controllability and verificationmore difficult, mostly because of uncertainty, complex continuousnonlinear dynamics, partial information, etc. In the case of open systems, the environmental influence produces randomevolutions increasing the complexity of verification and control problems. To address these issues, randomizedmodels havebeen considered and their class is usually denoted as stochastic hybrid systems [1].Mathematically, a stochastic hybrid systemcan be seen as an interleaving between a finite or countable family of diffusion processes (or, sometimes, deterministicdynamical systems only) and aMarkov chain. Modeling and analysis of these systems have been proved to be a very difficulttask, especially from foundational point of view. The stochastic analysis apparatus, employed to study their probabilisticproperties is complex and rather difficult to manage. This study involves the ability to combine tools available for diffusion

∗ Corresponding author.E-mail addresses:[email protected] (M.L. Bujorianu), [email protected] (M.C. Bujorianu).

1751-570X/$ – see front matter© 2010 Elsevier Ltd. All rights reserved.doi:10.1016/j.nahs.2010.10.008

http://dx.doi.org/10.1016/j.nahs.2010.10.008

http://www.elsevier.com/locate/nahs

http://www.elsevier.com/locate/nahs

mailto:[email protected]

mailto:[email protected]

http://dx.doi.org/10.1016/j.nahs.2010.10.008

M.L. Bujorianu, M.C. Bujorianu / Nonlinear Analysis: Hybrid Systems 5 (2011) 320–342 321

processes and jumpprocesses, in order to characterize the executions of these systems. The switchingmechanism (governedby a Markov chain in most cases) between the continuous dynamics of the modes, together with the interaction betweenpaths and boundaries, make the studying of the stochastic processes that arise in this way very difficult and challenging.

Reachability analysis is at the heart of any verification problem for stochastic systems that are discrete, or continuous, orhybrid discrete/continuous. The standard form of reachability analysis asks to compute/approximate the probability of allsystem paths that start from a given initial state and visit a target state set. In the discrete case, this problem is known asprobabilistic model checking and there exist relatively efficient algorithms for solving it. For continuous and hybrid stochasticsystems, the problem is more difficult and it is only partially solved. Theoretical solutions are based on martingale theory(see [2,3]), Bayesian statistics, as in [4], and optimal control [5]. Numerical solutions are mainly based on interactive particlesystems [6],Monte Carlo simulations [7],Markov chain approximations [8], and dynamic programming [9].

For practical reasons, the reachability analysis has got variants by inserting further conditions (constraints). Theconstraints can be formulated with respect to time or states see [10]. For example, one can ask to estimate the probability toreach a target state set in a finite horizon time (time-constrained reachability). Relative to the states, a variant of reachabilityanalysis may ask to evaluate the probability to reach a target state set by avoiding a given set of states (interpreted asdangerous situations) [11]. This problem is important in planning and stochastic control. From a practical site, this sortof problem arises from robotics, air traffic control, or military applications. In this paper, we formulate and investigate astate-constrained reachability problem for stochastic hybrid systems. We prove that the problem is solvable. Moreover, forsystems that are sufficiently ‘‘regular’’ (they can be approximated by diffusions), the problem can be solved symbolically ornumerically.

The paper’s structure is as follows. In the next section, we present a model of stochastic hybrid systems and itsproperties. In Section 3, we present the standard stochastic reachability problem, followed by the state-constrainedreachability problem. Then in Section 4, we give some characterizations of the state-constrained reachability probabilities.Generalizations of the state constrained reachability by introducing randomized targets are given in Section 5. Furtherdevelopments of the theory of state constrained reachability and dynamic programming solutions are given in the Section 6.In Section 7 a development framework is introduced and possible applications are discussed. The paper endswith some finalremarks.

2. Stochastic hybrid systemmodels

We adopt the general stochastic hybrid system model presented in [12,13]. In this section the model is described and thenotation is established.

Let Q be a finite/countable set of discrete states. For each q ∈ Q , we consider the Euclidean space Rd(q) with dimensiond(q) and we define an invariant as an open subset Xq of Rd(q). The hybrid state space is the set

X(Q , d, X) =

i∈Q

i × X i

and the hybrid state is defined as

x = (i, z i) ∈ X(Q , d, X).

The closure of the hybrid state space will be

X = X ∪ ∂X,

where

∂X =

i∈Q

i × ∂X i.

It is known that X can be endowed with a metric ρ whose restriction to any component X i is equivalent to the naturalEuclideanmetric of this component [14]. Then (X, B(X)) is a Borel space (i.e. homeomorphic to a Borel subset of a completeseparable metric space), where B(X) is the Borel σ -algebra of X . Let B(X) be the Banach space of bounded positivemeasurable functions on X with the norm given by the supremum.

Definition 1. A (general) stochastic hybrid system (SHS) [13] is a collection

H := ((Q , d, X), b, σ , Init, λ, R)

where

• Q is a countable/finite set of discrete states (modes);• d : Q → N is a map giving the dimensions of the continuous state invariants;• X : Q → Rd(.) maps each q ∈ Q into an open subset Xq of Rd(q);• b : X(Q , d, X) → Rd(.) is a vector field;• σ : X(Q , d, X) → Rd(·)×m is a X (·)-valued matrix,m ∈ N (more generally,m may also depend on the discrete mode),

322 M.L. Bujorianu, M.C. Bujorianu / Nonlinear Analysis: Hybrid Systems 5 (2011) 320–342

• Init : B(X) → [0, 1] is an initial probability measure on (X, B(X));• λ : X(Q , d, X) → R+ is a transition rate function;• R : X × B(X) → [0, 1] is a stochastic kernel.

Let us denote by X the whole space, i.e.

X = ∪(q, Xq)|q ∈ Q .

Define the boundary set ∂Xq:= Xq \ Xq of Xq and the whole space boundary

∂X = ∪(q, ∂Xq)|q ∈ Q .

The executions of an SHS form a stochastic process, called a stochastic hybrid process, which is built as a Markov string —see [13]. This is obtained by the concatenation of a sequence of diffusion processes (z it), i ∈ Q . The order in this sequence isprovided by means of a jumping mechanism given by a family of stopping times (S i) (defined below).

Letωi be a diffusion trajectory, which starts in (i, z i) ∈ X . Let t∗(ωi) be the first hitting time of ∂X i of the diffusion processcorresponding to the mode i. Define the multiplicative functional

F(t, ωi) = I(t<t∗(ωi)) exp

−

∫ t

0λ(i, z is(ωi))

ds. (1)

This functional is called the survivor function and is used to define the stopping time S i associated to the diffusion of themode i.S i is nothing else but a killing time when the diffusion process of the mode i is stopped and then another operatingmode with its continuous dynamics is started.

The full formal definition of a stochastic hybrid process is given below.

Definition 2 (SH Process). A stochastic hybrid (SH) process [13] is a stochastic process

xt = (q(t), z(t))

such that there exists a sequence of stopping times

T0 = 0 < T1 < T2 < · · ·

such that for each k ∈ N,

• x0 = (q0, zq00 ) is a Q × X-valued random variable extracted according to the probability measure Init;

• For t ∈ [Tk, Tk+1), qt = qTk is constant and z(t) is a solution of the stochastic differential equation (SDE):

dz(t) = b(qTk , z(t))dt + σ(qTk , z(t))dWt (2)

whereWt is the m-dimensional standard Wiener process (m might depend on qTk );• Tk+1 = Tk + S ik where S ik is a stopping time chosen according with the survivor function (1).• The probability distribution of x(Tk+1) is governed by the law R

(qTk , z(T

−

k+1)), ·.

Under standard assumptions (about the diffusion coefficients, non-Zeno executions, transition measure, etc.), explainedbelow, any SH process is a strong Markov process. The definition of a strong Markov process can be found in any classicalmonograph for Markov processes (see, for example, [15] or [16]).

The standard assumptions about the components of an SHS are presented as follows.

Assumption 1 (About Diffusion Coefficients). Suppose that

b : Q × X (·)→ Rd(·), σ : Q × X (·)

→ Rd(·)×m(·)

are bounded and Lipschitz continuous in z.

This assumption (about the diffusion coefficients) ensures that for any i ∈ Q , the existence and uniqueness of the solutionof the Eq. (2)

We denote by

Nt(ω) :=

−k

I(t≥Tk).

Assumption 2 (Non-Zeno Executions). For every starting point x ∈ X ,

ExNt < ∞,

for all t ∈ R+.

This assumption ensures that the stochastic hybrid process has no chattering.


Assumption 3 (Transition Structure). (A) λ : X → R+ is a measurable function such that

t → λ(xit(ωi))

is integrable on [0, ε(xi)), for some ε(xi) > 0, for each z i ∈ X i and each ωi starting at z i.(B)

(i) for all A ∈ B(X), R(·, A) is measurable;(ii) for all x ∈ X the function R(x, ·) is a probability measure;(iii) R(x, x) = 0 for x ∈ X .

This assumption ensures that the transition measure and the transition rate function are well-defined.Let

M = (xt , Px)

be the Markov hybrid process associated to H . Here, xt is a collection of X-valued random variables that represent thehybrid trajectories of H . The underlying probability space is denoted by (Ω, F ). Ft is the natural filtration of the process(the ‘history’ of the process). The (Wiener) probability

Px : (Ω, F ) → [0, 1]

is a probability measure such that Px(xt ∈ A) is B-measurable in x ∈ X , for each t ∈ [0, ∞) and A ∈ B, and

Px(x0 = x) = 1.

Themeaning of these elements associated toM is standard for continuous-parameterMarkov processes (see [17,15], or [14]).We adjoin an extra point ∆ (the cemetery) to X as an isolated point,

X∆ = X ∪ ∆.

The existence of ∆ is assumed in order to have a probabilistic interpretation of

Px(xt ∈ X) < 1,

i.e.∆ is the state where the process lies when it ‘dies’. Then, the ‘termination time’ ζ (ω) is the random timewhen the processM escapes to and is trapped at ∆.

The Markov property of the process M (in particular, the Chapman–Kolmogorov equations) allows us to introduce thefollowing semigroup of operators associated toM , denoted by

P := (Pt)t>0.

Each Pt (t > 0) maps B(X) into itself, and it is given as

Pt f (x) := Exf (xt), ∀x ∈ X,

where Ex is the expectation with respect to. Px.The infinitesimal generator L is the derivative of Pt at t = 0. Let D(L) ⊂ Bb(X) be the set of functions f for which the

following limit exists

limt0

1t(Pt f − f ).

In the case of existence, this limit is denoted by Lf .The following result is essential for the mathematical study of reachability properties.

Proposition 1 ([13]).

(a) Under the standard assumptions the stochastic hybrid process M defined above is a Borel right process.(b) The sample paths of M are right continuous with left limit (rcll), i.e. are cadlags.1

Recall [14] that a Borel right process is defined by the following properties:

(i) its sample paths t → xt are right-continuous almost sure.(ii) X is a separable metric space homeomorphic to a Borel subset of some compact metric space, equipped with Borel

σ -algebra B(X) or shortly B (i.e. X is a Lusin state space).(iii) The operator semigroup ofM , given by (2), maps B(X) into itself.(iv) M is a strong Markov process.

1 The French abbreviation for rcll.


Given a function f ∈ C1(Rn, R) and a vector field b : Rn→ Rn, we use Lbf to denote the Lie derivative of f along b

given by

Lbf (x) =

n−i=1

∂ f∂xi

(x)fi(x).

Given a function f ∈ C2(Rn, R), we use Hf to denote the Hamiltonian operator applied to f , i.e.

Hf (x) = (hij(x))i,j=1...n ∈ Rn×n,

where hij(x) =∂2f

∂xi∂xj(x).

AT denotes the transpose matrix of a matrix A = (aij)i,j=1...n ∈ Rn×m and Tr(A) denotes its trace. The infinitesimalgenerator of an SHS is an integro-differential operator. In [12], it has been proved, that the extended generator of an SHS hasthe following expression:

Lf (x) = Lcontf (x) + λ(x)∫X(f (y) − f (x))R(x, dy) (3)

where Lcontf (x) has the standard form of the diffusion infinitesimal operator

Lcontf (x) = Lbf (x) +12Tr(σ (x)σ (x)THf (x)). (4)

What makes this generator different from the generator of a Feller Markov process (see [15]) is its domain that contains atleast the set of second order differentiable functions that satisfy the boundary condition, as follows:

f (x) =

∫Xf (y)R(x, dy), x ∈ ∂X . (5)

3. Stochastic reachability with constraints: mathematical definitions

In this section, we extend the concept of state constrained reachability defined in the literature (see [10] and thereferences therein) only for discrete/continuous time Markov chains (with discrete state space) to continuous time/spaceMarkov processes. Further, we study this concept for stochastic hybrid processes. For a better understanding of this concept,first we revise the ‘‘classical’’ concept of stochastic reachability as it was defined in [2],

3.1. Stochastic reachability

Let us consider an SH process M = (xt , Px). Usually, the verification problem can be reduced to the following stochasticreachability problem. Given a target set, the stochastic reachability analysis aims to compute the probability that the systemtrajectories from an arbitrary initial state will reach the target set.

Formally, given a Borel measurable set A ∈ B(X) and a time horizon T > 0, we define the reach ‘‘event’’:

ReachT (A) := ω ∈ Ω | ∃t ∈ [0, T ] : xt(ω) ∈ A. (6)

For an infinite horizon time, we define:

Reach∞(A) := ω ∈ Ω | ∃t ≥ 0 : xt(ω) ∈ A. (7)

These two events are the sets of trajectories of M , which reach the set A (the flow that enters A) in the interval of time[0, T ] or [0, ∞). The reachability problem consists of determining the probabilities of such sets. Since the process M is aBorel process and has the cadlag property, the reachability problem is well-defined, i.e. ReachT (A), Reach∞(A) are indeedmeasurable sets — see [2].

The probabilities of reach events can be expressed as:

P(TA < T ) or P(TA < ζ)

where

• ζ is the life time ofM and TA is the first hitting time of A, i.e.

TA = inft > 0|xt ∈ A. (8)

• P is a probability on the measurable space (Ω, F ) of the elementary events associated toM .

Note that P can be chosen to be Px (if we want to consider the trajectories that start in x) or Pµ (if we want to consider thetrajectories that start with an initial condition given by the distribution µ). Recall that

Pµ(A) =

∫Px(A)dµ, A ∈ F .


Fig. 1. Obstacle avoidance reachability.

Remark 1. The probability P(TA < T ), which is the probability of the event defined by (6) can be thought of as a time-constrained reachability probability.

Denote by PA the hitting operator PA : B(X) → B(X) associated to the underlying Markov process (xt), i.e.

(PAu)(x) = Exu(xTA)|TA < ζ (9)

and TA is given by (8), and Ex is the expectation w.r.t. Px.The following fundamental result of stochastic reachability was proved in [3].

Proposition 2. For any x ∈ X and Borel set A ∈ B(X), we have

Px[Reach∞(A)] = PA1(x). (10)

Depending on the initial condition x, we can introduce, as a function of x, the infinite horizon time reachability probabilityas follows:

ϕA(x) := Px[Reach∞(A)]. (11)

According with the previous proposition, the reachability function ϕA is coming as a result of applying the hitting operatorPA to the constant function 1, i.e.

ϕA = PA1.

Next subsection is devoted to the introduction of the concept of state constrained reachability.

3.2. State-constrained reachability

State-constrained reachability analysis denotes a reachability problem with additional conditions (constraints) on thesystem trajectories. Let us consider A, B two Borel measurable sets of the state space X with disjoint closures, i.e.

A, B ∈ B(X) and A ∩ B = ∅.

We consider two fundamental situations. Suppose that the system paths start from a given initial state x and we areinterested in a target state set, let say B. These trajectories can hit the state set A or not. Therefore, we may define twonew concepts:

• Obstacle avoidance reachability. In this interpretation B is a safe set, whilst A is not. The goal is to compute the probability,denoted by

pB¬A(x),

of all trajectories that start from a given initial state x and hit the set B without hitting the state set A (as illustrated inFig. 1).

• Waypoint reachability. In this interpretation we are interested in computing the probability, denoted by

pBA(x),

of all trajectories that hit B only after hitting A (as illustrated in Fig. 2).

The connection between the two types of stochastic reachability is given by the formula

pB¬A(x) + pBA(x) = ϕB(x)

where ϕB is the reachability function for the target set B given by formula (11). Therefore, the computations of theprobabilities corresponding to the two types of reachability are equivalent. To have an easy notation, it is more convenientto work with the waypoint reachability, which will be called from now on just simply state-constrained reachability.

Now we consider the executions (paths) of the stochastic hybrid process that start in x = (q, z) ∈ X . When weinvestigate the state-constrained reachability, we ask the probability that these trajectories visit A before visiting eventuallyB. Mathematically, this is the probability of

ω|xt(ω) ∈ B, ∀t ≤ TA.


Fig. 2. Waypoint reachability.

Moreover, using the first hitting time TB of B, we are interested to compute

pBA(x) = Px[TA < TB]. (12)

Consequently, the state constrained reachability is related to some classical topics treated in the literature of Markovprocesses like the first passage problem [16], excursion theory [18], estimation of the equilibrium potential of the capacitorof the two sets [19]. These references provide theoretical characterizations for the probabilities (12) for different classesof Markov processes. The scope of this paper is not to survey all these characterizations, but to identify the appropriateanalytical solutions of this problem.

4. Stochastic reachability with constraints: mathematical characterizations

The main scope of this section is to prove that the state constrained reachability probabilities can be characterized assolutions of some boundary value problems expressed in terms of the infinitesimal generator of the given stochastic hybridprocess.

First we need to recall the concepts of excessive function and kernel (or Green) operator that can be introduced w.r.t. aMarkov processM = (xt , Px).

A nonnegative function f ∈ B(X) is called excessive (see [17]) w.r.t.M (in fact, w.r.t. the semigroup P ) if

(i) Pt f ≤ f for all t ≥ 0, and(ii) Pt f f as t 0.

In the theory of Markov processes, the excessive functions play the role of the superharmonic functions from the theory ofpartial differential equations (for e.g. a function f ≥ 0 is superharmonic with respect to the Laplace operator if ∆f ≤ 0).Moreover, in stochastic control, these functions are known also as stochastic Lyapunov functions. The reason for this is the factthat if f is an excessive function for M , then f (xt) is supermartingale. Therefore, the excessive functions are ‘‘expectationdecreasing’’ on the trajectories of M , property that is characteristic to the classical Lyapunov functions. Moreover, due tothe strong Markov property of M , the excessive functions are also right continuous on the process trajectories. For manycontrol problems, the cost functions are usually excessive functions. In particular, the reachability function is an excessivefunction [3].

The kernel operator is defined as

Vf (x) :=

∫∞

0Pt f (x)dt,

for all f ∈ B(X), x ∈ X .

The following assumption is essential for the results of this section.

Assumption 4. Suppose that M is a transient Markov process, i.e. there exists a strict positive Borel measurable function qsuch that Vq is a bounded function.

The transience of M means that for any Borel set E in X and for almost all trajectories there exists a finite stopping timet∗ such that xt ∈ E for all t > t∗. The transience ensures that the cone of excessive functions is nontrivial.

Using [19], we have the following characterization.

Proposition 3. The state-constrained reachability probability pBA has the following properties:

(i) 0 ≤ pBA ≤ 1 a.s. on X,(ii) pBA = 0 a.s. on B, and pBA = 1 on A,(iii) pBA is the potential of a signed measure ν such that the support of ν+ is contained in A and the support of ν− is contained

in B.


We can write, in a more compact manner

pBA(x) =

Px[TA < TB] if x ∈ A ∪ B1 if x ∈ A0 if x ∈ B.

An inclusion–exclusion argument leads to the following formula

pBA(x) = Px(TA < TB)= PA1(x) − PBPA1(x) + PAPBPA1(x) − · · · .

Let us make the following notations for:

• composition of the hitting operators corresponding to the target sets A and B

V A→B:= PA PB

where

PA(PBu)(x) = Ex( PBu)(xTA) = ExExTA u(xTB), u ∈ B(X)

provided that TA, TB < ζ .

• the probability of hitting A again after n excursions between A and B

pn := (V A→B)nϕA,

where ϕA is given by (11).• the probability of hitting A again after ‘infinitely many’ excursions between A and B

Γ :=

∞−n=0

pn.

Proposition 4. Then, we have the following recurrence formula:

pBA = (I − PB)Γ

where I : B(X) → B(X) is the identity operator.

Proof. Each pn is an excessive function, bounded by 1, and PBpn ≤ pn. Therefore,

pn − PBpn ∈ [0, 1].

Let us set T0 := 0 and T1, T2, T3, . . . to be the times of the successive visits to A, then to B, then back to A, and so on. Formally,these times are defined as:

T1 := TAT2 := TA + TB θTA

· · ·

T2n+1 := T2n + TA θT2nT2n+2 := T2n+1 + TB θT2n+1 .

An induction argument shows that

PT2n = (PAPB)n, n ∈ N.

Then, it can be easily checked that

Px[TA < TB, T2n+1 ≤ L ≤ T2n+2] = pn(x) − PBpn(x)

where L is the last exit time from A, i.e.

L = LA = supt > 0|xt ∈ A.

L is a.s. finite because usually we suppose that our process is transient, in the sense that if it enters a set then it must leaveit also.

Theorem 5. State-constrained reachability probability pBA solves the following boundary value problem:Lp(x) = 0 x ∈ X \ (A ∪ B)p(x) = 1 x ∈ Ap(x) = 0 x ∈ B

(13)

where L is the infinitesimal generator of the stochastic hybrid process given by (3).


This is the main theorem about the characterization of the state-constrained reachability. The theorem can be proved forBorel right processes that are SH processes. Stochastic hybrid processes have a continuous dynamics given by some diffusionprocesses, and a discrete dynamics described by a Markov chain. Therefore, the proof is a consequence of the following twolemmas, which are instantiations of the theorem for Brownian motion and Markov chains. We have not found the proofs inany monograph of stochastic processes that treat first time passage problems, excursion theory for Markov processes—forexample [16], therefore we sketch these proofs in the following.

Lemma 6. Let us consider a (discrete time, discrete state) Markov chain (Xt) with the state space Γ and the one-step transitionfunction p1(x, y). Given two disjoint sets A, B ⊂ Γ . Then the state-constrained reachability probability pBA (x) is the solution ofthe boundary value problem

(1 − p1)p(x) = 0 x ∈ Γ \ (A ∪ B)p(x) = 1 x ∈ Ap(x) = 0 x ∈ B.

Lemma 7. For a discrete space Markov chain, it is known that its infinitesimal generator is given by

L = 1 − p1.

Proof. If x ∈ A ∪ B, we make the elementary remark that the first step away leads either to B, and the event TA < TB failsto happen, or to A, in which case the event happens, or to another point y ∈ A ∪ B, in which case the event happens withprobability Py[TA < TB]. Therefore, we obtain

Px[TA < TB] =

−y∈A

p1(x, y) +

−y∈A∪B

Py[TA < TB].

Then for x ∈ A ∪ B, we obtain

p(x) =

−y∈Γ

p1(x, y)p(y).

This ends the proof.

Lemma 8. Let us consider W the standard d-dimensional Wiener process. Let A, B be two disjoint capacitable sets (see [20] forfull definition) of non-zero capacity such that A ∪ B is closed. The reachability probability p(x) satisfies the Laplace problem

∇2p(x) = 0

on X − (A ∪ B) with the boundary condition

p(x) =

1 if x ∈ A0 if x ∈ B.

Proof. Let x ∈ X − (A ∪ B) and H a ball of radius h and surface S in X − (A ∪ B) centred in x. Define the random variableT = inft|xt(ω) ∈ S. This has the property that Px[T < ∞] = 1. Let Hi = |W (i) − W (i − 1)| ≤ 2h. Then Px(A1) < 1 andlimn→∞ Px[T > n] = 0. This results from the inequality Px[T > n] ≤ Px(A1 ∩ · · · ∩ An) = Px(A1)

n.We have

p(x) =

∫y∈S

Px[TA < TB|W (T ) = y]f (y)dS

where f (y) = 1/|S| is the density function. This means that

p(x) =

∫y∈S

p(y)/|S|dS.

Theorem 5 characterizes the probabilities of the state constrained reachability as the solutions for a Dirichlet boundaryvalue problem (DBVP) associated to generator of the underlying stochastic hybrid process. This generator is a second orderelliptic integro-differential operator, and it is known that for this type of non-local operators, the value of the solutions forthe DBVP has to be prescribed not only on the boundary of the domain but also in its whole complementary set [21]. Understandard hypotheses, the existence and uniqueness of the solutions for such equations can be proved. The solutions arecalled potential functions for the underlying Markov process, and they play the same role like the harmonic functions forthe Laplace operator.

Dirichlet boundary value problems for such operators have been already addressed in the literature by using differenttheories:


• for a classical PDE approach using Sobolev spaces, see [21];• for a probabilistic approach using the operator semigroup, see [22].• for a viscosity solution approach, see [23].

For the verification problems defined in the context of stochastic hybrid systems, the DBVP defined in the Theorem 5willhave to be solved only locally in the appropriate modes. In this way, the quite difficult guard condition (5) can be avoided.Then, we consider that most of numerical solutions available for the Dirichlet boundary value problem corresponding tosecond order elliptic partial differential operators can be extended in a satisfactory manner to solve our problem.

Without anydoubts,weneed to consider the applications of state-constrained reachability in the framework of controlledstochastic hybrid systems. The control can be defined either

• at the ‘low level’, i.e. continuous dynamics in the operating modes are controlled diffusions,• or, at the ‘decision level’, i.e. the jumping structure is governed by a decision process (usually in the form of a Markov

Decision Process).

5. Stochastic reachability with constraints: randomized targets

In this section, we develop a possible generalization of the state constrained reachability when the ‘final goal’ is replacedby a ‘‘random event’’ that takes place at a random time.Wewill be interested to compute the probability of those trajectoriesthat visit a set before ‘something’ random happens.

5.1. Leveraging state constrained reachability

State constrained reachability analysis means to obtain estimations for Px[TA < TB], i.e. to find the probability of visitingB after the process has visited A. As we have seen in the previous section, this problem can be characterized as a boundaryvalue problem. In practice, it may happen that the sets A and B are not explicitly given. These sets can be characterized as:

• level sets for some given functions,• sets of states that validate some logical formulae;• metastable sets for the given process.

Sometimes, for the computation of the state constrained reachability probabilities more information is needed about atleast one of these sets. Often, the available information is about the set B that can be either

• the boundary of one mode of the stochastic hybrid process,• a cemetery set where the process evolution is stopped,• a set that is reached according with state dependent rate, etc.

Moreover, in the expression of the state constrained reachability probability, wemay replace the hitting time TB of Bwitha suitable random time that could be, for example,

• a switching time from one mode to another of the stochastic hybrid process,• a time of the apparition of a certain event, or• a time defined by the until operator in a suitable continuous stochastic logic associated to our hybrid Markov process.

Then the probabilities that should be computed are:

Px[TA < T ]

where T is a random time that will be defined properly in the following subsection.

5.2. Randomized stopping times

We enlarge the space of stopping times with the so-called randomized stopping times [24]. A randomized stopping timeT is defined to be a map

T : Ω × [0, 1] → [0, ∞]

such that T is a stopping time with respect to the σ -algebras (Ft × B1), where B1 represents the Borel σ -algebra of theinterval [0, 1]. It is required that for every ω ∈ Ω , T (ω, ·) is nondecreasing and left continuous on [0, 1].

If T is a randomized stopping time, then T (·, a) is an ordinary stopping time. A randomized stopping time T can becharacterized by a stopping time measure (ω-distribution) K induced by T . K is defined as the map

K : Ω × B[0, ∞] → [0, 1]K(ω, [0, t]) := supa : T (ω, a) ≤ t (14)

provided that K(ω, ·) is a measure on B[0, ∞]. K(ω, ·) is a version of the conditional distribution of T given the entiretrajectory ω.


Using the measure K , one can obtain back the randomized stopping time T by

T (ω, a) = inft : K(ω, [0, t]) ≥ a. (15)

Moreover, one can define a stopping time measure as an independent mathematical object as follows.A map

K : Ω × B[0, ∞] → [0, 1]

is called a stopping time measure if:

(i) K(ω, ·) is a probability measure for each ω ∈ Ω;(ii) K(·, [0, t]) is Ft-measurable for each t .

Then T can be defined by (15). Therefore, there exists a complete correspondence between the notions of stopping timemeasure and randomized stopping time.

Usually, the following notation is in use:

Kt := K((t, ∞]) = K(·, (t, ∞]).

If P is the underlying probability on (Ω, F ), andm1 is the Lebesgue measure on the unit interval [0, 1], then Kt is a versionof the conditional probability of T > t using the probability P × m1 with respect to the σ -algebra F .

5.3. Markovian randomized stopping times

Workingwith randomized stopping timesmight be a difficult task since theMarkov property is still true onlywith respectto the non-randomized stopping times (strong Markov Property). For the purposes of this paper, we will consider only theMarkovian randomized stopping times. Examples of this kind of random times are:

• Markov killing times: T is a Markov killing time forM if under Px the killed process (xt |0 ≤ t ≤ T ) is Markovian with thesub-Markovian semigroup (Γt)t≥0:

Γt f (x) := Px[f (xt)1(t<T )].

In addition, we assume that Γt f is B-measurable for all t > 0 and all positive B-measurable f .• Terminal times: An Ft-stopping time T : Ω → R+ is called a terminal time if

T = t + T θt (16)

identically on [t < T ].

In the definition of the terminal time we have used the shift operator or translation operator θt : Ω → Ω , that ischaracterized by the following property

xs θt = xt+s, t, s ≥ 0. (17)

Clearly, the relation (16) expresses the memoryless property of a terminal time,

T (ω) − t = T (θtω)

i.e. the value of T on a path ω after the time t has elapsed is equal with the value of T on the same path shifted/translatedwith the time t .

Examples of killing times are:

• the random time Tr with the stopping measure Kt = e−rtPt ;• ∞ = limr→0 Tr ;• the first entrance/last exit time of a suitable subset B of the state space X;• the random time obtained by killing the process at state-dependent rate k(xt), etc.

A finite fixed time T is not aMarkov killing time unless the process is set up as a space–time process, and then T becomesa hitting time. As shown by the example of last exit times, the killing times may not be stopping times of the process, incomparison with the terminal times, which should be necessarily stopping times.

Themost common examples of terminal times are provided by the hitting times of measurable subsets of the state spaceX . The jumping times in the definition of a stochastic hybrid process are terminal times.

5.4. Multiplicative functionals

A common methodology to obtain randomized stopping times is by using multiplicative functionals. These functionalshave a long history in the theory of Markov processes, and they have been mostly employed to describe transformations ofthe trajectories for these processes.


The seminal work about the properties of the trajectories of a Markov process in connection with multiplicativefunctionals belongs to Dynkin [25]. Under some regularity hypotheses, Dynkin proved that the transformed processescan be defined such that their trajectories are ‘‘restrictions’’ of the trajectories of the initial process. In [26], Kunita andWatanabe showed that the transformations of aMarkov process governed bymultiplicative functionals (whose expectationsare dominated by 1) preserve some regularity properties of the initial process such as (strong)Markovianity, right continuityof the trajectories, quasi-left continuity of the stopping time sequences, etc.

A systematic study of the multiplicative functionals of Markov processes has been done in [17], and later in [27]. In theAppendix, we provide a short background on multiplicative functionals that can be helpful for developing the theory ofconstrained reachability.

A stopping time measure α will be calledmultiplicative functional if for every s, t ≥ 0,

αt+s = αt(αs θt) a.s. (18)

We will assume that α is an exact multiplicative functional [17]. The property (18) ensures that the randomized stoppingtime generated by (αt) has the memoryless property.

6. Stochastic reachability with constraints: dynamic programming solutions

This section represents the development of the mathematical basis necessary for defining dynamic programmingsolutions for state constrained reachability. The main idea consists in taking the transformation of the initial SH processwith respect to the multiplicative functional that governs the randomized time (describing the final target). The procedureis, in fact, based on the idea to consider a subprocess of the initial Markov process dictated by this multiplicative functional.Then state constrained reachability becomes ‘‘classical’’ reachability for this subprocess.

Then, there are some steps that have to be followed:

1. construct the Lévy system associated to the subprocess;2. find the expression of the infinitesimal generator of the subprocess (using the Lévy system);3. use the characterization of stochastic reachability via an optimal stopping problem and derive the dynamic programming

equations.

6.1. Infinitesimal generator of the subprocess given by a multiplicative functional

Suppose we have given the following Markov process

M = (xt , θt , Px)

thought of as the realization of a stochastic hybrid system H . Suppose that (M, α) is the subprocess ofM determined by thefollowing multiplicative functional

αt = exp(−At) (19)

where A = (At) is a right continuous positive (strong) additive functional [17]. We assume that A0 = 0 a.s. that impliesPx(α0 = 1) = 1 for all x ∈ X . We suppose that A is right continuous.

The additive function A will be used to produce aMarkov additive process (MAP), as follows

(M, A) := (xt , At , θt , Px).

In general, a Markov additive process is a ‘‘two dimensional’’ process

(M, Y ) = (xt , yt , θt , Px)

where M = (xt , θt , Px) is a Markov process and Y = (yt) is a process defined on a fixed Euclidean space Rp with‘‘conditionally independent increments given the paths ofM ’’, i.e.

Px(xs θt ∈ C, ys θt ∈ B|Ft = Pxt (xs ∈ C, ys ∈ B

for each t, ≥ 0, x ∈ X∆, C ∈ B(X∆), B ∈ B(Rp).The Lévy system of the Markov additive process (M, A) is defined to be a pair (H,N), where H is an increasing additive

functional ofM and N is a stochastic kernel from (X, B(X)) into

(X × R+, B(X) ⊗ B(R+))

provided that, for any non-negative B(X) ⊗ B(X) ⊗ B(R+)-measurable function, the following identity holds for all x ∈ Xand t > 0

Ex

−s≤t

f (xs−, xs, ys − ys−)1xs−=xs or ys−=ys

= Ex

∫ t

0dHs

∫X×R+

f (xs, x, u)N(xs, dx, du)

.


For a stochastic hybrid processM , a non-continuous choice for the above additive functional H is:

Ht = Hspontt + H forced

t

where

Hspontt =

∫ t

0λ(xs), H forced

t =

−Tk≤t

1[x−Tk

∈∂X].

This Ht counts for the spontaneous jumps (through Hspontt ) and for the forced jumps (through H forced

t ) of M . Asking for H tobe continuous means that the forced (predictable) jumps are not allowed.

The existence of the Lévy systems of the Markov additive processes have been treated in a rather general setting in [28].In general, when A is increasing, the additive functional A can be decomposed as follows:

A = Ac+ Ap

+ Aq,

where Ac is continuous, Ap is predictable (that is, its jump time are predictable stopping times of M), and Aq is quasi-leftcontinuous. Note, that the only predictable stopping times of a stochastic hybrid process M are the jumping times thatappear in the discrete forced transitions (when a continuous diffusion path of the system hits the boundary of the operatingmode).

In order to find the expression of the infinitesimal generator of the subprocess Mα , we make the following assumptionabout the continuous part of A:

Assumption 5. Suppose that there exists a continuous bounded function a : X → R such that

Ac=

∫ t

0a(xs)ds.

Often, the function ahas the interpretation of a killing rate for the processM . The expression of the infinitesimal generatorLα of the subprocess Mα will depend on the kernel N that is the component of the Lévy system of the MAP (M, A) and ona. The dependence of Lα on the discrete transitions of Mα is realized through the kernel N . The following result is takenfrom [29] that is the only reference where we found a complete treatment of the generator of the subprocess determined bymultiplicative functionals. Most of the papers existing in the literature that develop the theory of such subprocesses workmostly with the concepts of operator semigroup or operator resolvent. The expression of Lα will enable us to write thedynamic programming equations for the constrained reachability problem.

Proposition 9. Let L be the infinitesimal generator, given by (3), of a stochastic hybrid process M and let f ∈ D(L). Supposethat the Assumption 5 is in forced. Then f ∈ D(Lα) and for every x ∈ X the following identity holds:

Lα f (x) = Lf (x) − f (x)a(x) −

∫X×R+

[1 − e−s]N(x, dy, ds)

(20)

where N is the Lévy kernel of the MAP (M, A).

6.2. Dynamic programming for constrained reachability

The constrained reachability problem for a stochastic hybrid process M (realization of an SHS H) can be reduced tothe study of the ‘‘classical’’ reachability problem corresponding to the hybrid subprocess corresponding to a multiplicativefunctional of M . Let us suppose that, we have a random time Tα associated to a MF α of M . The subordinate or subprocessof M determined by α, denoted by Mα , is also a stochastic hybrid process whose trajectories coincide with the trajectoriesofM up to the time T .

The following result is just a simple consequence of the definitions of the concepts that describe the subprocessMα , butit is important because it gives a characterization of the constrained reachability in terms of the subprocessMα .

Proposition 10. For any Borel measurable set A of X, the following formula holds Mα

Px[TA < Tα] = Qx[Tα

A < Sα], x ∈ Xα (21)

where

• Xα is the set of permanent points of α given by (31),• Tα

A is the first hitting time of A (in fact of A ∩ Xα) corresponding to Mα ,• Sα is the life time of α defined by (30), and• Qx represent the Wiener probabilities of Mα defined by (32).


Let us make the following notation for the constrained reachability function

wA(x) :=

Qx[Tα

A < Sα] x ∈ Xα

0 x ∈ X \ Xα.(22)

For A ∈ B(X), let us denote by g := 1A its indicator function.

Assumption 6 (About Boundary). To avoid the complications related to the boundary condition (5) of the domain of thegenerator, we suppose that either one of the following assumptions is true:(A1) if A ∩ ∂X = ∅ then the hybrid process never jumps in A, i.e. the reset map R satisfies the condition

R(x, A) = 0, ∀x ∈ ∂X .

(A2) if A∩ ∂X = ∅ then the hybrid process has a jump/switching from the boundary of A into A, i.e. the reset map R satisfiesthe condition

R(x, A) = 1, ∀x ∈ A ∩ ∂X .

In [13], it has been proved that the reach function (10) corresponding to a measurable set A is nothing else but the valuefunction for an optimal stopping problem for which the reward function is exactly the indicator function of A. Because wehave reduced the constrained reachability problem to the reachability problemof a subprocess of the initial stochastic hybridprocess, we can now characterize the constrained reach function wA given by (22) as the value function for the followingoptimal stopping problem.

Let Σα denote the set of stopping times (finite or not) of the process Mα . Consider the reward function g : X → R equalto the indicator function of A. Let (yt)t≥0 be the reward process defined by

yt = g(xαt ), t ≥ 0.

Themaximal payoff function (or the value function) is defined as:

Φ(x) := supEαx yτ |τ ∈ Σα

(23)

where Eαx is the expectation with respect to Qx. It is known that the connection between the expectations of the initial

stochastic hybrid process M and the expectations of its subprocess Mα can be derived from the expression (33) of theoperator semigroup corresponding toMα as follows:

Eαx (f (xα

t )) = Ex(f (xt)αt).

Proposition 11. If A ∈ B(X) then the reachability functions wA coincides with the value function of the optimal stoppingproblem corresponding to the reward process yt = 1A(xα

t ), i.e.

wA(x) = supQx(xατ ∈ A)|τ ∈ Σα

, ∀x ∈ X .

Characterizations of the value function corresponding to an optimal stopping problem as solution of some variationalinequalities abound in the literature. For stochastic hybrid processes we refer to [13,5] and the references therein.

Remark 2. The dynamic programming equation associated with value function (23) of the optimal stopping problem forthe hybrid subprocessMα is

min(−Lαu, u − g) = 0 in X, (24)

u(x) =

∫Xu(y)R(x, dy) on ∂X (25)

where, Lα is the generator associated toMα , given by (20).The existence of the viscosity solution of the Eq. (24) with the boundary condition (25) requires some assumptions about

• the diffusion terms to be non-degenerate;• the reset kernel R of the underlying stochastic hybrid process to provide a bounded linear operator.

Note that different and complex hypotheses regarding the parameters of a stochastic hybrid process may be necessaryin order to obtain smoothness properties of the value functions, sharp estimations of the reach probabilities, etc.

7. State-constrained reachability in a development setting

So far, the state constrained reachability has been defined and investigated in the abstract setting provided by the SHSmodel.We have shown that the reach probabilities can be computed as a solution of an elliptic integro-differential equation.


This problem can be efficiently solved in some cases, for example in the case when the equation can be reduced to an ellipticone. In this case, dedicated analogical hardware exists [30], and even optical computation can be used [31]. However, forcomplex systems, the integro-differential equation can be difficult to solve. In this case, the abstract model of SHS should bereplacedwithmoremanageablemodels. This can be done inmanyways, like using approximations, functional abstractions,model reductions, and so on. Based on existing research we propose a multilayer approach for describing a complex hybridsystem with stochastic behavior. The same system is described using a set of different models, each one constructed at adifferent level of abstraction. The models can be related by abstraction or refinement maps. This approach makes possibleto solve a specific problem for the given system at the right level of abstraction.

In this setting, we search for more manageable solutions for the state constrained reachability analysis and we give anapplication to trajectory design and optimization.

7.1. Multilayer stochastic hybrid systems

In this subsection, we introduce amultilayermodel for stochastic hybrid systems. This is inspired by the viewpoints modelfrom software engineering. There, a system is modularly developed from different perspectives. Each perspective providesa model of the system called viewpoint. Then, the viewpoints need to be consistently unified to provide the overall systemdescription. This methodology corresponds to a horizontal development philosophy, and for SHS has been introduced andstudied in [32]. In this paper, we proposed a vertical (or hierarchical) viewpoint approach corresponding to a methodologyin software engineering described in [33]. In this approach, the system is described by viewpoints constructed on top ofeach other, each one providing a partial model at a different abstraction level.

Mathematically, at the level j, a viewpoint is an SHS

H j:= ((Q j, dj, Xj), bj, σ j, Init j, λj, Rj)

and all its elements (discrete/continuous states, trajectories, jumping times, etc.) carry the superscript j.At the level 0, the corresponding viewpoint is an SHS

H0:= ((Q 0, d0, X0), b0, σ 0, Init0, λ0, R0).

The viewpoint H j is related to viewpoint of level (j − 1)

H j−1:= ((Q j−1, dj−1, Xj−1), bj−1, σ j−1, Init j−1, λj−1, Rj−1)

by a pair of maps (Φ, Ψ ), where

Φ : X j−1→ X j

relates the states, and

Ψ : Ω j→ Ω j−1

relates the trajectories. In relational algebra, such a pair is called a ‘‘twisted relation’’.The first map Φ is a surjective map that describes how H j simulates H j−1 by means of the following property

Lj−1(f Φ) = Ljf Φ, ∀f ∈ B(X j) (26)

where Lj−1 and Lj represent the infinitesimal generators for H j−1, and respectively, for H j. The relation (26) can be given,as well, in terms of transition probabilities or operator semigroups as follows

P j−1t (f Φ)(u) = (P j

t f )(Φ(u)), ∀f ∈ B(X j), u ∈ X j−1.

The second map Ψ can be defined in various ways, adding flexibility to the viewpoint’s modeling. For example, Ψ canbe generically defined by replacing a certain discrete transition in the dynamics of the viewpoint jwith a set of trajectoriesof the hybrid system described by the viewpoint (j − 1). In particular, a single discrete transition can be mapped into thetrajectories of a continuous dynamical system. In this way, a viewpoint described by a discrete transition system can berelated to another viewpoint described by a hybrid system. In a probabilistic setting, this sort of relationship has been studiedin detail in [34].

To illustrate the viewpoint approach let us consider a flying aircraft. At the lowest level of detail, its dynamics can beaccurately described by a switching diffusion process. In this model of SHS, the hybrid trajectories are continuous and arepiecewise diffusion paths. At the most abstract level, the flight can be modeled as a probabilistic timed automaton that isa sort of discrete transition system. In this viewpoint, the aircraft lies in a state for a certain time, and then with a givenrate it makes a discrete transition. An intermediate viewpoint in between the continuous and discrete ones, is modeledby a stochastic hybrid system. In the intermediate viewpoint, a discrete transition from the discrete viewpoint is refinedinto a continuous mode, and certain continuous paths from the continuous viewpoint are abstracted away into discretetransitions. Of course, in the stochastic setting there can be many subtle cases like rates of discrete transitions that dependon the diffusion evolution in a mode.

The utility of a multilayer model consists in the possibility to solve categories of problems at different levels/viewpoints.For example, the stability problems are more efficiently studied in a continuous viewpoint (corresponding to the lowest


abstraction level). A safety verification problem can be formally tackled in a discrete viewpoint (corresponding to the highestabstraction level). Many control problems can be suitable studied in the hybrid viewpoint (corresponding to an intermediatelevel of abstraction).

The state constrained reachability has been defined in a viewpoint corresponding to an intermediate discrete/continuouslevel of abstraction. Since in the previous sections, it has been proved that the approach in this viewpoint can lead toproblems with integro-differential operators that can be difficult to solve, it is worthy trying to study the problem at otherlevels of abstractions/viewpoints. In a discrete viewpoint, one can hope to use probabilistic model checking techniques.In a continuous viewpoint, the well-developed mathematical apparatus of diffusion processes is becoming available withspecific benefits.

In order tomake the state constrained reachability approach practical, a further refinement of themathematical model isnecessary. This refinement takes into account the Euclidean space, in which processes evolve.We call such processes spatialSHS. An SHS is called spatial if some of its parameters form together a subspace of the Euclidean spaces R, R2, or R3.

A multilayer model can be fruitfully in conjunction with spatial models for designing, or improving the control. Supposethat a system, which is an n-dimensional SHS, is a spatial process in a higher level of abstraction obtained by ignoring thenon-spatial parameters. In this way, one can obtain a viewpoint in which the system is modeled as a spatial Wiener process.The state constrained stochastic reachability problem becomes more tractable in this viewpoint, if the reach probabilityis high, and causes of this fact can be detected, then in the original viewpoint a control strategy can be considered thatminimizes the reach probability. For example, in the case of an air traffic control system, the spatial viewpoint can indicatethat the collision probability becomes higher in areas of dense traffic with no coordination. Then a control strategy willdesign a pathway for the aircraft that avoid the high traffic density regions.

The following two examples are inspired from [16].

Example 1. Let A be the sphere with radius ε and the centre at the origin of R3. Let us consider the continuous viewpointof an SHS modeled as a 3D Wiener process W with W0 = x0 ∈ A. The problem is to compute the probability that W visitsA. Let B be a sphere with the radius R and centre at the origin, where R is much larger than ε (ε ≪ R). We have to look for asolution of the Laplace’s equation in spherical polar coordinates:

∂

∂r

r2

∂p∂r

+

1sin θ

∂

∂θ

sin θ

∂p∂θ

+

1sin2 φ

∂2p∂φ2

= 0, (27)

subject to boundary conditions

p(x) =

1 if x ∈ A0 if x ∈ B.

Solutions for the Eq. (27) with spherical symmetry have the form

p(x) =c1r

+ c2 if x = (r, θ, φ).

Using the boundary conditions, the following solution can be obtained [16]:

pR(x) =r−1

− R−1

ε−1 − R−1.

Making R → ∞, we get

pR(x) → P(TA < ∞) =ε

r, r > ε.

Example 2. Let consider a 2D Wiener process W with W0 = x0, that can be thought of as another spatial viewpoint for anSHS. Again we use the polar coordinates (r, θ), and suppose thatW evolves in the set

−π < −α ≤ θ ≤ α < πT .

If A is the line θ = α, and B is the line θ = −α, we may ask for the probability thatW reaches A before B. We consider nowthe planar Laplace equation in polar coordinates

1r

∂

∂r

r∂p∂r

+

1r2

∂2p∂θ

= 0

with the boundary conditions

p = 1 on θ = α, and p = 0 on θ = −α.

It can be checked [16] that the function

p =θ + α

2α, −α ≤ θ ≤ α


is the required solution, and so

p(x1, x2) =12α

α + tan−1 x2

x1

.

Example 3. Let us consider a discrete time Markov chain that can be thought of as a discrete viewpoint for an SHS. In theliterature, the number of transitions (time) required before the state will move from i to j for the first time is referred to asthe first passage time. It is possible to calculate the average (or expected) number of transitions for the passage from state ito j. Let mij be the expected first passage time (number of transitions from state i to j). The probability of moving from i to jis pij took exactly one transition. If this is not the case then the state will change to k (= j). The probability of moving from ito k (for all k = j) would be the sum of all the probabilities pik for all k (= j), i.e.

∑k=j pik. We now need to move from i to j.

This may require many transitions and based on the Markov property, it is given by,∑

k=j pikmkj. Then

mij = pij +−k=j

pik +

−k=j

pikmkj

or, finally

mij = 1 +

−k=j

pikmkj.

7.2. Electrostatic analogy

It is known that the theory of Markov processes is intimately connected with the mathematical physics—see [17]. Thesolutions of the DBVP from the Theorem 5 can be characterized as some potential functions, defined on the underlyingMarkov process state space. These describe the ‘probability distributions’ of a charge that is distributed on the state spacewhere the set A (the obstacle) produces a repulsive force and the set B (the target) yields an attractive force.

In potential theory, the physical interpretation of the state-constrained reachability probability considered in this paperis related to the condenser problem—see [35,36]. This is described as follows: suppose there are given two disjoint compactconductors A, B in the Euclidean space R3 of positive capacity [37]. A positive electric unit charge placed on A and anegative unit charge on B, both allowed to distribute freely on the respective sets, will find a state of equilibrium, whichis characterized on one hand byminimal energy, and on the other hand by constant potential on A and on B (possibly takingout exceptional sets of zero capacity).

7.3. Trajectory design and optimization

Trajectory design in an abstract model is alternative formulation for the control problems. A system must follow somedesirable behaviors and a controller is designed to achieve that. The controller’s activity can be modeled by a process ofdesigning a ‘‘good’’ trajectory. With this respect, an analogy with the motion planning problem for robots can be useful.In this subsection, we use state constrained reachability for designing suitable trajectories in the SHS setting. The reachprobabilities are used to transform the original system state space into a new form called its uncertainty representation.For this representation, we use the physical analogy to design a trajectory. This trajectory, in the original state space, can beconsider as a system evolution that minimizes the risk modeled by the reach probabilities.

The trajectory design problem has to consider the following objects [38]:• an initial state and a final goal for the agent;• the state space of the agent (or area of operation);• the set of all possible actions that the agent is allowed to take;• a cost function that calculates the path efficiency.

The first step in trajectory design is to replace the state space with a mathematical model called the uncertaintyrepresentation. In fact, for practical reasons we will consider a complementary value of the obstacle avoidance probability(i.e the waypoint’s reachability probability).

Definition 3. The uncertainty representation of a stochastic hybrid systems is obtained by attaching to each state x the value1 − p(x) of the state constrained reachability probability p(x).

This representation depends on the target state set and the forbidden state region. When the target set is mobile, thisrepresentation is dynamic. A geometrical characteristic of this representation is the presence of ridges and valleys, with themaximumheight being one, and the lowest value zero. The forbidden regions will form the top of the ridges, as illustrated inFig. 3. The lowest valleys will represent the safest (less risky) areas of the state space that will lead to the target set. Anothergeometrical characteristic of the representation is the absence of saddle points as a consequence of the mini–max principleof the solutions of the boundary value problem.


1

0–2

–10

1

2

y

2

10

–1–2

x

Fig. 3. Trajectory in the uncertainty representation.

Example 4. Examples of situations when the uncertainty representation is useful are various:

– Consider a plane flying in stormweather conditions. The plane experiences a degree of turbulence where the clear spaceis diffused into storms with no sharp boundaries separating them.

– Consider a military unmanned aerial vehicle (UAV) passing through a hostile space. The UAV has incomplete knowledgeabout the placement of the enemy artillery. Then, the UAV will be helped to have a description of the state space on thedegree of possible harm.

– For a mobile robot operating in rough terrain, it is not useful to construct a binary description of the terrain usingforbidden and admissible areas. It is more useful to have a description based on the degree of difficulty of negotiating theterrain.

– The velocity of wind in a storm can be converted in a probability distribution function representing the suitability ofspace for navigation.

One way to construct a trajectory in the space described by uncertainty representation is to use a physical metaphor.Suppose that the system is a point with mass moving under gravity forces in the landscape provided by the uncertaintyrepresentation. This mass will naturally move away from the ridges and search for the lowest valleys. The simplest wayto simulate this movement is to use the gradient descent, a standard method for finding the local minimum of a function.Because the probabilities are potential functions (as solutions of a Dirichlet boundary value problem), they do not achievelocalminima, only a global one. In thisway, themethodwill generate a trajectory towards the target set free of saddle points.The lack of saddle points guarantees that mass will never get trapped into sinks and that it will always get to the target. Aset of techniques for path planning in the 3D terrains with ridges and valley have been developed in [39]. Interpreting theuncertainty representation as a 3D terrain, these techniques can be applied quite straightforward to generate a trajectory inthe uncertainty space.

Although natural and direct, this method does not provide the quickest way to descend from a ridge. In the original statespace, this means that trajectories will get close to the contours of the forbidden regions. The trajectories constructed in thisway are not the safest possible and are not reliable in case of moving forbidden areas. Other well known deficiencies of themethod are that it is relatively slow and that the generated trajectory might ‘zigzag’ down valleys.

New possibilities arise by using different physical metaphors. Another physical analogy appears when the gravitation isreplaced by electric fields. Imagine that the system is a charge without mass, moving under the influence of attracting andrepulsive forces generated by electric fields in the uncertainty space. The target state set has attached an attractive field andthe forbidden regions have attached repulsive fields. The reach probabilities will be used to construct the fields in a mannerthat will be rigorously described below.

An autonomous systemhas to determine the safest path for a vehicle tomove through the area of operation to accomplisha given mission.

Let us suppose the dynamics of an autonomous system (an agent), in different operational modes, can be described by ahybrid system. For example, anUAVdynamics can be described by a hybrid system thatmakes discrete transitionswheneverit reaches a waypoint. Considering different randomness factors (environment, noisy measurements, communicationfailures, etc.), wemay use a randomized version of the given hybrid system. Of course, the agent dynamics can be thought ofas a particularization of the SHS general model. For example, one can easily imagine that the agent has no proper jumps, butonly switchings from one continuous path to another. This randomized model of hybrid systemwill play a reference role. Itwill describe the desired dynamics. The description of this hybrid system should come with a fairly good representation ofits state space. That means we need to have the locations of threats, obstacles, and restricted fly areas.

Formally, suppose that the agent dynamics can be described by a stochastic hybrid process M with the state space X .The obstacle is represented by a measurable set A, and the goal is given by a measurable set B. Therefore, we can use as


uncertainty representation of X the following family of probability distributions

pBA(x)|x ∈ X (28)

where pBA(x) is the solution of the Dirichlet boundary value problem (13). The aim of the trajectory design problem for thisagent would be to find ‘‘trajectories’’ in the space (28) with low collision probability. Then, on this space, we can define anharmonic potential field (HPF) method that aims to identify the paths with the lowest obstacle collision probabilities. In thissetting, the potential field equations will be∇(1 − pBA(x))∇V (x) = 0 x ∈ X \ (A ∪ B)

V (x) = 1 x ∈ AV (x) = 0 x ∈ B.

(29)

A provably-correct path may be generated using the gradient dynamical system:.x= −∇V (x).

The modified differential operator in (29) will expand in

(1 − pBA(x))∇2V (x) + ∇(1 − pBA(x))∇V (x) = 0

which leads to

∇2V (x) = −

1(1 − pBA(x))

[−∇(1 − pBA(x))][−∇V (x)].

Notice that −∇V (x) is the direction at which motion is to be driven and −∇(1 − pBA(x)) = ∇pBA(x) is a vector pointing inthe direction of increasing risk.

The trajectories are generated using sampling. At a given state, the direction of moving is chosen by picking a state in theneighborhood with the smallest associated probability. In practice, the probabilities will not be computed for each state inthe neighborhood, but only for a finite set of states chosen according to a sampling policy. For example, a sampling policy inthe plane would be to construct a circle in a suitable metric and pick a point on this circle. The remaining states are obtainedby picking new points for each 45 degrees. Computationally, eight probabilities will be computed at each step. This samplingmethod could be made adaptive by considering the values of the probabilities on samples. If these values are small, fewersampleswill be necessary in the next step. To the contrary, for high values of the probabilities, more sampleswill help to finda quicker way down the ridge. The path generated by (29) will avoid the states from where the agent with a.s. probability 1will collide with obstacle, i.e. the state for which the pBA(x) = 1.

In Fig. 3. uncertainty representation of the state space corresponds to the graph of the harmonic function Re u(x, y)where

u(x, y) =e−(x+yI)2

(x + yI)2+ (x + yI)4

and x, y vary within the interval [−2, 2].The system’s trajectory is depicted as a thick curve. It can be observed that the trajectory follows a principle of getting

quicker down the ridges on to the ‘‘valleys’’.The uncertainty representation has an extra-dimension when compared to the state space. The generated trajectory will

have also an extra-dimension. The trajectory will carry, in addition to the state parameters, information on the uncertaintymeasure (in our case, the state constrained reach probability). One possibility to construct a trajectory in the original statespace is to construct an intermediatemodel, called colored SHS. In practice, only a finite number of values of state constrainedreachability probability can be computed. Therefore, the uncertainty representation of the state space will be a map coloredwith a finite number of colors.

Definition 4. A colored stochastic hybrid system is a collection

(H, C, )

where

• H := ((Q , d, X), b, σ , Init, λ, R) is a general stochastic hybrid system• C is a finite set, whose elements are called colors• : X → C is a function that we call coloring.

A colored stochastic hybrid system can be attached to its uncertainty representation by defining a color to the a value setof the state constrained reachability probability. In this way, coloring can be used as an effective tool for model reduction.


AA

A

A

A

A

x

B

Fig. 4. Trajectory in the colored state space.

Example 5. An example of colored SHS is illustrated in Fig. 4. In this example five colors are used for the uncertaintyrepresentation: the target state set is represented in light blue, the obstacle state set (which, in this example, is notconnected) is represented in black. The probability represented in the figure is the complementary ϕB(x) − pB

¬A = pBA ofthe obstacle avoidance probability pB

¬A (of reaching B, starting from x, by avoiding A). This probability has the value 1 onthe black colored areas (corresponding to the obstacles) and null value on the light blue colored area (corresponding tothe target). The areas colored in white, grey, dark grey and dark blue correspond to intermediate, increasing values of thecomplementary of the constrained reachability probability. For example, the white color corresponds to the states with theconstrained reachability probability values ranging from 0.001 to 0.250. The grey color corresponds to the states with theconstrained reachability probability values ranging from 0.251 to 0.700. Similarly, the dark grey colored states correspondto probability values ranging 0.701 and 0.850, and the blue colored states correspond to probability values ranging 0.851and 0.999. The trajectory is depicted in red. Remark that the trajectory ‘‘tries’’ to get quickly out of the dark colored stateareas.

Example 6. An immediate example at hand for colored state spaces is the case of flight within a geographical areacontaminated with ash clouds. It is reasonable to consider the probability directly proportional to ash concentration. TheNorth Atlantic Operations Bulletin 2010-009 imposes, from 16 May 2010: ‘‘Areas of Low Contamination’’—areas where it isforecast that the concentration of volcanic ash will be below 2×10−3 g/m3; ‘‘Areas of High Contamination’’—the forecastedconcentration of ash is between 2–4 mg/m3 and ‘‘No Fly Areas’’. These areas are modeled by colored sets in our model. Thelight blue state sets will be the surrounding area of the destination airport. The model states corresponding to the ‘‘No FlyAreas’’ will be considered colored in black. The area of high contamination will form the blue colored state regions. Thewhite state areas correspond to the physical areas of no contamination. The grey colored state regions will correspond tothe areas of low contamination.

It is important to underline that, in the previous example, the colors are dynamic (they change over time correspondingto the volcanic ash concentration determined by the ash clouds dynamics). The colored SHS model can be extended to copewith this situation by considered a time indexed family of coloring functions

s : X → Cs∈S⊂[0,∞).

A similar colored model, but in a different context, has been considered in [40]. The randomized approach from Section 5needs to be extended to randomized obstacles.

In the end of this section,we need tomention that a version of the state constrained reachability has been used for aircraftconflict prediction in [41], in a low abstraction level, i.e. the underlying model is a switching diffusion process.

8. Final remarks

The stochastic hybrid systems constitute well-established classes of realistic models of hybrid discrete/continuousdynamics subject to random perturbations, autonomous uncontrollable transitions, nondeterminism or uncertainty.Stochastic reachability analysis is a key factor in the verification and deployment of stochastic hybrid systems. Theencouraging recent progress led us to refine the problem to cover more realistic situations. We have extended the so-calledconstrained reachability problem from the probabilistic discrete case to stochastic hybrid systems. Then we have definedmathematically this problem, and we have obtained the reach probabilities as solutions of a boundary value problem andof some appropriate dynamic programming equations. The last problem is well studied and numerical, and even symbolicsolutions exist. These characterizations are useful in stochastic control and in probabilistic path planning. In this paper, thestochastic reachability problem for stochastic hybrid systems has been specialized by introducing constraints relative to thestate space. We have proved that the state-constrained stochastic reachability is solvable.


The major motivation for the approach presented in this paper comes from the area of autonomous open systems, in thelarger context of cyber physical systems. A cyber physical system is an interactive system with a complex behavior given bya rich interaction between physics and computation. Hybrid and real time systems can be thought of as abstract models ofcyber physical systems. An open system is a system that interacts with its environment. Safety analysis for open cyber-physical systems requires new techniques since safety verification methods for hybrid automata are based on a closedworld assumption. The behavior of an open system is affected by randomperturbations from its environment. Unpredictableeffects of large deviationsmake the traditional verificationmethods, likemodel checking and theorem proving, inapplicableand require a statistical approach for safety verification. In this line, we propose a stochastic certification method thatcharacterizes the reach probabilities for complex state configurations. In this way, the safety certification of open cyberphysical systems can be carried out numerically. When such systems are deployed in a dynamic random environment, theevaluations of the reach probabilities have to be iterated, and the control policies updated. This procedure is the key point forsafety certification of autonomous systems, i.e. self-managed systems. Autonomy is an important feature in modern systemengineering and the safety verification of autonomous systems is still unclear.

The applications of reachability analysis are ubiquitous. Its stochastic version constitutes a relaxed and statistical versionused for risk and safety assessment. The state constrained stochastic reachability analysis and its applications in control viamultilayer modeling and abstract trajectory design raises new application areas. More specifically, these are suitable forareas where objects have a imprecise contour like clouds, approximate positioning, or human organs. In some sense, thedevelopments presented in this paper give an alternative to the fuzzy logic based techniques. Case studies can be drawnfrom flight control and navigation in adverse weather conditions, in minimal invasive surgery, or in satellite docking.

The preliminary results of this paper were announced in [42].

Acknowledgement

This work was funded by the EPSRC project CICADA, ref. EP/E050441/1.

Appendix

This section contains a collection of standard concepts of Markov processes theory.

Multiplicative functional definition

A real-valued process α = (αt)t≥0 is called amultiplicative functional (MF) of a Markov processM = (xt , θt , Px) if:

1. t −→ αt(ω) is decreasing, right continuous and has values in [0, 1] for each ω ∈ Ω;2. α is adapted, i.e. αt is Ft-measurable for any t ≥ 0 (where Ft is the natural filtration ofM);3. αt+s(ω) = αt(ω) · αs(θtω) for any s, t ≥ 0 and ω ∈ Ω .4. In addition, an MF α is exact provided that for any t ≥ 0 and every sequence tn 0,

αt−tn → αt a.s. as n → ∞.

Here, θt is the time shift operator defined by (17).The set of all exact multiplicative functionals ofM is denoted byMF . For any α ∈ MF we make the following notations:

• the life time of α

Sα := inft ≥ 0|αt = 0 (30)

• the set of permanent points of α

Xα := x ∈ X |Px(α0 = 1) = 1. (31)

Xα is also the set of irregular points of Sα , i.e.

Xα = x ∈ X |Px(Sα > 0).

Wemake the following notation:

MF+ := α ∈ MF |Sα > 0 a.s..


Subprocesses

On the space Xα , we define theWiener probabilities Qx for the subordinate process, as follows:

Qx(A) := Px

∫∞

01A ktd(−αt), A ∈ F0 (32)

where α∞ := 0 and (kt)t≥0 are the killing operators on Ω defined by

kt(ω)(s) =

ω(s) if s < t∆ if s ≥ t.

If α is a right multiplicative functional, then (Ω, xt ,Qx) is also a right Markov process called the α-subprocess of M anddenoted by Mα or M with the state space (Xα, B(Xα)). We can make the convention that Qx(A) = 0 if x ∈ X \ Xα . Theoperator semigroup ofMα is given by

Qt f (x) = Px[f (xt)αt ]. (33)

Remark 3. If T is a terminal time, then 1[0,T )(t) is an MF for the process M. The life time of a multiplicative functional forthe processM is a terminal time forM .

Additive functionals

Let α be an MF. Following [17], we define an α-additive functional ofM as a positive increasing right continuous processA = (At)t≥0 provided that:

• At < ∞ for t < Sα ∧ ζ ;• for each t and s;

As+t = At + Mt · As θt , a.s. (34)

• A is adapted.

We denote by AF(α) the set of α-additive functionals, and AF for AF(1) the set of α-additive functionals of the process M .An additive functional is called strong additive functional if the additivity condition (34) holds also for the case when t isreplaced by a stopping time of the underlying Markov process.

Representation of multiplicative functional

It is known from the theory of themultiplicative functionals, that, under some standard assumptions about theunderlyingMarkov process, any α ∈ MF+ has the following decomposition:

αt := Ψt · exp−

∫ t

0a(xs)dAs

· 1[0,JB)(t) (35)

where

• Ψt = Π0<s≤t(1 − Φ(xs−, xs)) with 0 ≤ Φ < 1, Φ is B × B-measurable and vanishes on the diagonal D of X × X;• a is a positive B-measurable function;• A is a continuous additive function ofM .

Remark 4. B is a Borel subset of X × X which is disjoint from D and

Sα = JB := inft > 0|(xt−, xt) ∈ B.

Obviously, the decomposition (35) of a multiplicative functional is closely related to the expression of the survivor function(1) used in the definition of a stochastic hybrid process to obtain the jumping/switching times from one mode to another.One can think at a generalization of the definition of a stochastic hybrid process replacing the survivor function with a moregeneral multiplicative functional. In this way, the dynamics in modes may have somemathematical discontinuities and thejumping/switching times could depend on these discontinuities.

References

[1] C.G. Cassandras, J. Lygeros, Stochastic Hybrid Systems: Research Issues and Areas, CRC Press, 2007.[2] M.L. Bujorianu, J. Lygeros, Reachability questions in piecewise deterministic Markov processes HSCC, in: Springer Verlag Lectures Notes in Computer

Science, vol. 2623, 2003, pp. 126–140.


[3] M.L. Bujorianu, J. Lygeros, New insights on stochastic reachability, in: Proc. of the 46th Conference on Decision and Control, 2007, pp. 6172–6177.[4] M.L. Bujorianu, A statistical inference method for the stochastic reachability analysis, in: Proc. of the. 44th IEEE Conference on Decision and Control:

CDC-ECC’05, 2005, pp. 8088–8093.[5] M.L. Bujorianu, J. Lygeros, R. Langerak, Reachability analysis of stochastic hybrid systems by optimal control HSCC, in: Springer Verlag Lectures Notes

in Computer Science, vol. 4981, 2008, pp. 610–613.[6] H.A.P. Blom, G.J. Bakker, J. Krystul, Probabilistic reachability analysis for large scale stochastic hybrid systems, in: 46th IEEE Conference on Decision

and Control, 2007, pp. 3182–3189.[7] J. Krystul, H.A.P. Blom, Sequential Monte Carlo simulation of rare event probability in stochastic hybrid systems, in: 16th IFACWorld Congress, 2005.[8] M. Prandini, J. Hu, A stochastic approximation method for reachability computations, In [43], 2006, pp. 107–139.[9] X. Koutsoukos, D. Riley, Computational methods for verification of stochastic hybrid systems, IEEE Transactions on Systems, Man and Cybernetics.

Part A. 38 (2) (2008) 385–396.[10] C. Baier, B.J. Haverkort, H. Hermanns, J.P. Katoen, Reachability in continuous-time Markov reward decision processes, Logic and Automata: History

and Perspectives (2007) 53–71.[11] S. Summers, J. Lygeros, A probabilistic reach-avoid problem for controlled discrete time stochastic hybrid systems, in: IFAC Conference on Analysis

and Design of Hybrid Systems, 2009.[12] M.L. Bujorianu, J. Lygeros, General stochastic hybrid systems: modelling and optimal control, in: Proc. of 43th Conference in Decision and Control,

vol. 2, 2004, pp. 1872–1877.[13] M.L. Bujorianu, J. Lygeros, Towards modelling of general stochastic hybrid systems, In [43], 2006, pp. 3–30.[14] M.H.A. Davis, Markov Models and Optimization, Chapman and Hall, 1993.[15] S.N. Ethier, T.G. Kurtz, Markov Processes: Characterization and Convergence, John Wiley and Sons, New York, 1986.[16] G. Grimmett, D. Stirzaker, Probability and Random Processes, Oxford University Press, 1982.[17] R.M. Blumenthal, R.K. Getoor, Markov Processes and Potential Theory, Academic Press, 1968.[18] R.K. Getoor, Excursions of a Markov process, The Annals of Probability 7 (2) (1979) 244–266.[19] A. Bovier, Metastability, in: Springer Verlag LNM Series, vol. 1970, 2009.[20] M. Sion, On capacitability and measurability, Annales de l’Institut Fourier 13 (1) (1963) 83–98.[21] M.G. Garroni, J.L. Menaldi, Second Order Elliptic Integro-Differential Problems, Chapman & Hall/CRC, 2002.[22] K. Taira, Boundary value problems for elliptic integro-differential operator, Mathematische Zeitschrift 222 (1996) 305–327.[23] C. Barles, E. Chasseigne, C. Imbert, On the Dirichlet problem for second-order elliptic integro-differential equations. Preprint (2007).[24] J.R. Baxter, R.V. Chacon, Compactness of stopping times, Zeitschrift fur Wahrscheinlichkeitstheorie und Verwandte Gebiete 40 (3) (1977) 169–181.[25] E.B. Dynkin, Markov Processes, Springer Verlag, 1965.[26] H. Kunita, T. Watanabe, Notes on transformations of Markov processes connected with multiplicative functionals, Memoirs of the Faculty of Science,

Kyushu University Series A, Mathematics 17 (2) (1963) 181–191.[27] M.J. Sharpe, General Theory of Markov Processes, Academic Press, 1988.[28] B. Maisonneuve, Changement de temps d’un processus Markovien additif, in: Lecture Notes in Mathematicss, vol. 581, 1977, pp. 529–538.[29] H. Gzyl, On generator of subordinate semigroups, The Annals of Probability 6 (6) (1978) 975–983.[30] L. Tarassenko, A. Blake, Analogue computation of collision-free paths, in: IEEE Conference on Robotics and Automation, 1991, pp. 540–545.[31] M.B. Reid, Optical calculation of potential fields for robotic path planning, Applied Optics 33 (5) (1994) 881–896.[32] M.L. Bujorianu,M.C. Bujorianu, Viewpoint development of stochastic hybrid systems, in: Proc. of 45th IEEE Conference onDecision and Control CDC’06,

IEEE Computer Society Press, 2006, pp. 6241–6246.[33] M.C. Bujorianu, Integration of specification languages using viewpoints, in: Proceedings of Integrated Formal Methods: IFM’04, in: Springer Verlag

Lectures Notes in Computer Science, vol. 2999, 2004, pp. 421–440.[34] M.L. Bujorianu, M.C. Bujorianu, H.A.P. Blom, Approximate abstractions of stochastic hybrid systems, in: Proceedings of the 17th IFACWorld Congress,

Elsevier Science Press, 2008.[35] J. Bliedtner, M. Musat, The condenser problem, Potential Anal 21 (2) (2004) 177–192.[36] K.L. Chung, R.K. Getoor, The condenser problem, The Annals of Probability 5 (1) (1977) 82–86.[37] M. Ohtsuka, A general definition of capacity, Annales de l’Institut Fourier 25 (3–4) (1975) 499–507.[38] S.M. LaValle, Planning Algorithms, Cambridge University Press, 2006.[39] D.L. Page, A.F. Koschan, M.A. Abidi, J.L. Overholt, Ridge-valley path planning for 3D terrains, in: IEEE International Conference on Robotics and

Automation, 2006.[40] M.C. Bujorianu, M.L. Bujorianu, H. Barringer, A Formal framework for user centric control of probabilistic multi-agent cyber-physical systems,

in: Proceedings of CLIMA IX, Springer Verlag, 2008, pp. 97–116.[41] M. Prandini, J. Hu, Application of reachability analysis for stochastic hybrid systems to aircraft conflict prediction, IEEE Transactions on Automatic

Control 54 (4) (2009) 913–917.[42] M.L. Bujorianu, M.C. Bujorianu, State constrained reachability analysis, in: Proc. of the 2nd International Conference on Analysis and Design of Hybrid

Systems - ADHS, 2009.[43] H.A.P. Blom, J. Lygeros, Stochastic hybrid systems: theory and safety critical applications, in: Springer Verlag LNCIS Series, vol. 337, 2006.

state constrained reachability for stochastic hybrid systems

Documents