1

Robotic surveillance and Markov chains withminimal weighted Kemeny constant

Rushabh Patel, Pushkarini Agharkar, Francesco Bullo

Abstract—This article provides analysis and optimization re-sults for the mean first passage time, also known as the Kemenyconstant, of a Markov chain. First, we generalize the notion ofthe Kemeny constant to environments with heterogeneous traveland service times, denote this generalization as the weightedKemeny constant, and we characterize its properties. Second, forreversible Markov chains, we show that the minimization of theKemeny constant and its weighted counterpart can be formulatedas convex optimization problems and, moreover, as semidefiniteprograms. Third, we apply these results to the design of stochasticsurveillance strategies for quickest detection of anomalies innetwork environments. We numerically illustrate the proposeddesign: compared with other well-known Markov chains, theperformance of our Kemeny-based strategies are always betterand in many cases substantially so.

I. INTRODUCTION

A. Problem description and motivation

The subject of this paper is the analysis, generalization andminimization of the mean first passage time for a randomwalk. This problem is of general mathematical and engineeringinterest in the study of Markov chains and random walks;similar to the fastest mixing Markov chain, the mean firstpassage time is a metric by which to gauge the performance ofa random walk. In a robotic context, our motivating applicationis the design of surveillance algorithms for quickest detectionof intruders and anomalies. Specific examples include themonitoring of oil spills [12], the detection of forest fires[27], the tracking of border changes [39], and the periodicpatrolling of an environment [17], [35]. Other applications insingle and multi-agent systems include minimizing emergencyvehicle response times [5] as well as servicing tasks inrobotic warehouse management [41]. In this paper we proposestochastic surveillance strategies based on Markov chains.This study is motivated by the desire to design surveillancestrategies with pre-specified stationary distributions, that areeasily implementable and inherently unpredictable. In areasof research outside of robotics, the study of the mean firstpassage time is potentially useful in determining how quicklyinformation propagates in an online network [4] or howquickly an epidemic spreads through a contact network [40].

This work was supported in part by NSF grant CNS 1035917 and AROgrant W911NF-11-1-0092.

1 Rushabh Patel, Pushkarini Agharkar, and Francesco Bullo are withthe Mechanical Engineering Department and the Center for Control, Dy-namical Systems and Computation, University of California at SantaBarbara, CA 93106-5070, USA r_patel@engineering.ucsb.edu,agharkar@umail.ucsb.edu, bullo@enigneering.ucsb.edu

B. Literature review

For a random walk associated with a Markov chain, themean first passage time, also known as the Kemeny constant,of the chain is the expected time taken by a random walkerto travel from an arbitrary start node to a second randomly-selected node in a network. The Kemeny constant of a Markovchain first appeared in [26] and has since been studied byseveral scientists, e.g., see [25], [28] and references therein.Bounds on the mean first passage time for an arbitrary Markovchain over various network topologies appear in [25], [30]. Tothe best of our knowledge, no results are available so far onthe optimization of the mean first passage time of a Markovchain, nor on a framework for this quantity which accountsfor general weighted graphs.

The mean first passage time is closely related to other well-known metrics for graphs and Markov chains. We discuss twosuch quantities in what follows. First, the Kirchhoff index [29],also known as the effective graph resistance [15], is a relatedmetric quantifying the distance between pairs of verticesin an electric network. The relationship between electricalnetworks and random walks on graphs is explained elaboratelyin [14]. For an arbitrary graph, the Kirchoff index and theKemeny constant can be calculated from the eigenvalues of theconductance matrix and the transition matrix, respectively. Therelationship between these two quantities for regular graphs isestablished in [34]. Second, the mixing rate of an irreducibleMarkov chain is the rate at which an arbitrary distributionconverges to the chain’s stationary distribution [13]. It is well-know that the mixing rate is related to the second largesteigenvalue of the transition matrix of the Markov chain. Theinfluential text [31] provides a detailed review of the mixingrate and of other notions of mixing. Recently, [28] refers tothe Kemeny constant as the “expected time to mixing” andrelates it to the mixing rate.

In this paper we study strategies to surveil an environment,to provide a desired coverage frequency, and to detect anintruder in minimum time. The surveillance problem hasappeared in the literature in various manifestations. The au-thors of [38] look at minimizing time of detection of noisyanomalies via persistent surveillance strategies, and in [32]wireless sensor networks are utilized for intruder detection inpreviously unknown environments. In [3], the authors explorestrategies for surveillance using a multi-agent ground vehiclesystem which must maintain connectivity between agents. Anon-cooperative game framework is utilized in [11] to deter-mine an optimal strategy for intruder detection, and in [36]a similar framework is used to analyze intruder detection

2

for ad-hoc mobile networks. In our setup we model theenvironment as a graph and design random walks on thisgraph imposing restrictions on the stationary distribution ofthe walk. In other works with a similar setup, Markov chainMonte Carlo methods [19], [37] are used to design surveillancestrategies. In [19] convexity results for symmetric matricesare utilized to further optimize those strategies. Deterministicpolicies have been used to minimize the visit frequencies ina graph [16], [38], however, a main result of [37] shows thatdeterministic policies are ill-suited when designing strategieswith arbitrary constraints on those visit frequencies. We takean alternate approach and design policies using Markov chainswith minimal Kemeny constant.

C. Contribution

Before stating our contributions, it is worth mentioningthat all work to date on the mean first passage time andmixing rate of a Markov chain on a graph has been completeunder the assumption of homogeneous travel time along theedges of the graph. The contributions of this paper are thensix fold. First, we provide a convex optimization frameworkto minimize the Kemeny constant of a reversible Markovchain given the underlying graph topology of the randomwalk and the desired stationary distribution. Second, usingdoubly-weighted graphs we extend the formulation of themean first passage time to the network environments with non-homogeneous travel times, a generalization not yet looked at inthe literature. We denote this extension the weighted Kemenyconstant. Third, we derive a closed form solution for theweighted Kemeny constant and show its relation to the Kemenyconstant. Fourth, we provide a convex optimization frameworkto minimize the weighted Kemeny constant of a Markovchain with desired stationary distribution. Fifth, we provide asemidefinite program (SDP) formulation for the optimizationof the Kemeny constant and the weighted Kemeny constant.Finally, we look at two stochastic surveillance scenarios; inthe first scenario we provide a setup in which minimizingthe weighted Kemeny constant leads to the optimal Markov-chain strategy. In the second surveillance scenario we establishthrough numerical simulation that the Markov chain with theminimum weighted Kemeny constant performs substantiallybetter compared with other well-known Markov chains (i.e.,the fastest mixing chain and the Metropolis-Hastings Markovchain).

A preliminary short version of this article is to appear atthe 2014 CDC conference as [1] and is currently availableat http://motion.me.ucsb.edu. This article containsseveral addenda and updates not found in [1] including, butnot limited to, detailed proofs of theorems that were presentedin [1], the introduction of a new algorithm, and extensivesimulation results.

D. Organization

The paper is organized as follows. In Section I-E wesummarize the notation which we use throughout the paper andbriefly review properties of Markov chains. In Section II wegive relevant background for the Kemeny constant and present

our results for its minimization. In Section III we introduceand provide detailed characterization of the weighted Kemenyconstant as well as its minimization. In Section IV we providepractical surveillance applications of the weighted Kemenyconstant. In the final Section we present our conclusions andfuture research directions.

E. Notation

We use the notation A = [aij ] to denote a matrix Awith the element aij in its i-th row and j-th column and,unless otherwise indicated, use bold-faced letters to denotevectors. Letting δij denote the Kronecker delta, Ad = [δijaij ]represents the diagonal matrix whose diagonal elements arethe diagonal elements of the matrix A. The column vectorof all ones and length n is denoted by 1n ∈ Rn×1 and Irepresents the identity matrix of appropriate dimension. Weuse diag[b] to denote the diagonal matrix generated by vectorb and Tr[A] to denote the trace of matrix A.

A Markov chain is a sequence of random variables takingvalue in the finite set {1, . . . , n} with the Markov property,namely that, the future state depends only on the present state;see [23], [26] for more details. Let Xk ∈ {1, . . . , n} denotethe location of a random walker at time k ∈ {0, 1, 2, . . . }. Weare now ready to summarize some terminology and resultsfor Markov chains. (1) A Markov chain is time-homogeneousif P[Xn+1 = j|Xn = i] = P[Xn = j|Xn−1 = i] = pij ,where P ∈ Rn×n is the transition matrix of the Markov chain.(2) The vector π ∈ Rn×1 is a stationary distribution of Pif∑ni=1 πi = 1, 0 ≤ πi ≤ 1 for all i ∈ {1, . . . , n} and

πTP = πT . (3) A time-homogeneous Markov chain is saidto be reversible if πipij = πjpji, for all i, j ∈ {1, . . . , n}.For reversible Markov chains, π is always a steady statedistribution. (4) A Markov chain is irreducible if there existsa t such that for all i, j ∈ {1, . . . , n} , (P t)ij > 0. (5)If the Markov chain is irreducible, then there is a uniquestationary distribution π, and the corresponding eigenvaluesof the transition matrix, λi for i ∈ {1, . . . , n}, are such thatλ1 = 1, |λi| ≤ 1 and λi 6= 1 for i ∈ {2, . . . , n}. For furtherdetails on irreducible matrices and about results (4) and (5)see [33, Chapter 8]. In this paper we consider finite irreducibletime-homogeneous Markov chains.

II. THE KEMENY CONSTANT OF A MARKOV CHAIN ANDITS MINIMIZATION

Consider a undirected weighted graph G = (V, E, P ) withnode set V := {1, . . . , n}, edge set E ⊂ V × V , and weightmatrix P = [pij ] with the property that pij ≥ 0 if (i, j) ∈ Eand pij = 0 otherwise. We interpret the weight of edge (i, j) asthe probability of moving along that edge. Therefore, elementpij in the matrix represents the probability with which therandom walk visits node j from node i. Throughout thisdocument we assume that the underlying undirected graph(V, E) associated to the transition probabilities P is connected.

In this section we look into a discrete-time random walkdefined by a Markov chain on a graph G. At each time step(hop) of the random walk we move to a new node or stay at thecurrent node according to the transition probabilities described

3

by a transition matrix P as discussed above. We do this withthree objectives in mind. The first objective is to analyze therandom walk and characterize the average visit time betweennodes in the graph. The second objective is to minimize theaverage visit time between any two nodes and the final is toachieve a long term (infinite horizon) visit frequency πi atnode i. Here, the frequency πi is the ratio of visits to nodei divided by the total number of visits to all nodes in thegraph. Throughout the paper, we describe the random walkusing realizations of a Markov chain with transition matrixP = [pij ].

A. The mean first passage time for a weighted graph

Let Xk ∈ {1, . . . , n} denote the location of the randomwalker at time k ∈ {0, 1, 2, . . . }. For any two nodes i, j ∈{1, . . . , n}, the first passage time from i to j, denoted by Tij ,is the first time that the random walker starting at node i attime 0 reaches node j, that is,

Tij = min{k ≥ 1 | Xk = j given that X0 = i}.

It is convenient to introduce the shorthand mij = E[Tij ],and to define the mean first passage time matrix M to haveentries mij , for i, j ∈ {1, . . . , n}. The mean first passage timefrom start node i, denoted by ki, is the expected first passagetime from node i to a random node selected according to thestationary distribution π, i.e.,

ki =

n∑j=1

mijπj .

It is well known [25] that the mean first passage time from astart node is independent of the start node, that is, ki = kjfor all i, j ∈ {1, . . . , n}. Accordingly, we let K = ki, forall i ∈ {1, . . . , n}, denote the mean first passage time, alsoknown as the Kemeny constant, of the Markov chain.

Next, we provide formulas for these quantities. By defini-tion, the first passage time from i to j satisfies the recursiveformula:

Tij =

{1, with probability pij ,Tkj + 1, with probability pik, k 6= j.

Taking the expectation, we compute

mij = pij +

n∑k=1,k 6=j

pik(mkj + 1) = 1 +

n∑k=1,k 6=j

pikmkj ,

or in matrix notation,

(I − P )M = 1n1Tn − PMd, (1)

where P is the transition matrix of the Markov chain. If theMarkov chain is irreducible with stationary distribution π, thenone can show Md = diag[{1/π1, . . . , 1/πn}], and

πTMπ =

n∑i=1

πi

n∑j=1

πjmij =

n∑i=1

πiki = K.

Clearly, the Kemeny constant can be written as the functionP 7→ K(P ), however, to ease notation we simply write K

and use K(P ) only when we wish to emphasize the constant’sdependence on P .

The Kemeny constant K = πTMπ can be computedfrom equation (1) or can be expressed as a function of theeigenvalues of the transition matrix P as is stated in thefollowing theorem [25].

Theorem 1: (Kemeny constant of an irreducible Markovchain): Consider a Markov chain with an irreducible transitionmatrix P with eigenvalues λ1 = 1 and λi, i ∈ {2, . . . , n}. TheKemeny constant of the Markov chain is given by

K = 1 +

n∑i=2

1

1− λi.

Using Theorem 1, we derive the following equivalent ex-pression for reversible Markov chains in terms of the trace ofa symmetric positive definite matrix. Before stating our result,we first introduce some notation. Given a stationary distri-bution vector π ∈ Rn×1 for a Markov chain with transitionmatrix P ∈ Rn×n, we define the matrix Π ∈ Rn×n as Π =diag[π] and the vector q ∈ Rn×1 as qT = (

√π1, . . . ,

√πn).

We are now ready to state our first result.Theorem 2: (Kemeny constant of a reversible irreducible

Markov chain): The Kemeny constant of a reversible irre-ducible Markov chain with transition matrix P and stationarydistribution π is given by

K = Tr[(I −Π1/2PΠ−1/2 + qqT )−1

]. (2)

Proof: We start by noting that P is an irreduciblerow-stochastic matrix therefore the eigenvalues of P are{λ1 = 1, λ2, . . . , λn}, where |λi| ≤ 1 and λi 6= 1 fori ∈ {2, . . . , n}. It follows that the eigenvalues of (I − P ) are{0, 1− λ2, . . . , 1− λn}. Since P is irreducible and reversible,there exists a stationary distribution π ∈ Rn>0 implying Π isinvertible and that Π1/2(I − P )Π−1/2 = I − Π1/2PΠ−1/2

is symmetric. It can easily be verified that I − P andI − Π1/2PΠ−1/2 have the same eigenvalues and that q isthe eigenvector associated with the eigenvalue at 0. Next,notice the matrix (I − Π1/2PΠ−1/2 + qqT ) is symmet-ric and that (I − Π1/2PΠ−1/2 + qqT )q = q. Therefore,(I − Π1/2PΠ−1/2 + qqT ) has an eigenvalue at 1 associ-ated with the vector q. Since (I − Π1/2PΠ−1/2 + qqT ) issymmetric, the eigenvectors corresponding to different eigen-values are orthogonal; implying for eigenvector v 6= q that(I − Π1/2PΠ−1/2 + qqT )v = (I − Π1/2PΠ−1/2)v sincethe eigenvalue at 1 is simple. Therefore, the eigenvalues of(I −Π1/2PΠ−1/2 + qqT ) are {1, 1− λ2, . . . , 1− λn}. Thus,K = Tr

[(I −Π1/2PΠ−1/2 + qqT )−1

]= 1 + 1/(1− λ2) +

. . .+ 1/(1− λn) = K.

Given the above result, we are now ready to state our firstproblem of interest.

Problem 1: (Optimizing the Kemeny constant of a re-versible Markov chain): Given the stationary distribution πand graph G with vertex set V and edge set E, determine the

4

transition probabilities P = [pij ] solving:

minimize Tr[(I −Π1/2PΠ−1/2 + qqT )−1

]subject to

n∑j=1

pij = 1, for each i ∈ {1, . . . , n}

πipij = πjpji, for each (i, j) ∈ E0 ≤ pij ≤ 1, for each (i, j) ∈ Epij = 0, for each (i, j) /∈ E.

(3)

Remark 3: All feasible solutions P to Problem 1 are in-herently irreducible transition matrices: when P is not irre-ducible, the matrix (I−Π1/2PΠ−1/2+qqT ) is not invertible.Moreover, a feasible point always exists since the Metropolis-Hastings algorithm applied to any irreducible transition matrixassociated with G, generates a reversible transition matrixwhich is irreducible and satisfies the stationary distributionconstraint [21].The following theorem establishes the convexity of the Ke-meny constant for transition matrices with fixed stationarydistribution.

Theorem 4 (Convexity of Problem 1): Let Pπ denote theset of matrices associated to irreducible reversible Markovchains with stationary distribution π. Then, Pπ is a convexset and P 7→ K(P ) is a convex function over Pπ .

Proof: Let S denote the set of symmetric positive definitematrices, for any stationary distribution π ∈ Rn>0, denote theset SP,π := {I − Π1/2PΠ−1/2 + qqT | P ∈ Pπ}. We beginby showing that Pπ is a convex set. Let P1, P2 ∈ Pπ , thenPπ is convex if for an arbitrary θ ∈ [0, 1] that

θP1 + (1− θ)P2 = P3 ∈ Pπ. (4)

Pre and post multiplying (4) by Π1/2 and Π−1/2, respec-tively, we have that θΠ1/2P1Π−1/2 + (1− θ)Π1/2P2Π−1/2 =Π1/2P3Π−1/2. Then Π1/2P3Π−1/2 is symmetric sinceΠ1/2P1Π−1/2 and Π1/2P2Π−1/2 are symmetric. Pre multiply-ing (4) by πT we easily verify that the stationary distributionP3 is πT and similarly, post multiplying by 1n verifies that P3

is row stochastic. Finally taking the left hand side of (4) to then-th power gives (θP1+(1−θ)P2)n = θnPn1 +(1−θ)nPn2 +ζ,where ζ denotes the sum of all other terms in the expansionand has the property ζij ≥ 0 for all i, j since P1 andP2 are non-negative element-wise matrices. Moreover fromirreducibility, there exists a sufficiently large N such that forn > N , (Pn1 )ij > 0 and (Pn2 )ij > 0 for all i, j, which implies(Pn3 )ij > 0, therefore P3 ∈ Pπ and Pπ is convex.

Next we show that SP,π ⊂ S . From the proof of Theorem2 we have for P ∈ Pπ that I−Π1/2PΠ−1/2+qqT has eigen-values {1, 1 − λ2, . . . , 1 − λn}, where λi for i ∈ {1, . . . , n}are the eigenvalues of P , where λi ≤ |1| for all i andλi 6= 1 for i ∈ {2, . . . , n}. Therefore, all eigenvalues ofI−Π1/2PΠ−1/2 +qqT are strictly greater than zero. Finally,since P is reversible Π1/2PΠ−1/2 is symmetric implying(I − Π1/2PΠ−1/2 + qqT )T = I − Π1/2PΠ−1/2 + qqT andso SP,π ⊂ S.

Finally, define the mapping g : Pπ 7→ SP,π by g(X) =I −Π1/2XΠ−1/2 + qqT . This is an affine mapping from theconvex set Pπ to a subset of S. From [18] we know that

Tr[X−1] is convex for X ∈ S, therefore the compositionwith the affine mapping g : Pπ 7→ SP,π ⊂ S, Tr[g(X)−1] isalso convex [8, Chapter 3.2.2].

Problem 1 includes constraints on the stationary distributionof the transition matrix, a notion which has not been lookedat in the literature before. The author of [28] provides boundsto determine the set of transition matrices such that K isminimized and [25] gives special matrices for which theoptimal Kemeny constant can be found, but these are allapproached for the general setting with no constraint on theactual stationary distribution. In real-world settings, constraintson the stationary distribution are important and have manypractical interpretations. For example, it is often desirable tovisit certain regions more frequently than other based on eachregion’s relative importance.

B. SDP framework for optimizing the Kemeny constant

Here we show how Problem 1 can be equivalently rewrittenas an SDP by introducing a symmetric slack matrix.

Problem 2: (Optimizing the Kemeny constant of a re-versible Markov chain (SDP)): Given the stationary distribu-tion π and graph G with vertex set V and edge set E, determineX = [xij ] and the transition probabilities P = [pij ] solving:

minimize Tr[X]

subject to [I −Π1/2PΠ−1/2 + qqT I

I X

]� 0

n∑j=1

pij = 1, for each i ∈ {1, . . . , n}

πipij = πjpji, for each (i, j) ∈ E0 ≤ pij ≤ 1, for each (i, j) ∈ Epij = 0, for each (i, j) /∈ E.

The first inequality constraint in Problem 2 represents a linearmatrix inequality (LMI) and denotes that the matrix is positivesemidefinite. Since the matrix in the LMI has off-diagonalentries equal to the identity matrix, it holds true if and only ifX is positive definite and (I − Π1/2PΠ−1/2 + qqT ) −X−1is positive semidefinite [2, Theorem 1]. This implies (I −Π1/2PΠ−1/2 + qqT ) is positive definite and that X � (I −Π1/2PΠ−1/2+qqT )−1. Therefore, the SDP given by Problem2 minimizes the Kemeny constant.

III. THE WEIGHTED KEMENY CONSTANT OF MARKOVCHAIN AND ITS MINIMIZATION

In most practical applications, distance/time traveled andservice costs/times are important factors in the model of thesystem. We incorporate these concepts by allowing for anadditional set of weighted edges in our graph in addition tothe edge weights which describe the transition probabilities.Such a system can be represented by the doubly-weightedgraph G = (V, E, P,W ), where W = [ωij ] is a weightmatrix with the properties that: if (i, i) ∈ E, then ωii ≥ 0;if (i, j) ∈ E, i 6= j, then ωij > 0 ; and if (i, j) /∈ E, thenωij = 0. The weighted adjacency matrix P = [pij ] has the

5

1

23

p11

p13

p31 p22

p23

p32

1

23

p33

!32

!33

!23

!22!31

!11

!13

1

23

Fig. 1. Example of a doubly-weighted graph G = (V, E, P,W ) with threenodes: (a) shows the edge set, E, allowed for the graph with three nodes, (b)shows the probabilities, pij to move along each edge, and (c) shows the time(i.e., distance traveled), ωij to move along each edge.

same interpretation as before as an irreducible row-stochastictransition matrix P which governs the random walk on thegraph. An example of a doubly-weighted graph is shown inFigure 1. In the following, we will interpret ωij , i 6= j as thetime to travel between two nodes, i and j, in the graph and ωiias the service time at node i. We discuss another motivatingexample and interpretation for ωij in a later section.

Recall that Xk = i denotes that the random walker is atnode i at time k. If a sample trajectory of the random walkis X0 = i, X1 = j, X2 = k, then the time instant at which arandom walker arrives in state X2 is ωij +ωjk. Thus the timeinterval between two consecutive steps of this random walkdepends on the weighted adjacency matrix, W , of the graphand is not constant.

In the following analysis, we look at several characterizationand optimization objectives: The first involves extending thenotion of the Kemeny constant to doubly-weighted graphs andproviding a detailed characterization of this extension. Thesecond involves the minimization of the mean first passagetime of a doubly-weighted graph and the third involves char-acterization and minimization of the mean time to executea single hop. The first and second objectives are motivatedby the need to minimize visit times to nodes in the graph,and the third is motivated by the desire to minimize resourceconsumption when moving between nodes. We seek to designtransition matrices P with stationary distribution π whichoptimize each problem. We start with the first objective.

A. The mean first passage time for a doubly-weighted graph

The mean first passage time for the Markov chain on aweighted graph G = (V, E, P ) by definition, is simply itsKemeny constant. Recall that the mean first passage time fornode i, defined by ki, is determined by taking the expectationover the first passage times mij , from node i to all other nodesj. We present an analogous notion of the first passage timebetween two nodes on a doubly-weighted graph. We start withdefining the first passage time random variable for a randomwalk on a doubly-weighted graph and provide a recursiveformulation for its expectation.

As in Section II-A, for any two nodes i, j ∈ {1, . . . , n}, thefirst passage time from i to j is the first time that the random

walker starting at node i at time 0 reaches node j, that is,

Tij = min{ k−1∑n=0

wXn,Xn+1 , for k ≥ 1 | Xk = j

given that X0 = i}.

Lemma 5: (First passage time for a doubly-weightedgraph): Let nij = E[Tij ] denote the mean first passage time togo from i to j for a graph with weight matrix W and transitionmatrix P . Then

nij = pij(ωij) +∑k 6=j

pik(nkj + ωik), (5)

or, in matrix notation,

(I − P )N = (P ◦W )1n1Tn − PNd, (6)

where (P ◦W ) is the element-wise product between P andW and where Nd = [δijnij ].

Proof: By its definition, the first passage time satisfiesthe recursive formula:

Tij =

{ωij , with probability pij ,ωik + Tkj , with probability pik, k 6= j.

(7)

Therefore, the results follows from taking the expectation:

E[Tij ] = ωijpij +∑k 6=j

pik(E[Tkj ] + ωik).

The matrix N , which we call the mean first passage timematrix for a doubly-weighted graph thus satisfies an equationsimilar to (1) for the passage time matrix M of a graph witha single weight matrix, the transition matrix P . In fact, wesee that equation (6) is equivalent to (1) when wij = 1 for all(i, j) ∈ E (i.e., for an unweighted graph).

The random variable tracking the time interval betweenconsecutive visits to a node has been referred to as therefresh time of that node [35] and nii is the expected valueof the refresh time for the random walk. We now obtain arelation between π and the refresh times nii.

Theorem 6 (Refresh times for doubly-weighted graphs):Consider a Markov chain on a doubly-weighted graphG = (V, E, P,W ) with stationary distribution π andassociated mean first passage time matrix N . The refreshtime for node i is given by nii = (πT (P ◦ W )1n)/πi,implying that

Nd = πT (P ◦W )1nMd.

Proof: The stationary distribution of the transition matrixP is π ∈ Rn×1. Therefore, premultiplying equation (6) byπT , we obtain

0 = πT (P ◦W )1n1Tn − πTNd,

where the left hand side of equation (6) is zero since πT (I −P ) = πT − πT = 0. Now we have that (πT (P ◦W )1n)1Tn =πTNd. Since Nd is a diagonal matrix and πT (P ◦W )1n isa scalar, we get that πinii = πT (P ◦W )1n. Thus, dividingby πi we have that nii = πT (P ◦W )1n/πi, and in matrix

6

form Nd = πT (P ◦W )1ndiag({1/π1, . . . , 1 πn}) = πT (P ◦W )1nMd .This theorem implies that the refresh time nii of the randomwalk is directly proportional to the visit frequencies 1/πi.Therefore, the relative visit frequencies of one node comparedto another are not a function of the weight matrix W and onlydepend on the stationary distribution of the transition matrixP .

We now investigate the properties of the mean first passagetime of the weighted random walk. The mean first passagetime for a doubly-weighted graph G = (V, E, P,W ) withassociated passage times matrix N is given by KW = πTkW ,where kW = Nπ is the vector of first passage times and thei−th entry kW,i in kW denotes the mean time to go from ito any other node. We refer to the mean first passage timefor a doubly-weighted graph, KW , as the weighted Kemenyconstant. We now provide an analytic expression for the vectorkW ∈ Rn×1.

Lemma 7: (First passage times for a doubly-weightedgraph): Given a Markov chain on a doubly-weighted graphG = (V, E, P,W ) with stationary distribution π and associ-ated first passage time matrix N , the following equality holds:

(I − P )kW = (P ◦W )1n − 1nπT (P ◦W )1n, (8)

where kW = Nπ.Proof: Post multiplying equation (6) on both sides by π

gives

(I − P )Nπ =(P ◦W )1n1Tnπ − PNdπ,

(I − P )kW =(P ◦W )1n − P (πT (P ◦W )1n)1n

=(P ◦W )1n − 1nπT (P ◦W )1n.

The right hand side of (8) gives the insight that, in general,kW,i 6= kW,j on the doubly-weighted graph, unlike thecounterpart for the single-weighted graph (where ki = Kfor all i ∈ {1, . . . , n}). Interestingly enough however, theredoes exist a relation between the weighted Kemeny constantKW and the Kemeny constant K as is stated in the followingtheorem, whose proof is postponed to the Appendix.

Theorem 8: (Weighted Kemeny constant of a Markovchain): For the doubly-weighted graph G = (V, E, P,W ), theweighted Kemeny constant KW of the Markov chain is givenby

KW = πT (P ◦W )1nK, (9)

where K is the Kemeny constant associated with the irre-ducible transition matrix P with stationary distribution π .

Remark 9: The expected number of hops to go from onenode to another in a Markov chain with transition matrix Pis its Kemeny constant. The expected distance travelled (andhence time taken) executing one hop is

∑i πi

∑j pijωij =

π(P ◦ W )1n. Hence, it is consistent with intuition that theexpected time to travel from one node to another should beKπT (P ◦W )1n as is formally shown in the appendix.

Given the above results, we are now ready to state anotherproblem of interest.

Problem 3: (Optimizing the weighted Kemeny constant of areversible Markov chain): Given the stationary distribution πand graph G with vertex set V , edge set E and weight matrixW , determine the transition probabilities P = [pij ] solving:

minimize(πT (P ◦W )1n

) (Tr[(I −Π1/2PΠ−1/2 + qqT )−1

])subject to

n∑j=1

pij = 1, for each i ∈ {1, . . . , n}

πipij = πjpji, for each (i, j) ∈ E0 ≤ pij ≤ 1, for each (i, j) ∈ Epij = 0, for each (i, j) /∈ E.

The following theorem establishes the convexity of theweighted Kemeny constant for transition matrices with fixedstationary distribution.

Theorem 10 (Convexity of Problem 3): Given the graph Gwith vertex set V , edge set E and weight matrix W , let PG,πdenote the set of matrices associated with G that are irreduciblereversible Markov chains with stationary distribution π. Then,PG,π is a convex set and P 7→ πT (P ◦W )1nK(P ) is a convexfunction over PG,π .

Proof: Let S denote the set of symmetric positive definitematrices, for any stationary distribution π ∈ Rn>0, denotethe set SG,P,π := {I − Π1/2PΠ−1/2 + qqT | P ∈ PG,π}.The proof of convexity of the set PG,π is similar to thatof the proof of Pπ in Theorem 4 and so is omitted forbrevity. Then from the proof of Theorem 4 we know thereexists an affine mapping g(X) : PG,π 7→ SG,P,π given byg(X) = I − Π1/2PΠ−1/2 + qqT . We know from [18] thatf(X) = Tr[X−1] is convex, therefore the perspective functionh(X, t) = {tf(X/t) | t > 0} is also convex [8, Chapter 3.2.6].Moreover the composition of h(X, t) with the affine mappingg(X), h(g(X), t) is also convex. Let t = (πT (X ◦W )1n)1/2,and notice that πT (X ◦ W )1n > 0 for X ∈ PG,π andtherefore t > 0. Also notice that for a constant k ∈ Rn>0

and matrix X ∈ Rn×n that Tr[(Xk )−1] = kTr[X−1]. Thenh(g(X), t) = tTr[( g(X)

t )−1] = t2Tr[(g(X)−1] = πT (X ◦W )1nTr[(I −Π1/2XΠ−1/2 + qqT )−1] for X ∈ PG,π .

B. SDP framework for optimizing the weighted Kemeny con-stant

In a similar fashion to Problem 1, we can formulate Problem3 as an SDP by introducing the symmetric slack matrix X ∈Rn×n and the scalar variable t as is shown in the following.

Problem 4: (Optimizing the weighted Kemeny constant ofa reversible Markov chain (SDP)): Given the stationary dis-tribution π and graph G with vertex set V , edge set E and

7

weight matrix W , determine Y = [yij ], X and t solving:

minimize Tr[X]

subject to [t(I + qqT )−Π1/2YΠ−1/2 I

I X

]� 0

n∑j=1

yij = t, for each i ∈ {1, . . . , n}

πiyij = πjyji, for each (i, j) ∈ E0 ≤ yij ≤ t, for each (i, j) ∈ Eyij = 0, for each (i, j) /∈ EπT (Y ◦W )1n = 1

t ≥ 0.

Then, the transition matrix P is given by P = Y/t.

As in Problem 2, the first inequality constraint in Problem 4represents an LMI. Before noting when the LMI holds, firstnote that the constraints in Problem 4 imply that Pt = Yand that t = 1

πT (P◦W )1n. Hence, using a similar argument

as in Problem 2, the LMI constraint holds true if and only ifX � πT (P ◦W )1n(I−Π1/2PΠ−1/2+qqT )−1 where X andπT (P ◦W )1n(I − Π1/2PΠ−1/2 + qqT )−1 are both positivedefinite, and therefore the SDP given by Problem 4 minimizesthe weighted Kemeny constant.

C. Minimizing single hop distance

We now look at the objective of minimizing the mean timefor a single hop of a random walk. At time k, let Sij be thetime required to transition from i to j in a single hop alongan edge of length ωij . Then,

E [S] =

n∑i=1

n∑j=1

pijSij

=

n∑i=1

P [Xk = i]

n∑j=1

ωijP [Xk+1 = j]

=

n∑i=1

n∑j=1

πiωijpij = πT (P ◦W )1n. (10)

The function πT (P ◦ W )1n is clearly convex in P . If oneassumes that ωii = 0 for all i ∈ {1, . . . , n}, then minimiz-ing (10) over P yields the trivial solution P = I . We cantake into account both the single hop distance as well as theKemeny constant to design a Markov chain as follows.

Problem 5: (Optimizing Kemeny constant and mean dis-tance): Given the stationary distribution π and graph G withvertex set V , edge set E and weight matrix W , and givenuser specified constant α ∈ [0, 1], determine the transition

probabilities P = [pij ] solving:

minimize αTr[(I −Π1/2PΠ−1/2 + qqT )−1

]+ (1− α)πT (P ◦W )1n

subject ton∑j=1

pij = 1, for each i ∈ {1, . . . , n}

πipij = πjpji, for each (i, j) ∈ E0 ≤ pij ≤ 1, for each (i, j) ∈ Epij = 0, for each (i, j) /∈ E.

(11)

This problem is convex since the sum of two convex problemsis convex, moreover, it can be extended to an SDP utilizingthe LMI defined in Problem 2. In the context where ωii = 0for all i ∈ {1, . . . , n}, varying the parameter α can be usedto control the connectivity of the Markov chain; the choiceα = 1 ensures connectivity, and the choice α = 0 minimizesthe single hop distance while making the graph disconnected.

IV. APPLICATIONS OF THE MEAN FIRST PASSAGE TIME TOSURVEILLANCE

The results on mean first passage time for doubly-weightedgraphs (i.e., the weighted Kemeny constant) presented in thiswork provide a general framework which can potentially beapplied to the analysis and design in a myriad of fields. Wefocus on one application in particular; the intruder detectionand surveillance problem. We look at two variations of thisproblem:

a) Scenario IIn practical stochastic intruder detection and surveillance

scenarios, there is often a desire to surveil some regions morethan others (i.e.,have a pre- specified stationary distribution)while simultaneously minimizing the time any one region hasto wait before it is serviced. For this setup, in every stepof the random walk, the agent must move to a new regionand execute its surveillance task. Assuming we are workingon a doubly-weighted graph described by G = (V, E, P,W ),let us also assume there is a fixed persistent intruder in theenvironment and it takes si time for an agent to determineif the intruder is in region i ∈ V . Denote the time to movefrom region i to region j by dij , where we can assume, withoutloss of generality, that dii = 0. Then, we can define the weightcorresponding to the edge from i to j as ωij = dij+sj . In thisscenario we wish to minimize the expected time to capture thepersistent intruder when no prior knowledge of their positionis known.

b) Scenario IIIn this scenario we consider the intruder detection problem

and adopt a similar setup to Scenario I, however, we nowassume a set of intruders are distributed throughout the envi-ronment. Each intruder performs a malicious activity in its hostregion for a fixed duration of time, which we call the intruderlife-time, followed instantaneously by another intruder. Theintruder is caught only if the agent is in the same regionas the intruder for any duration of the intruder life-time. Forsimplicity only a single intruder appears at a time.

In the following subsections we analyze the performance ofvarious stochastic surveillance policies as applied to Scenario I

8

and Scenario II described above. More specifically, we gaugethe performance of other well-known Markov chain designalgorithms against the algorithms presented in this paper.

A. Optimal strategy for Scenario I

In the context of Scenario I the weighted Kemeny constantof the agent’s transition matrix, P , corresponds to the averagetime it takes to capture an intruder regardless of where theagent and intruder are in the environment. Therefore bydefinition of the Kemeny constant, we have the followingimmediate corollary for Scenario I.

Corollary 11 (Optimal surveillance and service strategy):The transition matrix P which has minimal mean first passagetime is the optimal strategy for the intruder detection problemdescribed by Scenario I.

This tells us that if we restrict ourselves to reversibleMarkov chains, then not only is the chain with minimal meanfirst passage time optimal, but given the results from Section IIand III, we can also optimally design this chain.

B. Numerical analysis of Scenario II

In Scenario II the transition matrix with minimum meanfirst passage time is not guaranteed to be the optimal policy,and thus to gauge its performance compared to other poli-cies we analyze both homogeneous (uniform service/traveltimes) and heterogeneous environment cases. To compareperformance we generate a random walk for the environmentusing the Metropolis-Hastings, fastest mixing Markov chain(FMMC) [7], and Kemeny constant algorithms. While gametheoretic frameworks [11], [6] also generate stochastic poli-cies, they are based on assumptions on the intruder behavior.We avoid such assumptions here and, therefore, omit themfrom our comparative analysis.

We first look at the homogeneous case which is describedby the discretized environment shown in Figure 2. We assumethat a single surveillance agent executes a random walk in theenvironment, spending 1 time unit in each region, and thatthe agent transitions between two connected regions instanta-neously. Also, we assume a uniform stationary distribution onthe transition matrix (each node in the region must be visitedwith equal likelihood). The Markov chain generated by theMetropolis-Hastings algorithm is generated by applying thealgorithm to the random walk described by pij = 1/di fori 6= j and pij = 0 for i = j, where di is the degree of a nodei (excluding self-loops) [21]. The intruder life-time is set to66 time units and 500 intruders appear per simulation run (thesequence in which the intruders appear is determined beforeeach simulation run), for a total simulation run of 33, 000time units. As stated in the scenario description, the intruderis caught if the surveillance agent is in the same region asthe intruder for any portion of the intruder life-time. TableI summarizes the statistical performance of each algorithmafter 200 runs of the simulation and justifies our use of theKemeny constant algorithm as a valid surveillance strategy; theKemeny constant algorithm captures intruders more frequentlythan the other two algorithms, and its worst case performanceis still better than the worst case performance of the other

Fig. 2. Environment with two obstacle represented by an unweighted graph.

two algorithms. Although results for an intruder life-time ofonly 66 time units are presented here, we have found that theKemeny constant algorithm always outperforms the other twoalgorithms or is equivalent; the algorithms become equivalentin the limiting case, when the intruder life-times are so lowthat no intruder can be caught, or when the intruder-life timesare so large that the intruder is always caught.

Algorithm Min Mean Max StdDev K

Kemeny constant 26.6 32.4 38.2 2.1 207FMMC 24.6 29.8 34.4 1.9 236

Metropolis-Hastings 24.8 31.1 37 2.1 231

TABLE ISTATISTICS ON THE PERCENTAGE OF INTRUDERS CAUGHT IN 200

SIMULATION RUNS FOR THE ENVIRONMENT IN FIG. 2.

For the heterogeneous case, we work with the environmentshown in Figure 3. In this environment the time taken by theagent to travel an edge is no longer instantaneous and hastravel weights as shown in the figure. Once in a new region,the agent is required to spend 1 time unit examining the regionfor malicious activities. We again assume that each node inthe region must be visited with equal likelihood. We againalso assume an intruder is caught if the surveillance agentis in the same region as a intruder for any portion of theintruder life-time, but now set the intruder life-time to 11 timeunits with a intruder appearing 500 times (total of 5500 timeunits per simulation run). Since the design of the FMMC andMetropolis-Hastings algorithms do not inherently account fornon-uniform travel and service times, we also compare theperformance of the random walk generated by the weightedKemeny constant algorithm against the random walk generatedby solving Problem 5 with α = 0.1 (smaller α emphasizesminimizing the length of the average edge traveled in thegraph). Table II summarizes the statistical performance ofeach algorithm after 200 runs of the simulation. The weightedKemeny constant algorithm’s performance compared to theFMMC and Metropolis-Hastings stochastic policies in thisscenario is significantly better than what was seen in the firstscenario. We also note that the random walk policy determinedby solving Problem 5 performs comparably to the weightedKemeny constant policy. This is to be expected since theMetropolis-Hastings and FMMC stochastic policies do notaccount for heterogeneous travel/service times on the graph.

9

ANC

JFK

SEA

ORL

LAX

ANC

JFK

SEA

ORL

LAX

1

1

1

1

1

5

6

2

12

15

3

6

6

5

3

Fig. 3. Various airport hub locations (top), and the corresponding weight map(bottom). Edge weights between two hubs account for travel time betweenhubs plus required service time once at hub. Self loops have no travel timeso encompass only service time required at hub.

To get a better understanding of each algorithm’s performancein this intruder scenario, the simulation is run for differentintruder life-times, the results of which can be seen in Figure 4.There are several key items worth noting from the simulation.First, we see that the weighted Kemeny constant algorithmsignificantly outperforms the other algorithms for a large rangeof intruder life-times. This matches our intuition since thealgorithm inherently minimizes average travel time betweennodes. Second, notice that the Markov chain generated bysolving Problem 5 (with α = 0.1) performs well for smallintruder life-times but its performance plateaus quickly. Thisis due to the fact that the transition matrix generated bysolving Problem 5 forces agents to stay at a given node ratherthan jump nodes; as one would suspect, once intruder life-times increase, a strategy which places emphasis on an agentthat moves between regions will begin to perform relativelybetter. Finally, observe that as intruder life-time increases,the algorithms’ capture rates start to converge. As in thehomogeneous case, this is due to the fact that once theintruder’s life-time is long enough, the agent will almost surelyreach the intruder regardless of the policy it follows.

Algorithm Min Mean Max StdDev KW

Weighted Kemeny 44 50.1 56 2.2 19.5Kemeny+Mean Dist. 40.6 47.1 53 2.2 23.1

FMMC 29.8 35.4 40.4 2.2 26.2Metropolis-Hastings 30.4 36 41.6 2.1 26.5

TABLE IISTATISTICS ON THE PERCENTAGE OF INTRUDERS CAUGHT IN 200

SIMULATION RUNS FOR THE ENVIRONMENT IN FIG. 3.

10 15 20 25 30 35 40 45 50 55 60

0.4

0.5

0.6

0.7

0.8

0.9

1

Intruder life−time

% o

f captu

res

Fig. 4. Percentage of intruders detected for varying intruder life-times by asurveillance agent executing a random walk according to the Markov chaingenerated by the mean first-passage time algorithm (circle), FMMC algorithm(square), M-H algorithm (asterisk), and the Markov chain generated by solvingProblem 5 (diamond). Average points and standard deviation error bars aretaken over 200 runs, where the intruder appears 500 times for each run.

To solve for the Markov chains with minimal Kemenyconstant (Problem 2 and Problem 4) and with fastest mixingrate, we use CVX, a Matlab-based package for convex pro-grams [20]. The execution time to solve each Markov chain forthe examples described above takes on the order of a coupleseconds using a computer with a 1.3 GHz processor.

V. CONCLUSIONS

We have studied the problem of how to optimally designa Markov chain which minimizes the mean first passage timeto go from one region to any other region in a connectedenvironment. We have presented the first formulation of themean first passage time for a doubly-weighted graph, whichwe refer to as the weighted Kemeny constant, and havealso provided a provably correct convex formulation for theminimization of both the Kemeny constant and the weightedKemeny constant. Finally, we have shown that both problemscan be written as SDPs and, moreover, have demonstrated theeffectiveness of using a Markov chain with minimal mean firstpassage time as a surveillance policy as compared to otherwell-known Markov chain policies.

This work leaves open various directions for further re-search. First, we designed surveillance policy only for singleagent systems and it would be of practical interest to considerthe case where there are multiple agents: [37], [9], [3], [10] areexamples of work in this direction. Second, it would be usefulto understand bounds on the design of of the mean first passagetime for general graph topologies. Finally, we treat only theoptimization of the transition matrix of the graph. It would beof interest to study how we can optimize the weight matrixW in conjunction with the transition matrix. This can havethe interpretation of optimizing the ”capacity” or ”resistance”of the graph, a topic in optimization which is of independentinterest [18].

10

APPENDIX APROOF OF THEOREM 8

Proof of Theorem 8: Let β = πT (P ◦W )1n, then fromTheorem 6 we have that Nd from (6) can be written as βMd.Now from Theorem 13 the general solution to (6) is

N = G((P ◦W )1n1Tn − βPMd) + (I −G(I − P ))U, (12)

where G is a generalized inverse of (I − P ) (see Theorem15) and U is an arbitrary matrix as long as the consistencycondition(

I − (I − P )G)(

(P ◦W )1n1Tn − βPMd)

= 0 (13)

holds. Substituting (18) from Lemma 16 in for (I − P )G in(13) gives that(

I − (I − P )G)(

(P ◦W )1n1Tn − βPMd)

=tπT

πT t((P ◦W )1n1Tn − βPMd),

=t

πT t(πT (P ◦W )1n1Tn − βπTPMd),

=t

πT t(β1Tn − β1Tn ) = 0,

and so we have that the system of equations is consistent. Thisimplies we can design U to reduce (12). We start by seeinghow the second term in (12) can be reduced. Using (19) fromLemma 16 we have that (I−G(I−P ))U = 1nu

T

uT 1nU = 1nh

T ,where hT = uTU

uT 1n. Hence, we can re-write (12) as

N = G((P ◦W )1n1Tn − βPMd) + 1nhT , (14)

designing U reduces to designing the n elements of h. LetH = diag[h], then 1nh

T = 1n1TnH . Also, let Ξ = 1nπT ,where Ξd = diag[π]. Utilizing these expressions in (14) andtaking the diagonal elements gives(

N = G((P ◦W )1n1Tn − βPMd) + 1n1TnH)

d,

=⇒ βMd = (G(P ◦W )Ξ)dMd − β(GP )dMd +H,

=⇒ H = βMd − (G(P ◦W )Ξ)dMd + β(GP )dMd,

where we use Lemma 14 to get the initial diagonal reduction.Substituting the expression for H into (14), and recalling that1nh

T = 1n1TnH gives

N =(G(P ◦W )Ξ− 1n1Tn (G(P ◦W )Ξ)d (15)

+ β(1n1Tn (GP )d −GP + 1n1Tn ))Md,

where we use the fact that 1n1Tn = ΞMd. Now from (19)we have that I − G − GP = 1nu

T

uT 1n. Notice that 1n1Tn (I −

G−GP )d = 1n1Tn ( 1nuT

uT 1n)d = 1nu

T

uT 1nand so this implies that

1n1Tn −1n1TnGd +1n1Tn (GP )d = I−G+GP , which impliesthat 1n1Tn+1n1Tn (GP )d−GP = I−G+1n1TnGd. Substitutingthis into (15) gives the following reduced form.

N =(G(P ◦W )Ξ− 1n1Tn (G(P ◦W )Ξ)d (16)

+ β(1n1TnGd + I −G))Md.

Observing the definition of the generalized inverse, G, givenby Theorem 15 part (ii) and recalling that Ξ = 1nπT , wecan rewrite the first term on the right hand side of (16) as

G(P ◦W )Ξ = (I − P + tuT )−1(P ◦W )1nπT . Substituting(20) in for the right hand side with t = (P ◦ W )1n givesG(P ◦W )Ξ = 1nπ

T

uT 1n= 1uT 1n

Ξ, and so 1n1Tn (G(P ◦W )Ξ)d =

1n1Tn ( 1uT 1n

Ξ)d = 1uT 1n

Ξ = G(P ◦W )Ξ. Therefore, the firsttwo terms in (16) cancel giving the equality

N = β(1n1TnGd + I −G)Md. (17)

We have already defined t in the generalized inverse G butnot u. Let u = π and multiply the right hand side of (17) byπ and the left hand side by πT . Utilizing equality (21) fromLemma 16 gives

πTNπ = πTβ(1n1TnGd + I −G)Mdπ

= β(1TnGd + πT − πT

β)1n

= β(1TnGd1n + 1− 1

β)

= β(Tr[G] + 1)− 1.

Noting that the eigenvalue at 1 for an irreducible row-stochastic matrix is unique, it can be easily verified using theorthogonality property of left and right eigenvectors that theeigenvalues of G−1 are λi = (1 − λi) for i ∈ {2, . . . , n},where λi are eigenvalues of P and λi 6= 1. Therefore, itonly remains to find λ1. Taking the trace of G−1 givesTr[I − P + (P ◦ W )1nπT ] = Tr[I − P ] + Tr[(P ◦W )1nπT ] =

∑ni=1(1 − λi) + πT (P ◦W )1n, which implies

that λ1 = πT (P ◦W )1n = β. Therefore, β(Tr[G] + 1)−1 =β( 1

β +∑ni=2

11−λi

+ 1)− 1 = β(1 +∑ni=2

11−λi

).

APPENDIX BSUPPLEMENTAL MATERIAL

For completeness, we include the following results whichare needed in the proof of Theorem 8. We begin with somestandard results on generalized inverses. For more details referto [23, Chapter 4] or [22] .

Definition 12 (Generalized inverse): A generalized inverseof an m×n matrix A is defined as any n×m matrix A− thathas the property

AA−A = A.

It should be noted that a generalized inverse always exists,although it is not always unique. However, for non-singularmatrices the generalized inverse is unique and corresponds tothe usual notion of a matrix inverse. The following theoremssummarize practical considerations when working with gener-alized inverses.

Theorem 13: The equation Ax = b admits a solution if andonly if every generalized inverse A− satisfies

AA−b = b.

Then, we say Ax = b is consistent and all its general solutionsare given by

x = A−b+ (A−A− I)z,

where z is an arbitrary vector. Moreover, a necessary andsufficient condition for the equation AX = C to be consistent

11

is that (I − AA−)C = 0, in which case the general solutionis given by

X = A−C + (I −A−A)U,

where U is an arbitrary matrix.The next two results come from [24, Chapter 7].

Lemma 14 (Diagonal matrix properties): For π with posi-tive non-zero elements, let 1nπT = Ξ, where Ξd = diag[π].Also, let Λ be any diagonal matrix, X any square matrix ofsame dimensions as Λ, and D = (Ξd)−1, then(i.) (XΛ)d = (Xd)Λ, and

(ii.) (X1n1Tn )d = (XΞ)dD, and(iii.) 1n1TnΞd = Ξ.

Theorem 15 (Generalized inverse of I − P ): LetP ∈ Rn×n be the transition matrix of a irreducibleMarkov Chain with stationary distribution π. Let u, t ∈ Rnbe any vectors such that uT1n 6= 0 and πT t 6= 0, then(i.) I − P + tuT is nonsingular, and

(ii.) (I − P + tuT )−1 is a generalized inverse of I − P .Lemma 16 (Properties of the generalized inverse of I − P ):

Let G = (V, E, P,W ) be a doubly-weighted graph withassociated weight matrix W and irreducible transition matrixP with stationary distribution π. Also let G = (I−P+tuT )−1

denote the generalized inverse of (I − P ), then the followingrelations hold.

(I − P )G = I − tπT

πT t, (18)

G(I − P ) = I − 1nuT

uT1n, and (19)

1nuT1n

= Gt. (20)

If t = (P ◦W )1n and uT = πT then

πTG =πT

πT (P ◦W )1n(21)

Proof: First, notice that (I−P+tuT )(I−P+tuT )−1 = Iimplies that

(I − P )G = I − tuTG. (22)

Multiplying both sides on the left by πT and noting thatπT (I−P ) = 0 gives that πT = (πT t)uTG. Dividing throughby (πT t) gives

πT

πT t= uTG, (23)

and substituting (23) into (22) gives (18).Following a similar procedure as before with (I − P +

tuT )−1(I − P + tuT ) = I , where we now multiply bothsides on the right by 1n and note that (I − P )1n = 0 resultsin (20), which after appropriate substitution gives (19).

For the proof of equality (21), first we check that t = (P ◦W )1n and uT = πT satisfy the conditions of Theorem 15.The definition of W guarantees that P ◦W has at least onenon-zero element which implies πT t = πT (P ◦W )1n 6= 0.Also, uT1n = πT1n = 1. Now substituting u and t into (23)gives (21).

REFERENCES

[1] P. Agharkar, R. Patel, and F. Bullo. Robotic surveillance and Markovchains with minimal first passage time. In IEEE Conf. on Decision andControl, pages 6603–6608, Los Angeles, CA, USA, December 2014.

[2] A. Albert. Conditions for positive and nonnegative definiteness in termsof pseudoinverses. SIAM Journal on Applied Mathematics, 17(2):434–440, 1969.

[3] D. A. Anisi, P. Ogren, and X. Hu. Cooperative minimum timesurveillance with multiple ground vehicles. IEEE Transactions onAutomatic Control, 55(12):2679–2691, 2010.

[4] E. Bakshy, I. Rosenn, C. Marlow, and L. Adamic. The role of socialnetworks in information diffusion. In Int. Conference on World WideWeb, pages 519–528, Lyon, France, 2012.

[5] T. H. Blackwell and J. S. Kaufman. Response time effectiveness:Comparison of response time and survival in an urban emergencymedical services system. Academic Emergency Medicine, 9(4):288–295,2002.

[6] B. Bosansky, V. Lisy, M. Jakob, and M. Pechoucek. Computing time-dependent policies for patrolling games with mobile targets. In Interna-tional Conference on Autonomous Agents and Multiagent Systems, pages989–996, 2011.

[7] S. Boyd, P. Diaconis, and L. Xiao. Fastest mixing Markov chain on agraph. SIAM Review, 46(4):667–689, 2004.

[8] S. Boyd and L. Vandenberghe. Convex Optimization. CambridgeUniversity Press, 2004.

[9] G. Cannata and A. Sgorbissa. A minimalist algorithm for multirobotcontinuous coverage. IEEE Transactions on Robotics, 27(2):297–312,2011.

[10] D. W. Casbeer, D. B. Kingston, R. W. Beard, T. W. Mclain, S.-M.Li, and R. Mehra. Cooperative forest fire surveillance using a team ofsmall unmanned air vehicles. International Journal of Systems Sciences,37(6):351–360, 2006.

[11] L. Chen and J. Leneutre. A game theoretical framework on intrusiondetection in heterogeneous networks. IEEE Transactions on InformationForensics and Security, 4(2):165–178, June 2009.

[12] J. Clark and R. Fierro. Mobile robotic sensors for perimeter detectionand tracking. ISA Transactions, 46(1):3–13, 2007.

[13] P. Diaconis and D. Stroock. Geometric bounds for eigenvalues ofMarkov chains. Annals of Applied Probability, 1(1):36–61, 1991.

[14] P. G. Doyle and J. L. Snell. Random Walks and Electric Networks.Mathematical Association of America, 1984.

[15] W. Ellens, F. M. Spieksma, P. Van Mieghem, A. Jamakovic, and R. E.Kooij. Effective graph resistance. Linear Algebra and its Applications,435(10):2491–2506, 2011.

[16] Y. Elmaliach, N. Agmon, and G. A. Kaminka. Multi-robot area patrolunder frequency constraints. Annals of Mathematics and ArtificialIntelligence, 57(3-4):293–320, 2009.

[17] Y. Elmaliach, A. Shiloni, and G. A. Kaminka. A realistic modelof frequency-based multi-robot polyline patrolling. In InternationalConference on Autonomous Agents, pages 63–70, Estoril, Portugal, May2008.

[18] A. Ghosh, S. Boyd, and A. Saberi. Minimizing effective resistance ofa graph. SIAM Review, 50(1):37–66, 2008.

[19] J. Grace and J. Baillieul. Stochastic strategies for autonomous roboticsurveillance. In IEEE Conf. on Decision and Control and EuropeanControl Conference, pages 2200–2205, Seville, Spain, December 2005.

[20] M. Grant and S. Boyd. CVX: Matlab software for disciplined convexprogramming, version 2.1. http://cvxr.com/cvx, October 2014.

[21] W. K. Hastings. Monte Carlo sampling methods using Markov chainsand their applications. Biometrika, 57(1):97–109, 1970.

[22] J. J. Hunter. Generalized inverses and their application to appliedprobability problems. Linear Algebra and its Applications, 45:157–198,1982.

[23] J. J. Hunter. Mathematical Techniques of Applied Probability, volume 1of Discrete Time Models: Basic Theory. Academic Press, 1983.

[24] J. J. Hunter. Mathematical Techniques of Applied Probability, volume 2of Discrete Time Models: Techniques and Applications. Academic Press,1983.

[25] J. J. Hunter. The role of Kemeny’s constant in properties of Markovchains. Communications in Statistics - Theory and Methods, 43(7):1309–1321, 2014.

[26] J. G. Kemeny and J. L. Snell. Finite Markov Chains. Springer, 1976.[27] D. B. Kingston, R. W. Beard, and R. S. Holt. Decentralized perimeter

surveillance using a team of UAVs. IEEE Transactions on Robotics,24(6):1394–1404, 2008.

12

[28] S. Kirkland. Fastest expected time to mixing for a Markov chain on adirected graph. Linear Algebra and its Applications, 433(11-12):1988–1996, 2010.

[29] D. J. Klein and M. Randic. Resistance distance. Journal of MathematicalChemistry, 12(1):81–95, 1993.

[30] M. Levene and G. Loizou. Kemeny’s constant and the random surfer.The American Mathematical Monthly, 109(8):741–745, 2002.

[31] D. A. Levin, Y. Peres, and E. L. Wilmer. Markov Chains and MixingTimes. American Mathematical Society, 2009.

[32] Y. Y. Li and L. E. Parker. Intruder detection using a wireless sensor net-work with an intelligent mobile robot response. In IEEE SoutheastCon,pages 37–42, Huntsville, AL, USA, April 2008.

[33] C. D. Meyer. Matrix Analysis and Applied Linear Algebra. SIAM, 2001.[34] J. L. Palacios. Closed-form formulas for Kirchhoff index. International

Journal of Quantum Chemistry, 81(2):135–140, 2001.[35] F. Pasqualetti, A. Franchi, and F. Bullo. On cooperative patrolling:

Optimal trajectories, complexity analysis and approximation algorithms.IEEE Transactions on Robotics, 28(3):592–606, 2012.

[36] A. Patcha and J. Park. A game theoretic approach to modelingintrusion detection in mobile ad hoc networks. In Information AssuranceWorkshop. Proceedings from the Fifth Annual IEEE SMC, pages 280–284, June 2004.

[37] K. Srivastava, D. M. Stipanovic, and M. W. Spong. On a stochasticrobotic surveillance problem. In IEEE Conf. on Decision and Control,pages 8567–8574, Shanghai, China, December 2009.

[38] V. Srivastava, F. Pasqualetti, and F. Bullo. Stochastic surveillance strate-gies for spatial quickest detection. International Journal of RoboticsResearch, 32(12):1438–1458, 2013.

[39] S. Susca, S. Martınez, and F. Bullo. Monitoring environmental bound-aries with a robotic sensor network. IEEE Transactions on ControlSystems Technology, 16(2):288–296, 2008.

[40] X. Wu and Z. Liu. How community structure influences epidemicspread in social networks. Physica A: Statistical Mechanics and itsApplications, 387(2):623–630, 2008.

[41] P. R. Wurman, R. D’Andrea, and M. Mountz. Coordinating hundreds ofcooperative, autonomous vehicles in warehouses. AI Magazine, 29(1):9–20, 2008.

Rushabh Patel (IEEE S’12) received the B.S. andM.S. degrees in aerospace engineering from Califor-nia Polytechnic State University, San Luis Obispo,California, in 2007. From 2007 through 2011 heworked as a performance analyst at Northrop Grum-man Aerospace Systems. He is currently pursuingthe Ph.D. degree in control and dynamical systemsat the University of California, Santa Barbara. Hisresearch interests include coordination and surveil-lance in networked systems.

Pushkarini Agharakar (IEEE S’12) received theB.Tech. and M.Tech. degrees in aerospace engineer-ing from Indian Institute of Technology, Madras, in2011. She is currently pursuing the Ph.D. degree inmechanical engineering at the University of Califor-nia at Santa Barbara. Her research interests includevehicle routing problems, robotic surveillance andmotion coordination for robotic systems.

Francesco Bullo (IEEE S’95-M’99-SM’03-F’10) isa Professor with the Mechanical Engineering De-partment and the Center for Control, DynamicalSystems and Computation at the University of Cali-fornia, Santa Barbara. He was previously associatedwith the University of Padova, the California Insti-tute of Technology and the University of Illinoisat Urbana-Champaign. His main research interestsare network systems and distributed control withapplication to robotic coordination, power grids andsocial networks. He is the coauthor of “Geometric

Control of Mechanical Systems” (Springer, 2004, 0-387-22195-6) and ”Dis-tributed Control of Robotic Networks” (Princeton, 2009, 978-0-691-14195-4).He received the 2008 IEEE CSM Outstanding Paper Award, the 2010 HugoSchuck Best Paper Award, the 2013 SIAG/CST Best Paper Prize, and the 2014IFAC Automatica Best Paper Award. He has served on the editorial boardsof IEEE Transactions on Automatic Control, ESAIM: Control, Optimization,and the Calculus of Variations, SIAM Journal of Control and Optimization,and Mathematics of Control, Signals, and Systems.