Image and Vision Computing 28 (2010) 851–864

Contents lists available at ScienceDirect

Image and Vision Computing

journal homepage: www.elsevier .com/ locate / imavis

Camera handoff with adaptive resource management for multi-cameramulti-object tracking

Chung-Hao Chen a,*, Yi Yao b, David Page c, Besma Abidi d, Andreas Koschan d, Mongi Abidi d

a Department of Mathematics and Computer Science, North Carolina Central University, NC 27713, USAb GE Global Research Center, Nikayuna, NY 12309, USAc Third Dimension Technologies LLC, Knoxville, TN 37920, USAd Imaging, Robotics, and Intelligent Systems Laboratory, Department of Electrical Engineering and Computer Science, The University of Tennessee Knoxville, TN 37996, USA

a r t i c l e i n f o

Article history:Received 5 September 2008Received in revised form 1 September 2009Accepted 26 October 2009

Keywords:Camera handoffMulti-camera multi-object trackingResource managementSurveillance system

0262-8856/$ - see front matter � 2009 Elsevier B.V. Adoi:10.1016/j.imavis.2009.10.013

* Corresponding author. Tel.: +1 919 530 6237; faxE-mail addresses: chchen@nccu.edu (C.-H. Chen

davidpage@ieee.org (D. Page), besma@utk.edu (B. AKoschan), abidi@utk.edu (M. Abidi).

a b s t r a c t

Camera handoff is a crucial step to obtain a continuously tracked and consistently labeled trajectory ofthe object of interest in multi-camera surveillance systems. Most existing camera handoff algorithmsconcentrate on data association, namely consistent labeling, where images of the same object are iden-tified across different cameras. However, there exist many unsolved questions in developing an efficientcamera handoff algorithm. In this paper, we first design a trackability measure to quantitatively evaluatethe effectiveness of object tracking so that camera handoff can be triggered timely and the camera towhich the object of interest is transferred can be selected optimally. Three components are considered:resolution, distance to the edge of the camera’s field of view (FOV), and occlusion. In addition, most exist-ing real-time object tracking systems see a decrease in the frame rate as the number of tracked objectsincreases. To address this issue, our handoff algorithm employs an adaptive resource management mech-anism to dynamically allocate cameras’ resources to multiple objects with different priorities so that therequired minimum frame rate is maintained. Experimental results illustrate that the proposed camerahandoff algorithm can achieve a substantially improved overall tracking rate by 20% in comparison withthe algorithm presented by Khan and Shah.

� 2009 Elsevier B.V. All rights reserved.

1. Introduction

With the increase in the scale and complexity of a surveillancesystem, it becomes increasingly difficult for a single camera toaccomplish object tracking and monitoring with the required reso-lution and continuity. Camera networks emerge and find extensiveapplications. The employment of multiple cameras not only im-proves coverage but also brings in more flexibility. However, theuse of multiple cameras induces problems such as camera handoff.Camera handoff is a decision process of transferring a mobile ob-ject from one camera to another, wherein consistent labeling solv-ing the identity problem among multiple observing cameras andlaying the foundation for camera handoff. In general, camera hand-off regulates the collaboration among multiple cameras and an-swers the questions of When and Who: when a handoff requestshould be triggered to secure sufficient time for a successful con-sistent labeling and who is the most qualified camera to take over

ll rights reserved.

: +1 919 530 6125.), yyao1@utk.edu (Y. Yao),bidi), akoschan@utk.edu (A.

the object of interest before it falls out of FOV of currently observ-ing camera.

Most existing camera handoff algorithms focus on developingefficient consistent labeling schemes. In the literature, consistentlabeling methods could be grouped into three main categories:(I) feature-based, (II) geometry-based, and (III) hybrid-based ap-proaches. In feature-based approach [1,2,33], color or other distin-guishing features of the tracked objects are matched, generatingcorrespondence among cameras. The geometry-based approachcan be divided into three sub-categories: location-based, align-ment-based, and homograph-based approaches. In location-basedapproach [3,4], consistent labeling can be established by projectingthe trace of the tracked object back into the world coordinate sys-tem, and then establishing equivalence between objects projectedonto the same location. In alignment-based approach [5,6], thetracks of the same object are recovered across different camerasafter being aligned by the geometric transformation between cam-eras. The homography-based approach [7,8,31,32] obtains positioncorrespondences between overlapped views in the 2D image plane.For instance, Calderara et al. [31] used the likelihood that is com-puted by warping the vertical axis of the new object on the FOVof the other cameras and computing the amount of match therein.This improves the algorithm’s capability in handling both the cases

852 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

of single individuals and groups. Hu et al. [32] selected principalaxes of people, homography, as the primary features for matching.The hybrid approach [9] is a combination of geometry and feature-based methods.

Most existing consistent labeling methods [5,6,23] need a cer-tain amount of time/frames to be carried out successfully. This val-idates the importance of when to trigger a handoff request,because a timely handoff ensures that a successful consistentlabeling before the object of interest falls out of FOV of currentlyobserving camera. Meanwhile, the question of Who is challengingwhen the object of interest can be observed by multiple cameras.Most existing handoff algorithms choose the camera to whichthe object of interest is approaching. This simple rule is frequentlyinsufficient and leads to unnecessary handoffs. Note that in es-sence, our proposed method mainly addresses multi-object track-ing with joint views. Although the works of Javed et al. [22,25],Kang et al. [26], and Lim et al. [24] can consistently label the objectin the case with disjoint views, those tracking systems cannot de-tect the occurrence of unusual events due to the lack of continuousobservations on the object. This may cause a serious loophole in asurveillance system. Therefore, the inspiration of introducing thetrackability measure in the paper is to assist the camera handoffalgorithm to prevent the occurrence of occlusion or discontinuityfor a multi-camera surveillance system with overlapped FOVs. Onthe other hand, the camera handoff algorithm can transfer theto-be-occluded or to-be-unseen objects to another proper camerabeforehand. In comparison, the works of Javed et al. [22,25], Kanget al. [26], and Lim et al. [24] can only be used for the compensa-tion purpose.

Due to the lack of research work addressing the questions ofWhen and Who, there is no clear formulation to govern the transi-tion between adjacent cameras. As a result, the abovementioned

Fig. 1. Image sequence examples of: (a) occlusion, (b) distan

camera handoff algorithms, concentrating on consistent labeling,are unable to optimize the system’s performance in terms of hand-off success rate. For instance, a handoff request in the work of Khanand Shan [5] is triggered when the object is close to the edge of thecamera’s FOV. No quantitative measure is given to describe the dis-tance that is considered as close to the edge of the camera’s FOV.One exemplary camera handoff approach in the work of De Silvaet al. [10] selects the successive camera by measuring which cam-era could obtain a better frontal view of the person to achieve abetter recognition rate. A quantitative measure is derived and issufficient for the face recognition applications. However, the mea-sure of frontal view is unable to evaluate the overall quality in ob-ject tracking. To select the optimal camera and minimizeunnecessary handoff requests, multiple criteria should be consid-ered including resolution, occlusion, and distance to the edge ofthe camera’s FOV. Fig. 1 shows examples of these criteria. Fig. 1aillustrates the scenarios where two objects of interest are movingtowards each other, leading to a high probability of occlusion. Itis desirable to transfer to-be-occluded object to another camerato avoid the potential occlusion. In Fig. 1b, the object of interestis moving toward the boundaries of the camera’s FOV, which alsorequires a transition between cameras before it falls out of theFOV of the current observing camera. In parallel, the object ofinterest in Fig. 1c is moving away from the camera along the cam-era’s optical axis. As a result, the resolution of the object decreasesto where it is infeasible for the camera to maintain its track. Undersuch condition, a handoff is necessary as well. Therefore, in this pa-per, we propose the trackability measure including these multiplecomponents, each of which describes different aspect of objecttracking. Equipped with the quantified and comprehensive mea-sure of the effectiveness of object tracking we can answer the ques-tions of When and Who with the optimized solution.

ce to the edge of the camera’s FOV, and (c) resolution.

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 853

In addition, most multiple objects tracking systems [11–14] findit difficult to maintain a constant frame rate given limited re-sources. Note that frame rates in this paper represent the numberof processed frames per second by the tracking system for execut-ing functions such as tracking, crowd segmentation, and behavioralunderstanding, instead of the number of read-in frames by cam-eras themselves. This difference occurs due to the tracking systemincapable of processing each read-in frame for accommodating theexecution of all functions in real-time given limited resources, eventhough cameras themselves are capable to acquire more frames.Herewith, resources include (I) CPU capacity for executing objecttracking, crowd segmentation, and behavior understanding in anautomated manner [16] and (II) network bandwidth for exchang-ing camera handoff information. The computational complexityof most existing tracking systems [11–14] is of the order fromNpO(n) to NpO(n3) [15], where Np is the number of tracked objectsand n represents the number of steps to execute the algorithm.There inherently exists an upper bound on the number of objectsthat can be tracked simultaneously without deteriorating the sys-tem’s frame rate.

Those unprocessed read-in frames may be dropped immedi-ately or reserved for future reference. Therefore, it is crucial for atracking system to be able to maintain a reasonable frame rate inreal-time. A lower frame rate may result in the following prob-lems: (I) the surveillance system’s real-time ability to automati-cally detect a threatening event degrades, causing possible obser-vation leaks. This dangerous loophole impedes the practical appli-cation of these real-time multi-camera multi-object trackingsystems [17] and (II) the decreased frame rate also affects theperformance of consistent labeling and consequently camera hand-off, because a successful execution of consistent labeling requiresaccumulated information of the object of interest over a periodof time [5,6,23]. The reduced frame rate leads to a decreased num-ber of available frames/information for carrying out consistentlabeling successfully.

In summary, the contributions of this paper are: (I) a trackabilitymeasure is introduced to quantitatively evaluate the effectiveness ofa camera in observing the tracked object. This gives a quantified met-

Handoff request side

No

H

C

Yes

Handoff Request

Handoff Success

Handoff Response

Yes

Yes

No

j*=argmax{Bij’}

Handoff Failure

TargetVisible

YesNo

No

CT,ij = 1

CE,ij* = 1

Consistent Labeling

Fig. 2. Flow chart of the propose

ric to direct camera handoff for continuous and automated trackingbefore the tracked object is occluded or falls out of the FOV of cur-rently observing camera, (II) an adaptive resource managementalgorithm that automatically and dynamically allocates resourcesto objects with different priority ranks is developed, and (III) basedon the trackability measure and adaptive resource management, acamera handoff algorithm is designed. The proposed handoff algo-rithm can achieve a significantly improved overall tracking ratewhile maintaining a constant frame rate of each camera.

The remainder of this paper is organized as follows. Section 2illustrates the overall system architecture of our proposed camerahandoff algorithm. Section 3 defines the trackability measure. Sec-tion 4 presents the adaptive resource management algorithm.Experimental results are demonstrated in Sections 5 and 6 con-cludes the paper.

2. Camera handoff

The flow chart of the proposed camera handoff algorithm isshown in Fig. 2, where operations are carried out at the handoff re-quest and handoff response sides. Let the jth camera be the handoffrequest side and the ith object be the one that needs a transfer. Tomaintain persistent and continuous object tracking, a handoff re-quest is triggered before the object of interest is untraceable orunidentifiable in the currently observing camera. The object ofinterest may become untraceable or unidentifiable due to the fol-lowing reasons: (I) the object is being occluded by other objects,(II) the object is leaving the camera’s FOV, and (III) the object’s res-olution is getting low. Accordingly, three criteria are defined in thetrackability measure to determine when to trigger a handoff re-quest: occlusion (MO), distance to the edge of the camera’s FOV(MD), and resolution (MS). Let MO,ij, MD,ij, and MS,ij be the MO, MD,and MS values of the ith object observed by the jth camera, respec-tively. These three components MO,ij, MD,ij, and MS,ij, to be discussedin details in Section 3, are scaled to [1] where zero means that theobject is untraceable or unidentifiable and one means that thecamera has the best effectiveness in tracking the object.

Handoff response side

Resource Management

Handoff Rejectandoff Response

NoYes

rjth

r

krj Nn ,',

1,' ≤∑

=

onsistent Labeling

Update rjthN ,',

d camera handoff algorithm.

854 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

Define the trigger criterion CT,ij as:

CT;ij ¼ ðMO;ij < TOÞ ^dMO;ij

dt< 0

� �� �_ ðMD;ij < TDÞ ^

dMD;ij

dt< 0

� �� �_ ðMS;ij < TSÞ ^

dMS;ij

dt< 0

� �� �; ð1Þ

where ^ and _, both logical symbols, represent ‘and’ and ‘or’operations, respectively. TO, TD, and TS, associated with MO, MD,and MS, represent the predefined thresholds for triggering handoffand are mainly determined by the time needed for handoff exe-cution and the objects’ maximal moving speed. A handoff request,therefore, is triggered and broadcasted, if CT,ij = 1, which suggeststhat at least one of the three components is below the predefinedthreshold and is decreasing. The decreasing criterion in Eq. (1)may appear to be redundant since at the first instance that oneof the components becomes below the predefined threshold, itsderivative should be necessarily negative. However, it requires acertain amount of execution time from the point when a handoffis triggered to the point when a handoff is granted. During thisperiod of time, the trackability measure may change due to thedynamics of the object of interest. Chances are the measure isstill below the threshold but its value is increasing. For instance,the object of interest may change its motion direction and beginto move towards the center of the camera’s FOV. To address theaforementioned situation and avoid back-and-forth transitions,the decreasing criterion is included. Furthermore, since each com-ponent of the trackability measure is computed from the esti-mated position of the object of interest, noise and possiblejittering exist primarily due to the limited detection and trackingaccuracy. Kalman filtering is, therefore, employed to smooth thetrackability measure by exploiting the dynamics of the object.In so doing, the probability of back-and-forth transition can bereduced, which in turn improves the efficiency of the proposedhandoff algorithm.

Afterwards, the jth camera keeps tracking the ith object andwaits for confirmation responses from adjacent cameras whilethe object is still visible. At the handoff response side, the (j0)thcamera examines its current load. Let Nth;j0 ;r denote the maximumnumber of objects with a priority rank smaller than or equal to rthat can be tracked simultaneously and nj0 ;r the number of objectswith a priority rank r that have been tracked by the (j0)th camera.A positive handoff response for the ith object is granted, ifPr

k¼1nj0 ;k < Nth;j0 ;r , which represents that the total number oftracked objects from different priority ranks has to be less thanthe maximum number of tracked objects in the system. Toachieve a higher acceptance rate or equivalently a higher handoffsuccess rate, the thresholds Nth;j0 ;r should be adaptively adjustedaccording to the system’s current load. Given limited capacity,more resources should be allocated to objects with higher prior-ities at the cost of dropping out objects with lower priorities.Such a system provides a higher threat awareness level comparedto systems where objects have the same priority ranks. Some-times additional requirements on the overload probabilities of ob-jects with different priority ranks are given. To meet theserequirements, we need an online learning process to automati-cally adjust the distribution of the capacities according to esti-mated system load.

Since the priority rank plays an important role in evaluatingthe load of a camera, before continuing with the proposed hand-off algorithm, we stop here to clarify the possible selection meth-ods of the priority rank. The priority rank assignment isapplication dependent. Frequently, the priority rank depends onthe object’s behavior. For example, in an airport surveillance sys-

tem, the moving direction is a good hint to allocate the priorityrank. Passengers walking in the opposite direction of an exit hallway should be assigned a higher priority rank than passengersfollowing the regulated direction. Another example is a workplacesurveillance system, where a close observation is necessary onworkers handling valuable assets [34]. Workers in the closewhereabouts of these high valuable assets should be assigned ahigher priority rank. The initial priority ranks are obtained fromlow level behavior understanding that can be performed easilyonce the object of interest is detected. Fore example, the motiondirection and location as discussed in the previous two examples.These initial priority ranks can be adjusted and refined as theobservation is long enough to carry out more complex behaviorunderstanding.

Back to the handoff request side, if no positive handoff responseis received before the jth camera loses the track of the ith object, ahandoff failure is issued. Otherwise, consistent labeling is carriedout between the handoff request side and all available candidatecameras. A handoff failure means that the ith object is no longertracked or monitored by any camera in the system. It might bepicked up by one camera later on once it enters the camera’sFOV and the camera has resource to process it. However, withoutsuccessful handoff its original identity is lost and a new identifyis assigned instead. In order to select the most appropriate candi-date camera to take over the object of interest in the pool of can-didate cameras, the one with the lowest system load PO,ij0 and thehighest trackability measure Qij0 is chosen:

Bij0 ¼ ð1� PO;ij0 ÞQij0 ; ð2Þ

where PO,ij0 is the overload probability of the ith object in the (j0)thcamera and Qij0 denotes the trackability measure of the ith object inthe (j0)th camera. The detailed definition of Qij0 and PO;ij0 are given inSections 3 and 4, respectively. The term (1 � PO,ij0) is included to re-duce the chances of choosing a camera with high system load,which ensures an evenly distributed system load across all cameras.

The execution criterion CE;ij� is defined as:

CE;ij� ¼ ðMO;ij < MO;ij� Þ _dMO;ij�

d> 0

� �� �^ ðMD;ij < MD;ij� Þ _

dMD;ij�

d> 0

� �� �^ ðMS;ij < MS;ij� Þ _

dMS;ij�

d> 0

� �� �: ð3Þ

Since an efficient tracking system should be able to direct camerahandoff for continuous and automated tracking before the trackedobject is occluded or falls out of the FOV of currently observingcamera. In the meanwhile, system load can be evenly distributedwithout deteriorating the frame rate of each camera. Thus, the ithobject is transferred to the (j�)th camera if CE;ij� ¼ 1.

Note that in some applications such as the work of Lien andHuang [28], each object of interest is tracked by multiple cam-eras to obtain more or better monitoring results. Our proposedhandoff algorithm can easily be applied to these applications, be-cause each camera still needs to handoff the object of interest toanother camera that is not tracking the object when eitherocclusion, low resolution, or falling out of its FOV occurs. Forone extreme case where the object of interest is tracked or mon-itored by all the cameras that can see it, our algorithm can beemployed with the following modifications. The handoff trigger-ing and camera selection processes are not necessary. Once anobject of interest enters the field view of a camera, the cameracomputes its current load to determine whether it has sparecomputational resources to accept the object of interest. If theobject of interest is accepted, consistent labeling is performedbetween this camera and adjacent cameras.

1 2 NmaxNmax-10

µ 2µ Nmax µ

n

nµ

∑ =prNrg gλ prN

λ∑ =prN

g g1λ ∑ =prN

g g1λ

11 22 NmaxNmax-100

µ 2µ Nmax µ

nn

nµ

∑ =prNrg gλ prN

λ∑ =prN

g g1λ ∑ =prN

g g1λ

Fig. 3. Illustration of the state transition of an M/M/Nmax/Nmax/1/FCFS queuingsystem, which is used to model a multi-object tracking system.

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 855

3. Trackability measure

In the following discussion, formulas are derived for a singletarget observed by a single camera. For clear representation, thesubscripts i and j are omitted. Assume that from object trackingthe target image’s relative scale q and center of mass g = [gx gy]T

are estimated. The size preserving tracking algorithm discussedin [18] can be used for this purpose. The resolution componentMS is defined as:

MS ¼ aSfZr¼ aSq; ð4Þ

where f represents the camera’s focal length, Zr is the average targetdepth, and aS denotes the normalization coefficient. Let fmax denotethe maximum focal length of the camera and Zr,min the minimumdistance between the target and the camera so that the target canbe observed completely. The normalization coefficient aS is given by

aS ¼Zr;min

fmax:

To reserve enough computation time for the execution of thehandoff between cameras, the object should remain at a distancefrom the boundaries of the camera’s FOV. This margin distance isalso affected by the object’s depth. When the object is at a closerdistance to the observing camera, its projected image undergoesa larger displacement in the image plane. Therefore, a larger mar-gin should be reserved. In our definition, a varying polynomialpower is used to achieve different decreasing/increasing ratesand in turn different margin distances. The MD term is defined as:

MD ¼12

1� 2gx

Nx� 1

���� ����� �2

þ 1�2gy

Ny� 1

���� ����� �2" #( )b1qþb0

; ð5Þ

where Nx (Ny) denotes the width (height) of the image. The MD com-ponent evaluates the distance from the four image boundaries de-fined by x ¼ � Nx

2 and y ¼ � Ny

2 . The coefficients b1, and b0 are usedto adjust the polynomial power according to the target depth. Inour experiments, we choose b1, and b0 according to

b1fmax=Zr;min þ bo ¼ 12b1fmax=Zr;min þ bo ¼ 0:5

�, which leads to b1 ¼ �

Zr;min2f max

and b0 ¼ 1:5.

The above equations are obtained empirically.In order to continuously track multiple objects, the system

should be able to transfer the tracked object with latent occlusionto another camera with a clear view. Therefore, the occlusioncaused by objects’ motion is also considered. The MO term is de-fined as:

MO ¼ aO mini–j

ðgx;i � gx;jÞ2 þ ðgy;i � gy;jÞ

2n o� �b1qþb0

; ð6Þ

where aO is a normalization weight. [gx,i gy,i]T and [gx,j gy,j]T denotethe centers of mass of any pair of objects in the field of view of cur-rently tracking camera. Occlusion can be caused by stationaryobstacles, such as tables and cabinets, or other moving pedestriansin the environment. Thus, those objects do not only represent mo-bile objects, but stationary. Nevertheless, how to differentiatewhich one is in the front or in the back is not the scope of this paper.Interesting readers can refer the work of Hoiem et al. [27]. In con-clusion, the trackability measure is given by:

Q ¼ MOðwSMS þwDMDÞ; ð7Þ

where wS and wD are importance weights for the resolution and dis-tance components, respectively. The sum of these importanceweights is one. The selection of the importance weights is applica-tion dependent. We purposefully reserve the freedom for users tochoose different importance weights according to their special

requirements to increase our algorithm’s flexibility. Meanwhile, de-fault values can be used if the corresponding variables are not spec-ified by users. The default values of wS and wD are simply 0.5. Theresolution and distance components describe two aspects of thesame interaction between the target and the observing camera.Summation is used to combine the quantitative measures of thesetwo aspects. In contrast, the occlusion component measures theinteraction between two targets, which is independent of the inter-action between the target and the camera. Therefore, the occlusioncomponent measure is included via multiplication.

4. Adaptive resource management

In this section, we first derive the overload probabilities of ob-jects at different priority ranks and then introduce our resourcemanagement algorithm. In the following discussion, formulas arederived for any single camera. For clear representation, the sub-script j is omitted.

4.1. Probability of camera overload

Assume that the arrival of objects with priority rank r follows aPoisson distribution with a rate kr. The amount of time that an ob-ject remains within the camera’s FOV, is independent and followsan exponential distribution with mean of 1

l. The exponential distri-bution describes a process where events occur continuously andindependently at a constant average rate. The exponential distribu-tion has been proved to be a good approximate model for servicerates and has been widely used in queueing systems such as bankservice and wireless communication [21,35,36]. The case of an ob-ject of interest entering the FOV of a surveillance system for theservice of ‘‘tracking” and ‘‘monitoring” is similar to the case of acustomer entering a bank for the service of account transactionsand the case of a mobile call entering the base station for the ser-vice of wireless communication. Therefore, the exponential distri-bution is chosen to formulate the amount of time an objectremains within the FOV. Let Nth,r be the maximum number of ob-jects with a priority rank smaller than or equal to r that can betracked simultaneously. We deliberately add Nth,0 = 0 to simplifythe formulation. Let the maximum number of objects that can betracked simultaneously be Nmax and the total number of priorityranks be Npr.

To derive the overload probability of objects at different pri-ority ranks, a multi-object tracking system is modeled as an M/M/Nmax/Nmax/FCFS queuing system, where M represents arrival ordeparture distribution as a Poisson distribution, and the servingrule is first come first serve (FCFS) [19,20]. Such a system consti-tutes a Markov process of the birth–death type, as shown inFig. 3. We examine the queuing system when it is at equilib-rium. Under proper conditions, such equilibrium will be reachedafter the system has been operating for a period of time. It alsoimplies that the probability of n objects being tracked, P(n),eventually becomes stable, where n ranges from 0 to Nmax.Therefore, the probability of the nth state can be computed giventhe probability of the (n�1)th state:

856 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

PðnÞ ¼PNpr

k¼rkk

nlPðn� 1Þ; ð8Þ

where Nth,r�1 < n 6 Nth,r. This relation leads to:

PðnÞ ¼ Pð0Þn!

Yr�1

k¼1

XNpr

l¼k

kl

l

!Nth;k�Nth;k�1 XNpr

k¼r

kk

l

!n�Nth;r�1

; ð9Þ

where

Pð0Þ ¼XNmax

n¼0

1n!

Yr�1

g¼1

XNpr

h¼g

kh

l

!Nth;g�Nth;g�1 XNpr

g¼r

kg

l

!n�Nth;r�124 358<:

9=;�1

:

ð10Þ

According to (9) and (10), the overload probability for the objectwith a priority rank of r is given by

PO;r ¼XNmax

n¼Nth;r

PðnÞ: ð11Þ

The overload probability is one important criterion to evaluate theperformance of a multi-camera system fulfilling multiple objecttracking. It determines the number of objects that may be droppeddue to limited resources. Therefore, in practice, it is desirable to dis-tribute the resources dynamically according to the system’s currentload and the object’s priority rank. From the above derivations, welearn that Nth,r determines the overload probabilities. Given theoverload probabilities for objects at different priority ranks, wecould adjust these thresholds to achieve the requirements. If thereal-time estimated overload probability, bPO;r , for the object witha priority rank r exceeds the desired overload probability, Pth,r, weneed to decrease the thresholds Nth,k with 1 6 k < r or increase thethresholds Nth,k with r 6 k 6 Npr. Based on this key concept, we de-velop our adaptive resource management algorithm.

4.2. Algorithm description

The flow chart of our resource management algorithm is illus-trated in Fig. 4. If given the known arrival rate kr with 1 6 r 6 Npr,

the initial thresholds Nth,r can be computed as Nth;r ¼Pr

k¼1kkPNpr

k¼1kk

Nmax.

If not, the initial values can be set to Nth;r ¼ rNmaxNpr

. Let nr be the num-

ber of tracked objects with priority rank r. As we mentioned before,ifPr

k¼1nk < Nth;r , the handoff request is accepted. Otherwise the

Reject object

No Yes

No

Pth,rrOP ,ˆ

Fr-1 = Fr-1-r and Fr = Fr+r

Update rOP ,ˆ

rth

pr

i i Nn ,1<∑ =

Yes

If Fk<-Fth, then Nth,k= Nth,k-1, Fk=0and

If Fk>Fth, then Nth,k= Nth,k+1, Fk=0,where k = 1,…, Npr

Accept object

Update rλ̂

Reject object

No Yes

No

Pth,rrOP ,ˆ Pth,rrOP ,ˆ

Fr-1 = Fr-1-r and Fr = Fr+r

Update rOP ,ˆUpdate rOP ,ˆ

rth

pr

i i Nn ,1<

=

Yes

If Fk<-Fth, then Nth,k= Nth,k-1, Fk=0and

If Fk>Fth, then Nth,k= Nth,k+1, Fk=0,where k = 1,…, Npr

Accept object

Update rλ̂

∑

Fig. 4. Flow chart of the proposed adaptive resource management scheme. Ingeneral, if the real-time estimated overload probability, bPO;r , for the object with apriority rank r exceeds the predefined or desired overload probability Pth,r, we needto decrease the thresholds Nth,k with 1 6 k < r or increase the thresholds Nth,k withr 6 k 6 Npr.

handoff request is rejected. Afterwards, the real-time arrival ratesof objects with different ranks k̂r are estimated during the timeframe 1

l. Note that even in scenarios with known average arrival

rates, it is also necessary to estimate the real-time arrival rates soas to adjust resource allocation among objects with different ranksaccording to current system load. Given the estimated k̂r , the real-

time overload probability, bPO;r , for objects with rank r can be com-

puted according to Eq. (11). The estimated overload probability bPO;r

is then compared with the predefined or desired overload probabil-

ity Pth,r. If bPO;r > Pth;r , the thresholds Nth,r�1 and Nth,r should be ad-justed. Ideally, we want to increase Nth,r and decrease Nth,r�1.

However, varying Nth,r�1 and Nth,r also affects the overload prob-ability of objects from other ranks. In addition, the estimated over-load probability bPO;r may fluctuate, which in turn inducesunnecessary adjustment of the thresholds. Therefore, to smooththe decisions over a period of time and incorporate the require-ments from objects of other ranks, a flag is set up for the thresholdsat each priority rank, which is defined as Fr. If bPO;r > Pth;r , decreaseFr�1 by r suggesting that a decrease in Nth,r�1 is requested and in-crease Fr by r suggesting that an increase in Nth,r is preferred. Sinceit is cumulative, Fr takes the previous decision into consideration aswell. If multiple handoff requests are received, the same procedurerepeats for each object and the decisions from multiple objects arecombined in Fk with k = 1, . . ., Npr. The contribution in Fk from eachobject is associated with its priority rank. In so doing, more impor-tance is assigned to the decisions from objects with higher priori-ties and the following adjustment of the thresholds favors asmaller overload probability for objects with higher priorities. Inaddition, a more prompt response is also achieved for objects withhigher priorities. In addition, the priority rank is included to im-prove the system’s level of threat awareness. Priority ranks canbe assigned to tracked objects according to their behaviors. Forexample, in the surveillance of an airport, passengers moving alongthe indicated direction (from the gates to exit) in the hall way areassigned with a lower priority while passengers moving in theopposite direction are assigned with a higher priority.

After all the objects have been processed, the thresholds are up-dated. If Fk > Fth, Nth,k is increased by one, where Fth is a predefinedthreshold. If Fk < �Fth, Nth,k is reduced by one. After the adjustmentof Nth,k, the corresponding Fk is reset to zero. Nth,k remains the same

if |Fk| 6 Fth. The complexity of computing bPO;r and Fk is of the order

OPNpr

k¼1nk

� . The adjustment of thresholds Nth,k has a computational

complexity of O(Npr). As a result, the proposed resource adjustmentis able to dynamically relocate the available resources with mar-ginally increased computational cost in comparison with the com-plexity of multiple object tracking and consistent labeling.

4.3. Example system

To further study the effect of adjusting Nth,r for adaptive re-source management, we consider the asset monitoring system asan example. In this application, people who are close to or carrythe valued asset should be adaptively allocated more resourcewhen the system’s load is high. This is because the tracking systemneeds to continuously track the people to immediately detect anythreats to the valued asset. A system with Npr = 2, therefore, repre-sents a system with only two types of objects, high and low prior-ities. Let kH and kL be the arrival rate of objects with high and lowpriorities. The probability of n tracked objects is given by

PðnÞ ¼Pð0Þ

n!kHþkL

l

� n0 � n � Nth;

Pð0Þn!

kHþkLl

� Nth kHl

� n�NthNth < n � Nmax;

8><>: ð12Þ

1 2 3 4 5 610-2

10-1

100

Nth

Ove

rload

pro

babi

lity

PO,L

PO,H

Initial value: Nth=2

Adjusted value:Nth=5

Pth,L=0.2

Fig. 5. Illustration of the overload probabilities PO,H and PO,L as functions of Nth.kHl ¼ 2; kL

l ¼ 1, Nmax = 6. The corresponding PO,H and PO,L are 0.015 and 0.710,respectively. In the beginning, the PO,L is much higher than the probabilityPth,L = 0.2. Our resource management algorithm is able to increase Nth by one atone time so as to decrease PO,L. At equilibrium, we arrive at Nth = 5 resulting inPO,H = 0.035 and PO,L = 0.142.

10 meters

13 meters

11 meters

10 meters

9.5 meters

: Camera without covering entry door

: Field of view

: Entry door

Height: 3 metersCamera 1

Camera 2

: Camera covering entry door

Camera 5

Camera 4

Camera 3

Camera 6

Camera 7

10 meters

13 meters

11 meters

10 meters

9.5 meters

: Camera without covering entry door

: Field of view

: Entry door

Height: 3 metersCamera 1

Camera 2

: Camera covering entry door

Camera 5

Camera 4

Camera 3

Camera 6

Camera 7

Fig. 6. Floor plan of the experimental environment.

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 857

with

Pð0Þ ¼XNth

n¼0

1n!

kH þ kL

u

� �n

þXNmax

n¼Nthþ1

1n!

kH þ kL

u

� �Nth kH

u

� �n�Nth !�1

:

ð13Þ

The overload probabilities for the object of high and low prioritiesare PO,H = P(Nmax) and PO;L ¼

PNmaxn¼Nth

PðnÞ. These two probabilitiesare monotonously increasing and decreasing functions of thethreshold Nth as shown in Fig. 5. Suppose we have kH

l ¼ 2; kLl ¼ 1,

and Nmax = 6. The initial Nth is initialized byPr

g¼1kgPNpr

g¼1kg

Nmax ¼ 2. The cor-

responding PO,H and PO,L are 0.015 and 0.710, respectively. The PO,L ismuch higher than the probability Pth,L = 0.2. Our resource manage-ment algorithm is able to increase Nth by one at one time so as todecrease PO,L. At equilibrium, we arrive at Nth = 5 resulting inPO,H = 0.035 and PO,L = 0.142. Fig. 5 also depicts the adjustmentprocess.

Fig. 7. The computed resolution component MS from frames acquired by a real-time tracking system as the object of interest moves toward the camera along theoptical axis.

5. Experiment results

In this section, we study the individual and combined effects ofthe three components, MS, MD, and MO, defined in the trackabilitymeasure. Afterwards, experiments are conducted to verify the effec-tiveness of our proposed camera handoff algorithm via video se-quences generated by ourselves and dataset S7 in PETS’ 2006 [29].Fig. 6 shows the floor plan of the experimental environment. Yaoet al.’s camera placement algorithm [30] is used in our experimentto optimally preserve overlapped FOVs. Static perspective cameraswith a resolution of 640 � 480 are placed along the walls at a heightof 3 m with a tilt angle hT of�30�. Two priority levels are assigned tothe objects, Npr = 2. The maximum number of objects that can betracked simultaneously is three for all cameras, Nmax = 3 in our case.The thresholds TO, TD, and TS are 0.2 to comply with the time neededfor executing camera handoff (5 s average) and the maximal movingspeed of the objects (0.6 m/s). The surveillance system in our exper-iment includes behavioral understanding in addition to multiple ob-ject tracking algorithm. The behavioral understanding part isnecessary for assigning different priorities to tracked objects. As a re-sult, the surveillance system illustrated in our experiment can only

sustain at most three tracked objects without deteriorating the sys-tem’s frame rate. In other words, the system only includes multi-ob-ject tracking. Thus, it can monitor 10 objects without deterioratingthe frame rate. This observation also exemplifies the importance ofresource management in a real-life scenario. Since the focus of thispaper is not developing object tracking and consistent labeling algo-rithms, we use existing algorithms for multi-object tracking andconsistent labeling. Image difference and homography-based ap-proaches are implemented for object tracking and consistent label-ing, respectively.

5.1. Experiments on trackability measure

From the definition of the trackability measure, we first studythe individual effect of MS, MD, and MO based on real-time trackingsystem where camera 2 indicated in Fig. 6 is used in this experi-ment. According to the derivation introduced in (4) and (5), we no-tice that the components MS and MD mainly describe the variationsalong and orthogonal to the camera’s optical axis, respectively. Asexpected, in Figs. 7 and 8, MS increases as the target moves towardthe camera along the optical axis and MD increases as the targetmoves toward the image center. In Fig. 9, two targets walk

Fig. 8. The computed distance component MD from frames acquired by a real-timetracking system as the object of interest moves toward the image center.

Fig. 9. The computed occlusion component MO from frames acquired by a real-timetracking system. Two objects move across the camera’s FOV at different speeds,resulting in a decreased relative distance between them.

858 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

diagonally across the camera’s FOV with the same direction at dif-ferent speeds. As a result, the relative distance between them de-creases. This variation is indicated by a decreased MO, as shownin Fig. 9.

Fig. 10 illustrates sampled frames at fn and fn+15 from a real-timetracking sequence 1 with two static perspective cameras. The cam-eras’ positions are specified in Fig. 6 as cameras 1 and 2. Table 1lists MS,ij, MD,ij, and MO,ij for the ith object observed by the jth cam-era at frames fn and fn+15, where i ranges from 1 to 5 and j is either 1

Fig. 10. Illustration of the effectiveness of our proposed trackability measure in the came

Table 1The illustration of MO,ij, MD,ij, and MS,ij shown in Fig. 10.

Object 1 (i = 1) Object 2 (i = 2)

fn fn+15 fn fn+15

Camera 1 (j = 1) MO,ij 0.31 0.15 0.41 0.15MD,ij 0.6 0.5 0.5 0.4MS,ij 0.43 0.41 0.41 0.42

Camera 2 (i = 2) MO,ij 0 0.25 0 0MD,ij 0.6 0.15 0.6 0.6MS,ij 0.42 0.41 0.43 0.41

or 2. Fig. 11 illustrates continuous trackability measures, MS,ij, MD,ij,and MO,ij, of objects 1, 2, 3, 4, and 5 from frame fn to fn+20 in real-time tracking sequence 1. In frame fn, object 4 is blocked by object3 in camera 1 while object 1 is blocked by object 2 in camera 2.Both objects can be observed without occlusion in the other cam-era. Thus, objects 4 and 1 are transferred to cameras 2 and 1,respectively. Object 5 in the camera 1 is close to objects 3 and 4.Its MO,51 is 0.18, less than TO = 0.2. A handoff request is, therefore,

ra handoff procedure at sampled frames fn and fn+15 in real-time tracking sequence 1.

Object 3 (i = 3) Object 4 (i = 4) Object 5 (i = 5)

fn fn+15 fn fn+15 fn fn+15

0 0 0 0 0.18 0.250.45 0.5 0.43 0.4 0.38 0.140.43 0.42 0.3 0.3 0.45 0.6

0.6 0.5 0.24 0 0.5 0.60.9 0.6 0.85 0.56 0.43 0.150.42 0.42 0.41 0.41 0.40 0.41

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0 4 8 12 16 20Frame

Trac

kabi

lity

Mea

sure

MO,11MO,12MD,11MD,12MS,11MS,12S,12

S,11D,12D,11

O,12

MO,11

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Frame

Trac

kabi

lity

Mea

sure

MO,21MO,22MD,21MD,22MS,21MS,22S,22

S,21D,22

D,21

O,22

MO,21

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Frame

Trac

kabi

lity

Mea

sure

MO,31MO,32MD,31MD,32MS,31MS,32S,32

S,31D,32D,31

O,32

MO,31

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Frame

Trac

kabi

lity

Mea

sure

MO,41MO,42MD,41MD,42MS,41MS,42S,42

S,41

D,42D,41

O,42

MO,41

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Frame

Trac

kabi

lity

Mea

sure

MO,51MO,52MD,51MD,52MS,51MS,52S,52

S,51

D,52D,51

O,52

MO,51

0 4 8 12 16 20

0 4 8 12 16 20 0 4 8 12 16 20

0 4 8 12 16 20

Fig. 11. Illustration of continuous trackability measures, MS,ij, MD,ij, and MO,ij, of objects 1, 2, 3, 4, and 5 from frame fn to fn+20 in real-time tracking sequence 1.

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 859

triggered for object 5. On the other hand, camera 1 sends out ahandoff request to its adjacent camera 2 and receives a positive re-sponse. As a result, object 5 in camera 1 will be transferred to cam-era 2, as marked by a yellow rectangle. Similarly, in frame fn+15,object 5 in camera 2 is close to the edge of the camera’s FOV, whereits MD,52 is 0.15 and less than TD = 0.2. It requires camera handoff.On the other hand, camera 2 sends out the handoff request to itsadjacent camera 1 and the request is granted, which is markedwith a yellow rectangle in the camera 1. In general, we can see thattrackability measure gives a quantified metric to direct the camerahandoff successfully and smoothly before the tracked object is oc-cluded or falls out of FOV of currently observing camera.

5.2. Experiments on adaptive resource management

In order to illustrate the importance of our proposed adaptive re-source management in camera handoff, Fig. 12 illustrates sampledframes at fn and fn+15 from real-time tracking sequence 2 with three

static perspective cameras. The cameras’ positions are specified inFig. 6 as cameras 3, 4 and 5. To illustrate the effectiveness of adaptiveresource management, we focus on object 1. In frame fn, even thoughcamera 5 can see object 1, it does not track the object. This is becausecamera 4 tracks object 1 first and does not send out handoff requestto adjacent cameras. In frame fn+15, object 1 is moving out of FOV ofcamera 4 and camera 4 had send out handoff request to adjacentcameras 3 and 5 before frame fn+15. Since camera 3 has reached itsmaximum system load (PO;130 ¼ 0:9 and PO;150 ¼ 0:1) and MS, MD,and MO are not dominant factors in the camera selection process,camera 5 is the next best camera to track object 1. In general, ouradaptive resource management is able to guide the camera handoffprocedure to choose the least system load.

5.3. Experiments on overall performance

In order to examine the overall performance of our proposedcamera handoff algorithm including trackability measure and

Fig. 12. Illustration of the effectiveness of our proposed adaptive resource management in the camera handoff procedure at sampled frames fn and fn+15 in real-time trackingsequence 2.

860 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

adaptive resource management, the algorithm discussed in [5] isimplemented and serves as the comparison reference. The refer-ence algorithm simply triggers a handoff request whenever the ob-ject of interest is close to the edge of camera’s FOV withoutregarding the system’s load, object priority, and the next best cam-era to track the object. Note that since there is no direct works cor-responding to ours to the best of our knowledge, we choose Khanand Shah’ work as a symbolic algorithm to demonstrate problemswe face and then overcome in a real-life case. To accommodateKhan and Shah’s work to our experiments, we make the followingadjustments for their algorithms: (I) we trigger a handoff requestwhen its distance to the edge of the camera’s FOV (MD) is smallerthan the predefined threshold, TD, (II) we choose the next best cam-era by merely the biggest MD in adjacent cameras, and (III) accord-ing to our experiment, average 10 frames is necessary for Khan andShah’s work to carry out a successful consistent labeling in a gen-eral situation. The failure of consistent labeling may occur whenless than 10 frames are collected before the object is moving outof FOV of currently observing camera. One solution to reduce thepossibility of failure of consistent labeling is to increase the over-lapped views among adjacent cameras. This leads to the decreasedoverall coverage, thus, requiring more cameras to cover the area.This may not be practical in many cases. Thus, optimizing thetradeoff between coverage and overlapped views [30] is used inthis experiment. As a result, accumulating sufficient number offrames is necessary before objects fall out of FOV of currentlyobserving camera to avoid the failure of handoff process.

In our experiment, we first illustrate how frame rates fluctuatewhen not considering adaptive resource management scheme inthe tracking system. The overall tracking rate, the ratio betweenthe time of objects being tracked by the system and the total timeof objects staying in the FOV of the system, is used to describe the

system’s overall performance. To obtain a statistically valid estima-tion of the overall tracking rate, simulations are carried out to en-able a large amount of tests under various conditions. Severalpoints of interest are generated randomly to form a pedestriantrace. Overall tracking rate is obtained from simulation results of300 randomly generated traces. In order to understand the behav-ior of our proposed camera handoff algorithm facing varying arri-val rates of the objects with low and high priority, the ratio kL/kH

is set to vary from 0.8 to 1.2. The expected probability of cameraoverload for objects with low and high priorities is Pth,L = 0.2 andPth,H = 0.2. Note that once we lose the track of the object due to fail-ure of camera handoff, we will not recover it until the object movesto another adjacent camera.

Fig. 13 compares the performance of our adaptive resourcemanagement method and the reference algorithm [5] with variouskL/kH in term of the handoff success rate. The notation Adaptive-0.8suggests a system using our proposed resource management meth-od with kL/kH = 0.8 and the notation KS-0.8 means the referencesystem [5] with kL/kH = 0.8. Fig. 13a illustrates the system equippedwith our adaptive resource management can keep a steady framerate of 8 fps while the frame rate of the system based on the refer-ence algorithm varies between 3 fps and 8 fps. In addition, inFig. 13b and c, regardless of kL/kH, the overall tracking rate of ouradaptive approach is higher than that of the static approach. A con-siderable improvement in overall tracking rate by 20% is achievedin comparison with the Khan and Shah’s work. The observed infe-rior overall tracking rate of the reference method results from itsfluctuating frame rate. When the frame rate is low, less informa-tion is acquired for the execution of consistent labeling, hencedeteriorating the accuracy of identity matching and then the over-all tracking rate. In other word, the continuity of objects beingtracked in the system is compromised.

0

1

2

3

4

5

6

7

8

9

20 40 60 80 100 120 140 160 180 200

Time (Minutes)

Fram

e ra

te (f

ram

es/s

econ

d) Adaptive-1.2, 1, and 0.8KS--1.2KS--1KS--0.8

0

10

20

30

40

50

60

70

80

90

Time (Minutes)

Ove

rall

track

ing

rate

(%) f

or o

bjec

ts

with

hig

h pr

iorit

y

Adaptive-1.2KS--1.2Adaptive-1 KS--1Adaptive-0.8KS--0.8

0

10

20

30

40

50

60

70

80

Time (Minutes)

Ove

rall

Trac

king

rat

e (%

) fo

r ob

ject

s w

ith lo

w p

riorit

y

Adaptive-1.2KS--1.2Adaptive-1 KS--1Adaptive-0.8KS--0.8

20 40 60 80 100 120 140 160 180 200

20 40 60 80 100 120 140 160 180 200

a b

c

Fig. 13. Comparisons of camera handoff approaches with our proposed adaptive and Khan and Shah’ static resource management methods with various kLkH

: (a) the illustrationof how frame rates fluctuate when not considering the adaptive resource management scheme in the system. (b) Handoff success rate for objects with high priority and (c)handoff success rate for objects with low priority. In (a), (b), and (c), adaptive and KS denote our proposed adaptive and Khan and Shah’ static resource management methodsrespectively.

Fig. 14. Illustration of the effectiveness of our proposed camera handoff procedure including trackability measure and adaptive resource management at sampled frames fn

and fn+30 in real-time tracking sequence 3.

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 861

Fig. 14 illustrates sampled frames from fn to fn+30 from real-timetracking sequence 3 with three static perspective cameras. In this

sequence, since objects 1 and 2 are carrying valuable materials, re-duced frame rates is not allowed for the sake of security. Thus, in

Table 2The illustration of MO,ij, MD,ij, MS,ij, and PO;ij0 shown in Fig. 14.

Object 1 (i = 1) Object 2 (i = 2) Object 3 (i = 3)

fn fn+10 fn+20 fn+30 fn fn+10 fn+20 fn+30 fn fn+10 fn+20 fn+30

Camera 7 (i = 7) MO,ij 0.31 0 � � 0.31 0 � 0.7 � � � �MD,ij 0.16 0.85 � � 0.35 0.84 � 0.39 � � � �MS,ij 0.5 0.49 � � 0.48 0.48 � 0.8 � � � �PO,ij’ 0.6 0.6 0.1 0.3 0.6 0.6 0.1 0.3 0.6 0.6 0.1 0.3

Camera 1 (i = 1) MO,ij � 0.23 0.25 0.3 � 0.23 0.28 0.29 � � � �MD,ij � 0.3 0.32 0.34 � 0.2 0.15 0.1 � � � �MS,ij � 0.7 0.7 0.71 � 0.7 0.65 0.6 � � � �PO,ij’ 0.1 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6 0.6

Camera 6 (i = 6) MO.ij � � � � � � � � 0.99 0.89 0.69 0.59MDi,j � � � � � � � � 0.8 0.8 0.8 0.8MS,ij � � � � � � � � 0.75 0.75 0.75 0.75PO,ij’ 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3 0.3

862 C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864

this experiment, objects 1 and 2 represent the high priority rank.Object 3 represents the low priority rank. The cameras’ positionsare specified in Fig. 6 as cameras 1, 6, and 7. Table 2 lists MS,ij, MD,ij,MO,ij, and PO,ij0 for the ith object observed by the jth camera atframes from fn to fn+30, where i ranges from 1 to 3 and j is either1, 6 or 7. In frame fn, objects 1 and 2 are tracked by camera 7. Ob-ject 3 is tracked by camera 6. In frame fn+10, object 2 is occluded byobject 1 in camera 7. However, our trackability measure has trig-gered the camera handoff procedure before the occlusion happens.Even though object 1 can be seen by cameras 1 and 6 and repre-sents similar MS,ij, MD,ij, and MO,ij in both cameras, camera 1 hasthe lowest computational load as compared with camera 6(PO;110 ¼ 0:1 and PO;160 ¼ 0:3). Thus, object 1 is transferred to camera

Fig. 15. Illustration of the effectiveness of our proposed camera handoff procedure inclf1147, f1225, f1292, f1348, and f1414 in PETS’ 2006 dataset S7.

1. In frame fn+20, object 2 is under camera handoff procedure sinceit is moving out of FOV of camera 1 (MD,21 = 0.15). In frame fn+30,object 2 had been successfully handed over to camera 7. In general,we can see that the newly defined trackability measure gives aquantified metric to direct the camera handoff successfully andsmoothly before the tracked object is occluded or falls out of FOVof currently observing camera. Also, our adaptive resource man-agement is able to effectively guide camera handoff to choosethe camera with the least system load. This can reduce the proba-bility of missing critical events and improve the system’s level ofthreat awareness. The maintained frame rate also stabilizes theperformance of consistent labeling and leads to an improved hand-off success rate.

uding trackability measure and adaptive resource management at sampled frames

C.-H. Chen et al. / Image and Vision Computing 28 (2010) 851–864 863

5.4. Experiment on PETS’ video sequence

Fig. 15 illustrates sampled frames at f1147, f1225, f1292, f1348, andf1414 from PETS’ 2006 dataset S7 where it contains a single personwith a suitcase who loiters before leaving the item of luggage unat-tended and four cameras are monitoring the scene. During thisevent other people move in close proximity to the item of luggage.Two priority levels are assigned to the objects, Npr = 2. The maxi-mum number of objects that can be tracked simultaneously is alsothree for all cameras, Nmax = 3. The thresholds TO, TD, and TS are 0.2to comply with the time needed for executing camera handoff (5 saverage) and the maximal moving speed of the objects (0.6 m/s). Inthis sequence, since object 1 is leaving his luggage unattended inthe scene, which may post a threat to the area, reduced frame ratesare not allowed. To illustrate the effectiveness of our proposedhandoff algorithm, we focus on object 1. In the beginning, object1 is tracked by camera first. In frame f1292, because object 4 is goingto occlude object 1 (MO,1A = 0.18), handoff request from camera A issent out to adjacent cameras B, C, and D. Since camera C has thelowest system load (PO;1B0 ¼ 0:4; PO;1C0 ¼ 0:1 and PO;1D0 ¼ 0:3), theresolution of object 1 in camera B is too low (MS,1B = 0.13), and ob-ject 1 has similar MS,ij, MD,ij, and MO,ij in both cameras C and D, ob-ject 1 is transferred to camera C. In general, we can see that ourdefined trackability measure gives a quantified metric to directthe camera handoff successfully and smoothly before the trackedobject is occluded by other objects. Also, our adaptive resourcemanagement is able to effectively guide camera handoff to choosethe camera with the least system load. This can reduce the proba-bility of missing critical events and improve the system’s level ofthreat awareness.

6. Conclusion

Most existing camera handoff algorithms leave two crucial un-solved problems: (I) no quantitative measure is given to guide thetransitions between adjacent cameras and (II) it is difficult tomaintain a constant frame rate given limited resources. Thesetwo problems lead to a deteriorated performance of consistentlabeling and possible observation leaks. As a result, the surveil-lance system is unable to continuously track the object of interestand immediately detect threatening events in the monitored area.In this paper, we first defined a trackability measure based on res-olution, distance to the edge of the camera’s FOV, and occlusion toquantitatively evaluate the effectiveness of object tracking. Thetrackability measure is used to determine when to trigger a hand-off request and to select the optimal camera to which the object ofinterest is transferred. We also developed an adaptive resourcemanagement algorithm based on system’s current load to adap-tively allocate the resources among multiple objects with differentprivileges. Experimental results illustrated that our handoff algo-rithm outperforms Khan and Shah’s method by keeping a higheroverall tracking rate and a more stable frame rate. This improvesthe reliability of the tracking system for continuously trackingmultiple objects across multiple cameras.

Acknowledgment

This work was supported in part by the University ResearchProgram in Robotics under Grant DOE-DE-FG52-2004NA25589.

References

[1] M. Balcells, D. DeMenthon, D. Doermann, An appearance-based approach forconsistent labeling of humans and objects in video, Pattern and Application(2005) 373–385.

[2] D.G. Lowe, Distinctive image features from scale-invariant keypoints,International Journal of Computer Vision 60 (2) (2004) 91–110.

[3] P. Kelly, A. Katkere, D. Kuramura, S. Moezzi, S. Chatterjee, R. Jain, Anarchitecture for multiple perspective interactive video, in: Proceedings ofACM Multimedia 95, May 1995.

[4] J. Black, T. Ellis, Multiple camera image tracking, in: Proc. PerformanceEvaluation of Tracking and Surveillance Conference (PETS 2001), with CVPR2001, December 2001.

[5] S. Khan, M. Shah, Consistent labeling of tracked objects in multiple cameraswith overlapping fields of view, IEEE Transactions on PAMI 25 (10) (2003)1355–1361.

[6] F. Fluuret, J. Berclaz, R. Lengagne, P. Fua, Multicamera people tracking with aprobabilistic occupancy map, IEEE Transactions on PAMI 30 (2) (2008) 267–273.

[7] L. Lee, R. Romano, G. Stein, Monitoring activities from multiple video streams:establishing a common coordinate frame, IEEE Transactions on PAMI 22 (8)(2000) 758–767.

[8] S. Calderara, A. Prati, R. Vezzani, R. Cucchiara, Consistent labeling for multi-camera object tracking, in: 13th International Conference on Image Analysisand Processing, September 2005.

[9] J. Kang, I. Cohen, G. Medioni, Continuous tracking within and across camerastreams, in: IEEE International Conference on Computer Vision and PatternRecognition, June 2003.

[10] G.C. De Silva, T. Yamasaki, T. Ishikawa, K. Aizawa, Video handover for retrievalin a ubiquitous environment using floor sensor data, in: IEEE InternationalConference on Multimedia and Expo, July 2005.

[11] C. Beleznai, B. Fruhstuck, H. Bischof, Multiple object tracking using local PCA,in: 18th International Conference on Pattern Recognition, June 2006.

[12] X. Luo, S.M. Bhandarkar, Multiple object tracking using elastic matching, in:IEEE Conference on Advanced Video and Signal based Surveillance, September2005.

[13] Y. Yao, B. Abidi, M. Abidi, Fusion of omnidirectional and PTZ cameras foraccurate cooperative tracking, in: IEEE International Conference on AdvancedVideo and Signal Based Surveillance, Sydney, Australia, November 2006.

[14] M. Han, W. Xu, H. Tao, Y. Gong, An algorithm for multiple object trajectorytracking, in: IEEE Conference on Computer Vision and Pattern Recognition,2004.

[15] I.O. Sebe, S. You, U. Neumann, Globally optimum multiple object tracking, in:SPIE Defense and Security Symposium, 2005.

[16] W. Hu, T. Tan, L. Wang, S. Maybank, A survey on visual surveillance of objectmotion and behaviors, IEEE Transactions on Systems, Man, and Cybernetics 34(3) (2004) 334–352.

[17] M. Shah, Understanding human behavior from motion imagery, MachineVision and Applications 14 (September) (2003) 210–214.

[18] Y. Yao, B. Abidi, M. Abidi, 3D target scale estimation for size preserving in PTZvideo tracking, in: IEEE International Conference on Image Processing, Atlanta,GA, Oct. 2006.

[19] L. Klennrock, Queuing Systems, Theory, vol. 1, Wiley, New York, 1975.[20] L. Huang, S. Kumar, C.-C. Jay Kuo, Adaptive resource allocation for multimedia

Qos management in wireless networks, IEEE Transactions on VehicularTechnology 53 (2) (2004) 547–558.

[21] Y. Nagai, T. Kobayashi, Statistical characteristics of pedestrians’ motion andeffects on teletraffic of mobile communication networks, in: Second IFIPInternational Conference on Wireless and Optical Communications Networks,March 2005.

[22] O. Javed, Z. Rasheed, K. Shafique, M. Shah, Tracking across multiple cameraswith disjoint views, in: IEEE International Conference on Computer Vision,October 2003.

[23] S. Guler, J.M. Griffith, I.A. Pushee, Tracking and handoff between multipleperspective camera views, in: IEEE Proceedings of the 32nd Applied ImageryPattern Recognition Workshop (AIPR 03), USA, October 2003.

[24] F.L. Lim, W. Leoputra, T. Tan, Non-overlapping distributed tracking systemutilizing particle filter, The Journal of VLSI Signal Processing 49 (3) (2007) 343–362.

[25] O. Javed, K. Shafique, M. Shas, Appearance modeling for tracking in multiplenon-overlapping cameras, in: IEEE Conference on Computer Vision and PatternRecognition, June 2005.

[26] J. Kang, I. Cohen, G. Medioni, Persistent objects tracking across multiple nonoverlapping cameras, in: IEEE Workshop on Motion and Video Computing,2005.

[27] D. Hoiem, A.N. Stein, A.A. Efros, M. Hebert, Recovering occlusion boundariesfrom a single image, in: IEEE International Conference on Computer Vision,October 2007.

[28] K.-C. Lien, C.-L. Huang, Multi-view-based cooperative tracking of multiplehuman objects in clustered scenes, in: 18th International Conference onPattern Recognition, June 2006.

[29] PETS: Performance Evaluation of Tracking and Surveillance. <http://www.cvg.rdg.ac.uk/slides/pets.html>.

[30] Y. Yao, C.-H. Chen, B. Abidi, D. Page, A. Koschan, M. Abidi, Sensor planning forautomated and persistent object tracking with multiple cameras, in:Proceedings IEEE Conference on Computer Vision and Pattern Recognition,June 2008.

[31] S. Calderara, R. Cucchiara, A. Prati, Bayesian-competitive consistentlabeling for people surveillance, IEEE Transactions on PAMI 30 (2) (2008)354–360.

[32] W. Hu, M. Hu, X. Zhou, T. Tan, J. Lou, S. Maybank, Principal axis-basedcorrespondence between multiple cameras for people tracking, IEEETransactions on PAMI 28 (4) (2006) 663–671.

864 C.-H. Chen et al. / Image and Vision

[33] J.-Y. Choi, J.-W. Choi, Y.-K. Yang, Improved tracking of multiple vehicles usinginvariant feature-based matching, Pattern Recognition and MachineIntelligence 4815 (2007) 649–656.

[34] C.-H. Chen, Y. Yao, D. Page, B. Abidi, A. Koschan, M. Abidi, Video-based Multi-Camera Automated Surveillance of High Value Assets in Nuclear Facilities,Transactions of the American Nuclear Society, Washington, DC, November 2007.

[35] R. Panneerselvam, Research Methodology, Prentice Hall, 2004.[36] K.S. Tivedi, Probability and Statistics with Reliability, Queuing, and Computer

Science Applications, first ed., Viley-Interscience, 2001.

Chung-Hao Chen received his B.S. and M.S. both inComputer Science and Information Engineering fromFu-Jen University, Taiwan 1997 and 2001, respectively.He received his Ph.D. in the department of ElectricalEngineering and Computer Science at the University ofTennessee, Knoxville in 2009. His research interestsinclude object tracking, robotics and image processing.He is currently an assistant professor in the departmentof Mathematics and Computer Science at North CarolinaCentral University.

Yi Yao received her B.S. and M.S. both in Electrical

Engineering from Nanjing University of Aeronautics andAstronautics, China in 1996 and 2000, respectively. Shereceived her Ph.D. in the department of ElectricalEngineering and Computer Science at the University ofTennessee, Knoxville in 2008. Her research interestsinclude object tracking and multi-camera surveillancesystems. She is currently with the Global ResearchCenter, General Electric.

David Page received the B.S. and M.S. degrees in elec-

trical engineering from Tennessee Technological Uni-versity, Cookeville, in 1993 and 1995, respectively, andthe Ph.D. degree in electrical engineering from theUniversity of Tennessee (UT), Knoxville, in 2003. Aftergraduation, he was a Civilian Research Engineer withthe Naval Surface Warfare Center, Dahlgren, VA. From2003 to 2008, he was a Research Assistant Professorwith the Imaging, Robotics, and Intelligent SystemsLaboratory, Department of Electrical and ComputerEngineering, UT. He is currently a partner with ThirdDimension Technologies LLC, a Knoxville-based startup.

His research interests include 3-D scanning and modeling for computer visionapplications, robotic vision systems, and 3-D shape analysis for object description.

Besma Abidi is a Research Assistant Professor with theDepartment of Electrical and Computer Engineering at

the University of Tennessee, Knoxville, which she joinedin 1998. She was a research scientist at the Oak RidgeNational Laboratory from 1998 until 2001. From 1985 to1988 she was Assistant Professor at the National Engi-neering School of Tunis, Tunisia. Dr. Abidi obtained twoM.S. in 1985 and 1986 in image processing and RemoteSensing with honors from the National EngineeringSchool of Tunis. She received her Ph.D. from the Uni-versity of Tennessee in 1995. Her general areas ofresearch are in senor positioning and geometry, video

tracking, sensor fusion, nano-vision, and biometrics. She is a senior member of IEEE,member of SPIE, Tau Beta Pi, Eta Kappa Nu, Phi Kappa Phi, and The Order ofEngineer.

Computing 28 (2010) 851–864

Andreas Koschan received his Diploma (M.S.) in Com-puter Science and his Dr.-Ing. (Ph.D.) in ComputerEngineering from the Technical University Berlin, Ger-many in 1985 and 1991, respectively. Currently he is aResearch Associate Professor in the Department ofElectrical and Computer Engineering at the University ofTennessee, Knoxville. His work focused on color imageprocessing and 3D computer vision including stereovision and laser range finding techniques. He is acoauthor of two textbooks on 3D image processing andhe is a member of IS&T and IEEE.

Mongi Abidi, Professor and Associate Department Head

in the Department of Electrical and Computer Engi-neering, directs activities in the Imaging, Robotics, andIntelligent Systems Laboratory. He received his Ph.D. inElectrical Engineering at The University of Tennessee in1987, M.S. in Electrical Engineering at The University ofTennessee in 1985, and Principal Engineer in ElectricalEngineering at the National Engineering School of Tunis,Tunisia in 1981. Dr. Abidi conducts research in the fieldof 3D imaging, specifically in the areas of scene building,scene description, and data visualization.