1

Autonomous robotic cameras

for collaborative target localization

(… and a privacy filter)

Andrea Cavallaro

@smartcamerassmartcameras

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

outline

2

From traditional camera views …

pedestrian tracking on MOT16 challenge dataset

X

Y

… to a mini-drone view

motion patterns

Trajectory clustering for motion pattern extraction in aerial videos

T. Nawaz, A. Cavallaro, B. Rinner

Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014

3

Extraction of motion patterns

Multi-feature object trajectory clustering for video analysis [source code]

N. Anjum, A. Cavallaro

IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008

Trajectory clustering for motion pattern extraction in aerial videos

T. Nawaz, A. Cavallaro, B. Rinner

Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014

Compensated trajectories

input trajectoriescompensated trajectories overlaid

on the scene mosaic

4

Motion pattern extraction

Mean-shift clustering with incremental bandwidth selection

clusters motion patterns

Multi-feature object trajectory clustering for video analysis [source code]

N. Anjum, A. Cavallaro

IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

5

From single to multiple hovering cameras

The problem

Continuous estimation of the state of an object (target)

given a set of measurements (observations)

obtained from spatially distributed sensing nodes (cameras)

Measurements

State estimation

)zzz( Z N

k

2

k

1

kk

), x, Zf(Zx :k-:kkk 1011

1

kz

2

kz

3

kz

4

kz

6

Top view

Camera allocation for target tracking

• Constraints

– limited resources

– wireless communication

• packet errors

• packet losses

• Distributed tracking via camera grouping (coalitions)

– beneficial to distributed fusion

• quicker convergence

• greater tracking accuracy

– communication efficient

• comparable cost to decentralized fusion

• cheaper than distributed fusion without coalitions

7

Proposed coalition framework

Consensus protocols for distributed tracking in wireless camera networks

S. Katragadda, J. SanMiguel, A. Cavallaro

In Proc. of Information Fusion (FUSION), Salamanca, July 7-10, 2014

Coalition

candidate

announcement

Communication-

aware

coalition

formation

Distributed

fusion

(EIWCF)

[Katragadda2014]

Bridge

identification

Three-stage coalition formation

Stage 1 Stage 2 Stage 3

EIWCF: Extended Information

Weighted Consensus Filter

Coalition formation for distributed multi-target tracking using information consensus

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015

Coalition candidate announcement

• After target detection

OR

Camera to join multiple coalitions

Announcement to neighbours

c1

c6 c7

c2

c5

c4

c3

measurement

prior estimate

Coalition formation for distributed multi-target tracking using information consensus

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015

8

Communication-aware coalition formation

• For each candidate coalition

– information contribution to neighbours

• joining coalition

– contribute information for tracking

– traffic packet reception ratio

• not joining coalition

– not contributing information for tracking

– traffic packet reception ratio

– join a coalition if , where

U : utility

i : camera index

j : target index

- : excluding

Coalition formation for distributed multi-target tracking using information consensus

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015

Bridge identification

• Distributed fusion requires a connected network

Bridge camera

Broadcast notification if ‘lonely’

Camera not in the coalition

c1

c6 c7

c2

c5

c4

c3

Coalition formation for distributed multi-target tracking using information consensus

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015

9

Coalitions for target tracking

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

10

Tracking on demand: overview

• Hovering (static) cameras

– detect targets and form groups to fuse information

– send help requests to mobile cameras

• Mobile cameras

– select the target to track

– move to track the target jointly with hovering (static) cameras

static camera

mobile camera

Objective

• Collaboration with ‘on-call’ mobile cameras for

– prioritized target observation

– tracking accuracy

– energy efficiency

tracking

accuracy

prioritized

target

observation

energy

efficiency

Optimal controller for active tracking

Local robot-target assignment

Prioritized target tracking with active collaborative cameras

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016

11

Challenges

• Limited on-board energy

– maintain tracking performance with smaller energy cost

• Limited communication range

– local decision making

Local target selection

request to

mobile cameras

after receiving request each robot selects a target to track

Prioritized target tracking with active collaborative cameras

Y. Wang, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016

12

Motion planning

• Model Predictive Control (MPC) [Findeisen2002]

– 𝑇ℎ: time horizon

– compute next robot state using the predicted state at 𝑡 + 𝑇ℎ

– minimize a weighted sum of cost functions: 𝐽1 and 𝐽2

𝐽1 aims to center target in FoV 𝐽2 aims to reduce energy [Liu2014]

cc

3 targets & 2 mobile cameras

Switching to track target

with higher priority 𝑜𝑗

𝑤𝑗: target 𝑜𝑗 with priority 𝑤𝑗

𝑐𝑖 receives a request to track target 𝑜𝑗

𝑜𝑗 𝑐𝑖𝑚𝑖𝑅

13

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

Team of flying cameras

1

4

3

2 d1,2

d1,4

y

x

z

xt(k)

Rt(k)

e1

e2

e3

b1,1

b2,1

b3,1

x1(k)

R1(k)

Objective: to achieve formation on target for monitoring and filming

14

Views of the target

Examples of flying camera views

target

A simple force-based model

goal force

interaction

forces

GPS sensing

positions of

the neighbors

position of the target

quadrotor

dynamics

Self-positioning of a team of flying smart cameras

F. Poiesi, A. Cavallaro

in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015

15

Self-positioning of a team of flying smart cameras

F. Poiesi, A. Cavallaro

in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015

How to follow a target without GPS?

velocities predictions

vision-based

inference

(position mapping)

video

analyticsdistributed

consensus

formation

maintenance

quadrotor

dynamics

• Noisy target detections could generate incorrect control

• Target could exit the fields of view

Distributed vision-based flying cameras to film a moving target

F. Poiesi, A. Cavallaro

in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015

16

• Video 2

Distributed vision-based flying cameras to film a moving target

F. Poiesi, A. Cavallaro

in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

17

The (physical) risks for mini-drones …

Detection of fast incoming objects with a moving camera

F. Poiesi, A. Cavallaro

Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016

Safe-point estimation for collision avoidance

Detection of fast incoming objects with a moving camera

F. Poiesi, A. Cavallaro

Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016

18

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

Location-based privacy: geo-fencing

https://www.noflyzone.org/

19

Where are privacy-related data?

• Direct

– face

– body

– licence plate

• Indirect

– location

– time

– activity

Motivation

• How to protect privacy of individuals captured by

airborne cameras?

• Objective:

– to maintain high fidelity of the visual data

• Solution

– to define & explore the privacy design space

– to configure an adaptive privacy filter

20

Adaptive privacy protection

.

nadir direction camera axis

Design space exploration for adaptive privacy protection in airborne images

O. Sarwar, B. Rinner, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016

Adaptive blurring: example

Original OptimalOverFixed Under

Design space exploration for adaptive privacy protection in airborne images

O. Sarwar, B. Rinner, A. Cavallaro

Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016

21

mini-drone view

coalitions for tracking

tracking on demand

collective target

following

moving obstacle

avoidance

privacy preservation

summary

a.cavallaro@qmul.ac.uk

www.eecs.qmul.ac.uk/~andrea

@smartcameras

smartcameras

Autonomous robotic cameras

for collaborative target localization

Acknowledgments

R. Sánchez Matilla, F. Poiesi, T. Nawaz, Y. Wang, B. Rinner, O. Sarwar

EU Artemis JU project COPCAMS

Cognitive & perceptive cameras

EU Erasmus Mundus Joint Doctorate ICE

Interactive & Cognitive Environments

22

References: single camera

Detection of fast incoming objects with a moving camera

F. Poiesi, A. Cavallaro

in Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016

Design space exploration for adaptive privacy protection in airborne images

O. Sarwar, B. Rinner, A. Cavallaro

in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, Aug 23-26, 2016

Trajectory clustering for motion pattern extraction in aerial videos

T. Nawaz, A. Cavallaro, B. Rinner

in Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014

References: teams of cameras

Coalition formation for distributed multi-target tracking using information consensus

Y. Wang, A. Cavallaro

in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Karlsruhe, August 25-28, 2015

Prioritized target tracking with active collaborative cameras

Y. Wang, A. Cavallaro

in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016

Self-positioning of a team of flying smart cameras

F. Poiesi, A. Cavallaro

in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015

Distributed vision-based flying cameras to film a moving target

F. Poiesi, A. Cavallaro

in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015