![Page 1: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/1.jpg)
1
Autonomous robotic cameras
for collaborative target localization
(… and a privacy filter)
Andrea Cavallaro
@smartcamerassmartcameras
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
outline
![Page 2: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/2.jpg)
2
From traditional camera views …
pedestrian tracking on MOT16 challenge dataset
X
Y
… to a mini-drone view
motion patterns
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
![Page 3: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/3.jpg)
3
Extraction of motion patterns
Multi-feature object trajectory clustering for video analysis [source code]
N. Anjum, A. Cavallaro
IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
Compensated trajectories
input trajectoriescompensated trajectories overlaid
on the scene mosaic
![Page 4: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/4.jpg)
4
Motion pattern extraction
Mean-shift clustering with incremental bandwidth selection
clusters motion patterns
Multi-feature object trajectory clustering for video analysis [source code]
N. Anjum, A. Cavallaro
IEEE Transactions on Circuits and Systems for Video Technology, Vol. 18, Issue 11, November 2008
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
![Page 5: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/5.jpg)
5
From single to multiple hovering cameras
The problem
Continuous estimation of the state of an object (target)
given a set of measurements (observations)
obtained from spatially distributed sensing nodes (cameras)
Measurements
State estimation
)zzz( Z N
k
2
k
1
kk
), x, Zf(Zx :k-:kkk 1011
1
kz
2
kz
3
kz
4
kz
![Page 6: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/6.jpg)
6
Top view
Camera allocation for target tracking
• Constraints
– limited resources
– wireless communication
• packet errors
• packet losses
• Distributed tracking via camera grouping (coalitions)
– beneficial to distributed fusion
• quicker convergence
• greater tracking accuracy
– communication efficient
• comparable cost to decentralized fusion
• cheaper than distributed fusion without coalitions
![Page 7: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/7.jpg)
7
Proposed coalition framework
Consensus protocols for distributed tracking in wireless camera networks
S. Katragadda, J. SanMiguel, A. Cavallaro
In Proc. of Information Fusion (FUSION), Salamanca, July 7-10, 2014
Coalition
candidate
announcement
Communication-
aware
coalition
formation
Distributed
fusion
(EIWCF)
[Katragadda2014]
Bridge
identification
Three-stage coalition formation
Stage 1 Stage 2 Stage 3
EIWCF: Extended Information
Weighted Consensus Filter
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
Coalition candidate announcement
• After target detection
OR
Camera to join multiple coalitions
Announcement to neighbours
c1
c6 c7
c2
c5
c4
c3
measurement
prior estimate
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
![Page 8: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/8.jpg)
8
Communication-aware coalition formation
• For each candidate coalition
– information contribution to neighbours
• joining coalition
– contribute information for tracking
– traffic packet reception ratio
• not joining coalition
– not contributing information for tracking
– traffic packet reception ratio
– join a coalition if , where
U : utility
i : camera index
j : target index
- : excluding
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
Bridge identification
• Distributed fusion requires a connected network
Bridge camera
Broadcast notification if ‘lonely’
Camera not in the coalition
c1
c6 c7
c2
c5
c4
c3
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surveillance (AVSS), Karlsruhe, August 25-28, 2015
![Page 9: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/9.jpg)
9
Coalitions for target tracking
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
![Page 10: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/10.jpg)
10
Tracking on demand: overview
• Hovering (static) cameras
– detect targets and form groups to fuse information
– send help requests to mobile cameras
• Mobile cameras
– select the target to track
– move to track the target jointly with hovering (static) cameras
static camera
mobile camera
Objective
• Collaboration with ‘on-call’ mobile cameras for
– prioritized target observation
– tracking accuracy
– energy efficiency
tracking
accuracy
prioritized
target
observation
energy
efficiency
Optimal controller for active tracking
Local robot-target assignment
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
![Page 11: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/11.jpg)
11
Challenges
• Limited on-board energy
– maintain tracking performance with smaller energy cost
• Limited communication range
– local decision making
Local target selection
request to
mobile cameras
after receiving request each robot selects a target to track
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
![Page 12: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/12.jpg)
12
Motion planning
• Model Predictive Control (MPC) [Findeisen2002]
– 𝑇ℎ: time horizon
– compute next robot state using the predicted state at 𝑡 + 𝑇ℎ
– minimize a weighted sum of cost functions: 𝐽1 and 𝐽2
𝐽1 aims to center target in FoV 𝐽2 aims to reduce energy [Liu2014]
cc
3 targets & 2 mobile cameras
Switching to track target
with higher priority 𝑜𝑗
𝑤𝑗: target 𝑜𝑗 with priority 𝑤𝑗
𝑐𝑖 receives a request to track target 𝑜𝑗
𝑜𝑗 𝑐𝑖𝑚𝑖𝑅
![Page 13: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/13.jpg)
13
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
Team of flying cameras
1
4
3
2 d1,2
d1,4
y
x
z
xt(k)
Rt(k)
e1
e2
e3
b1,1
b2,1
b3,1
x1(k)
R1(k)
Objective: to achieve formation on target for monitoring and filming
![Page 14: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/14.jpg)
14
Views of the target
Examples of flying camera views
target
A simple force-based model
goal force
interaction
forces
GPS sensing
positions of
the neighbors
position of the target
quadrotor
dynamics
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
![Page 15: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/15.jpg)
15
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
How to follow a target without GPS?
velocities predictions
vision-based
inference
(position mapping)
video
analyticsdistributed
consensus
formation
maintenance
quadrotor
dynamics
• Noisy target detections could generate incorrect control
• Target could exit the fields of view
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015
![Page 16: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/16.jpg)
16
• Video 2
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
![Page 17: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/17.jpg)
17
The (physical) risks for mini-drones …
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
Safe-point estimation for collision avoidance
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
![Page 18: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/18.jpg)
18
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
Location-based privacy: geo-fencing
https://www.noflyzone.org/
![Page 19: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/19.jpg)
19
Where are privacy-related data?
• Direct
– face
– body
– licence plate
• Indirect
– location
– time
– activity
Motivation
• How to protect privacy of individuals captured by
airborne cameras?
• Objective:
– to maintain high fidelity of the visual data
• Solution
– to define & explore the privacy design space
– to configure an adaptive privacy filter
![Page 20: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/20.jpg)
20
Adaptive privacy protection
.
nadir direction camera axis
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
Adaptive blurring: example
Original OptimalOverFixed Under
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
![Page 21: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/21.jpg)
21
mini-drone view
coalitions for tracking
tracking on demand
collective target
following
moving obstacle
avoidance
privacy preservation
summary
www.eecs.qmul.ac.uk/~andrea
@smartcameras
smartcameras
Autonomous robotic cameras
for collaborative target localization
Acknowledgments
R. Sánchez Matilla, F. Poiesi, T. Nawaz, Y. Wang, B. Rinner, O. Sarwar
EU Artemis JU project COPCAMS
Cognitive & perceptive cameras
EU Erasmus Mundus Joint Doctorate ICE
Interactive & Cognitive Environments
![Page 22: Autonomous robotic cameras for collaborative target …andrea/dwnld/2016.08.23_Colorado...2016/08/23 · Detection of fast incoming objects with a moving camera F. Poiesi, A. Cavallaro](https://reader035.vdocuments.mx/reader035/viewer/2022071214/6042d44f39ad5e7b311951bc/html5/thumbnails/22.jpg)
22
References: single camera
Detection of fast incoming objects with a moving camera
F. Poiesi, A. Cavallaro
in Proc. of British Machine Vision Conference (BMVC), York, September 19-22, 2016
Design space exploration for adaptive privacy protection in airborne images
O. Sarwar, B. Rinner, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, Aug 23-26, 2016
Trajectory clustering for motion pattern extraction in aerial videos
T. Nawaz, A. Cavallaro, B. Rinner
in Proc. of IEEE Int. Conference on Image Processing (ICIP), Paris, October 27-30, 2014
References: teams of cameras
Coalition formation for distributed multi-target tracking using information consensus
Y. Wang, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Karlsruhe, August 25-28, 2015
Prioritized target tracking with active collaborative cameras
Y. Wang, A. Cavallaro
in Proc. of IEEE Int. Conf. on Adv. Video and Signal based Surv. (AVSS), Colorado Springs, August 23-26, 2016
Self-positioning of a team of flying smart cameras
F. Poiesi, A. Cavallaro
in Proc. Int. Conf. on Intelligent Sensors, Sensor Networks and Inf. Proc. (ISSNIP), Singapore, April 2015
Distributed vision-based flying cameras to film a moving target
F. Poiesi, A. Cavallaro
in Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), Hamburg, Germany, Sept-Oct. 2015