multi-sensor fusion for target tracking · multi-sensor fusion for target tracking ... •critical...

This project has received funding from the European Union’s Horizon 2020 research

and innovation programme under grant agreement No 731667 (MULTIDRONE)

Multi-sensor Fusion for Target Tracking

J. Ramiro Martínez-de Dios

[email protected]

Robotics, Computer Vision Group

University of Seville, Spain



• Localization and tracking in GNSS-denied environments are open R&D topics

• Critical for many applications in cinematography

Issues:

• Unstructured, complex and dynamic environments: sensor fusion

• Real-time problem: efficient use of resources

• On-board processing

• Robustness: decentralization

• Changing conditions: responsiveness, dynamic adaptation

Motivation and introduction

2



• Introduction

• The multi-sensor tracking problem

• Tools for multi-sensor tracking

• An example: fusion of camera and RSSI measurements

• Active tracking

• Conclusions

Outline

3



• Estimate target location and velocities

• General case: 6D (position and orientation) in location and velocities

• The tracking problems has been largely analyzed in the literature

• Data association is still not a solved problem: problem-based solutions

• This presentation focusses on multi-sensor

fusion for tracking

The Multi-sensor Tracking Problem

4



• One only sensor cannot capture tracking information in all conditions

• Unexpected issues event:

• Loss of GNSS signal

• Occlusions

• Shadows and lighting conditions

• Scenario changes

• Sensor or drone failure

• Communication failures

• Scenario with many uncertainty sources

Multi-sensor Fusion in Complex Scenarios

5




Accurate localization in GNSS-denied environments

Sensors: ◦ Velodyne HDL-32E

◦ ZED stereo camera

◦ ToF range sensors

◦ UWB ToF sensors

◦ IMU

◦ Laser altimeter

◦ RTK D-GPS

Processing:

◦ Intel NUC

◦ Jetson TX2

6




Pros Cons

Camera Accurate with short distances with the objects

Sensitive to lighting conditions

2D/3D LIDAR Accurate at moderate distances Requires feature-rich scenarios

Range-sensors (radio) - Naturally solves data association problem - Available at large distances

- Requires deploying sensors in the environment

- Radio interference with metallic objects

7



RTK GPS VS Multi-sensor Localization

8



J. R. Martínez-de Dios Email: [email protected]

Multi-sensor Localization

9



• USE, CATEC, AIRBUS D&S

• One of the two Finalists of EUROC-Challenge 3,

out of more than 35 teams.

• Award “Best Dron-based Solution”, EU Parliament, January 2017

• Drones as co-workers in factories

• Drone indoor auton. navigation

• Massive use of sensor-fusion

techniques

• High robustness

ARCOW: Aerial Robot CO-Worker

10




11




12



• Introduction




• Active tracking

• Conclusions

Outline

13



Recursive Bayesian Filters

• Strong mathematical foundation

• Explicit consideration of uncertainty in models and sensors Good performance in presence of noise

in sensors and models

• Very high flexibility: higher than traditional data fusion methods

• Allows modeling realistic systems: observations and systems under uncertainty

• Flexible approach: can be combined with other modules such as dynamic model-learning or uncertainty-

based supervisors

• It enables reasoning in terms of INFORMATION enabling combination with Information-based methods

& tools; e.g. POMDPs

Tools for Multi-sensor Tracking

14



• Optimal performance in presence of noise in sensors and models

• Very high flexibility: higher than traditional data fusion methods

• Allows modeling realistic systems: observations and systems under uncertainty

Probabilistic Bayesian Filters: RBFs

Actions

Disturbances

Estimator

Estimation

State

System Sensors

Noise

Sensor model

System model

Observations

Update

Prediction

measurements

xk+1|k

xk|k

zk

15



Kalman Filter:

• Use parametric models for the system and observations:

(mean vector) (covariance matrix)

• Assumes Gaussian noise (observation and model)

• Assumes Linear models

• Extension to non-linear models: Extended Kalman Filter (EKF)

Information Filter:

• Dual to the KF. Uses the canonical representation:

• Uses same assumptions as KF

• KF and IF have similar burden complexity

• KF are efficient in the Prediction Step

• IF are efficient in the Update Step. They can be decentralized

Scales well with the number of measurements


16



Particle Filter:

• Non-parametric representation: cloud of particles that represent the p.d.f. of the vector state

• Pros: no constrained in noise or system representations

• Pros: allows multi-hypothesis cases

• Cons: High computational burden (>100 particles)


17



Information Filters for Multi-sensor Tracking

Kalman Filter Information Filter Particle Filter

Decentralization Complex Natural Complex

Computational efficiency + +++

Flexibility, adaptability + +++

Scalability + +++

Numerical Stability +++

18



Decentralized Information Filter for Multi-sensor Tracking

non-head head

Notation: “Probabilistic Robotics”, Thrun et al.

19



• Distributed Extended Information Filter (EIF):

• State: object location and velocities

• Measurements: center on image plane (pin-hole nonlinear model)

• Prediction model:

Object follows a locally rectilinear trajectory

Decentralized Information Filter for Multi-sensor Tracking

A de San Bernabe, J.R. Martinez-de Dios, A Ollero, “Efficient cluster-based tracking mechanisms for camera-based wireless sensor networks”, IEEE Transactions on Mobile Computing 14 (9), 1820-1832

20



Drones perceiving the same object organize autonomously in a cluster

Advantages of cluster-based tracking: Scalable and robust

• Local processing of information: avoids transmission of normally

heavy traffic

• Tracking of several objects simultaneously, each with its cluster

The cluster head (CH) is responsible for:

• Collecting and fusing measurements from all the cluster nodes

• Managing inclusion/exclusion from the cluster

• Managing changing/rotation of cluster heads

Other Tracking Functionalities

21



• Introduction




• Active tracking

• Conclusions

Outline

22



Mechanisms:

A distributed EIF fuse the measurements

An entropy-based active sensing method for inclusion and exclusion of camera nodes in the cluster

A method that calibrates RSSI using camera measurements

Fusion of Cameras and RSSI for Tracking

A de San Bernabé, JR Martinez-de Dios, A Ollero, “Efficient integration of RSSI for tracking using Wireless Camera Networks” Information Fusion 36, 296-312

23



• Integrates available RSSI and camera measurements

Uses RSSI-range models

Fusion of Cameras and RSSI for Tracking

24



Motivation: In static scenarios RSSI has low variability

RSSI-range Training using Camera Measurements

Approach: use estimations of the object location to train in real-time RSSI-range models

The computed RSSI-model will be valid locally around the current target location

25



Estimate the model: linear for simplicity (it is only local)

Estimate the uncertainty of the trained model:

𝑅𝑆𝑆𝐼𝑖,𝑡 = 𝑎𝑑𝑖,𝑡 + 𝑏 𝑎𝑖 = 𝑅𝑆𝑆𝐼𝑖,𝑡𝑑𝑖,𝑡 − 𝑅𝑆𝑆𝐼𝑖 𝑑𝑖,𝑡𝑡𝑡

𝑑𝑖,𝑡2− 𝑑 𝑖 𝑑𝑖,𝑡𝑡𝑡

,

𝑏𝑖 = 𝑅𝑆𝑆𝐼𝑖 − 𝑎𝑑 𝑖 ,

RSSI-range Training using Camera Measurements

26



Experiments: Integrated Testbed

27



• Experimental results:

• Settings:

• 31 camera nodes

• One Pioneer 3-AT robot

Experiments: Integrated Testbed

3 Methods

1. Decentralized EIF with no mechanisms

2. 1 with node activation/deactivation

3. 2 with RSSI-range training

Proposed scheme with sensor activation/deactivation

28



• Introduction




• Active tracking

• Conclusions

Outline

29



• The CH dynamically activates/deactivates nodes balancing the usefulness of the measurements and

their costs

Active Perception for Reducing Tracking Uncertainty

Action 1

Action 2

A

B

30



• Balance of the usefulness of the measurements and their costs

• Greedy approach: at each time selects the action that maximizes the difference between reward and

cost

• Cost: energy spent to gather a new measurement

• Reward: the expected information gain from executing the action. Assuming Gaussian distribution, it can

be computed from the information matrices of the predicted and the prior states:


31




32



• Introduction




• Active tracking

• Conclusions

Outline

33



• In general using only one sensor cannot provide sufficient information for accurate

tracking in complex, dynamic scenarios

• Probabilistic estimation tools are very interesting tools:

• Can explicitly consider uncertainty

• Can naturally fuse measurements

• Can be complemented with supervisors and other modules

• Can be used to reason on information gain enabling active perception schemes

Conclusions

34



Multi-sensor Fusion for Target Tracking

J. Ramiro Martínez-de Dios Univ. de Sevilla

[email protected]

Robotics, Computer Vision Group

University of Seville, Spain

multi-sensor fusion for target tracking · multi-sensor fusion for target tracking ... •critical...

Documents