multi-sensor fusion for target tracking · multi-sensor fusion for target tracking ... •critical...
Post on 12-Jul-2018
252 Views
Preview:
TRANSCRIPT
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Multi-sensor Fusion for Target Tracking
J. Ramiro Martínez-de Dios
jdedios@us.es
Robotics, Computer Vision Group
University of Seville, Spain
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Localization and tracking in GNSS-denied environments are open R&D topics
• Critical for many applications in cinematography
Issues:
• Unstructured, complex and dynamic environments: sensor fusion
• Real-time problem: efficient use of resources
• On-board processing
• Robustness: decentralization
• Changing conditions: responsiveness, dynamic adaptation
Motivation and introduction
2
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Introduction
• The multi-sensor tracking problem
• Tools for multi-sensor tracking
• An example: fusion of camera and RSSI measurements
• Active tracking
• Conclusions
Outline
3
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Estimate target location and velocities
• General case: 6D (position and orientation) in location and velocities
• The tracking problems has been largely analyzed in the literature
• Data association is still not a solved problem: problem-based solutions
• This presentation focusses on multi-sensor
fusion for tracking
The Multi-sensor Tracking Problem
4
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• One only sensor cannot capture tracking information in all conditions
• Unexpected issues event:
• Loss of GNSS signal
• Occlusions
• Shadows and lighting conditions
• Scenario changes
• Sensor or drone failure
• Communication failures
• Scenario with many uncertainty sources
Multi-sensor Fusion in Complex Scenarios
5
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Multi-sensor Fusion in Complex Scenarios
Accurate localization in GNSS-denied environments
Sensors: ◦ Velodyne HDL-32E
◦ ZED stereo camera
◦ ToF range sensors
◦ UWB ToF sensors
◦ IMU
◦ Laser altimeter
◦ RTK D-GPS
Processing:
◦ Intel NUC
◦ Jetson TX2
6
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Multi-sensor Fusion in Complex Scenarios
Pros Cons
Camera Accurate with short distances with the objects
Sensitive to lighting conditions
2D/3D LIDAR Accurate at moderate distances Requires feature-rich scenarios
Range-sensors (radio) - Naturally solves data association problem - Available at large distances
- Requires deploying sensors in the environment
- Radio interference with metallic objects
7
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
RTK GPS VS Multi-sensor Localization
8
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
J. R. Martínez-de Dios Email: jdedios@us.es
Multi-sensor Localization
9
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• USE, CATEC, AIRBUS D&S
• One of the two Finalists of EUROC-Challenge 3,
out of more than 35 teams.
• Award “Best Dron-based Solution”, EU Parliament, January 2017
• Drones as co-workers in factories
• Drone indoor auton. navigation
• Massive use of sensor-fusion
techniques
• High robustness
ARCOW: Aerial Robot CO-Worker
10
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
ARCOW: Aerial Robot CO-Worker
11
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
ARCOW: Aerial Robot CO-Worker
12
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Introduction
• The multi-sensor tracking problem
• Tools for multi-sensor tracking
• An example: fusion of camera and RSSI measurements
• Active tracking
• Conclusions
Outline
13
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Recursive Bayesian Filters
• Strong mathematical foundation
• Explicit consideration of uncertainty in models and sensors Good performance in presence of noise
in sensors and models
• Very high flexibility: higher than traditional data fusion methods
• Allows modeling realistic systems: observations and systems under uncertainty
• Flexible approach: can be combined with other modules such as dynamic model-learning or uncertainty-
based supervisors
• It enables reasoning in terms of INFORMATION enabling combination with Information-based methods
& tools; e.g. POMDPs
Tools for Multi-sensor Tracking
14
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Optimal performance in presence of noise in sensors and models
• Very high flexibility: higher than traditional data fusion methods
• Allows modeling realistic systems: observations and systems under uncertainty
Probabilistic Bayesian Filters: RBFs
Actions
Disturbances
Estimator
Estimation
State
System Sensors
Noise
Sensor model
System model
Observations
Update
Prediction
measurements
xk+1|k
xk|k
zk
15
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Kalman Filter:
• Use parametric models for the system and observations:
(mean vector) (covariance matrix)
• Assumes Gaussian noise (observation and model)
• Assumes Linear models
• Extension to non-linear models: Extended Kalman Filter (EKF)
Information Filter:
• Dual to the KF. Uses the canonical representation:
• Uses same assumptions as KF
• KF and IF have similar burden complexity
• KF are efficient in the Prediction Step
• IF are efficient in the Update Step. They can be decentralized
Scales well with the number of measurements
Probabilistic Bayesian Filters: RBFs
16
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Particle Filter:
• Non-parametric representation: cloud of particles that represent the p.d.f. of the vector state
• Pros: no constrained in noise or system representations
• Pros: allows multi-hypothesis cases
• Cons: High computational burden (>100 particles)
Probabilistic Bayesian Filters: RBFs
17
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Information Filters for Multi-sensor Tracking
Kalman Filter Information Filter Particle Filter
Decentralization Complex Natural Complex
Computational efficiency + +++
Flexibility, adaptability + +++
Scalability + +++
Numerical Stability +++
18
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Decentralized Information Filter for Multi-sensor Tracking
non-head head
Notation: “Probabilistic Robotics”, Thrun et al.
19
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Distributed Extended Information Filter (EIF):
• State: object location and velocities
• Measurements: center on image plane (pin-hole nonlinear model)
• Prediction model:
Object follows a locally rectilinear trajectory
Decentralized Information Filter for Multi-sensor Tracking
A de San Bernabe, J.R. Martinez-de Dios, A Ollero, “Efficient cluster-based tracking mechanisms for camera-based wireless sensor networks”, IEEE Transactions on Mobile Computing 14 (9), 1820-1832
20
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Drones perceiving the same object organize autonomously in a cluster
Advantages of cluster-based tracking: Scalable and robust
• Local processing of information: avoids transmission of normally
heavy traffic
• Tracking of several objects simultaneously, each with its cluster
The cluster head (CH) is responsible for:
• Collecting and fusing measurements from all the cluster nodes
• Managing inclusion/exclusion from the cluster
• Managing changing/rotation of cluster heads
Other Tracking Functionalities
21
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Introduction
• The multi-sensor tracking problem
• Tools for multi-sensor tracking
• An example: fusion of camera and RSSI measurements
• Active tracking
• Conclusions
Outline
22
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Mechanisms:
A distributed EIF fuse the measurements
An entropy-based active sensing method for inclusion and exclusion of camera nodes in the cluster
A method that calibrates RSSI using camera measurements
Fusion of Cameras and RSSI for Tracking
A de San Bernabé, JR Martinez-de Dios, A Ollero, “Efficient integration of RSSI for tracking using Wireless Camera Networks” Information Fusion 36, 296-312
23
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Integrates available RSSI and camera measurements
Uses RSSI-range models
Fusion of Cameras and RSSI for Tracking
24
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Motivation: In static scenarios RSSI has low variability
RSSI-range Training using Camera Measurements
Approach: use estimations of the object location to train in real-time RSSI-range models
The computed RSSI-model will be valid locally around the current target location
25
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Estimate the model: linear for simplicity (it is only local)
Estimate the uncertainty of the trained model:
𝑅𝑆𝑆𝐼𝑖,𝑡 = 𝑎𝑑𝑖,𝑡 + 𝑏 𝑎𝑖 = 𝑅𝑆𝑆𝐼𝑖,𝑡𝑑𝑖,𝑡 − 𝑅𝑆𝑆𝐼𝑖 𝑑𝑖,𝑡𝑡𝑡
𝑑𝑖,𝑡2− 𝑑 𝑖 𝑑𝑖,𝑡𝑡𝑡
,
𝑏𝑖 = 𝑅𝑆𝑆𝐼𝑖 − 𝑎𝑑 𝑖 ,
RSSI-range Training using Camera Measurements
26
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Experiments: Integrated Testbed
27
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Experimental results:
• Settings:
• 31 camera nodes
• One Pioneer 3-AT robot
Experiments: Integrated Testbed
3 Methods
1. Decentralized EIF with no mechanisms
2. 1 with node activation/deactivation
3. 2 with RSSI-range training
Proposed scheme with sensor activation/deactivation
28
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Introduction
• The multi-sensor tracking problem
• Tools for multi-sensor tracking
• An example: fusion of camera and RSSI measurements
• Active tracking
• Conclusions
Outline
29
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• The CH dynamically activates/deactivates nodes balancing the usefulness of the measurements and
their costs
Active Perception for Reducing Tracking Uncertainty
Action 1
Action 2
A
B
30
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Balance of the usefulness of the measurements and their costs
• Greedy approach: at each time selects the action that maximizes the difference between reward and
cost
• Cost: energy spent to gather a new measurement
• Reward: the expected information gain from executing the action. Assuming Gaussian distribution, it can
be computed from the information matrices of the predicted and the prior states:
Active Perception for Reducing Tracking Uncertainty
31
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Active Perception for Reducing Tracking Uncertainty
32
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• Introduction
• The multi-sensor tracking problem
• Tools for multi-sensor tracking
• An example: fusion of camera and RSSI measurements
• Active tracking
• Conclusions
Outline
33
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
• In general using only one sensor cannot provide sufficient information for accurate
tracking in complex, dynamic scenarios
• Probabilistic estimation tools are very interesting tools:
• Can explicitly consider uncertainty
• Can naturally fuse measurements
• Can be complemented with supervisors and other modules
• Can be used to reason on information gain enabling active perception schemes
Conclusions
34
This project has received funding from the European Union’s Horizon 2020 research
and innovation programme under grant agreement No 731667 (MULTIDRONE)
Multi-sensor Fusion for Target Tracking
J. Ramiro Martínez-de Dios Univ. de Sevilla
jdedios@us.es
Robotics, Computer Vision Group
University of Seville, Spain
top related