stochastic geometry - team.inria.fr · surveillance – now more than ever human benefits wildlife...
TRANSCRIPT
Stochastic geometry
for
automatic object detection and tracking
in
remotely sensed image sequences
Paula CR����� https://team.inria.fr/ayin/paula-craciun/
This work has been done in collaboration with dr. Josiane ������ ���������and dr. Mathias Ortner ������������� ����������������������
���!
Paul
a Cr
aciu
n –
���
����
����
�
Surveillance – now more than ever
Human benefits Wildlife benefits
2 "�������#��$%&�� ����%��$���'()* "�+��$��+��������� �����%���
Paul
a Cr
aciu
n –
���
����
����
�
Optical airborne and space-
borne systems • ��,��0��������������$�1�&��$��3
• ���-meter ground sampling resolution imagery • ���%��$���$�% ���
• Low-orbit satellites
• ���-meter ground sampling resolution imagery • �%��$���$�% ��� • High-�� ���%����1������ ����%��4(����������%�5(� ������6�������
• Geostationary satellites
• )7��8����������$��8�����$�%�������8��9 • Low temporal frequency … 5
Paul
a Cr
aciu
n –
���
����
����
�
Challenges
*
• ���$$�object size • Large number of objects • �&���#�
• ����������%��������6�object motion
• Time requirements
Paul
a Cr
aciu
n –
���
����
����
�
Multiple Object Tracking (MOT)
5
• Two sub-problems • Where are the possible targets? - ��%��%����� �targets • Which detection corresponds to each target? - ��$1��%&��
data association problem
• Two data-handling approaches • Sequential – �%���%�1�$9����$9<�� ���������%������$�order • Batch processing – ���$9<��%&����%����1������%�once
• =#�����������$�����$1��8�approaches • Tracking by detection • Track before detect
• Goal: Extract object trajectories throughout a 1����
Paul
a Cr
aciu
n –
���
����
����
�
Data-association based methods
6
NN-app DD-MCMCDA
?@�����'((AB ?D�'((FB [��$����'()5B
MHT
Paul
a Cr
aciu
n –
���
����
����
�
RFS-based methods
7
RFS PHD
?��&$��'((5B ?,�'((KB [,�'((AB
?@���'())B
?,�'()5B ?,�'()*B
?@���'()KB
L-RFS / GLMB
Paul
a Cr
aciu
n –
���
����
����
�
Patterns and stochastic geometry
• Object tracking as a
spatio-temporal marked point process
• How to model and simulate such a spatio-temporal point process?
F
Paul
a Cr
aciu
n –
���
����
����
�
Thesis at a glance
• Marked point process models for object detection and
tracking
• Linear programming for automatic or semi-automatic parameter learning
• Model simulation using �����1�� 1��������� �Q����
4
Paul
a Cr
aciu
n –
���
����
����
�
Overview
• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model
• Parameter learning • Linear programming • Parameter learning as a linear program
• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC
• Results • ����$������������������%�1��
)(
Paul
a Cr
aciu
n –
���
����
����
�
• Center of the ellipse is a point in the point process • Marks:
• Geometric marks: semi-��W����X��������-�������X���������%�%��� • ����%����$����7: label
Marked point process of ellipses
11 �
x
y
Paul
a Cr
aciu
n –
���
����
����
�
� Multiple object tracking problem � �����&��8� ���%&�����%�$�7�$9���� �8���%�����XX %&�%� �%��%&��8�1���
image sequence Y � ��$�%��� � X ��������$�<�%����� �%&��Y�������������8�1����9Z
� =&�����%�$�7�$9���� �8���%�������8�1����9Z
� The process energy is composed of two energy terms:
Marked Point Process for Multiple
Object Tracking
External energy Internal energy
12
0)3
0'3
053
Paul
a Cr
aciu
n –
���
����
����
�
Internal energy
����%��%�1�$���%9�����$ Long smooth trajectories ����1��$�����8���W��%�
)5
Paul
a Cr
aciu
n –
���
����
����
�
External energy
Quality model
• !�W��%��1�������%&���8&�frame differencing
• Contrast distance measure between interior and exterior of ellipse
Statistical model
)*
Paul
a Cr
aciu
n –
���
����
����
�
Overview
• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model
• Parameter learning • Linear programming • Parameter learning as a linear program
• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC
• Results • ����$������������������%�1��
)A
Paul
a Cr
aciu
n –
���
����
����
�
Objective function
• Quality model energy formulation
• !�W��%�1��function
)F
Paul
a Cr
aciu
n –
���
����
����
�
Gathering constraints
• Only the ratio is needs to be computed • We can create inequalities of the form
• � �we &�1��ground truth information
• Or more specifically the constraints can be written as )4
0)3
0'3
053
Paul
a Cr
aciu
n –
���
����
����
�
Overview
• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model
• Parameter learning • Linear programming • Parameter learning as a linear program
• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC
• Results • ����$������������������%�1��
')
Paul
a Cr
aciu
n –
���
����
����
�
Related samplers
Paul
a Cr
aciu
n –
���
����
����
�
22
RJMCMC MBD and MBC
?Y����)44KB ?���������'((4B ?Y���$'())B
?,�����'()'B
P-RJMCMC
Classic RJMCMC
• Why?
• Highly non-���1�X energy MCMC • ��7��#� number of objects RJ 0��1�����$� W���3
• Core idea
• Create a ���7�1�chain • �%���%�1�$9 perturb the current state of the chain • ��%�$ ���1��8�����is reached
'5
Paul
a Cr
aciu
n –
���
����
����
�
Standard perturbation kernels
• ��%&��������%& • ��%&Z
• ��������#���W��%�%��%&����� �8���%��� • ���%&Z
• ���1��������W��%� ����%&����� �8���%���
• Local transformations Rotation Translation ���$�
'*
Paul
a Cr
aciu
n –
���
����
����
�
RJMCMC sampler
25
iX
1�iX
1�iX
2�iX
2�iX
^
…
)�- ^
)�- ^
^
1�i
)
2�i
Paul
a Cr
aciu
n –
���
����
����
�
Adding Kalman-inspired births
���%��$�<� Kalman ��$%��
����%�� Kalman ��$%��
Perturbation accepted
Perturbation rejected
End iteration
Perturbation accepted
26
Paul
a Cr
aciu
n –
���
����
����
�
Did time efficiency increase?
Kalman-inspired births reduce computation times!
Experimental results ��%�$$�%����%�
0*���W��%��6� ����3
RJMCMC with Kalman like moves ���1��8������&� ��%������������%��%&��
standard RJMCMC
27
Paul
a Cr
aciu
n –
���
����
����
�
Parallel implementation of
RJMCMC [Verdie2012]
• Data-driven space
partitioning
• Locally conditional
independent
perturbations
Paul
a Cr
aciu
n –
���
����
����
�
'F Image with boats © Airbus D&S
Parallel implementation of
RJMCMC
• Data-driven space
partitioning
• Locally conditional
independent
perturbations
Paul
a Cr
aciu
n –
���
����
����
�
'4 Probability that objects exist in each
part of the image
Parallel implementation of
RJMCMC
• Data-driven space
partitioning
• Locally conditional
independent
perturbations
Paul
a Cr
aciu
n –
���
����
����
�
5( Color coding of quad-tree leafs
Parallel perturbations [Verdie2012]
• A color is randomly
chosen
• Perturbations are
performed in all
cells of the chosen
color in parallel
Paul
a Cr
aciu
n –
���
����
����
�
5) Color blue is randomly chosen
Our improvement to the
parallel sampler Problem Solution
Paul
a Cr
aciu
n –
���
����
����
�
5' Large boat is split between
two neighboring cells
Take the configurations in
the neighboring cells into
consideration
Did time efficiency increase?
55 Parallel implementation significantly reduces computation times!
RJMCMC RJMCMC + Kalman Parallel RJMCMC without Kalman
Paul
a Cr
aciu
n –
���
����
����
�
Overview
• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model
• Parameter learning • Linear programming • Parameter learning as a linear program
• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC
• Results • ����$������������������%�1��
5*
Paul
a Cr
aciu
n –
���
����
����
�
Data sets
• 2 different data sets:
• ��,�0��������������$�1�&��$�3���%��0@��$����1��$��$����%����%3
• ��%�$$�%����%��0��������� ��������������3 • v�#�%������$� ��{����9�0�� |)-2}<3 • }�8&�%������$� ��{����9�05(}<3
5K
Paul
a Cr
aciu
n –
���
����
����
�
UAV data – low temporal frequency
Original image ?@��7�W'())B Proposed
COLUMBUS LARGE IMAGE FORMAT 0�v��3�'((A���%����% @��1������9Z =&�����������%������8����%��9�%�����U�U��������� https://www.sdms.afrl.af.mil
5A
UAV data – low temporal frequency
5\
Original image ?@��7�W'())B Proposed Paul
a Cr
aciu
n –
���
����
����
�
Satellite data – low temporal frequency
=���7��8�����$%��"������6��D��
5F
�1���8�������%�%����%���Z��)'�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�
Paul
a Cr
aciu
n –
���
����
����
�
Satellite data – high temporal frequency
Tracking results "������6��D��
54
�1���8�������%�%����%���Z��F�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�
Paul
a Cr
aciu
n –
���
����
����
�
Satellite data – high temporal frequency
=���7��8�����$%��"������6��D��
*(
�1���8�������%�%����%���Z��)(-))�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�
Paul
a Cr
aciu
n –
���
����
����
�
Overview
• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model
• Parameter learning • Linear programming • Parameter learning as a linear program
• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC
• Results • ����$������������������%�1��
*)
Paul
a Cr
aciu
n –
���
����
����
�
Conclusions
• Two ��1�$�spatio-temporal marked point process models for the detection and %���7��8�� ���1��8�objects
• ��%���%����������-automatic parameter estimation using linear programming
• ��%�8��%���RJMCMC sampler with Kalman-like ��1�� • Efficient parallel implementation of the RJMCMC sampler • Good results on different types of data
*'
Paul
a Cr
aciu
n –
���
����
����
�
Critical analysis
Advantages
• ��%��%��� of weakly contrasted objects
• Consistent trajectories • Object interactions
modeling • Robustness to noise and
data quality • Good results on different
data sets
Drawbacks
• Real-time processing only in exceptional cases
• ����$���&��������$��8
*5
Paul
a Cr
aciu
n –
���
����
����
�
Perspectives
• ����8��a hierarchical model that integrates both low-$�1�$�
����%����%����%#��������1����$�objects and high-$�1�$�constraints between trajectories
• Multi-marked process to distinguish between 1���������W��%�classes
• Model traffic density instead of ����1����$ trajectories • Optimization ���������&��$����� ��%&��������1���to make
���&�����$�������%�%�1�
**
Paul
a Cr
aciu
n –
���
����
����
�
References
• ?���������'((4B X. �����������U�Minlos�������U�Zhizhina. Object extraction using a stochastic birth-and-���%&��9�������������%�����U�Q��,��55Z5*\5K4��'((4U�
• ?Y���$'())B �U�Gamal Eldin���U������������YU�Charpiat������QU�ZerubiaU��� ��%���$%��$��birthand ��%��$8���%&������8���$�� ������8�%���U�@���U�� ����@����8���'F)5'F)A��'())U
• ?Y����)44KB�@U�Y����U��1�����$��W�������7�1��&�������%�����$�������%�%���������9������model determination. ����%��7���F'0*3Z\))\5'��)44KU�
• ?��&$��'((5B�R. Mahler. Multitarget �9�� ltering 1���rst-��������$%��%��8�%������%�U������=����U�������������������$��%�������9�%�����540*3Z))K'))\F��'((5U�
• ?@���'())B��U�@���U��%��&��%�������$��������%&���� �����$%�-��W��%�%���7��8U�@&��%&���������1����%������������%�=��&��$�8�����������X��'())U�
• ?@���'()KB��U�Papi��U=U�,����U���{��$������U�U�,�U�Y�����$�<���$���$�����$%�-�����$$��approximation of multi-��W��%������%���U������=����U���8��$�@��������8��A50'(3Z�K*F\K*4\��'()KU
• ?@�����'((AB �UYU�U�Perera���U������1�����U�Hoogs��YU����7��9������+U�Hu. Multi-object tracking through simultaneous long occlusions and split-���8�������%����U�@���U�� ��,@��'((AU
• ?��$����'()5B��U���$���� �����U��&�&U�Multiframe many-���9�����%���������������� ���1�&��$��%���7��8����&�8&������%9�#��������������$�1�����U��Q�,��)(*0'3Z)4F')4��'()5U
• ?,�����'()5B�DU�,�����U�Modelisation de scènes urbaines à partir de donnèes aeriennes. @&� %&���������1����%� ��������- ���&�����%���$����'()5U
• ?,�'()*B�U-�U�,���U-=U�,��������U�Phung. Labeled random finite sets and the �9�� ��$%�%��8�%�%���7��8 �$%��U������=����U���8��$�@��������8��A'0'*3ZAKK*AKA\��'()*U
• ?,�'()5B�U-=U�,������U-�U�,�U�v���$����������nite sets and multi-��W��%����W�8�%��������U������=����U���8��$�@��������8��A)0)53Z5*A(5*\K��'()5U
• ?D�'((4B�Q. Yu and G. Medioni. Multiple-target tracking by spatio-temporal Monte Carlo ���7�1�chain ��%���������%���U������TransU�@�����5)0)'3Z')4A'')(��'((4U
*K