stochastic geometry - team.inria.fr · surveillance – now more than ever human benefits wildlife...

45
Stochastic geometry for automatic object detection and tracking in remotely sensed image sequences Paula CR https://team.inria.fr/ayin/paula-craciun/ This work has been done in collaboration with dr. Josiane and dr. Mathias Ortner Paula Craciun –

Upload: others

Post on 02-Sep-2019

2 views

Category:

Documents


0 download

TRANSCRIPT

Stochastic geometry

for

automatic object detection and tracking

in

remotely sensed image sequences

Paula CR����� https://team.inria.fr/ayin/paula-craciun/

This work has been done in collaboration with dr. Josiane ������ ���������and dr. Mathias Ortner ������������� ����������������������

���!

Paul

a Cr

aciu

n –

���

����

����

Surveillance – now more than ever

Human benefits Wildlife benefits

2 "�������#��$%&�� ����%��$���'()* "�+��$��+��������� �����%���

Paul

a Cr

aciu

n –

���

����

����

Optical airborne and space-

borne systems • ��,��0��������������$�1�&��$��3

• ���-meter ground sampling resolution imagery • ���%��$���$�% ���

• Low-orbit satellites

• ���-meter ground sampling resolution imagery • �%��$���$�% ��� • High-�� ���%����1������ ����%��4(����������%�5(� ������6�������

• Geostationary satellites

• )7��8����������$��8�����$�%�������8��9 • Low temporal frequency … 5

Paul

a Cr

aciu

n –

���

����

����

Challenges

*

• ���$$�object size • Large number of objects • �&���#�

• ����������%��������6�object motion

• Time requirements

Paul

a Cr

aciu

n –

���

����

����

Multiple Object Tracking (MOT)

5

• Two sub-problems • Where are the possible targets? - ��%��%����� �targets • Which detection corresponds to each target? - ��$1��%&��

data association problem

• Two data-handling approaches • Sequential – �%���%�1�$9����$9<�� ���������%������$�order • Batch processing – ���$9<��%&����%����1������%�once

• =#�����������$�����$1��8�approaches • Tracking by detection • Track before detect

• Goal: Extract object trajectories throughout a 1����

Paul

a Cr

aciu

n –

���

����

����

Data-association based methods

6

NN-app DD-MCMCDA

?@�����'((AB ?D�'((FB [��$����'()5B

MHT

Paul

a Cr

aciu

n –

���

����

����

RFS-based methods

7

RFS PHD

?��&$��'((5B ?,�'((KB [,�'((AB

?@���'())B

?,�'()5B ?,�'()*B

?@���'()KB

L-RFS / GLMB

Paul

a Cr

aciu

n –

���

����

����

Patterns and stochastic geometry

• Object tracking as a

spatio-temporal marked point process

• How to model and simulate such a spatio-temporal point process?

F

Paul

a Cr

aciu

n –

���

����

����

Thesis at a glance

• Marked point process models for object detection and

tracking

• Linear programming for automatic or semi-automatic parameter learning

• Model simulation using �����1�� 1��������� �Q����

4

Paul

a Cr

aciu

n –

���

����

����

Overview

• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model

• Parameter learning • Linear programming • Parameter learning as a linear program

• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC

• Results • ����$������������������%�1��

)(

Paul

a Cr

aciu

n –

���

����

����

• Center of the ellipse is a point in the point process • Marks:

• Geometric marks: semi-��W����X��������-�������X���������%�%��� • ����%����$����7: label

Marked point process of ellipses

11 �

x

y

Paul

a Cr

aciu

n –

���

����

����

� Multiple object tracking problem � �����&��8� ���%&�����%�$�7�$9���� �8���%�����XX %&�%� �%��%&��8�1���

image sequence Y � ��$�%��� � X ��������$�<�%����� �%&��Y�������������8�1����9Z

� =&�����%�$�7�$9���� �8���%�������8�1����9Z

� The process energy is composed of two energy terms:

Marked Point Process for Multiple

Object Tracking

External energy Internal energy

12

0)3

0'3

053

Paul

a Cr

aciu

n –

���

����

����

Internal energy

����%��%�1�$���%9�����$ Long smooth trajectories ����1��$�����8���W��%�

)5

Paul

a Cr

aciu

n –

���

����

����

External energy

Quality model

• !�W��%��1�������%&���8&�frame differencing

• Contrast distance measure between interior and exterior of ellipse

Statistical model

)*

Paul

a Cr

aciu

n –

���

����

����

Total energy

Quality model

Statistical model

)K

Paul

a Cr

aciu

n –

���

����

����

Overview

• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model

• Parameter learning • Linear programming • Parameter learning as a linear program

• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC

• Results • ����$������������������%�1��

)A

Paul

a Cr

aciu

n –

���

����

����

Linear programming

)\

0)3

0'3

Paul

a Cr

aciu

n –

���

����

����

Objective function

• Quality model energy formulation

• !�W��%�1��function

)F

Paul

a Cr

aciu

n –

���

����

����

Gathering constraints

• Only the ratio is needs to be computed • We can create inequalities of the form

• � �we &�1��ground truth information

• Or more specifically the constraints can be written as )4

0)3

0'3

053

Paul

a Cr

aciu

n –

���

����

����

How many constraints?

'(

Paul

a Cr

aciu

n –

���

����

����

Overview

• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model

• Parameter learning • Linear programming • Parameter learning as a linear program

• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC

• Results • ����$������������������%�1��

')

Paul

a Cr

aciu

n –

���

����

����

Related samplers

Paul

a Cr

aciu

n –

���

����

����

22

RJMCMC MBD and MBC

?Y����)44KB ?���������'((4B ?Y���$'())B

?,�����'()'B

P-RJMCMC

Classic RJMCMC

• Why?

• Highly non-���1�X energy MCMC • ��7��#� number of objects RJ 0��1�����$� W���3

• Core idea

• Create a ���7�1�chain • �%���%�1�$9 perturb the current state of the chain • ��%�$ ���1��8�����is reached

'5

Paul

a Cr

aciu

n –

���

����

����

Standard perturbation kernels

• ��%&��������%& • ��%&Z

• ��������#���W��%�%��%&����� �8���%��� • ���%&Z

• ���1��������W��%� ����%&����� �8���%���

• Local transformations Rotation Translation ���$�

'*

Paul

a Cr

aciu

n –

���

����

����

RJMCMC sampler

25

iX

1�iX

1�iX

2�iX

2�iX

^

)�- ^

)�- ^

^

1�i

)

2�i

Paul

a Cr

aciu

n –

���

����

����

Adding Kalman-inspired births

���%��$�<� Kalman ��$%��

����%�� Kalman ��$%��

Perturbation accepted

Perturbation rejected

End iteration

Perturbation accepted

26

Paul

a Cr

aciu

n –

���

����

����

Did time efficiency increase?

Kalman-inspired births reduce computation times!

Experimental results ��%�$$�%����%�

0*���W��%��6� ����3

RJMCMC with Kalman like moves ���1��8������&� ��%������������%��%&��

standard RJMCMC

27

Paul

a Cr

aciu

n –

���

����

����

Parallel implementation of

RJMCMC [Verdie2012]

• Data-driven space

partitioning

• Locally conditional

independent

perturbations

Paul

a Cr

aciu

n –

���

����

����

'F Image with boats © Airbus D&S

Parallel implementation of

RJMCMC

• Data-driven space

partitioning

• Locally conditional

independent

perturbations

Paul

a Cr

aciu

n –

���

����

����

'4 Probability that objects exist in each

part of the image

Parallel implementation of

RJMCMC

• Data-driven space

partitioning

• Locally conditional

independent

perturbations

Paul

a Cr

aciu

n –

���

����

����

5( Color coding of quad-tree leafs

Parallel perturbations [Verdie2012]

• A color is randomly

chosen

• Perturbations are

performed in all

cells of the chosen

color in parallel

Paul

a Cr

aciu

n –

���

����

����

5) Color blue is randomly chosen

Our improvement to the

parallel sampler Problem Solution

Paul

a Cr

aciu

n –

���

����

����

5' Large boat is split between

two neighboring cells

Take the configurations in

the neighboring cells into

consideration

Did time efficiency increase?

55 Parallel implementation significantly reduces computation times!

RJMCMC RJMCMC + Kalman Parallel RJMCMC without Kalman

Paul

a Cr

aciu

n –

���

����

����

Overview

• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model

• Parameter learning • Linear programming • Parameter learning as a linear program

• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC

• Results • ����$������������������%�1��

5*

Paul

a Cr

aciu

n –

���

����

����

Data sets

• 2 different data sets:

• ��,�0��������������$�1�&��$�3���%��0@��$����1��$��$����%����%3

• ��%�$$�%����%��0��������� ��������������3 • v�#�%������$� ��{����9�0�� |)-2}<3 • }�8&�%������$� ��{����9�05(}<3

5K

Paul

a Cr

aciu

n –

���

����

����

UAV data – low temporal frequency

Original image ?@��7�W'())B Proposed

COLUMBUS LARGE IMAGE FORMAT 0�v��3�'((A���%����% @��1������9Z =&�����������%������8����%��9�%�����U�U��������� https://www.sdms.afrl.af.mil

5A

UAV data – low temporal frequency

5\

Original image ?@��7�W'())B Proposed Paul

a Cr

aciu

n –

���

����

����

Satellite data – low temporal frequency

=���7��8�����$%��"������6��D��

5F

�1���8�������%�%����%���Z��)'�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�

Paul

a Cr

aciu

n –

���

����

����

Satellite data – high temporal frequency

Tracking results "������6��D��

54

�1���8�������%�%����%���Z��F�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�

Paul

a Cr

aciu

n –

���

����

����

Satellite data – high temporal frequency

=���7��8�����$%��"������6��D��

*(

�1���8�������%�%����%���Z��)(-))�����6� �����������$��%���#�%&�K)'������ ���8����<�Z�)A((�X�4((���X�$�

Paul

a Cr

aciu

n –

���

����

����

Overview

• Models • Model formulation • Quality ����$�1�U��%�%��%���$ model

• Parameter learning • Linear programming • Parameter learning as a linear program

• ����$�%��� • RJMCMC with Kalman inspired ��1�� • Parallel implementation of RJMCMC

• Results • ����$������������������%�1��

*)

Paul

a Cr

aciu

n –

���

����

����

Conclusions

• Two ��1�$�spatio-temporal marked point process models for the detection and %���7��8�� ���1��8�objects

• ��%���%����������-automatic parameter estimation using linear programming

• ��%�8��%���RJMCMC sampler with Kalman-like ��1�� • Efficient parallel implementation of the RJMCMC sampler • Good results on different types of data

*'

Paul

a Cr

aciu

n –

���

����

����

Critical analysis

Advantages

• ��%��%��� of weakly contrasted objects

• Consistent trajectories • Object interactions

modeling • Robustness to noise and

data quality • Good results on different

data sets

Drawbacks

• Real-time processing only in exceptional cases

• ����$���&��������$��8

*5

Paul

a Cr

aciu

n –

���

����

����

Perspectives

• ����8��a hierarchical model that integrates both low-$�1�$�

����%����%����%#��������1����$�objects and high-$�1�$�constraints between trajectories

• Multi-marked process to distinguish between 1���������W��%�classes

• Model traffic density instead of ����1����$ trajectories • Optimization ���������&��$����� ��%&��������1���to make

���&�����$�������%�%�1�

**

Paul

a Cr

aciu

n –

���

����

����

References

• ?���������'((4B X. �����������U�Minlos�������U�Zhizhina. Object extraction using a stochastic birth-and-���%&��9�������������%�����U�Q��,��55Z5*\5K4��'((4U�

• ?Y���$'())B �U�Gamal Eldin���U������������YU�Charpiat������QU�ZerubiaU��� ��%���$%��$��birthand ��%��$8���%&������8���$�� ������8�%���U�@���U�� ����@����8���'F)5'F)A��'())U

• ?Y����)44KB�@U�Y����U��1�����$��W�������7�1��&�������%�����$�������%�%���������9������model determination. ����%��7���F'0*3Z\))\5'��)44KU�

• ?��&$��'((5B�R. Mahler. Multitarget �9�� ltering 1���rst-��������$%��%��8�%������%�U������=����U�������������������$��%�������9�%�����540*3Z))K'))\F��'((5U�

• ?@���'())B��U�@���U��%��&��%�������$��������%&���� �����$%�-��W��%�%���7��8U�@&��%&���������1����%������������%�=��&��$�8�����������X��'())U�

• ?@���'()KB��U�Papi��U=U�,����U���{��$������U�U�,�U�Y�����$�<���$���$�����$%�-�����$$��approximation of multi-��W��%������%���U������=����U���8��$�@��������8��A50'(3Z�K*F\K*4\��'()KU

• ?@�����'((AB �UYU�U�Perera���U������1�����U�Hoogs��YU����7��9������+U�Hu. Multi-object tracking through simultaneous long occlusions and split-���8�������%����U�@���U�� ��,@��'((AU

• ?��$����'()5B��U���$���� �����U��&�&U�Multiframe many-���9�����%���������������� ���1�&��$��%���7��8����&�8&������%9�#��������������$�1�����U��Q�,��)(*0'3Z)4F')4��'()5U

• ?,�����'()5B�DU�,�����U�Modelisation de scènes urbaines à partir de donnèes aeriennes. @&� %&���������1����%� ��������- ���&�����%���$����'()5U

• ?,�'()*B�U-�U�,���U-=U�,��������U�Phung. Labeled random finite sets and the �9�� ��$%�%��8�%�%���7��8 �$%��U������=����U���8��$�@��������8��A'0'*3ZAKK*AKA\��'()*U

• ?,�'()5B�U-=U�,������U-�U�,�U�v���$����������nite sets and multi-��W��%����W�8�%��������U������=����U���8��$�@��������8��A)0)53Z5*A(5*\K��'()5U

• ?D�'((4B�Q. Yu and G. Medioni. Multiple-target tracking by spatio-temporal Monte Carlo ���7�1�chain ��%���������%���U������TransU�@�����5)0)'3Z')4A'')(��'((4U

*K