Double-constrained RPCA based on Saliency Maps for Foreground Detection in Automated Maritime Surveillance

Andrews Sobral, Thierry Bouwmans and El-hadi ZahZahPh.D. Student, Computer Vision

Lab. L3I/MIA – University of La Rochelle, France

Summary

▪ Context: Maritime Surveillance

▪ Understanding an Intelligent Video Surveillance Framework

▪ Introduction to Background Subtraction

▪ Robust Principal Component Analysis (RPCA)

▪ The SCM-RPCA approach

▪ Definition of shape and confidence map

▪ Solving the SCM-RPCA

▪ Experimental results

▪ Conclusions

Context: Maritime Surveillance

http://www.asv.fr/en/mediaImages from:

▪ The automatic video surveillance of maritime environment aims to:

▪ Detect and identify specific ships, vessels or boats on the sea and their activity.

▪ Extend the capabilities to identify maritime risks: piracy, trafficking, immigration, etc.

▪ However, the development of automatic video surveillance applications for maritime environment is a very difficult task due to the complexity of the scenes: moving water, waves, etc.

▪ The motion of the objects of interest (i.e. ships or boats) can be mixed with the dynamic behavior of the background (non-regular patterns).

!

Video Content Analysis(VCA) or Video Analytics

Behavior Analysis

Image acquisition and preprocessing

ObjectDetection

ObjectTracking

event location

Intrusion detection Collision

prevention

Target detection and tracking

Anomaly detection

Target behavior analysis

Traffic data collection and

analysis

activity report

Understanding an Intelligent Video Surveillance Framework

supervisor

Behind the Scenes of an Intelligent Video Surveillance Framework

!

Example of automatic incident detection

our focus

Introduction to Background Subtraction

Initialize Background Model

frame modelForegroundDetection

Background Model Maintenance

Background subtraction methods

Traditional methods:• Basic methods, mean and variance over time• Fuzzy based methods• Statistical methods • Non-parametric methods• Neural and neuro-fuzzy methods

Matrix and Tensor Factorization methods:• Eigenspace-based methods (PCA / SVD)• RPCA, LRR, NMF, MC, ST, etc.• Tensor Decomposition, NTF, etc.

BGSLibrary (C++)https://github.com/andrewssobral/bgslibrary

A large number of algorithms have been proposed for background subtraction over the last few years:

LRSLibrary (MatLab)https://github.com/andrewssobral/lrslibrary

our focus

Andrews Sobral and Antoine Vacavant. A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos. Computer Vision and Image Understanding (CVIU), 2014. http://dx.doi.org/10.1016/j.cviu.2013.12.005

Robust Principal Component Analysis (RPCA)

▪ RPCA can be formulated as the problem of decomposing a data matrix (M) into two components (L) and (S), where (M) is the sum of a low-rank matrix (L) and a sparse matrix (S):

▪ M = L + S

Sparse error matrix

SL

Underlying low-rank matrix

M

Matrix of corrupted observations

Robust Principal Component Analysis (RPCA)

http://perception.csl.illinois.edu/matrix-rank/home.html

M L S

(tradeoff between rank and sparsity)

What about RPCA for dynamic background?

▪  

The SCM-RPCA approach

▪ We propose to combine some ideas of Oreifej et al. (2013) and Yang et al. (2015).

▪ The weighting matrix proposed by Yang et al. (2015) can be used as a shape constraint (or region constraint),

▪ While the confidence map proposed by Oreifej et al. (2013) reinforces the pixels belonging from the moving objects.

▪ The original 3WD was modified adding the shape constraint as has been done in the RMAMR.

▪ We chose to modify the 3WD instead of RMAMR due its capacity to deal more robustly with the multimodality of the background.

Definition of shape and confidence map

▪ The second contribution of this paper refers to the way of building the shape constraint and confidence map.

▪ Instead of using dense optical flow (temporal descriptor) as a preliminary step, we suggest to use a saliency detector (spatial descriptor).

▪ In some cases where:▪ a) the object of interest moves very slowly (i.e long distance boats) or

▪ b) the background is very dynamic (i.e boats in the sea), the optical flow may not be enough to ensure the object detection.

▪ In addition, the computation of the dense optical flow usually request more computational cost than saliency maps.▪ In this work, the BMS method proposed by Zhang and Sclaroff (2014) was selected by its speed performance and

visual results.

http://cs-people.bu.edu/jmzhang/BMS/BMS.html

Block diagram of the SCM-RPCA

Is important to note that the double constraints (confidence map and shape) can be built from two different types of source (i.e. from spatial, temporal, or spatio-temporal information), but in this work we focus only on spatial saliency maps.

Solving the SCM-RPCA

Experimental results

UCSD MarDT

The UCSD and MarDT data sets consists of 18 and 28 video sequences, respectively, both acquired from a stationary and moving camera, but in this work (due to page limits) we have selected only four sequences from UCSD and three sequences from MarDT.

http://www.svcl.ucsd.edu/projects/background\_subtraction/ucsdbgsub\_dataset.htmhttp://www.dis.uniroma1.it/~labrococo/MAR/index.htm

Quantitative results on UCSD data set

The quantitative results in Table 2 show that the SCM-RPCA outperforms the previous methods, with the highest F-measure average over the selected video sequences.

Is important to note that all constrained RPCA evaluated in this paper have been used the saliency maps as the input constraint.

Benchmark evaluation over UCSD data set

The algorithms were implemented in MATLAB (R2014a) running on a laptop computer with Windows 7 Professional 64 bits, 2.7 GHz Core i7-3740QM processor and 32Gb of RAM.

We can see that the combination of shape constraint and confidence map did not changed significantly the number of iterations and computation time over original 3WD.

Visual results on UCSD data set

From left to right: (a) input frame, (b) saliency map generated by BMS, (c) ground truth, (d) proposed approach, (e) 3WD, and (f) RMAMR.

Visual results on MarDT data set

Is important to note that in the UCSD scenes we have used the original spatial saliency map provided by BMS, while for the MarDT scenes we have subtracted its temporal median due to the high saliency from the buildings around the river.

Conclusions

▪ The experimental results indicate a better enhancement of the object foreground mask when compared with its direct competitors.

▪ As shown in quality evaluation, the combination with confidence map and shape constraint can reduce the amount of false positive pixels.

▪ In the future work, we will investigate how the combination of different types of source can improve the foreground detection.

▪ We are also interested to perform the decomposition incrementally as in Online RPCA proposed by Feng et al. (2013).

▪ MATLAB codes available: https://sites.google.com/site/scmrpca