Spatio-temporal Segmentation of Video by Hierarchical Mean

Shift AnalysisDaniel DeMenthon

SMVP 2002

Motivation

• Semantic understanding of video

• Object segmentation

• Video compression

• Event detection

• Video surveillance

Related Work

• Jojic et al. Flexible sprites

• Layer extraction

• Shi and Malik normalized cuts

• Irani et al. Event detection

Space – Time Volume Segmentation

Frame to Frame

Video Stack segmentation

Patch motion (1,u,v)

Feature Space

• 7 D feature vector, three color features in CIE L*u*v*, 2 motion angles, 2 motion distances.

ux arctan180

90

vy arctan180

90

xxx ttxxD cos)2/(sin)2/( maxmax

yyy ttyyD cos)2/(sin)2/( maxmax

Mapping Pixels in Feature Space

Mean Shift Clustering

• Introduced by Fukunaga (1990) and applied to image analysis by Yizhong Cheng and Comaniciu and Meer (1997)

• Natural borders (Leung et al.)

Range Search

• ATRIA tree

• O(N log N) for small radii

• O(N) for large radii

Hierarchical Mean Shift

• First standard mean shift is run until competition with very small radius

• Weights are assigned to cluster centers equal to the sum of the weights of the member points

• Clusters are now treated as the points, and radius is multiplied with factor of 1.25 or 1.50

• Repeat until desired radius or the desired number of regions is reached

“Flower Garden” Video Sequence

88 x 60, 12 frames

Video Strands

Color Segmentation

Motion Segmentation

Faster lateral motion corresponds to lighter color

Comparison of Two Segmentation Algorithms

Comments on this Approach

•Spatially distant color patches can be clustered together

•Experiment was with small number of frames

•It is not clear if it can handle the case when video object changes the direction of motion, or when video object stops

•All features (color features and motion features) are scaled using heuristics, and that might not work for different video sequences

Conclusions and Further Work

• Hierarchical mean shift analysis is of lower empirical complexity then standard mean shift analysis

• Segmentation can be improved, the bounding areas of moving areas are jagged, usually by post-processing

• Leung et al. suggested non parametric segmentation that can be applied here