Video Trails: Representing and Visualizing Structure in Video Sequences

Vikrant Kobla

David Doermann

Christos Faloutsos

Outline

Background and Motivation Overview Video Trails Trail Segmentation Trail Classification Gradual Transition Detection Experiments and Results Conclusion

Background and Motivation

Video is a valuable information resource There are still few efficient ways to provide

access to the information the video contains Early work on indexing video treated video

sequence as collections of still images, ignored the temporal structure

Efficient analysis and representation of the temporal structure of a video is necessary

Overview

1. Generate a trail of points (Video Trails) in a low-dimensional space

2. Segment the video trails

3. Classify each of those segmented trails into two types:

Stationary (low activity) VS Transitional (high activity)

4. Detect gradual transition

Video Trails

Definition: A trail of points in a low-dimensional space where each point is derived from physical features of a single frame in the video clip

Features: DC coefficients of the luminance and chrominance components of an MPEG frame

Dimensionality Reduction (FastMap) initial feature vector

a vector in that dimensional

target dimension spaceFastMap

Example

Consider a video clip with a 320x240 frame size Each frame has 20x15 MBs( Macroblock) Each MB contains 6 DC coefficients ( 4

luminance and 2 chrominance) Totally, 20x15x6=1800 coefficients (initial vector) 1800-by-1 vector

(X1,X2,X3)

3 (target dimension) FastMap

Example

Example

Trail Segmentation

Segment the video in order to determine regions of high activity corresponding to transitions and low activity corresponding to individual shots

The problem of segmenting the video into sets of frames is transformed into the problem of splitting the video trails into smaller trails corresponding to segments of video

Splitting Algorithm

1. Start by placing the first point in a new trail

2. Consider each successive point in the sequence in order

3. Perform a test for “inclusion” of this point in the current trail

4. if (the test pass)

5. Include the point in the current trail

6. Move to the next point

7. Goto 2

8. else

9. Close the current trail with the previous point as the last one

10. Start a new trail with only the current point

11. Goto 2

“Inclusion Test”

Marginal Cost:Total cost per point in the trail Consider a clip with N frames Assume there are m points in the current

trail, denoted by set , and be the point being considered for inclusion

Define ,d is the dimensionality So the new marginal cost is

new marginal cost > previous one : not include

new marginal cost < previous one : include

Example

Example (close-up)

The sequence of frames that yield the sparse transition between the two dense clusters

Trail Classification

Classify each of those segmented trails into: Stationary (low activity) or Transitional (high activity)

Classification Criteria– Monotonicity W1=0.4 – Sparsity W2=0.3– Convex Hull Volume Ratio W3=0.2– MBR Shape W4=0.1

Monotonicity

If a trail is (close to) monotonic, in some direction,it’s likely transitional projection of distance along k

projected distance ratio

the length of MBR dimension k Minimum projected distance ratio

Monotonicity (Normalization)

Recall: W1 is the weight of monotonicity Tlow is the lower bound=1.1 Tup is the upper bound=2.0

Sparsity

Sparsity: total MBR volume per point

Average Sparsity

Sparsity Ratio

Normalize

Convex Hull Volume Ratio

The ratio of volume of the convex hull of points in a trail to the volume of MBR

Normalize

MBR Shape

Cuboidal Planar Elongated

Classification

Gradual Transition Detection

Dissolves, Fades, Wipes Difficulty: activity arising from camera or

large object motion also yields trails similar to trails resulting from gradual edits

Filter out any kind of global motion leading to a transitional trail, Analysis global motion

Results

Conclusion

Provide a compact representation of a video sequence structure

Reduce a sequence MPEG frames to a trail of points in a low dimensional space

Segment trails and classify each segment as either stationary or transitional

Detect gradual edits