Automatic Video Shot Detection

from MPEG Bit StreamJianping Fan

Department of Computer ScienceUniversity of North Carolina at CharlotteCharlotte, NC 28223

jfan@uncc.edu

http://www.cs.uncc.edu/~jfan

Why we need video shots?

a. Text Retrieval: Keyword Extraction Indexing

Document Storage

Reverse File Indexing

Why we need video shots?

b. Database Query: Entity Extraction

sid name login age gpa

53666 Jones jones@cs 18 3.453688 Smith smith@eecs 18 3.253650 Smith smith@math 19 3.8

Indexing

Database Storage

B-Tree Indexing

Why we need video shots?

c. Image Retrieval: Object Indexing Indexing

Database Storage

Ontology Indexing

Why we need video shots?

MPEG Video Sequence

Video shot

frames

Why we need video shots?

Story Unit in News Video

Each story unit may consist of multiple connected video shots!

Why we need video shots?

Indexing

Video Shots in Storage

Shot Indexing

???

Storage Database

Query Processing

Server

Networks

???Video shot =?= keyword in video?

Shot is used as basic unit for video indexing!

1. Why we need shot detection?

Potential Search on Video Shots:

a. Give me some videos which consist of similar video shots!

b. Give me all the videos which are related to this video shot!

2. Content Structure of Video Sequence

Video sequence

Video shot

frames

Shot-Based Video Content Interpretation!

3. Coding Structure in MPEG Video

Video sequence

GOP

MPEG encoding

shot?

Interesting Questions

If you are leader of MPEG around 1994, how can you design new MPEG standards which can be used for video indexing?

New MPEG Standard with Video Shot

Video sequence

GOP GOP GOP

Video shot

frames

GOP is Video Shot!

4. What’s mean of video shot?

(a) Scene Cuts;

(b) Fade in & Fade out;

© Dissolves & Wipes

Why We Have Shots: (a) Human Editing; (b) Content Change & Camera Motion

Shot Types:

4. What’s mean of video shot?

time

video

Shot 1

a. Scene Cuts:

Sudden change of video content or focus

b. Fade: gradual transition between a screen and a

4. What’s mean of video shot?

constant image (fade out).

c. Dissolve: gradual transition from one screen to

4. What’s mean of video shot?

another, the first screen fade & the second one fade out.

4. What’s mean of video shot?

Dissolve

d. Wipe: a line moves across the screen & new appears behind the line

4. What’s mean of video shot?

4. What’s mean of video shot?

Wipe

Video Sequence

Video shot

frames

Conclusion: Why we have video shots?

Big change between two continuous video frames!

How can we detect video shots? Why we can define them as cuts,

wipe, ….? Because they are changed between neighboring video frames!!!!

How can we measure the changes or similarities between the neighboring video frames?

How can make decision on whether they are change or not?

5. How to detect video shot?

Major Components for Shot Detection:

a. Visual Representation of Video Frames: Color, Texture…

b. Difference Calculation for Neighboring Video Frames

c. Threshold for Decision Making: How Large is Enough?

5. How to detect video shot?

How to measure statistical property of video frames?

Color Histogram

00.050.1

0.150.2

0.250.3

0.350.4

(0,0,0) (110,112,80)

3-D Column 1RGB

5. How to detect video shot?

How to measure statistical property of video frames?

Color Histogram

5. How to detect video shot?

How to measure statistical property of video frames?

Texture Histogram

5. How to detect video shot?

5. How to detect video shot?

How we can do this more efficient on MPEG videos?

MPEG videos

MPEG Decoder

Color Histogram Difference

Scene Cut Frame Output

Automatic Threshold

Determination

a. Easy but not smart

Scene Cut

Effects on color histogram difference by scene cuts:

5. How to detect video shot?

5. How to detect video shot?

How to measure the change of video content?

Color Histogram Difference

5. How to detect video shot?

How we can select the threshold automatically?

a. J. Fan, D.K.Y. Yau, W.G. Aref, A. Rezgui, ``Adaptive motion- compensated video coding scheme towards content-based bit rate allocation”, Journal of Electronic Imaging, vol.9, no.4, pp. 521-533, 2000.

This algorithm can adapt the thresholds to different videos.

b. J. Fan, et al., ``ClassView: Hierarchical video shot classification and retrieval”, IEEE Trans. on Multimedia, 2004.

This algorithm can adapt thresholds to different video shots.

5. How to detect video shot?

Relationships among continuous frames can defined as:

scene cuts versus non-scene cuts

probability

0

1

1T

non-scene cut scene cut

5. How to detect video shot?

How to obtain T automatically?

Entropy for non-scene cut frames:

Interesting Questions

What will happen on MPEG bit stream if the current video frame is a shot boundary?

I B PHow can we use such properties for shot detection?

5. How to detect video shot?

b. Complex but smart approach

GOP MPEG encoding

If scene cuts happen on I frames in MPEG Video?

How to calculate color histograms of I frames efficiently?

Only decode DC coefficient!!!

5. How to detect video shot?

b. Complex but smart approach

What will happen on MPEG bit stream if there is a shot boundary?

1.5Mb/s

4Mb/s

Picture Quality high

low

Reference frame( I or P)

Current P frame

Most macroblocks can not find their correspondences!!

If content change happens on P frames in MPEG video?

Only decode the block coding types in P frames!!!

Previous Reference

Current B frame Future Reference

Most macroblocks in B current frame can only be predicted by backward way!!

If content change happens on B frames in MPEG video?

Only decode the coding types in B frames!!!!

6. Shot Detection via Edge Extraction

Edges indicate the content structure of video frames!

6. Shot Detection via Edge Extraction

MPEG videos

MPEG Decoder

Color Edge Detection

Scene Cut Frame Output

Automatic Threshold

Determination

a. Easy but not smart

6. Shot Detection via Edge Extraction

a. J. Fan, W.G. Aref, M.-S. Hacid and A.K. Elmagarmid, ``An automatic isotropic color edge detection technique”, Pattern Recognition Letters, vol.22, pp.1419-1429, 2001.

1 2 1

0 0 0

-1 -2 -1

1 0 -1

2 0 -2

1 0 -1

2 1 0

1 0 -1

0 -1 -2

0 1 2

-1 0 1

-2 -1 0

6. Shot Detection via Edge Extraction

b. complex but smartOnly decode only part of MPEG videos

255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255255 255 255 255 255 255 255 255

DCT

2550 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 00 0 0 0 0 0 0 0

What you find from this figure?

6. Shot Detection via Edge Extraction

1. Calculate the directional edge histogram

Horizontal edgesVertical edges Northwest diagonal edgesNortheast diagonal edges

2. Calculate the differences of the directional edge histogram

3. Threshold to obtain the scene cuts

Effects on color histogram different by Fades & Dissolves

7. Complex Shot Detection

Wipe Modeling

Dissolve Modeling

7. Complex Shot Detection

8. Camera Motion Detection

a. Zoom in

8. Camera Motion Detection

b. Zoom out

8. Camera Motion Detection

c. Left Pan

8. Camera Motion Detection

d. Right Pan

Why We Need Keyframe Extraction?

Why We Need Keyframe Extraction?

Keyframe Extraction

What are keyframes? Different from others!

How to measure the significance of video frames?

How to make the decision?

9. Background Modeling for Shot

Detection

9. Background Modeling for Shot

Detection

9. Background Modeling for Shot

Detection

9. Background Modeling for Shot

Detection

9. Background Modeling for Shot

Detection

9. Background Modeling for Shot

Detection

10. Shot-Based Video Content Representation

Statistical Information

Keyframe-Based Information

(a)Color histogram & variance;

(b) edge histogram & variance

© motion histogram & variance;

(a)Color histogram; (b) edge histogram

© motion histogram

11. Shot-Oriented Video Semantics Interpretation

Who cares?

12. Who works on this topic?

1. IBM research center

2. Intel

3. Microsoft Redmond & Beijing

4. Kodak at Rochester

5. HP at Polo Alto

6. Google

7. Yahoo!

12. Requirement

If you are asked to work on video shot detection from compressed video stream, what you should know (basic requirement)?a. MPEG video coding standard:

how to decode and read MPEG file

b. Knowledge on video shot analysis:

Color, motion, edge, fade, dissolve, …

c. Skills on C++

Project Introduction

Design a system for automatic shot detection from MPEG video streams

Interface design: read in MPEG files and display shot boundaries, or even decision process!