video summarization by video structure analysis and graph optimization m. phil 2 nd term...
Post on 20-Dec-2015
221 views
TRANSCRIPT
Video summarization by Video summarization by vvideo ideo sstrtructure analysis and graph optimizucture analysis and graph optimiz
ationation
M. Phil 2M. Phil 2ndnd Term Presentation Term Presentation
Lu ShiLu Shi
Dec 5, 2003Dec 5, 2003
OutlineOutline
MotivationMotivation Video structureVideo structure Video skim length distributionVideo skim length distribution Spatial-temporal graph modeling Spatial-temporal graph modeling Optimization based video shot selectionOptimization based video shot selection Experimental resultsExperimental results
MotivationMotivation
Huge volume of video data are distributed over the Huge volume of video data are distributed over the WebWeb
Browsing and management in the huge video Browsing and management in the huge video database are time consumingdatabase are time consuming
Help the user to quickly grasp the content of a videoHelp the user to quickly grasp the content of a video
Two kinds of applications:Two kinds of applications: Video skimming (dynamic)Video skimming (dynamic) Video static summary (static)Video static summary (static)
GoalsGoals
ConcisenessConciseness Content coverageContent coverage
Spatial and temporalSpatial and temporal CoherencyCoherency
Not too jumpyNot too jumpy
FlowchartFlowchart
Video structureVideo structure
Video narrates a story just like an article doesVideo narrates a story just like an article does Video (story)Video (story) Video scenes (paragraph)Video scenes (paragraph) Video shot groups Video shot groups Video shots (sentence)Video shots (sentence) Video framesVideo frames
Video structure Video structure Graphical exampleGraphical example
Video structureVideo structure
Can be built up in a bottom-up mannerCan be built up in a bottom-up manner Video shot detectionVideo shot detection Video shot groupingVideo shot grouping Video scene formation Video scene formation
Video structureVideo structure
Video shot detectionVideo shot detection Video slice image [1]Video slice image [1] Column - pairwise distanceColumn - pairwise distance Filtering and thresholdingFiltering and thresholding
… …… …
Video structureVideo structure
Video shot groupingVideo shot grouping Window-sweeping algorithm [2]Window-sweeping algorithm [2] Spatial similaritySpatial similarity Temporal distanceTemporal distance Intersected video shot groups form loop scenesIntersected video shot groups form loop scenes
Video structureVideo structure
Summarize each video scene respectivelySummarize each video scene respectively Loop scenes and progressive scenesLoop scenes and progressive scenes
Loop scenes depict an event happened at a placeLoop scenes depict an event happened at a place Progressive scenes: “transition” between events or Progressive scenes: “transition” between events or
dynamic eventsdynamic events
Video structureVideo structure
Scene importance: length and complexityScene importance: length and complexity Content entropy for loop scenesContent entropy for loop scenes Measure the complexity for a loop sceneMeasure the complexity for a loop scene
)log()(i
j
i
j
Sc
Sg
j Sc
Sg
i l
l
l
lScEntropy
Video structureVideo structure
Determine each video scene’s target skim lengthDetermine each video scene’s target skim length Determine each progressive scenes’ skim lengthDetermine each progressive scenes’ skim length
If , discard it, else If , discard it, else
Determine each loop scenes’ skim lengthDetermine each loop scenes’ skim length If ,discard itIf ,discard it
Redistribute to remaining scenesRedistribute to remaining scenes
1tL
Ll
v
vsSci
v
vsScSc L
Llskim
ii
2)(
)(' t
ScEntropyl
ScEntropylLskim
jjSc
iScvsSc
j
i
i
vsL'
Graph modelingGraph modeling
Spatial-temporal dissimilarity functionSpatial-temporal dissimilarity function Linear with visual dissimilarityLinear with visual dissimilarity Exponential with temporal distanceExponential with temporal distance
)),((),(1),( ji shshsTemporalDikjiji eshshVisualSimshshDis
Graph modelingGraph modeling
The spatial temporal relation graph The spatial temporal relation graph Each vertex corresponds to a video shotEach vertex corresponds to a video shot Each edge corresponds to the dissimilarity function betweeEach edge corresponds to the dissimilarity function betwee
n shotsn shots Directional and completeDirectional and complete
Skim generationSkim generation
The goal of video summarizationThe goal of video summarization Conciseness: given the target skim lengthConciseness: given the target skim length Content coverageContent coverage The spatial temporal dissimilarity functionThe spatial temporal dissimilarity function
The spatial temporal relation graph The spatial temporal relation graph A path corresponds to a series of video shotsA path corresponds to a series of video shots Vertex weight summationVertex weight summation Path length is the summation of the dissimilarity between Path length is the summation of the dissimilarity between
consecutive shot pairsconsecutive shot pairs
vsL
Skim generationSkim generation
Objectives:Objectives: Search for a path in the graph such that:Search for a path in the graph such that: Maximize the path length (dissimilarity Maximize the path length (dissimilarity
summation)summation) Vertex weight summation should be close to Vertex weight summation should be close to
but not exceed itbut not exceed it The objective function The objective function
vsL
vssvsspvssobj LpVWSLpVWSwLLpfs
)(),)((),(
sp
Skim generationSkim generation
Global optimal solutionGlobal optimal solution Let denote the paths begin with , whose Let denote the paths begin with , whose
vertex weight summation is upper bounded byvertex weight summation is upper bounded by The optimal path is denoted by The optimal path is denoted by
The target is The target is )( ,0
ovobj
vsLpf
}{ ,i
Lv rxp
rL
xv
)(max)( ,,i
Lvxobjio
Lvobj rrxpfpf
Skim generationSkim generation
Optimal substructureOptimal substructure
Dynamic programmingDynamic programming Effective way to compute the global optimal Effective way to compute the global optimal
solution solution Trace back to find the optimal pathTrace back to find the optimal path Time complexity , space complexity Time complexity , space complexity )( 2
vsLnO
nxlwshshDispfpfishiri
n
xirx shixo
lLvobjv
vvo
Lvobj ),),()((max)( ,1,
)()( , snrn vsho
Lvobj Llwpf
)( vsLnO
ExperimentsExperiments Key frames of selected video shotsKey frames of selected video shots
ExperimentsExperiments There is no ground truth so that it is hard to objectively evaluate a video There is no ground truth so that it is hard to objectively evaluate a video
skimskim Subjective experimentSubjective experiment Parameters:Parameters: 250,01.0sec,4sec,3 21 kwtt
ConclusionConclusion
Video structure analysisVideo structure analysis Scene boundaries, sub-skim length determinationScene boundaries, sub-skim length determination
Graph scene modelingGraph scene modeling Optimization based sub skim generationOptimization based sub skim generation Generate a video skimGenerate a video skim
ReferenceReference
[1] C. W. Ngo, Analysis of spatial temporal sli[1] C. W. Ngo, Analysis of spatial temporal slices for video content representation, Ph. D theces for video content representation, Ph. D thesis, HKUST, Aug.2000sis, HKUST, Aug.2000
[2] [2] Y. Rui, T.S. Huang, and S. Mehrotra, Constructing table-of content for videos, ACM Multimedia Systems Journal, Special Issue Multimedia Systems on Video Libraries, vol. 7, no.5, pp. 359~368, Sept 1999.
Q & AQ & A
Thank you!!Thank you!!