![Page 1: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/1.jpg)
Video summarization by Video summarization by vvideo ideo sstrtructure analysis and graph optimizucture analysis and graph optimiz
ationation
M. Phil 2M. Phil 2ndnd Term Presentation Term Presentation
Lu ShiLu Shi
Dec 5, 2003Dec 5, 2003
![Page 2: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/2.jpg)
OutlineOutline
MotivationMotivation Video structureVideo structure Video skim length distributionVideo skim length distribution Spatial-temporal graph modeling Spatial-temporal graph modeling Optimization based video shot selectionOptimization based video shot selection Experimental resultsExperimental results
![Page 3: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/3.jpg)
MotivationMotivation
Huge volume of video data are distributed over the Huge volume of video data are distributed over the WebWeb
Browsing and management in the huge video Browsing and management in the huge video database are time consumingdatabase are time consuming
Help the user to quickly grasp the content of a videoHelp the user to quickly grasp the content of a video
Two kinds of applications:Two kinds of applications: Video skimming (dynamic)Video skimming (dynamic) Video static summary (static)Video static summary (static)
![Page 4: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/4.jpg)
GoalsGoals
ConcisenessConciseness Content coverageContent coverage
Spatial and temporalSpatial and temporal CoherencyCoherency
Not too jumpyNot too jumpy
![Page 5: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/5.jpg)
FlowchartFlowchart
![Page 6: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/6.jpg)
Video structureVideo structure
Video narrates a story just like an article doesVideo narrates a story just like an article does Video (story)Video (story) Video scenes (paragraph)Video scenes (paragraph) Video shot groups Video shot groups Video shots (sentence)Video shots (sentence) Video framesVideo frames
![Page 7: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/7.jpg)
Video structure Video structure Graphical exampleGraphical example
![Page 8: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/8.jpg)
Video structureVideo structure
Can be built up in a bottom-up mannerCan be built up in a bottom-up manner Video shot detectionVideo shot detection Video shot groupingVideo shot grouping Video scene formation Video scene formation
![Page 9: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/9.jpg)
Video structureVideo structure
Video shot detectionVideo shot detection Video slice image [1]Video slice image [1] Column - pairwise distanceColumn - pairwise distance Filtering and thresholdingFiltering and thresholding
… …… …
![Page 10: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/10.jpg)
Video structureVideo structure
Video shot groupingVideo shot grouping Window-sweeping algorithm [2]Window-sweeping algorithm [2] Spatial similaritySpatial similarity Temporal distanceTemporal distance Intersected video shot groups form loop scenesIntersected video shot groups form loop scenes
![Page 11: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/11.jpg)
Video structureVideo structure
Summarize each video scene respectivelySummarize each video scene respectively Loop scenes and progressive scenesLoop scenes and progressive scenes
Loop scenes depict an event happened at a placeLoop scenes depict an event happened at a place Progressive scenes: “transition” between events or Progressive scenes: “transition” between events or
dynamic eventsdynamic events
![Page 12: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/12.jpg)
Video structureVideo structure
Scene importance: length and complexityScene importance: length and complexity Content entropy for loop scenesContent entropy for loop scenes Measure the complexity for a loop sceneMeasure the complexity for a loop scene
)log()(i
j
i
j
Sc
Sg
j Sc
Sg
i l
l
l
lScEntropy
![Page 13: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/13.jpg)
Video structureVideo structure
Determine each video scene’s target skim lengthDetermine each video scene’s target skim length Determine each progressive scenes’ skim lengthDetermine each progressive scenes’ skim length
If , discard it, else If , discard it, else
Determine each loop scenes’ skim lengthDetermine each loop scenes’ skim length If ,discard itIf ,discard it
Redistribute to remaining scenesRedistribute to remaining scenes
1tL
Ll
v
vsSci
v
vsScSc L
Llskim
ii
2)(
)(' t
ScEntropyl
ScEntropylLskim
jjSc
iScvsSc
j
i
i
vsL'
![Page 14: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/14.jpg)
Graph modelingGraph modeling
Spatial-temporal dissimilarity functionSpatial-temporal dissimilarity function Linear with visual dissimilarityLinear with visual dissimilarity Exponential with temporal distanceExponential with temporal distance
)),((),(1),( ji shshsTemporalDikjiji eshshVisualSimshshDis
![Page 15: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/15.jpg)
Graph modelingGraph modeling
The spatial temporal relation graph The spatial temporal relation graph Each vertex corresponds to a video shotEach vertex corresponds to a video shot Each edge corresponds to the dissimilarity function betweeEach edge corresponds to the dissimilarity function betwee
n shotsn shots Directional and completeDirectional and complete
![Page 16: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/16.jpg)
Skim generationSkim generation
The goal of video summarizationThe goal of video summarization Conciseness: given the target skim lengthConciseness: given the target skim length Content coverageContent coverage The spatial temporal dissimilarity functionThe spatial temporal dissimilarity function
The spatial temporal relation graph The spatial temporal relation graph A path corresponds to a series of video shotsA path corresponds to a series of video shots Vertex weight summationVertex weight summation Path length is the summation of the dissimilarity between Path length is the summation of the dissimilarity between
consecutive shot pairsconsecutive shot pairs
vsL
![Page 17: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/17.jpg)
Skim generationSkim generation
Objectives:Objectives: Search for a path in the graph such that:Search for a path in the graph such that: Maximize the path length (dissimilarity Maximize the path length (dissimilarity
summation)summation) Vertex weight summation should be close to Vertex weight summation should be close to
but not exceed itbut not exceed it The objective function The objective function
vsL
vssvsspvssobj LpVWSLpVWSwLLpfs
)(),)((),(
sp
![Page 18: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/18.jpg)
Skim generationSkim generation
Global optimal solutionGlobal optimal solution Let denote the paths begin with , whose Let denote the paths begin with , whose
vertex weight summation is upper bounded byvertex weight summation is upper bounded by The optimal path is denoted by The optimal path is denoted by
The target is The target is )( ,0
ovobj
vsLpf
}{ ,i
Lv rxp
rL
xv
)(max)( ,,i
Lvxobjio
Lvobj rrxpfpf
![Page 19: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/19.jpg)
Skim generationSkim generation
Optimal substructureOptimal substructure
Dynamic programmingDynamic programming Effective way to compute the global optimal Effective way to compute the global optimal
solution solution Trace back to find the optimal pathTrace back to find the optimal path Time complexity , space complexity Time complexity , space complexity )( 2
vsLnO
nxlwshshDispfpfishiri
n
xirx shixo
lLvobjv
vvo
Lvobj ),),()((max)( ,1,
)()( , snrn vsho
Lvobj Llwpf
)( vsLnO
![Page 20: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/20.jpg)
ExperimentsExperiments Key frames of selected video shotsKey frames of selected video shots
![Page 21: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/21.jpg)
ExperimentsExperiments There is no ground truth so that it is hard to objectively evaluate a video There is no ground truth so that it is hard to objectively evaluate a video
skimskim Subjective experimentSubjective experiment Parameters:Parameters: 250,01.0sec,4sec,3 21 kwtt
![Page 22: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/22.jpg)
ConclusionConclusion
Video structure analysisVideo structure analysis Scene boundaries, sub-skim length determinationScene boundaries, sub-skim length determination
Graph scene modelingGraph scene modeling Optimization based sub skim generationOptimization based sub skim generation Generate a video skimGenerate a video skim
![Page 23: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/23.jpg)
ReferenceReference
[1] C. W. Ngo, Analysis of spatial temporal sli[1] C. W. Ngo, Analysis of spatial temporal slices for video content representation, Ph. D theces for video content representation, Ph. D thesis, HKUST, Aug.2000sis, HKUST, Aug.2000
[2] [2] Y. Rui, T.S. Huang, and S. Mehrotra, Constructing table-of content for videos, ACM Multimedia Systems Journal, Special Issue Multimedia Systems on Video Libraries, vol. 7, no.5, pp. 359~368, Sept 1999.
![Page 24: Video summarization by video structure analysis and graph optimization M. Phil 2 nd Term Presentation Lu Shi Dec 5, 2003](https://reader030.vdocuments.mx/reader030/viewer/2022032800/56649d445503460f94a20972/html5/thumbnails/24.jpg)
Q & AQ & A
Thank you!!Thank you!!