automatic keyframe selection based on mutual reinforcement algorithm
DESCRIPTION
Ventura, C.; Giro-i-Nieto, X.; Vilaplana, V.; Giribet, D.; Carasusan, E., "Automatic keyframe selection based on mutual reinforcement algorithm," Content-Based Multimedia Indexing (CBMI), 2013 11th International Workshop on , vol., no., pp.29,34, 17-19 June 2013 doi: 10.1109/CBMI.2013.6576548 This paper addresses the problem of video summarization through an automatic selection of a single representative keyframe. The proposed solution is based on the mutual reinforcement paradigm, where a keyframe is selected thanks to its highest and most frequent similarity to the rest of considered frames. Two variations of the algorithm are explored: a first one where only frames within the same video are used (intraclip mode) and a second one where the decision also depends on the previously selected keyframes of related videos (interclip mode). These two algorithms were evaluated by a set of professional documentalists from a broadcaster’s archive, and results concluded that the proposed techniques outperform the semi-manual solution adopted so far in the company. More details: https://imatge.upc.edu/web/publications/automatic-keyframe-selection-based-mutual-reinforcement-algorithmTRANSCRIPT
AUTOMATIC KEYFRAME SELECTION
BASED ON
MUTUAL REINFORCEMENT ALGORITHM
C. Ventura, X. Giró-i-Nieto, V. Vilaplana et al
1. Introducing the problem
Automatic selection of the representative
keyframe
2
1.1. What is the application?3
1.2. Current implementation4
ARBITRARY SAMPLING
MANUAL SELECTION
BY
PROFESSIONAL
2. Designing the system
2 scenarios:
Intra-clip mode
Inter-clip mode
Database
Textual search
to retrieve
related videos
5
2.1. General scheme6
Reranking
Frame
extraction
visual
features
Textual
search
Similarity
graph
Mutual
Reinforcement
Inter-clip
mode
21
3
12
3
2.2. Intra-clip mode
Mutual Reinforcement Algorithm (Joshi04)
Gets the frame with maximum coverage
(Joshi04) D. Joshi et al. The story picturing engine: finding elite
images to illustrate a story using mutual reinforcement. In MIR ‘04
7
2.3. Inter-clip mode
Reranking (based on Liu11)
(Liu11) C. Liu et al. Query sensitive dynamic web
video thumbnail generation. In ICIP ‘11
8
INTRA INTER
INPUT FRAMESREPRESENTATIVE KEYFRAMES
FROM TEXTUAL SEARCHER
FRAME HAAR WAVELET MORPHOLOGY
2.4. Post-processing block
9 Text filtering
Goal: To avoid representative keyframes with
textual captions
3. Experiments
Qualitative evaluation
Quantitative evaluation
MOS Test
Experimental dataset
10
3.1. Qualitative evaluation
Intra-clip mode
11
3.1. Qualitative evaluation
Inter-clip mode
Ranking scores
after mutual
reinforcement
(INTRA-CLIP
MODE)
Representative
keyframes of the
retrieved videos
12
3.1. Qualitative evaluation
Inter-clip mode
Final ranking scores after reranking:
13
3.2. Quantitative evaluation
MOS (Mean Opinion Score) test
Performed by TVC professionals
Scores
14
EXCELLENTGOODBAD
NON
ACCEPTABLEACCEPTABLE
NEWS DOMAIN
3.2. Quantitative evaluation
Database
15
POLITICS
INTERNATIONAL
ECONOMY
MORNING SHOW DOMAIN
INTERVIEW
DISCUSSION
3.2. Quantitative evaluation
MOS test
4 different approaches
Intra-clip
Inter-clip
Random
Current
16
3.2. Quantitative evaluation17
NEWS
MORNING
SHOW
3.2. Quantitative evaluation18
NEWSMORNING
SHOW
3. Experiments
Database and results are available on:
imatge.upc.edu
19
4. Conclusions
Keyframe selection based on mutual reinforcement algorithm To get the frame with maximum coverage within the
video in the intra-clip approach
Inter-clip approach Textual similarity to retrieve related videos
Linear fusion to get the new ranking scores
MOS test The semi-manual system (from TVC) can be
replaced by the automatic approach.
Inter-clip approach outperforms intra-clip in controlled environments.
20
21
2.3. Inter-clip mode
Textual search
2 modalities:
Textual searcher binary
TF-IDF descriptors
22