tsinghua @ trecvid2007.search - nist · c tf idf c d freq c d concept relevance concept specificity...
TRANSCRIPT
![Page 1: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/1.jpg)
Tsinghua @ TRECVID2007.search
Zhikun Wang, Dong Wang, Huiyi Wang, Tongchun Xiao,
Duanpeng Wang, Yingyu Liang, Yang Pang
Jianmin Li, Fuzong Lin, Bo Zhang
![Page 2: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/2.jpg)
Outline
System Overview
Concept-Based Search
Experiments & Results
Conclusion
![Page 3: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/3.jpg)
Outline
System Overview
Concept-Based Search
Experiments & Results
Conclusion
![Page 4: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/4.jpg)
Automatic Search System
Framework
Find shots of water
with boats or ships
Multimedia Query
Text-based
Retrieval
Visual-based
Retrieval
ConceptConcept-
based
Retrieval
Multi-
Modal
Fusion
Text
Visual
![Page 5: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/5.jpg)
Automatic Search System
Text-based search
Keywords: expanded by WordNet
Transcript segmentation:
shot-level, story-level, video-level
Result expansion for shot-level search:
scores spread along the timeline
![Page 6: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/6.jpg)
Automatic Search System
Text-based search
Visual-based search
Richer feature set
Feature selection & fixed-value fusion weight:
MAP & consistency
5 features involved
Several SVM classifiers for each feature
Weighted average multi-feature fusion
![Page 7: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/7.jpg)
Automatic Search System
Text-based search
Visual-based search
Concept-based search
Query-concept mapping
Text-concept mapping
Example-concept mapping
More details come later.
![Page 8: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/8.jpg)
Automatic Search System
Text-based search
Visual-based search
Concept-based search
Fusion
Weighted average
Query-independent
![Page 9: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/9.jpg)
Interactive Search System
User interface
faster, faster and faster
Browsing functions
Server end
Several options
![Page 10: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/10.jpg)
Interactive Search System: UI
Double-screen interface
Multi-thread browsing
Temporal thread
Visual neighbor thread
Frame-level browsing
Browsing function
Forward, Backward, Bookmark
Hotkey
![Page 11: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/11.jpg)
Browsing
Rank listTemporal thread
Visual neighbor thread
Faster Browsing
Story browserFrame-level browsing
![Page 12: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/12.jpg)
Labeling:Hotkey & Mouse
![Page 13: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/13.jpg)
RefiningPositive samplesNegative samples
Uncertain samples
Bookmark
![Page 14: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/14.jpg)
Server end
Distributed server end
More options
1 text-based server
4 SVM models with different features
2 concept-based servers
manually adjusted options Vs. default options
![Page 15: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/15.jpg)
Outline
System Overview
Concept-Based Search
Experiments & Results
Conclusion
![Page 16: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/16.jpg)
Concept-Based Search
Well established approach
Need theoretical guidance for practical issues
Query-Concept Mapping (QUCOM)
Query Image
Boat/Ship, Waterscape, …
![Page 17: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/17.jpg)
Possible Solutions for QUCOM
User choice?
Text Match ([Snoek, 2006], [Chang, 2006], etc)
Effective if well matched
Fails to consider visual correlation
concept performance
concept distribution over the collection
Concept Space
Search in Full Space (e.g. SVM, KNN
[Natsev, 2006], PMIWS
[Zheng, 2006])
Search in
Concept Subspace
![Page 18: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/18.jpg)
Concept Selection via c-tf-idf Metric
Concept Relevance Ranking
tf: frequency of a term in a document term popularity
idf: inverse document frequency of a term term specificity
Definition in text area
c: concept, d: shot
c-tf-idf: tf-idf for concept
))(
log()|())(
log(),(:),(cfreq
NdcP
cfreq
Ndcfreqdcidftfc
![Page 19: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/19.jpg)
Insight of the tf-idf based Principle)
)(log(),(:),(
cfreq
Ndcfreqdcidftfc
Concept Relevance
Concept Specificity
Query-Dependent
Rank
Query-Independent
Rank
c-tf-idf is a good combination of query-dependent ranksand query-independent ranks, and a promising solutionfor QUCOM.
![Page 20: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/20.jpg)
Two Settings for QUCOM
Automatic video retrieval (AVR) limited information as text input, and possibly, image examples
Interactive video retrieval (IVR) unrealistic to ask user provide relevant concepts
Infer the implicit semantic concepts by explicit user feedback
QUCOM should be On a per query analysis basis, on-the-fly,
Combat against varied concept detection performance
Scalable to
Concepts in a given lexicon
Video archive size
![Page 21: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/21.jpg)
Concept-Based Search: Search
Search in concept subspace
Impact of dimension of subspace
0.31
0.315
0.32
0.325
0.33
0.335
1 2 3 4 5 6
MAP
MAP
Experiment on TRECVID 2006, interactive search
Experiment on TRECVID 2006, automatic search
![Page 22: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/22.jpg)
Inferring implicit concepts through
explicit feedback: Interactive Search
Interactive Search
Using relevance feedback as examples
Higher efficiency: Vs. user-provided examples
Pre-computed offline
Lower user labor: Vs. manual concept selection
Better performance: Vs. previous system
65% improvement upon previous method (without using concepts)
experiment on TRECVID 2006, interactive search
![Page 23: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/23.jpg)
Concept-Based Search: Lexicons
LSCOM-lite
39 concept detectors from HLF task
LSCOM
374 concepts chosen from LSCOM
Impact of quality & quantity?
Experiment on TRECVID 2006, interactive search
![Page 24: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/24.jpg)
Outline
System Overview
Concept-Based Search
Experiments & Results
Conclusion
![Page 25: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/25.jpg)
Automatic runs
Run1:text :0.011
Run2:image + LSCOM-Lite :0.042
Run3:text+image :0.038
Run4:text+image+LSCOM :0.043
0
0.05
0.1
0.15
0.2
0.25
MAP 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220
Run1: text Run2: image+LSCOM-Lite Run3: text+image Run4: text+image+LSCOM
![Page 26: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/26.jpg)
Run1
Text-based search
Helpful to topics about Object
Useless to topics about Event or Scene
Unsatisfactory upon non-news video
![Page 27: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/27.jpg)
Run2, Run3, Run4
Run2 Vs. Run 4 Concept detectors from LSCOM(except 39 concepts
from HLF) are trained upon different dataset.
Run2 Vs. Run3 Involving concept-based search brings improvement.
0
0.01
0.02
0.03
0.04
0.05
Run1: text Run2:image+LSCOM-lite Run3:text+image Run4:text+image+LSCOMMAP 0.0104 0.041 0.0376 0.0426
MAP
![Page 28: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/28.jpg)
Interactive runs
Run5: expert with manually adjusted options :0.209
Run6: expert with default options :0.171
RunS: novice with default options :0.149
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
MAP 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220
Run5: expert with manually adjusted options Run6: expert with default options RunS: novice
![Page 29: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/29.jpg)
Relevant results retrieved
0
200
400
600
800
1000
1200
ret_rel rel
![Page 30: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/30.jpg)
Outline
System Overview
Concept-Based Search
Experiments & Results
Conclusion
![Page 31: Tsinghua @ TRECVID2007.search - NIST · c tf idf c d freq c d Concept Relevance Concept Specificity Query-Dependent Rank Query-Independent Rank c-tf-idf is a good combination of query-dependent](https://reader035.vdocuments.mx/reader035/viewer/2022071216/604886099d6a0a6f4f21fa98/html5/thumbnails/31.jpg)
Conclusion
Concept-based search is fruitful and complement to text and visual search
A easy-to-use UI is essential to interactive search
User can make-up the drop in automatic