context-based visual concept detection using domain adaptive semantic diffusion yu-gang jiang, jun...
TRANSCRIPT
![Page 1: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/1.jpg)
Context-based Visual Concept Context-based Visual Concept Detection Using Domain Adaptive Detection Using Domain Adaptive
Semantic DiffusionSemantic Diffusion
Yu-Gang Jiang‡, Jun Wang‡, Shih-Fu Chang‡, Chong-Wah Ngo† † VIREO Research Group (VIREO), City University of Hong Kong‡ Digital Video and Multimedia Lab (DVMM), Columbia University
1NIST TRECVID Workshop, Nov. 2009
![Page 2: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/2.jpg)
Overview: framework
Local FeatureLocal Feature Global FeatureGlobal Feature
SVM ClassifiersSVM Classifiers
66
55
1-41-4
VIREO-374:374 LSCOM
concept detectors
VIREO-374:374 LSCOM
concept detectors
Domain Adaptive Semantic DiffusionDomain Adaptive
Semantic Diffusion
![Page 3: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/3.jpg)
3
Overview: performance
DASDLocal + global features
Local feature alone
Local feature is still the most powerful component (MAP=0.150) Global features help a little bit (MAP=0.156) DASD further contributes incrementally to the final detection
![Page 4: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/4.jpg)
Overview: framework
Local FeatureLocal Feature Global FeatureGlobal Feature
SVM ClassifiersSVM Classifiers
66
55
1-41-4
VIREO-374:374 LSCOM
concept detectors
VIREO-374:374 LSCOM
concept detectors
Domain Adaptive Semantic DiffusionDomain Adaptive
Semantic Diffusion
![Page 5: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/5.jpg)
Local feature representation
5
Chang et al TRECVID 2008; Jiang, Yang, Ngo & Hauptmann, IEEE TMM, to appear
Keypoint extraction
Visual word vocabulary 1
SIF
T fe
atur
e sp
ace
......... .........
Visual word vocabulary 2
DoG Hessian Affine
+ + +
+ + ++ + ++ + +
+
- - -
- - -
- - -
- - - - - -
- - - - - - - - -
SVM classifiers
......
+ + +
+ + ++ + ++ + +
+
- - -
- - -
- - -
- - - - - -
- - - - - - - - -
+ + +
+ + ++ + +
- - -
- - -
- - -
- - - - - -
- - - - - - - - -
Vocabulary Generation BoW Representation
+ + +
+ + ++ + +
- - -
- - -
- - -
- - - - - -
- - - - - - - - -
BoW histograms Using Soft-weighting
![Page 6: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/6.jpg)
Context-based concept detection
Local FeatureLocal Feature Global FeatureGlobal Feature
SVM ClassifiersSVM Classifiers
66
55
1-41-4
VIREO-374:374 LSCOM
concept detectors
VIREO-374:374 LSCOM
concept detectors
DASD: Domain Adaptive Semantic
Diffusion
DASD: Domain Adaptive Semantic
Diffusion
![Page 7: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/7.jpg)
DASD - motivation
• Most existing methods aim at the assignment of concept labels individually– but concepts do not occur in isolation!
military personnel
smoke
explosion_fire
road outdoor
vehicle
building
7
![Page 8: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/8.jpg)
8
Documentary Videos
Broadcast News Videos
• Most existing methods aim at the assignment of concept labels individually– but concepts do not occur in isolation!
• Domain change between training and testing data was not considered
DASD - motivation
![Page 9: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/9.jpg)
9Jiang, Wang, Chang & Ngo, ICCV 2009
DASD - overview
road vehicle
0.05
0.19
0.80
0.46
0.13
0.01
0.12
0.91
0.18
0.05
water
0.11
0.58
0.10
0.13
0.02
sky
0.01
0.36
0.53
0.17
0.23
![Page 10: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/10.jpg)
DASD - overview
• Domain adaptive semantic diffusion (DASD)– Semantic graph
• Nodes are concepts• Edges represent
concept correlation
– Graph diffusion• Smooth concept
detection scores w.r.t the concept correlation
vehicle
road
Water sky
0.10.20.80.50.1…0.4
0.10.60.10.10.0…0.8
0.00.40.50.20.8…0.7
0.00.10.90.20.1…0.3
10
![Page 11: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/11.jpg)
DASD - formulation
• Energy function
11
Concept affinityConcept affinity
Detection score of concept ci on test samples Detection score of concept ci on test samples
![Page 12: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/12.jpg)
DASD - formulation (cont.)
• Gradually smooth the function makes the detection scores in accordance with the concept relationships
Detection score smoothing processDetection score smoothing process
12
![Page 13: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/13.jpg)
DASD - formulation (cont.)
• Graph adaptation
Graph adaptation processGraph adaptation process
13
![Page 14: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/14.jpg)
Iteration: 8Iteration: 12
0.16
0.09VEHICLE
0.640.20
0.24
0.29
DESERT
SKY
CLOUDS
WEAPON
0.10
PARKING_LOT
CAR0.16
0.09VEHICLE
0.640.19
0.18
0.32
DESERT
SKY
CLOUDS
WEAPON
0.13
PARKING_LOT
CAR
Iteration: 0Iteration: 4
0.15
0.09VEHICLE
0.640.17
0.12
0.34
DESERT
SKY
CLOUDS
WEAPON
0.16
PARKING_LOT
CAR0.15
0.08VEHICLE
0.640.17
0.05
0.38
DESERT
SKY
CLOUDS
WEAPON
0.19
PARKING_LOT
CAR
Iteration: 16
0.15
0.08VEHICLE
0.640.16
0.00
0.42
DESERT
SKY
CLOUDS
WEAPON
0.24
PARKING_LOT
CAR0.15
0.08VEHICLE
0.640.16
0.00
0.43
DESERT
SKY
CLOUDS
WEAPON
0.27
PARKING_LOT
CAR
Iteration: 20
Graph adaptation - example
Broadcast news video domainBroadcast news video domain Documentary video domainDocumentary video domain
14
![Page 15: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/15.jpg)
Experiments on TV ’05-’07
• Baseline detectors– VIREO-374
• Graph construction:– Ground-truth labels on TRECVID 2005
SPORTSSPORTS WEATHERWEATHER
OFFICEOFFICE BUILDINGBUILDING
DESERTDESERT MOUNTAINMOUNTAIN
WALKINGWALKING
PEOPLE-PEOPLE-MARCHINGMARCHING
EXPLOSION-EXPLOSION-FIREFIRE
MAPMAP
TRUCKTRUCK
CORP. LEADERCORP. LEADER
SPORTS WEATHER OFFICE
DESERTDESERT MOUNTAINMOUNTAIN WATERWATER
POLICEPOLICE MILITARYMILITARY ANIMALANIMAL TWO PEOPLETWO PEOPLE
NIGHT TIMENIGHT TIME TELEPHONETELEPHONE
STREETSTREET
CLASSROOMBUS
TRECVID 05/06 (Broadcast News Videos)TRECVID 05/06 (Broadcast News Videos) TRECVID 07 (Documentary Videos)TRECVID 07 (Documentary Videos)
15
![Page 16: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/16.jpg)
Results on TV ’05-’07
• Performance gain on TRECVID 05-07 Datasets
TRECVID- 2005 2006 2007
# of evaluated concepts 39 20 20
Baseline (MAP) 0.166 0.154 0.099
SD 11.8% 15.6% 12.1%
DASD 11.9% 17.5% 16.2%
16
SD: semantic diffusion (without graph adaptation) Consistent improvement over all 3 data sets
DASD: domain adaptive semantic diffusion Graph adaptation further improves the performance
![Page 17: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/17.jpg)
Results on TV ’05-’07 (cont.)TRECVID 2006 Test Data
17
TRECVID Jiang et al Aytar et al Weng et al DASD
2005 2.2% 4.0% N/A 11.9%
2006 N/A N/A 16.7% 17.5%
Comparison with the state-of-the-artsComparison with the state-of-the-arts
![Page 18: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/18.jpg)
18
Results on TRECVID ’09
30%
10%
5%5%
![Page 19: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/19.jpg)
Results on TRECVID ’09 (cont.)
19
• Quality of contextual detectors (VIREO-374)
Context VIREO-374
TV09 detectors
TV06 detectors
TV07 detectors
18%
16%
5% DASD performance gain
![Page 20: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/20.jpg)
DASD - computational time
• Complexity is O(mn) – m: # concepts; n: # video shots
• Only 2 milliseconds per shot/keyframe!
20
TRECVID 05 TRECVID 06 TRECVID 07
SD 59s 84s 12s
DASD 89s 165s 28s
![Page 21: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/21.jpg)
Summary
21
• A well-designed approach using local features achieves good results for concept detection.
• Context information is helpful !– Domain adaptive semantic diffusion
• effective for enhancing concept detection accuracy
• can alleviate the effect of data domain changes
• highly efficient !
– Future directions include:• detector reliability: diffusion over directed graph
• web data annotation: utilize contextual information to improve the quality of tags
– Source code available for download from DVMM lab research page
![Page 22: Context-based Visual Concept Detection Using Domain Adaptive Semantic Diffusion Yu-Gang Jiang, Jun Wang, Shih-Fu Chang, Chong-Wah Ngo VIREO Research Group](https://reader036.vdocuments.mx/reader036/viewer/2022062619/551517f8550346c77d8b4e22/html5/thumbnails/22.jpg)
22