unsupervised detection of regions of interest using iterative link analysis gunhee kim 1 antonio...
TRANSCRIPT
![Page 1: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/1.jpg)
Unsupervised Detection of Regions of Interest
Using Iterative Link Analysis
Gunhee Kim1 Antonio Torralba2
1: SCS, CMU2: CSAIL, MIT
Neural Information Processing Systems 2009
November 30, 2009
![Page 2: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/2.jpg)
Unsupervised Detection of ROIs
A set of images…
Rectangular Regions of Interest
![Page 3: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/3.jpg)
Why Is the ROI Detection Useful ?
Scene recognition[Quattoni&Torralba, CVPR09]
Training for Recognition[Bosch et al, ICCV07]
Flickr Notes
![Page 4: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/4.jpg)
Alternating Optimization
• One of widely used heuristics for iterative optimization
• Optimization over two sets of variables is not easy
• But affordable to optimize one while the other is fixed
![Page 5: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/5.jpg)
• Goal: Find correspondences between two sets of point clouds [Besl&McKay,1992]
Example – Iterative Closest Point Algorithm
Trans-formation
Estimate transformation parameters
Corres-pondences
Associate points by NN criteria
![Page 6: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/6.jpg)
• Goal: Clustering
Example – K-means
ClusterMembershi
p
Find nearest cluster center
ClusterCenters
Take mean
Initialization
Pictures from Bishop’s book
![Page 7: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/7.jpg)
• Goal: Find best ROIs in each image of dataset
Unsupervised Detection of ROIs
RefineROIs
Detection or Localization
FindExamplars
Modeling or Ranking
examplars
Where is butterfly?
What are examplars?
![Page 8: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/8.jpg)
Our Approach
• Inspired by alternating optimization
• Based on link analysis of hypothesis network.
• Find Examplars = Central and diverse Hubs
• Refine ROIs = Highly-ranked Hypotheses in each image wrt examplars
• Easy, Fast and Dynamic– Simple heuristic for linearity of computation wrt dataset size.– Ex. 4.5 hours / 200k images with naïve matlab implementation.
![Page 9: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/9.jpg)
ROI Candidates and Description
• For each , define– At least one of would be good
• Description: Spatial pyramids of visual words and HOG
• Similarity measure: Cosine similarity
An image 15 segments 43 ROI hypotheses
Visual words Edge Gradient
![Page 10: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/10.jpg)
Algorithm - Input
• Image set and its ROI hypothesis set
![Page 11: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/11.jpg)
Algorithm - Initialization
• Best ROI = Image itself !
![Page 12: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/12.jpg)
Algorithm - Initialization
• Initialization is essential for the success !
• Why is it a feasible idea for Web images ?– Most pictures are taken from a canonical view so that an object
of interest is located in a center with significant size.– Given a similarity network of a sufficiently large number of
images, democratic voting reveals the most dominant visual information as hubs [Kim et al 08]
Examples of top-ranked Images
![Page 13: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/13.jpg)
Algorithm – First Hub Seeking
• Generate a similarity network and find a hub set
![Page 14: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/14.jpg)
Algorithm – First ROI Refinement
• Bipartite graph between hub sets and All ROIs of an image
![Page 15: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/15.jpg)
Algorithm – Second Hub Seeking
• Keep iterating…
![Page 16: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/16.jpg)
Hub Seeking with Centrality & Diversity
• Mean-shift like hub seeking algorithm
Mean Shift[Comaniciu and Meer,
PAMI 2002]
K-NN similarity matrix PageRank vector
G(t)
K-NN graph
Degree distribution ~ PageRank vector
![Page 17: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/17.jpg)
Hub Seeking with Centrality & Diversity
• Mean-shift like hub seeking algorithm
0.05
0.20.5
0.25
0.80.5
0.1
Max P-value !
Fixed radius window= max. reachable probability d (= 0.1)
Mean Shift
![Page 18: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/18.jpg)
ROI Refinement
• Augmented Bipartite Graph
(1-α)Wo
WoT
αWi
ROI hypothesis Hub set vector
RO
I hyp
othe
ses
Hub
set
PageRank
Argmax ( )i
![Page 19: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/19.jpg)
ROI Refinement
• What does α do?
(1-α)Wo
WoT
αWi
α = 0 α = 0.1
Wo
WoT
![Page 20: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/20.jpg)
Example - ROI Refinement
T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7
T=0 T=1 T=2 T=3 T=4 T=5 T=6 T=7
![Page 21: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/21.jpg)
Scalability Setting
• Bottleneck: Quadratic computation to generate a similarity matrix of selected ROIs
• If dataset size is too large, – Run the algorithm with N number of images (N = 10,000)– Re-use x % of previous high-ranked images.
Dataset
N
N N
N
![Page 22: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/22.jpg)
Experiments
• Performance Test– PASCAL VOC 2006 Dataset– Weakly-supervised1 and Unsupervised2
• Scalability Test– Five objects: {butterfly+insect (69,990), classic+car (265,731),
motorcycle+bike (106,590), sunflower (165,235), giraffe+zoo (53,620)}
– Weakly-supervised1
1: Input imageset consists of a single object type (only localization is required) 2: Input imageset consists of multiple object types (localization and clustering are required)
![Page 23: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/23.jpg)
Performance Tests
• Weakly Supervised Localization (PR-Curves)
[Russell et al. CVPR 2006]http://www.di.ens.fr/russell/projects/mult seg discovery/index.html
X-axis: RecallY-axis: Precision
![Page 24: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/24.jpg)
Performance Tests
• Unsupervised Classification & Localization
X-axis: RecallY-axis: Precision
X-axis: FP rateY-axis: TP rate
ROCCurves
PRCurves
![Page 25: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/25.jpg)
Scalability Tests
• Weakly-supervised Localization
X-axis: RecallY-axis: Precision
![Page 26: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/26.jpg)
Perturbation Tests
• Robustness of ROI detection of each image against random network formation – 100 random sets of size of 200 images
Entropy: 0.2419 1.6846 2.4331
Dataset
An image of interest
X-axis: ROI hypothesesY-axis: Frequencies
![Page 27: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/27.jpg)
Localization Examples
![Page 28: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/28.jpg)
Conclusion
• Alternating optimization based Unsupervised ROI detection
• Simple and Fast
• Competitive performance on PASCAL 06
• Scalable Test with more than 200K Flickr images
• Critic: Analysis for convexity, convergence, sensitivity to initialization, quality of solution
![Page 29: Unsupervised Detection of Regions of Interest Using Iterative Link Analysis Gunhee Kim 1 Antonio Torralba 2 1: SCS, CMU 2: CSAIL, MIT Neural Information](https://reader035.vdocuments.mx/reader035/viewer/2022062421/56649c9c5503460f9495ad5a/html5/thumbnails/29.jpg)
Algorithm