scaling object instance retrievalkosecka/cs682/lect-instance-retrieval.pdf · scaling object...
TRANSCRIPT
![Page 1: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/1.jpg)
Scaling Object Instance Retrieval
CS 682
![Page 2: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/2.jpg)
Previously
• Advanced Features • Feature Correspondences • Optical flow • Texture
• Scaling up matching using local features • More general image aligment problems
![Page 3: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/3.jpg)
4m
50 Thousand Images
Slides adapted from. D. Nister and H. Stewenius Scalable Matching Using Hierarchical Vocabulary Trees, CVPR 2006
![Page 4: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/4.jpg)
110,000,000 Images in 5.8 Seconds
![Page 5: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/5.jpg)
![Page 6: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/6.jpg)
Adding, Querying and Removing Images at full speed
Add Remove
Query
![Page 7: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/7.jpg)
Training and Addition are Separate
Common Approach Our approach
![Page 8: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/8.jpg)
Normalize patch
Detect patches [Mikojaczyk and Schmid ’02]
[Mata, Chum, Urban & Pajdla, ’02]
[Sivic & Zisserman, ’03]
Compute SIFT
descriptor [Lowe’99]
Slide credit: Josef Sivic
1. Feature extraction
![Page 9: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/9.jpg)
…
1. Feature extraction
![Page 10: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/10.jpg)
2. Learning the visual vocabulary
…
![Page 11: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/11.jpg)
2. Learning the visual vocabulary
Clustering
…
Slide credit: Josef Sivic
![Page 12: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/12.jpg)
2. Learning the visual vocabulary
Clustering
…
Slide credit: Josef Sivic
Visual vocabulary
![Page 13: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/13.jpg)
K-means clustering
• Want to minimize sum of squared Euclidean distances between points xi and their nearest cluster centers mk
Algorithm: • Randomly initialize K cluster centers • Iterate until convergence:
• Assign each data point to the nearest center • Recompute each cluster center as the mean of all points
assigned to it
∑ ∑ −=k
ki
ki mxMXDcluster
clusterinpoint
2)(),(
![Page 14: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/14.jpg)
From clustering to vector quantization
• Clustering is a common method for learning a visual vocabulary or codebook • Unsupervised learning process • Each cluster center produced by k-means becomes a
codevector • Codebook can be learned on separate training set • Provided the training set is sufficiently representative, the
codebook will be “universal”
• The codebook is used for quantizing features • A vector quantizer takes a feature vector and maps it to the
index of the nearest codevector in a codebook • Codebook = visual vocabulary • Codevector = visual word
![Page 15: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/15.jpg)
Image patch examples of visual words
Sivic et al. 2005
![Page 16: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/16.jpg)
Visual vocabularies: Issues
• How to choose vocabulary size? • Too small: visual words not representative of all patches • Too large: quantization artifacts, overfitting
• Generative or discriminative learning? • Computational efficiency
• Vocabulary trees (Nister & Stewenius, 2006)
![Page 17: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/17.jpg)
Hierarchical k-means
We have many, many of these features 100000 images ~1000 features per image If we can get repeatable, discriminative features, then recognition can scale to very large databases using the vocabulary tree and indexing approach • Quantize the feature descriptor space + efficient search • Flat k-means , Approximate Nearest Neightbour Methods • Hierarchical k-means - Nister&Stewenius [CVPR 2006] • Visual vocabulary trees
Slides from Nister & Stewenius 06
![Page 18: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/18.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 19: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/19.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 20: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/20.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 21: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/21.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 22: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/22.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 23: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/23.jpg)
Building Visual Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 24: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/24.jpg)
Slides from Nister & Stewenius 06
![Page 25: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/25.jpg)
Slides from Nister & Stewenius 06
![Page 26: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/26.jpg)
Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 27: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/27.jpg)
Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 28: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/28.jpg)
Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 29: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/29.jpg)
Vocabulary Tree
Slides from Nister & Stewenius 06
![Page 30: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/30.jpg)
Vocabulary Trees • Easy to add/remove images from the database • Suitable for incremental approach • Suitable for creating single generic vocabulary Approach • Extract descriptors from many/many images • Acquire enough statistics about the descriptor distribution • Run k-means hierarchically k- is the branching factor of the tree E.g. Branching factor of 10 and 6 levels – million leaves
Slides from Nister & Stewenius 06
![Page 31: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/31.jpg)
Vocabulary Trees
• Training phase – add images to the database • Extract descriptors – drop it down the tree • Each node has an inverted file index • Index to that image is added to all inverted files • When we want to query image • Pass each descriptor down the tree • Accumulate scores for each image in the database At each level do dot products total of dot products For leafs and integer descriptors we need bytes for 1M leaf nodes use 143 MB of memory
kkL DkL
kL
Slides from Nister & Stewenius 06
![Page 32: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/32.jpg)
Slides from Nister & Stewenius 06
![Page 33: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/33.jpg)
Slides from Nister & Stewenius 06
![Page 34: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/34.jpg)
Slides from Nister & Stewenius 06
![Page 35: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/35.jpg)
Slides from Nister & Stewenius 06
![Page 36: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/36.jpg)
Slides from Nister & Stewenius 06
![Page 37: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/37.jpg)
TF-IDF scoring
• TF-IDF term frequency – inverse document frequency • Used in the information retrieval and text mining • To evaluate how important is a word to document
• Importance depends on how many times the word appears in document – offset by number of occurrence of that word in the whole document corpus
![Page 38: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/38.jpg)
TF-IDF scoring
• TF-IDF term frequency – inverse document frequency • Number of occurrences of a word in a document / number of
occurrences of all words in the document • Number of documents / number of documents where term appears • High weight of a word/term is when it has high frequency and low
term document frequency
tfi,j =ni,j�k nk,j
idfi,j = log|D|
|{d : ti � d}||D|
|d : ti � d|
tfidfi,j = tfi,j � idfi
![Page 39: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/39.jpg)
Inverted file index • Idea – large number of images (documents) – how to find out
quickly which image does the word occur ?
• Forward index – list of words per document
• Inverted index – list of documents per word • Once the word is encountered – we can quickly figure out which
image it belongs to
Word 1 {1, 2 , 10, 12} Word 2 {0, 2, 100, 7, 18} Word 3 {5, 10, 12} …
![Page 40: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/40.jpg)
Size Matters Improves Retrieval
Improves Speed
Performance improves with the Size of the database
Here the results of particular object instance retrieval, database Of ~ 40,000 objects, real-time performance
Slides from Nister & Stewenius 06
![Page 41: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/41.jpg)
Alternative NN search
• K-d trees with best bin first • Split the space along different dimensions recursively • Each split has dimension and split value • Balance the tree and have small number of levels • Query time – locate the bin and and search nearby bins
![Page 42: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/42.jpg)
Nearest Neighbor Classifier
• Assign label of nearest training data point to each test data point
Voronoi partitioning of feature space for 2-category 2-D and 3-D data
from Duda et al.
Source: D. Lowe
![Page 43: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/43.jpg)
• For a new point, find the k closest points from training data • Labels of the k points “vote” to classify • Works well provided there is lots of data and the distance function is
good
K-Nearest Neighbors
k = 5
Source: D. Lowe
![Page 44: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/44.jpg)
Functions for comparing histograms
• L1 distance
• χ2 distance
• Quadratic distance (cross-bin)
∑=
−=N
iihihhhD
12121 |)()(|),(
Jan Puzicha, Yossi Rubner, Carlo Tomasi, Joachim M. Buhmann: Empirical Evaluation of Dissimilarity Measures for Color and Texture. ICCV 1999
( )∑= +
−=
N
i ihihihihhhD
1 21
221
21 )()()()(),(
∑ −=ji
ij jhihAhhD,
22121 ))()((),(
![Page 45: Scaling Object Instance Retrievalkosecka/cs682/lect-instance-retrieval.pdf · Scaling Object Instance Retrieval CS 682 . Previously • Advanced Features ... [Mikojaczyk and Schmid](https://reader036.vdocuments.mx/reader036/viewer/2022063003/5f72213bf2bdcc482e5208b5/html5/thumbnails/45.jpg)
Earth Mover’s Distance
• Each image is represented by a signature S consisting of a set of centers {mi } and weights {wi }
• Centers can be codewords from universal vocabulary, clusters of features in the image, or individual features (in which case quantization is not required)
• Earth Mover’s Distance has the form where the flows fij are given by the solution of a transportation problem
Y. Rubner, C. Tomasi, and L. Guibas: A Metric for Distributions with Applications to Image Databases. ICCV 1998
∑=ji ij
jiij
fmmdf
SSEMD,
2121
),(),(