patch descriptors - university of washingtonshapiro/ee596/notes/descriptors.pdf · patch...
TRANSCRIPT
![Page 1: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/1.jpg)
Patch Descriptors
CSE 576
Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi
![Page 2: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/2.jpg)
How can we find corresponding points?
![Page 3: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/3.jpg)
How can we find correspondences?
![Page 4: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/4.jpg)
How do we describe an image patch?
![Page 5: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/5.jpg)
How do we describe an image patch?
Patches with similar content should have similar descriptors.
![Page 6: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/6.jpg)
Raw patches as local descriptors
The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector. But this is very sensitive to even small shifts, rotations.
![Page 7: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/7.jpg)
SIFT descriptor Full version
• Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor
Adapted from slide by David Lowe
![Page 8: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/8.jpg)
Full version • Divide the 16x16 window into a 4x4 grid of cells (2x2 case shown below) • Compute an orientation histogram for each cell • 16 cells * 8 orientations = 128 dimensional descriptor • Threshold normalize the descriptor:
SIFT descriptor
Adapted from slide by David Lowe
0.2
such that:
![Page 9: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/9.jpg)
Properties of SIFT Extraordinarily robust matching technique
• Can handle changes in viewpoint – Up to about 30 degree out of plane rotation
• Can handle significant changes in illumination – Sometimes even day vs. night (below)
• Fast and efficient—can run in real time • Lots of code available
– http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT
![Page 10: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/10.jpg)
NASA Mars Rover images with SIFT feature matches Figure by Noah Snavely
Example
![Page 11: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/11.jpg)
Example: Object Recognition
Lowe, IJCV04
SIFT is extremely powerful for object instance recognition, especially for well-textured objects
![Page 12: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/12.jpg)
Example: Google Goggle
![Page 13: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/13.jpg)
panorama? • We need to match (align) images
![Page 14: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/14.jpg)
Matching with Features
•Detect feature points in both images
![Page 15: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/15.jpg)
Matching with Features
•Detect feature points in both images
•Find corresponding pairs
![Page 16: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/16.jpg)
Matching with Features
•Detect feature points in both images
•Find corresponding pairs
•Use these matching pairs to align images - the required mapping is called a homography.
![Page 17: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/17.jpg)
Automatic mosaicing
http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html
![Page 18: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/18.jpg)
Recognition of specific objects, scenes
Rothganger et al. 2003 Lowe 2002
Schmid and Mohr 1997 Sivic and Zisserman, 2003
Kristen Grauman
![Page 19: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/19.jpg)
When does SIFT fail?
Patches SIFT thought were the same but aren’t:
![Page 20: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/20.jpg)
Other methods: Daisy
SIFT
Daisy
Picking the best DAISY, S. Winder, G. Hua, M. Brown, CVPR 09
Circular gradient binning
![Page 21: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/21.jpg)
Other methods: SURF For computational efficiency only compute gradient histogram with 4 bins:
SURF: Speeded Up Robust Features Herbert Bay, Tinne Tuytelaars, and Luc Van Gool, ECCV 2006
![Page 22: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/22.jpg)
Other methods: BRIEF
Daisy
BRIEF: binary robust independent elementary features, Calonder, V Lepetit, C Strecha, ECCV 2010
Randomly sample pair of pixels a and b. 1 if a > b, else 0. Store binary vector.
![Page 23: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/23.jpg)
Feature distance How to define the difference between two features f1, f2?
• Simple approach is SSD(f1, f2) – sum of square differences between entries of the two descriptors – can give good scores to very ambiguous (bad) matches
I1 I2
f1 f2
![Page 24: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/24.jpg)
Feature distance How to define the difference between two features f1, f2?
• Better approach: ratio distance = SSD(f1, f2) / SSD(f1, f2’) – f2 is best SSD match to f1 in I2 – f2’ is 2nd best SSD match to f1 in I2
– gives large values (~1) for ambiguous matches
I1 I2
f1 f2 f2'
![Page 25: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/25.jpg)
Eliminating bad matches Throw out features with distance > threshold
• How to choose the threshold?
50 75
200
feature distance
false match
true match
![Page 26: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/26.jpg)
True/false positives The distance threshold affects performance
• True positives = # of detected matches that are correct – Suppose we want to maximize these—how to choose threshold?
• False positives = # of detected matches that are incorrect – Suppose we want to minimize these—how to choose threshold?
50 75
200
feature distance
false match
true match
![Page 27: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/27.jpg)
Local Descriptors: Shape Context
Count the number of points inside each bin, e.g.:
Count = 4
Count = 10 ...
Log-polar binning: more precision for nearby points, more flexibility for farther points.
Belongie & Malik, ICCV 2001 K. Grauman, B. Leibe
![Page 28: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/28.jpg)
Texture
• The texture features of a patch can be considered a descriptor.
; Varma & Zisserman, 2002, 2003;, 2003
![Page 29: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/29.jpg)
Bag-of-words models • Orderless document representation: frequencies of words
from a dictionary Salton & McGill (1983)
![Page 30: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/30.jpg)
Bag-of-words models
US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
![Page 31: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/31.jpg)
Bag-of-words models
US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
![Page 32: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/32.jpg)
Bag-of-words models
US Presidential Speeches Tag Cloud http://chir.ag/phernalia/preztags/
• Orderless document representation: frequencies of words from a dictionary Salton & McGill (1983)
![Page 33: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/33.jpg)
Bags of features for image classification 1. Extract features
![Page 34: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/34.jpg)
1. Extract features 2. Learn “visual vocabulary”
Bags of features for image classification
![Page 35: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/35.jpg)
1. Extract features 2. Learn “visual vocabulary” 3. Quantize features using visual vocabulary
Bags of features for image classification
![Page 36: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/36.jpg)
1. Extract features 2. Learn “visual vocabulary” 3. Quantize features using visual vocabulary 4. Represent images by frequencies of
“visual words”
Bags of features for image classification
![Page 37: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/37.jpg)
Texture representation
Universal texton dictionary
histogram
Julesz, 1981; Cula & Dana, 2001; Leung & Malik 2001; Mori, Belongie & Malik, 2001; Schmid 2001; Varma & Zisserman, 2002, 2003; Lazebnik, Schmid & Ponce, 2003
![Page 38: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/38.jpg)
• Regular grid • Vogel & Schiele, 2003 • Fei-Fei & Perona, 2005
• Interest point detector • Csurka et al. 2004 • Fei-Fei & Perona, 2005 • Sivic et al. 2005
1. Feature extraction
![Page 39: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/39.jpg)
Normalize patch
Detect patches [Mikojaczyk and Schmid ’02]
[Mata, Chum, Urban & Pajdla, ’02]
[Sivic & Zisserman, ’03]
Compute SIFT descriptor
[Lowe’99]
Slide credit: Josef Sivic
1. Feature extraction
![Page 40: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/40.jpg)
…
1. Feature extraction
Lots of feature descriptors for the whole image or set of images.
![Page 41: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/41.jpg)
2. Discovering the visual vocabulary
…
feature vector space
![Page 42: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/42.jpg)
2. Discovering the visual vocabulary
Clustering
…
Slide credit: Josef Sivic
![Page 43: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/43.jpg)
2. Discovering the visual vocabulary
Clustering
…
Slide credit: Josef Sivic
Visual vocabulary
![Page 44: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/44.jpg)
Clustering and vector quantization • Clustering is a common method for learning a visual
vocabulary or codebook • Unsupervised learning process • Each cluster center produced by k-means becomes a
codevector • Codebook can be learned on separate training set • Provided the training set is sufficiently representative, the
codebook will be “universal”
• The codebook is used for quantizing features • A vector quantizer takes a feature vector and maps it to the
index of the nearest codevector in a codebook • Codebook = visual vocabulary • Codevector = visual word
![Page 45: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/45.jpg)
Example visual vocabulary
Fei-Fei et al. 2005
![Page 46: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/46.jpg)
Example codebook
…
Source: B. Leibe
Appearance codebook
![Page 47: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/47.jpg)
Another codebook
Appearance codebook …
…
… …
…
Source: B. Leibe
![Page 48: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/48.jpg)
Visual vocabularies: Issues
• How to choose vocabulary size? • Too small: visual words not representative of all patches • Too large: quantization artifacts, overfitting
• Computational efficiency • Vocabulary trees
(Nister & Stewenius, 2006)
![Page 49: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/49.jpg)
3. Image representation: histogram of codewords
…..
freq
uenc
y
codewords
![Page 50: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/50.jpg)
Image classification • Given the bag-of-features representations of images from
different classes, learn a classifier using machine learning
![Page 51: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/51.jpg)
But what about layout?
All of these images have the same color histogram
![Page 52: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/52.jpg)
Spatial pyramid representation • Extension of a bag of features • Locally orderless representation at several levels of resolution
level 0
Lazebnik, Schmid & Ponce (CVPR 2006)
![Page 53: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/53.jpg)
level 0 level 1
Lazebnik, Schmid & Ponce (CVPR 2006)
Spatial pyramid representation • Extension of a bag of features • Locally orderless representation at several levels of resolution
![Page 54: Patch Descriptors - University of Washingtonshapiro/EE596/notes/Descriptors.pdf · Patch Descriptors CSE 576 . Many slides from Larry Zitnick, Steve Seitz, Ali Farhadi . ... SURF](https://reader033.vdocuments.mx/reader033/viewer/2022042106/5e85770047e02058d33135c7/html5/thumbnails/54.jpg)
level 0 level 1 level 2
Lazebnik, Schmid & Ponce (CVPR 2006)
Spatial pyramid representation • Extension of a bag of features • Locally orderless representation at several levels of resolution