dense correspondences across scenes and scales tal hassner the open university of israel cvpr’14...
TRANSCRIPT
![Page 1: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/1.jpg)
Dense correspondences across scenes and scales
Tal HassnerThe Open University of Israel
CVPR’14 Tutorial on
Dense Image Correspondences for Computer Vision
![Page 2: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/2.jpg)
Tal HassnerDense correspondences across scenes and scales
Matching Pixels
Invariant detectors + robust descriptors +
matching
In different views, scales, scenes, etc.
![Page 3: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/3.jpg)
Tal HassnerDense correspondences across scenes and scalesSource: Szeliski’s book
Observation:
Invariant detectors require dominant scales
BUTMost pixels do not have such
scales
![Page 4: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/4.jpg)
Tal HassnerDense correspondences across scenes and scales
Observation:
Invariant detectors require dominant scales
BUTMost pixels do not have such
scales
But what happens if we want dense
matches with scale differences?
Source: Szeliski’s book
![Page 5: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/5.jpg)
Tal HassnerDense correspondences across scenes and scales
Solution 1:
Ignore scale differences – Dense-SIFT
Dense matching with scale differences
![Page 6: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/6.jpg)
Tal HassnerDense correspondences across scenes and scales
Dense SIFT (DSIFT)
Arbitrary scale selection
• A. Vedaldi and B. Fulkerson, VLFeat: An open and portable library of computer vision algorithms, in Proc. int. conf. on Multimedia (ICMM), 2010
![Page 7: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/7.jpg)
Tal HassnerDense correspondences across scenes and scales
SIFT-Flow
Left photo Right photo Left warped onto Right
“The good”: Dense flow between different scenes!
• C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. Freeman, SIFT flow: dense correspondence across different scenes, in European Conf. Comput. Vision (ECCV), 2008
• C. Liu, J. Yuen, and A. Torralba, SIFT flow: Dense correspondence across scenes and its applications, Trans. Pattern Anal. Mach. Intell. (TPAMI), vol. 33, no. 5, pp. 978–994, 2011
![Page 8: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/8.jpg)
Tal HassnerDense correspondences across scenes and scales
SIFT-Flow
Left photo Right photo Left warped onto Right
“The bad”: Fails when matching different scales
• C. Liu, J. Yuen, A. Torralba, J. Sivic, and W. Freeman, SIFT flow: dense correspondence across different scenes, in European Conf. Comput. Vision (ECCV), 2008
• C. Liu, J. Yuen, and A. Torralba, SIFT flow: Dense correspondence across scenes and its applications, Trans. Pattern Anal. Mach. Intell. (TPAMI), vol. 33, no. 5, pp. 978–994, 2011
![Page 9: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/9.jpg)
Tal HassnerDense correspondences across scenes and scales
What’s happening?
20% 50% 80%
This is what happens when one image is zoomed!!!
…yet remains
robust even until
20% scale errors
![Page 10: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/10.jpg)
Tal HassnerDense correspondences across scenes and scales
Solution 2:
Multi-scale descriptors
Dense matching with scale differences
Scale Invariant Descriptors (SID) [Kokkinos and Yuille’08]
Scale-Less SIFT (SLS) [Hassner, Mayzels, Zelnik-Manor’12]
• Kokkinos and Yuille, Scale Invariance without Scale Selection, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2008
![Page 11: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/11.jpg)
Tal HassnerDense correspondences across scenes and scales
SID: Log-Polar sampling
𝑟
𝜃
![Page 12: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/12.jpg)
Tal HassnerDense correspondences across scenes and scales
SID: Rotation + scale -> translation
𝑟
𝜃
𝑟
𝜃
𝑟
𝜃
![Page 13: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/13.jpg)
Tal HassnerDense correspondences across scenes and scales
SID: Translation invarianceAbsolute of the Discrete-Time Fourier Transform
![Page 14: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/14.jpg)
Tal HassnerDense correspondences across scenes and scales
SID-FlowLeft Right
DSIFT SID
![Page 15: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/15.jpg)
Tal HassnerDense correspondences across scenes and scales
SID-Flow
Left Right
DSIFT SID
![Page 16: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/16.jpg)
Tal HassnerDense correspondences across scenes and scales
Solution 2:
Multi-scale descriptors
Dense matching with scale differences
Scale Invariant Descriptors (SID) [Kokkinos and Yuille’08]
Scale-Less SIFT (SLS) [Hassner, Mayzels, Zelnik-Manor’12]
• T. Hassner, V. Mayzels, and L. Zelnik-Manor, On SIFTs and their Scales, IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2012
![Page 17: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/17.jpg)
Tal HassnerDense correspondences across scenes and scales
SIFTs at multiple scales
1, ,
k h h
1ˆ , , bH h h
Compute basis (e.g., PCA)
This low-dim subspace reflects SIFT behavior through scales at a single pixel
![Page 18: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/18.jpg)
Tal HassnerDense correspondences across scenes and scales
MatchingUse subspace to subspace distance:
2
2ˆ ˆ( , ) ( , ) sindist dist p qp q H H θ
![Page 19: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/19.jpg)
Tal HassnerDense correspondences across scenes and scales
To Illustrate
…if SIFTs were 2D
h𝜎𝑖
′h𝜎𝑖
❑
Comparing DSIFTs (single scales)
![Page 20: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/20.jpg)
Tal HassnerDense correspondences across scenes and scales
To Illustrate
…if SIFTs were 2D
kh
1'h
1h
'kh
h𝜎𝑖
′h𝜎𝑖
❑
Comparing SIFTs at multiple scales
![Page 21: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/21.jpg)
Tal HassnerDense correspondences across scenes and scales
To Illustrate
kh
1'h
1h
'kh
θ
Comparing subspaces of SIFTs from multiple scales!
![Page 22: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/22.jpg)
Tal HassnerDense correspondences across scenes and scales
The Scale-Less SIFT (SLS)
Map these subspaces to points!
11 2212 1 23, , , , , ,
2 2 2DD
D
a a aSLS Vec a a a
pp A
ˆ ˆ Tp p pA H H
For each pixel p
[Basri, Hassner, Zelnik-Manor, CVPR’07, ICCVw’09, TPAMI’11]
2 2 ˆ ˆ( ) ( ) ,p qSLS SLS dist p q H H
![Page 23: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/23.jpg)
Tal HassnerDense correspondences across scenes and scales
The Scale-Less SIFT (SLS)
Map these subspaces to points!
11 2212 1 23, , , , , ,
2 2 2DD
D
a a aSLS Vec a a a
pp A
ˆ ˆ Tp p pA H H
For each pixel p
[Basri, Hassner, Zelnik-Manor, CVPR’07, ICCVw’09, TPAMI’11]
2 2 ˆ ˆ( ) ( ) ,p qSLS SLS dist p q H H
A point representation for the subspace spanning
SIFT’s behavior in scales!!!
![Page 24: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/24.jpg)
Tal HassnerDense correspondences across scenes and scales
SLS-FlowUsing SIFT-Flow to compute the flow
LeftPhoto
Right Photo
DSIFT SID [Kokkinos & Yuille, CVPR’08] Our SLS
![Page 25: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/25.jpg)
Tal HassnerDense correspondences across scenes and scales
Solution 3:
Scale-space sift flow
Dense matching with scale differences
• W. Qiu, X. Wang, X. Bai, A. Yuille, and Z. Tu, Scale-space sift flow, in Proc. Winter Conf. on Applications of Comput. Vision. IEEE, 2014
Previous talk!
![Page 26: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/26.jpg)
Tal HassnerDense correspondences across scenes and scales
Solution 4:
Scale propagation
Dense matching with scale differences
• M. Tau, T. Hassner, “Dense Correspondences Across Scenes and Scales”, arXiv:1406.6323 (Available online from: http://arxiv.org/abs/1406.6323) Longer version in submission. Please see http://www.openu.ac.il/home/hassner/publications.html for updates.
![Page 27: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/27.jpg)
Tal HassnerDense correspondences across scenes and scales
Similar Pixels -> Similar Scales
Only 0.1% of pixels selected by multi-scale feature detector
![Page 28: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/28.jpg)
Tal HassnerDense correspondences across scenes and scales
Similar Pixels -> Similar Scales
Scales at neighboring pixels likely to be very similar
![Page 29: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/29.jpg)
Tal HassnerDense correspondences across scenes and scales
Similar Pixels -> Similar Scales
Propagate scales from detected points to neighbors!
![Page 30: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/30.jpg)
Tal HassnerDense correspondences across scenes and scales
Global cost of scale assignmentC
“…the scale at each pixel p should be close to the weighted average of its neighbors q”
Constrained by scales assigned by feature detector
Large, sparse system of equations with efficient solvers
![Page 31: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/31.jpg)
Tal HassnerDense correspondences across scenes and scales
To illustrate
Problem: Many scales do not matchSolution: Propagate scales only from
corresponding points!
![Page 32: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/32.jpg)
Tal HassnerDense correspondences across scenes and scales
Space and run-timeRepresentation Dim. SIFT-Flow time
![Page 33: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/33.jpg)
Tal HassnerDense correspondences across scenes and scales
Space and run-timeRepresentation Dim. SIFT-Flow time
DSIFT 128D 0.8 sec
SID 3,328D 5 sec.
SLS 8,256D 13 sec.
Proposed 128D 0.8 sec
* Measured on 78 x 52 pixel images
* Propagation required 0.06 sec.
![Page 34: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/34.jpg)
Tal HassnerDense correspondences across scenes and scales
QualitativeSource Target DSIFT SID SLS This
![Page 35: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/35.jpg)
Tal HassnerDense correspondences across scenes and scales
Quantitative
…in the paper
but ~SotA!
![Page 36: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/36.jpg)
Tal HassnerDense correspondences across scenes and scales
What we saw
Dense matching, even when scenes and scales are different
![Page 37: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/37.jpg)
Tal HassnerDense correspondences across scenes and scales
Thank you!
www.openu.ac.il/home/hassner
![Page 38: Dense correspondences across scenes and scales Tal Hassner The Open University of Israel CVPR’14 Tutorial on Dense Image Correspondences for Computer Vision](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649ea95503460f94bacd47/html5/thumbnails/38.jpg)
Tal HassnerDense correspondences across scenes and scales
Some resources• SIFT-Flow
– http://people.csail.mit.edu/celiu/SIFTflow/
• DSIFT (vlfeat)– http://www.vlfeat.org/
• SID– http://vision.mas.ecp.fr/Personnel/iasonas/code.html
• SLS– http://www.openu.ac.il/home/hassner/projects/siftscales/
• Scale propagation– Code coming soon! (see my webpage for updates)
• Me!– http://www.openu.ac.il/home/hassner– [email protected]