feature tracking class 5
DESCRIPTION
Feature tracking Class 5. Read Section 4.1 of course notes http://www.cs.unc.edu/~marc/tutorial/node49.html Read Shi and Tomasi’s paper on good features to track http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf Read Lowe’s paper on SIFT features - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/1.jpg)
Feature trackingClass 5
Read Section 4.1 of course noteshttp://www.cs.unc.edu/~marc/tutorial/node49.html
Read Shi and Tomasi’s paper on good features to track
http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf
Read Lowe’s paper on SIFT featureshttp://www.unc.edu/courses/2004fall/comp/290/089/papers/Lowe_ijcv03.pdf
![Page 2: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/2.jpg)
Don’t forget: Assignment 1(due by next Tuesday before class)
• Find a camera• Calibration approach 1
• Build/use calibration grid (2 orthogonal planes)• Perform calibration using (a) DLT and (b) complete
gold standard algorithm (assume error only in images, model radial distortion, ok to click points by hand)
• Calibration approach 2• Build/use planar calibration pattern• Use Bouguet’s matlab calibration toolbox (≈Zhang’s
approach)http://www.vision.caltech.edu/bouguetj/calib_doc/
(or implement it yourself for extra points)• Compare results of approach 1(a),1(b) and 2• Make short report of findings and be ready to
discuss in class
![Page 3: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/3.jpg)
Single view metrology
• Allows to relate height of point to height of camera
![Page 4: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/4.jpg)
Single view metrology
• Allows to transfer point from one plane to another
![Page 5: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/5.jpg)
Feature trackingClass 5
Read Section 4.1 of course noteshttp://www.cs.unc.edu/~marc/tutorial/node49.html
Read Shi and Tomasi’s paper on good features to track
http://www.unc.edu/courses/2004fall/comp/290/089/papers/shi-tomasi-good-features-cvpr1994.pdf
Read Lowe’s paper on SIFT featureshttp://www.unc.edu/courses/2004fall/comp/290/089/papers/Lowe_ijcv03.pdf
![Page 6: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/6.jpg)
Feature matching vs. tracking
Extract features independently and then match by comparing descriptors
Extract features in first images and then try to find same feature back in next view
What is a good feature?
Image-to-image correspondences are key to passive triangulation-based 3D reconstruction
![Page 7: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/7.jpg)
Compare intensities pixel-by-pixel
Comparing image regions
I(x,y) I´(x,y)
Sum of Square Differences
Dissimilarity measures
![Page 8: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/8.jpg)
Compare intensities pixel-by-pixel
Comparing image regions
I(x,y) I´(x,y)
Zero-mean Normalized Cross Correlation
Similarity measures
![Page 9: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/9.jpg)
Feature points
• Required properties:• Well-defined
(i.e. neigboring points should all be different)
• Stable across views(i.e. same 3D point should be extracted as feature for neighboring viewpoints)
![Page 10: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/10.jpg)
Feature point extraction
homogeneous
edge
corner
Find points that differ as much as possible from all neighboring points
![Page 11: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/11.jpg)
Feature point extraction
• Approximate SSD for small displacement Δ• Image difference, square difference for
pixel• SSD for window
![Page 12: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/12.jpg)
Feature point extraction
homogeneous
edge
corner
Find points for which the following is maximum
i.e. maximize smallest eigenvalue of M
![Page 13: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/13.jpg)
Harris corner detector
• Only use local maxima, subpixel accuracy through second order surface fitting
• Select strongest features over whole image and over each tile (e.g. 1000/image, 2/tile)
• Use small local window:• Maximize „cornerness“:
![Page 14: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/14.jpg)
Simple matching• for each corner in image 1 find the corner
in image 2 that is most similar (using SSD or NCC) and vice-versa
• Only compare geometrically compatible points
• Keep mutual best matches
What transformations does this work for?
![Page 15: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/15.jpg)
Feature matching: example
0.96 -0.40
-0.16
-0.39
0.19
-0.05
0.75 -0.47
0.51 0.72
-0.18
-0.39
0.73 0.15 -0.75
-0.27
0.49 0.16 0.79 0.21
0.08 0.50 -0.45
0.28 0.99
1 5
24
3
1 5
24
3
What transformations does this work for? What level of transformation do we need?
![Page 16: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/16.jpg)
Wide baseline matching • Requirement to cope with larger
variations between images• Translation, rotation, scaling• Foreshortening• Non-diffuse reflections• Illumination
geometric transformations
photometric changes
![Page 17: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/17.jpg)
Wide-baseline matching example
(Tuytelaars and Van Gool BMVC 2000)
![Page 18: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/18.jpg)
Lowe’s SIFT features
Recover features with position, orientation and scale
(Lowe, ICCV99)
![Page 19: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/19.jpg)
Position
• Look for strong responses of DOG filter (Difference-Of-Gaussian)
• Only consider local maxima
![Page 20: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/20.jpg)
Scale• Look for strong responses of DOG
filter (Difference-Of-Gaussian) over scale space
• Only consider local maxima in both position and scale
• Fit quadratic around maxima for subpixel
![Page 21: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/21.jpg)
Orientation
• Create histogram of local gradient directions computed at selected scale
• Assign canonical orientation at peak of smoothed histogram
• Each key specifies stable 2D coordinates (x, y, scale, orientation) 0 2
![Page 22: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/22.jpg)
Minimum contrast and “cornerness”
![Page 23: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/23.jpg)
SIFT descriptor• Thresholded image gradients are sampled
over 16x16 array of locations in scale space• Create array of orientation histograms• 8 orientations x 4x4 histogram array = 128
dimensions
![Page 24: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/24.jpg)
![Page 25: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/25.jpg)
Matas et al.’s maximally stable regions
• Look for extremal regions
http://cmp.felk.cvut.cz/~matas/papers/matas-bmvc02.pdf
![Page 26: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/26.jpg)
Mikolaczyk and Schmid LoG Features
![Page 27: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/27.jpg)
Feature tracking
• Identify features and track them over video• Small difference between frames• potential large difference overall
• Standard approach: KLT (Kanade-Lukas-Tomasi)
![Page 28: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/28.jpg)
Good features to track
• Use same window in feature selection as for tracking itself
• Compute motion assuming it is small
Affine is also possible, but a bit harder (6x6 in stead of 2x2)
![Page 29: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/29.jpg)
Example
![Page 30: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/30.jpg)
Example
![Page 31: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/31.jpg)
Synthetic example
![Page 32: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/32.jpg)
Good features to keep tracking
Perform affine alignment between first and last frameStop tracking features with too large errors
![Page 33: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/33.jpg)
Live demo
• OpenCV (try it out!)LKdemo
![Page 34: Feature tracking Class 5](https://reader035.vdocuments.mx/reader035/viewer/2022062305/5681682e550346895dddcbbe/html5/thumbnails/34.jpg)
Next class: triangulation and reconstruction
C1m1
L1
m2
L2
M
C2
Triangulation- calibration- correspondences