AN UPDATE ON FEATURES
SURF, BRISK, ORB and FREAK
Stefan Haller ↦ [email protected]
Dresden, 12th January 2015
Outline
1. Introduction to Features
2. SURF – Speeded Up Robust Features
3. BRISK – Binary Robust Invariant Scalable Keypoints
4. ORB – Oriented FAST and Rotated BRIEF
5. FREAK – Fast Retina Keypoints
6. Comparison
7. References
8. Discussion
TU Dresden, 12th January 2015 An Update on Features Folie 2 von 51
Image Featuresachievement :
• detect image features: unique interest points
• describe image feature: sample region of image patcharound point
applications include:
• object recognition and tracking
• image matching and stitching
• robotic mapping and 3D modeling
important :
• repeatability and reliability
• good performance (description, matching)
• speed
TU Dresden, 12th January 2015 An Update on Features Folie 4 von 51
1st Step: Feature Detection
By Retardo(http://en.wikipedia.org/wiki/File:Corner.png)[Public domain], viaWikimedia Commons
TU Dresden, 12th January 2015 An Update on Features Folie 5 von 51
2nd Step: Feature Description
Bay, H., Tuytelaars, T., & VanGool, L. (2006). Surf: Speededup robust features. InComputer Vision–ECCV2006 (pp. 404-417). SpringerBerlin Heidelberg.
Lowe, D. G. (2004).Distinctive image featuresfrom scale-invariantkeypoints. Internationaljournal of computer vision,60(2), 91-110.
TU Dresden, 12th January 2015 An Update on Features Folie 6 von 51
History
Traditional (slower, accurate):
1999 Scale Invariant Feature Transform (Lowe)
2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)
Binary (faster, real time, smartphone, performance):
2010 Binary Robust Independent Elementary Features(Michael Calonder, et al.)
2011 Oriented FAST and Rotated BRIEF (Ethan Rublee et al.)
2011 Binary Robust Invariant Scalable Keypoints(Leutenegger, Chli, Siegwart)
2012 Fast Retina Keypoint (Alahi, Ortiz, Vandergheynst)
TU Dresden, 12th January 2015 An Update on Features Folie 7 von 51
History
Traditional (slower, accurate):
1999 Scale Invariant Feature Transform (Lowe)
2006 Speeded Up Robust Features (Bay, Tuytelaars, Van Gool)
Binary (faster, real time, smartphone, performance):
2010 Binary Robust Independent Elementary Features(Michael Calonder, et al.)
2011 Oriented FAST and Rotated BRIEF (Ethan Rublee et al.)
2011 Binary Robust Invariant Scalable Keypoints(Leutenegger, Chli, Siegwart)
2012 Fast Retina Keypoint (Alahi, Ortiz, Vandergheynst)
TU Dresden, 12th January 2015 An Update on Features Folie 7 von 51
Properties
• scale and rotation invariant (Upright SURF only scaleinvariant)
• faster computation and faster matching
• achieved by integral images and smaller feature vector size
• ideas of SIFT, but much more simplified
TU Dresden, 12th January 2015 An Update on Features Folie 9 von 51
Feature Detection: Fast-Hessian• based on Hessian matrix (Laplacian of Gaussian, LoG)• reminder: LoG approximated by Difference of Gaussian,
DoG• SURF uses “Fast-Hessian Detector”: approximation withbox filters
TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51
Feature Detection: Fast-Hessian• based on Hessian matrix (Laplacian of Gaussian, LoG)• reminder: LoG approximated by Difference of Gaussian,
DoG• SURF uses “Fast-Hessian Detector”: approximation withbox filters
TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51
Feature Detection: Fast-Hessian• based on Hessian matrix (Laplacian of Gaussian, LoG)• reminder: LoG approximated by Difference of Gaussian,
DoG• SURF uses “Fast-Hessian Detector”: approximation withbox filters
box filters→very fast for arbitrary size (integral imagemethod)
Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In ComputerVision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.
TU Dresden, 12th January 2015 An Update on Features Folie 10 von 51
Feature Detection: Scale invariance
• use of scale space pyramid
• box filters allow to blur base image of current octave
• parallel creation of scales for octave possible!
Lowe, D. G. (2004).Distinctive image featuresfrom scale-invariantkeypoints. Internationaljournal of computer vision,60(2), 91-110.
TU Dresden, 12th January 2015 An Update on Features Folie 11 von 51
Feature Description: OrientationAssignment
• (step is skipped for USURF)• consider circular region around image patch (size
dependes on scale)• calculate Haar wavelet responses in 𝑥 and 𝑦 direction
(integral images → 6 ops for any sampling point)
-1.5
-1.0
-0.5
0.0
0.5
1.0
1.5
0 1
“Haar wavelet”. Licensed under CC BY-SA 3.0 viaWikimedia Commons -http://commons.wikimedia.org/wiki/File:Haar_wavelet.svg
• representation in vector space• sliding window → sum up responses → choose largestvector
TU Dresden, 12th January 2015 An Update on Features Folie 12 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description: Dominant Direction
verticalresponse
horizontalresponse
TU Dresden, 12th January 2015 An Update on Features Folie 13 von 51
Feature Description Vector
• square region around IP (rotated)
• split into 4 × 4 subregion
• for each subregion: 5 × 5 regularly spaced sample points
• compute Haar wavelet response horiz. and vert.
• for each: sum of d𝑥, d𝑦, |d𝑥|, |d𝑦|
→ 4 × 4 region, respectively 4 dim. vector → 64 dim. vector
TU Dresden, 12th January 2015 An Update on Features Folie 14 von 51
Feature Description Vector
Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. In ComputerVision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.
TU Dresden, 12th January 2015 An Update on Features Folie 15 von 51
BRISK – Binary RobustInvariant Scalable
Keypoints
TU Dresden, 12th January 2015 An Update on Features Folie 16 von 51
Properties
• dramatically lower computational complexity
• assembly of bit-string vector (no gradients, etc!)
• general property of binary descriptors: use of Hammingdistance instead of Euclidean distance
• Hamming distance ≙ XOR both vectors and count “1”s→ boosted by SSE/AVX
• faster than SIFT/SURF, comparable performance
(binary descriptors) especially suited for:
• real time requirements
• low power devices
TU Dresden, 12th January 2015 An Update on Features Folie 17 von 51
Scale-Space Keypoint Detection
• extension of AGAST which is itself extension FAST
• invariant of scale
• estimates keypoint scale in continuous scale-space
TU Dresden, 12th January 2015 An Update on Features Folie 18 von 51
Keypoint Description
• composed of binary string, simple brightness comparisontests
• neighborhood sampling: deterministic, equally spaced,concentric circles
Leutenegger, S., Chli, M., & Siegwart,R. Y. (2011, November). BRISK:
Binary robust invariant scalablekeypoints. In Computer Vision(ICCV), 2011 IEEE International
Conference on (pp. 2548-2555). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 19 von 51
Long and Short Pairs𝒜 = {(𝐩𝑖, 𝐩𝑗) ∈ ℝ2 × ℝ2 ∣ 𝑖, 𝑗 ∈ ℕ, 𝑗 < 𝑖 < 𝑁}
𝒮 = {(𝐩𝑖, 𝐩𝑗) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖‖ < 𝛿𝑚𝑎𝑥} ⊆ 𝒜ℒ = {(𝐩𝑖, 𝐩𝑗) ∈ 𝒜 ∣ ‖𝐩𝑗 − 𝐩𝑖‖ > 𝛿𝑚𝑖𝑛} ⊆ 𝒜
Leutenegger, S., Chli, M., & Siegwart,R. Y. (2011, November). BRISK:
Binary robust invariant scalablekeypoints. In Computer Vision(ICCV), 2011 IEEE International
Conference on (pp. 2548-2555). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 20 von 51
Orientation Computation
• long distance pairs used• calculate local gradient between pairs• sum up all local gradients
𝐠(𝐩𝑖, 𝐩𝑗) = (𝐩𝑗 − 𝐩𝑖) ⋅ 𝐼(𝐩𝑗, 𝜎𝑗) − 𝐼(𝐩𝑖, 𝜎𝑖)‖𝐩𝑗 − 𝐩𝑖‖2
𝐠 = (𝑔𝑥𝑔𝑦
) = 1𝐿 ⋅ ∑
(𝐩𝑖,𝐩𝑗)∈ℒ𝐠(𝑝𝑖, 𝑝𝑗)
𝜃 = arctan2(𝑔𝑦, 𝑔𝑥)TU Dresden, 12th January 2015 An Update on Features Folie 21 von 51
Building the Descriptor
• binary descriptor: comparison of points with binaryoutcome
• short distance pairs used
𝑏 = {1 if 𝐼(𝐩𝛼𝑗 , 𝜎𝑗) > 𝐼(𝐩𝛼
𝑖 , 𝜎𝑖)0 otherwise
∀(𝐩𝛼𝑖 , 𝐩𝛼
𝑗 ) ∈ 𝒮
• Hamming distance for matching
• XOR two vectors and count resulting “1”s
TU Dresden, 12th January 2015 An Update on Features Folie 22 von 51
ORB – Oriented FASTand Rotated BRIEF
TU Dresden, 12th January 2015 An Update on Features Folie 23 von 51
Properties
• fast, computational efficient
• real time and low power device
• (all other properties of binary descriptors)
TU Dresden, 12th January 2015 An Update on Features Folie 24 von 51
Detector (oriented FAST)
• modified FAST: oriented FAST
• measures intensity between center pixel and those incircular ring around center
shortcomings of FAST approach:
1. no quality measure (“cornerness”)solution: use Harris cornerness measure
2. not scale invariantsolution: use scale pyramid (like SIFT)
3. not rotation invariantsolution: calculate intensity centroid
TU Dresden, 12th January 2015 An Update on Features Folie 25 von 51
Intensity Centroid (Rotation Invariance)
assumption: corner’s intensity is offset from center
∀ 𝑝, 𝑞 ∈ {0, 1}⏟⏟⏟⏟⏟binary selector
for x and ydirection
∶ 𝑚𝑝𝑞 = ∑𝑥,𝑦⏟
circularwindow
𝑥𝑝 𝑦𝑞⏟weighted
bycoordinate
𝐼(𝑥, 𝑦)⏟image
function
𝐶 = (𝑚10𝑚00
, 𝑚01𝑚00
)
dominant direction ⃗⃗⃗⃗ ⃗⃗ ⃗⃗⃗ ⃗⃗ ⃗𝑂𝐶 : 𝜃 = atan2(𝑚01, 𝑚10)
TU Dresden, 12th January 2015 An Update on Features Folie 26 von 51
Intensity Weighted Center of Mass
TU Dresden, 12th January 2015 An Update on Features Folie 27 von 51
Descriptor (modified BRIEF)
• binary descriptor, no sampling pattern
• 𝜏(𝑝; 𝑥, 𝑦) = {1 if 𝑝(𝑥) ≥ 𝑝(𝑦)0 otherwise
• feature vector 𝑓𝑛(𝑝) ≔ ∑1≤𝑖≤𝑛
2𝑖−1 𝜏(𝑝; 𝑥𝑖, 𝑦𝑖)
• vector length 𝑛 = 256 bit
TU Dresden, 12th January 2015 An Update on Features Folie 28 von 51
Learning of Pairs
• 256 bit, but thousands of possible pairs
• Which pairs provide best information for given patch?
• We want to maximize variance and thus minimizecorrelation.
• Paper suggests learning algorithm
• Input: reference image series / Output: static list of pairs
⌛ Not enough time to explain in detail→ Discussion afterwards?
TU Dresden, 12th January 2015 An Update on Features Folie 29 von 51
Properties and Motivation
• binary descriptor, no feature detector specified
• inspired by human visual system, precisely retina
• at same time enforcing low computational complexity
• tries to be faster and more robust than SIFT, SURF orBRISK
• (all other properties of binary descriptors)
TU Dresden, 12th January 2015 An Update on Features Folie 31 von 51
Biological Background• human retina extracts details from images using DoGs• differences are encoded into action potentials• several photo-receptors influence ganglion cell
Alahi, A., Ortiz, R., &Vandergheynst, P. (2012,June). Freak: Fast retinakeypoint. In ComputerVision and PatternRecognition (CVPR),2012 IEEE Conferenceon (pp. 510-517). Ieee.
TU Dresden, 12th January 2015 An Update on Features Folie 32 von 51
Biological Background• human retina extracts details from images using DoGs• differences are encoded into action potentials• several photo-receptors influence ganglion cell• size and dendritic field increases with radial distance
from fovea• spatial distribution of ganglion cells reduces exponentially
Alahi, A., Ortiz, R., &Vandergheynst, P. (2012,June). Freak: Fast retinakeypoint. In ComputerVision and PatternRecognition (CVPR), 2012IEEE Conference on (pp.510-517). Ieee.
TU Dresden, 12th January 2015 An Update on Features Folie 32 von 51
FREAK’s Sampling Pattern
• sampling pattern similar to retinal ganglion cells
• receptive fields are overlapping → redundancy
• increases performance, better discriminative power
Alahi, A., Ortiz, R., & Vandergheynst,P. (2012, June). Freak: Fast retina
keypoint. In Computer Vision andPattern Recognition (CVPR), 2012 IEEE
Conference on (pp. 510-517). Ieee.
TU Dresden, 12th January 2015 An Update on Features Folie 33 von 51
FREAK’s methods
⌛ skipped:
orientation: based on BRISK
sampling pairs: learningalgorithm based on ORB
the results are quite interesting:coarse-to-fine ordering isautomatically preferred!
Alahi, A., Ortiz, R., & Vandergheynst, P. (2012,June). Freak: Fast retina keypoint. In ComputerVision and Pattern Recognition (CVPR), 2012 IEEEConference on (pp. 510-517). Ieee.
TU Dresden, 12th January 2015 An Update on Features Folie 34 von 51
Saccadic Search• eyes perform discontinuous individual movements
(“saccades”)
• fovea captures high-resolution → high density• critical role during recognition and matching
• perifoveal area captures less detailed information• used to compile first estimates
“Szakkad” by Original file: SpooSpa. Derivative:Simon Viktória - Derivative work from File:Faceof SpooSpa.jpg. Licensed under CC BY-SA 2.0 viaWikimedia Commons -http://commons.wikimedia.org/wiki/File:Szakkad.jpg
TU Dresden, 12th January 2015 An Update on Features Folie 35 von 51
Saccadic Search
• FREAK performs several steps
• at first parse first 16 byte of descriptor → corresponds toouter coarse information
• ≥ 90% of candidates discarded with first 16 bytes
• use SIMD: comparing 16 byte as fast as 1 byte
TU Dresden, 12th January 2015 An Update on Features Folie 36 von 51
Summary
real valued detector descriptorSIFT LoG pyramid gradient valuesSURF box filter pyramid Haar wavelets
binary detector sample pattern orientation sampling pairsBRISK AGAST concentric circles gradient short pairsORB oFAST none moments learnedFREAK N/A overlapping con-
centric circlesgradient (pre-selected pairs)
learned
TU Dresden, 12th January 2015 An Update on Features Folie 38 von 51
Performance Evaluation
• source: “Evaluation of Local Detectors and Descriptors forFast Feature Matching”
• comparison of matching speed and precision/accuracy
• only publicly available implementations can be tested(→ OpenCV)
TU Dresden, 12th January 2015 An Update on Features Folie 39 von 51
Implementation Details
• matching: nearest neighborhood search (NN)1. kd-trees (geometric, partition of dimensions)
→ for real-valued descriptors
2. hashing (project similar data into same bucket, Hamming space)→ for binary descriptors
• ϵ-ANN: approximated NN
• image data set: standard graffiti image sequences
• 24× 3.47 GHz, 12 MB cache, Intel Xeon X5690, 99 GBRAM, x64 Ubuntu 10.04
• SSE 4.2 enabled (fast POPCNT)
• algorithms used with default parameters (except FAST)
TU Dresden, 12th January 2015 An Update on Features Folie 40 von 51
Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors forfast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.2681-2684). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 41 von 51
Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors forfast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.2681-2684). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 42 von 51
Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors forfast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.2681-2684). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 43 von 51
Performance Measures
recall = #true positive#true positive + #false negative
precision = #true positive#true positive + #false positive
TU Dresden, 12th January 2015 An Update on Features Folie 44 von 51
Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors and descriptors forfast feature matching. In Pattern Recognition (ICPR), 2012 21st International Conference on (pp.2681-2684). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 45 von 51
My Conclusion
• SIFT: slow + good quality
• BRISK: fast + good quality
• FREAK ≈ BRISK
• ORB: faster + littlebit less quality
TU Dresden, 12th January 2015 An Update on Features Folie 46 von 51
References (Primary)
📖 Bay, H., Tuytelaars, T., & Van Gool, L. (2006). Surf: Speeded up robust features. InComputer Vision–ECCV 2006 (pp. 404-417). Springer Berlin Heidelberg.
📖 Rublee, E., Rabaud, V., Konolige, K., & Bradski, G. (2011, November). ORB: an efficientalternative to SIFT or SURF. In Computer Vision (ICCV), 2011 IEEE InternationalConference on (pp. 2564-2571). IEEE.
📖 Leutenegger, S., Chli, M., & Siegwart, R. Y. (2011, November). BRISK: Binary robustinvariant scalable keypoints. In Computer Vision (ICCV), 2011 IEEE InternationalConference on (pp. 2548-2555). IEEE.
📖 Alahi, A., Ortiz, R., & Vandergheynst, P. (2012, June). Freak: Fast retina keypoint. InComputer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 510-517).Ieee.
📖 Miksik, O., & Mikolajczyk, K. (2012, November). Evaluation of local detectors anddescriptors for fast feature matching. In Pattern Recognition (ICPR), 2012 21stInternational Conference on (pp. 2681-2684). IEEE.
TU Dresden, 12th January 2015 An Update on Features Folie 48 von 51
References (Secondary)
📖 Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. Internationaljournal of computer vision, 60(2), 91-110.
📖 Rosten, E., & Drummond, T. (2006). Machine learning for high-speed corner detection. InComputer Vision–ECCV 2006 (pp. 430-443). Springer Berlin Heidelberg.
📖 Juan, L., & Gwun, O. (2009). A comparison of sift, pca-sift and surf. International Journal ofImage Processing (IJIP), 3(4), 143-152.
📖 Calonder, M., Lepetit, V., Strecha, C., & Fua, P. (2010). Brief: Binary robust independentelementary features. In Computer Vision–ECCV 2010 (pp. 778-792). Springer BerlinHeidelberg.
📖 Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors.In Image Processing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753
📖 Figat, J., Kornuta, T., & Kasprzak, W. (2014). Performance Evaluation of Binary Descriptorsof Local Features. In Computer Vision and Graphics (pp. 187-194). Springer InternationalPublishing.
TU Dresden, 12th January 2015 An Update on Features Folie 49 von 51
Discussion
• real valued vs. binary descriptor
• simplicity vs. complexity
• learning of optimal pairs for ORB/FREAK
• Is evaluation of paper exhaustive?
• real time?
• object tracking?
TU Dresden, 12th January 2015 An Update on Features Folie 51 von 51
Two Supplementary Evaluation PapersBekele, D., Teutsch, M., & Schuchert, T. (2013). Evalua-tion of binary keypoint descriptors. In Image Proces-sing (ICIP), 2013 20th IEEE International Conference (pp.3652-3656). doi: 10.1109/ICIP.2013.6738753
compares BRIEF, ORB, BRISK andFREAK
data sets: Oxford and Stanford MobileVisual Search (the last w/o groundtruth values)
conlusion: paper recommends BRISK
BRISK needs significant morecomputational effort compared to ORB
FREAK faster than BRISK, has lessmemory load, slightly less performance
SIFT has state-of-the-art performance
Figat, J., Kornuta, T., & Kasprzak, W. (2014). PerformanceEvaluation of Binary Descriptors of Local Features. InComputer Vision andGraphics (pp. 187-194). Springer In-ternational Publishing.
compares BRIEF, ORB, BRISK andFREAK; each with all kinds of detectors
data set: Oxford
conclusion: recommends ORB detectorwith BRISK descr. or FREAK descr.(depending on distortion type)
FREAK for viewpoint changes muchbetter than BRISK/ORB
SIFT has state-of-the-art performance
TU Dresden, 12th January 2015 An Update on Features Folie 57 von 51
Two Supplementary Evaluation Papers
Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In ImageProcessing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753
TU Dresden, 12th January 2015 An Update on Features Folie 58 von 51
Two Supplementary Evaluation Papers
Bekele, D., Teutsch, M., & Schuchert, T. (2013). Evaluation of binary keypoint descriptors. In ImageProcessing (ICIP), 2013 20th IEEE International Conference (pp. 3652-3656). doi:10.1109/ICIP.2013.6738753
TU Dresden, 12th January 2015 An Update on Features Folie 58 von 51