9300 harris corners pkwy, charlotte, ncslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · the...
TRANSCRIPT
![Page 1: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/1.jpg)
Keypoint extraction: Corners
9300 Harris Corners Pkwy, Charlotte, NC
![Page 2: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/2.jpg)
Why extract keypoints?• Motivation: panorama stitching
• We have two images – how do we combine them?
![Page 3: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/3.jpg)
Why extract keypoints?• Motivation: panorama stitching
• We have two images – how do we combine them?
Step 1: extract keypointsStep 2: match keypoint features
![Page 4: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/4.jpg)
Why extract keypoints?• Motivation: panorama stitching
• We have two images – how do we combine them?
Step 1: extract keypointsStep 2: match keypoint featuresStep 3: align images
![Page 5: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/5.jpg)
Characteristics of good keypoints
• Compactness and efficiency• Many fewer keypoints than image pixels
• Saliency• Each keypoint is distinctive
• Locality• A keypoint occupies a relatively small area of the image; robust to
clutter and occlusion
• Repeatability• The same keypoint can be found in several images despite geometric
and photometric transformations
![Page 6: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/6.jpg)
Applications Keypoints are used for:
• Image alignment • 3D reconstruction• Motion tracking• Robot navigation• Database indexing and retrieval• Object recognition
![Page 7: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/7.jpg)
Corner detection: Basic idea
![Page 8: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/8.jpg)
• We should easily recognize the point by looking through a small window
• Shifting a window in any direction should give a large change in intensity
“edge”:no change along the edge direction
“corner”:significant change in all directions
“flat” region:no change in all directions
Corner detection: Basic idea
![Page 9: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/9.jpg)
Corner Detection: Derivation
Change in appearance of window W for the shift [u,v]:
I(x, y)E(u, v)
E(3,2)
E(u,v) = [I(x +u, y+ v)− I(x, y)]2(x,y)∈W∑
![Page 10: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/10.jpg)
Corner Detection: Derivation
I(x, y)E(u, v)
E(0,0)
Change in appearance of window W for the shift [u,v]:
E(u,v) = [I(x +u, y+ v)− I(x, y)]2(x,y)∈W∑
![Page 11: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/11.jpg)
Corner Detection: Derivation
We want to find out how this function behaves for small shifts
E(u, v)
Change in appearance of window W for the shift [u,v]:
E(u,v) = [I(x +u, y+ v)− I(x, y)]2(x,y)∈W∑
![Page 12: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/12.jpg)
Corner Detection: DerivationFirst-order Taylor approximation for small motions [u, v]:
Let’s plug this into E(u,v):
vIuIyxIvyuxI yx ++»++ ),(),(
E(u,v) = [I(x +u, y+ v)− I(x, y)]2(x,y)∈W∑
≈ [I(x, y)+ Ixu+ Iyv− I(x, y)]2
(x,y)∈W∑
= [Ixu+ Iyv]2
(x,y)∈W∑ = Ix
2u2 + 2IxIyuv+ Iy2v2
(x,y)∈W∑
![Page 13: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/13.jpg)
Corner Detection: Derivation
E(u,v) ≈ u2 Ix2
x,y∑ + 2uv IxIy
x,y∑ + v2 Iy
2
x,y∑
= u v#$
%&
Ix2
x,y∑ IxIy
x,y∑
IxIyx,y∑ Iy
2
x,y∑
#
$
''''
%
&
((((
uv
#
$'
%
&(
E(u,v) can be locally approximated by a quadratic surface:
In which directions does this surface have the fastest/slowest change?
E(u, v)
![Page 14: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/14.jpg)
Corner Detection: Derivation
E(u,v) ≈ u2 Ix2
x,y∑ + 2uv IxIy
x,y∑ + v2 Iy
2
x,y∑
= u v#$
%&
Ix2
x,y∑ IxIy
x,y∑
IxIyx,y∑ Iy
2
x,y∑
#
$
''''
%
&
((((
uv
#
$'
%
&(
E(u,v) can be locally approximated by a quadratic surface:
Second moment matrix M
![Page 15: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/15.jpg)
A horizontal “slice” of E(u, v) is given by the equation of an ellipse:
Interpreting the second moment matrix
const][ =úû
ùêë
évu
Mvu
![Page 16: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/16.jpg)
Consider the axis-aligned case (gradients are either horizontal or vertical):
Interpreting the second moment matrix
M =
Ix2
x,y∑ IxIy
x,y∑
IxIyx,y∑ Iy
2
x,y∑
"
#
$$$$
%
&
''''
= a 00 b
"
#$
%
&'
[u v] a 00 b
!
"#
$
%&
uv
!
"#
$
%&=1
au2 + bv2 =1u2
(a−1/2 )2+
v2
(b−1/2 )2=1
a-1/2
b-1/2
Major axisMin
or a
xis
![Page 17: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/17.jpg)
Consider the axis-aligned case (gradients are either horizontal or vertical):
Interpreting the second moment matrix
M =
Ix2
x,y∑ IxIy
x,y∑
IxIyx,y∑ Iy
2
x,y∑
"
#
$$$$
%
&
''''
= a 00 b
"
#$
%
&'
If either a or b is close to 0, then this is not a corner, so we want locations where both are large
![Page 18: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/18.jpg)
Interpreting the second moment matrix
RRM úû
ùêë
é= -
2
11
00l
l
The axis lengths of the ellipse are determined by the eigenvalues and the orientation is determined by R:
direction of the slowest change
direction of the fastest change
(lmax)-1/2(lmin)-1/2
In the general case, need to diagonalize M:
![Page 19: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/19.jpg)
Visualization of second moment matrices
![Page 20: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/20.jpg)
Visualization of second moment matrices
![Page 21: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/21.jpg)
Interpreting the eigenvalues
l1
l2
“Corner”l1 and l2 are large,l1 ~ l2;E increases in all directions
l1 and l2 are small;E is almost constant in all directions
“Edge” l1 >> l2
“Edge” l2 >> l1
“Flat” region
Classification of image points using eigenvalues of M:
![Page 22: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/22.jpg)
Corner response function
“Corner”R > 0
“Edge” R < 0
“Edge” R < 0
“Flat” region
|R| small
22121
2 )()(trace)det( llalla +-=-= MMR
α: constant (0.04 to 0.06)
![Page 23: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/23.jpg)
The Harris corner detector
1. Compute partial derivatives at each pixel2. Compute second moment matrix M in a
Gaussian window around each pixel:
C.Harris and M.Stephens, A Combined Corner and Edge Detector, Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
úúú
û
ù
êêê
ë
é=
åååå
yxy
yxyx
yxyx
yxx
IyxwIIyxw
IIyxwIyxwM
,
2
,
,,
2
),(),(
),(),(
![Page 24: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/24.jpg)
The Harris corner detector
1. Compute partial derivatives at each pixel
2. Compute second moment matrix M in a
Gaussian window around each pixel
3. Compute corner response function R
C.Harris and M.Stephens, A Combined Corner and Edge Detector, Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
![Page 25: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/25.jpg)
Harris Detector: Steps
![Page 26: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/26.jpg)
Harris Detector: StepsCompute corner response R
![Page 27: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/27.jpg)
The Harris corner detector
1. Compute partial derivatives at each pixel2. Compute second moment matrix M in a
Gaussian window around each pixel 3. Compute corner response function R4. Threshold R5. Find local maxima of response function
(nonmaximum suppression)
C.Harris and M.Stephens, A Combined Corner and Edge Detector, Proceedings of the 4th Alvey Vision Conference: pages 147—151, 1988.
![Page 28: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/28.jpg)
Harris Detector: StepsFind points with large corner response: R > threshold
![Page 29: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/29.jpg)
Harris Detector: StepsTake only the points of local maxima of R
![Page 30: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/30.jpg)
Harris Detector: Steps
![Page 31: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/31.jpg)
Robustness of corner features• What happens to corner features when the image undergoes
geometric or photometric transformations?
![Page 32: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/32.jpg)
Affine intensity change
• Only derivatives are used, so invariant to intensity shift I ® I + b
• Intensity scaling: I ® a I
R
x (image coordinate)
threshold
R
x (image coordinate)
Partially invariant to affine intensity change
I ® a I + b
![Page 33: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/33.jpg)
Image translation
• Derivatives and window function are shift-invariant
Corner location is covariant w.r.t. translation
![Page 34: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/34.jpg)
Image rotation
Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same
Corner location is covariant w.r.t. rotation
![Page 35: 9300 Harris Corners Pkwy, Charlotte, NCslazebni.cs.illinois.edu/spring19/lec08_corner.pdf · The Harris corner detector 1. Compute partial derivatives at each pixel 2. Compute second](https://reader034.vdocuments.mx/reader034/viewer/2022042512/5f8f9189d76661560d66cf45/html5/thumbnails/35.jpg)
Scaling
All points will be classified as edges
Corner
Corner location is not covariant w.r.t. scaling!