machine perception cis 580 - penn engineeringderpanis/steerable... · agenda • steerable filters...
TRANSCRIPT
Machine Perception CIS 580
Kostas Derpanis
February 8, 2012
Thursday, February 9, 2012
Agenda
• Steerable filters (recap)
• Harris corner detector
• Binomial filters
Thursday, February 9, 2012
Steerable filters
Definition: Steerable filters are a class of filters where a filter of arbitrary orientation is synthesized as a linear combination of a set of basis filters [Freeman and Adelson, 1991].
Thursday, February 9, 2012White - positive filter lobesBlack - negative filter lobes
Steerable filters
Definition: Steerable filters are a class of filters where a filter of arbitrary orientation is synthesized as a linear combination of a set of basis filters [Freeman and Adelson, 1991].
second derivative of Gaussian
Thursday, February 9, 2012White - positive filter lobesBlack - negative filter lobes
Steerable derivatives of Gaussian
Thursday, February 9, 2012
Steerable filter architecture
Inputimage
Basisfilterbank
Summingjunction
Adaptivelyfiltered image
Gainmaps
Steerable Filter Architecture
ki(!)
��
i
ki(θ)Bi
�∗ I ≡
�
i
ki(θ)IBi
Thursday, February 9, 2012
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20
0.2
0.4
0.6
0.8
1
Derivatives of Gaussian frequency spectrum
normalized magnitude spectrum
ω
n
Gn(x) ←→ (jω)nG(ω)
Thursday, February 9, 2012
Agenda
• Steerable filters (recap)
• Harris corner detector
• Binomial filters
Thursday, February 9, 2012
Motivation
?
“What stuff in one image matches with stuff in another?”
Thursday, February 9, 2012
Motivation
“What stuff in one image matches with stuff in another?”
Thursday, February 9, 2012
What makes a good feature?
• Repeatability• Same feature can be found in other
images despite geometric and photometric transformations.
• Saliency• Each feature is distinctive.
• Compactness and efficiency• Many fewer features than image
pixels.
• Locality• Feature occupies a relatively small area
of the image; robust to clutter and occlusion.
Thursday, February 9, 2012
Detectors
Moravec [Moravec ’77]Hessian [Beaudet ’78]Harris/Forstner [Harris ’88, Forstner ‘87]Laplacian [Lindeberg ‘98]DoG [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] FAST [Rosten et al. ‘2010]
Many Others…
Thursday, February 9, 2012
Detectors
Moravec [Moravec ’77]Hessian [Beaudet ’78]Harris/Forstner [Harris ’88, Forstner ‘87]Laplacian [Lindeberg ‘98]DoG [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] FAST [Rosten et al. ‘2010]
Many Others…
Thursday, February 9, 2012
Harris corner?
Thursday, February 9, 2012
Harris detector: Intuition
• Analyze local variation of signal.
• “Corner”: Shifting window in any direction yields a large change in appearance.
Thursday, February 9, 2012
• Analyze local variation of signal.
• “Corner”: Shifting window in any direction yields a large change in appearance.
Harris detector: Intuition
“flat” region:no change in all directions
Thursday, February 9, 2012
• Analyze local variation of signal.
• “Corner”: Shifting window in any direction yields a large change in appearance.
Harris detector: Intuition
“edge”:no change along the edge direction
“flat” region:no change in all directions
Thursday, February 9, 2012
• Analyze local variation of signal.
• “Corner”: Shifting window in any direction yields a large change in appearance.
Harris detector: Intuition
“edge”:no change along the edge direction
“corner”:significant change in all directions
“flat” region:no change in all directions
Thursday, February 9, 2012
Harris detector: Derivation
(∆x,∆y)Variation of intensity for the shift :
E(∆x,∆y) =�
x,y
w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2
Thursday, February 9, 2012
Harris detector: Derivation
For nearly constant patches:
E(∆x,∆y) ≈ 0
Variation of intensity for the shift :(∆x,∆y)
E(∆x,∆y) =�
x,y
w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2
Thursday, February 9, 2012
Harris detector: Derivation
For “edge” patches: E(∆x,∆y)
variation large along one orientation
(∆x,∆y)Variation of intensity for the shift :
E(∆x,∆y) =�
x,y
w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2
Thursday, February 9, 2012
Harris detector: Derivation
(∆x,∆y)
For “corner” patches: E(∆x,∆y)
variation large along two orientations
Variation of intensity for the shift :
E(∆x,∆y) =�
x,y
w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2
Thursday, February 9, 2012
Review: Taylor series
f(x+ u, y + v) = f(x, y) + fx(x, y)u+ fy(x, y)v
+1
2[fxx(x, y)u
2 + fxy(x, y)uv + fyy(x, y)v2]
+ h.o.t.
Thursday, February 9, 2012
Review: Taylor series
f(x+ u, y + v) = f(x, y) + fx(x, y)u+ fy(x, y)v
+1
2[fxx(x, y)u
2 + fxy(x, y)uv + fyy(x, y)v2]
+ h.o.t.
≈
Thursday, February 9, 2012
Harris detector: Derivation
E(∆x,∆y) =�
x,y
[I(x, y)− I(x+∆x, y +∆y)]2
Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience
Taylor series approximation --> simplify --> rewrite
Harris detector: Derivation
E(∆x,∆y) =�
x,y
[I(x, y)− I(x+∆x, y +∆y)]2
≈�
x,y
[I(x, y)− I(x, y)− Ix(x, y)∆x− Iy(x, y)∆y]2
Taylor series expansion
Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience
Taylor series approximation --> simplify --> rewrite
Harris detector: Derivation
E(∆x,∆y) =�
x,y
[I(x, y)− I(x+∆x, y +∆y)]2
≈�
x,y
[I(x, y)− I(x, y)− Ix(x, y)∆x− Iy(x, y)∆y]2
Taylor series expansion
Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience
Taylor series approximation --> simplify --> rewrite
Harris detector: Derivation
E(∆x,∆y) =�
x,y
[I(x, y)− I(x+∆x, y +∆y)]2
≈�
x,y
[I(x, y)− I(x, y)− Ix(x, y)∆x− Iy(x, y)∆y]2
=�
x,y
[Ix(x, y)2∆x2 + 2Ix(x, y)Iy(x, y)∆x∆y + Iy(x, y)
2∆y ]
Taylor series expansion
expand
Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience
Taylor series approximation --> simplify --> rewrite
Harris detector: Derivation
E(∆x,∆y) =�
x,y
[I(x, y)− I(x+∆x, y +∆y)]2
rewrite as matrix
≈�
x,y
[I(x, y)− I(x, y)− Ix(x, y)∆x− Iy(x, y)∆y]2
=�
x,y
[Ix(x, y)2∆x2 + 2Ix(x, y)Iy(x, y)∆x∆y + Iy(x, y)
2∆y ]
Taylor series expansion
expand
Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience
Taylor series approximation --> simplify --> rewrite
Harris detector: Derivation
E(∆x,∆y) =�∆x ∆y
�M
�∆x∆y
�.
M =�
x,y
w(x, y)
�I2x IxIyIxIy I2y
�.
where
captures the variation of the gradients within the local patchM
Thursday, February 9, 2012M is generally a positive definite matrix
Note: Windowing function reintroduced and products of derivatives NOT second derivativesM: second-order moment, covariance matrix or scatter matrix
Back to intuition
1. Treat gradient vectors as a set of 2D points, (Ix, Iy).
2. Compute scatter matrix/ellipse.
3. Analyze scatter matrix/ellipse shape.
Thursday, February 9, 2012
Intuition
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
“Flat” “Corner”“Edge”
Ix
Iy
notice distribution of gradients vary for different types of patches
Thursday, February 9, 2012
Intuition
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
−5 −4 −3 −2 −1 0 1 2 3 4 5−5
−4
−3
−2
−1
0
1
2
3
4
5
“Flat” “Corner”“Edge”
λ1 >> λ2λ1 λ2≈ λ1 λ2≈small large
Ix
Iy
eigenvalues, λ1 and λ1, of scatter matrix, M, quantify variation in the principle directions (i.e., eigenvectors of M) of the data
Thursday, February 9, 2012
Corner response function
r = detM− k(trace M)2
detM = λ1λ2 trace M = λ1 + λ2
where
? ?
k is empirically set constant: 0.04 - 0.06
Thursday, February 9, 2012
Corner response function
r = detM− k(trace M)2
detM = λ1λ2 trace M = λ1 + λ2
where
?
k is empirically set constant: 0.04 - 0.06
Thursday, February 9, 2012
Corner response function
r = detM− k(trace M)2
detM = λ1λ2 trace M = λ1 + λ2
where
k is empirically set constant: 0.04 - 0.06
Thursday, February 9, 2012
Classification of image patches
λ1
λ2
“Corner”
“Edge”
“Edge”
“Flat” region
Classification of image points using eigenvalues of (or r)M
Thursday, February 9, 2012
Classification of image patches
λ1
λ2
“Corner”
“Edge”
“Edge”
“Flat” region
Classification of image points using eigenvalues of (or r)M
E is almost constant in all directions
λ1 and λ2 are small|r| small
Thursday, February 9, 2012
Classification of image patches
λ1
λ2
“Corner”
“Edge”
“Edge”
“Flat” region
Classification of image points using eigenvalues of (or r)M
E increases about the vertical orientation
λ1 >> λ2
r < 0
E increases about the horizontal orientation
λ2 >> λ1
r < 0
E is almost constant in all directions
λ1 and λ2 are small|r| small
Thursday, February 9, 2012
Classification of image patches
λ1
λ2
“Corner”
“Edge”
“Edge”
“Flat” region
Classification of image points using eigenvalues of (or r)
E increases in all directionsλ1 λ2
λ1 and λ2 are large, r > 0 and |r| >> 0
≈
M
E increases about the vertical orientation
λ1 >> λ2
r < 0
E increases about the horizontal orientation
λ2 >> λ1
r < 0
E is almost constant in all directions
λ1 and λ2 are small|r| small
Thursday, February 9, 2012
Harris corner example
Thursday, February 9, 2012
Extension to video
Thursday, February 9, 2012Video exampleFeature detectors operating at multiple scales
Agenda
• Steerable filters (recap)
• Harris corner detector
• Binomial filters
Thursday, February 9, 2012
Binomial filter
• Class of smoothing filter ( Gaussian).
• Attenuates high-frequencies and retains low frequencies.
• Filter set generated by a single filter combined with itself.
≈
Thursday, February 9, 2012
Starting point
• Generating filter:
• Binomial filter set:
• By the Central-Limit Theorem, the shape of the Binomial filters tend to a Gaussian as R increases.
B1 =1
2
�1 1
�.
.
�BR =
1
2
�1 1
�∗ 1
2
�1 1
�∗ . . . ∗ 1
2
�1 1
��.
Thursday, February 9, 2012Notice that the binomial filters are closed under convolution.
Sum of iid random variables tend to a Gaussian (or normal distribution) as the number of random variables gets larger.
Sum of two or more independent variables is equivalent to the convolution of their densities.
Binomial filter: Spatial domain
n
0
Thursday, February 9, 2012
Binomial filter: Spatial domain
n
0
Thursday, February 9, 2012
Equivalence to ...
1 1 B1B2B3B4B5
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
B6 1 6 15 20 15 6 1 ...
Thursday, February 9, 2012
Equivalence to ...
1 1 B1B2B3B4B5
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
B6 1 6 15 20 15 6 1 ...
Thursday, February 9, 2012
Equivalence to ...
1 1 B1B2B3B4B5
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1
B6 1 6 15 20 15 6 1 ...
Pascal’s Triangle
.
�nr
�
.
nth rowrth element
Thursday, February 9, 2012
Binomial derivatives
1 1
1 2 1
1 3 3 1
1 4 6 4 1
1 5 10 10 5 1...
1
Thursday, February 9, 2012
-1 -2 0 2 1
-1 -3 -2 2 3 1
-1 -4 -5 0 5 4 1
-1 -1 1 1
-1 0 1
-1 1
Binomial derivatives
...
Thursday, February 9, 2012
B2 =�1 2 1
�
..
Frequency domain
DTFT:
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
B2 =�1 2 1
�
..
B(ω) =∞�
n=−∞B2[n]e
−iωn
..
Frequency domain
DTFT:
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
B2 =�1 2 1
�
..
B(ω) =∞�
n=−∞B2[n]e
−iωn
..=
∞�
n=−∞
�δ[n+ 1] 2δ[n] δ[n+ 1]
�e−iωn
..
+ +
Frequency domain
DTFT:
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
B2 =�1 2 1
�
..
B(ω) =∞�
n=−∞B2[n]e
−iωn
..=
∞�
n=−∞
�δ[n+ 1] 2δ[n] δ[n+ 1]
�e−iωn
..
+ +
Frequency domain
=∞�
n=−∞δ[n+ 1]e−iωn +
∞�
n=−∞2δ[n]e−iωn +
∞�
n=−∞δ[n+ 1]e−iωn
..
DTFT:
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
B2 =�1 2 1
�
..
B(ω) =∞�
n=−∞B2[n]e
−iωn
..=
∞�
n=−∞
�δ[n+ 1] 2δ[n] δ[n+ 1]
�e−iωn
..
+ +
Frequency domain
=∞�
n=−∞δ[n+ 1]e−iωn +
∞�
n=−∞2δ[n]e−iωn +
∞�
n=−∞δ[n+ 1]e−iωn
..
DTFT:
= 2 + e−iω + eiω
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
B2 =�1 2 1
�
..
B(ω) =∞�
n=−∞B2[n]e
−iωn
..=
∞�
n=−∞
�δ[n+ 1] 2δ[n] δ[n+ 1]
�e−iωn
..
+ +
Frequency domain
=∞�
n=−∞δ[n+ 1]e−iωn +
∞�
n=−∞2δ[n]e−iωn +
∞�
n=−∞δ[n+ 1]e−iωn
..
DTFT:
= 2 + 2 cos(ω) Euler’s formula
(not normalized)
Thursday, February 9, 2012Note: using i instead of j for complex number representation
Binomial filter frequency domain
ω
0
Thursday, February 9, 2012
Binomial filter frequency domain
ω
0
Thursday, February 9, 2012
2D Binomial filter
• 2D Binomial filters are constructed by 1D horizontal and vertical filters.
Example:
B2D2 = Bx
R ∗ByR
.
B2D2 =
1
4
�1 2 1
�∗ 1
4
121
=1
16
1 2 12 4 21 2 1
.
separable convolution
Thursday, February 9, 2012
2D Binomial filter smoothing example
Thursday, February 9, 2012
2D Binomial filter smoothing example
Thursday, February 9, 2012