computer vision: filtering
TRANSCRIPT
![Page 1: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/1.jpg)
Computer Vision: Filtering
Raquel Urtasun
TTI Chicago
Jan 10, 2013
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 1 / 82
![Page 2: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/2.jpg)
Today’s lecture ...
Image formation
Image Filtering
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 2 / 82
![Page 3: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/3.jpg)
Readings
Chapter 2 and 3 of Rich Szeliski’s book
Available online here
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 3 / 82
![Page 4: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/4.jpg)
How is an image created?
The image formation process that produced a particular image depends on
lighting conditions
scene geometry
surface properties
camera optics
[Source: R. Szeliski]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 4 / 82
![Page 5: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/5.jpg)
Image formation
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 5 / 82
![Page 6: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/6.jpg)
What is an image?
!"#"$%&'(%)*+%'
,-*'./*'
0*1&&'23456'37'$-*6*'"7'$-"6'4&%66'
[Source: A. Efros]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 6 / 82
![Page 7: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/7.jpg)
From photons to RGB values
Sample the 2D space on a regular grid.
Quantize each sample, i.e., the photons arriving at each active cell areintegrated and then digitized.
[Source: D. Hoiem]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 7 / 82
![Page 8: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/8.jpg)
What is an image?
A grid (matrix) of intensity values
!"
#$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" #$$" #%" %" #$$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" #$$" &$" &$" &$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" &$" '$" '$" &$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" '(" )#&" )*$" )&$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" )#&" )*$" )&$" )&$" )&$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" )#&" )*$" #%%" #%%" )&$" )&$" '$" #$$" #$$" #$$"
#$$" #$$" )#&" )*$" #%%" #%%" )&$" )&$" '$" *&" #$$" #$$"
#$$" #$$" )#&" )*$" )*$" )&$" )#&" )#&" '$" *&" #$$" #$$"
#$$" #$$" &*" )#&" )#&" )#&" '$" '$" '$" *&" #$$" #$$"
#$$" #$$" #$$" &*" &*" &*" &*" &*" &*" #$$" #$$" #$$"
#$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$"
#$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$" #$$"
Common to use one byte per value: 0=black, 255=white)
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 8 / 82
![Page 9: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/9.jpg)
What is an image?
We can think of a (grayscale) image as a function f : <2 → < giving theintensity at position (x , y)
x
y
f (x, y)
A digital image is a discrete (sampled, quantized) version of this function
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 9 / 82
![Page 10: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/10.jpg)
Image Transformations
As with any function, we can apply operators to an image
!g (x,y) = f (x,y) + 20 !g (x,y) = f (-x,y)
We’ll talk about special kinds of operators, correlation and convolution(linear filtering)
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 10 / 82
![Page 11: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/11.jpg)
Filtering
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 11 / 82
![Page 12: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/12.jpg)
Question: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Take lots of images and average them!
What’s the next best thing?
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82
![Page 13: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/13.jpg)
Question: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Take lots of images and average them!
What’s the next best thing?
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82
![Page 14: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/14.jpg)
Question: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Take lots of images and average them!
What’s the next best thing?
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82
![Page 15: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/15.jpg)
Question: Noise reduction
Given a camera and a still scene, how can you reduce noise?
Take lots of images and average them!
What’s the next best thing?
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 12 / 82
![Page 16: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/16.jpg)
Image filtering
Modify the pixels in an image based on some function of a localneighborhood of each pixel
!" #"$"
#" %"#"
!" &"#'"
()*+,"-.+/0"1+2+"
%"
3)1-401"-.+/0"1+2+"
5).0"678*9)8"
[Source: L. Zhang]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 13 / 82
![Page 17: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/17.jpg)
Applications of Filtering
Enhance an image, e.g., denoise, resize.
Extract information, e.g., texture, edges.
Detect patterns, e.g., template matching.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82
![Page 18: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/18.jpg)
Applications of Filtering
Enhance an image, e.g., denoise, resize.
Extract information, e.g., texture, edges.
Detect patterns, e.g., template matching.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82
![Page 19: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/19.jpg)
Applications of Filtering
Enhance an image, e.g., denoise, resize.
Extract information, e.g., texture, edges.
Detect patterns, e.g., template matching.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 14 / 82
![Page 20: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/20.jpg)
Noise reduction
Simplest thing: replace each pixel by the average of its neighbors.
This assumes that neighboring pixels are similar, and the noise to beindependent from pixel to pixel.
[Source: S. Marschner]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82
![Page 21: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/21.jpg)
Noise reduction
Simplest thing: replace each pixel by the average of its neighbors.
This assumes that neighboring pixels are similar, and the noise to beindependent from pixel to pixel.
[Source: S. Marschner]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82
![Page 22: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/22.jpg)
Noise reduction
Simplest thing: replace each pixel by the average of its neighbors.
This assumes that neighboring pixels are similar, and the noise to beindependent from pixel to pixel.
[Source: S. Marschner]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 15 / 82
![Page 23: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/23.jpg)
Noise reduction
Simplest thing: replace each pixel by the average of its neighbors
This assumes that neighboring pixels are similar, and the noise to beindependent from pixel to pixel.
Moving average in 1D: [1, 1, 1, 1, 1]/5
[Source: S. Marschner]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 16 / 82
![Page 24: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/24.jpg)
Noise reduction
Simpler thing: replace each pixel by the average of its neighbors
This assumes that neighboring pixels are similar, and the noise to beindependent from pixel to pixel.
Non-uniform weights [1, 4, 6, 4, 1] / 16
[Source: S. Marschner]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 17 / 82
![Page 25: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/25.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 26: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/26.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 27: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/27.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 28: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/28.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 29: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/29.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 30: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/30.jpg)
Moving Average in 2D
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 18 / 82
![Page 31: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/31.jpg)
Linear Filtering: Correlation
Involves weighted combinations of pixels in small neighborhoods.
The output pixels value is determined as a weighted sum of input pixel values
g(i , j) =∑k,l
f (i + k , j + l)h(k , l)
The entries of the weight kernel or mask h(k, l) are often called the filtercoefficients.
This operator is the correlation operator
g = f ⊗ h
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82
![Page 32: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/32.jpg)
Linear Filtering: Correlation
Involves weighted combinations of pixels in small neighborhoods.
The output pixels value is determined as a weighted sum of input pixel values
g(i , j) =∑k,l
f (i + k , j + l)h(k , l)
The entries of the weight kernel or mask h(k , l) are often called the filtercoefficients.
This operator is the correlation operator
g = f ⊗ h
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82
![Page 33: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/33.jpg)
Linear Filtering: Correlation
Involves weighted combinations of pixels in small neighborhoods.
The output pixels value is determined as a weighted sum of input pixel values
g(i , j) =∑k,l
f (i + k , j + l)h(k , l)
The entries of the weight kernel or mask h(k , l) are often called the filtercoefficients.
This operator is the correlation operator
g = f ⊗ h
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82
![Page 34: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/34.jpg)
Linear Filtering: Correlation
Involves weighted combinations of pixels in small neighborhoods.
The output pixels value is determined as a weighted sum of input pixel values
g(i , j) =∑k,l
f (i + k , j + l)h(k , l)
The entries of the weight kernel or mask h(k , l) are often called the filtercoefficients.
This operator is the correlation operator
g = f ⊗ h
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 19 / 82
![Page 35: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/35.jpg)
Smoothing by averaging
What if the filter size was 5 x 5 instead of 3 x 3?
[Source: K. Graumann]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 20 / 82
![Page 36: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/36.jpg)
Gaussian filter
What if we want nearest neighboring pixels to have the most influence onthe output?
Removes high-frequency components from the image (low-pass filter).
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 21 / 82
![Page 37: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/37.jpg)
Smoothing with a Gaussian
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 22 / 82
![Page 38: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/38.jpg)
Mean vs Gaussian
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 23 / 82
![Page 39: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/39.jpg)
Gaussian filter: Parameters
Size of kernel or mask: Gaussian function has infinite support, but discretefilters use finite kernels.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 24 / 82
![Page 40: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/40.jpg)
Gaussian filter: Parameters
Variance of the Gaussian: determines extent of smoothing.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 25 / 82
![Page 41: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/41.jpg)
Gaussian filter: Parameters
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 26 / 82
![Page 42: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/42.jpg)
Is this the most general Gaussian?
No, the most general form for x ∈ <d
N (x; µ,Σ) =1
(2π)d/2|Σ|1/2exp
(−1
2(x− µ)TΣ−1(x− µ)
)
But the simplified version is typically use for filtering.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 27 / 82
![Page 43: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/43.jpg)
Properties of the Smoothing
All values are positive.
They all sum to 1.
Amount of smoothing proportional to mask size.
Remove high-frequency components; low-pass filter.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 28 / 82
![Page 44: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/44.jpg)
Properties of the Smoothing
All values are positive.
They all sum to 1.
Amount of smoothing proportional to mask size.
Remove high-frequency components; low-pass filter.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 28 / 82
![Page 45: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/45.jpg)
Properties of the Smoothing
All values are positive.
They all sum to 1.
Amount of smoothing proportional to mask size.
Remove high-frequency components; low-pass filter.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 28 / 82
![Page 46: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/46.jpg)
Properties of the Smoothing
All values are positive.
They all sum to 1.
Amount of smoothing proportional to mask size.
Remove high-frequency components; low-pass filter.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 28 / 82
![Page 47: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/47.jpg)
Example of Correlation
What is the result of filtering the impulse signal (image) F with the arbitrarykernel H?
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 29 / 82
![Page 48: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/48.jpg)
Convolution
Convolution operator
g(i , j) =∑k,l
f (i − k , j − l)h(k , l) =∑k,l
f (k , l)h(i − k, j − l) = f ∗ h
and h is then called the impulse response function.
Equivalent to flip the filter in both dimensions (bottom to top, right to left)and apply cross-correlation.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 30 / 82
![Page 49: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/49.jpg)
Convolution
Convolution operator
g(i , j) =∑k,l
f (i − k , j − l)h(k , l) =∑k,l
f (k , l)h(i − k, j − l) = f ∗ h
and h is then called the impulse response function.
Equivalent to flip the filter in both dimensions (bottom to top, right to left)and apply cross-correlation.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 30 / 82
![Page 50: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/50.jpg)
Matrix form
Correlation and convolution can both be written as a matrix-vector multiply,if we first convert the two-dimensional images f (i , j) and g(i , j) intoraster-ordered vectors f and g
g = Hf
with H a sparse matrix.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 31 / 82
![Page 51: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/51.jpg)
Correlation vs Convolution
Convolution
g(i , j) =∑k,l
f (i − k , j − l)h(k, l)
G = H ∗ F
Correlation
g(i , j) =∑k,l
f (i + k , j + l)h(k, l)
G = H ⊗ F
For a Gaussian or box filter, how will the outputs differ?
If the input is an impulse signal, how will the outputs differ? h ∗ δ?, andh ⊗ δ?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 32 / 82
![Page 52: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/52.jpg)
Example
What’s the result?
[Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 53: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/53.jpg)
Example
What’s the result?
[Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 54: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/54.jpg)
Example
What’s the result?
[Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 55: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/55.jpg)
Example
What’s the result?
[Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 56: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/56.jpg)
Example
What’s the result?
Original!
!"!"!"
!"!"!"
!"!"!"
#"#"#"
#"$"#"
#"#"#" -! %"* [Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 57: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/57.jpg)
Example
What’s the result?
Original!
!"!"!"
!"!"!"
!"!"!"
#"#"#"
#"$"#"
#"#"#" -!!"#$%&'(')*+,-&$*
%&''()*+&*(,"(-.(,/*
0"* [Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 33 / 82
![Page 58: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/58.jpg)
Sharpening
[Source: D. Lowe]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 34 / 82
![Page 59: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/59.jpg)
Gaussian Filter
Convolution with itself is another Gaussian
* !"
Convolving twice with Gaussian kernel of width σ is the same as convolvingonce with kernel of width σ
√2
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 35 / 82
![Page 60: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/60.jpg)
Sharpening revisited
What does blurring take away?
!"#$#%&'( )*!!+,-.(/0102(
!"
.-+&#'(
#"
Let’s add it back
!"#$%&'&()
!"
*$+,+'#-) (&.#+-)
#"$"
[Source: S. Lazebnik]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 36 / 82
![Page 61: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/61.jpg)
Sharpening
!"#$%&'&()
#$%&'&()
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 37 / 82
![Page 62: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/62.jpg)
”Optical” Convolution
Camera Shake
* !"Figure: Fergus, et al., SIGGRAPH 2006
Blur in out-of-focus regions of an image.
Figure: Bokeh: Click for more info
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 38 / 82
![Page 63: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/63.jpg)
Correlation vs Convolution
The convolution is both commutative and associative.
The Fourier transform of two convolved images is the product of theirindividual Fourier transforms.
Both correlation and convolution are linear shift-invariant (LSI)operators, which obey both the superposition principle
h ◦ (f0 + f1) = h ◦ fo + h ◦ f1
and the shift invariance principle
if g(i , j) = f (i + k, j + l)↔ (h ◦ g)(i , j) = (h ◦ f )(i + k, j + l)
which means that shifting a signal commutes with applying the operator.
Is the same as saying that the effect of the operator is the same everywhere.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 39 / 82
![Page 64: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/64.jpg)
Correlation vs Convolution
The convolution is both commutative and associative.
The Fourier transform of two convolved images is the product of theirindividual Fourier transforms.
Both correlation and convolution are linear shift-invariant (LSI)operators, which obey both the superposition principle
h ◦ (f0 + f1) = h ◦ fo + h ◦ f1
and the shift invariance principle
if g(i , j) = f (i + k , j + l)↔ (h ◦ g)(i , j) = (h ◦ f )(i + k, j + l)
which means that shifting a signal commutes with applying the operator.
Is the same as saying that the effect of the operator is the same everywhere.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 39 / 82
![Page 65: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/65.jpg)
Correlation vs Convolution
The convolution is both commutative and associative.
The Fourier transform of two convolved images is the product of theirindividual Fourier transforms.
Both correlation and convolution are linear shift-invariant (LSI)operators, which obey both the superposition principle
h ◦ (f0 + f1) = h ◦ fo + h ◦ f1
and the shift invariance principle
if g(i , j) = f (i + k , j + l)↔ (h ◦ g)(i , j) = (h ◦ f )(i + k, j + l)
which means that shifting a signal commutes with applying the operator.
Is the same as saying that the effect of the operator is the same everywhere.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 39 / 82
![Page 66: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/66.jpg)
Correlation vs Convolution
The convolution is both commutative and associative.
The Fourier transform of two convolved images is the product of theirindividual Fourier transforms.
Both correlation and convolution are linear shift-invariant (LSI)operators, which obey both the superposition principle
h ◦ (f0 + f1) = h ◦ fo + h ◦ f1
and the shift invariance principle
if g(i , j) = f (i + k , j + l)↔ (h ◦ g)(i , j) = (h ◦ f )(i + k, j + l)
which means that shifting a signal commutes with applying the operator.
Is the same as saying that the effect of the operator is the same everywhere.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 39 / 82
![Page 67: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/67.jpg)
Boundary Effects
The results of filtering the image in this form will lead to a darkening of thecorner pixels.
The original image is effectively being padded with 0 values wherever theconvolution kernel extends beyond the original image boundaries.
A number of alternative padding or extension modes have been developed.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 40 / 82
![Page 68: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/68.jpg)
Boundary Effects
The results of filtering the image in this form will lead to a darkening of thecorner pixels.
The original image is effectively being padded with 0 values wherever theconvolution kernel extends beyond the original image boundaries.
A number of alternative padding or extension modes have been developed.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 40 / 82
![Page 69: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/69.jpg)
Boundary Effects
The results of filtering the image in this form will lead to a darkening of thecorner pixels.
The original image is effectively being padded with 0 values wherever theconvolution kernel extends beyond the original image boundaries.
A number of alternative padding or extension modes have been developed.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 40 / 82
![Page 70: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/70.jpg)
Boundary Effects
The results of filtering the image in this form will lead to a darkening of thecorner pixels.
The original image is effectively being padded with 0 values wherever theconvolution kernel extends beyond the original image boundaries.
A number of alternative padding or extension modes have been developed.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 40 / 82
![Page 71: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/71.jpg)
Separable Filters
The process of performing a convolution requires K 2 operations per pixel,where K is the size (width or height) of the convolution kernel.
In many cases, this operation can be speed up by first performing a 1Dhorizontal convolution followed by a 1D vertical convolution, requiring 2Koperations.
If his is possible, then the convolution kernel is called separable.
And it is the outer product of two kernels
K = vhT
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 41 / 82
![Page 72: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/72.jpg)
Separable Filters
The process of performing a convolution requires K 2 operations per pixel,where K is the size (width or height) of the convolution kernel.
In many cases, this operation can be speed up by first performing a 1Dhorizontal convolution followed by a 1D vertical convolution, requiring 2Koperations.
If his is possible, then the convolution kernel is called separable.
And it is the outer product of two kernels
K = vhT
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 41 / 82
![Page 73: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/73.jpg)
Separable Filters
The process of performing a convolution requires K 2 operations per pixel,where K is the size (width or height) of the convolution kernel.
In many cases, this operation can be speed up by first performing a 1Dhorizontal convolution followed by a 1D vertical convolution, requiring 2Koperations.
If his is possible, then the convolution kernel is called separable.
And it is the outer product of two kernels
K = vhT
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 41 / 82
![Page 74: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/74.jpg)
Separable Filters
The process of performing a convolution requires K 2 operations per pixel,where K is the size (width or height) of the convolution kernel.
In many cases, this operation can be speed up by first performing a 1Dhorizontal convolution followed by a 1D vertical convolution, requiring 2Koperations.
If his is possible, then the convolution kernel is called separable.
And it is the outer product of two kernels
K = vhT
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 41 / 82
![Page 75: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/75.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 42 / 82
![Page 76: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/76.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
What does this filter do?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 42 / 82
![Page 77: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/77.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 43 / 82
![Page 78: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/78.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
What does this filter do?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 43 / 82
![Page 79: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/79.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 44 / 82
![Page 80: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/80.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
What does this filter do?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 44 / 82
![Page 81: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/81.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 45 / 82
![Page 82: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/82.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
What does this filter do?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 45 / 82
![Page 83: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/83.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 46 / 82
![Page 84: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/84.jpg)
Let’s play a game...
Is this separable? If yes, what’s the separable version?
What does this filter do?
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 46 / 82
![Page 85: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/85.jpg)
How can we tell if a given kernel K is indeed separable?
Inspection... this is what we were doing.
Looking at the analytic form of it.
Look at the singular value decomposition (SVD), and if only one singularvalue is non-zero, then it is separable
K = UΣVT =∑i
σiuivTi
with Σ = diag(σi ).√σ1u1 and
√σ1vT
1 are the vertical and horizontal kernels.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 47 / 82
![Page 86: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/86.jpg)
How can we tell if a given kernel K is indeed separable?
Inspection... this is what we were doing.
Looking at the analytic form of it.
Look at the singular value decomposition (SVD), and if only one singularvalue is non-zero, then it is separable
K = UΣVT =∑i
σiuivTi
with Σ = diag(σi ).
√σ1u1 and
√σ1vT
1 are the vertical and horizontal kernels.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 47 / 82
![Page 87: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/87.jpg)
How can we tell if a given kernel K is indeed separable?
Inspection... this is what we were doing.
Looking at the analytic form of it.
Look at the singular value decomposition (SVD), and if only one singularvalue is non-zero, then it is separable
K = UΣVT =∑i
σiuivTi
with Σ = diag(σi ).√σ1u1 and
√σ1vT
1 are the vertical and horizontal kernels.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 47 / 82
![Page 88: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/88.jpg)
How can we tell if a given kernel K is indeed separable?
Inspection... this is what we were doing.
Looking at the analytic form of it.
Look at the singular value decomposition (SVD), and if only one singularvalue is non-zero, then it is separable
K = UΣVT =∑i
σiuivTi
with Σ = diag(σi ).√σ1u1 and
√σ1vT
1 are the vertical and horizontal kernels.
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 47 / 82
![Page 89: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/89.jpg)
Application of filtering: Template matching
Filters as templates: filters look like the effects they are intended to find.
Use normalized cross-correlation score to find a given pattern (template)in the image.
Normalization needed to control for relative brightnesses.
[Source: K. Grauman]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 48 / 82
![Page 90: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/90.jpg)
Template matching
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 49 / 82
![Page 91: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/91.jpg)
More complex Scenes
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 50 / 82
![Page 92: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/92.jpg)
Let’s talk about Edge Detection
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 51 / 82
![Page 93: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/93.jpg)
Filtering: Edge detection
Map image from 2d array of pixels to a set of curves or line segments orcontours.
More compact than pixels.
Look for strong gradients, post-process.
Figure: [Shotton et al. PAMI, 07]
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 52 / 82
![Page 94: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/94.jpg)
Origin of edges
Edges are caused by a variety of factors
depth discontinuity
surface color discontinuity
illumination discontinuity
surface normal discontinuity
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 53 / 82
![Page 95: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/95.jpg)
What causes an edge?
[Source: K. Grauman]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 54 / 82
![Page 96: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/96.jpg)
Looking more locally...
[Source: K. Grauman]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 55 / 82
![Page 97: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/97.jpg)
Images as functions
Edges look like steep cliffs
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 56 / 82
![Page 98: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/98.jpg)
Characterizing Edges
An edge is a place of rapid change in the image intensity function.
[Source: S. Lazebnik]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 57 / 82
![Page 99: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/99.jpg)
How to Implement Derivatives with Convolution
How can we differentiate a digital image F[x,y]?
Option 1: reconstruct a continuous image f , then compute the partialderivative as
∂f (x , y)
∂x= limε→0
f (x + ε, y)− f (x)
ε
Option 2: take discrete derivative (finite difference)
∂f (x , y)
∂x≈ f [x + 1, y ]− f [x ]
1
What would be the filter to implement this using convolution?
1 -1 -1
1 !" !"
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 58 / 82
![Page 100: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/100.jpg)
How to Implement Derivatives with Convolution
How can we differentiate a digital image F[x,y]?
Option 1: reconstruct a continuous image f , then compute the partialderivative as
∂f (x , y)
∂x= limε→0
f (x + ε, y)− f (x)
ε
Option 2: take discrete derivative (finite difference)
∂f (x , y)
∂x≈ f [x + 1, y ]− f [x ]
1
What would be the filter to implement this using convolution?
1 -1 -1
1 !" !"
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 58 / 82
![Page 101: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/101.jpg)
How to Implement Derivatives with Convolution
How can we differentiate a digital image F[x,y]?
Option 1: reconstruct a continuous image f , then compute the partialderivative as
∂f (x , y)
∂x= limε→0
f (x + ε, y)− f (x)
ε
Option 2: take discrete derivative (finite difference)
∂f (x , y)
∂x≈ f [x + 1, y ]− f [x ]
1
What would be the filter to implement this using convolution?
1 -1 -1
1 !" !"
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 58 / 82
![Page 102: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/102.jpg)
How to Implement Derivatives with Convolution
How can we differentiate a digital image F[x,y]?
Option 1: reconstruct a continuous image f , then compute the partialderivative as
∂f (x , y)
∂x= limε→0
f (x + ε, y)− f (x)
ε
Option 2: take discrete derivative (finite difference)
∂f (x , y)
∂x≈ f [x + 1, y ]− f [x ]
1
What would be the filter to implement this using convolution?
1 -1 -1
1 !" !"
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 58 / 82
![Page 103: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/103.jpg)
Partial derivatives of an image
Figure: Using correlation filters
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 59 / 82
![Page 104: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/104.jpg)
Finite Difference Filters
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 60 / 82
![Page 105: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/105.jpg)
Image Gradient
The gradient of an image ∇f =[∂f∂x ,
∂f∂y
]The gradient points in the direction of most rapid change in intensity
The gradient direction (orientation of edge normal) is given by:
θ = tan−1(∂f
∂y/∂f
∂x
)The edge strength is given by the magnitude ||∇f || =
√( ∂f∂x )2 + ( ∂f∂y )2
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 61 / 82
![Page 106: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/106.jpg)
Image Gradient
The gradient of an image ∇f =[∂f∂x ,
∂f∂y
]The gradient points in the direction of most rapid change in intensity
The gradient direction (orientation of edge normal) is given by:
θ = tan−1(∂f
∂y/∂f
∂x
)
The edge strength is given by the magnitude ||∇f || =√
( ∂f∂x )2 + ( ∂f∂y )2
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 61 / 82
![Page 107: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/107.jpg)
Image Gradient
The gradient of an image ∇f =[∂f∂x ,
∂f∂y
]The gradient points in the direction of most rapid change in intensity
The gradient direction (orientation of edge normal) is given by:
θ = tan−1(∂f
∂y/∂f
∂x
)The edge strength is given by the magnitude ||∇f || =
√( ∂f∂x )2 + ( ∂f∂y )2
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 61 / 82
![Page 108: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/108.jpg)
Image Gradient
The gradient of an image ∇f =[∂f∂x ,
∂f∂y
]The gradient points in the direction of most rapid change in intensity
The gradient direction (orientation of edge normal) is given by:
θ = tan−1(∂f
∂y/∂f
∂x
)The edge strength is given by the magnitude ||∇f || =
√( ∂f∂x )2 + ( ∂f∂y )2
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 61 / 82
![Page 109: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/109.jpg)
Image Gradient
[Source: S. Lazebnik]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 62 / 82
![Page 110: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/110.jpg)
Effects of noise
Consider a single row or column of the image.
Plotting intensity as a function of position gives a signal.
!"#$%&#'()*&#+,-.&
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 63 / 82
![Page 111: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/111.jpg)
Effects of noise
Smooth first, and look for picks in ∂∂x (h ∗ f ).
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 64 / 82
![Page 112: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/112.jpg)
Derivative theorem of convolution
Differentiation property of convolution
∂
∂x(h ∗ f ) = (
∂h
∂x) ∗ f = h ∗ (
∂f
∂x)
It saves one operation
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 65 / 82
![Page 113: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/113.jpg)
2D Edge Detection Filters
Gaussian Derivative of Gaussian (x)
hσ(x , y) = 12πσ2 exp−
u2+v2
2σ2 ∂∂x hσ(u, v)
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 66 / 82
![Page 114: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/114.jpg)
Derivative of Gaussians
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 67 / 82
![Page 115: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/115.jpg)
Laplacian of Gaussians
Edge by detecting zero-crossings of bottom graph
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 68 / 82
![Page 116: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/116.jpg)
2D Edge Filtering
with ∇2 the Laplacian operator ∇2f = ∂2f∂x2 + ∂2f
∂y2
[Source: S. Seitz]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 69 / 82
![Page 117: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/117.jpg)
Effect of σ on derivatives
The detected structures differ depending on the Gaussian’s scale parameter:
Larger values: larger scale edges detected.
Smaller values: finer features detected.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 70 / 82
![Page 118: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/118.jpg)
Derivatives
Use opposite signs to get response in regions of high contrast.
They sum to 0 so that there is no response in constant regions.
High absolute value at points of high contrast.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 71 / 82
![Page 119: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/119.jpg)
Derivatives
Use opposite signs to get response in regions of high contrast.
They sum to 0 so that there is no response in constant regions.
High absolute value at points of high contrast.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 71 / 82
![Page 120: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/120.jpg)
Derivatives
Use opposite signs to get response in regions of high contrast.
They sum to 0 so that there is no response in constant regions.
High absolute value at points of high contrast.
[Source: K. Grauman]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 71 / 82
![Page 121: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/121.jpg)
Band-pass filters
The Sobel and corner filters are band-pass and oriented filters.
More sophisticated filters can be obtained by convolving with a Gaussianfilter
G (x , y , σ) =1
2πσ2exp
(−x2 + y2
2σ2
)and taking the first or second derivatives.
These filters are band-pass filters: they filter low and high frequencies.
The second derivative of a two-dimensional image is the laplacian operator
∇2f =∂2f
∂x2+∂2f
∂y2
Blurring an image with a Gaussian and then taking its Laplacian is equivalentto convolving directly with the Laplacian of Gaussian (LoG) filter,
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 72 / 82
![Page 122: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/122.jpg)
Band-pass filters
The Sobel and corner filters are band-pass and oriented filters.
More sophisticated filters can be obtained by convolving with a Gaussianfilter
G (x , y , σ) =1
2πσ2exp
(−x2 + y2
2σ2
)and taking the first or second derivatives.
These filters are band-pass filters: they filter low and high frequencies.
The second derivative of a two-dimensional image is the laplacian operator
∇2f =∂2f
∂x2+∂2f
∂y2
Blurring an image with a Gaussian and then taking its Laplacian is equivalentto convolving directly with the Laplacian of Gaussian (LoG) filter,
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 72 / 82
![Page 123: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/123.jpg)
Band-pass filters
The Sobel and corner filters are band-pass and oriented filters.
More sophisticated filters can be obtained by convolving with a Gaussianfilter
G (x , y , σ) =1
2πσ2exp
(−x2 + y2
2σ2
)and taking the first or second derivatives.
These filters are band-pass filters: they filter low and high frequencies.
The second derivative of a two-dimensional image is the laplacian operator
∇2f =∂2f
∂x2+∂2f
∂y2
Blurring an image with a Gaussian and then taking its Laplacian is equivalentto convolving directly with the Laplacian of Gaussian (LoG) filter,
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 72 / 82
![Page 124: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/124.jpg)
Band-pass filters
The Sobel and corner filters are band-pass and oriented filters.
More sophisticated filters can be obtained by convolving with a Gaussianfilter
G (x , y , σ) =1
2πσ2exp
(−x2 + y2
2σ2
)and taking the first or second derivatives.
These filters are band-pass filters: they filter low and high frequencies.
The second derivative of a two-dimensional image is the laplacian operator
∇2f =∂2f
∂x2+∂2f
∂y2
Blurring an image with a Gaussian and then taking its Laplacian is equivalentto convolving directly with the Laplacian of Gaussian (LoG) filter,
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 72 / 82
![Page 125: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/125.jpg)
Band-pass filters
The Sobel and corner filters are band-pass and oriented filters.
More sophisticated filters can be obtained by convolving with a Gaussianfilter
G (x , y , σ) =1
2πσ2exp
(−x2 + y2
2σ2
)and taking the first or second derivatives.
These filters are band-pass filters: they filter low and high frequencies.
The second derivative of a two-dimensional image is the laplacian operator
∇2f =∂2f
∂x2+∂2f
∂y2
Blurring an image with a Gaussian and then taking its Laplacian is equivalentto convolving directly with the Laplacian of Gaussian (LoG) filter,
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 72 / 82
![Page 126: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/126.jpg)
Band-pass filters
The directional or oriented filter can obtained by smoothing with aGaussian (or some other filter) and then taking a directional derivative∇u = ∂
∂uu · ∇(G ∗ f ) = ∇u(G ∗ f ) = (∇uG ) ∗ f
with u = (cos θ, sin θ).
The Sobel operator is a simple approximation of this:
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 73 / 82
![Page 127: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/127.jpg)
Practical Example
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 74 / 82
![Page 128: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/128.jpg)
Finding Edges
Figure: Gradient magnitude
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 75 / 82
![Page 129: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/129.jpg)
Finding Edges
!"#$#%&'%("#%#)*#+%
Figure: Gradient magnitude
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 75 / 82
![Page 130: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/130.jpg)
Non-Maxima Suppression
Figure: Gradient magnitude
Check if pixel is local maximum along gradient direction: requiresinterpolation
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 76 / 82
![Page 131: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/131.jpg)
Finding Edges
Figure: Thresholding
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 77 / 82
![Page 132: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/132.jpg)
Finding Edges
Figure: Thinning: Non-maxima suppression
[Source: N. Snavely]Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 77 / 82
![Page 133: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/133.jpg)
Canny Edge Detector
Matlab: edge(image,’canny’)
1 Filter image with derivative of Gaussian
2 Find magnitude and orientation of gradient
3 Non-maximum suppression
4 Linking and thresholding (hysteresis):
Define two thresholds: low and highUse the high threshold to start edge curves and the low threshold tocontinue them
[Source: D. Lowe and L. Fei-Fei]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 78 / 82
![Page 134: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/134.jpg)
Canny edge detector
Still one of the most widely used edge detectors in computer vision
J. Canny, A Computational Approach To Edge Detection, IEEE Trans.Pattern Analysis and Machine Intelligence, 8:679-714, 1986.
Depends on several parameters: σ of the blur and the thresholds
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 79 / 82
![Page 135: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/135.jpg)
Canny edge detector
large σ detects large-scale edges
small σ detects fine edges
Canny with Canny with original
[Source: S. Seitz]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 80 / 82
![Page 136: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/136.jpg)
Scale Space (Witkin 83)
larger
Gaussian filtered signal
first derivative peaks
Properties of scale space (w/ Gaussian smoothing)
edge position may shift with increasing scale (σ)
two edges may merge with increasing scale
an edge may not split into two with increasing scale
[Source: N. Snavely]
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 81 / 82
![Page 137: Computer Vision: Filtering](https://reader034.vdocuments.mx/reader034/viewer/2022052318/587f20061a28ab8a5f8c0528/html5/thumbnails/137.jpg)
Next class ... more on filtering and image features
Raquel Urtasun (TTI-C) Computer Vision Jan 10, 2013 82 / 82