csci 1290: comp photo - brown...
TRANSCRIPT
CSCI 1290: Comp PhotoFall 2018 @ Brown University
James Tompkin
Many slides thanks to James Hays’ old CS 129 course, along with all of its acknowledgements.
Things I forgot on Thursday
• Grads are not required to do 10pts of extra credit
- This is not a 2000-level course
• Question: How many of you do not have laptops (for in-class labs)?
What is an image?
>> I = rand(256,256);
Think-Pair-Share:
- What is this? What does it look like?
- Which values does it take?
- How many values can it take?
- Is it an image?
>> I = rand(256,256);
>> imshow(I);
Danny Alexander
Dimensionality of an Image
• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.
• Some depiction of all possible scenes would fit into this memory.
Dimensionality of an Image
• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.
• Some depiction of all possible scenes would fit into this memory.
• We have to operate or reason over this extremely high-dimensional space.• Subspace of ‘natural’ images.
• Deriving low-dimensional, explainable, physically-meaningful models.
Each part of an image is a pixel
y
x
Each part of an image is a pixel
• Pixel -> picture element
y
x
I(x,y)
‘138’
Image as a 2D sampling of signal
• Signal: function depending on some variable with physical meaning.
• Image: sampling of that function.• 2 variables: xy coordinates
• 3 variables: xy + time (video)
• ‘Brightness’ is the value of the function for visible light
• Can be other physical values too: temperature, pressure, depth …
Danny Alexander
Example 2D Images
Danny Alexander
Sampling in 1D
• Sampling in 1D takes a ‘function’, and returns a vector whose elements are values of that function at the sample points.
Danny Alexander
x
f(x)
[2,6,5,2]
x
Sampling in 2D
• Sampling in 2D takes a function and returns a matrix.
Danny Alexander
Grayscale Digital Image as ‘Height Map’
Brightnessor intensity
x y
Danny Alexander
What is each part of a photograph?
• Pixel -> picture element
y
x
I(x,y)
‘127’
Integrating light over a range of angles.
Output Image
Camera Sensor
James Hays
Resolution – geometric vs. spatial resolution
Both images are ~500x500 pixels
Quantization
Quantization Effects – Radiometric Resolution
8 bit – 256 levels 1 bit – 2 levels2 bit – 4 levels4 bit – 16 levels
Images in Matlab• NxM image “im”
– im(1,1) = top-left pixel value
– im(y, x) = y pixels down, x pixels to right
– im(N, M) = bottom-right pixel
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
rowcolumn
James Hays
Grayscale intensity0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
Color R
G
B
James Hays
Images in Matlab• NxM RGB image “im”
– im(1,1,1) = top-left pixel value in R-channel
– im(y, x, b) = y pixels down, x pixels to right in the bth channel
– im(N, M, 3) = bottom-right pixel in B-channel
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
R
G
B
row
column
James Hays
Cameras with Three Sensors
Lens
[Edmund Optics; Adam Wilt]
Cheaper/More Compact Color Sensing: Bayer Filter
• Estimate RGBat ‘G’ cells from neighboring values
Steve Seitz
Camera Color Response
MaxMax.com
Canon 450D
What’s going on over here?
How to capture color?
• Back in the old days…
Take three photographs each with a different color filter
• How to display?
Light is additive (linear), so justproject three photographs on top of each other!
[High Point Scientific]
The Prokudin-Gorsky Photo Collection
Prokudin-Gorsky (1863-1944)
Similar design camera ofAdolf Miethe
[Nadezhda Stanulevich, Russian Academy of Fine Arts]
The Prokudin-Gorsky Photo Collection
Similar design viewing projector of Adolf Miethe
Prokudin-Gorsky (1863-1944)
[Nadezhda Stanulevich, Russian Academy of Fine Arts]
[Wikipedia]
Alim Khan, emir of Bukhara,1911.
‘Good result’
• Six or seven hours of manual work…
• Geometric correction
• Radiometric (color) correction
• Blemish removal
Too much! Let’s go computational!
[Wikipedia]
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
How to align R,G,B channels?
Think-pair-share!
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
What is a good metric? (open research problem)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
Candidate metrics:• Sum of Squared Differences (SSD):
I = Image to alignP = Image to comparex,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
Candidate metrics:• Normalized Cross Correlation (NCC):
I = Image to align Ī = Mean of I
P = Image to compare 𝑃= Mean of Px,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Next problem: It’s expensive!
Especially for high-resolution images.
Coarse-to-fine strategy:
1. Compare at low resolution; find alignment
2. At higher resolution, search around this alignment
3. Repeat
Multi-resolution Image Pyramids
Wikipedia – Image Pyramids
Good for:
Search over translationsLike project 1Classic coarse-to-fine strategy
Search over scaleTemplate matchingE.g. find a face at different scales
Pre-computationReal-time graphics (texture mapping)
CompositingIn a couple of lectures…
A bar in the big
images is a hair on
the zebra’s nose;
in smaller images,
a stripe; in the
smallest, the
animal’s nose
Figure from David Forsyth
Algorithm for downsampling by factor of 2
1. Start with image(h, w)
2. Sample every other pixel
• im_small = image( 1:2:end, 1:2:end );
3. Repeat until im_small is 1 pixel large.
Hays
Image sub-sampling
Throw away every other row and
column to create a 1/2 size image.
1/4
1/8
Steve Seitz
Subsampling without filtering
1/4 (2x subsample) 1/8 (4x subsample)1/2
Steve Seitz
Subsampling and aliasing
• 1D example (sinewave):
Source: S. Marschner
Aliasing problem
Source: S. Marschner
• 1D example (sinewave):
Aliasing problem
What’s happening?
Input signal:
x = 0:.05:5; imagesc(sin((2.^x).*x))
Plot as image:
Aliasing!
Not enough
samples
• Sub-sampling may be dangerous….
• Characteristic errors may appear: • “Car wheels rolling the wrong way in movies”
• “Striped shirts look funny on color television”
• “Checkerboards disintegrate in ray tracing”
Source: D. Forsyth; Wikipedia
Aliasing problem
Moiré patterns!
Source: A. Efros
Aliasing in graphics
Gong 96, 1932, Claude Tousignant, Musée des Beaux-Arts de Montréal
Aliasing in video
Slide by Steve Seitz
Videos
[YouTube; JoinBuzzirk; phrancque]
http://blogs.discovermagazine.com/badastronomy/2009/06/24/the-blue-and-the-green/
The blue and green colors are actually the same
• When sampling a signal at discrete intervals, the sampling frequency must be 2 fmax
• fmax = max frequency of the input signal
• This will allows to reconstruct the original perfectly from the sampled version
good
bad
Nyquist-Shannon Sampling Theorem
How to fix aliasing?
Solutions?
Better sensors
Solutions:
• Sample more often
Anti-aliasing
Solutions:
• Sample more often
• Get rid of all frequencies that are greater than half the new sampling frequency– Will lose information
– But it’s better than aliasing
– Apply a smoothing (low pass) filter
Hays
Anti-aliasing
Forsyth and Ponce 2002
Subsampling without filtering
1/4 (2x subsample) 1/8 (4x subsample)1/2
Steve Seitz
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Steve SeitzGaussian Pyramid [Burt and Adelson, 1983]
IMAGE FILTERING
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
],[],[],[,
lnkmIlkfnmhlk
++=I=imagef=filterh=output
2d coords=m,n2d coords=k,l
[ ] [ ][ ]
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Example: box filter
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
],[],[],[,
lnkmIlkfnmhlk
++=
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
?
],[],[],[,
lnkmIlkfnmhlk
++=
0 10 20 30 30
50
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
?
],[],[],[,
lnkmIlkfnmhlk
++=
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
[.,.]h[.,.]I
Image filtering111
111
111],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
What does it do?
• Replaces each pixel with
an average of its
neighborhood
• Achieve smoothing effect
(remove sharp features)
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Box Filter
What does it do?
• Replaces each pixel with
an average of its
neighborhood
• Achieve smoothing effect
(remove sharp features)
• Why does it sum to one?
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Box Filter
Smoothing with box filterJames Hays
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
• Really important!
– Enhance images• Denoise, resize, increase contrast, etc.
– Extract information from images• Texture, edges, distinctive points, etc.
– Detect patterns• Template matching
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
Think-Pair-Share time
000
010
000
1.
000
100
000
2.
-101
-202
-101
111
111
111
000
020
000
-
3.
4.
1. Practice with linear filters
000
010
000
Original
?
Source: D. Lowe
1. Practice with linear filters
000
010
000
Original Filtered
(no change)
Source: D. Lowe
2. Practice with linear filters
000
100
000
Original
?
Source: D. Lowe
2. Practice with linear filters
000
100
000
Original Shifted left
By 1 pixel
Source: D. Lowe
3. Practice with linear filters
-101
-202
-101
Vertical Edge
(absolute value)
Sobel
David Lowe
3. Practice with linear filters
-1-2-1
000
121
Horizontal Edge
(absolute value)
Sobel
David Lowe
4. Practice with linear filters
Original
111
111
111
000
020
000
- ?
(Note that filter sums to 1)
Source: D. Lowe
4. Practice with linear filters
Original
111
111
111
000
020
000
-
Sharpening filter
- Accentuates differences with local
average
Source: D. Lowe
4. Practice with linear filters
Source: D. Lowe
Correlation and Convolution
• 2d correlation
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
h=filter2(f,I); or h=imfilter(I,f);
Correlation and Convolution
• 2d correlation
• 2d convolution
],[],[],[,
lnkmIlkfnmhlk
−−=
],[],[],[,
lnkmIlkfnmhlk
++=
conv2(I,f)is the same as filter2(rot90(f,2),I)
Correlation and convolution are identical when the filter is symmetric.
James Hays
h=filter2(f,I); or h=imfilter(I,f);
h=conv2(f,I); or h=imfilter(I,f,’conv’);
Key properties of linear filters
Linearity:imfilter(I, f1 + f2) =
imfilter(I,f1) + imfilter(I,f2)
Shift invariance:Same behavior regardless of pixel locationimfilter(I,shift(f)) = shift(imfilter(I,f))
Any linear, shift-invariant operator can be represented as a convolution.
Source: S. Lazebnik
Convolution properties
Commutative: a * b = b * a
– Conceptually no difference between filter and signal
– But particular filtering implementations might break this equality, e.g., image edges
Associative: a * (b * c) = (a * b) * c
– Often apply several filters one after another: (((a * b1) * b2) * b3)
– This is equivalent to applying one filter: a * (b1 * b2 * b3)
Source: S. Lazebnik
Convolution properties
Commutative: a * b = b * a
– Conceptually no difference between filter and signal
– But particular filtering implementations might break this equality, e.g., image edges
Associative: a * (b * c) = (a * b) * c
– Often apply several filters one after another: (((a * b1) * b2) * b3)
– This is equivalent to applying one filter: a * (b1 * b2 * b3)
– Correlation is _not_ associative (rotation effect)
– Why important?
Source: S. Lazebnik
Convolution properties• Commutative: a * b = b * a
– Conceptually no difference between filter and signal– But particular filtering implementations might break this equality,
e.g., image edges
• Associative: a * (b * c) = (a * b) * c– Often apply several filters one after another: (((a * b1) * b2) * b3)– This is equivalent to applying one filter: a * (b1 * b2 * b3)– Correlation is _not_ associative (rotation effect)– Why important?
• Distributes over addition: a * (b + c) = (a * b) + (a * c)
• Scalars factor out: ka * b = a * kb = k (a * b)
• Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a
Source: S. Lazebnik
• Weight contributions of neighboring pixels by nearness
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Slide credit: Christopher Rasmussen
Important filter: Gaussian
xy
x
y
Smoothing with box filterJames Hays
111
111
111
],[ f
Smoothing with Gaussian filterJames Hays
Gaussian filters
• Remove “high-frequency” components from the image (low-pass filter)– Images become more smooth
• Gaussian convolved with Gaussian……is another Gaussian
– So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have
– Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2
• Separable kernel– Factors into product of two 1D Gaussians
Source: K. Grauman
Separability of the Gaussian filter
Source: D. Lowe
Separability example
*
*
=
=
2D convolution
(center location only)
Source: K. Grauman
The filter factors
into a product of 1D
filters:
Perform convolution
along rows:
Followed by convolution
along the remaining column:
=
Separability
Why is separability useful in practice?
Separability
Why is separability useful in practice?
MxN image, PxQ filter
• 2D convolution: ~MNPQ multiply-adds
• Separable 2D: ~MN(P+Q) multiply-adds
Speed up = PQ/(P+Q)
9x9 filter = ~4.5x faster
How big should the filter be?• Values at edges should be near zero
• Gaussians have infinite extent…
• Rule of thumb for Gaussian: set filter half-width to about 3 σ
Practical matters
James Hays
Practical matters
• What about near the edge?
– the filter window falls off the edge of the image
– need to extrapolate
– methods:
• clip filter (black)
• wrap around
• copy edge
• reflect across edge
Source: S. Marschner
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Make alignment operation fast using image pyramid.
Remove aliasing from pyramid by filtering.
Coarse-to-fine strategy:
0. Build Gaussian Pyramid
1. Compare at low resolution; find alignment
2. At higher resolution, search around this alignment
3. Repeat