computational photography cse 590 tamara berg filtering & pyramids
TRANSCRIPT
Computational PhotographyCSE 590
Tamara Berg
Filtering & Pyramids
Image filtering
• Image filtering: compute function of local neighborhood at each position
• Really important!– Enhance images
• Denoise, resize, increase contrast, etc.
– Extract information from images• Texture, edges, distinctive points, etc.
– Detect patterns• Template matching
111
111
111
Slide credit: David Lowe (UBC)
],[g
Example: box filter
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
[.,.]h[.,.]f
Image filtering111
111
111
],[g
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
?
],[],[],[,
lnkmflkgnmhlk
0 10 20 30 30
50
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111
111
111
],[g
Credit: S. Seitz
?
],[],[],[,
lnkmflkgnmhlk
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
[.,.]h[.,.]f
Image filtering111111111],[g
Credit: S. Seitz
],[],[],[,
lnkmflkgnmhlk
What does it do?• Replaces each pixel with an
average of its neighborhood
• Achieve smoothing effect (remove sharp features)
111
111
111
Slide credit: David Lowe (UBC)
],[g
Box Filter
Smoothing with box filter
Practice with linear filters
000
010
000
Original
?
Source: D. Lowe
Practice with linear filters
000
010
000
Original Filtered (no change)
Source: D. Lowe
Practice with linear filters
000
100
000
Original
?
Source: D. Lowe
Practice with linear filters
000
100
000
Original Shifted leftBy 1 pixel
Source: D. Lowe
Practice with linear filters
Original
111
111
111
000
020
000
- ?
Source: D. Lowe
Practice with linear filters
Original
111
111
111
000
020
000
-
Sharpening filter- Accentuates differences with local
average
Source: D. Lowe
Sharpening
Source: D. Lowe
Characterizing edges• An edge is a place of rapid change in the image
intensity function
imageintensity function
(along horizontal scanline) first derivative
edges correspond toextrema of derivative
source: Svetlana Lazebnik
Edge filters
Approximations of derivative filters:
Source: K. Grauman
Convolve filter with image to get edge map
Edge filters
Approximations of derivative filters:
Source: K. GraumanRespond highly to vertical edges
Edge filters
Approximations of derivative filters:
Source: K. GraumanRespond highly to horizontal edges
Edges: example
source: Svetlana Lazebnik
Gaussian Kernel
• Constant factor at front makes volume sum to 1 (can be ignored, as we should re-normalize weights to sum to 1 in any case)
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Source: C. Rasmussen
Example: Smoothing with a Gaussian
source: Svetlana Lazebnik
Gaussian filters
• Remove “high-frequency” components from the image (low-pass filter)– Images become more smooth
• Convolution with self is another Gaussian– So can smooth with small-width kernel, repeat, and
get same result as larger-width kernel would have• Separable kernel
– Factors into product of two 1D Gaussians
Source: K. Grauman
Separability of the Gaussian filter
Source: D. Lowe
The filter factorsinto a product of 1D
filters:
Practical matters• What is the size of the output?• MATLAB: imfilter(g, f, shape)
– shape = ‘full’: output size is sum of sizes of f and g– shape = ‘same’: output size is same as f– shape = ‘valid’: output size is difference of sizes of f and g
f
gg
gg
f
gg
gg
f
gg
gg
full same valid
Source: S. Lazebnik
Practical matters• What about near the edge?
– the filter window falls off the edge of the image– need to extrapolate– methods:
• clip filter (black)• wrap around• copy edge• reflect across edge
Source: S. Marschner
Practical matters
– methods (MATLAB):• clip filter (black): imfilter(f, g, 0)• wrap around: imfilter(f, g, ‘circular’)• copy edge: imfilter(f, g, ‘replicate’)• reflect across edge: imfilter(f, g, ‘symmetric’)
Source: S. Marschner
Q?
Template matching• Goal: find in image
• Main challenge: What is a good similarity or distance measure between two patches?– Correlation– Zero-mean correlation– Sum Square Difference– Normalized Cross Correlation
Side by Derek Hoiem
Matching with filters• Goal: find in image• Method 0: filter the image with eye patch
Input Filtered Image
],[],[],[,
lnkmflkgnmhlk
What went wrong?
f = imageg = filter
Side by Derek Hoiem
Matching with filters• Goal: find in image• Method 1: filter the image with zero-mean eye
Input Filtered Image (scaled) Thresholded Image
)],[()],[(],[,
lnkmgflkfnmhlk
True detections
False detections
mean of f
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD) Thresholded Image
2
,
)],[],[(],[ lnkmflkgnmhlk
True detections
Matching with filters• Goal: find in image• Method 2: SSD
Input 1- sqrt(SSD)
2
,
)],[],[(],[ lnkmflkgnmhlk
What’s the potential downside of SSD?
Side by Derek Hoiem
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
5.0
,
2,
,
2
,,
)],[()],[(
)],[)(],[(
],[
lknm
lk
nmlk
flnkmfglkg
flnkmfglkg
nmh
mean image patchmean template
Side by Derek Hoiem
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
Matching with filters• Goal: find in image• Method 3: Normalized cross-correlation
Input Normalized X-Correlation Thresholded Image
True detections
Q: What is the best method to use?
• A: Depends• Zero-mean filter: fastest but not a great
matcher• SSD: next fastest, sensitive to overall
intensity• Normalized cross-correlation: slowest,
invariant to local average intensity and contrast
Side by Derek Hoiem
Image half-sizing
This image is too big tofit on the screen. Howcan we reduce it?
How to generate a half-sized version?
Image sub-sampling
Throw away every other row and
column to create a 1/2 size image- called image sub-sampling
1/4
1/8
Slide by Steve Seitz
Image sub-sampling
1/4 (2x zoom) 1/8 (4x zoom)
Aliasing! What do we do?
1/2
Slide by Steve Seitz
Anti-aliasing
Get rid of all frequencies that are greater than half the new sampling frequency
– Will lose information– But it’s better than aliasing– Apply a smoothing filter
Slide by Derek Hoiem
Algorithm for downsampling by factor of 2
1. Start with image(h, w)2. Apply low-pass filter
im_blur = imfilter(image, fspecial(‘gaussian’, 7, 1))
3. Sample every other pixelim_small = im_blur(1:2:end, 1:2:end);
Slide by Derek Hoiem
Gaussian (lowpass) pre-filtering
G 1/4
G 1/8
Gaussian 1/2
Solution: filter the image, then subsample
Slide by Steve Seitz
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Slide by Steve Seitz
Compare with...
1/4 (2x zoom) 1/8 (4x zoom)1/2
Slide by Steve Seitz
Image Pyramids
Known as a Gaussian Pyramid [Burt and Adelson, 1983]• In computer graphics, a mip map [Williams, 1983]• A precursor to wavelet transform
Slide by Steve Seitz
A bar in the big images is a hair on the zebra’s nose; in smaller images, a stripe; in the smallest, the animal’s nose
Figure from David Forsyth
Coarse-to-fine Image Registration1. Compute Gaussian pyramid2. Align with coarse pyramid3. Successively align with finer
pyramids– Search smaller range
Why is this faster?
What are pyramids good for?
• Improve Search– Search over translations
• Like HW1• Classic coarse-to-fine strategy
– Search over scale• Template matching• E.g. find a face at different scales
• Pre-computation– Need to access image at different blur levels– Useful for texture mapping at different resolutions (called
mip-mapping) – Useful for compositing/blending – coming soon!