![Page 1: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/1.jpg)
CSCI 1290: Comp PhotoFall 2018 @ Brown University
James Tompkin
Many slides thanks to James Hays’ old CS 129 course, along with all of its acknowledgements.
![Page 2: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/2.jpg)
Things I forgot on Thursday
• Grads are not required to do 10pts of extra credit
- This is not a 2000-level course
• Question: How many of you do not have laptops (for in-class labs)?
![Page 3: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/3.jpg)
What is an image?
![Page 4: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/4.jpg)
>> I = rand(256,256);
Think-Pair-Share:
- What is this? What does it look like?
- Which values does it take?
- How many values can it take?
- Is it an image?
![Page 5: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/5.jpg)
>> I = rand(256,256);
>> imshow(I);
Danny Alexander
![Page 6: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/6.jpg)
Dimensionality of an Image
• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.
• Some depiction of all possible scenes would fit into this memory.
![Page 7: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/7.jpg)
Dimensionality of an Image
• @ 8bit = 256 values ^ 65,536• Computer says ‘Inf’ combinations.
• Some depiction of all possible scenes would fit into this memory.
• We have to operate or reason over this extremely high-dimensional space.• Subspace of ‘natural’ images.
• Deriving low-dimensional, explainable, physically-meaningful models.
![Page 8: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/8.jpg)
Each part of an image is a pixel
y
x
![Page 9: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/9.jpg)
Each part of an image is a pixel
• Pixel -> picture element
y
x
I(x,y)
‘138’
![Page 10: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/10.jpg)
Image as a 2D sampling of signal
• Signal: function depending on some variable with physical meaning.
• Image: sampling of that function.• 2 variables: xy coordinates
• 3 variables: xy + time (video)
• ‘Brightness’ is the value of the function for visible light
• Can be other physical values too: temperature, pressure, depth …
Danny Alexander
![Page 11: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/11.jpg)
Example 2D Images
Danny Alexander
![Page 12: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/12.jpg)
Sampling in 1D
• Sampling in 1D takes a ‘function’, and returns a vector whose elements are values of that function at the sample points.
Danny Alexander
x
f(x)
[2,6,5,2]
x
![Page 13: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/13.jpg)
Sampling in 2D
• Sampling in 2D takes a function and returns a matrix.
Danny Alexander
![Page 14: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/14.jpg)
Grayscale Digital Image as ‘Height Map’
Brightnessor intensity
x y
Danny Alexander
![Page 15: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/15.jpg)
What is each part of a photograph?
• Pixel -> picture element
y
x
I(x,y)
‘127’
![Page 16: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/16.jpg)
Integrating light over a range of angles.
Output Image
Camera Sensor
James Hays
![Page 17: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/17.jpg)
Resolution – geometric vs. spatial resolution
Both images are ~500x500 pixels
![Page 18: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/18.jpg)
Quantization
![Page 19: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/19.jpg)
Quantization Effects – Radiometric Resolution
8 bit – 256 levels 1 bit – 2 levels2 bit – 4 levels4 bit – 16 levels
![Page 20: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/20.jpg)
Images in Matlab• NxM image “im”
– im(1,1) = top-left pixel value
– im(y, x) = y pixels down, x pixels to right
– im(N, M) = bottom-right pixel
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
rowcolumn
James Hays
![Page 21: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/21.jpg)
Grayscale intensity0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
![Page 22: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/22.jpg)
Color R
G
B
James Hays
![Page 23: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/23.jpg)
Images in Matlab• NxM RGB image “im”
– im(1,1,1) = top-left pixel value in R-channel
– im(y, x, b) = y pixels down, x pixels to right in the bth channel
– im(N, M, 3) = bottom-right pixel in B-channel
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
0.92 0.93 0.94 0.97 0.62 0.37 0.85 0.97 0.93 0.92 0.99
0.95 0.89 0.82 0.89 0.56 0.31 0.75 0.92 0.81 0.95 0.91
0.89 0.72 0.51 0.55 0.51 0.42 0.57 0.41 0.49 0.91 0.92
0.96 0.95 0.88 0.94 0.56 0.46 0.91 0.87 0.90 0.97 0.95
0.71 0.81 0.81 0.87 0.57 0.37 0.80 0.88 0.89 0.79 0.85
0.49 0.62 0.60 0.58 0.50 0.60 0.58 0.50 0.61 0.45 0.33
0.86 0.84 0.74 0.58 0.51 0.39 0.73 0.92 0.91 0.49 0.74
0.96 0.67 0.54 0.85 0.48 0.37 0.88 0.90 0.94 0.82 0.93
0.69 0.49 0.56 0.66 0.43 0.42 0.77 0.73 0.71 0.90 0.99
0.79 0.73 0.90 0.67 0.33 0.61 0.69 0.79 0.73 0.93 0.97
0.91 0.94 0.89 0.49 0.41 0.78 0.78 0.77 0.89 0.99 0.93
R
G
B
row
column
James Hays
![Page 24: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/24.jpg)
Cameras with Three Sensors
Lens
[Edmund Optics; Adam Wilt]
![Page 25: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/25.jpg)
Cheaper/More Compact Color Sensing: Bayer Filter
• Estimate RGBat ‘G’ cells from neighboring values
Steve Seitz
![Page 26: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/26.jpg)
Camera Color Response
MaxMax.com
Canon 450D
What’s going on over here?
![Page 27: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/27.jpg)
How to capture color?
• Back in the old days…
Take three photographs each with a different color filter
• How to display?
Light is additive (linear), so justproject three photographs on top of each other!
[High Point Scientific]
![Page 28: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/28.jpg)
The Prokudin-Gorsky Photo Collection
Prokudin-Gorsky (1863-1944)
Similar design camera ofAdolf Miethe
[Nadezhda Stanulevich, Russian Academy of Fine Arts]
![Page 29: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/29.jpg)
The Prokudin-Gorsky Photo Collection
Similar design viewing projector of Adolf Miethe
Prokudin-Gorsky (1863-1944)
[Nadezhda Stanulevich, Russian Academy of Fine Arts]
![Page 30: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/30.jpg)
[Wikipedia]
Alim Khan, emir of Bukhara,1911.
![Page 31: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/31.jpg)
‘Good result’
• Six or seven hours of manual work…
• Geometric correction
• Radiometric (color) correction
• Blemish removal
Too much! Let’s go computational!
[Wikipedia]
![Page 32: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/32.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
How to align R,G,B channels?
Think-pair-share!
![Page 33: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/33.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
![Page 34: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/34.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
![Page 35: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/35.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
What is a good metric? (open research problem)
![Page 36: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/36.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
Candidate metrics:• Sum of Squared Differences (SSD):
I = Image to alignP = Image to comparex,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)
![Page 37: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/37.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Approach we will take: sliding window.
Compute metric for similarity at each position.
Candidate metrics:• Normalized Cross Correlation (NCC):
I = Image to align Ī = Mean of I
P = Image to compare 𝑃= Mean of Px,y = Pixel coordinatesu,v = Alignment offset coordinatesN = ‘Neighbourhood’ of pixels to compare (e.g., whole image)
![Page 38: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/38.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Next problem: It’s expensive!
Especially for high-resolution images.
Coarse-to-fine strategy:
1. Compare at low resolution; find alignment
2. At higher resolution, search around this alignment
3. Repeat
![Page 39: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/39.jpg)
Multi-resolution Image Pyramids
Wikipedia – Image Pyramids
Good for:
Search over translationsLike project 1Classic coarse-to-fine strategy
Search over scaleTemplate matchingE.g. find a face at different scales
Pre-computationReal-time graphics (texture mapping)
CompositingIn a couple of lectures…
![Page 40: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/40.jpg)
A bar in the big
images is a hair on
the zebra’s nose;
in smaller images,
a stripe; in the
smallest, the
animal’s nose
Figure from David Forsyth
![Page 41: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/41.jpg)
Algorithm for downsampling by factor of 2
1. Start with image(h, w)
2. Sample every other pixel
• im_small = image( 1:2:end, 1:2:end );
3. Repeat until im_small is 1 pixel large.
Hays
![Page 42: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/42.jpg)
Image sub-sampling
Throw away every other row and
column to create a 1/2 size image.
1/4
1/8
Steve Seitz
![Page 43: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/43.jpg)
Subsampling without filtering
1/4 (2x subsample) 1/8 (4x subsample)1/2
Steve Seitz
![Page 44: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/44.jpg)
Subsampling and aliasing
![Page 45: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/45.jpg)
• 1D example (sinewave):
Source: S. Marschner
Aliasing problem
![Page 46: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/46.jpg)
Source: S. Marschner
• 1D example (sinewave):
Aliasing problem
![Page 47: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/47.jpg)
What’s happening?
Input signal:
x = 0:.05:5; imagesc(sin((2.^x).*x))
Plot as image:
Aliasing!
Not enough
samples
![Page 48: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/48.jpg)
• Sub-sampling may be dangerous….
• Characteristic errors may appear: • “Car wheels rolling the wrong way in movies”
• “Striped shirts look funny on color television”
• “Checkerboards disintegrate in ray tracing”
Source: D. Forsyth; Wikipedia
Aliasing problem
Moiré patterns!
![Page 49: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/49.jpg)
Source: A. Efros
Aliasing in graphics
![Page 50: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/50.jpg)
Gong 96, 1932, Claude Tousignant, Musée des Beaux-Arts de Montréal
![Page 51: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/51.jpg)
Aliasing in video
Slide by Steve Seitz
![Page 52: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/52.jpg)
Videos
[YouTube; JoinBuzzirk; phrancque]
![Page 53: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/53.jpg)
![Page 54: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/54.jpg)
http://blogs.discovermagazine.com/badastronomy/2009/06/24/the-blue-and-the-green/
The blue and green colors are actually the same
![Page 55: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/55.jpg)
• When sampling a signal at discrete intervals, the sampling frequency must be 2 fmax
• fmax = max frequency of the input signal
• This will allows to reconstruct the original perfectly from the sampled version
good
bad
Nyquist-Shannon Sampling Theorem
![Page 56: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/56.jpg)
How to fix aliasing?
Solutions?
![Page 57: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/57.jpg)
Better sensors
Solutions:
• Sample more often
![Page 58: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/58.jpg)
Anti-aliasing
Solutions:
• Sample more often
• Get rid of all frequencies that are greater than half the new sampling frequency– Will lose information
– But it’s better than aliasing
– Apply a smoothing (low pass) filter
Hays
![Page 59: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/59.jpg)
Anti-aliasing
Forsyth and Ponce 2002
![Page 60: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/60.jpg)
Subsampling without filtering
1/4 (2x subsample) 1/8 (4x subsample)1/2
Steve Seitz
![Page 61: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/61.jpg)
Subsampling with Gaussian pre-filtering
G 1/4 G 1/8Gaussian 1/2
Steve SeitzGaussian Pyramid [Burt and Adelson, 1983]
![Page 62: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/62.jpg)
IMAGE FILTERING
![Page 63: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/63.jpg)
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
![Page 64: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/64.jpg)
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
],[],[],[,
lnkmIlkfnmhlk
++=I=imagef=filterh=output
2d coords=m,n2d coords=k,l
[ ] [ ][ ]
![Page 65: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/65.jpg)
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Example: box filter
![Page 66: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/66.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
Credit: S. Seitz
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 67: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/67.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 68: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/68.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 69: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/69.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 70: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/70.jpg)
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 71: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/71.jpg)
0 10 20 30 30
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
?
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 72: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/72.jpg)
0 10 20 30 30
50
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
[.,.]h[.,.]I
Image filtering
111
111
111
],[ f
Credit: S. Seitz
?
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 73: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/73.jpg)
0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 90 0 90 90 90 0 0
0 0 0 90 90 90 90 90 0 0
0 0 0 0 0 0 0 0 0 0
0 0 90 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0
0 10 20 30 30 30 20 10
0 20 40 60 60 60 40 20
0 30 60 90 90 90 60 30
0 30 50 80 80 90 60 30
0 30 50 80 80 90 60 30
0 20 30 50 50 60 40 20
10 20 30 30 30 30 20 10
10 10 10 0 0 0 0 0
[.,.]h[.,.]I
Image filtering111
111
111],[ f
Credit: S. Seitz
],[],[],[,
lnkmIlkfnmhlk
++=
![Page 74: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/74.jpg)
What does it do?
• Replaces each pixel with
an average of its
neighborhood
• Achieve smoothing effect
(remove sharp features)
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Box Filter
![Page 75: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/75.jpg)
What does it do?
• Replaces each pixel with
an average of its
neighborhood
• Achieve smoothing effect
(remove sharp features)
• Why does it sum to one?
111
111
111
Slide credit: David Lowe (UBC)
],[ f
Box Filter
![Page 76: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/76.jpg)
Smoothing with box filterJames Hays
![Page 77: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/77.jpg)
Image filtering
• Image filtering:
– Compute function of local neighborhood at each position
• Really important!
– Enhance images• Denoise, resize, increase contrast, etc.
– Extract information from images• Texture, edges, distinctive points, etc.
– Detect patterns• Template matching
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
![Page 78: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/78.jpg)
Think-Pair-Share time
000
010
000
1.
000
100
000
2.
-101
-202
-101
111
111
111
000
020
000
-
3.
4.
![Page 79: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/79.jpg)
1. Practice with linear filters
000
010
000
Original
?
Source: D. Lowe
![Page 80: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/80.jpg)
1. Practice with linear filters
000
010
000
Original Filtered
(no change)
Source: D. Lowe
![Page 81: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/81.jpg)
2. Practice with linear filters
000
100
000
Original
?
Source: D. Lowe
![Page 82: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/82.jpg)
2. Practice with linear filters
000
100
000
Original Shifted left
By 1 pixel
Source: D. Lowe
![Page 83: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/83.jpg)
3. Practice with linear filters
-101
-202
-101
Vertical Edge
(absolute value)
Sobel
David Lowe
![Page 84: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/84.jpg)
3. Practice with linear filters
-1-2-1
000
121
Horizontal Edge
(absolute value)
Sobel
David Lowe
![Page 85: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/85.jpg)
4. Practice with linear filters
Original
111
111
111
000
020
000
- ?
(Note that filter sums to 1)
Source: D. Lowe
![Page 86: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/86.jpg)
4. Practice with linear filters
Original
111
111
111
000
020
000
-
Sharpening filter
- Accentuates differences with local
average
Source: D. Lowe
![Page 87: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/87.jpg)
4. Practice with linear filters
Source: D. Lowe
![Page 88: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/88.jpg)
Correlation and Convolution
• 2d correlation
],[],[],[,
lnkmIlkfnmhlk
++=
James Hays
h=filter2(f,I); or h=imfilter(I,f);
![Page 89: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/89.jpg)
Correlation and Convolution
• 2d correlation
• 2d convolution
],[],[],[,
lnkmIlkfnmhlk
−−=
],[],[],[,
lnkmIlkfnmhlk
++=
conv2(I,f)is the same as filter2(rot90(f,2),I)
Correlation and convolution are identical when the filter is symmetric.
James Hays
h=filter2(f,I); or h=imfilter(I,f);
h=conv2(f,I); or h=imfilter(I,f,’conv’);
![Page 90: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/90.jpg)
Key properties of linear filters
Linearity:imfilter(I, f1 + f2) =
imfilter(I,f1) + imfilter(I,f2)
Shift invariance:Same behavior regardless of pixel locationimfilter(I,shift(f)) = shift(imfilter(I,f))
Any linear, shift-invariant operator can be represented as a convolution.
Source: S. Lazebnik
![Page 91: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/91.jpg)
Convolution properties
Commutative: a * b = b * a
– Conceptually no difference between filter and signal
– But particular filtering implementations might break this equality, e.g., image edges
Associative: a * (b * c) = (a * b) * c
– Often apply several filters one after another: (((a * b1) * b2) * b3)
– This is equivalent to applying one filter: a * (b1 * b2 * b3)
Source: S. Lazebnik
![Page 92: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/92.jpg)
Convolution properties
Commutative: a * b = b * a
– Conceptually no difference between filter and signal
– But particular filtering implementations might break this equality, e.g., image edges
Associative: a * (b * c) = (a * b) * c
– Often apply several filters one after another: (((a * b1) * b2) * b3)
– This is equivalent to applying one filter: a * (b1 * b2 * b3)
– Correlation is _not_ associative (rotation effect)
– Why important?
Source: S. Lazebnik
![Page 93: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/93.jpg)
Convolution properties• Commutative: a * b = b * a
– Conceptually no difference between filter and signal– But particular filtering implementations might break this equality,
e.g., image edges
• Associative: a * (b * c) = (a * b) * c– Often apply several filters one after another: (((a * b1) * b2) * b3)– This is equivalent to applying one filter: a * (b1 * b2 * b3)– Correlation is _not_ associative (rotation effect)– Why important?
• Distributes over addition: a * (b + c) = (a * b) + (a * c)
• Scalars factor out: ka * b = a * kb = k (a * b)
• Identity: unit impulse e = [0, 0, 1, 0, 0], a * e = a
Source: S. Lazebnik
![Page 94: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/94.jpg)
• Weight contributions of neighboring pixels by nearness
0.003 0.013 0.022 0.013 0.0030.013 0.059 0.097 0.059 0.0130.022 0.097 0.159 0.097 0.0220.013 0.059 0.097 0.059 0.0130.003 0.013 0.022 0.013 0.003
5 x 5, = 1
Slide credit: Christopher Rasmussen
Important filter: Gaussian
xy
x
y
![Page 95: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/95.jpg)
Smoothing with box filterJames Hays
111
111
111
],[ f
![Page 96: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/96.jpg)
Smoothing with Gaussian filterJames Hays
![Page 97: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/97.jpg)
Gaussian filters
• Remove “high-frequency” components from the image (low-pass filter)– Images become more smooth
• Gaussian convolved with Gaussian……is another Gaussian
– So can smooth with small-width kernel, repeat, and get same result as larger-width kernel would have
– Convolving two times with Gaussian kernel of width σ is same as convolving once with kernel of width σ√2
• Separable kernel– Factors into product of two 1D Gaussians
Source: K. Grauman
![Page 98: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/98.jpg)
Separability of the Gaussian filter
Source: D. Lowe
![Page 99: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/99.jpg)
Separability example
*
*
=
=
2D convolution
(center location only)
Source: K. Grauman
The filter factors
into a product of 1D
filters:
Perform convolution
along rows:
Followed by convolution
along the remaining column:
=
![Page 100: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/100.jpg)
Separability
Why is separability useful in practice?
![Page 101: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/101.jpg)
Separability
Why is separability useful in practice?
MxN image, PxQ filter
• 2D convolution: ~MNPQ multiply-adds
• Separable 2D: ~MN(P+Q) multiply-adds
Speed up = PQ/(P+Q)
9x9 filter = ~4.5x faster
![Page 102: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/102.jpg)
How big should the filter be?• Values at edges should be near zero
• Gaussians have infinite extent…
• Rule of thumb for Gaussian: set filter half-width to about 3 σ
Practical matters
James Hays
![Page 103: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/103.jpg)
Practical matters
• What about near the edge?
– the filter window falls off the edge of the image
– need to extrapolate
– methods:
• clip filter (black)
• wrap around
• copy edge
• reflect across edge
Source: S. Marschner
![Page 104: CSCI 1290: Comp Photo - Brown Universitycs.brown.edu/courses/csci1290/lectures/02_ImagingBasics.pdf · •Computer says ZInf combinations. •Some depiction of all possible scenes](https://reader034.vdocuments.mx/reader034/viewer/2022050212/5f5e9548a8182c7b0e63ed90/html5/thumbnails/104.jpg)
Project 1: Image Alignment to Recombine the Prokudin-Gorsky Photo Collection
Make alignment operation fast using image pyramid.
Remove aliasing from pyramid by filtering.
Coarse-to-fine strategy:
0. Build Gaussian Pyramid
1. Compare at low resolution; find alignment
2. At higher resolution, search around this alignment
3. Repeat