cs6640 computational photography - cornell university · cs6640 computational photography 7....
Embed Size (px)
TRANSCRIPT
-
© 2012 Steve Marschner
CS6640 Computational Photography
7. Digital camera processing pipeline
1
-
Cornell CS6640 Fall 2012
Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map
2
-
Cornell CS6640 Fall 2012
Demosaicking• First question: how to spell it
I cite “picnicking”• Each photosite
senses only onecolor
• We need three measurements ateach pixel
3http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html
http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.htmlhttp://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html
-
Cornell CS6640 Fall 2012
Bayer array• Simple solution: make a half-resolution image
• Want sensor-resolution image? Make up 2/3 of the data!
4
? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?
??
??
??
??
??
??
?
?
?
?
?
?
??????
?
?
?
??????
?
?
?
??????
?
?
?
-
Cornell CS6640 Fall 2012 5
-
Cornell CS6640 Fall 2012 6
-
Cornell CS6640 Fall 2012 6
bayer
-
Cornell CS6640 Fall 2012 6
bayerbayer
-
Cornell CS6640 Fall 2012 7
-
Cornell CS6640 Fall 2012 8
-
Cornell CS6640 Fall 2012 8
bayer
-
Cornell CS6640 Fall 2012 8
bayerbayer
-
Cornell CS6640 Fall 2012
Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel
Easy to code up in one line of Matlab. But what is wrong with this?
9
-
Cornell CS6640 Fall 2012
Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel
Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution
9
-
Cornell CS6640 Fall 2012
Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel
Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution2. produces subpixel shifts in color planes!
9
-
Cornell CS6640 Fall 2012 10
-
Cornell CS6640 Fall 2012 10
bayer
-
Cornell CS6640 Fall 2012 10
bayerblock
-
Cornell CS6640 Fall 2012 10
bayerblockbayer
-
Cornell CS6640 Fall 2012 10
bayerblockbayerblock
-
Cornell CS6640 Fall 2012 11
-
Cornell CS6640 Fall 2012 11
bayer
-
Cornell CS6640 Fall 2012 11
bayerblock
-
Cornell CS6640 Fall 2012 11
bayerblockbayer
-
Cornell CS6640 Fall 2012 11
bayerblockbayerblock
-
Cornell CS6640 Fall 2012
Centered half-resolution• Average pixels in groups that all have the same
“center of gravity”avoids major color fringing
12
? ? ?? ? ?? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?
??
??
??
??
??
??
?
?
?
?
?
?
?
????
?
?
?
??????
?
?
?
??????
?
?
?
-
Cornell CS6640 Fall 2012 13
-
Cornell CS6640 Fall 2012 13
bayer
-
Cornell CS6640 Fall 2012 13
bayerblock
-
Cornell CS6640 Fall 2012 13
bayerblockcentered
-
Cornell CS6640 Fall 2012 13
bayerblockcenteredbayer
-
Cornell CS6640 Fall 2012 13
bayerblockcenteredbayerblock
-
Cornell CS6640 Fall 2012 13
bayerblockcenteredbayerblockcentered
-
Cornell CS6640 Fall 2012 14
naïve full-resdcrawnaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayernaïve full-resdcrawnaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayerblocknaïve full-resdcrawnaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayerblockcenterednaïve full-resdcrawnaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayerblockcenterednaïve full-resdcrawbayernaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayerblockcenterednaïve full-resdcrawbayerblocknaïve full-resdcraw
-
Cornell CS6640 Fall 2012 14
bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw
-
Cornell CS6640 Fall 2012
Naïve full-resolution interpolation• What if we don’t want to throw away so much sharpness?
15
? ? ?? ?? ? ?? ? ?? ? ?? ?? ? ?? ? ?? ? ?
??
?
??
??
??
??
?
?
?
?
?
?
??
???
?
?
?
??????
?
?
?
??????
?
?
?
-
Cornell CS6640 Fall 2012 16
dcrawdcraw
-
Cornell CS6640 Fall 2012 16
bayerdcrawdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockdcrawdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcentereddcrawdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcenterednaïve full-resdcrawdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcenterednaïve full-resdcrawbayerdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcenterednaïve full-resdcrawbayerblockdcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcenterednaïve full-resdcrawbayerblockcentereddcraw
-
Cornell CS6640 Fall 2012 16
bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw
-
Cornell CS6640 Fall 2012 17
-
Cornell CS6640 Fall 2012 17
bayer
-
Cornell CS6640 Fall 2012 17
bayerblock
-
Cornell CS6640 Fall 2012 17
bayerblockcentered
-
Cornell CS6640 Fall 2012 17
bayerblockcenterednaïve full-res
-
Cornell CS6640 Fall 2012 17
bayerblockcenterednaïve full-resbayer
-
Cornell CS6640 Fall 2012 17
bayerblockcenterednaïve full-resbayerblock
-
Cornell CS6640 Fall 2012 17
bayerblockcenterednaïve full-resbayerblockcentered
-
Cornell CS6640 Fall 2012 17
bayerblockcenterednaïve full-resbayerblockcenterednaïve full-res
-
slide by Frédo Durand, M
IT
Results of simple linear
-
slide by Frédo Durand, M
IT
Results - not perfect
-
slide by Frédo Durand, M
IT
Questions?
-
slide by Frédo Durand, M
IT
The problem• Imagine a black-on-white corner• Let’s focus on the green channel for now
??
??
??
??
??
??
?
?
?
?
?
?
-
slide by Frédo Durand, M
IT
The problem• Imagine a black-on-white corner
??
??
??
??
??
??
?
?
?
?
?
?
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
-
slide by Frédo Durand, M
IT
The problem• Imagine a black-on-white corner
??
??
??
??
??
??
?
?
?
?
?
?
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
00
0
1
.25.75
.25
.751
0
.25
1
1
.1
1
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
.25 .75.75
-
slide by Frédo Durand, M
IT
The problem• Imagine a black-on-white corner
??
??
??
??
??
??
?
?
?
?
?
?
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
00
0
1
.25.75
.25
.751
0
.25
1
1
.1
1
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
.25 .75.75
-
slide by Frédo Durand, M
IT
Yep, that’s what we saw
-
slide by Frédo Durand, M
IT
Green channel
-
slide by Frédo Durand, M
IT
Edge-based Demosaicking
-
slide by Frédo Durand, M
IT
Idea• Take into account structure in image
- Here, 1D edges• Interpolate along preferred direction
- In our case, only use 2 neighbors
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
-
slide by Frédo Durand, M
IT
How do we decide• Look at the similarity of recorded neighbors
- Compare |up-down| and |right-le|- Be smart- See pset 4
• Called edge-based demosaicking
0 0 10 0 1
0 0 10 0 1
1 1 11 1 1
-
slide by Frédo Durand, M
IT
Green channel -- naive
-
slide by Frédo Durand, M
IT
Green channel -- edge-based
-
slide by Frédo Durand, M
IT
Challenge with other channels
-
slide by Frédo Durand, M
IT
Problem• What do we do with red and blue? • We could apply the edge-based principle• But we’re missing more information• But color transitions might be shifted
-
slide by Frédo Durand, M
IT
Example• Black on white corner• Notion of edge-based unclear for pixels in
empty rows or columns
?
?
?
-
slide by Frédo Durand, M
IT
Example• Black on white corner• Even if we imagine we can do some decent job
for each channel
-
slide by Frédo Durand, M
IT
Example• Black on white corner• Even if we imagine we can do some decent job
for each channel• The channels don’t line up
- Because they are not recorded at the same location
-
slide by Frédo Durand, M
IT
• Bad color fringes!
Example
-
slide by Frédo Durand, M
IT
Recall color artifacts
-
slide by Frédo Durand, M
IT
Green-based Demosaicking
-
slide by Frédo Durand, M
IT
Green-based demosaicking• Green is a better color channel
- Twice as many pixels- Oen better SNR- We know how to do edge-based green interpolation
• Do the best job you can and get high resolution from green
• Then use green to guide red & blue interpolation
-
slide by Frédo Durand, M
IT
Interpolate difference to green• Interpolate green
- using e.g. edge-based• For recorded red pixels
- compute R-G• At empty pixels
- Interpolate R-G naively- Add G
• Same for blue
-
slide by Frédo Durand, M
IT
Black on white corner
-
slide by Frédo Durand, M
IT
Measurements
-
slide by Frédo Durand, M
IT
Edge-based green
-
slide by Frédo Durand, M
IT
Red-Green difference• Zero everywhere!
-
slide by Frédo Durand, M
IT
Red-Green interpolation• Easy!
-
slide by Frédo Durand, M
IT
Add back green
-
slide by Frédo Durand, M
IT
Same for blue
-
slide by Frédo Durand, M
IT
Fully naive
-
slide by Frédo Durand, M
IT
Edge-based green, naive red blue
-
slide by Frédo Durand, M
IT
Green-based blue and red
-
slide by Frédo Durand, M
IT
Still not 100% perfect• But will be good enough for pset 4
-
Cornell CS6640 Fall 2012 53
-
Cornell CS6640 Fall 2012 53
bayer
-
Cornell CS6640 Fall 2012 53
bayerblock
-
Cornell CS6640 Fall 2012 53
bayerblockcentered
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-res
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-based
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcraw
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayer
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayerblock
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based
-
Cornell CS6640 Fall 2012 53
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw
-
Cornell CS6640 Fall 2012 54
-
Cornell CS6640 Fall 2012 54
bayer
-
Cornell CS6640 Fall 2012 54
bayerblock
-
Cornell CS6640 Fall 2012 54
bayerblockcentered
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-res
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-based
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcraw
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayer
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayerblock
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based
-
Cornell CS6640 Fall 2012 54
bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw
-
Cornell CS6640 Fall 2012 55
Noise reduction• Users want noise out of their images
shot noise increases as sqrt(exposure)relative shot noise decreases as sqrt(exposure)relative readout noise increases prop. to exposurehigher ISO settings try to reduce relative readout noise
• Basic approach: average together measurements• Be clever, spatially, to avoid blurring image structure
• One approach: bilateral filtermore on this later
• Camera makes all have their proprietary methods
-
Cornell CS6640 Fall 2012 56
worth a look:http://www.dpreview.com/reviews/nikon-d3200/12
dpreview.comcamera noise
comparison tool
-
Cornell CS6640 Fall 2012 56
worth a look:http://www.dpreview.com/reviews/nikon-d3200/12
dpreview.comcamera noise
comparison tool
-
slide by Frédo Durand, M
IT
Denoising & Demosaicking• http://research.microsoft.com/en-us/UM/
people/yasumat/papers/lowlight_CVPR11.pdf
http://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdf
-
Cornell CS6640 Fall 2012
Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map —more on this in the project and papers
58
-
slide by Frédo Durand, M
IT
Links from Frédo- http://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdf
• http://www.unc.edu/~rjean/demosaicing/demosaicing.pdf• http://www.pages.drexel.edu/~par24/rawhistogram/
40D_Demosaicing/40D_DemosaicingArtifacts.html• http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm• http://www.cambridgeincolour.com/tutorials/RAW-file-
format.htm• http://www.cambridgeincolour.com/tutorials/camera-
sensors.htm
http://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdfhttp://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdfhttp://www.unc.edu/~rjean/demosaicing/demosaicing.pdfhttp://www.unc.edu/~rjean/demosaicing/demosaicing.pdfhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.guillermoluijk.com/tutorial/dcraw/index_en.htmhttp://www.guillermoluijk.com/tutorial/dcraw/index_en.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htm
-
slide by Frédo Durand, M
IT
More• http://www.ists.dartmouth.edu/library/
edf0205.pdf• http://ieeexplore.ieee.org/
iel5/79/30519/01407714.pdf?isnumber=30519&prod=JNL&arnumber=1407714&arSt=+44&ared=+54&arAuthor=Gunturk%2C+B.K.%3B+Glotzbach%2C+J.%3B+Altunbasak%2C+Y.%3B+Schafer%2C+R.W.%3B+Mersereau%2C+R.M.
http://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdf