cs6640 computational photography - cornell university · cs6640 computational photography 7....

of 127 /127
© 2012 Steve Marschner CS6640 Computational Photography 7. Digital camera processing pipeline 1

Author: others

Post on 26-Apr-2020

87 views

Category:

Documents


0 download

Embed Size (px)

TRANSCRIPT

  • © 2012 Steve Marschner

    CS6640 Computational Photography

    7. Digital camera processing pipeline

    1

  • Cornell CS6640 Fall 2012

    Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map

    2

  • Cornell CS6640 Fall 2012

    Demosaicking• First question: how to spell it

    I cite “picnicking”• Each photosite

    senses only onecolor

    • We need three measurements ateach pixel

    3http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html

    http://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.htmlhttp://www.currentprotocols.com/WileyCDA/CPUnit/refId-ns0204.html

  • Cornell CS6640 Fall 2012

    Bayer array• Simple solution: make a half-resolution image

    • Want sensor-resolution image? Make up 2/3 of the data!

    4

    ? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 5

  • Cornell CS6640 Fall 2012 6

  • Cornell CS6640 Fall 2012 6

    bayer

  • Cornell CS6640 Fall 2012 6

    bayerbayer

  • Cornell CS6640 Fall 2012 7

  • Cornell CS6640 Fall 2012 8

  • Cornell CS6640 Fall 2012 8

    bayer

  • Cornell CS6640 Fall 2012 8

    bayerbayer

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this?

    9

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution

    9

  • Cornell CS6640 Fall 2012

    Half-resolution demosaic• Idea 1: treat each block of four pixels as a pixel

    Easy to code up in one line of Matlab. But what is wrong with this?1. throws away too much resolution2. produces subpixel shifts in color planes!

    9

  • Cornell CS6640 Fall 2012 10

  • Cornell CS6640 Fall 2012 10

    bayer

  • Cornell CS6640 Fall 2012 10

    bayerblock

  • Cornell CS6640 Fall 2012 10

    bayerblockbayer

  • Cornell CS6640 Fall 2012 10

    bayerblockbayerblock

  • Cornell CS6640 Fall 2012 11

  • Cornell CS6640 Fall 2012 11

    bayer

  • Cornell CS6640 Fall 2012 11

    bayerblock

  • Cornell CS6640 Fall 2012 11

    bayerblockbayer

  • Cornell CS6640 Fall 2012 11

    bayerblockbayerblock

  • Cornell CS6640 Fall 2012

    Centered half-resolution• Average pixels in groups that all have the same

    “center of gravity”avoids major color fringing

    12

    ? ? ?? ? ?? ?? ? ?? ? ?? ? ?? ? ?? ? ?? ? ?

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    ?

    ????

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 13

  • Cornell CS6640 Fall 2012 13

    bayer

  • Cornell CS6640 Fall 2012 13

    bayerblock

  • Cornell CS6640 Fall 2012 13

    bayerblockcentered

  • Cornell CS6640 Fall 2012 13

    bayerblockcenteredbayer

  • Cornell CS6640 Fall 2012 13

    bayerblockcenteredbayerblock

  • Cornell CS6640 Fall 2012 13

    bayerblockcenteredbayerblockcentered

  • Cornell CS6640 Fall 2012 14

    naïve full-resdcrawnaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayernaïve full-resdcrawnaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblocknaïve full-resdcrawnaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcrawnaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcrawbayernaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcrawbayerblocknaïve full-resdcraw

  • Cornell CS6640 Fall 2012 14

    bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw

  • Cornell CS6640 Fall 2012

    Naïve full-resolution interpolation• What if we don’t want to throw away so much sharpness?

    15

    ? ? ?? ?? ? ?? ? ?? ? ?? ?? ? ?? ? ?? ? ?

    ??

    ?

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    ??

    ???

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

    ??????

    ?

    ?

    ?

  • Cornell CS6640 Fall 2012 16

    dcrawdcraw

  • Cornell CS6640 Fall 2012 16

    bayerdcrawdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockdcrawdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcentereddcrawdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcrawdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcrawbayerdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcrawbayerblockdcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcrawbayerblockcentereddcraw

  • Cornell CS6640 Fall 2012 16

    bayerblockcenterednaïve full-resdcrawbayerblockcenterednaïve full-resdcraw

  • Cornell CS6640 Fall 2012 17

  • Cornell CS6640 Fall 2012 17

    bayer

  • Cornell CS6640 Fall 2012 17

    bayerblock

  • Cornell CS6640 Fall 2012 17

    bayerblockcentered

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-resbayer

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-resbayerblock

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-resbayerblockcentered

  • Cornell CS6640 Fall 2012 17

    bayerblockcenterednaïve full-resbayerblockcenterednaïve full-res

  • slide by Frédo Durand, M

    IT

    Results of simple linear

  • slide by Frédo Durand, M

    IT

    Results - not perfect

  • slide by Frédo Durand, M

    IT

    Questions?

  • slide by Frédo Durand, M

    IT

    The problem• Imagine a black-on-white corner• Let’s focus on the green channel for now

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

  • slide by Frédo Durand, M

    IT

    The problem• Imagine a black-on-white corner

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

  • slide by Frédo Durand, M

    IT

    The problem• Imagine a black-on-white corner

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

    00

    0

    1

    .25.75

    .25

    .751

    0

    .25

    1

    1

    .1

    1

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

    .25 .75.75

  • slide by Frédo Durand, M

    IT

    The problem• Imagine a black-on-white corner

    ??

    ??

    ??

    ??

    ??

    ??

    ?

    ?

    ?

    ?

    ?

    ?

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

    00

    0

    1

    .25.75

    .25

    .751

    0

    .25

    1

    1

    .1

    1

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

    .25 .75.75

  • slide by Frédo Durand, M

    IT

    Yep, that’s what we saw

  • slide by Frédo Durand, M

    IT

    Green channel

  • slide by Frédo Durand, M

    IT

    Edge-based Demosaicking

  • slide by Frédo Durand, M

    IT

    Idea• Take into account structure in image

    - Here, 1D edges• Interpolate along preferred direction

    - In our case, only use 2 neighbors

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

  • slide by Frédo Durand, M

    IT

    How do we decide• Look at the similarity of recorded neighbors

    - Compare |up-down| and |right-le|- Be smart- See pset 4

    • Called edge-based demosaicking

    0 0 10 0 1

    0 0 10 0 1

    1 1 11 1 1

  • slide by Frédo Durand, M

    IT

    Green channel -- naive

  • slide by Frédo Durand, M

    IT

    Green channel -- edge-based

  • slide by Frédo Durand, M

    IT

    Challenge with other channels

  • slide by Frédo Durand, M

    IT

    Problem• What do we do with red and blue? • We could apply the edge-based principle• But we’re missing more information• But color transitions might be shifted

  • slide by Frédo Durand, M

    IT

    Example• Black on white corner• Notion of edge-based unclear for pixels in

    empty rows or columns

    ?

    ?

    ?

  • slide by Frédo Durand, M

    IT

    Example• Black on white corner• Even if we imagine we can do some decent job

    for each channel

  • slide by Frédo Durand, M

    IT

    Example• Black on white corner• Even if we imagine we can do some decent job

    for each channel• The channels don’t line up

    - Because they are not recorded at the same location

  • slide by Frédo Durand, M

    IT

    • Bad color fringes!

    Example

  • slide by Frédo Durand, M

    IT

    Recall color artifacts

  • slide by Frédo Durand, M

    IT

    Green-based Demosaicking

  • slide by Frédo Durand, M

    IT

    Green-based demosaicking• Green is a better color channel

    - Twice as many pixels- Oen better SNR- We know how to do edge-based green interpolation

    • Do the best job you can and get high resolution from green

    • Then use green to guide red & blue interpolation

  • slide by Frédo Durand, M

    IT

    Interpolate difference to green• Interpolate green

    - using e.g. edge-based• For recorded red pixels

    - compute R-G• At empty pixels

    - Interpolate R-G naively- Add G

    • Same for blue

  • slide by Frédo Durand, M

    IT

    Black on white corner

  • slide by Frédo Durand, M

    IT

    Measurements

  • slide by Frédo Durand, M

    IT

    Edge-based green

  • slide by Frédo Durand, M

    IT

    Red-Green difference• Zero everywhere!

  • slide by Frédo Durand, M

    IT

    Red-Green interpolation• Easy!

  • slide by Frédo Durand, M

    IT

    Add back green

  • slide by Frédo Durand, M

    IT

    Same for blue

  • slide by Frédo Durand, M

    IT

    Fully naive

  • slide by Frédo Durand, M

    IT

    Edge-based green, naive red blue

  • slide by Frédo Durand, M

    IT

    Green-based blue and red

  • slide by Frédo Durand, M

    IT

    Still not 100% perfect• But will be good enough for pset 4

  • Cornell CS6640 Fall 2012 53

  • Cornell CS6640 Fall 2012 53

    bayer

  • Cornell CS6640 Fall 2012 53

    bayerblock

  • Cornell CS6640 Fall 2012 53

    bayerblockcentered

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-based

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcraw

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayer

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblock

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based

  • Cornell CS6640 Fall 2012 53

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw

  • Cornell CS6640 Fall 2012 54

  • Cornell CS6640 Fall 2012 54

    bayer

  • Cornell CS6640 Fall 2012 54

    bayerblock

  • Cornell CS6640 Fall 2012 54

    bayerblockcentered

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-based

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcraw

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayer

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblock

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcentered

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-res

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-based

  • Cornell CS6640 Fall 2012 54

    bayerblockcenterednaïve full-resedge-baseddcrawbayerblockcenterednaïve full-resedge-baseddcraw

  • Cornell CS6640 Fall 2012 55

    Noise reduction• Users want noise out of their images

    shot noise increases as sqrt(exposure)relative shot noise decreases as sqrt(exposure)relative readout noise increases prop. to exposurehigher ISO settings try to reduce relative readout noise

    • Basic approach: average together measurements• Be clever, spatially, to avoid blurring image structure

    • One approach: bilateral filtermore on this later

    • Camera makes all have their proprietary methods

  • Cornell CS6640 Fall 2012 56

    worth a look:http://www.dpreview.com/reviews/nikon-d3200/12

    dpreview.comcamera noise

    comparison tool

  • Cornell CS6640 Fall 2012 56

    worth a look:http://www.dpreview.com/reviews/nikon-d3200/12

    dpreview.comcamera noise

    comparison tool

  • slide by Frédo Durand, M

    IT

    Denoising & Demosaicking• http://research.microsoft.com/en-us/UM/

    people/yasumat/papers/lowlight_CVPR11.pdf

    http://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdfhttp://research.microsoft.com/en-us/UM/people/yasumat/papers/lowlight_CVPR11.pdf

  • Cornell CS6640 Fall 2012

    Camera processing pipeline• read image out from sensor —see Sensors lecture• optional: HDR assembly —see Homework 2• color balance —see Color lecture• demosaic• noise processing• color matrix —see Color lecture• tone map —more on this in the project and papers

    58

  • slide by Frédo Durand, M

    IT

    Links from Frédo- http://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdf

    • http://www.unc.edu/~rjean/demosaicing/demosaicing.pdf• http://www.pages.drexel.edu/~par24/rawhistogram/

    40D_Demosaicing/40D_DemosaicingArtifacts.html• http://www.guillermoluijk.com/tutorial/dcraw/index_en.htm• http://www.cambridgeincolour.com/tutorials/RAW-file-

    format.htm• http://www.cambridgeincolour.com/tutorials/camera-

    sensors.htm

    http://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdfhttp://www.csee.wvu.edu/~xinl/papers/demosaicing_survey.pdfhttp://www.unc.edu/~rjean/demosaicing/demosaicing.pdfhttp://www.unc.edu/~rjean/demosaicing/demosaicing.pdfhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.pages.drexel.edu/~par24/rawhistogram/40D_Demosaicing/40D_DemosaicingArtifacts.htmlhttp://www.guillermoluijk.com/tutorial/dcraw/index_en.htmhttp://www.guillermoluijk.com/tutorial/dcraw/index_en.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/RAW-file-format.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htmhttp://www.cambridgeincolour.com/tutorials/camera-sensors.htm

  • slide by Frédo Durand, M

    IT

    More• http://www.ists.dartmouth.edu/library/

    edf0205.pdf• http://ieeexplore.ieee.org/

    iel5/79/30519/01407714.pdf?isnumber=30519&prod=JNL&arnumber=1407714&arSt=+44&ared=+54&arAuthor=Gunturk%2C+B.K.%3B+Glotzbach%2C+J.%3B+Altunbasak%2C+Y.%3B+Schafer%2C+R.W.%3B+Mersereau%2C+R.M.

    http://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdfhttp://www.ists.dartmouth.edu/library/edf0205.pdf