computational photography and video - eth zurich€¦ · what is computational photography •...

of 89 /89
Computational Photography and Video Prof. Marc Pollefeys Dr. Gabriel Brostow

Author: vuhanh

Post on 25-Aug-2018

229 views

Category:

Documents


1 download

Embed Size (px)

TRANSCRIPT

  • ComputationalPhotographyandVideo

    Prof.MarcPollefeys

    Dr.GabrielBrostow

  • Todaysschedule

    IntroductionofComputationalPhotography Coursefacts Syllabus DigitalPhotography

  • Whatiscomputationalphotography

    Convergenceofimageprocessing,computervision,computergraphicsandphotography

    Digitalphotography: Simplyreplacestraditionalsensorsandrecordingbydigitaltechnology

    Involvesonlysimpleimageprocessing Computationalphotography

    Moreelaborateimagemanipulation,morecomputation

    Newtypesofmedia(panorama,3D,etc.) Cameradesignthattakecomputationintoaccount

    (severalslidesinspired/borrowedfromFredoDurand,MIT)

  • Tonemapping

    Before After

    DurandandDorsey.Siggraph02

    http://www-users.mat.uni.torun.pl/~olka/http://www.loc.gov/exhibits/empire/

  • Flash/NoFlash

    Petschniggetal.Siggraph04

  • Photomontage

    Agarwalaetal.Siggraph04

  • Panoramicimages

    BrownandLoweICCV03

  • Defocusmatting

    McGuireetal.Siggraph05

  • Videotextures

    Schoedletal.Siggraph00

  • Motionmagnification

    Liuetal.Siggraph05

  • Todaysschedule

    IntroductionofComputationalPhotography Coursefacts Syllabus DigitalPhotography

  • Administrivia

    Staff Prof.MarcPollefeys Dr.GabrielBrostow RolandAngst

    Timeandlocation: Lectures:Wednesday1315inCABH57 Exercises:Thursday1112inCABH56

    Webpage:http://www.inf.ethz.ch/personal/pomarc/courses/CompPhoto/

    http://www.inf.ethz.ch/personal/pomarc/courses/CompPhoto/

  • Courseorganization

    Lectures Exercises

    Firstafewassignments Laterprojectandpaperpresentations

    Smallclassproject Individualorsmallgroups

  • Gradingpolicy

    50%assignments 10%paperpresentation 40%classproject(report+presentation) Bonusforparticipation

    Noseparateexam

  • Textbook

    Notextbookrequired Slidesavailableoncoursewebpage Lotmoreresourcesonline

    Interestingreference:ComputationalPhotography:MasteringNewTechniquesforLenses,Lighting,andSensors.RaskarandTumblin,toappearsoon,AKPeters.

  • Todaysschedule

    IntroductionofComputationalPhotography Coursefacts Syllabus DigitalPhotography

  • Topics

    Imageformation,Imagesensor,Optics Pixelresolution,Exposure,Aperture,Focus,DynamicRange Color,whitebalance,Bayerpattern,demosaicking, Motionblur,shutter,deblurring Dynamicrange,HDRimaging,tonemapping,bilateralfiltering Imagepyramids,opticalflow,gradients Mattingandcompositing,graphcuts Warpingandmorphing,panoramas Texturesynthesis Illumination,flash/noflash,depthedges Codedaperture,defocus Videotextures,timelapse,videosummarization Lightfield imaging

  • Todaysschedule

    IntroductionofComputationalPhotography Coursefacts Syllabus DigitalPhotography

  • Overview

    Lensandviewpointdetermineperspective Apertureandshutterspeeddetermineexposure Apertureandothereffectsdeterminedepthoffield Sensorrecordsimage

    (thisandfollowingslidesborrowedfromFredoDurand,MIT)

  • Reference

    http://en.wikipedia.org/wiki/Lens_(optics)

    Theslidesuseillustrationsfromthesebooks

    http://en.wikipedia.org/wiki/Lens_(optics)

  • Morereferences

  • Plan

    Pinholeoptics Lenses Exposure

  • Whyistherenoimageonawhitepieceofpaper?

    Itreceiveslightfromalldirections

    FromPhotography,Londonetal.

  • Pinhole

    FromPhotography,Londonetal.

  • Focallength

    pinhole

    Film/sensor scene

    f

    s

  • Focallength:pinholeoptics

    Whathappenswhenthefocallengthisdoubled? Projectedobjectsize Amountoflightgathered

    pinhole

    Film/sensor scene

    f d

    s

    2f

    isdoubledisdividedby4

  • Questions?

  • Pinholesize?

    FromPhotography,Londonetal.

  • Diffractionlimit

    Optimalsizeforvisiblelight:sqrt(f)/28(inmillimiters)wherefisfocallength

    FromWandell

  • Problemwithpinhole?

    Notenoughlight! Diffractionlimitssharpness

  • Solution:refraction!

    FromPhotography,Londonetal.

  • Lenses

    gathermorelight!

    Butneedtobefocused

    FromPhotography,Londonetal.

  • Thinlensoptics

    Simplificationofgeometricalopticsforwellbehavedlenses Allparallelraysconvergetoonepointonaplanelocatedatthe

    focallengthf

    Allraysgoingthroughthecenterarenotdeviated Hencesameperspectiveaspinhole

    f

  • Howtotracerays

    Startbyraysthroughthecenter

  • Howtotracerays

    Startbyraysthroughthecenter Choosefocallength,traceparallels

    f

  • Howtotracerays Startbyraysthroughthecenter Choosefocallength,traceparallels Yougetthefocusplaneforagivensceneplane

    Allrayscomingfrompointsonaplaneparalleltothelensarefocusedonanotherplaneparalleltothelens

    f

  • Focusing

    Tofocuscloserthaninfinity Movethesensor/filmfurther thanthefocallength

    f

  • Thinlensformula

    fDD

  • Thinlensformula

    fDD

    Similartriangleseverywhere!

  • Thinlensformula

    fDD

    Similartriangleseverywhere!

    y

    y

    y/y=D/D

  • Thinlensformula

    fDD

    Similartriangleseverywhere!

    y

    y

    y/y=D/D

    y/y=(Df)/D

  • Thinlensformula

    fDD

    1D D

    1 1f

    + =

  • Minimumfocusingdistance

    Bysymmetry,anobjectatthefocallengthrequiresthefilmtobeatinfinity.

    sensor

    Raysfrominfinity

    Raysfromobjectatf

  • Extensionstubes

    Allowustoputsensorfartherfocuscloser

  • Fieldofview&focusing

    Whathappenstothefieldofviewwhenonefocusescloser? It'sreduced

    sensorfocused at infinity

    sensorfocused close

  • Questions?

    http://www.pinhole.cz/en/pinholecameras/dirkon_01.html

  • Focallengthinpractice

    24mm

    50mm

    135mm

  • Perspectivevs.viewpoint

    Telephotomakesiteasiertoselectbackground(asmallchangeinviewpointisabigchangeinbackground).

  • Perspectivevs.viewpoint

    Movescameraasyouzoomin HitchcockVertigoeffect

  • Perspectivevs.viewpoint

    Portrait:distortionwithwideangle Why?

    Wideangle Standard Telephoto

  • Focallength&sensor Whathappenswhenthefilmishalfthesize? Application:

    Realfilmis36x24mm Onthe20D,thesensoris22.5x15.0mm Conversionfactoronthe20D? OntheSD500,itis1/1.8"(7.18x5.32mm) Whatisthe7.723.1mmzoomontheSD500?

    pinhole

    Film/sensor

    scene

    f d

    s

  • Sensorsize

    Similartocropping

    source:canonredbook

  • http://www.photozone.de/3Technology/digital_1.htm

  • Recap

    Pinholeisthesimplestmodelofimageformation Lensesgathermorelight

    Butgetonlyoneplanefocused Focusbymovingsensor/film Cannotfocusinfinitelyclose

    Focallengthdeterminesfieldofview Fromwideangletotelephoto Dependsonsensorsize

    Moreinthelenslecture

  • Questions?

  • Exposure

    Gettherightamountoflighttosensor/film Twomainparameters:

    Shutterspeed Aperture(areaoflens)

  • Shutterspeed

    Controlshowlongthefilm/sensorisexposed Prettymuchlineareffectonexposure Usuallyinfractionofasecond:

    1/30,1/60,1/125,1/250,1/500 Getthepattern?

    Onanormallens,normalhumanscanhandholddownto1/60 Ingeneral,theruleofthumbsaysthatthelimitistheinverseoffocallength,e.g.1/500fora500mm

  • Maineffectofshutterspeed

    Motionblur

    FromPhotography,Londonetal.

  • Effectofshutterspeed

    Freezingmotion

    1/125 1/250 1/500 1/1000

    Walking people Running people Car Fast train

  • Shutter

    Varioustechnologies Goal:achieveuniformexposureacrossimage

    FromCameraTechnology,Goldberg

  • Flashsynchspeed?

    Fastestshutterspeedforwhichtheshutteropenscompletelyatsomeinstant.

    Forfasterspeeds,itopensandclosesatthesametimeandexposesaslit.

    Modernhighspeedflashsynchusesmultipleflashbursts

    FromPhotography,Londonetal.

  • Aperture

    Diameterofthelensopening(controlledbydiaphragm) Expressedasafractionoffocallength,infnumber

    f/2.0ona50mmmeansthattheapertureis25mm f/2.0ona100mmmeansthattheapertureis50mm

    Disconcerting:smallfnumber=bigaperture Whathappenstotheareaoftheaperturewhengoingfromf/2.0to

    f/4.0? Typicalfnumbersare

    f/2.0,f/2.8,f/4,f/5.6,f/8,f/11,f/16,f/22,f/32 Seethepattern?

  • Maineffectofaperture

    Depthoffield

    FromPhotography,Londonetal.

  • Depthoffield

    lenssensor

    Pointinfocus

    Objectwithtexture

  • Depthoffield

    lenssensor

    Pointinfocus

    Objectwithtexture

    Weallowforsometolerance

    lenssensor

    Pointinfocus

    Objectwithtexture

    Maxacceptablecircleofconfusion

    Depthoffield

    Depthoffocus

  • Depthoffield

    Whathappenswhenweclosetheaperturebytwostop? Aperturediameterisdividedbytwo Depthoffieldisdoubled

    lenssensor

    Pointinfocus

    Objectwithtexture

    Diaphragm

  • Depthoffield

    FromPhotography,Londonetal.

  • Depthoffield&focusingdistance

    Whathappenswhenwedividefocusingdistancebytwo? Similartriangles=>dividedbytwoaswell

    lenssensor

    Pointinfocus

    Halfdepthoffield

    Halfdepthoffield

  • Depthoffield&focusingdistance

    Whathappenswhenwedividefocusingdistancebytwo? Similartriangles=>dividedbytwoaswell

    FromPhotography,Londonetal.

  • SLRviewfinder&aperture

    Bydefault,anSLRalwaysshowsyouthebiggestaperture

    Brighterimage Shallowdepthoffieldhelpjudgefocus Depthoffieldpreviewbutton:

    Stopsdowntotheapertureyouhavechosen Darkerimage Largerdepthoffield

  • Questions?

  • Exposure

    Twomainparameters: Aperture(infstop) Shutterspeed(infractionofasecond)

    ReciprocityThesameexposureisobtainedwitanexposuretwiceaslongandanapertureareahalfasbig

    Hencesquarerootoftwoprogressionoffstopsvs.poweroftwoprogressionofshutterspeed

    Reciprocitycanfailforverylongexposures

    FromPhotography,Londonetal.

  • Reciprocity

    Assumeweknowhowmuchlightweneed Wehavethechoiceofaninfinityofshutterspeed/aperturepairs

    Whatwillguideourchoiceofashutterspeed? Freezemotionvs.motionblur,camerashake

    Whatwillguideourchoiceofanaperture? Depthoffield,diffractionlimit

    Oftenwemustcompromise Openmoretoenablefasterspeed(butshallowDoF)

  • FromPhotography,Londonetal.

  • FromPhotography,Londonetal.

  • FromPhotography,Londonetal.

  • Questions?

  • Metering

    Photosensitivesensorsmeasuresceneluminance UsuallyTTL(throughthelens) Simpleversion:centerweightedaverage

    Assumption?Failurecases? Usuallyassumesthatasceneis18%gray Problemwithdarkandbrightscenes

  • FromPhotography,Londonetal.

  • Metering

    Centeredaverage

    Spot

    Smartmetering Nikon3Dmatrix Canonevaluative

    Incident Measureincominglight

    Fromtheluminouslandscape

    Nextslide

    ChoiceonNikon

    http://www.mir.com.my//

  • Nikon3DColorMatrix

    http://www.mir.com.my/rb/photography/hardwares/classics/NikonF5/metering/

    Learningfromdatabaseof30,000photos

    Multiplecaptors(segments) Exposuredependson

    Brightnessfromeachsegments Color Contrast Distance Focus(whereisthesubject)

  • Exposure&metering

    Thecamerameteringsystemmeasureshowbrightthesceneis InApertureprioritymode,thephotographersetstheaperture,

    thecamerasetstheshutterspeed

    InShutterspeedprioritymode,thephotographerssetstheshutterspeedandthecameradeducestheaperture

    Inbothcases,reciprocityisexploited InProgrammode,thecameradecidesbothexposureand

    shutterspeed(middlevaluemoreorless)

    InManual,theuserdecideseverything(butcangetfeedback)

  • Prosandconsofvariousmodes

    Aperturepriority Directdepthoffieldcontrol Cons:canrequireimpossibleshutterspeed(e.g.withf/1.4fora

    brightscene) Shutterspeedpriority

    Directmotionblurcontrol Cons:canrequireimpossibleaperture(e.g.whenrequestinga

    1/1000speedforadarkscene) Notethatapertureissomewhatmorerestricted

    Program Almostnocontrol,butnoneedforneurons

    Manual Fullcontrol,buttakesmoretimeandthinking

  • Recap:Metering

    Measurescenebrightness Someadvancedmodesthattakemultiplesourcesofinformation

    Stillanopenproblem

  • Questions?

  • Sensitivity(ISO)

    Thirdvariableforexposure Lineareffect(200ISOneedshalfthelightas100ISO) Filmphotography:tradesensitivityforgrain

    Digitalphotography:tradesensitivityfornoise

    From

    dpreview.com

    Computational Photography and Video Todays scheduleWhat is computational photographyTone mappingFlash/No-FlashPhotomontagePanoramic imagesDefocus mattingSlide Number 9Video texturesMotion magnificationTodays scheduleAdministriviaCourse organizationGrading policyTextbookTodays scheduleTopics Todays scheduleOverviewReferenceMore referencesPlanWhy is there no image on a white piece of paper?PinholeFocal lengthFocal length: pinhole opticsQuestions?Pinhole size? Diffraction limitProblem with pinhole?Solution: refraction!Lenses Thin lens opticsHow to trace raysHow to trace raysHow to trace raysFocusingThin lens formulaThin lens formulaThin lens formulaThin lens formulaThin lens formulaMinimum focusing distanceExtensions tubesField of view & focusingQuestions?Focal length in practicePerspective vs. viewpointPerspective vs. viewpointPerspective vs. viewpointFocal length & sensorSensor sizehttp://www.photozone.de/3Technology/digital_1.htmRecapQuestions?ExposureShutter speedMain effect of shutter speedEffect of shutter speedShutter Slide Number 63Flash synch speed?ApertureMain effect of apertureDepth of fieldDepth of fieldSlide Number 69Depth of fieldDepth of fieldDepth of field & focusing distanceDepth of field & focusing distanceSLR viewfinder & apertureQuestions?ExposureReciprocitySlide Number 78Slide Number 79Slide Number 80Questions?MeteringSlide Number 83MeteringNikon 3D Color MatrixExposure & meteringPros and cons of various modesRecap: MeteringQuestions?Sensitivity (ISO)