Statistical and numerical techniquesfor photorealistic image synthesis

Kartic Subr

Who am I?

• Born in India– Bangalore University (Bachelor of Engineering) 2001– Hewlett Packard, India/Singapore

• 6 years in USA– PhD, University of California Irvine, 2008– Advisor: Jim Arvo (PhD Yale University), pioneered methods in light transport

• 2 years in France– Post doctoral researcher, ARTIS, INRIA-Grenoble (2008-2010)

My goal: Generating realistic visuals

Gustave Courbet, Stone-Breakers, 1849.

Wilhelm Oswald Gustav Achenbach, Abendstimmung in der Campagna, 1850.

Realism in art

Gustave Courbet, Stone-Breakers, 1849.

Wilhelm Oswald Gustav Achenbach, Abendstimmung in der Campagna, 1850.

Realism in art

Photograph: Nicéphore Niépce, 1826

My goal: Generating realistic visuals

Notion of “realism” depends on technology

Gerhard Richter, 1983

Pedro Campos

Hyperrealism

Realistic image synthesis

Image synthesis involves light transport

Image ?

Digital models of scene (geometry + materials)

Light sources

Virtual camera

Image synthesis adds visual impact

AvatarCaptured video+ digital model

Digital model

Applications of image synthesis

Defense

Advertising

Entertainment

Virtual prototyping

Biomedical imaging

Multidisciplinary nature of the problem

• Physically based optical simulations

• Mathematical tools for analysis

• Numerical techniques for light transport solution

• Understanding biological processeseg. Perception et cognition

Reflection of light is an integration

Light transport: multi-domain integration

• Combinatorial explosion from sampling each domain

Image spaceAperture

Exposure time

Visible spectrum

Reflectance Direct illumination Indirect illumination

[Efficient sampling strategies for Monte Carlo integration (my PhD thesis)]

Talk outline

• Recent contributions– Simulating defocus– Rendering translucent materials

• Research plan– Core problems in image synthesis– Model representation and abstraction

My contributions: Fourier depth of field

Defocus blur is important in photography

Defocus is due to aperture integraion

Image Aperture

Pixel p

Lens

Defocus

Image Aperture

Pixel p

Lens Scene

Pixel p

Monte Carlo estimation of aperture integral

Image Aperture

NA primary rays per pixel

Integrateat p

Aperture integration is very costly

ImageAperture

NP pixels

NP x NA Primary rays

NA Aperture samples

64 x #primary rays of the pinhole image

Paradox: Blurry image is costlier to compute!

Observation 1: Image

Blurry regionsshould not require

dense samplingof the image

Observation 2: Lens

Regions in focusshould not requireprofuse samplingof the lens for diffuse objects

Fourier depth of field

• Fourier domain analysis of finite aperture cameras• Adaptive sampling• Speedup of around 20 over existing algorithms

[ACM Transactions on Graphics 2009. Presented at ACM SIGGRAPH 09]

Collaborators: MIT

My contributions: Translucent materials

TranslucentOpaque

Translucency: Sub-surface scattering

• Brute force Monte Carlo: prohibitively expensive• Diffusion approximations: severe constraints on geometry

Finite difference method on new domain

• Approximation: diffusion equation

• Domain: Dual graph of tetrahedralization

Diffuse flux

Rendering translucent materials

[Computer Graphics Forum 2010. To be presented at Eurographics 2010]

[Collaborators: Microsoft Research, Tsinghua University]

• Arbitrary geometry• Heterogenous materials• Dynamically deforming shapes• In real-time!

Research program

Realistic image synthesisModel representation

and abstraction

1. Realistic image synthesis

• Bandwidth driven sampling– Transport of local light field spectrum– Derive spatial / angular sampling rates– co-advising PhD student Laurent Belcour (ARTIS)

• Importance vs radiance– Tracing from eye vs tracing from light– Monte Carlo matrix chain multiplication

Short-term

Long-term

Importance vs radiance

Radiance

Importance vs radiance

Importance

Importance vs radiance

MC matrix-chain product estimator

MC matrix-chain product estimator

Related to optimal matrix chain multiplication using dynamic programming?

2. Model representation and abstraction

• Abstracting detail in geometry– First step: images (published at SIGGRAPH Asia 09)

• Alternate representation– Voxel data to represent geometry and materials

Short-term

Long-term

Detail = oscillations between extrema

Input

Local maxima

Local minima

Image multiscale decomposition

+

+

Medium

Pixels

Intensity

Input

Fine

Coarse

1D

Allows smoothing high-contrast detail

Input

Smoothed

Thank you!• Collaborators

– Established • MIT, USA• Microsoft Research• Tsinghua University, China• University of California, Irvine

– Current• Cornell University, USA• University of California, Berkeley

– Potential• Indian Institute of Information Technology

• International journal publications– Computer Graphics Forum 2010: Translucent materials. 4th author of 6– TOG 2009: Multiscale image decomposition. 1st author of 3– TOG 2009: Fourier Depth of Field. 2nd author out of 5

• Refereed international conference papers– Pacific Graphics 2007: Statistical hypotheses. 1st author of 2– Interactive raytracing 2007: Steerable importance sampling. 1st author of 2– ICIAP 2005: Contrast enhancement. 1st author of 3

Merci !• Collaborators

– Established • MIT, USA• Microsoft Research• Tsinghua University, China• University of California, Irvine• LJK Grenoble

– Current• Cornell University, USA• University of California, Berkeley

– Potential• Indian Institute of Information Technology

• International journal publications– Computer Graphics Forum 2010: Translucent materials. 4th author of 6– TOG 2009: Multiscale image decomposition. 1st author of 3– TOG 2009: Fourier Depth of Field. 2nd author out of 5

• Refereed international conference papers– Pacific Graphics 2007: Statistical hypotheses. 1st author of 2– Interactive raytracing 2007: Steerable importance sampling. 1st author of 2– ICIAP 2005: Contrast enhancement. 1st author of 3

• Teaching– Columbia University, USA (120 h)– University of California, Irvine (360 h)

• Industry– Rhythm and Hues Studios– NVIDIA Corporation– Hewlett Packard