Real-Time Relighting

Digital Image SynthesisYung-Yu Chuang1/10/2008

with slides by Ravi Ramamoorthi, Robin Green and Milos Hasan

Realistic rendering

• We have talked about photorealistic rendering for complex materials, complex geometry and complex lighting. They are realistic but slow.

Real-time rendering

• Its goal is to achieve interactive rendering with reasonable quality. It’s important in many applications such as games, visualization, computer-aided design, …

Real-Time relighting

• Lighting is the process of adjusting lights. It is an important but time-consuming step in animation production pipeline.

• Relighting algorithms for two kinds of lights– Distant environment lights– Near-field lights for production

Relighting algorithms for distant environment lights

Natural illumination

People perceive materials more easily under natural illumination than simplified illumination.

Images courtesy Ron Dror and Ted Adelson

Natural illumination

Rendering with natural illumination is more expensive compared to using simplified illumination

directional source natural illumination

Reflection maps

Blinn and Newell, 1976

Environment maps

Miller and Hoffman, 1984

HDR lighting

Examples of complex environment light

Examples of complex environment light

Direct lighting with complex illumination

)ωp,( ooL )ω,p( oeL

iiiio ωθcos)ωp,()ω,ωp,(2

dLf is

)ωp,( oB iiiio ωθcos)ωp,()ω,ωp,(2

dLf ds)ω( opB iiiiω, ωθcos)ω()ω(

2 odLf ds p

p q

Function approximation

• G(x): the function to approximate• B1(x), B2(x), … Bn(x): basis functions• We want

• Storing a finite number of coefficients ci gives an approximation of G(x)

)()(1

xBcxG i

n

ii

Function approximation• How to find coefficients ci?

– Minimize an error measure• What error measure?

– L2 error

• Coefficients

2])()([

2 I i

iiL xBcxGE

X

iii dxxBxGBGc )()(

• Basis Functions are pieces of signal that can be used to produce approximations to a function

1c

2c

3c

Function approximation

• We can then use these coefficients to reconstruct an approximation to the original signal

1c

2c

3c

Function approximation

• We can then use these coefficients to reconstruct an approximation to the original signal

xBcN

iii

1

Function approximation

Orthogonal basis functions

• Orthogonal Basis Functions– These are families of functions with special

properties

– Intuitively, it’s like functions don’t overlap each other’s footprint• A bit like the way a Fourier transform breaks

a functions into component sine waves

ji

jidxxBxB ji 0

1

Integral of product

iiij

i jiji

i jjjii

GFdxgfdxxBxBgf

dxxBgxBfdxxGxF

ˆˆ)()(

)()(

dxxGxFI

i

ii xBfxF )( j

jj xBgxG )(

)ω( opB iiiiω, ωθcos)ω()ω(2 o

dLf ds p

Basis functions

• Transform data to a space in which we can capture the essence of the data better

• Spherical harmonics, similar to Fourier transform in spherical domain, is used in PRT.

Real spherical harmonics

• A system of signed, orthogonal functions over the sphere

• Represented in spherical coordinates by the function

where l is the band and m is the index within the band

0

0

0

,cos

,cossin2

,coscos2

,00 m

m

m

PK

PmK

PmK

y

ll

ml

ml

ml

ml

ml

Real spherical harmonics

Reading SH diagrams

–+

Not thisdirection

Thisdirection

Reading SH diagrams

–+

Not thisdirection

Thisdirection

The SH functions

00y

11y

11y

12y

22y

02y

12y2

2y

The SH functions

Spherical harmonics

Spherical harmonics

-1-2 0 1 2

0

1

2

( , )lmY

xy z

xy yz 23 1z zx 2 2x y

1

m

l

SH projection

• First we define a strict order for SH functions

• Project a spherical function into a vector ofSH coefficients

S

ii dssysfc

mlli 1

SH reconstruction

• To reconstruct the approximation to a function

• We truncate the infinite series of SH functions to give a low frequency approximation

2

0

~ N

iii sycsf

Examples of reconstruction

An example

• Take a function comprised of two area light sources– SH project them into 4 bands = 16 coefficients

2380042508370317000106420

27800417009400908093006790

3291

.,,.,.,.,.,.,.,,.,,.

,.,.,.,.

Low frequency light source

• We reconstruct the signal– Using only these coefficients to find a low frequency

approximation to the original light source

Harr wavelets• Scaling functions (Vj)

• Wavelet functions (Wj)

• The set of scaling functions and wavelet functions forms an orthogonal basis

Harr wavelets

Example for wavelet transform

• Delta functions, f=(9,7,3,5) in V2

Wavelet transform

• V1, W1

Example for wavelet transform

• V0, W0 , W1

Example for wavelet transform

Quadratic B–spline scaling and wavelets

2D Harr wavelets

Example for 2D Harr wavelets

Applications

19%5% L2

1%15% L2

3%10% L2

Relighting algorithms for animation production

Relighting for production

• Lighting is a time-consuming process.• Artists adjust lighting parameters and wait for

a couple of hours or days to get feedback.• Local shading with complex scene and many li

ghts• Interactive relighting

– Interative visual eedback– Fixed scene and camera– Lower quality – Scalable with sene complexity and number of light

s

Deep framebuffer• Gershbein and Hanrahan, SIGGRAPH 2000

Deep framebuffer

Deep framebuffer

LPICS

• Pixar, SIGGRPH 2005. A practical realization for the deep framebuffer approach on GPUs

LPICS0.1s

Final renderer2,000s

video

Lightspeed• ILM, SIGGRAPH 2007• An even more practical system with automatic

shader conversion. (2.7s v.s. 57m)

Direct-to-indirect transfer

• Hasan et. al. SIGGRAPH 2006• Deep framebuffer approaches only support loc

al shading, but not indirect lighting

direct lighting With indirect lighting

Concept

• Distribute gather samples on scene surfaces

Concept

• Direct illumination on both gather samples and view samples

Concept

• Inter-reflections between gather samples

Concept

• Final gather on view samples

Inter-reflections between gather samples

gather sample

gather sample

Inter-reflections between gather samples• Assume all gather samples are diffuse

Inter-reflections between gather samples

Inter-reflections between gather samples

Final gathering

Direct on gather

Indirect on view

Final

Transfer

matrix

Direct on view

Concept

Scene: Still Life

Precomputation: 1.6 hours

11.4 – 18.7 fps Polygon: 107k

Scene: Temple

Precomputation: 2.5 hours

8.5 – 25.8 fps Polygon: 2M

Scene: Hair Ball

Precomputation: 2.9 hours

9.7 – 24.7 fpsPolygon: 320k

Scene: Sponza Atrium

Precomputation: 1.5 hours

13.7 – 24.9 fps Polygon: 66k

Comparison

DTI: 8-25 fps (2.5 hr precomputation)

Monte Carlo path tracer: 32 hours