Direct-to-Indirect Transfer for Cinematic Relighting
Direct-to-Indirect Transfer for Cinematic Relighting
Milos Hasan (Cornell University)
Fabio Pellacini (Dartmouth College)
Kavita Bala (Cornell University)
IntroductionIntroduction
• Cinematic Relighting [Gershbein 2000, Pellacini 2005]
– Interactive local light movement, fixed camera
– Procedural shaders, various materials
– High-complexity scenes
– Only direct illumination
• The goal of our system:
– Add multiple bouncesof indirect illumination
Direct-to-Indirect TransferDirect-to-Indirect Transfer
• Direct illumination from local lights
– Use known techniques
– Deep frame-buffer, shadow maps
• Indirect illumination – key idea:
– Precompute direct-to-indirect transfer matrix
– Compute indirect from direct on the fly
Direct Indirect Final
Transfer matrix
Related WorkRelated Work
• Cinematic relighting engines
– [Gershbein 00; Pellacini 05, Tabellion 04]
• Precomputed radiance transfer
– [Sloan 02, 03, 05; Kautz 02; Ng 03, 04; Liu 04; Wang 04, 05; Annen 04, etc.]
• Linear combinations of full solutions
– [Kristensen 05; Dorsey 91; Debevec 00; etc.]
Related WorkRelated Work
• Sparse sampling– [Walter 99, 02; Bala 99, 03; Ward 99; Tole 02;
Nijasure 03; Gautron 05; Dayal 05; Simmons 01 etc.]
• Hierarchical / instant radiosity methods– [Hanrahan 91; Keller 97; Drettakis 97; Christensen
97; Dachsbacher 05, 06; etc.]
• Wavelet radiance transport
– [Kontkanen 06] (concurrent work)
OutlineOutline
• Key concepts
• Transfer matrix
– Efficient precomputation
– Efficient compression
– Efficient evaluation
• Results
• Conclusions and Future work
Key ConceptsKey Concepts
• View samples
• Gather samples
• Transfer: A large matrix
– Contributions of gather samples to view samples
i = T d
Indirect on view
Transfer matrix
Direct on gather
Direct IlluminationDirect Illumination
• Standard techniques: GPU, shaders, shadow maps
Direct on gather samples Direct on view samples
Conceptual OverviewConceptual Overview
Direct on gather
Indirect on view
Final
Transfer matrix
Direct on view
OutlineOutline
• Key concepts
• Transfer matrix
– Efficient precomputation
– Efficient compression
– Efficient evaluation
• Results
• Conclusions and future work
Why Is Precomputation HardWhy Is Precomputation Hard
• Transfer matrix is huge!
– View samples:640 x 480 = 300k rows
– Gather samples: 64k columns
– 19 billion elements
• Columns:
– Images with 1 light and full global illumination
64k gather
300k view
Full global illumination
images
Multi-bounce & Final GatherMulti-bounce & Final Gather
• Split full transfer matrix:
– Multiple-bounce matrix (lower accuracy)
– Final gather matrix (higher accuracy)
Full transfer:Many gather-to-view
Multi-bounce:Many gather-to-gather
Final gather:One gather-to-view
Final Gather Matrix ElementsFinal Gather Matrix Elements
Fij = ??
View sample i
Gather sample j
Single-bounce contribution
Multi-bounce Matrix:Sparse ApproximationMulti-bounce Matrix:Sparse Approximation
• Photon mapping [Jensen 96] analogy
– But, don’t know light positions
• Shoot photons from gather samples instead!
– Photons carry gather sample ID
– Like standard photon mapping with 64k lights!
Multi-bounce Matrix:Computing by Photon MappingMulti-bounce Matrix:Computing by Photon Mapping
Gather sample i
i-th row of M = ?? A gather sample
K nearest photons
Conceptual Overview, UpdatedConceptual Overview, Updated
Direct on gather Indirect on gather Dir + Ind on gather
Indirect on view Final
Multi-bounce matrix
Add direct
Final gather matrix
Add direct
OutlineOutline
• Key concepts
• Transfer matrix
– Efficient precomputation
– Efficient compression
– Efficient evaluation
• Results
• Conclusions and future work
Defining a Wavelet BasisDefining a Wavelet Basis
• Want to use wavelets
– How? There is no obvious domain to apply them…
– Unstructured cloud of gather samples
• Still possible to define wavelets on the cloud!
– Flatten gather cloud to a 2D array
– Maintain coherence
– Use 2D Haar wavelets on the 2D array
Defining a Wavelet BasisDefining a Wavelet Basis
64k 256 x 256
Gather samples
Flattened in a 2D array
Wavelet transform
256 x 256
Wavelet CompressionWavelet Compression
• Transform rows into Haar wavelets
• Remove small coefficients
64k
300k
Dense
64k
300k
Sparse
64k
64k
Even more sparse
64k
64k
Sparse
Final gather matrixMulti-bounce matrix
Columns of the Wavelet-Transformed MatrixColumns of the Wavelet-Transformed Matrix
Images lit by wavelet lights
Final gather matrix in wavelets
Full System OverviewFull System Overview
Direct on gather Dir + Ind on gather
Indirect on view
Direct on gather in wavelet space
Dir + Ind on gather in wavelet space
Wavelet Xform
Wavelet Xform
Multi-bounce matrix
in waveletspace
Final gather matrix
in waveletspace
Final Image
Add direct
OutlineOutline
• Key concepts
• Transfer matrix
– Efficient precomputation
– Efficient compression
– Efficient evaluation
• Results
• Conclusions and future work
Sparse Matrix-Vector MultiplicationSparse Matrix-Vector Multiplication
Sparse matrix
Sparse vector
Linear combination of columns
Sparse Matrix-Vector Multiplication on the GPUSparse Matrix-Vector Multiplication on the GPU
• Problem:
– Columns are sparse themselves
– How to represent on the GPU?
• Solution:
– Non-zero elements tend to cluster…
– Cut out rectangular blocks
– Pack blocks into texture atlases
– Converted problem to blending!
Scene: Sponza AtriumScene: Sponza Atrium
Precomputation: 1.5 hours Frame-rate: 13.7 – 24.9 Polygon count: 66k
Conclusions and Future WorkConclusions and Future Work
• Conclusion
– Extend cinematic relighting with indirect illumination
– Interactive performance (GPU), complex scenes
– Arbitrary light shaders, efficient precomputation
• Future work
– Environment mapping
– Subsurface scattering
– Moving camera
Questions?Questions?
• Acknowledgements
– NSF grant CCF-0539996
– Bruce Walter (code, support)
– Veronica Sunstedt, Patrick Ledda (Temple)
– Marko Dabrovic (Sponza atrium)
– Cornell animation TA’s (Still life)
– Pixar, NVidia
• E-mail: [email protected]