a signal-processing framework for forward and inverse rendering ravi ramamoorthi...
Post on 20-Dec-2015
219 views
TRANSCRIPT
![Page 1: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/1.jpg)
A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering
A Signal-Processing Framework for A Signal-Processing Framework for Forward and Inverse RenderingForward and Inverse Rendering
Ravi [email protected]
Ravi [email protected]
![Page 2: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/2.jpg)
OutlineOutlineOutlineOutline
• Motivation• Forward Rendering• Inverse Rendering• Object Recognition
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation• Forward Rendering• Inverse Rendering• Object Recognition
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
![Page 3: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/3.jpg)
Interactive RenderingInteractive RenderingInteractive RenderingInteractive Rendering
Directional Source Complex Illumination
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
![Page 4: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/4.jpg)
Reflection MapsReflection MapsReflection MapsReflection Maps
Blinn and Newell, 1976
![Page 5: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/5.jpg)
Environment MapsEnvironment MapsEnvironment MapsEnvironment Maps
Miller and Hoffman, 1984Later, Greene 86, Cabral et al. 87
![Page 6: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/6.jpg)
Reflectance Space ShadingReflectance Space ShadingReflectance Space ShadingReflectance Space Shading
Cabral, Olano, Nemec 1999
![Page 7: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/7.jpg)
Reflectance MapsReflectance MapsReflectance MapsReflectance Maps
• Reflectance Maps (Index by N)• Horn, 1977
• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984
• Reflectance Maps (Index by N)• Horn, 1977
• Irradiance (N) and Phong (R) Reflection Maps• Hoffman and Miller, 1984
Mirror Sphere Chrome Sphere Matte (Lambertian) Sphere
Irradiance Environment Map
![Page 8: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/8.jpg)
Complex IlluminationComplex Illumination Complex IlluminationComplex Illumination
• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)
• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)
• New representation for lighting design, IBR
• Must (pre)compute hemispherical integral of lighting• Efficient Prefiltering (> 1000x faster)
• Traditionally, requires irradiance map textures• Real-Time Procedural Rendering (no textures)
• New representation for lighting design, IBR
Directional Source Complex LightingIllumination Irradiance
Environment Map
![Page 9: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/9.jpg)
Photorealistic RenderingPhotorealistic RenderingPhotorealistic RenderingPhotorealistic Rendering
Geometry
70s, 80s: Splines 90s: Range Data
Materials/Lighting(Texture, reflectance [BRDF], Lighting)
Realistic Input Models Required
Arnold Renderer: Marcos Fajardo
Rendering Algorithm
80s,90s: Physically-based
![Page 10: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/10.jpg)
Inverse RenderingInverse RenderingInverse RenderingInverse Rendering• How to measure realistic material models, lighting?
• From real photographs by inverse rendering
• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data
• How to measure realistic material models, lighting?• From real photographs by inverse rendering
• Can then change viewpoint, lighting, reflectance• Rendered images very realistic: they use real data
Illumination:Mirror Sphere
Grace Cathedralcourtesy
Paul Debevec
BRDF (reflectance): Images using
point light source
![Page 11: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/11.jpg)
FlowchartFlowchartFlowchartFlowchart
Lighting
BRDF NewView
NewView,Light
![Page 12: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/12.jpg)
ResultsResultsResultsResults
Photograph Computer rendering
New view, new lighting
Ramamoorthi and Hanrahan, SIGGRAPH 2001a
![Page 13: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/13.jpg)
Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals
• Complex (possibly unknown) illumination
• Estimate both lighting and reflectance (factorization)
• Complex (possibly unknown) illumination
• Estimate both lighting and reflectance (factorization)
Photographs of 4 spheres in 3 different
lighting conditions
courtesy Dror and Adelson
![Page 14: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/14.jpg)
Factorization AmbiguitiesFactorization AmbiguitiesFactorization AmbiguitiesFactorization Ambiguities
Width of Light Source
SurfaceRoughness
![Page 15: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/15.jpg)
Inverse ProblemsInverse ProblemsInverse ProblemsInverse Problems
• Sometimes ill-posed• No solution or several solutions given data
• Often numerically ill-conditioned• Answer not robust, sensitive to noise
• Need general framework to address these issues• Mathematical theory for complex illumination
• Sometimes ill-posed• No solution or several solutions given data
• Often numerically ill-conditioned• Answer not robust, sensitive to noise
• Need general framework to address these issues• Mathematical theory for complex illumination
Directional
Source
Area source
Same reflectance
![Page 16: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/16.jpg)
Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
Peter Belhumeur: Yale Face Database A
![Page 17: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/17.jpg)
OutlineOutlineOutlineOutline
• Motivation
• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Signal Processing Framework: Reflection as Convolution• Reflection Equation (2D)• Fourier Analysis (2D)• Spherical Harmonic Analysis (3D)• Examples
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
![Page 18: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/18.jpg)
Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)Reflection as Convolution (2D)
2
2
ˆ( , ) ( , ) ( , )o i i o iB L d
( , ) ( ) ( )i i iL L L
o
L
i i
o B
BRDF
( , )i o
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
B L
![Page 19: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/19.jpg)
Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)Fourier Analysis (2D)
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
iIl
pleL ,ˆ i oIl I
lp
qp
p
e e ,oIp
pIl
ql
p
B e e
, ,ˆ2l p l l pB L
![Page 20: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/20.jpg)
Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)Spherical Harmonics (3D)
-1-2 0 1 2
0
1
2
.
.
.
( , )lmY
xy z
xy yz 23 1z zx 2 2x y
![Page 21: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/21.jpg)
Spherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic AnalysisSpherical Harmonic Analysis
, ,ˆ2l p l l pB L
lL ,ˆl p,l pB
2
2
ˆ( , ) ( ) ( , )o i i o iB L d
2D:
3D:,lm pqB
lmL ,ˆlq pqIsotropic
, ,ˆlm pq l lm lq pqB L
![Page 22: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/22.jpg)
InsightsInsightsInsightsInsights
• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
• Inverse rendering is deconvolution
• Our contribution: Formal Frequency-space analysis
• Signal processing framework for reflection• Light is the signal• BRDF is the filter• Reflection on a curved surface is convolution
• Inverse rendering is deconvolution
• Our contribution: Formal Frequency-space analysis
![Page 23: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/23.jpg)
Example: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDFExample: Mirror BRDF
• BRDF is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to lighting
• Mirror Sphere (Gazing Ball)
• BRDF is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to lighting
• Mirror Sphere (Gazing Ball)
![Page 24: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/24.jpg)
Phong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet ModelsPhong, Microfacet Models
• Rough surfaces blur highlight
• Analytic Formula• Approximately Gaussian
• Rough surfaces blur highlight
• Analytic Formula• Approximately Gaussian
Mirror Matte
2
exp2l l
l
s
Roughness
![Page 25: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/25.jpg)
Example: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDFExample: Lambertian BRDF
2 1
2
2
2 1 ( 1) !ˆ 2
4 ( 2)( 1) 2 !
l
l l l
l ll even
l l
Ramamoorthi and Hanrahan, JOSA 2001
![Page 26: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/26.jpg)
Second-Order ApproximationSecond-Order ApproximationSecond-Order ApproximationSecond-Order Approximation
Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial
Lambertian: 9 parameters only• order 2 approx. suffices• Quadratic polynomial
L=0 L=1 L=2 Exact
-1-2 0 1 2
0
1
2
( , )lmY
xy z
xy yz 23 1z zx 2 2x ySimilar to Basri & Jacobs 01
![Page 27: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/27.jpg)
Dual RepresentationDual Representation Dual RepresentationDual Representation
• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation
• Practical Representation• Diffuse localized in frequency space• Specular localized in angular space• Dual Angular, Frequency-Space representation
Frequency9 param.
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Angular SpaceFrequency9 param.
![Page 28: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/28.jpg)
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
![Page 29: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/29.jpg)
VideoVideoVideoVideo
Ramamoorthi and Hanrahan, SIGGRAPH 2001b
![Page 30: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/30.jpg)
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
![Page 31: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/31.jpg)
Lighting effects in recognitionLighting effects in recognitionLighting effects in recognitionLighting effects in recognition
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
• Space of Images (Lighting) is Infinite Dimensional• Prior empirical work: 5D subspace captures variability
• We explain empirical data, subspace methods
Peter Belhumeur: Yale Face Database A
![Page 32: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/32.jpg)
Face Basis FunctionsFace Basis FunctionsFace Basis FunctionsFace Basis Functions
• 5 basis functions capture 95% of image variability
• Linear combinations of spherical harmonics
• Complex illumination not much harder than points
• 5 basis functions capture 95% of image variability
• Linear combinations of spherical harmonics
• Complex illumination not much harder than points
Frontal Lighting Side Above/Below Extreme Side Corner
![Page 33: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/33.jpg)
Inverse LightingInverse LightingInverse LightingInverse Lighting
• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:
contradicts theorem in Preisendorfer 76
• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters
• Well-posed unless equals zero in denominator• Cannot recover radiance from irradiance:
contradicts theorem in Preisendorfer 76
• Well-conditioned unless small• BRDF should contain high frequencies : Sharp highlights• Diffuse reflectors are ill-conditioned : Low pass filters
B L
BL
![Page 34: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/34.jpg)
Inverse LambertianInverse LambertianInverse LambertianInverse LambertianSum l=2 Sum l=4True Lighting
Mirror
Teflon
![Page 35: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/35.jpg)
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary
![Page 36: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/36.jpg)
Inverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: GoalsInverse Rendering: Goals• Formal study: Well-posedness, conditioning
• General Complex (Unknown) Illumination
• Formal study: Well-posedness, conditioning
• General Complex (Unknown) Illumination
Bronze Delrin Paint Rough Steel
Photographs
Renderings(Recovered
BRDF)
Quantitative Pixel error approximately 5%
![Page 37: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/37.jpg)
Factoring the Light FieldFactoring the Light FieldFactoring the Light FieldFactoring the Light Field
• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)
(½ 3D) -- Make use of reciprocity
• The light field may be factored to estimate both the BRDF and the lightingKnowns B (4D)Unknowns L (2D)
(½ 3D) -- Make use of reciprocity
,,
1 lm pqlq pq
l lm
B
L
,00
00, 0
1 lmlm
l l
BL
B
00
1
l
L
B L
![Page 38: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/38.jpg)
Algorithms ValidationAlgorithms ValidationAlgorithms ValidationAlgorithms ValidationRendering
Unknown
Lighting
Photograph
Known
Lighting
Kd 0.91 0.89 0.87
Ks 0.09 0.11 0.13
1.85 1.78 1.48
0.13 0.12 .14
Recovered
Light
Estimate by ratio of
intensity andtotal energy
Marschner
![Page 39: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/39.jpg)
Complex GeometryComplex GeometryComplex GeometryComplex Geometry
3 photographs of a sculptureComplex unknown illuminationGeometry KNOWNEstimate BRDF and Lighting
![Page 40: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/40.jpg)
FlowchartFlowchartFlowchartFlowchart
Lighting
BRDF NewView
NewView,Light
![Page 41: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/41.jpg)
ComparisonComparisonComparisonComparison
RenderedKnown Lighting
Photograph RenderedUnknown Lighting
![Page 42: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/42.jpg)
New View, LightingNew View, LightingNew View, LightingNew View, Lighting
Photograph Computer rendering
![Page 43: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/43.jpg)
Textured ObjectsTextured ObjectsTextured ObjectsTextured Objects
Real RenderingComplex, Known Lighting
![Page 44: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/44.jpg)
OutlineOutlineOutlineOutline
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary • Conclusions• Pointers
• Motivation
• Reflection as Convolution
• Efficient Rendering: Environment Maps
• Lighting Variability in Object Recognition
• Deconvolution, Inverse Rendering
• Summary • Conclusions• Pointers
![Page 45: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/45.jpg)
SummarySummarySummarySummary
• Reflection as Convolution
• Signal-Processing Framework
• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter
• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization
• Efficient Forward Rendering (Environment Maps)
• Lighting Variability in Object Recognition
• Reflection as Convolution
• Signal-Processing Framework
• Frequency-space analysis yields insights• Lambertian: approximated with 9 parameters• Phong/Microfacet: acts like Gaussian filter
• Inverse Rendering• Formal Study: Well-posedness, conditioning• Dual Representations• Practical Algorithms: Complex Lighting, Factorization
• Efficient Forward Rendering (Environment Maps)
• Lighting Variability in Object Recognition
![Page 46: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/46.jpg)
PapersPapersPapersPapers
• http://graphics.stanford.edu/~ravir/research.html
• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]
• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]
• http://graphics.stanford.edu/~ravir/research.html
• Theory• Flatland or 2D using Fourier analysis [SPIE 01]• Lambertian: radiance from irradiance [JOSA 01]• General 3D, Isotropic BRDFs [SIGGRAPH 01a]
• Applications• Inverse Rendering [SIGGRAPH 01a]• Forward Rendering [SIGGRAPH 01b]• Lighting variability [In preparation]
![Page 47: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/47.jpg)
AcknowledgementsAcknowledgementsAcknowledgementsAcknowledgements
• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• John Parissenti
• Jean Gleason
• Scanned cat sculpture is “Serenity” by Sue Dawes
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant #0085864: “Interacting with the Visual World”
• Marc Levoy
• Szymon Rusinkiewicz
• Steve Marschner
• John Parissenti
• Jean Gleason
• Scanned cat sculpture is “Serenity” by Sue Dawes
• Hodgson-Reed Stanford Graduate Fellowship
• NSF ITR grant #0085864: “Interacting with the Visual World”
![Page 48: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/48.jpg)
The EndThe EndThe EndThe End
![Page 49: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/49.jpg)
Related WorkRelated WorkRelated WorkRelated Work
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
![Page 50: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/50.jpg)
Related WorkRelated WorkRelated WorkRelated Work
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case
Graphics: Prefiltering Environment Maps• Qualitative observation of reflection as convolution• Miller and Hoffman 84, Greene 86• Cabral, Max, Springmeyer 87 (use spherical harmonics)• Cabral et al. 99
Vision, Perception• D’Zmura 91: Reflection as frequency-space operator• Basri and Jacobs 01: Lambertian reflection as convolution• Recognition: Appearance models e.g. Belhumeur et al.
Our Contributions• Explicitly derive frequency-space convolution formula• Formal Quantitative Analysis in General 3D Case
![Page 51: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/51.jpg)
Example: Directional SourceExample: Directional SourceExample: Directional SourceExample: Directional Source
• Lighting is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter
• Lighting is delta function• Harmonic Transform is constant (infinite width)
• Reflected light field corresponds directly to BRDF• Impulse response of BRDF filter
![Page 52: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/52.jpg)
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Incomplete sparse data: Few views• Use practical Dual Representation
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Angular SpaceFrequency
![Page 53: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/53.jpg)
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Concavities: Self Shadowing
• Incomplete sparse data: Few views• Use practical Dual Representation
• Concavities: Self Shadowing
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Reflected RayShadowed?
IntegrateLighting
SourceShadowed?
![Page 54: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/54.jpg)
Practical IssuesPractical IssuesPractical IssuesPractical Issues• Incomplete sparse data: Few views
• Use practical Dual Representation
• Concavities: Self Shadowing
• Textures: Spatially Varying Reflectance
• Incomplete sparse data: Few views• Use practical Dual Representation
• Concavities: Self Shadowing
• Textures: Spatially Varying Reflectance
B Bd
diffuse
Bs,slow
slow specular(area sources)
Bs,fast
fast specular(directional)
= + +
Reflected RayShadowed?
IntegrateLighting
SourceShadowed?
Kd(x) Ks(x)
![Page 55: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/55.jpg)
Inverse BRDFInverse BRDFInverse BRDFInverse BRDF
• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting
• Well-conditioned unless L small• Lighting should have sharp features (point sources, edges)• Ill-conditioned for soft lighting
Directional
Source
Area source
Same BRDF
B L
B
L
![Page 56: A Signal-Processing Framework for Forward and Inverse Rendering Ravi Ramamoorthi ravir@graphics.stanford.edu Ravi Ramamoorthi ravir@graphics.stanford.edu](https://reader035.vdocuments.mx/reader035/viewer/2022081516/56649d415503460f94a1bf5a/html5/thumbnails/56.jpg)
ComparisonComparisonComparisonComparisonRenderingPhotograph
Kd 0.91 0.89
Ks 0.09 0.11
1.85 1.78
0.13 0.12
Marschner Our method
KnownLighting
Pixel ErrorApproximately 5%