Inferring Reflectance FunctionsInferring Reflectance Functionsfrom Wavelet Noisefrom Wavelet Noise
Pieter Peers
Philip Dutré
Pieter Peers
Philip Dutré
June 30th 2005
Department of Computer Science
Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting
Scene(fixed viewpoint)
Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting
…
Scene(fixed viewpoint)
Novel Incident Illumination
+
Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting
… …
Scene(fixed viewpoint)
Novel Incident Illumination Compute Relit Image
+ =
Image-based Relighting / Environment MattingImage-based Relighting / Environment Matting
… …
Scene(fixed viewpoint)
Novel Incident Illumination Compute Relit Image
+ =
ReflectanceFunction
Examples of Reflectance FunctionsExamples of Reflectance Functions
Diffuse BallSpecular Ball
Examples of Reflectance FunctionsExamples of Reflectance Functions
Diffuse BallSpecular Ball
Examples of Reflectance FunctionsExamples of Reflectance Functions
Diffuse BallSpecular Ball
Reflectance Function Reflectance Function
Reflectance Functions (frequency domain)Reflectance Functions (frequency domain)
Diffuse BallSpecular Ball
Reflectance Function (frequency domain) Reflectance Function (frequency domain)
Reflectance Functions (wavelet domain)Reflectance Functions (wavelet domain)
Diffuse BallSpecular Ball
Reflectance Function (wavelet domain) Reflectance Function (wavelet domain)
Relight a PixelRelight a Pixel
Novel Incident IlluminationSpecular Ball
Relit pixel value?
Reflectance Function (wavelet space)
Relight a PixelRelight a Pixel
Novel Incident IlluminationSpecular Ball
Reflectance Function (wavelet space) Incident Illumination (wavelet space)
Relight a PixelRelight a Pixel
Novel Incident IlluminationSpecular Ball
Reflectance Function (wavelet space) Incident Illumination (wavelet space)
( )
Relight a PixelRelight a Pixel
Novel Incident IlluminationSpecular Ball
Reflectance Function (wavelet space) Incident Illumination (wavelet space)
( )
Onlynon-zero
coefficients
Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions
Controlled Incident IlluminationPhotograph of Specular Ball
Emit(e.g. from CRT)
Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions
Controlled Incident IlluminationPhotograph of Specular Ball
ReflectanceFunction
(unknown)
Observed pixel
Controlled Incident Illumination (wavelet space)
Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions
Controlled Incident IlluminationPhotograph of Specular Ball
Unknown Reflectance Function (wavelet space)
( )
Controlled Incident Illumination (wavelet space)
Directly Observing Reflectance FunctionsDirectly Observing Reflectance Functions
Controlled Incident IlluminationPhotograph of Specular Ball
Controlled Incident Illumination (wavelet space)
( )
Onlynon-zero
coefficients
Observed coefficient
Unknown Reflectance Function (wavelet space)
Number of ObservationsNumber of Observations
Specular Ball
Reflectance Function (wavelet space)
#Photographs=
#Illumination pixels
Incident Illumination
Number of Observations ProblemNumber of Observations Problem
Specular Ball
Reflectance Function (wavelet space)
Incident Illumination
1000x
1000
1000x
1000
#Photographs=
#Illumination pixels
Wavelet Noise IlluminationWavelet Noise Illumination
Wavelet Noise
•Normal distribution of wavelet coefficients
•Mean : 0.0
•Standard deviation : 1.0
•Rescale Wavelet Noise Pattern to fit into [0..1] range
Wavelet Noise Pattern
Wavelet Noise Pattern (wavelet space)
Advantages
•Arbitrary number of different patterns possible
•Any reflectance function gives a non-zero response
•Constant average luminance
Estimating Wavelet Coefficients Estimating Wavelet Coefficients
(Unknown)Reflectance Function
Wavelet Noise
Assume: positions of are knownQuestion: what are the magnitudes?
( ) =Observed
Pixel Value
Estimating Wavelet Coefficients Estimating Wavelet Coefficients
( ) =
Leave out zero coefficients(of the reflectance function)
Wavelet Noise (linearized)
Reflectance Function(linearized)
Observed Pixel Value
Estimating Wavelet Coefficients Estimating Wavelet Coefficients
= …
Multiple observations matrix-vector multiplication
…
Wavelet NoiseReflectance
Function
Observed PixelValues
# em
itted
pat
tern
s # observations
Estimating Wavelet Coefficients Estimating Wavelet Coefficients
=
Finding magnitudes : Linear Least Squares problem
… …
Wavelet NoiseReflectance
Function
Observed PixelValues
Estimating Wavelet Coefficients Estimating Wavelet Coefficients
=
Estimation error when onlya part is approximated?
… …
Wavelet NoiseReflectance
Function
Observed PixelValues
Partial EstimationPartial Estimation
+… … …= = …
Wavelet NoiseReflectance
Function
ObservedPixel Values
Partial EstimationPartial Estimation
According to a normal distribution
+… … …= = …
Wavelet NoiseReflectance
Function
ObservedPixel Values
Partial EstimationPartial Estimation
According to a normal distribution
+… … …= = …
Wavelet NoiseReflectance
Function
ObservedPixel Values
Normal Normal distributiondistribution
Partial EstimationPartial Estimation
+… …= = …
Wavelet NoiseReflectance
Function
ObservedPixel Values
Finding the best approximation for : Linear Least Squares problem
NoIse
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Reflectance Function(2D wavelet space)
Inferring Reflectance FunctionsInferring Reflectance Functions
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
Inferring Reflectance FunctionsInferring Reflectance Functions
Reflectance Function(2D wavelet space)
Priority Queueof Candidates
OverviewOverview
Record photographs
Emit
Wavelet Noise
Predetermined number of photographs
OverviewOverview
Record photographs
Infer Reflectance Functions
Reflectance Function
Progressive Algorithm
For each pixel
OverviewOverview
Record photographs
Infer Reflectance Functions
Compute Relit Image
Relight
Incident Illumination
ResultsResults
64 Haar Wavelet Coefficients256 Photographs
Reference Photograph
ResultsResults
64 Haar Wavelet Coefficients256 Photographs
Reference Photograph
ResultsResults
64 Haar Wavelet Coefficients256 Photographs
Reference Photograph
ResultsResults
64 Haar Wavelet Coefficients256 Photographs
Reference Photograph
ResultsResults
64 Haar Wavelet Coefficients256 Photographs
Reference Photograph
ResultsResults
128 Haar Wavelet Coefficients512 Photographs
Reference Photograph
Results: High Frequency IlluminationResults: High Frequency Illumination
Conclusion & Future WorkConclusion & Future Work
Inferring Reflectance Functions from Wavelet Noise– No restriction on material properties– Stochastic illumination patterns– Trade-off quality versus acquisition time
Future Work– Noise filtering– Higher-order wavelets