reflectance modeling for vision and graphics
TRANSCRIPT
1
Reflectance Modelingfor Vision and Graphics
Todd ZicklerHarvard School of Engineering and Applied Sciences
Reflectance Modeling
Acknowledgements
Satya Mallick, UCSD
Sebastian Enrique, EA Sports
Ravi Ramamoorthi, Columbia University
Peter Belhumeur, Columbia University
David Kriegman, UCSD
Jeffrey Ho, UFL
Jean Ponce, UIUC, ENS
Funding: NSF
2
Reflectance Modeling
Appearance
f -1( I ) = ?
I = f (shape, reflectance)illumination,
Reflectance Modeling
Reflectance: BRDF
n(θi,φi)
fr(θi,φi; θo,φo)
(θo,φo)
Bi-directional Reflectance Distribution Function
3
Reflectance Modeling
Typical Assumption: Lambertian Reflectance
LAMBERTIAN:IDEALLY DIFFUSE
Reflectance Modeling
Handling Complex Reflectance
1. Ignore (treat as noise)2. Detect and remove3. Model parametrically
Proposed Approach:
Exploit common reflectance phenomena (reciprocity, isotropy, separability,…)
4
Reflectance Modeling
♦ Reciprocity:Helmholtz stereopsis– reconstruction
Outline
u
v
θh
2φd
♦ Separability:Color subspaces– reconstruction, recognition, motion
estimation, segmentation
♦ Spatial coherence, compressibility:Reflectance sharing– modeling, appearance capture
Reflectance Modeling
i e
n
Helmholtz Reciprocity
ie
n
[Helmholtz 1925; Minnaert 1941; Nicodemus et al. 1977]
)i,e()e,i( rr ff =
5
Reflectance Modeling
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
Reflectance Modeling
Stereo vs. Helmholtz Stereo
STEREO HELMHOLTZ STEREO
6
Reflectance Modeling
In Practice
HELMHOLTZ STEREO
Reflectance Modeling
Reciprocal Images
Specularities “fixed” to surface
el er
Relation between el and er independent of BRDF
7
Reflectance Modeling
Reciprocity Constraint
n
vl^ vr
^
x
ol or
=
vl^ vr
^
x
ol or
n
er(x) = fr(vl(x), vr(x))slvl(x) · n(x)|ol − x|2el(x) = fr(vr(x), vl(x))
srvr(x) · n(x)|or − x|2
sr sl
Reflectance Modeling
Reciprocity Constraint
n
vl^ vr
^
x
ol or
vl^ vr
^
x
ol or
n
sr sl
µel(x)
slvl(x)
|ol − x|2 − er(x)srvr(x)
|or − x|2¶· n(x) = 0.
8
Reflectance Modeling
Reciprocity Constraint
µel(x)
slvl(x)
|ol − x|2 − er(x)srvr(x)
|or − x|2¶· n(x) = 0.
Near-field reciprocity constraint:
Far-field reciprocity constraint:
(el(x)slvl − er(x)srvr) · n(x) = 0
[Zickler et al., ECCV 2002]
Reflectance Modeling
Example
[Zickler et al., ECCV 2002]
9
Reflectance Modeling
Reciprocal Images: Typical Dataset
SOURCEVIEW
Reflectance Modeling
Reciprocal Images: Typical Dataset
SOURCE
VIEW
Conventional Stereo• Constant brightness (Lambertian)• No structure in textureless regions
10
Reflectance Modeling
Reciprocal Images: Typical Dataset
SOURCEVIEW
Conventional Stereo• Constant brightness (Lambertian)• No structure in textureless regions
Photometric Stereo• Needs reflectance model• No direct depth estimates
Reflectance Modeling
Reciprocal Images: Typical Dataset
SOURCE
VIEW
Conventional Stereo• Constant brightness (Lambertian)• No structure in textureless regions
Photometric Stereo• Needs reflectance model• No direct depth estimates
Helmholtz Stereo• No assumed reflectance• Gives depth and surface normals
11
Reflectance Modeling
Auto-calibration
1. Cameras 2. Source strengths3. Surface shape
Reflectance Modeling
Auto-calibration
x0 = Ax; n0 =A−>n|A−>n| ;
s0i = si|Avi|; P0i = PiA−1;
f 0r(u, v; θ) = fr(u, v; θ)|A−>n(u, v)|
12
Reflectance Modeling
Auto-calibration Strategy
INPUT SPARSE FEATURES
EPIPOLAR GEOMETRY/SPARSE RECONSTRUCTION
DENSE, METRICRECONSTRUCTION
[Zickler, CVPR 2006]
Reflectance Modeling
Specular Highlights as Features
INPUT
13
Reflectance Modeling
Auto-calibration Example
♦ Five ‘far-field’ cameras/sources♦ Dense reconstruction: [Zickler et al., 2002]
{si} = {0.74, 0.89, 0.89, 0.92, 1}
{si} = {1, 1, 1, 1, 1}
[Zickler, CVPR 2006]
Reflectance Modeling
♦ Reciprocity (& isotropy):Helmholtz stereopsis– reconstruction
Outline
u
v
θh
2φd
♦ Separability:Color subspaces– reconstruction, recognition, motion
estimation, segmentation
♦ Spatial coherence, compressibility:Reflectance sharing– modeling, appearance capture
14
Reflectance Modeling
Separability
DIFFUSE
= +
SPECULAR
[Shafer, 1985]
Reflectance Modeling[Shafer, 1985]
λ
E
λ
R
λ
kCDk =
ZE(λ)R(λ)Ck(λ)dλ
IRGB = (n · i)D+ fs(i, e)(n · i)S
Separability: Dichromatic Model
15
Reflectance Modeling[Shafer, 1985]
λ
E
λ
R
λ
kCDk =
ZE(λ)R(λ)Ck(λ)dλ
IRGB = (n · i)D+ fs(i, e)(n · i)S
Separability: Dichromatic Model
Reflectance Modeling
Separability: Dichromatic Model
[Shafer, 1985]
λ
E
λ
R
λ
kC
Sk =
ZE(λ)Ck(λ)dλ
Dk =
ZE(λ)R(λ)Ck(λ)dλ
IRGB = (n · i)D+ fs(i, e)(n · i)S
16
Reflectance Modeling
Dichromatic Materials
[Tominga and Wandell, 1989; Healey, 1989; Lee et al., 1990]
Reflectance Modeling
Explicit Separation
DIFFUSE
= +
SPECULAR
= +σd(u)D(u) σs(u)SIRGB(u)
17
Reflectance Modeling
Explicit Separation
DIFFUSE
= +
SPECULAR
[Klinker et al., 1988; Bajscy et al., 1996; Criminisi et al., 2005; Lee and Bajscy, 1992; Lin et al., 2002; Lin and Shum, 2001; Miyazaki et al., 2003; Nayar et al., 1997; Ragheb and Hancock, 2001; Sato and Ikeutchi, 1994; Tan and Ikeutchi, 2005; Wolfe and Boult, 1991;…]
= +σd(u)D(u) σs(u)SIRGB(u)
Reflectance Modeling
Observation:Explicit Separation not Required
Gr2r1
S
IRGB
B
R J
IRGB = σdD+ σsS
Jl =< IRGB , rl >= σdr>l D
1. INVARIANT TOSPECULAR REFLECTIONS
2. BEHAVES ‘LAMBERTIAN’
18
Reflectance Modeling
Observation:Explicit Decomposition not Required
Gr2r1
S
IRGB
B
RJ
IRGB = σdD+ σsS
Jl =< IRGB , rl >= σdr>l D
IRGB || J ||
[Mallick, Zickler et al., CVPR 2005; Zickler et al., CVPR 2006]
Reflectance Modeling
Generalization: Mixed Illumination
Gr2r1
S
IRGB
B
R Jr1
S1
IRGB
B
G
R
S2
J
SINGLE ILLUMINANT MIXED ILLUMINATION
[Mallick, Zickler et al., CVPR 2005; Zickler et al., CVPR 2006]
19
Reflectance Modeling
Generalization: Mixed Illumination
Reflectance Modeling
Example: Optical Flow
[Algorithm: Black and Anandan, 1993]
Conv
entio
nal
Gra
ysca
le(R
-+G
+B)/3
Spec
ular
In
varia
nt,
||J||
(blu
e ill
umin
ant)
Spec
ular
Inv
aria
nt,
||J||
(blu
e &
yel
low
ill
umin
ants
)
Ground truth flow
20
Reflectance Modeling
Example: Binocular Stereo
[Algorithm: Boykov, Veksler and Zabih, CVPR 1998]
Conventional Grayscale(R+G+B)/3
Specular Invariant, ||J||(blue illuminant)
Specular Invariant, ||J||(blue & yellow illuminants)
One image from input stereo pair
Reco
vere
d de
pth
Reflectance Modeling
Generalized Hue
Gr2r1
S
IRGB
B
R Jψ
ψ = tan−1(J1/J2) = tan−1(r>1 D/r>2 D)
Jl =< IRGB , rl >= σdr>l D
[Zickler et al., CVPR 2006]
21
Reflectance Modeling
Example: Material-based Segmentation
Input image
Conventional Grayscale Specular Invariant ||J||
Conventional Hue Generalized Hue ψ
[Zickler et al., CVPR 2006]
Reflectance Modeling
Example: Photometric Stereo
[Mallick, Zickler et al., CVPR 2005]
J behaves ‘Lambertian’→ Linear function of surface normal
Jl =< IRGB , rl >= σdr>l D = (n · i)r>l D
22
Reflectance Modeling
Example: Photometric Stereo
J behaves ‘Lambertian’→ Linear function of surface normal
[Mallick, Zickler et al., CVPR 2005]
Jl =< IRGB , rl >= σdr>l D = (n · i)r>l D
Reflectance Modeling
Example: Photometric Stereo
J behaves ‘Lambertian’→ Linear function of surface normal
[Mallick, Zickler et al., CVPR 2005]
Jl =< IRGB , rl >= σdr>l D = (n · i)r>l D
23
Reflectance Modeling
Example: Photometric Stereo
[Mallick, Zickler et al., CVPR 2005]
Reflectance Modeling
♦ Reciprocity (& isotropy):Helmholtz stereopsis– reconstruction
Outline
u
v
θh
2φd
♦ Separability:Color subspaces– reconstruction, recognition, motion
estimation, segmentation
♦ Spatial coherence, compressibility:Reflectance sharing– modeling, appearance capture
24
Reflectance Modeling
Modeling; Appearance Capture
1. SHAPE
+
2. REFLECTANCE
Reflectance Modeling
5º sampling: 1,000,000 images >106 MB1º sampling: 625,000,000 images >109 MB
n
ˆ ˆ( , )xf i er
Spatially-varying BRDF (SBRDF)From Images and Shape
~x
25
Reflectance Modeling
ˆ ˆ( ; , )x xf i eαr rr
ˆ ˆ( , )xf i er
1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…
Existing Approaches
Reflectance Modeling
ˆ ˆ( , )xf i er
1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…
2. Data-Driven (Non-parametric)[Debevec et al., 2000][Wood et al., 2000][Matusik et al., 2002]
Existing Approaches
26
Reflectance Modeling
1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…
2. Data-Driven (Non-parametric)[Debevec et al., 2000][Wood et al., 2000][Matusik et al., 2002]
ˆ ˆ( , )xf i er
Subset of reflectance
12,0002000 600# IMAGES:
Existing Approaches
Reflectance Modeling
1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…
2. Data-Driven (Non-parametric)[Debevec et al., 2000][Wood et al., 2000][Matusik et al., 2002]
PRO: Sparse Images
Efficient Rendering
CON: Limited Generality
PRO: General BRDFs
CON: Expensive ( , $ )
Cumbersome
Existing Approaches
27
Reflectance Modeling
1. Parametric BRDF models[Sato, Wheeler, Ikeuchi, 1997][Yu et al., 1999][Boivin, Gagalowicz, 2001][Lensch, et al., 2001] [McAllister, Lastra, Heidrich, 2002][Georghiades, 2003]…
2. Data-Driven (Non-parametric)[Debevec et al., 2000][Wood et al., 2000][Matusik et al., 2002]
PRO: Sparse ImagesEfficient Rendering
CON: Limited Generality
PRO: General BRDFs
CON: Expensive
Cumbersome
Existing Approaches
Reflectance Modeling
Reflectance Sharing: Three BRDF properties
n(θi,φi)
(θo,φo)
fr(θi,φi; θo,φo) −→ fr(θi, θo,φi − φo)
ISOTROPY
28
Reflectance Modeling
Spatial coherence Compressibility
Reflectance Sharing: Three BRDF properties
[Zickler et al., T-PAMI 2006]
fr(x, y, θi, θo,φi − φo)
Reflectance Modeling
Evaluation: No spatial variation
uv
w
SLO
W
RAPID
(θi, θo,φi − φo) −→ (u, v, w) = ~q
29
Reflectance Modeling
Evaluation: No spatial variation
uv
w
( )1
( ) ( ) -N
i ii
f q p q q qλψ=
= +∑r r r r%
(θi, θo,φi − φo) −→ (u, v, w) = ~q
Reflectance Modeling
uv
w
( )1
( ) ( ) -N
i ii
f q p q q qλψ=
= +∑r r r r%
LAFORTUNE LAMBERTIAN LAFORTUNE WARDRBFACTUAL RBFACTUAL
[Oren, Nayar 2001] [Matusik et al., 2003]
Evaluation: No spatial variation
30
Reflectance Modeling
Example: Human Face
DIFFUSE SPECULAR
θr
xr
( , , , , )q x y u v w=r
aRGB(x, y)
Reflectance Modeling
hθ
2 dφ
Example: Human Face
32
Reflectance Modeling
Results
Reflectance Modeling
♦ Reciprocity (& isotropy):Helmholtz stereopsis– reconstruction
Summary
u
v
θh
2φd
♦ Separability:Color subspaces– reconstruction, recognition, motion
estimation, segmentation
♦ Spatial coherence, compressibility:Reflectance sharing– modeling, appearance capture
33
Reflectance Modeling
Some Future Work
♦Ubiquitous appearance capture– Spatial reflectance discontinuities– Interreflections; sub-surface scattering
[Nayar et al., 2004]
www.eecs.harvard.edu/[email protected]