shape from shading and photometric stereotwo shape-from-x methods that use shading 1.shape from...
TRANSCRIPT
1
CSE152, Spring 2020
Shape from Shadingand
Photometric Stereo
Introduction to Computer VisionCSE 152
Lecture 11
1
CSE152, Spring 2020
Announcements
• HW2 posted, due Friday 2/14– HW as modified (1.b extra credit)
• Presidents day (No Class), Mon 2/17• Midterm, Wed 2/19
– Sample questions are posted to the class web page.
2
2
CSE152, Spring 2020
MidtermWed, Feb. 19
• In class• Full period• Coverage – everything up to today’s lecture• The primary source of material to understand is
what was presented in class and on HW. If it only in the readings, then it’s very unlikely to be in the midterm.
• “Cheat sheet” – you can prepare a one-sided sheet of notes. It must be handwritten. (After the midterm, save your sheet since you can use the other side for the final).
• No calculators.
3
CSE152, Spring 2020
Incomplete list of topics covered…• Camera models• Factors in producing
images• Homogenous Coordinates, • Vanishing points• Lenses, Distortion• Sensors• Rigid transformation• Intrinsic and extrinsic
camera models
• Quantization/Resolution • Illumination• Reflectance
– BRDF– Lambertian– Specular– Phong
• Photometric Stereo– Lambertian– Normal estimation
4
3
CSE152, Spring 2020
Topics cont.• Noise
– Additive, Gaussian noise• Filtering, linear,
convolution with Kernel– Averaging/smoothing– correlation– Sharpening– Derivatives– Gaussian filter– Seperability– Median filers
• Corner detection• SIFT
• Stereo– Epipolar constraint– epipoles– Rectification– Reconstruction– Essential matrix– Fundamental matrix– 8 point algorithm– Calibration– Matching – trinocular
• Structure from Motion– RANSAC– Constraints– Bundle adjustment
5
CSE152, Spring 2020
A few more words on Photometric Image Formation
6
4
CSE152, Spring 2020
Last lecture: Photometric Image Formation
• Light gathering limitation of pinhole camera & lenses
• Thin lens – equation, focus, depth of field, circle of confusion,
• field of view• Aberations – sphereical, astymatism,
chromatic, distortion, vignetting• Light sources, • Color cameras• Radiometry – radiance, irradiance• BRDF7
CSE152, Spring 2020
Radiance• Power traveling at some point
in a specified direction, per unit area perpendicular to the direction of travel, per unit solid angle
• Units: watts per square meter per steradian : w/(m2sr1)
Irradiance• How much light is arriving at a
surface?• Irradiance E(x) – power per unit
area: W/cm2
• Total power arriving at the surface is given by adding irradiance over all incoming angles
xx
dA
q
dw
(q, f)
wq ddAPL)cos(
=
8
5
CSE152, Spring 2020
BRDF• Bi-directional Reflectance
Distribution Function r(qin, fin ; qout, fout)
• Function of– Incoming light direction:
qin , fin– Outgoing light direction:
qout , fout
• Ratio of incident irradiance to emitted radiance
n(qin,fin)
(qout,fout)
Li
Lo E
I
• Radiance Li from light source strikes surface and leads to radiance Lo which strikes image plane with Irradiance E causing pixel intensity I.
9
CSE152, Spring 2020where ωi=(θi, ϕi)
Emitted radiance Lr(x,ωr) in direction ωr for incoming radiances over the hemisphere
Lr (x,ωr ) = ρ(x,ωi ,ωr )H 2∫ Li (x,ωi )cosθidωi
10
6
CSE152, Spring 2020
Surface Reflectance Models
• Lambertian (1 parameter)• Phong (4 parameters)• Physics-based
– Specular [Blinn 1977], [Cook-Torrance 1982], [Ward 1992]
– Diffuse [Hanrahan, Kreuger 1993]
– Generalized Lambertian [Oren, Nayar 1995]
– Thoroughly Pitted Surfaces [Koenderink et al 1999]
– Hair, Skin, Cloth
Common Models Arbitrary Reflectance
• Phenomenological[Koenderink, Van Doorn 1996]
• Non-parametric models• Anisotropic• Non-uniform over surface• BRDF Measurement [Dana et
al, 1999], [Marschner ]
11
CSE152, Spring 2020
Lambertian Surface
The intensity (irradiance) I(u,v) of a pixel at (u,v) is:
• a(u,v) is the albedo of the surface projecting to (u,v)• n(u,v) is the direction of the surface normal• s0 is the light source intensity• s is the direction to the light source
ns
a
I(u,v)
^
I (u,v) = a(u,v)n(u,v)• s0s
12
7
CSE152, Spring 2020
Full Phong Reflectance Model
where
ia, im,d, im,s are brightness of ambient, diffuseand specular light sources.
ka, kd, ks,𝛼 parameterize Phong BRDF at a point
13
CSE152, Spring 2020
Shadows cast by a point source• A point that can’t see the source is in shadow• For point sources, two types of shadows: cast
shadows & attached shadows
Cast Shadow
Attached Shadow
14
8
CSE152, Spring 2020
Penumbra and Umbra(soft vs. hard shadow)
15
CSE152, Spring 2020
Hard vs soft shadows for different light source
Hard shadow(point light)
Soft shadow(small square light source)
Z. Dimitry, https://slideplayer.com/slide/5184420/
16
9
CSE152, Spring 2020
Hard vs Soft Shadow
17
CSE152, Spring 2020Figure from “Mutual Illumination,” by D.A. Forsyth and A.P. Zisserman, Proc. CVPR, 1989, copyright 1989 IEEE
At the top, geometry of a gutter with triangular cross-section; below, predicted radiosity solutions, scaled to lie on top of each other, for different albedos of the geometry. When albedo is close to zero, shading follows a local model; when it is close to one, there are substantial reflexes.
Inter-reflections
18
10
CSE152, Spring 2020
Shading reveals 3-D surface geometry
19
CSE152, Spring 2020
Shape-from-X• Where X is
– Shading– Photometric stereo– Stereo– Motion– Texture– Blur– Focus– Structured light– ….
20
11
CSE152, Spring 2020
Two Shape-from-X methods that use shading1. Shape from shading: Single Image, known
light direction, known BRDF.
2. Photometric stereo: Single viewpoint, multiple images under different lighting.
21
CSE152, Spring 2020
Multi-view stereo vs. Photometric Stereo
• Multi-view (binocular) Stereo – Multiple images– Dynamic scene– Multiple viewpoints– Fixed lighting– Correspondence Hard
• Photometric Stereo– Multiple images– Static scene– Fixed viewpoint– Multiple lighting conditions– Correspondence trivial
22
12
CSE152, Spring 2020
Photometric Stereo Rigs:Fixed camera, changing lighting
Because of single fixed viewpoint, a pixel location sees the same point of the scene across all images.
23
CSE152, Spring 2020
An example of photometric stereo
Input Images
24
13
CSE152, Spring 2020
Photometric Stereo Process
1. From a single viewpoint of a static scene, acquire k images under different lighting.
2. Independently, estimate the surface normal for each pixel location from k measurements.
3. Integrate the normals to estimate depth across the surface.
25
CSE152, Spring 2020
Assumptions
We generally make the following simplifying assumptions:1. Light source is distant2. BRDF is known and has a few parameters
26
14
CSE152, Spring 2020
BRDF
• Bi-directional Reflectance Distribution Function
r(qin, fin ; qout, fout)
• Function of– Incoming light direction:
qin , fin– Outgoing light direction:
qout , fout
• Ratio of incident irradiance to emitted radiance
n(qin,fin)
(qout,fout)
27
CSE152, Spring 2020
Coordinate system
x
y
f(x,y)
Surface: s(x,y) =(x,y, f(x,y))
(x,y)
∂s(x, y)∂x
= 1,0, ∂f∂x
⎛
⎝⎜
⎞
⎠⎟
∂s(x, y)∂y
= 0,1, ∂f∂y
⎛
⎝⎜
⎞
⎠⎟
Tangent vectors:
s(x,y)
Normal vector
n = ∂s∂x×∂s∂y
= −∂f∂x,−∂f∂y,1
⎛
⎝⎜
⎞
⎠⎟
Not unit length
𝜕𝑠𝜕𝑥
𝜕𝑠𝜕𝑦n
28
15
CSE152, Spring 2020
Representing Surface Normals• The surface normal is a direction (vector)
orthogonal to the tangent plane• Unit vector n where |n| = 1. • n lies on the unit sphere• We can also represent normal in Gradient
Space as slant 𝑝 = ()(*
and tilt q= ()(+
^^
^
29
CSE152, Spring 2020
Photometeric Stereo for Lambertian Surface with Distant Known
Lighting
30
16
CSE152, Spring 2020
Distant Light Source
s
e(u,v)
• When light source is “distant”, like the sun1. The direction ,𝒔 to the light source is constant over the
surface2. The distance from points on the surface to light location is
constant, and so brightness of light doesn’t vary.• So, we can treat lighting as a constant direction ,𝒔 and strength s0
31
CSE152, Spring 2020
Lambertian Surface
At image location (u,v), the intensity of a pixel x(u,v) is:
e(u,v) = [a(u,v) n(u,v)] · [s0s ] = b(u,v) · swhere
– a(u,v) is the albedo of the surface projecting to (u,v).– /𝒏(u,v) is the direction of the surface normal.– s0 is the light source intensity.– ,𝒔 is the direction to the light source.
And– 𝒔 = 𝒔𝟎,𝒔– 𝒃 𝑢, 𝑣 = 𝑎(𝑢, 𝑣)/𝒏(𝑢, 𝑣)
ns
^ ^
a
e(u,v)
32
17
CSE152, Spring 2020
Normal Field – Normals at every location
33
CSE152, Spring 2020
Lambertian Photometric stereo• If the light sources s1, s2, and s3 are known, then we can
recover b from as few as three images. [Silver 80, Woodham81]
• For a single pixel location in 3 gray scale images, we have
[e1 e2 e3 ] = bT[s1 s2 s3 ] where e1, e2, and e3 are measured pixel intensities and we know s1, s2, and s3. We can then compute b at the pixel location by solving a linear system.
• Normal is: = b/|b|, albedo is: a = |b|
[ ][ ] 1321-= 321
T sssb eeen
34
18
CSE152, Spring 2020
Computing b for every pixel• Note: S-1 = [s1 s2 s3 ]-1 only needs to be computed. Not per
pixel.• Slow way: Loop over every pixel and apply
• Better: For an image with n pixels,• Let
where ei,j is the measured image intensity at pixel i in image j, and row i of B is the surface normal scaled by the albedo (b) for pixel i
• So image formation equation is E = BS• Solving for B, we have: B = ES-1
E =
e1,1 e1,2 e1,3e2,1 e2,2 e2,3e3,1 e3,2 e3,3
!
⎡
⎣
⎢⎢⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥⎥⎥
, B =
b1,1 b1,2 b1,3b2,1 b2,2 b2,3b3,1 b3,2 b3,3
!
⎡
⎣
⎢⎢⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥⎥⎥
, S = s1 s2 s3⎡⎣⎢
⎤⎦⎥
[ ][ ] 1321-= 321
T sssb eee
35
CSE152, Spring 2020
What if we have more than 3 Images?Linear Least Squares
[e1 e2 e3…en] =bT[s1 s2 s3…sn]
Rewrite as e = Sb
wheree is n by 1b is 3 by 1S is n by 3
Let the residual ber = e - Sb
Squaring this: r2 = rTr = (e-Sb)T (e-Sb)
= eTe - 2bTSTe + bTSTSb
∇𝒃(r2) = 0 --zero gradient with respect to b is a necessary condition for a minimum, or-2STe+2STSb=0;
Solving for b yields
b= (STS)-1STe
36
19
CSE152, Spring 2020
Plastic Baby Doll: Normal Field
37
CSE152, Spring 2020
Next step:Go from normal field to surface
38
20
CSE152, Spring 2020
Depth recover as solution to partial differential equation
• From estimated n =(nx, ny, nz) at each pixel (x,y), compute the slant p(x,y) and tilt q(x,y)
p(x,y) = - nx(x,y)/nz(x,y)q(x,y) = - ny(x,y)/nz(x,y)
• But p and q are just just the partial derivatives of the depth function f(x,y) with respect to x and y.
• System of two first order partial differential equations (pde)
• Assume surface and derivatives are continuous and then solve pde for f(x,y)
∂f (x, y)∂x
= p(x, y)
∂f (x, y)∂y
= q(x, y)
39
CSE152, Spring 2020
Step 2: Recovering the surface f(x,y)Many methods: Simplest approach1. Integrate along the top row (x,0) to get f(x,0)
2. Then integrate along each column starting with f(x,0) of the first row
f(x,0)
∂f∂x
= p(x, y)
∂f∂y
= q(x, y)
f (x,0) = p(x ',0)dxx '=0
x
∫ ' or for a discrete image f(i,0) = p( j,0)j=1
i
∑ = f (i −1,0)+ p(i,0)
40
21
CSE152, Spring 2020
What might go wrong?
• Height z(x,y) is obtained by integration along a curve from (x0, y0).
• If one integrates the derivative field along any closed curve, on expects to get back to the starting value.
• Might not happen because of noisy estimates of (p,q)
ò ++=),(
),(00
00
)(),(),(yx
yx
qdypdxyxzyxz
[not responsible for this on exams]41
CSE152, Spring 2020
What might go wrong?
yf
xxf
y ¶¶
¶¶
=¶¶
¶¶
xq
yp
¶¶
=¶¶
Integrability. If f(x,y) is the height function, we expect that
In terms of estimated gradient (p,q), this means:
where p=-nx/nz, q=-ny/nz with n=[nx ny nz]
But n (and in turn p, q) are estimated independently at each pixel, equality is not going to exactly hold
[not responsible for this on exams]42
22
CSE152, Spring 2020
Horn’s Method[ “Robot Vision, B.K.P. Horn, 1986 ]
• Formulate estimation of surface height z(x,y) from gradient field by minimizing cost functional:
where (p,q) are estimated components of the gradient while zx and zy are partial derivatives of best fit surface
• Solved using calculus of variations – iterative updating
• z(x,y) can be discrete or represented in terms of basis functions.
• Integrability is naturally satisfied.
dxdyqzpz yx22
Image
)()( -+-òò
[not responsible for this on exams]43
CSE152, Spring 2020
Input Images
44
23
CSE152, Spring 2020
Recovered albedo
45
CSE152, Spring 2020
Recovered normal field(shown on recovered 3D structure)
46
24
CSE152, Spring 2020
Surface recovered by integration
47
CSE152, Spring 2020
Lambertian Photometric Stereo
Reconstruction with Albedo Map
48
25
CSE152, Spring 2020
Without the albedo map
49
CSE152, Spring 2020
Another person
50
26
CSE152, Spring 2020
No Albedo map
51
CSE152, Spring 2020
Artifacts from assuming Lambertian Model
52