1 Ellen L. Walker

3D Vision

Why? The world is 3D

Not all useful information is readily available in 2D

Why so hard? “Inverse problem”: one image = many scenes

Complex relationship between objects & pixels

Noise, occlusion, etc.

What can we do? Use "hints" from our knowledge of the world

Add more information to the problem!

2 Ellen L. Walker

Labeling Image Edges

Many edge labels carry 3D information Occluding blade (>)

Occluding surface to right along arrow

Convex crease (+) Edge is closer than both surfaces

Concave crease (–) Edge is further than both surfaces

Limb (>>) Edge is "horizon" of curving-away surface

Others are about reflectance or illumination changes Mark (M)

Change due to paint or material boundary

Illumination Boundary (S) Shadow edge

3 Ellen L. Walker

Intrinsic Image Pixel Contents

Depth (range)

Orientation (surface normal)

Illumination

Albedo (reflectance)

Given this information, the picture (intensity values) can be completely reconstructed

4 Ellen L. Walker

3D Cues in Single 2D Images

Occlusion Occluding objects are closer to the camera The crossbar of a T-junction belongs to the occluding

object

Perspective scaling and foreshortening If two copies of the same object appear in a picture, the

smaller one is further away. Scaling is parallel to the image plane Foreshortening is perpendicular to the image plane

Texture Gradient Since texture is composed of repeated patterns, changes

in size and density of texture convey depth cues

5 Ellen L. Walker

"Shape-From" Methods

Use a cue (e.g. texture, shading) from a small region of an image

Cues generally give partial surface orientation information E.g. degree of tilt

Related cues can give "boundary conditions" to start from

Solve for continuous surfaces that satisfy both the general and boundary constraints

6 Ellen L. Walker

Example: Shape From Shading

Figure 12.2

7 Ellen L. Walker

Gradient Space Represents Surface Normal

p

q

8 Ellen L. Walker

Reflectance Geometry

Three directions are important: normal to the surface, surface to light source, and surface to camera

surface patch

light source

normal vector

Camera

9 Ellen L. Walker

Types of Reflection (Review)

Specular reflection (mirror) Color depends on light source color Limited scattering: angle of incidence = angle of reflection Camera sees light if it’s pointed in the right direction Nearly all light is reflected

Lambertian reflection (matte) Color depends on material properties of object Light evenly scattered throughout half-space Camera sees light if surface is visible Amount of light reflected is proportional to angle between

surface and light source

10 Ellen L. Walker

Shape From Shading

Lambertian surface, known light source direction Reflective component (highlights) can be subtracted out in

preprocessing

Relative brightness of surface patches constrain their directions Darker patches are more tilted away A given brightness value represents a circle in gradient

space

Boundary pixels indicate surface at 90 degrees from normal (if smooth surface)

Solve an optimization problem: brightness term and smoothness term

11 Ellen L. Walker

Shape From Shading (cont’d)

Brightness term Intensity is a function of reflectance, and reflectance is a

function of surface normals (p,q) and light source direction (vx, vy, vz)

Smoothness term Try to minimize integral of partial derivatives of p and q in

x and y direction€

I(x,y)= R(p(x,y),q(x,y))

R(p,q) = max(0,ρpvx +qvy + vz1+ p2 +q2

)

€

ES = px2∫ + py

2 +qx2 +qy

2dxdy

12 Ellen L. Walker

Photometric Stereo

Extension to multiple "images" Lambertian surface, several light sources

Each image has one light source, constrains surfaces

Solve an overdetermined linear (matrix) system - like camera calibration with extra points

Implementation Surround your object with a frame containing inward-

pointing lights

Take an image with each light in turn

Use images and known light directions to solve the equations.

13 Ellen L. Walker

More Variations

SHINY Photometric stereo system for highly reflective materials

Used to accurately characterize welds

Accurate color determination (plastic objects) Separate highlights from matte portion

Determine illumination color from highlight

Determine object color

Create “matte object” for photometric stereo or shape from shading

14 Ellen L. Walker

Shape From Texture

Figure 12.3

15 Ellen L. Walker

Shape from Texture

Transformation of original texture related to surface normal

Solve for affine transformation between original texel and viewed texel

Transformation depends on surface normal & distance

(Assume camera is far enough to avoid worst perspective distortion)

If the original texel is known, transformations can be computed directly

If the original texel is unknown, assume the largest visible texel is directly facing the camera

Use smoothness or shape constraints to eliminate alternatives

16 Ellen L. Walker

Shape from Focus

Vary the focal length of the camera (i.e. zoom lens)

Objects at different distances will become clear at different focal lengths

Can use comparisons between pairs of images to get relative distances

17 Ellen L. Walker

Choice Depends on Object

Solid, matte object (or matte separated from specular) Shape from shading, photometric stereo

Highly reflective object Shape from specular reflection

Regular textured object Shape from texture, stereo, focus

Irregular textured object Stereo, focus