Reconstruction of 3D Models from Intensity Images and Partial Depth

L. Abril Torres-Méndez and Gregory Dudek

Centre for Intelligent Machines

School of Computer Science

McGill University

Our Application

• Automatic generation of 3D maps.• Robot navigation, localization

- Ex. For rescue and inspection tasks.

• Robots are commonly equipped with camera(s) and laser rangefinder.

Would like a full range map of the

the environment. Simple acquisition of data

Problem Context

• Pure vision-based methods – Shape-from-X remains challenging, especially in

unconstrained environments.

• Laser line scanners are commonplace, but– Volume scanners remain exotic, costly, slow.– Incomplete range maps are far easier to obtain that complete

ones.

Proposed solution: Combine visual and partial depth

Shape-from-(partial) Shape

Problem Statement

From incomplete range data combined with intensity, perform scene recovery.

From range scans like thisinfer the rest of the map

Overview of the Method

• Approximate the composite of intensity and range data at each point as a Markov process.

• Infer complete range maps by estimating joint statistics of observed range and intensity.

What knowledge does Intensity provide about Surfaces?

• Two examples of kind of inferences:

Intensity image Range image

surface smoothness

variations in depth

surface smoothness

far

close

What about outliers?

• Changes in intensity from texture-like regions

Intensity Range

Not critical for reconstruction since very close neighborhoods represent the same type of smooth surface.

What about Edges?

• Edges often detect depth discontinuities• Very useful in the reconstruction process!

Intensity Rangeedges edges

Range synthesis basis

Range and intensity images are correlated, in complicated ways, exhibiting useful structure.

- Basis of shape from shading & shape from darkness, but they are based on strong assumptions.

The variations of pixels in the intensity and range images are related to the values elsewhere in the image(s).

Markov Random Fields

Related Work

• Probabilistic updating has been used for – image restoration [e.g. Geman & Geman, TPAMI

1984] as well as

– texture synthesis [e.g. Efros & Leung, ICCV 1999].

• Problems: Pure extrapolation/interpolation:– is suitable only for textures with a stationary

distribution

– can converge to inappropriate dynamic equilibria

MRFs for Range Synthesis

States are described as augmented voxels V=(I,R,E).

ZZmm=(x,y):1≤x,y≤m=(x,y):1≤x,y≤m: mxm lattice over which the image are described.

I = {II = {Ix,yx,y}, (x,y)}, (x,y) Z Zmm: intensity (gray or color) of the input image

E is a binary matrix (1 if an edge exists and 0 otherwise).

R={RR={Rx,yx,y}, (x,y)}, (x,y) Z Zmm: incomplete depth values

We model V as an MRF. I and R are random variables.

RI vx,y

AugmentedRange Map

IR

Markov Random Field Model

Definition: A stochastic process for which a

voxel value is predicted by its neighborhood

in range and intensity.

€

P(Vx,y = vx,y |Vk,l = vk,l ,(k, l) ≠ (x,y)) =

€

P(Vx,y = vx,y |Vk,l = vk,l ,(k, l)∈ Nx,y )

Nx,y is a square neighborhood of size nxn centered at voxel Vx,y.

Computing the Markov Model

• From observed data, we can explicitly compute

€

P(Vx,y = vx,y |Vk,l = vk,l ,(k, l)∈ Nx,y )

intensity

intensity & range

Vx,y

Nx,y

• This can be represented parametrically or via a table.–To make it efficient, we use the sample data itself as a table.

Further, we can do this even with partial

neighborhood information.

Estimation using the Markov Model

• From

what should an unknown range value be?

For an unknown range value with a known

neighborhood, we can select the maximum

likelihood estimate for Vx,y.

€

P(Vx,y = vx,y |Vk,l = vk,l ,(k, l)∈ Nx,y )

Even further, if both intensity and range are

missing we can marginalize out the unknown

neighbors.

intensity

intensity & range

Interpolate PDF• In general, we cannot uniquely solve the desired neighborhood

configuration, instead assume

€

P(Rx,y = rx,y | Ix,y = ix,y , Vk,l = vk,l , (k, l)∈ N x,y ) ≈

P(Ru,v = ru,v | Iu,v = iu,v , Vp,q = v p,q , ( p, q)∈ N u,v )

The values in Nu,v are similar to the values in Nx,y, (x,y) ≠ (u,v).

Similarity measureSimilarity measure:: Gaussian-weighted SSD (sum of squared differences).

Update schedule is purely causal and deterministic.

Order of Reconstruction

• Dramatically reflects the quality of the result.• Our reconstruction sequence is based on the amount

of reliable information surrounding each of the voxels to be synthesized.

• Edge information used to defer reconstruction of voxels with edges as much as possible.

Info-edge-driven ordering Correct result With the spiral-ordering

Experimental Evaluation

• Obtain full range and intensity maps of the same environment.

• Remove most of the range data, then try and

estimate what it is.

• Use the original ground truth data to

estimate accuracy of the reconstruction.

Arbitrary shape of unknown range data

Input Intensity Synthesized result

Scharstein & Szeliski’s Data Set Middlebury College

Compact case

Ground truth rangeInput range imageSynthesized resultGround truth range

Input IntensityGround truth range Input range image

Arbitrary shape of unknown range data

Scharstein & Szeliski’s Data Set Middlebury College

Less compact case

Synthesized result

Arbitrary shape of unknown range data

Compact DistributedSynthesized results

Ground truth range

• Expected quality of reconstruction degrades with distance from known boundaries• Need broader distribution of range-intensity combinations in the sampling

Input IntensityEdge map

Stripes along the x- and y-axis

Data courtesy of Oak Ridge National Labs & Univ. of Florida

Ground truth rangeApproximated depth size: 650cms.

Input range 66% missing range

Synthesized ResultMean Absolute Residual (MAR) Error: 5.76 cms.

Input intensity and range data Synthesized results Ground truth range

Input range data. 66% of missing range

Achromatic vs. Color Results

Using achromatic image Using color image

Synthesized range images

Greyscale image Color imageGround truth range

More Experimental Results

Initial range scans

Results on Realistic Sampling

Not in the paper

Real intensity image

Ground truth range

Edge map

Synthesized range image

What happened?

• Problems with surfaces that slope Intensity-range combinations from the initial range do not capture the underlying structure.

• Solution: Surface normal-based MRF range synthesis

– Surface normals from initial range data is added

to our MRF model so that depth values are computed according to the surface being reconstructed.

Adding Surface NormalsNot in the paper

• We compute the normals by fitting a plane (smooth surface) in windows of nxn pixels.

• Normal vector: Eigenvector with the smallest eigenvalue of the covariance matrix.

• Similarity is now computed between surface normals instead of range values.

Initial range scans

Preliminary Results Not in the paper

Synthesized range image Ground truth range

Edge map Real intensity image Initial range dataReal intensity image Edge map

Conclusions

• Works very well -- is this consistent?• Can be more robust than standard methods (e.g.

shape from shading) due to limited dependence on a priori reflectance assumptions.

• Depends on adequate amount of reliable range as input.

• Depends on statistical consistency of region to be constructed and region that has been measured.

Discussion & Ongoing Work

• Surface normals are needed when the input range data do not capture the underlying structure

• Data from real robot – Issues: non-uniform scale, registration,

correlation on different type of data

Questions ?

Input intensity Ground truth range

Range synthesis using edge info

Initial range data (white regions are to be estimated)

Range synthesis (no edge info)

Animated example

Input intensity Ground truth range

Range synthesis using edge info

Initial range data (white regions are to be estimated)

Range synthesis (no edge info)

Animated example

I

V(x,y)

R

Algorithm Outline

• The depth value of the augmented voxel Vx,y is synthesized as follows:

d

Synthesis orderingSynthesis ordering: according to available information. (maximum number of “full” neighbors)

argmax(argmax(PP))

Compactness• Reconstruction facilitated by less compact

regions.

• Experimental illustration: keep area constant, vary the shape of area being recovered.

Not in the paper

Compactness

Average errors are 2.8% and 1.6%.

Expected quality of reconstruction degrades with distance from known boundaries.

Not in the paper

Res

idua

l Err

or in

of

Rec

onst

ruct

ed D

epth

(fro

m 0

to 2

0%)

Amount to data missing

Not in the paper

Depth histogram: ground truth & reconstructedNot in the paper

Technical Aside• Really should have independent relations for

R(x,y) and I(x,y).

€

P(Rx,y = rx,y | Ix,y = ix,y,Vk,l = vk,l ,(k, l)∈ N)

Augmented Range Map

• Incorporate appearance values, expressing both spatial occupancy as well as appearance under specific viewing conditions, into augmented range map.

Mode of Absolute Residual Error

The Neighborhood

• The neighborhood N{x,y} is a square mask of size nxn centered at the augmented voxel.

• Observations: The neighborhood is causal The size is important to capture relevant characteristics

v(x,y)

IR

Neighborhood of v(x,y)

N{x,y}

Initial range (none)

Novel View Synthesis (ongoing work)

Intensity image Range image

Known intensity image

• Infer range map from intensity of current view and from range and intensity of other views.

Other view

Current view to estimaterange map

Synthesized range Ground truth range

Previous results Results using EDGES Ground truth range MAR Errors (cms.)

10.40 8.58

16.58 13.48

12.16 11.39

19.17 7.12

Case 2 Result

Mean Absolute Residual (MAR) Error: 18.98 cms.

75% missing rangeInput intensity

Case 3 Result

Mean Absolute Residual (MAR) Error: 20.23 cms.

78% missing rangeInput intensity

More examplesSynthesized results Ground truth rangeThe input intensity and range data

Order of Reconstruction

• Dramatically influences the quality of final result.• Spiral scan-ordering (a.k.a.“onion peeling”) Disadvantage:

strong dependence from the previous assigned depth value.

With the spiral-ordering Correct result

Adding Surface NormalsNot in the paper

• We compute the normals by fitting a plane (smooth surface) in windows of nxn pixels.

• The eigenvector with the smallest eigenvalue of the covariance matrix is the normal vector.

– The smallest eigenvalue measures the quality of the fit.

• In the range synthesis, the similarity is now computed between surface normals instead of range values.