from 2d images to 3d tangible models: reconstruction and visualization of martian rocks

EURON Summer School 2003

From 2D Images to 3D Tangible Models:Reconstruction and Visualization

of Martian Rocks

Cagatay Basdogan, Ph.D.College of Engineering

Koc University

EURON Summer School 2003

The Old Story ... (July 04, 1997)

Sojourner

EURON Summer School 2003

The New Story ... January 04, 2004

EURON Summer School 2003

Rock

Our goal ...

EURON Summer School 2003From 2D Images to 3D Models

rock

Range surfaces

registrationintegration

Acquisition Visualization

Reconstruction Transmission

Left Right 1

2

4

3

EURON Summer School 2003

Limitations ...

• CPU: 2MHz 2 GHz • Memory: 768Kb 256 Mb + 40 Gb• Transmission Rate: <5 bytes/sec 100 Mb/sec• Transmission Delay ~ 20 min. < 200 msec• Transmission Interval ~ 2 hours anytime• Transmission Reliability poor good

Sojourner Laptop

EURON Summer School 2003

Octree-BasedData Representation

and Integration

Range surfaces

ProgressiveTransmission

Merged Range Data

Multi-Resolution Transmitted Range Data

Iso-surfaceExtraction

3D Model

Registration

Transformed Range surfaces1

2

3

1 2

3

Image-Based Rendering

Data Acquisition

EURON Summer School 2003

Data Acquisition

JPL – Marsyard (http://marsyard.jpl.nasa.gov)

EURON Summer School 2003

Input: Range Scans

Left Image Right Image

3D range surface:coordinatesconnectivity

EURON Summer School 2003

3D Reconstruction

Range Surface 2

Range Surface 1 1. Registration

2. Integration

Range Surface 3

Top View

1

2

3

EURON Summer School 2003

3D Registration

Pair-wise registration problem: Given 2 overlapping range scans what is the rigid transformation, T,that minimizes the distance between them ?

T

QP

||||2

iiE TqpN

i

Solution: ICP algorithm (Besl and Kay, 1992)

doIdentify corresponding pointsCompute the optimal T

while (E < threshold)

EURON Summer School 2003

3D Registration : Computation of T

||)(||2

tRqp iiEN

i

R: Rotation Matrixt: translation vector

2/1)( MMMR T

Solution by Horn, 1990:

where,

Tii

N

i

qqppM ))((

qRpt

EURON Summer School 2003

3D Registration: Corresponding Points

Q

P

We now have a closed form solution, how do we get the corresponding points in two scans?

Nearest points along the direction of the normal at qi

Top View

1 2

3

EURON Summer School 2003

3D Integration

Problem: Given a set of registered range scans, reconstruct a 3D surface that closely approximates the original shape.

Methods:

• Delaunay-Based (Amenta et al., Siggraph’98)• Surface-Based (Turk and Levoy, Siggraph’94)• Volumetric (Curless and Levoy, Siggraph’96)

Computationally ComplexityRobustnessMemory Usage...

Which one is better ?

EURON Summer School 2003

3D Integration: Volumetric Approach

Our implementation:

Step 1: Merge the registered range surfaces using an octree

Step 2:Extract an isosurface using the Marching Cubes algorithm

Input:registered

range surfaces

VolumetricIntegration

Output:3D Mesh

EURON Summer School 2003

3D Integration (Step 1): Merge Range Images

rock

Mesh 1

Mesh 2

Mesh 3

Registered

Merged(no connectivity)

EURON Summer School 2003

root Level 0

Level 1

Level 2

3D Integration (Step 1): Data Representation Using an Octree

Weightedaveraging

Yemez & Schmitt, 1999

EURON Summer School 2003

Original Model: 47724 points

Max 50 points/octant Total: 1870 points

Max 100 points/octant Total: 1052 points

Max 500 points/octant Total: 273 points

2D Example: QuadtreeMax 1 point/rectangle

Data Reduction Using an Octree

EURON Summer School 2003Octree Representation: Our Implementation

root

1 2 3 4

1 2 3 4 1 2 3 4

1 2 3 4

1 2 3 4

Level 0

Level 1

Level 2

Level 3

1 2

3 4Reference:

Path : 1 3 1 1center

=estimatedcoordinateand shape

EURON Summer School 2003

0 1 1 0 0 0 1 0

1 0 1 1 0 0 0 0 0 0 0 1 0 0 1 0

0 12 3

4

56

0 1

2 345

6

0 1

2 345

6

0 1

2 345

6

1 0 0 0 0 0 0 1

0 1

2 345

6

1 0 0 0 1 0 0 0

octree root layer 0

layer 16

0

4

layer 2

layer 3

Octree Encoding

Yemez & Schmitt, 1999

Our implementation:

EURON Summer School 2003

Octrees for 3D Progressive Data Transmission

root Level 0

Level 1

Level 2

Level 5

Level 8

.

.

.

.

215 points

2813 points

5823 points

EURON Summer School 2003

3D Integration (Step 2): Iso-surface Extraction

ProgressiveTransmission

Merged Data

Transmitted Data

Iso-surfaceExtraction:

Marching Cubes

3D Mesh

v1x v1y v1zv2x v2y v2z…rgb1rgb2...n1x n1y n1zn2x n2y n2z...

Coordinates

Colors

Normals

High Priority

Low Priority

EURON Summer School 2003

2-D Example

INOUT

24 = 16 possible cases

3D Integration (Step 2): Marching Cubes

isoline

+0.3 -0.5

+0.4+0.8

EURON Summer School 2003

28 = 256 possible cases(a look-up table reduces to 15 cases)

(Lorensen et al., Siggraph’87)Marching Cubes Algorithm : 3D Case

EURON Summer School 2003

3D Integration: Step 2: Signed Distance Computation

isos

urfa

ce

-0.7 +0.1

+0.4-0.3

Voxel (IN)

Voxel (OUT)n1 n2

v1

v2

dot(v1,n1) > 0 IN, distance = -|v1|

dot(v2,n2) < 0 OUT, distance = +|v2|

Signed Distance:

EURON Summer School 2003

Can I estimate Normals?

Problem: Given a set of P unorganized sample points, estimate the point normals.

Algorithm by Hoppe (Siggraph’92):

Step 1: Tangent Plane Estimation

Step 2: Consistent Tangent Plane Orientation

EURON Summer School 2003

1. Tangent Plane Estimation

R ~ (

Noise Density

R

Sample point

Centeroid

Neighbor

N

Use covariance matrix to compute NSVD : eigenvalues :

eigenvectors : vvv

Nv

v

N ?

N ?

EURON Summer School 2003

Incorrect Surface Normals

Corrected Surface Normals

MST: A minimal spanning tree fora connected graph

4

8 7

9

10

144

6

2

12

11

8

2. Consistent Tangent Plane Orientation: Graph Optimization Problem

Cost on edge: 1 –Ni.Nj

Propagate along directions of low curvature!

Ni NjNj

Ni

EURON Summer School 2003

point-based rendering with colors

EURON Summer School 2003

Iso-surfaceExtraction

3D Mesh3D Texture

3D Visualization

EURON Summer School 2003

Autostereoscopic Visualization

3D Visualization without any eye wear !

Stereoscopic viewing Autostereoscopic viewing

Src: Stereographics Inc.

EURON Summer School 2003

LCDRLCDL

HolographicOptical Element

eyeReyeL

Stereoscopic viewing Autostereoscopic viewing

Stereoscopic Visualization

EURON Summer School 2003

Autostereoscopic Displays

Relatively new area !

Hale et al., 1997, SiggraphPerlin et al., 2000, Siggraph

• Re-imaging displays• Volumetric displays• Parallax displays

Classification by Hale et al.:

• Holograms• Parallax Barrier Displays• Lenticular Sheet Displays• Holographic Stereograms• Electro-Holography

holographic optical element

EURON Summer School 2003

CameraPosition

Far

Dis

tanc

e

Nea

r D

ista

nce

Width A

ngle

Height Angle

HolographicPlate

Origin (0,0,0)

CameraPosition

b) AsymmetricFrustum

a) SymmetricFrustum

ShearDirection

Stereo Rendering : Shear Transform

EURON Summer School 2003

AutostereoscopicDisplay

HapticInterface

r

x

yz

v

wL

wR

1

0)(tan

wvQ

wvI T

S

Shear Transform:

1000

01

0010

0001

z

y

z

xcamera

r

r

r

rS

zyx rrrEye pos:

EURON Summer School 2003

Haptic Visualization

• Rendering rock textures• Displaying 3D rock shapes• Tele-science experiments• Guiding user’s movements• Positioning rover instruments

HIP

EURON Summer School 2003

Haptic Display of Shape

CollisionDetection

CollisionResponse

RockGeometry

MaterialPropertiesof Rock

Object Database

PositionOrientation

ContactInformation

ForceTorque

Rock

EURON Summer School 2003

Mapping Between Visual and Haptic Workspaces

3D modelof a rock

Haptic Workspace

Visual Workspace

T3= T1.T2

T1

T2

EURON Summer School 2003

Synchronization of Cursor Movements

HIP T2Display

Visual Cursor

HIP (T2)-1 CollisionDetection

CollisionResponse(T2)

FFNEW

EURON Summer School 2003

EURON Summer School 2003

EURON Summer School 2003

• 3D Registration: problems with ICP, global registration• 3D Integration: robustness, storage requirements• 3D Transmission: 3D geometry comp. vs 3D data comp.• More effective transmission of normals and colors• 3D Visual and Haptic Texturing• Image-Based rendering• Optimized computation (e.g. efficient data structures such as ADFs)• Missing link between image analysis and 3d modeling• More efficient graphical rendering (e.g. point-based rendering)• Missing link between real-time 3D modeling and rover navigation• Unified data structures for transmission of multi-modal data

Unsolved/Untouched Problems

On-

boar

dC

ompu

tati

on

Resources

EURON Summer School 2003Acknowledgements:

Supporting NASA Programs: IPN, ISE, CISMISS, USRP, SBIR, JPL/Caltech-UROP

JPLAndres Castano (acquisition, registration)Aaron Keily (encoding)Ed Chow, Jose Salcedo (autostereoscopic visualization)Larry Bergman (human-rover interactions)

Industry (Physical Optics Corporation)Steve Cupiac (autostereoscopic visualization)Andrew Kostrzewski, Kirill Kolesnikov (autostereoscopic display; hardware integration)

StudentsMitch Lum, UW (autostereoscopic and haptic visualization)Elaine Ou, Caltech (vision and touch)

UniversityProf. Marc Levoy, Stanford (3D range scans of “bunny”, Scanalyze registration software; personal communication)Prof. Brian Curless, UW (3D photography notes; public domain)Dr. Huges Hoppe, Microsoft (Ph.D. Thesis and 3D reconstruction code; public domain)

EURON Summer School 2003

0.60

1.20

1.80

2.40

3.00%

Mea

n E

rror

Rel

ativ

e to

BB

ox

Dia

gon

al o

f H

igh

Res

olu

tion

Mod

el

4 6 8

Number of Octree Layers

AveragedCenter

(857 pts) (4834 pts) (5823 pts)

High ResolutionModel (35947 pts)

3(215 pts)

5 7

EURON Summer School 2003

0.60

1.20

1.80

2.40

3.00

% M

ean

Err

or R

elat

ive

to B

Box

D

iago

nal

of

Hig

h R

esol

uti

on M

odel

4 6 8

Number of Octree Layers

Using Estimated NormalsUsing Computed Normals

(931 pts) (13156 pts) (35781 pts)

High ResolutionModel (35947 pts)

3(220 pts)

5 7

3.60

EURON Summer School 2003

Octrees for 3D Data Transmission: Encoding

0 1

2 3

4

56

3 bites/octant

000 0001 1010 2..111 7

8 layers = 28 X 28 X 28 cubesResolution = 1000 mm / 28 = ~ 4 mm !!!

a*22 + b*2 + c

abc

1m

1m

1m

Path to a leaf node: 3 4 1 7 0 3 3 2

Ref: Yemez and Schmitt, 1999

EURON Summer School 2003

Path to a leaf node: 3 4 1 7 0 3 3 2

Layer 1 … Layer 8

Octrees for Progressive Transmission

All data is transmitted at once (Maximum 8 layers):

28 X 28 X 28 cubes * 3 bites/cube * 1 byte/8 bits = ~ 6.3 MB !!

Progressive Transmission:

If we transmit the differencebetween layer 4 to 5 : ~ 10 KB ! (not even compressed)

EURON Summer School 2003

EURON Summer School 2003

A Simple 3D Example:

Outputv1x v1y v1zv2x v2y v2z…v10x v10y v10zn1x n1y n1zn2x n2y n2z…n10x n10y n10z

Input Data:

Voxelization

EURON Summer School 2003

2D:• Perspective • Occlusion• Lighting, shadows• Relative motion• Texture

Stereoscopic Visualization: Depth Perception

3D:• Binocular disparity • Accommodation• Convergence

Stereo Pairs

EURON Summer School 2003

Stereo Rendering

Incorrect ! Correct !