Reverse Engineering

Niloy J. Mitra

Reverse Engineering

Capturing Geometry: 3D Scanners

laser scanner 3D geometry

Reverse Engineering

Shape Acquisition

Reverse Engineering

Shape Acquisition

Reverse Engineering

Shape Acquisition

Reverse Engineering

Shape Acquisition

Reverse Engineering

Shape Acquisition

Reverse Engineering

Partial Shape Similarity

partial similarity

Reverse Engineering

Total vs Partial Matching• Total matching is easy

PCA (Principal Component Analysis)

Reverse Engineering

Total vs Partial Matching• Partial matching is difficultWhich region matches which other region(s)?Space of rigid transforms rotation + translation

Brute force approach not feasible

Instead of exhaustive searching, use local geometry to guide where to search

Easy to verify a transform

Reverse Engineering

Descriptor Based Alignment

[Kazhan et al. `03][Osada et al. `02]

• Feature based alignment• Combinatorial search, need multiple objects together

• Descriptor based alignment • Fails for partial similarity

Reverse Engineering

• What is registration?• Finding a one-to-one mapping between two or more coordinate systems such that

corresponding features of models in the different systems are mapped to each other• Using the mapping to align a model(s)

• Pair-wise model alignment• Transformation to a canonical pose/coordinate system

Audette 2000 M. Kazhdan

Registration

Reverse Engineering

• What is the resulting alignment/pose used for?• Object recognition in scenes• Stitching together parts of a model captured from different views• Alignment for pose-dependent shape descriptors

Funkhouser, COS 597D Class NotesChang and Krumm, 1999

OR

Registration

Reverse Engineering

Choice of Transformation• Rigid: mutual distances of points within a model are conserved

during transformation

• R is a rotation matrix and t is a translation vector

• Non-rigid• Account for surface deformations in the transformation• Affine transformation

Audette 2000

Registration

Reverse Engineering

• Use PCA to place models into a canonical coordinate frame

Covariance MatrixComputation

Principal Axis Alignment

M. Kazhdan

Normalization

Reverse Engineering

• Translate point set {pi} to origin by center of mass:

• Result is new point set {qi}cpq ii

n

iin 1

1pc

Steps for finding principal axes

Reverse Engineering

• Calculate second-order covariance matrix:

n

i zi

zi

yi

zi

xi

zi

zi

yi

yi

yi

xi

yi

zi

xi

yi

xi

xi

xi

qqqqqq

qqqqqq

qqqqqq

n 1

1M

Steps for finding principal axes

Reverse Engineering

• Decompose symmetric covariance matrix:

• Matrix U contains 3 principal axes (eigenvectors) as rows: A, B, C

• Matrix S contains eigenvalues

tUSUM

zyx

zyx

zyx

CCC

BBB

AAA

U

c

b

a

00

00

00

S

Steps for finding principal axes

Reverse Engineering

• Start with manual initial alignment

[Pulli][Pulli]S. Rusinkiewicz

Aligning Scans

Reverse Engineering

• Improve alignment using ICP algorithm

[Pulli][Pulli]S. Rusinkiewicz

Aligning Scans

Reverse Engineering

Pairwise Rigid Registration Goal

Align two partially-overlapping meshesgiven initial guessfor relative transform

S. Rusinkiewicz

Reverse Engineering

Aligning 3D DataIf correct correspondences are known, can find correct relative rotation/translation

S. Rusinkiewicz

Reverse Engineering

Aligning 3D Data• How to find correspondences: User input? Feature

detection? Signatures?• Alternative: assume closest points correspond

S. Rusinkiewicz

Reverse Engineering

Aligning 3D Data• … and iterate to find alignment

• Iterative Closest Points (ICP) [Besl & McKay 92]

• Converges if starting position “close enough”

S. Rusinkiewicz

Reverse Engineering

Basic ICP• Select e.g. 1000 random points• Match each to closest point on other scan,

using data structure such as k-d tree• Reject pairs with distance > k times median• Construct error function:

• Minimize (closed form solution in [Horn 87])

2 ii qtRpE2 ii qtRpE

S. Rusinkiewicz

Reverse Engineering

ICP VariantsVariants on the following stages of ICPhave been proposed:

1.1. SelectingSelecting source points (from one or both meshes) source points (from one or both meshes)2.2. MatchingMatching to points in the other mesh to points in the other mesh3.3. WeightingWeighting the correspondences the correspondences4.4. RejectingRejecting certain (outlier) point pairs certain (outlier) point pairs5.5. Assigning an Assigning an error metricerror metric to the current transform to the current transform6.6. MinimizingMinimizing the error metric w.r.t. transformation the error metric w.r.t. transformation

S. Rusinkiewicz

Reverse Engineering

Point-to-Plane Error MetricUsing point-to-plane distance instead of point-to-point lets flat regions slide along each other [Chen & Medioni 91]

S. Rusinkiewicz

Reverse Engineering

Point-to-Plane Error Metric• Error function:

where R is a rotation matrix, t is translation vector• Linearize (i.e. assume that sin , cos 1):

• Result: overconstrained linear system

2)( iii nqtRpE 2)( iii nqtRpE

z

y

x

iiiiii

r

r

r

rntnprnqpE where,)()(2

z

y

x

iiiiii

r

r

r

rntnprnqpE where,)()(2

S. Rusinkiewicz

Reverse Engineering

Point-to-Plane Error Metric• Overconstrained linear system

• Solve using least squares

22

111

222

11

)2(

)(1,,

,

nqp

nqp

t

t

t

r

r

r

nnp

nnp

bx

bx

z

y

x

z

y

x

A

A

22

111

222

11

)2(

)(1,,

,

nqp

nqp

t

t

t

r

r

r

nnp

nnp

bx

bx

z

y

x

z

y

x

A

A

bx

bxT1T

TT

AAA

AAA

bx

bxT1T

TT

AAA

AAA

S. Rusinkiewicz

Reverse Engineering

Closest Compatible Point• Closest point often a bad approximation to

corresponding point• Can improve matching effectiveness by restricting

match to compatible points• Compatibility of colors [Godin et al. 94]• Compatibility of normals [Pulli 99]

• Other possibilities: curvatures, higher-order derivatives, and other local features

S. Rusinkiewicz

Reverse Engineering

Global Registration Goal• Given: n scans around an object• Goal: align them all• First attempt: ICP each scan to one other

S. Rusinkiewicz

Reverse Engineering

Global Registration Goal• Want method for distributing accumulated error

among all scans

S. Rusinkiewicz

Reverse Engineering

• Search the whole model for correspondences• Range query for descriptor values• Cluster and pick representatives

Correspondence Space

PQ

Reverse Engineering

Evaluating Correspondences• Coordinate root mean squared distance

• Requires best aligning transform• Looks at correspondences individually

Reverse Engineering

Rigidity Constraint• Pair-wise distances between features and

correspondences should be the same

PQ

Reverse Engineering

Rigidity Constraint• Pair-wise distances between features and

correspondences should be the same

PQ

Reverse Engineering

Rigidity Constraint• Pair-wise distances between features and

correspondences should be the same

PQ

Reverse Engineering

Alignment Results

Input: 2 scans Our alignment Refined by ICP

Reverse Engineering

Alignment Results

Input: 10 scans

Our alignment Refined by ICP

Reverse Engineering

Non-rigid Alignment• Algorithm Input

• Set of human range images• All in similar poses

• Set of colored feature markers• Algorithm Goal

• Develop correspondence from template to target• Compute affine transform for each vertex• Minimize error function

T. Funkhouser

[Allen et al. 2003]

Reverse Engineering

Optimization Variables• Algorithm viewed as optimization problem

• Given an initial template surface with vertices vi

• Corresponding affine transformation matrices Ti

• Current state is Ti vi for all i (see diagram)

• Find values of Ti to minimize objective function• Attempts to find a “good fit” (blue) of template (cyan) to target

(red)

T. Funkhouser

Reverse Engineering

Objective Function• Objective Function has three weighted terms

• Data error• Smoothness error• Marker error

• Will use different weights in each phase of process• Multistep / Multi-resolution fitting process

T. Funkhouser

Reverse Engineering

Objective Function – Data Error• Data Error term requires current match to be close to target

• Uses distance from each transformed vertex to the target surface• Weighted by confidence measure (from scanning)• Hole regions have weight = 0• Sums total error

• Distance function• Uses transformed template vertex• Takes minimum distance to “compatible” vertices in target• Compatible defined as those with normal w/in 90 deg.

T. Funkhouser

Reverse Engineering

Objective Function – Smoothness Error• Measures smoothness of deformation applied to template

• Problem is under-constrained using data error• E_s measures change in T_I between adjacent vertices• Encourages similarly-shaped features to be mapped to each

other

• Uses Frobenius norm (vector L2 norm)

T. Funkhouser

Reverse Engineering

Objective Function – Marker Error• Data and Smoothness Error can hit local minima

• Example: left arm transformed to right arm• Solution: Use pre-labeled markers on the test subjects

• Viewed as white dots in the range image• Correspondences set up beforehand (as in Consistent Mesh

Parameterization)• 74 markers per subject (not all are used, however)

• Measure distance from template marker to target marker• K_I are the indices of the markers in template, m_I are target

markers

T. Funkhouser

Reverse Engineering

Algorithm Procedure• Minimize error function using L-BFGS-B algorithm

• Quasi-Newton method with limited memory usage

T. Funkhouser

Reverse Engineering

Motivation – Correct Hole Filling

T. Funkhouser

Reverse Engineering

More Information• http://www.cs.princeton.edu/~bjbrown/iccv05_course/• http://www.cs.princeton.edu/courses/archive/fall03/cs597

D/index.html

• http://www.cs.princeton.edu/courses/archive/fall03/cs597D/resources.html#Registration

• http://graphics.stanford.edu/papers/fasticp/• http://www.geometrie.tuwien.ac.at/ig/

(Surface Matching and Registration)