Speed and Robustness Speed and Robustness inin

3D Model Registration3D Model Registration

Szymon RusinkiewiczSzymon Rusinkiewicz

Princeton UniversityPrinceton University

No corpora callosa were harmed in the making of this talk.

ProblemProblem

• Align two partially-Align two partially-

overlapping meshesoverlapping meshes

given initial guessgiven initial guess

for relative transformfor relative transform

OutlineOutline

• ICP: Iterative Closest PointsICP: Iterative Closest Points

• Classification of ICP variantsClassification of ICP variants

• Better robustnessBetter robustness

• Faster alignmentFaster alignment

• ICP as function minimizationICP as function minimization

• Looking aheadLooking ahead

Aligning 3D DataAligning 3D Data

• If correct correspondences are If correct correspondences are known, can find correct relative known, can find correct relative rotation/translationrotation/translation

Aligning 3D DataAligning 3D Data

• How to find correspondences: User How to find correspondences: User input? Feature detection? input? Feature detection? Signatures?Signatures?

• Alternative: assume Alternative: assume closestclosest points points correspondcorrespond

Aligning 3D DataAligning 3D Data

• … … and iterate to find alignmentand iterate to find alignment– Iterative Closest Points (ICP) [Besl & Iterative Closest Points (ICP) [Besl &

McKay 92]McKay 92]

• Converges if starting position “close Converges if starting position “close enough“enough“

ICP VariantsICP Variants

• Variants on the following stages of Variants on the following stages of ICPICPhave been proposed:have been proposed:

1.1. SelectingSelecting source points (from one or both meshes) source points (from one or both meshes)2.2. MatchingMatching to points in the other mesh to points in the other mesh3.3. WeightingWeighting the correspondences the correspondences4.4. RejectingRejecting certain (outlier) point pairs certain (outlier) point pairs5.5. Assigning an Assigning an error metricerror metric to the current transform to the current transform6.6. MinimizingMinimizing the error metric w.r.t. transformation the error metric w.r.t. transformation

Performance of VariantsPerformance of Variants

• Can analyze various aspects of Can analyze various aspects of performance:performance:– SpeedSpeed– StabilityStability– Tolerance of noise and/or outliersTolerance of noise and/or outliers– Maximum initial misalignmentMaximum initial misalignment

• Comparisons of many variants inComparisons of many variants in[Rusinkiewicz & Levoy, 3DIM 2001][Rusinkiewicz & Levoy, 3DIM 2001]

OutlineOutline

• ICP: Iterative Closest PointsICP: Iterative Closest Points

• Classification of ICP variantsClassification of ICP variants

• Better robustnessBetter robustness

• Faster alignmentFaster alignment

• ICP as function minimizationICP as function minimization

• Looking aheadLooking ahead

ICP VariantsICP Variants

1.1. Selecting source points (from one or both meshesSelecting source points (from one or both meshes))2.2. MatchingMatching to points in the other mesh to points in the other mesh3.3. Weighting the correspondencesWeighting the correspondences4.4. Rejecting certain (outlier) point pairsRejecting certain (outlier) point pairs5.5. Assigning an error metric to the current transformAssigning an error metric to the current transform6.6. Minimizing the error metric w.r.t. transformationMinimizing the error metric w.r.t. transformation

Closest Compatible PointClosest Compatible Point

• Closest point often a bad approximation Closest point often a bad approximation to corresponding pointto corresponding point

• Can improve matching effectiveness by Can improve matching effectiveness by restricting match to restricting match to compatiblecompatible points points– Compatibility of colors [Godin et al. 94]Compatibility of colors [Godin et al. 94]– Compatibility of normals [Pulli 99]Compatibility of normals [Pulli 99]– Other possibilities: curvatures, higher-Other possibilities: curvatures, higher-

order derivatives, and other local featuresorder derivatives, and other local features

ICP VariantsICP Variants

1.1. SelectingSelecting source points (from one or both meshes) source points (from one or both meshes)2.2. Matching to points in the other meshMatching to points in the other mesh3.3. Weighting the correspondencesWeighting the correspondences4.4. Rejecting certain (outlier) point pairsRejecting certain (outlier) point pairs5.5. Assigning an error metric to the current transformAssigning an error metric to the current transform6.6. Minimizing the error metric w.r.t. transformationMinimizing the error metric w.r.t. transformation

Selecting Source PointsSelecting Source Points

• Use all pointsUse all points

• Uniform subsamplingUniform subsampling

• Random samplingRandom sampling

• Normal-space samplingNormal-space sampling– Ensure that samples have normals Ensure that samples have normals

distributeddistributedas uniformly as possibleas uniformly as possible

Normal-Space SamplingNormal-Space Sampling

Uniform SamplingUniform Sampling Normal-Space SamplingNormal-Space Sampling

ResultResult

• Conclusion: normal-space sampling Conclusion: normal-space sampling better for mostly-smooth areas with better for mostly-smooth areas with sparse featuressparse features

Random samplingRandom sampling Normal-space samplingNormal-space sampling

Selection vs. WeightingSelection vs. Weighting

• Could achieve same effect with Could achieve same effect with weightingweighting

• Hard to ensure enough samples in Hard to ensure enough samples in features except at high sampling ratesfeatures except at high sampling rates

• However, have to build special data However, have to build special data structurestructure

• Preprocessing / run-time cost tradeoffPreprocessing / run-time cost tradeoff

OutlineOutline

• ICP: Iterative Closest PointsICP: Iterative Closest Points

• Classification of ICP variantsClassification of ICP variants

• Better robustnessBetter robustness

• Faster alignmentFaster alignment

• ICP as function minimizationICP as function minimization

• Looking aheadLooking ahead

ICP VariantsICP Variants

1.1. Selecting source points (from one or both meshes)Selecting source points (from one or both meshes)2.2. Matching to points in the other meshMatching to points in the other mesh3.3. Weighting the correspondencesWeighting the correspondences4.4. Rejecting certain (outlier) point pairsRejecting certain (outlier) point pairs5.5. Assigning an Assigning an error metricerror metric to the current transform to the current transform6.6. Minimizing the error metric w.r.t. transformationMinimizing the error metric w.r.t. transformation

Point-to-Plane Error MetricPoint-to-Plane Error Metric

• Using point-to-plane distance instead Using point-to-plane distance instead of point-to-point lets flat regions slide of point-to-point lets flat regions slide along each other [Chen & Medioni along each other [Chen & Medioni 91]91]

ICP VariantsICP Variants

1.1. Selecting source points (from one or both meshes)Selecting source points (from one or both meshes)2.2. MatchingMatching to points in the other mesh to points in the other mesh3.3. Weighting the correspondencesWeighting the correspondences4.4. Rejecting certain (outlier) point pairsRejecting certain (outlier) point pairs5.5. Assigning an error metric to the current transformAssigning an error metric to the current transform6.6. Minimizing the error metric w.r.t. transformationMinimizing the error metric w.r.t. transformation

Projection to Find Projection to Find CorrespondencesCorrespondences

• Finding closest point is most expensive stage Finding closest point is most expensive stage ofofthe ICP algorithmthe ICP algorithm

• Idea: use a simpler algorithm to find Idea: use a simpler algorithm to find correspondencescorrespondences

• For range images, can simply project point For range images, can simply project point [Blais 95][Blais 95]

Projection-Based MatchingProjection-Based Matching

• Slightly worse performance per Slightly worse performance per iterationiteration

• Each iteration is one to two orders of Each iteration is one to two orders of magnitude faster than closest-pointmagnitude faster than closest-point

• Result: can alignResult: can aligntwo range imagestwo range imagesin a few milliseconds,in a few milliseconds,vs. a few secondsvs. a few seconds

ApplicationApplication

• Given:Given:– A scanner that returns range images in A scanner that returns range images in

real timereal time– Fast ICPFast ICP– Real-time merging and renderingReal-time merging and rendering

• Result: 3D model acquisitionResult: 3D model acquisition– Tight feedback loop with userTight feedback loop with user– Can see and fill holes while scanningCan see and fill holes while scanning

Scanner LayoutScanner Layout

PhotographPhotograph

Real-Time ResultReal-Time Result

Theoretical Analysis of ICP Theoretical Analysis of ICP VariantsVariants

• One way of studying performance is One way of studying performance is via empirical tests on various scenesvia empirical tests on various scenes

• How to analyze performance How to analyze performance analytically?analytically?

• For example, when does point-to-For example, when does point-to-plane help? Under what conditions plane help? Under what conditions does projection-based matching does projection-based matching work?work?

What Does ICP Do?What Does ICP Do?

• Two ways of thinking about ICP:Two ways of thinking about ICP:– Solving the correspondence problemSolving the correspondence problem– Minimizing point-to-surface squared Minimizing point-to-surface squared

distancedistance

• ICP is like Newton’s methodICP is like Newton’s method– Computing local “derivative” to obtain Computing local “derivative” to obtain

next stepnext step

What Does ICP Do?What Does ICP Do?

• Two ways of thinking about ICP:Two ways of thinking about ICP:– Solving the correspondence problemSolving the correspondence problem– Minimizing point-to-surface squared Minimizing point-to-surface squared

distancedistance

• ICP is like Newton’s methodICP is like Newton’s method– Computing local “derivative” to obtain Computing local “derivative” to obtain

next stepnext step

What Does ICP Do?What Does ICP Do?

• Two ways of thinking about ICP:Two ways of thinking about ICP:– Solving the correspondence problemSolving the correspondence problem– Minimizing point-to-surface squared Minimizing point-to-surface squared

distancedistance

• ICP is like Newton’s methodICP is like Newton’s method– Computing local “derivative” to obtain next Computing local “derivative” to obtain next

stepstep– ICP variants affect shape ofICP variants affect shape of

global error function global error function ororlocal approximationlocal approximation

Point-to-Surface DistancePoint-to-Surface Distance

Point-to-Point DistancePoint-to-Point Distance

Point-to-Plane DistancePoint-to-Plane Distance

Point-to-Multiple-Point Point-to-Multiple-Point DistanceDistance

Point-to-Multiple-Point Point-to-Multiple-Point DistanceDistance

Soft Matching and Distance Soft Matching and Distance FunctionsFunctions

• Soft matching equivalent to standard ICP Soft matching equivalent to standard ICP on (some) filtered surfaceon (some) filtered surface

• Produces filtered version of distance Produces filtered version of distance functionfunction fewer local minima fewer local minima

• Multiresolution minimization [Turk & Multiresolution minimization [Turk & Levoy 94] or softassign with simulated Levoy 94] or softassign with simulated annealing [Gold et al. 98]annealing [Gold et al. 98]

SummarySummary

• Classified and compared variants of Classified and compared variants of ICPICP

• Normal-space sampling for smooth Normal-space sampling for smooth meshes with sparse featuresmeshes with sparse features

• Particular combination of variants Particular combination of variants can align two range images in a few can align two range images in a few tens of ms.tens of ms.– Real-time range scanningReal-time range scanning– Model-based trackingModel-based tracking

Future WorkFuture Work

• Continuum between ICP with fancy Continuum between ICP with fancy selectionselection(or weighting) and feature matching(or weighting) and feature matching

• Better approximations to distance Better approximations to distance functions:functions:– Switch between point-to-point andSwitch between point-to-point and

point-to-plane depending on curvaturepoint-to-plane depending on curvature– Elliptical approximationsElliptical approximations– Higher-order approximationsHigher-order approximations

For More InformationFor More Information

http://www.cs.princeton.edu/~smr/