a low-dimensional representation for robust partial...
TRANSCRIPT
A Low-Dimensional Representation for Robust PartialIsometric Correspondences Computation
Alan Brunton∗ Michael Wand∗ Stefanie Wuhrer† Hans-Peter Seidel∗ Tino Weinkauf∗
Abstract
Intrinsic shape matching has become the standard ap-proach for pose invariant correspondence estimationamong deformable shapes. Most existing approaches as-sume global consistency. While global isometric match-ing is well understood, only a few heuristic solutions areknown for partial matching. Partial matching is particu-larly important for robustness to topological noise, whichis a common problem in real-world scanner data. Weintroduce a new approach to partial isometric matchingbased on the observation that isometries are fully deter-mined by local information: a map of a single point and itstangent space fixes an isometry. We develop a new repre-sentation for partial isometric maps based on equivalenceclasses of correspondences between pairs of points andtheir tangent-spaces. We apply our approach to registerpartial point clouds and compare it to the state-of-the-artmethods, where we obtain significant improvements overglobal methods for real-world data and stronger guaran-tees than previous partial matching algorithms.
1 Introduction
Modern computer graphics has experienced a paradigmshift: Traditional manual modeling is increasingly com-plemented by data-driven techniques where measureddata, such as 3D scans, are used as a basis for building3D models. An important data source are 3D scans ofdeformable models, such as humans or animals in vary-ing poses. Today, there exist sophisticated scanning se-tups for acquiring moving geometry in real-time [1–4]and there are even consumer devices on the market suchas the Microsoft KinectTM. This leads to new applica-tions such as virtual cinematography [5], or the creationof data-driven generative shape models of deformable ob-jects [6–9]. Finding correspondences among such data is afundamental problem for all of these applications: Almostany further processing, such as the registration of partialscans into a complete shape, editing of sequences, or sta-tistical analysis, requires dense correspondences betweensurface points.
Matching deformable shapes is in many cases a difficultproblem: If we permit rather general deformations this
∗Max Planck Institute for Informatics, Germany,{abrunton,mwand,weinkauf,hpseidel}@mpi-inf.mpg.de†Cluster of Excellence, Multi-Modal Computing and Interaction,
Saarland University, Germany, [email protected]
might require many parameters that have to be exploredduring matching. The size of this search space is expo-nential with respect to the available degrees of freedom.However, for the important special case of a single objectin different poses, we can often assume that the deforma-tion is approximately isometric, i.e., preserving the intrin-sic metric structure. Concretely, the distances along sur-faces of objects such as humans, animals, plants, or clothsdo not change a lot without serious injury or damage. Thisrestriction leads to a strongly constrained search space.Lipman et al. [10] argue that isometries between topo-logical disks are a special case of conformal mappings,thereby limiting the degrees of freedom to six (three point-to-point correspondences are sufficient). Ovsjanikov etal. [11] show that a single point correspondence is suf-ficient for a special class of shapes where the spectrumof the Laplace-Beltrami operator is not degenerate, thusshowing that there are only two degrees of freedom in thisspecial case. Additional cues, such as carefully selectedconstellations of local features [12], can even reduce thecomplexity for many shapes to the point of leaving a triv-ial search space, eliminating all degrees of freedom. Asapproximate isometry makes the correspondence problemfeasible while still permitting significant pose changes,many of the recent shape matching algorithms are basedon this assumption [10–19].
However, the isometric matching problem is not yetsolved: Because of the intrinsic view of the geometry, it isnaturally sensitive to topological noise. In case of holes,missing data, or contacts, intrinsic distances become dis-torted and thus no longer constitute invariants that can beexploited for matching. One solution is to replace thenotion of distance. For example, by using diffusion dis-tances, or variants thereof, one can reduce the sensitiv-ity to topological artifacts [17, 20, 21] when the pieces ofgeometry that cause the problem are small in relation tothe overall shape. Nonetheless, these invariants still breakdown in case of large artifacts (wide contacts, large holes,as shown for example in Section 5 of this paper). Un-fortunately, real-world 3D scanner data, one of the mainpractical application areas, is almost universally troubledby substantial artifacts of this kind.
Formulated more generally, we have to address the prob-lem of partial matching, where not the whole manifoldcan be brought into correspondence by a (near-) isomet-ric mapping but only an excerpt of the surface can bematched. In this case, typical invariants (geodesic paths,Laplacian eigenfunctions) become unreliable because theyutilize global information. Importantly, the excerpts of the
1
surfaces that can be matched are not known a priori (oth-erwise, we could just restrict a traditional method accord-ingly) but need to be determined along with the matching.This seems to re-introduce prohibitive complexity as wenow have to choose from an exponential number of suchsubsets [22]. The most important contribution of this pa-per is to show that with an appropriate matching modelthis is not the case. The search space is not much largerthan in the global problem and we give an efficient algo-rithm for computing such matches.
The core of our method is based on the observation fromdifferential geometry that an isometric map can be fullyspecified by using local information up to first order only:An isometry between Riemannian manifolds is fixed by asingle point correspondence and an orthogonal map of thetangent spaces (see for example [23, p. 201]). In the caseof surfaces, this means that the map of a point and a localdirection (plus orientation, in case of unoriented surfaces)is sufficient to determine an isometry. We will sketch aconstructive proof in Section 3 that directly yields a prop-agation algorithm for computing matches: Starting with asingle oriented point match, correspondence informationis incrementally propagated to the neighborhood, therebyflood-filling a partially consistent region of isometric ge-ometry. Being local in nature, the method handles partialmatches naturally and is robust to topological noise, whichis reported naturally as boundary of partiality.
From a structural point of view, we can understand partialisometries among smooth, connected manifolds as equiv-alence classes of mappings of a pair of points and theirtangent spaces on the two surfaces involved. In the caseof oriented 2-manifolds, each such object has six degreesof freedom (a point and an angle around the surface nor-mal for each for source and target surface, respectively).The mapping is captured by a equivalence class of such 6-tuples. This set is redundant in that one degree of freedom(the sum of the angles) can be chosen arbitrarily and twofurther parameters (either the starting or the end point) areonly needed to select the partial region to be mapped. Thisleaves three degrees of freedom that need to be actuallyexplored densely, with false starting points being rejectedin O(1) time.
In this view, we can perform a relaxation: In order toalso find approximate isometries, we can cast this problemas the task of finding approximate equivalence classes.Kim et al. [18] have used this idea in the context of glob-ally consistent isometric mappings in order to efficientlyfind approximate isometries. In our paper, we demon-strate how our partial matching framework can be adaptedto perform the same task in a partial matching scenario.We perform agglomerative clustering [24] in the space ofnear-isometric mappings, which are concisely representedusing the tuples introduce above.
We validate our algorithm on standard benchmark datasets and raw scanner data, and compare the results to pre-vious work. We show a significant improvement in qual-ity over global methods in shapes with topological noise.Our algorithm yields similar or better results as previousheuristics for partial matching, but with stronger guaran-
tees of discovering existing isometries as outlined above.
In summary, our paper proposes a systematic frameworkand new algorithms for extending isometric matching tothe case of partial consistency, thereby making the follow-ing specific contributions:
• We characterize partial isometries of shapes bysingle-point maps up to first order, which yields atight bound on the inherent degrees of freedom.
• This leads to a novel matching algorithm that pro-vides a systematic approach to the general setting ofpartial intrinsic matching, where both surfaces maybe incomplete, including robustness to strong topo-logical noise.
• By interpreting partial matching as a problem of find-ing approximate equivalent classes in our novel rep-resentation, we obtain an algorithm for approximatepartial isometric matching.
Our algorithmic pipeline for approximate partial isometricmatching is summarized as follows. We identify distinc-tive feature points on both surfaces. We compute orientedpoint correspondences by matching feature descriptors,with orientation determined by nearby features. From ori-ented point matches, we perform local metric propagation,stopping the propagation when the stretching becomes toolarge. This gives us a set of partial isometries, coveringdifferent, but possibly overlapping regions of both sur-faces. We define a dissimilarity measure between partialisometries, and use this to cluster the partial matches intoequivalence classes. The cluster (equivalence class) withthe smallest total intra-cluster distance is merged by tak-ing the geodesic centroid of all candidate correspondencesfor each point on the source manifold.
The steps in this pipeline exhibit sensitivity to certainchallenges. Most prominent are the sensitivity of featurematching to surface noise, missing data and topologicalnoise, and the sensitivity of the clustering step to under-sampling of the space of isometric mappings, which canbe exacerbated by problems in the feature matching stage,in addition to the lack of a proper distance metric betweenpartial matches. The difficulty of reliable feature match-ing on real data can be alleviated by the metric propaga-tion algorithm, which will typically produce smaller par-tial isometries for incorrect feature matches than for cor-rect ones since we expect the local metric to be vary sig-nificantly for different parts of the surface. However, inthe presence of strong continuous intrinsic symmetries thisassumption breaks down. We thoroughly discuss and ex-plore in which situations and to what extent these chal-lenges create problems in the final result in Section 5.
2 Complexity of Isometric ShapeMatching
In this section, we discuss different isometric matchingmodels and their implications on the difficulty for finding
2
a globally optimal solution to the matching problem. Tothe best of our knowledge, this has not yet been analyzedexplicitly in literature. We do not aim to give an extensivereview of surface registration methods; for this, we referthe reader to recent surveys such as van Kaick et al. [25]or Tam et al. [26].
We start by introducing some formal notations.
Manifolds: We consider smooth, orientable 2-manifoldsM ⊂ R3 embedded in three-dimensional space. In orderto represent partial data (such as scans with acquisitionholes), we permit boundaries, denoted by ∂M. The orien-tation ofM might (optionally) be prescribed by orientedsurface normals n(x), x ∈M, pointing outwards.
Tangent space: We further use TxM to denote the tan-gent space ofM at point x ∈ M. For its representation,we choose two arbitrary but fixed orthogonal tangent vec-tors u(x),v(x), i.e.: TxM = span{u(x),v(x)}.
Distances: We use distM(x,y) for x,y ∈ M to denotethe intrinsic or geodesic distance between the two pointsx and y. A geodesic connecting x and y is a curve that hasno geodesic curvature, which means that the derivative ofthe curve at a point s projected to TsM is zero. We call theshortest geodesic connecting x and y a shortest geodesicpath inM.
Mappings and isometries: Consider two manifolds Sand T , and a mapping f : U → T from an open sub-set U ⊆ S to T . Let fu(x) and fv(x) denote the partialderivatives of f with respect to the tangent space direc-tions u and v in R3 of S. The first fundamental formIf (x) of f at point x ∈ U is then given by:
If (x) =
(fu(x) · fu(x) fu(x) · fv(x)fv(x) · fu(x) fv(x) · fv(x)
).
The function f is an isometry if and only if If (x) = I forall x ∈ U .
Equipped with this notation, we will now identify and an-alyze three classes of approaches for finding surface cor-respondences. The difference is in how the notion of ap-proximate isometry is handled, leading to different com-plexity characteristics and algorithms.
2.1 Global Approximate Isometry
The global model assumes that geodesic distances areglobal invariants of the shape, being consistent at leastup to an error margin ν > 0 that accounts only for a smallfraction of the object size. This means, the energy
Eglobal = supx,y∈U
|distS(x,y)− distT (f(x), f(y))| (1)
must be smaller than ν. For two points x,y ∈ U Eq. 1corresponds to an additive error of at most ν, i.e.
|distS(x,y)− distT (f(x), f(y))| ≤ ν. (2)
The global consistency criterion is sometimes modified toallow for a multiplicative error of ν instead, as
max
(distS(x,y)
distT (f(x), f(y)),distT (f(x), f(y))
distS(x,y)
)≤ (1 + ν).
(3)
The former error model considers absolute errors, whilethe latter one considers relative errors.
In case of exact isometry, i.e., ν = 0, the set of match-ing candidates becomes strongly constrained. The isom-etry assumption has been used to embed the intrinsicgeometry of a shape in a Euclidean space using multi-dimensional scaling, such that embeddings of isometricshapes become identical, which facilitates shape recogni-tion and matching [27–29]. Alternatively, the geometryof shape S can be embedded into shape T using gen-eralized multi-dimensional scaling, thereby computing across-parameterization directly [14].
Exact isometry results in a set of matching candidates withfew degrees of freedom. Lipman and Funkhouser [10]have noticed that isometries are special cases of conformalmaps, thereby having only the degrees of freedom givenby the Mobius group, which are fixed by three pointwisematches on spherical topologies. Ovsjanikov et al. [11]have shown that even a single point match is sufficient tofix an isometry if the Laplace-Beltrami spectrum of S isnon-degenerate. In this paper, we exploit a different wayto uniquely describe an isometry: fixing one point, a tan-gential direction, and the surface orientation is necessaryand sufficient to specify an isometric mapping [23]. Thisprovides the fewest possible degrees of freedom while stillcovering all cases including shapes with global intrinsicsymmetries.
In case of small, global error margins, statistical triangula-tion algorithms can be applied that compute all correspon-dences from a few landmark matches [15, 16] or, simi-larly, by voting for several approximate solutions [10,11].Depending on the geometry of the shape, errors can be-come amplified so that a bit more than only the minimalset of initial correspondences are required [12]. Nonethe-less, matching according to this model is efficient and canbe considered a more or less solved problem.
The global approximate isometry criterion has also beenemployed to study matching partial overlaps of completesurfaces. Van Kaick et al. [30] use a pair of features to de-fine a map that captures a local geodesic region betweenthe two features, and show how this descriptor can be usedfor partial matching. Here, using two features instead ofone is crucial because two features encode orientation in-formation on the surface, while one feature does not.
The drawback of the global consistency model is its sen-sitivity to topological noise. To make globally consistentapproaches more robust w.r.t. topological changes, severalapproaches have been proposed to perform a band-limitedanalysis in the eigenspace of the manifold’s Laplace-Beltrami operator [17,20,21,31]. Unlike prior approachesbased on embedding the intrinsic geometry of the shapedirectly [14, 27–29], these approaches successfully han-dle small topological errors. However, they break down
3
in the presence of large artifacts, such as wide contactsor missing pieces. A notable exception is the heuris-tic region growing approach by Sharma et al. [32]. Ittries to match points with similar spectral signatures us-ing an expectation-maximization framework, which hasbeen shown to perform well in the presence of large con-tacts. Despite good practical performance, the algorithmis heuristic in nature and it remains unclear under whichconditions it will find a correct solution. In particular, iflocal descriptors are not unique, the greedy region grow-ing might fail and the EM-algorithm does not guaranteeto recover a correct global match. In contrast, our regiongrowing algorithm, which is based on propagation of met-ric information rather than potentially ambiguous descrip-tor matching, comes with the theoretical guarantee to findcorrect results for exact isometries while requiring an ini-tialization from a small search space with very few degreesof freedom.
2.2 Global Approximate Isometry in Par-tial Regions
The global consistency model is incompatible with the no-tion of partial matching, since distances have to be mea-sured on the complete surfaces S and T , which might notbe available. Xu et al. [22] modify the criterion in Eq. 1by restricting paths to the partially matched region U :
Epartial = supx,y∈U
|distU (x,y)− distf(U)(f(x), f(y))|.
(4)
Note that Epartial considers again absolute errors
|distU (x,y)− distf(U)(f(x), f(y))| ≤ ν (5)
and can be modified to consider relative errors as in Eq. 3.
The drawback is that the shortest geodesic paths and thusthe energy depend on the shape of the domain, whichmakes it difficult to optimize Epartial; changing the do-main U influences which pairs of points are mapped ina geodesically consistent way. For this reason, Xu et al.restrict their method to considering geodesically convexregions U , which are defined as regions where for any twopoints x,y in U , there exists a shortest geodesic path be-tween x and y in U . In this special case, both Epartial andEglobal measure shortest paths on S and T . The solutionproposed by Xu et al. optimizes the scale of consistent re-gions and the consistent points separately, which leads toa rather complex algorithm. Further, the notion of a scaleparameter is not canonically related to the original match-ing problem.
Sahilloglu and Yemez [33] consider the case where oneof the surfaces is complete and the other an incomplete,deformed part of that surface. Using a coarse samplingand matching strategy between shape extremities, they candirectly estimate a scale parameter between the two sur-faces, which allows them to define a scale-invariant iso-metric distortion measure. This results in one-sided partialdense intrinsic matching up to an arbitrary scale. Their ap-proach also allows matching of semantically similar, but
non-isometric complete surfaces. Our approach allowsboth surfaces to be incomplete, at the cost that they mustbe scaled consistently beforehand. Given real-world scan-ners often provide data in known units, our method is com-patible with the scenario of matching surfaces acquiredwith different modalities. While in this paper we do notexplore the latter scenario, our approach would be com-patible with this task given a reliable way to estimate scalechange during metric propagation, possibly using shapeextremities or other intrinsic features.
Bronstein et al. [34] introduced a general framework toevaluate partial similarity using Pareto optimality. In caseof partial intrinsic shape matching, this method aims tofind large parts of two surfaces that are similar to eachother, where similarity is measured according to Eq. 2.In practice, the parts are found using generalized multi-dimensional scaling. Raviv et al. [35] use a similar tech-nique to find partial intrinsic symmetries.
2.3 Local Approximate Isometry
Another common way to relax the requirement of ex-act isometry towards approximate matching is to main-tain the metric tensor in a least-squares-sense. Again, letf : U → T ,U ⊆ S be a mapping between two manifolds,where U is the open subset of S that should be mappedto T in a distortion minimizing way. We can measure thedistortions for example by minimizing a matrix norm ofthe Green deformation tensor (difference of the first fun-damental form to identity):
Elocal(f) =
∫U‖If (x)− I‖2F dx. (6)
This criterion is purely local and thus well suited for par-tial matching. It is worth noting that extrinsic elasticdeformation techniques, such as [36–38] are closely re-lated: They either include the preservation of the curvaturetensor in the objective function to maintain the extrinsicshape, or apply Eq. 6 to the 3-manifold of the embeddingEuclidean volume [39, 40]. All of these methods are de-signed for partial matching.
The problem with both intrinsic and extrinsic elasticmatching models is that the search space becomes verylarge, rendering any approach based on exhaustive searchprohibitively expensive. The structure of the search spacecan be approximately understood by a linearized analy-sis. In order to understand the degrees of freedom of thelocal matching model, we can use the tool of modal anal-ysis of such elastic models, first introduced by Pentlandand Williams [41] to the field of computer graphics (theaim of their paper was actually to speed up the simulationof extrinsic elastic deformations of solid objects in three-space). Modal analysis represent the deformation of anobject as a linear combination of the eigenmodes of theobject’s vibration, which are found using a spectral analy-sis of a linearized deformation energy.
Typically, these energies are related to the Laplacian ofthe deformed domain, thus leading to eigenvalues that are
4
only decreasing rather slowly. Many modes need to be re-tained to represent the space of low-energy (i.e., plausible)deformations adequately. If the local matching is not verystiff, the size of the search space explodes. This is intu-itive: Permitting local deformations creates a large varietyof permissible shape variants, for example by adding dif-ferent local dents and combinations of those everywhere.Due to this large search space, it is impractical to matchshapes purely based on the deformation model of Eq. 6.In order to reduce the large search space for elastic match-ing, existing methods use additional constraints, such as,template models [40], temporal coherence [39], or fairlylarge sets of coherent feature correspondences [37].
Recently, Kim et al. [18] proposed a new paradigm forthe elastic matching problem. First, the method computesmultiple dense maps between two shapes by assuming aglobal near-isometric deformation model. Multiple mapsare obtained by fixing different triples of correspondingpoints for the computation of global conformal maps [10].Second, the method computes local weights for pairs ofmaps, depending on their local adherence to isometry, andperforms a spectral analysis to combine multiple weightedmaps into a single global map. The key idea is to findcliques of similar mappings by clustering approximatelycompatible maps. The method was shown to perform wellin many interesting cases. However, it cannot handle par-tial mapping; in particular constellations with large topo-logical noise cannot be handled: Each global conformalmap can is highly distorted due to the lack of global con-sistency. This introduces distortions in many of the lo-cal matches, and it is not always possible to remove thesedistortions in the final blended result (as demonstrated inSection 5).
Our clustering method (Section 4.4) is based on the sameidea, but it combines partial maps instead of global maps,therefore avoiding the mentioned problems of artifacts dueto only partial consistency. On the technical side, the mainchallenge is that we cannot measure the distance betweenall pairs of candidate maps but only between actually over-lapping ones. This is a problem for the original spectralclustering, which we substitute by an alternative techniquegeared towards sparse pairwise constraints [24].
2.4 Previous Work on Local Isometry
In previous work, there have been a number of attemptsto find local isometric mappings, similar to our approach.However, these did not consider mappings between gen-eral surfaces but only local planar parametrizations.
Different techniques have been proposed to locally param-eterize the intrinsic geometry of a surface to a plane usinga local approximate isometry criterion. Schmidt et al. [42]used exponential maps to transport a local coordinate sys-tem along a surface for the purpose of texture mapping.More recently, Schmidt [43] used transported exponen-tial maps to produce a parameterization of a local surfacepatch to the plane that has low metric distortion. The lo-cal surface patch is provided through user interaction as
an input stroke on the surface. Malvaer et al. [44] pro-pose an alternative parameterization based on an extensionof polar coordinates to surfaces. In our work, we cross-parameterize local surface patches from S to T . Thiscross-parameterization is more challenging than a param-eterization to the plane due to the arbitrary geometry ofT . Our cross-parameterization task is further complicatedas in our application scenario, both S and T are scannedpoint clouds with missing surface information and scannernoise. Hence, the reviewed methods cannot be applied ina straight forward manner in our application.
3 Local Metric Matching
We saw that for general surface matching using aglobal approximate isometry criterion for partial matching(Eq. 4) is difficult since the domain changes, and that usinga local approximate isometry criterion (Eq. 6) is difficultsince this results in a large search space. In this section,we outline our new method: starting from a point s ∈ Sand attached direction in TsS with known correspondencef(s) ∈ T and corresponding attached direction in Tf(s)T ,we find the largest domain U such that Eq. 5 is satisfied.
The key assumption of our approach is that S and T canbe matched using a global approximate isometry in somepartial region U containing s, implying that Eq. 5 holds.Our goal is to find the largest region U for which this as-sumption is satisfied. This assumption holds in scenar-ios where we know that S and T are actually related bya near-isometric map but the data does not comprise allof the original input and/or contains additional unrelatedgeometry or contacts. The most important practical ex-ample where this assumption holds is the acquisition of asurface that deforms near-isometrically but the scanningequipment introduces areas of missing data (shadowed tothe scanner by other object parts) and cannot correctly re-solve the topology in contact areas (such as the hand of aperson being in contact with the body).
Our approach can be viewed as a hybrid approach be-tween global and local approximate isometric matching.We use the assumption that S and T can be matched us-ing a global approximate isometry in some partial regionU , and we compute U by growing a region using a localapproximate isometry criterion. This allows us to com-bine the advantage of the global methods of having a low-dimensional search space with the advantage of the localmethods of being well suited to describe partial isometricmatches.
Let Θ denote the parameter domain of all partial isometricmatchings between S and T . Our goal is to compute allpartial isometries {fθ,Uθ}θ∈Θ that map maximal subsetsU of S to T . The vector θ ∈ Θ parametrizes the set of allsuch mappings.
In the following, we will discuss that:
• Isometric deformations of S have three degrees offreedom.
5
• A partial isometry can be parametrized by Θ = S ×SO(1)× T × SO(1), where SO(1) denotes the unitcircle.
• There exists a global representation redundancy thatidentifies all choices of parameters s ∈ U ⊂ S andds ∈ TsS.
3.1 Three Degrees of Freedom
For a (sufficiently) smooth Riemannian manifold S fixingone point s on S, a tangential direction ds in TsS, anda surface normal ns at s suffices to specify an isometricmapping [23]. For details on the smoothness criteria, re-fer to [45]. The following proof sketch aims to give theintuition behind this statement for 2-dimensional surfacesembedded in R3 by showing that we can transfer the localcoordinate frame defined by ds and ns to any point on Sin a canonical way.
We start with two definitions. The injectivity radius ρsS ats is the largest radius, such that for any point x on S withgeodesic distance less than ρsS from s, there is a uniqueshortest geodesic path between s and x. The injectivityradius of S is defined as ρS = infs∈S (ρsS).
For closed surfaces, the injectivity radius can be boundedfrom below by the minimum of π/
√supK, where K
is the Gaussian curvature, and half of the length of thesmallest periodic geodesic [23, Thm. 10]. It follows thatρS > 0 holds for closed Riemannian surfaces with fi-nite Gaussian curvature. For general surfaces with bound-ary, the injectivity radius may become zero. Note that inthis paper, we consider Riemannian surfaces with finiteGaussian curvature with boundaries. The boundaries arepresent because the closed Riemannian surface of interestis only partially observed by the acquisition device. In thisspecial case, the injectivity radius is still positive.
Imagine that S is covered by overlapping regions of intrin-sic radius less than ρS. These regions are all topologicallyequivalent to disks. Consider a disk D containing a points as shown in Figure 1. In a small neighborhood of s,D is arbitrarily close to the tangent plane TsS. Note thats, ds, and ns fix a local orthonormal coordinate frame inR3. This coordinate frame can be transported to a pointx : x ∈ D,x 6= s along the (unique) shortest geodesicpath Psx from s to x in D by parallel transport, which canbe thought of as repeatedly projecting the direction ds tothe tangent planes of consecutive points along Psx in in-finitesimally small steps (for details on parallel transportsee for example Berger [23, Chapter 3.1]). Let dx denotethe transported direction. The transferred direction lies inthe tangent plane TxS, and can again be used to fix a localorthonormal coordinate frame at x. This transfer can berepeated by chaining together disks until every point y onS was assigned a fixed tangential direction dy ∈ TyS byparallel transport along a path connecting s and y that con-sists of an arbitrary but fixed sequence of (unique) shortestgeodesic paths within the chained disks. Note that by con-struction, we only use intrinsic information to propagatethe direction ds to the entire surface. Hence, the resulting
D
s xPsx
ds dx
Figure 1: Propagating a local coordinate system along S.
local coordinate frames are invariant with respect to iso-metric deformations of S when encoded relative to fixedlocal coordinate systems u(y),v(y),n(y) on S.
This implies that any isometric deformation of an orientedsurface S can be specified using three degrees of freedom:one point s on S (accounting for two degrees of freedom)and a direction in the tangent plane of s.
3.2 Representation
We can use the fact that any isometric deformation of anoriented surface can be specified using three degrees offreedom to derive a representation θ for intrinsic map-pings. Specifically, identifying one corresponding pointand one corresponding tangential direction completely de-termines an isometric mapping between two oriented sur-faces. More formally, to define an isometric mapping be-tween (subsets of) S and T , it suffices to specify a points ∈ S, its intrinsically corresponding point t ∈ T , atangential direction ds in TsS, and its intrinsically cor-responding tangential direction dt in TtT .
Starting from this information, and assuming that S andT are isometric, we can propagate the correspondence in-formation by mapping the metric structure of S onto Tas follows. Starting from the corresponding points s andt along with the corresponding directions ds and dt, wecan propagate the correspondence information to a suffi-ciently small geodesic neighborhood Ns of s by simulta-neously walking along corresponding geodesic paths start-ing at s and t, respectively, and by matching points that arereached at the same time. Here, Ns is sufficiently small ifits geodesic radius is below ρS. Once the correspondenceinformation is computed for Ns, we continue propagatingthe correspondence information from a point close to theboundary of Ns to its geodesic neighborhood, and iterateuntil every point on S has a correspondence.
The assumption that S and T are (near-) isometric can alsobe used to detect the boundary of the largest region U ⊆ Scontaining s for which the mapping is near-isometric. Thereason is that the propagation algorithm allows us to mea-sure the difference in intrinsic geometry in newly mappedparts of S and T directly. Hence, we can stop the re-gion growing algorithm if a newly added correspondencewould induce a stretching larger than ν.
Figure 2 illustrates the near-isometric region growing pro-cess. The plane on the left (S) and the plane with a hill on
6
(a) Oriented point match. (b) Some growing.
(c) More growing. (d) Final mapping.
Figure 2: Illustration of the near-isometric region growingprocess. Corresponding points share the same color.
the right (T ) are isometric except for the hill. Beginningwith a point and direction match in Figure 2a, the isomet-ric region grows outward in all directions where the isom-etry condition is locally satisfied. This way, the largestnear-isometric partial match is identified, as shown in Fig-ure 2d. Note that this region can have a complex topologyand geometry.
3.3 Representation Redundancy
The representation discussed above contains redundant in-formation: Infinitely many θ may represent the same near-isometric mapping.
The first degree of redundancy is the choice of the direc-tion ds in the tangent plane TsS. Changing this directionmerely rotates the field of directions dx in TxS (and sim-ilarly, the field of corresponding directions dy in TyT ).Hence, the choice of this direction has no influence on thefinal result. To remove this redundancy, we start from anarbitrary but fixed direction ds and precompute and fix dx
for all x on S.
The second redundancy is the choice of the start point s.Let U ⊆ S be the maximal set within which an isometryf : U → T can be constructed. We can replace s in θ byany other point s′ ∈ U . See Fig. 3. This requires an up-date of the remaining parameters. The direction ds′ is setto the precomputed value (see above), and the remainingparameters can be updated using the computed mappingf . More specifically, t′ = f(s′) and the direction of dt′
is set to the direction of f(s′ + εds′) − f(s′), where εis set small enough that s′ + εds′ is in U . Thus, in thecase of a global isometry, or when we know beforehandthe isometric region U , the mapping f has three degreesof freedom. In the general partial isometry case, however,where U is unknown, the starting point s ∈ S is not fullyredundant; it still selects the equivalence class that rep-resents a certain partial map; computationally, this is thepatch to be matched by the propagation algorithm. Thisdoes not increase the complexity strongly as we just haveto restart the matching in case multiple partial matches ex-ist. Ideally, we would sample one starting point per par-tial isometric region, which in practice will be far fewerthan there are samples on S. While we do not determinethis lower-bound beforehand, we maintain low complexity
s tU ⊆ S f(U) ⊆ T
s′ t′s′′ t′′
s′′′ t′′′
Figure 3: For a given set U and a corresponding isome-try (shown in blue), the choice of the starting point s isarbitrary.
by using features to identify potential starting points, andmarking a starting point s as redundant if another startingpoint s′ produces an isometric region U containing s. Incase of a mismatch, it can be discovered quickly if the tar-get area does not match, which will become evident withinconstant time.
In summary, all mappings represented by s′,ds′ , t′,dt′
form an equivalence class in the parameter space Θ. Sincewe can remove one degree of redundancy by fixing the tan-gential directions on S, for each mapping f we have twodegrees of freedom that vary among equivalent maps (thechoice of the start point on S), which along with the man-ifold structure of U implies that each equivalence classforms a 2-manifold in Θ, which can be computed directlyusing the propagation algorithm introduced in Section 3.2.
In practice, we can take advantage of this representationalredundancy as follows. Since S and T are discretized andcorrupted by noise, the error of the correspondence in-formation computed using the propagation algorithm in-creases with increasing distance from the start point ofthe propagation. Hence, it is possible that the propaga-tion stops prematurely due to discretization artifacts andthe influence of noise, thereby identifying a region U thatis smaller than the correct solution. Thanks to the redun-dancy in the representation, we can start the propagationalgorithm from multiple oriented point pairs, identify a setof equivalent mapping functions fi, and them into a singlemapping function f that covers a larger area of S and isless influenced by noise than the individual fi. The follow-ing section discusses a direct implementation of this theo-retically motivated method, which we will use to computecorrespondences of noisy scanner data.
4 Pairwise Intrinsic Matching
From the preceding analysis, we derive an algorithm forcomputing partial near-isometries between two surfaces Sand T . We start our discussion by outlining how surfacesare represented and how basic steps of the algorithm areimplemented (Section 4.1). Our direct, non-optimized im-plementation is based on enumerating the non-equivalentchoices by selecting different oriented point matches (Sec-tion 4.2), growing the isometric region locally from thereuntil no more points locally satisfy ν (Section 4.3), andfinally clustering the partial maps into equivalence classes(Section 4.4). Figure 4 gives a graphical overview of ourmatching pipeline.
7
Features Oriented PointMatches
PartialMatches
Clustered andMerged Result
Figure 4: Overview of the pairwise matching pipeline. First, features are detected, second, oriented point matches arecomputed, third, starting from the oriented point matches, partial isometric matches are found, and finally, different partialmatches are clustered and merged.
4.1 Surface Representation
In our implementation, the surfaces S and T are eitherrepresented as oriented point clouds or meshes. Whilemost parts of the algorithm can be extended to discretizedsurfaces in a straight forward manner, for some parts ofthe algorithm we require a continuous surface represen-tation. We obtain a continuous surface representation asimplicit moving-least-squares (MLS) manifolds using therobust method of Oztireli et al. [46]. Using this method,a discretized surface S is represented continuously as thezero level-set of an implicit function derived from the ori-ented vertices of S .
Using a MLS representation allows a point to be projectedto a continuous representation of S in the case where Sis given as oriented point cloud or mesh. In the follow-ing, whenever we refer to projecting a point x onto a dis-cretized surface S, this projection is implemented as pro-jecting x to the MLS representation derived from S.
One basic operation needed by our algorithm is the com-putation of geodesic distances and paths on S. In someparts of the algorithms, rough estimates of geodesic dis-tances and paths suffice, and these are computed using Di-jkstra’s algorithm. In other parts of the algorithm, it is cru-cial to have accurate estimates of geodesic distances andpaths. In these cases, we initialize a geodesic path P tothe Dijkstra path and refine P by minimizing the length ofP using the constraint that P must not leave S. This opti-mization is carried out iteratively using a conjugate gradi-ent method. After each step, P is projected back to S. Inthe following, we refer to these refined geodesic distancesand paths as smoothed geodesic distances and paths.
A core part of our approach is to compare distances mea-sure on the target surface to distances measured on thesource for corresponding points. Comparing geodesics be-tween all pairs of correspondences quickly becomes pro-hibitively expensive and would limit the practical appli-cability of our algorithm. To remedy this, we construct atopology hierarchy on S similar to [47] as follows. We de-fine level 0 of the hierarchy to be the original set of verticesand their connectivity–either the original triangle mesh orthe k-nn graph for a point cloud. (In all our experimentsinvolving point clouds, we set k = 8.) The sample spac-ing ε0 is defined as the average edge length. Level 1 ofthe hierarchy is constructed by selecting an evenly spacedsubset of the level 0 vertices and connecting vertices ina topology preserving way. Subsequent levels j + 1 areconstructed in the same way as a subset of level j. Ateach level the subset for the next level is determined bydoubling the desired sample spacing, εj+1 = 2εj . At acoarser level of this hierarchy fewer vertices are connectedto any others, but at a greater distance to each other. In ouralgorithm, we only consider geodesic distances betweenvertices that share an edge in at least one level of the hi-erarchy, which results in a sparse set of distances to beoptimized, even for a dense set of correspondences.
This hierarchical structure ensures that locality is re-spected, as only geodesics between points that are neigh-bors in some level of the hierarchy are considered. Hav-ing a structure that respects locality is crucial to allow forpartial matching. The number of levels in the hierarchydetermines the trade-off between local and global isome-try constraints. We use 5 levels in all experiments in thispaper.
8
Note that during the execution of the matching algorithm,the vertices on S are fixed, as we aim to find a correspon-dence for every vertex of S on T (if such a correspondenceexists). Hence, we can precompute all geodesic distanceson S for vertices connected in some level of the hierarchy.This way, only distances on T need to be updated duringthe metric optimization.
4.2 Finding Oriented Point Matches
In principle, by exhaustively trying all possible startingpoints and directions, our algorithm will recover all partialisometries. However, since this is infeasible, we reducethe search space using sparse feature matches. This sec-tion outlines an algorithm to find features matches. How-ever, note that this part is not a novel contribution of thispaper and only given for completeness; in principle, anyfeature matches can be used to reduce the search space, asdemonstrated in Section 5, where we show a result usingimage-based feature matches from a multi-view capturesetup. We denote the feature descriptor at a point s ∈ Swith a vector DS(s), and similarly for points on T .
We identify features on S and T using the geodesic fin-gerprint descriptor DS(s) [48], which compares how anisocontour of the geodesic distance from a point s ∈ Sdeviates in length from the isocontour of the same 2DEuclidean distance. We use 10 isocontours in all exper-iments, for geodesic radii between rmin and rmax. Thevalues of rmin and rmax need to be varied slightly depend-ing on the amount of noise in the input data, and are dis-cussed in Section 5. We define the distinctiveness FS(s)of a point s ∈ S as the sum of L1 distances to the rest ofthe vertices of S in descriptor space. Features are pointsthat locally maximize distinctiveness. The left-most boxin Figure 4 shows features color-coded by distinctiveness(red for most distinct, blue for least distinct).
To find feature matches, we begin by computing the Carte-sian product of L2 descriptor distances between featureson S and T . The vast majority of these are not correctmappings between the surfaces. We filter these potentialfeature matches using both the L2 distances between de-scriptors and the distinctiveness of the features. More pre-cisely, for each feature s on S, we only consider the Kfeatures of T that have the closest descriptor matches to s.(We use K = 10 in all our experiments.) The initial dis-similarity δ(init)
s,t between two features s ∈ S and t ∈ T isdefined as
δ(init)s,t = − log (FS(s)) + ωD ‖DS(s)−DT (t)‖22 , (7)
where ωD is a weight. (We use ωD = 400 in all our exper-iments.) This dissimilarity is minimized for features withhigh distinctiveness that are similar in descriptor space.
This procedure often produces a good set of sorted featurematches on clean data. For noisy data from real scanners,however, the descriptors will be less discriminative, andconsidering spatial relations between features in additionto descriptors and distinctiveness leads to more reliablefeature matches.To do this, we iteratively build (possibly overlapping)clusters Ci of consistent feature matches. In each itera-
tion, the feature match (s, t) with the next lowest δ(init)s,t
is chosen as a starting point for Ci. The cluster Ci isbuilt by repeatedly adding the close-by feature match thathas the lowest dissimilarity to Ci. More precisely, letCi = {(sj , tj)}. In the next step, all features s′ 6∈ Cithat are neighbors of sj in the topology hierarchy are con-sidered, along with their K potential matches t′ ∈ T . Thedissimilarity δCi,(s′,t′) between a feature match (s′, t′)and cluster Ci is defined as
δCi,(s′,t′) = δ
(init)s,t + ωC
∑(sj ,tj)
∣∣distS(sj , s′)− distT (tj , t
′)∣∣2 ,
(8)
where ωC is a weight. (In all our experiments, we set ωCto one over 8× the square of the average edge length onS.) We repeatedly add the match (s′, t′) with smallest dis-similarity δCi,(s′,t′) to Ci as long as δCi,(s′,t′) is below athreshold. (We use threshold 11.5 ≈ − log 10−5 in all ourexperiments.) We stop adding new clusters once the sumof the cardinalities of the clusters Ci exceeds the initialnumber of feature matches.
Note that the above clustering scheme is equivalent tomodeling both the descriptor distances and the summedstretches of geodesic distances of a feature match to acluster as normally distributed. Hence, the above stop-ping criteria correspond to stopping the clustering oncethe joint probability of a match belonging to a cluster be-comes small.
After the clustering, we place the feature matches (s, t)in a min-priority queue, so that we start isometric re-gion growing first from the matches we expect to bemost reliable. A feature match (s, t) is assigned prior-ity ∞ if (s, t) is not part of any cluster Ci and priorityminCi:(s,t)∈Ci
δCi,(s′,t′) otherwise. We repeatedly takethe minimum element from queue and use it to generatepartial isometric mappings using the region growing ofSection 4.3.
However, so far we have only established a positionalcorrespondence between features, and we need a direc-tional correspondence as well to fix the partial isometrywe wish to grow. To establish direction, we build a min-priority queue as outlined above, but only with the subsetof features in the neighborhood of the current positionalmatch in the topology hierarchy. Matches of neighboringfeatures allow us to find corresponding tangent plane di-rections from corresponding smoothed geodesic paths be-tween feature points.
Once a partial isometry has been computed, we increasethe priority of feature matches that are redundant given thealready computed partial match. If the same source andtarget points are already matched, by our model, the resultof running the growing again from those same points willbe equivalent.
To keep the run-time of our method bounded, we stop aftera fixed number of oriented point matches have been tried.(We use 200 in all our experiments.) A more general stop-ping criterion could be devised based on determining themaximal coverage of S and T subject to a consistent map-ping. The second box from the left of Figure 4 shows threeoriented feature matches found using this method.
9
4.3 Isometric Region Growing
Starting from an oriented point match s,ds, t,dt, wegrow the region U by adding matches incrementally in thelocal neighborhood of the boundary of U . For a new pointu near the boundary of U such that u /∈ U , let N1(u)denote the neighborhood of u in level 1 of our topologyhierarchy. We use parallel transport along correspondingdirections in S and T emanating from an oriented pointmatch s′,ds′ , t
′,dt′ , where s′ ∈ U ∩ N1(u) is in a lo-cal neighborhood of u. We know the full path betweens′ and u on S, and from ds′ and dt′ we know the corre-sponding start direction on T . Hence, we can transport thestart direction along corresponding paths on S and T untilwe have traveled distS(s′,u). In practice, we implementparallel transport by moving in small steps in the tangentplane, and by reprojecting the resulting point to the surfaceand updating the tangent to the path. We use smoothedgeodesic paths to transport the position of the new matchon T to reduce discretization errors for differently sam-pled surfaces. For robustness, we use all available orientedpoint matches inN1(u) and take the Riemannian center ofmass [49] of the transported points, where all transportedpoints have equal mass.
The newly matched source point u is added to U only if itlocally respects the stretch factor ν as given in Eq. 5. Thisis necessary for two reasons. First, subsequent matcheswill be estimated based on the assumption that the exist-ing matches are near-isometric. Second, as discussed next,we use a nonlinear least-squares optimization to refine thepositions of the matched points on T , which effectivelydistributes the error evenly over the matched region. Toavoid introducing errors, it is therefore important to ver-ify that each newly added point match locally respects thestretch factor ν.
To reduce the effect of quantization errors and noise, weoptimize the metric matching of U using a non-linear op-timization technique [14] every time the area of U hasdoubled. This means more frequent optimizations at thestart of the growing process. This optimization reduces theamount of drift as it re-aligns the matched regions whiletaking into account long geodesics, between neighbors inthe top level of our topology hierarchy, in U . For increasedefficiency, we only consider edges in the topology hierar-chy of S (explained in Section 4.1) during the optimiza-tion.
An important difference to the global approach used byBronstein et al. [14] is that we optimize the metric onlyusing geodesic paths which are entirely within U andf(U). This models the isometry criterion in partial re-gions and is crucial to handling topology changes andmissing data. Note that such defects may cause largedifferences between distS(x,y) and distU (x,y), as wellas distT (f(x), f(y)) and distf(U)(f(x), f(y)), respec-tively. The second box from the right in Figure 4 showsthe first three partial isometries found by growing isomet-rically from oriented feature matches in our priority queue.
The proper value for the stretching threshold ν dependson a number of factors: material properties, the resolu-
tion at which the surface is sampled (as it affects the accu-racy of the Dijkstra paths), and the noise of the acquisitionprocess. In our experiments, we do not consider materialproperties, and we assume that the acquisition noise hasan equal influence on both source and target. Hence, weset ν to 0.5ε0 to account for quantization effects in thecomputation of geodesics.
4.4 Combining Equivalent Partial Maps
It remains to identify and merge a set of partial mappingsthat represent the same mapping function f . If the surfaceswere related by exact isometries and noise was negligible,the following step would not be required. However, forreal-world data, the identification of functions that are ap-proximately equivalent improves the results substantially.
The problem at this point similar to the blending problemby Kim et al. [18]. Recall that their approach uses a spec-tral method to find blending weights for different maps.This is a good approach in their case as blending weightsare given as the solution to a quadratic energy function.Note that since in our model, equivalent mappings form a2-manifold in parameter space Θ, and the parameter spaceΘ is non-linear, it is not appropriate to use a spectral ap-proach.
However, we can take advantage of the property thatequivalent mappings form a 2-manifold in Θ. We employthe agglomerative clustering algorithm of Zhang et al. [24]for discovering manifold structures in high-dimensionaldata based on the in-degree and out-degree of the nearest-neighbor graph of points in high-dimensional space. Inour case, we consider the nearest neighbor graph of par-tial isometric mappings, where the dissimilarity betweenthese points in Θ is measured as follows.
We compare different maps fi and fj using a dissimilaritymeasure based on their domains Ui and Uj :
dΘ(fi, fj) = W1
∫Uij
distT (fi(x), fj(x))dx
+ W2
∫fi(Uij)∩fj(Uij)
distS(f−1i (y), f
−1j (y))dy(9)
where Uij = Ui ∩ Uj , W1 = 1A(Uij) , W2 =
1A(fi(Uij)∩fj(Uij)) , and A(·) denotes the surface area. Inpractice, we compute a discrete version of this dissim-ilarity by replacing integrals over regions by sums oververtices in the region. We cluster different mappings to-gether until the maximum affinity between any two clus-ters is greater than a threshold ρ. Affinity is computedfrom the weighted graph degree between mappings, wherethe weights have a double-exponential fall-off as dΘ in-creases. See Zhang et al. [24] for details. While the di-rect relation of ρ to the allowed stretch is not easily de-fined, it should be lower when we want to allow for greaterstretching between partial maps. We only have to ad-just this value in a few cases in our experiments, as dis-cussed in Section 5. Following the clustering, we selectfor our final mapping the cluster with the highest intra-cluster affinity, or connectivity, as proposed by Zhang et
10
al. [24]. This most often correlates with the cluster thatcovers the largest portion of the surface.
We merge clustered maps by computing a weightedgeodesic average on T for each source point in the unionof the mapped regions as
f(x) = arg miny∈T
∑i
wi(x) distT (y, fi(x)), (10)
where fi are the clustered partial isometries, whichwe want to merge into f . The weights are com-puted as an exponential distribution in the geodesic dis-tance from the starting point match si as wi(x) =exp(−λds distUi(x, si)), reflecting that we expect errorsto accumulate as the growing proceeds because of dis-cretization artifacts, data noise, and deviations from isom-etry. We set λds = 1/(5ε1), where ε1 is the sample spac-ing of the level 1 of the topology hierarchy described inSection 4.1. Note that this is equivalent to finding the Rie-mannian center of mass [49] of the estimates fi(x). Theright-most box of Figure 4 shows the final clustered andmerged result.
5 Experiments
To validate our theoretical analysis, we evaluate a directimplementation of our algorithm and compare our resultsto state of the art approaches. In the following, we demon-strate that our method achieves results that are either com-parable or superior to the state of the art, which demon-strates that our algorithm not only has theoretical advan-tages, but is applicable in practice as well.
5.1 Implementation Details
We implemented the algorithm described in Section 4 inC++, and conducted our evaluations on a standard laptopPC. During the evaluation, all but two types of parame-ters are fixed. The first type of parameters that is variedis {rmin, rmax}, which controls the size of the neighbor-hood used to compute surface descriptors. If the radii areset higher, then the method is more robust with respect tonoise at the cost of potentially missing features of smallscale. In our experiments, we only use two settings forthese parameters. The first setting, rmin = 0.9R andrmax = 1.7R, is for relatively clean data, and the secondsetting, rmin = 1.5R and rmax = 3.4R, is for noisy data.Here, R is set to 5% of the diameter of S. The secondparameter that is varied is the threshold ρ used to controlthe clustering. Lower values of ρ allow for less isometricpatches to be clustered together. We vary ρ ∈ {0, 1, 1.9}in our experiments.
For the examples discussed below, our algorithm takes be-tween 30 minutes and 8 hours to compute the final result.To give an idea of the distribution of the time, we dis-cuss the running time for one pair of models (the space-carved samba models shown in Fig. 9) in more detail. Forthis pair, finding oriented feature matches takes about 2
minutes, growing partial mappings takes about 1.5 hours,clustering the mappings takes about 9 minutes, and merg-ing the patches takes about 44 minutes. Hence, the totaltime to compute the results is about 2.4 hours. Note how-ever, that the running time of our method depends signif-icantly on the distinctiveness of the intrinsic geometry ofthe surfaces, relative to the noise level. For example, thetemplate-fitted samba models shown in Fig. 6 take about1.5 hours in total, despite having more than twice as manyvertices as the space-carved versions, because the approx-imate isometry criterion is more discriminative (geodesicdistances are less perturbed by surface noise).
5.2 Comparison to State of the Art
We compare the performance of our algorithm against fourexisting methods, namely heat kernel maps (HKM) [11],blended intrinsic maps (BIM) [18], the method of Sharmaet al. [32], and the method of Tevs et al. [50]. We se-lect these methods for comparison because they representthe state of the art for matching between surfaces. Morespecifically, HKM is derived from the theoretical com-plexity of isometric mappings, BIM can match surfacesthat exhibit local deviations from isometry, and the meth-ods of Sharma et al. and Tevs et al. are heuristics that havebeen specifically designed for matching partial data withtopological noise. For HKM we use our own implementa-tion, which uses two feature correspondences to initializethe mapping, for BIM we use the authors’ implementa-tion, for comparisons with Sharma et al., we run our codeon their data, and for the method of Tevs et al., the authorswere kind enough to run their algorithm on our data.
We show comparative evaluations on a variety of types ofdata. The first type is a synthetic data set that helps inunderstanding the major difference between our approachand HKM, and illustrates the partial isometric matchingmodel. The second type is data acquired using either alaser scanner or an image-based reconstruction system thatwas processed by fitting a template to the data. In thiscase, we treat the result of the template fitting as groundtruth. The third, and most challenging, type is unprocessedreal-world data acquired using different acquisition sys-tems.
To compare our approach to previous methods, we use twoevaluation methodologies. In cases where ground truth isavailable, we compare quantitatively by evaluating the ac-curacy of different results with respect to the ground truth.For data that has no ground truth, we rely on visual evalua-tion. Our visualization scheme is as follows. A texture onS is mapped to corresponding points on T , and regions ofT that have no correspondence in S are colored red. As atexture we combine constant coloring of semantically dis-tinct parts (where applicable) with a checkerboard pattern.This type of texture simultaneously shows both global se-mantic accuracy and fine-scale distortion.
11
Heat kernel maps [11]
This paper
Figure 5: Results of our method and HKM on globallynon-isometric data. Red indicates unmatched area.
5.3 Synthetic Data
We start by comparing our algorithm against HKM usingthe synthetic example shown in Fig. 5, where we map froma plane to a plane with three peaks. This experiment illus-trates the difference between a global isometric matchingmodel and partial one: the two surfaces are globally non-isometric, but the planar part is isometric. In this example,we use two ground truth correspondences to initialize bothalgorithms to remove differences due to different featurematching approaches. Since HKM aims to map the shapesusing a global isometry, the result maps planar parts to thepeaks, while our method successfully detects the largestpart of the surfaces that can be isometrically mapped.
5.4 Template-Fitted Scan Data
Next, we consider acquisitions of real-world data that wasprocessed by fitting a template shape to the raw data. Forall experiments in this section, we use rmin = 0.9R andrmax = 1.7R.
We first use two frames of the samba sequence by Vlasicet al. [51]. These frames are locally very close to isomet-ric, but globally have high non-isometric distortion due tothe dress being connected to the legs. We therefore setρ = 0 in the clustering step. Vlasic et al. provide a pro-cessed version of the frames, where a template was fittedto the data. This processing ensures that the models havethe same topology, and the processed data can be used asground truth correspondence.
We compare our method to HKM and BIM using the pro-cessed frames of the samba sequence. Figure 6 shows theresults. Note that while HKM leads to a result with visualartifacts, the results of BIM and our method are visuallypleasing. Furthermore, since we have ground truth cor-respondences, we show the cumulative error distributionsfor all three methods in Figure 7. Here, geodesic error ismeasured as a fraction of the square root of the surfacearea of T . Note that our method numerically outperformsthe two other methods.
The experiments conducted so far have shown that HKM,while being based on a solid theoretical foundation, leadsto results of low quality when the aim is to find densecorrespondences in datasets that contain noise and non-isometric distortion. Hence, in the following, we exclude
0
20
40
60
80
100
0 0.05 0.1 0.15 0.2 0.25
% M
atch
es
Geodesic Error
This paperBlended intrinsic maps
Heat kernel maps
Figure 7: Cumulative error distributions for BIM, HKM,and our method based on the known ground truth. Themodels are shown in Figure 6.
0
20
40
60
80
100
0 0.05 0.1 0.15 0.2 0.25
% M
atch
es
Geodesic Error
This paperBlended intrinsic maps
Figure 8: Cumulative error distributions for BIM andour method based on the known ground truth for SCAPEdataset.
this method from our comparisons.
Second, we test our algorithm on the SCAPE dataset [52]consisting of 71 scans of a male scanned in different pos-tures. In our experiment, we match the neutral posture toall 70 remaining postures using BIM and our approach.The cumulative error distributions for all 70 mappingsare shown in Figure 8. This data differs from the sambaframes in that it is globally near-isometric, but containslocal areas of high-distortion (at joints for example). Forthis reason it provides a different kind of near-isometrictest, and poses a greater challenge for our method, whichdoes not exploit global assumptions. This is reflected inour error curve being below that of BIM. We set ρ = 1.9 inthis experiment to reduce the influence of partial isometricpatches that were thrown off by high local distortions.
5.5 Raw Scan Data
Finally, we consider raw scan data acquired using differentacquisition systems. This type of data is noisy and incom-plete, and is therefore significantly more challenging tomatch than the data used in the previous experiments. Byusing data from a variety of acquisition systems, we showour methods robustness to different types of acquisitionnoise. We also show our method’s robustness to other ac-quisition artifacts that violate global isometry: large holesand contacts.
12
source and target Heat kernel maps [11] Blended intrinsic maps [18] This paper
Figure 6: Comparison to HKM and BIM on models with ground truth. For error rates see Figure 7. Red indicatesunmatched area; all further colors and the checker board have been painted on the source surface in order to visualizecorrespondences.
First, we use the same two frames of the samba sequenceby Vlasic et al. [51] that were used above. However,this time, we consider the geometry reconstructed by aspace-carving algorithm rather than template fitting. Inthis case, the frames we choose have different topology(as the hands merge with the body at the hips in one of themodels) and are severely corrupted by noise. For this rea-son, we set rmin = 1.5R, rmax = 3.4R, and ρ = 1. Weuse these models to compare our method to BIM, Tevs etal.’s method, and Sharma et al.’s method. Figure 9 showsthe results. Note that the results using BIM and the methodby Tevs et al. match parts of the body of S to the arms andlegs of T , thereby leading to visual artifacts. (See enlargedareas in Figure 9.) The method of Sharma et al., which isespecially well suited to the scenario of handling contacts,leads to a result that covers the surfaces well. For thisexample, our method is run using two feature sets: first,using the standard features of our method and second, us-ing the same image-based features used by Sharma et al.Our method produces a visually accurate mapping in bothcases. However, when using standard features, the resultof our method does not cover the right foot of the targetsurface, while the entire target surface is covered whenusing image-based features. Note that both the result bySharma et al. and our results detect the contacts correctlyand stop the growing in these regions. Hence, all areas ofthe surface are matched well. The improved performancewith image-based features illustrates some of the technicalchallenges. The high level of surface noise makes match-ing geometric features difficult, which results in poorersampling of the space of isometries, which in turn resultsin a poorer clustering result. We note however, that usingthe same features as Sharma et al., we obtain equal cov-erage and accuracy, and that when using purely geomet-ric information, we outperform the other purely geometricmethods tested.
Second, we compare our method to BIM and Tevs etal.’s method using two models of the BU-3DFE facedatabase [53]. The two models contain numerous smallholes and outliers. Furthermore, the models have differ-ent topology because the mouth is closed in one modeland open in the other one. For these reasons, we setrmin = 0.9R, rmax = 1.7R, and ρ = 1. Figure 10shows the results. Note that our method obtains a map-
source and targetBlended intrinsic
maps [18]
Tevs et al. [50] Sharma et al. [32]
This paperThis paper with
image-based features
Figure 9: Comparison in presence of large contacts. Eachpair shows S on the left and T on the right. Different datathan in Figure 6. Red again indicates unmatched area
13
ping with higher visual accuracy than both BIM and themethod of Tevs et al. In the case of BIM, this is to beexpected, since the topological change breaks the globalisometry. (Note the lips mapped to the side of the face.)The method of Tevs et al. produces a much more accu-rate result, but with still significant overall distortion andoutliers (speckle-like effect). Our method leverages lo-cal information to fix partial isometries, and produces amapping that largely preserves the semantic coloring withsignificantly lower distortion and without outliers.
Third, we compare our method to that of Tevs et al. on twopoint clouds acquired using a laser scanner. The two mod-els contain numerous holes and outliers. As the models arepoint clouds and BIM requires input meshes, we do notcompare our result to BIM for this experiment. For thesemodels, we set rmin = 1.5R, rmax = 3.4R, and ρ = 1.The results are shown in Figure 11. As can be seen, themethod of Tevs et al. obtains better coverage, however ourmethod has fewer outliers–islands of incorrectly mappedpoints within larger smoothly mapped regions (see the en-larged parts of Figure 11). The suboptimal coverage ofour method is due to the difficulty in matching featureson surfaces plagued by missing data. A feature descriptorless sensitive to holes could improve this.
5.6 Limitations
In the previous sections, we have demonstrated that ourmethod not only has theoretical advantages, but also com-putes results that improve upon the state of the art resultsin challenging cases where the input data is a pair of rawscans with topological noise.
We now discuss some limitations of our algorithm. Our al-gorithm is based on growing near-isometric mappings be-tween partial regions of two surfaces and then clusteringconsistent mappings together. In this way, our algorithmenumerates a set of near-isometric mappings. For mod-els that exhibit a large number of partial intrinsic sym-metries, this technique enumerates a large set of near-isometric mappings where many of the mappings are in-consistent with each other. Since we stop growing newnear-isometric regions after the first 200 oriented featurepoint matches have been considered, for shapes with alarge number of partial near-isometric symmetries, it mayhappen that there is no cluster of consistent mappings cov-ering a large area of S.
An example where the clustering step fails to identify acluster of consistent mappings covering a large area of Sis shown in Figure 12. The two frames of the flashkickdataset [54] contain a large topological change due to amerge of the subject’s pants in one of the models. Notethat our algorithm correctly stops the growing of singlepartial mapping in this area, as shown in Figure 12. How-ever, since the legs and core of the target body are intrinsi-cally symmetric (similar to cylindrical), many inconsistentpartial mappings are found by the algorithm, and they can-not be clustered in a consistent way.
We should also note that the computational costs are quite
input: source (left) and target (right)
Blended intrinsic maps [18]
Tevs et al. [50]
This paper
Figure 10: Comparison on data with significant topologi-cal changes and acquisition noise. Each pair shows S onthe left and T on the right. Unmatched area marked in red.
14
front back
Tevs et al. [50]
front back
This paper
Figure 11: Comparison on data with significant holes andacquisition noise. The first row shows the method of Tevset al., while the second row shows our method. Withineach row, the front and back are shown, in each case withS on the left and T on the right. Unmatched area shownin red.
Figure 12: A single partial mapping for an example withlarge topological changes. The red area is unmatched.
high. This is partially due to unoptimized code, butan algorithmic shortcoming is the rather simple feature-matching algorithm for finding start positions and direc-tions. Optimizing this was not the focus of this work; ourmethod should rather be understood as an alternative forthe dense matching step where it can replace previous ap-proaches based on conformal maps ( [10, 18] and follow-ups), heat-kernel maps ( [11] and follow-ups), or geodesictriangulation with landmarks ( [15, 16] and follow-ups) inorder to handle partial matching.
For future work, to address this limitation, we plan to com-bine our approach with an approach that detects continu-ous intrinsic symmetries [55], thereby reducing the searchspace for the initial feature matching and allowing us toefficiently enumerate all partial matches. Similarly, ourframework could be extended to find partial intrinsic sym-metries within a single object.
6 Conclusion
We have analyzed the complexity of the isometric match-ing problem under global and local isometry assumptionsand based on this analysis we have introduced a new ap-proach to solve the partial isometric matching problemusing a representation for partial isometries that is bothlow-dimensional and redundant. Underpinning this is thefundamental observation that isometric mappings can bedetermined using purely local information and have onlythree degrees of freedom on 2-manifolds. The local met-ric propagation algorithm we derived from this observa-tion is designed to handle topological noise that could af-fect large portions of the model, including both large holesand contacts. The redundancy in the representation can beexploited to increase robustness of and to combine partialmatches.We have shown how a direct implementation ofthis theoretical framework can be used to match challeng-ing surfaces with different types of topological noise.
The insights gained by studying the partial isometricmatching problem have the potential to impact other shapeprocessing tasks. For instance, the representation for par-tial isometric matches introduced in this paper can be usedto derive new algorithms to detect partial symmetries ofshapes. For future work, we plan to further investigatethis option.
Acknowledgements
The authors thank Art Tevs, Aurela Shehu, Waqar Khanand Avinash Sharma for their help in conducting the com-parative evaluation, Vladimir Kim for making his codeavailable, and Art Tevs, Silke Jansen, Alexander Berner,Qi-Xing Huang, and Leonidas Guibas for discussions. Wealso thank the anonymous reviewers for their valuablecomments that we used to improve the paper.
15
References
[1] J. Davis, D. Nehab, R. Ramamoorthi, andS. Rusinkiewicz, “Spacetime stereo: A unifyingframework for depth from triangulation,” IEEEPAMI, vol. 27, no. 2, pp. 296–302, 2005.
[2] S. Konig and S. Gumhold, “Image-based motioncompensation for structured light scanning of dy-namic scenes,” Int. J. of Int. Sys. Tech. App., vol. 5,no. 3/4, pp. 434 – 441, 2008.
[3] T. Weise, B. Leibe, and L. V. Gool, “Fast 3d scan-ning with automatic motion compensation,” in Proc.CVPR, 2007, pp. 1–8.
[4] D. Vlasic, P. Peers, I. Baran, P. Debevec, J. Popovic,S. Rusinkiewicz, and W. Matusik, “Dynamic shapecapture using multi-view photometric stereo,” TOG,vol. 28, no. 5, pp. 174:1–174:11, 2009.
[5] P. Debevec, “Virtual cinematography: Relightingthrough computation,” IEEE Computer, vol. 39,no. 8, pp. 57–65, August 2006.
[6] V. Blanz and T. Vetter, “A morphable model for thesynthesis of 3d faces,” in Proc. SIGGRAPH, 1999,pp. 187–194.
[7] B. Allen, B. Curless, and Z. Popovic, “The space ofhuman body shapes: Reconstruction and parameter-ization from range scans,” TOG, vol. 22, no. 3, pp.587–594, 2003.
[8] N. Hasler, C. Stoll, M. Sunkel, B. Rosenhahn, andH.-P. Seidel, “A statistical model of human pose andbody shape,” CGF (Proc. Eurographics), vol. 28,no. 2, 2009.
[9] T. Weise, S. Bouaziz, H. Li, and M. Pauly, “Realtimeperformance-based facial animation,” TOG, vol. 30,no. 4, pp. 77:1–77:10, 2011.
[10] Y. Lipman and T. Funkhouser, “Mobius voting forsurface correspondence,” TOG, vol. 28, no. 3, pp.72:1–72:12, 2009.
[11] M. Ovsjanikov, Q. Merigot, F. Memoli, and L. J.Guibas, “One point isometric matching with the heatkernel,” CGF, vol. 29, no. 5, pp. 1555–1564, 2010.
[12] A. Tevs, A. Berner, M. Wand, I. Ihrke, and H.-P. Sei-del, “Intrinsic shape matching by planned landmarksampling,” CGF, vol. 29, no. 2, pp. 543–552, 2011.
[13] D. Anguelov, P. Srinivasan, H.-C. Pang, D. Koller,S. Thrun, and J. Davis, “The correlated correspon-dence algorithm for unsupervised registration ofnonrigid surfaces,” in Proc. NIPS, 2005, pp. 33–40.
[14] A. M. Bronstein, M. M. Bronstein, and R. Kimmel,“Generalized multidimensional scaling: a frame-work for isometry-invariant partial surface match-ing,” PNAS, vol. 103, no. 5, pp. 1168–1172, 2006.
[15] Q.-X. Huang, B. Adams, M. Wicke, and L. J.Guibas, “Non-rigid registration under isometric de-
formations,” CGF, vol. 27, no. 5, pp. 1449–1457,2008.
[16] A. Tevs, M. Bokeloh, M. Wand, A. Schilling, and H.-P. Seidel, “Isometric registration of ambiguous andpartial data,” in Proc. CVPR, 2009, pp. 1185–1192.
[17] J. Sun, M. Ovsjanikov, and L. Guibas, “A conciseand provably informative multi-scale signature basedon heat diffusion,” CGF, vol. 28, no. 5, pp. 1383–1392, 2009.
[18] V. G. Kim, Y. Lipman, and T. Funkhouser, “Blendedintrinsic maps,” TOG, vol. 30, no. 4, pp. 79:1–79:12,2011.
[19] M. Ovsjanikov, M. Ben-Chen, J. Solomon,A. Butscher, and L. Guibas, “Functional maps:a flexible representation of maps between shapes,”TOG, vol. 31, no. 4, pp. 30:1–30:11, 2012.
[20] A. M. Bronstein, M. M. Bronstein, R. Kim-mel, M. Mahmoudi, and G. Sapiro, “A Gromov-Hausdorff framework with diffusion geometry fortopologically-robust non-rigid shape matching,”IJCV, vol. 89, no. 2-3, pp. 266–286, 2010.
[21] Y. Lipman, R. M. Rustamov, and T. A. Funkhouser,“Biharmonic distance,” TOG, vol. 29, no. 3, pp.27:1–27:11, 2010.
[22] K. Xu, H. Zhang, W. Jiang, R. Dyer, Z. Cheng,L. Liu, and B. Chen, “Multi-scale partial intrinsicsymmetry detection,” TOG, vol. 31, no. 6, pp. 181:1–181:11, 2012.
[23] M. Berger, A Panoramic View of Riemannian Geom-etry. Springer, 2002.
[24] W. Zhang, X. Wang, D. Zhao, and X. Tang, “Graphdegree linkage: Agglomerative clustering on a di-rected graph,” in Proc. ECCV, 2012, pp. 428–441.
[25] O. van Kaick, H. Zhang, G. Hamarneh, andD. Cohen-Or, “A survey on shape correspondence,”CGF, vol. 30, no. 6, pp. 1681–1707, 2011.
[26] G. Tam, Z.-Q. Cheng, Y.-K. Lai, F. Langbein, Y. Liu,D. Marshall, R. Martin, X.-F. Sun, and P. Rosin,“Registration of 3d point clouds and meshes: A sur-vey from rigid to non-rigid,” TVCG, To appear.
[27] A. Elad and R. Kimmel, “On bending invariant sig-natures for surfaces,” TPAMI, vol. 25, no. 10, pp.1285–1295, 2003.
[28] V. Jain, H. Zhang, and O. van Kaick, “Non-rigidspectral correspondence of triangle meshes,” IJSM,vol. 13, no. 1, pp. 101–124, 2007.
[29] S. Wuhrer, Z. B. Azouz, C. Shu, and P. Bose, “Pos-ture invariant correspondence of incomplete triangu-lar manifolds,” IJSM, vol. 13, no. 2, pp. 139–157,2007.
[30] O. van Kaick, H. Zhang, and G. Hamarneh, “Bilat-eral maps for partial matching,” CGF, vol. to appear,2013.
16
[31] M. Aubry, U. Schlickewei, and D. Cremers, “Thewave kernel signature: A quantum mechanical ap-proach to shape analysis,” in Proc. ICCV Workshops,2011, pp. 1626–1633.
[32] A. Sharma, R. P. Horaud, J. Cech, and E. Boyer,“Topologically-robust 3d shape matching based ondiffusion geometry and seed growing,” in Proc.CVPR, 2011, pp. 2481–2488.
[33] Y. Sahillioglu and Y. Yemez, “Scale normalizationfor isometric shape matching,” Computer GraphicsForum (Proc. Pacific Graphics), vol. 31, no. 7, 2012.
[34] A. Bronstein, M. Bronstein, A. Bruckstein, andR. Kimmel, “Partial similarity of objects, or how tocompare a centaur to a horse,” IJCV, vol. 84, no. 2,pp. 163–183, 2009.
[35] D. Raviv, A. Bronstein, M. Bronstein, and R. Kim-mel, “Full and partial symmetries of non-rigidshapes,” IJCV, vol. 89, no. 1, pp. 18–39, 2010.
[36] D. Hahnel, S. Thrun, and W. Burgard, “An extensionof the ICP algorithm for modeling nonrigid objectswith mobile robots,” in Proc. IJCAI, 2003, pp. 915–920.
[37] H. Zhang, A. Sheffer, D. Cohen-Or, Q. Zhou, O. vanKaick, and A. Tagliasacchi, “Deformation-drivenshape correspondence,” CGF, vol. 27, no. 5, pp.1431–1439, 2008.
[38] H. Li, R. W. Sumner, and M. Pauly, “Global corre-spondence optimization for non-rigid registration ofdepth scans,” CGF, vol. 27, no. 5, pp. 1421–1430,2008.
[39] M. Wand, B. Adams, M. Ovsjanikov, A. Berner,M. Bokeloh, P. Jenke, L. Guibas, H.-P. Seidel, andA. Schilling, “Efficient reconstruction of nonrigidshape and motion from real-time 3d scanner data,”TOG, vol. 28, no. 2, pp. 1–15, 2009.
[40] H. Li, B. Adams, L. J. Guibas, and M. Pauly, “Ro-bust single-view geometry and motion reconstruc-tion,” TOG, vol. 28, no. 5, pp. 175:1–175:10, 2009.
[41] A. Pentland and J. Williams, “Good vibrations:modal dynamics for graphics and animation,” inProc. SIGGRAPH, 1989, pp. 215–222.
[42] R. Schmidt, C. Grimm, and B. Wyvill, “Interactivedecal compositing with discrete exponential maps,”TOG, vol. 25, no. 3, pp. 605–613, 2006.
[43] R. Schmidt, “Stroke parametrization,” CGF, vol. 32,no. 3, 2013.
[44] E. L. Malvaer and M. Reimers, “Geodesic polar co-ordinates on polugonal meshes,” CGF, vol. 31, no. 8,pp. 2423–2435, 2012.
[45] R. Palais, “On the differentiability of isometries,”Proc. of the AMS, vol. 8, no. 4, pp. 805–807, 1957.
[46] A. Oztireli, G. Guennebaud, and M. Gross, “Featurepreserving point set surfaces based on non-linear ker-nel regression,” CGF, vol. 28, no. 2, 2009.
[47] F. Memoli and G. Sapiro, “A theoretical and com-putational framework for isometry invariant recogni-tion of point cloud data,” FoCM, vol. 5, no. 3, pp.313–347, 2005.
[48] Y. Sun and M. Abidi, “Surface matching by 3dpoint’s fingerprint,” in Proc. ICCV, 2001, pp. 263–269.
[49] R. Rustamov, “Barycentric coordinates on surfaces,”CGF, vol. 29, no. 5, 2010.
[50] A. Tevs, A. Berner, M. Wand, I. Ihrke, M. Bokeloh,J. Kerber, and H.-P. Seidel, “Animation cartography -intrinsic reconstruction of shape and motion,” TOG,vol. 31, no. 2, p. 15, April 2012.
[51] D. Vlasic, I. Baran, W. Matusik, and J. Popovic, “Ar-ticulated mesh animation from multi-view silhou-ettes,” TOG, vol. 27, no. 3, pp. 97:1–97:9, 2008.
[52] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun,J. Rodgers, and J. Davis, “Scape: Shape completionand animation of people,” TOG, vol. 24, no. 3, pp.408–416, 2005.
[53] L. Yin, X. Wei, Y. Sun, J. Wang, and M. J. Rosato,“A 3d facial expression database for facial behaviorresearch,” in Proc. FG, 2006, pp. 211–216.
[54] J. Starck and A. Hilton, “Surface capture for perfor-mance based animation,” CG&A, vol. 27, no. 3, pp.21–31, 2007.
[55] M. Ben-Chen, A. Butscher, J. Solomon, andL. Guibas, “On discrete killing vector fields and pat-terns on surfaces,” CGF, vol. 29, no. 5, 2010.
17