Joint 3D Shape Matching

December 2015

Abstract

In this paper we extend the global optimization-basedapproach of jointly matching a set of images to jointlymatching a set of 3D meshes. The estimated correspon-dences simultaneously maximize pairwise feature affini-ties and cycle consistency across multiple models. Weshow that the low-rank matrix recovery problem can beefficiently applied to the 3D meshes as well. The fastalternating minimization algorithm helps to handle realworld practical problems with thousands of features. Ex-perimental results show that, unlike the state-of-the-artalgorithm which rely on semi-definite programming, ouralgorithm provides an order of magnitude speed-up alongwith competitive performance. Along with the joint shapematching we propose an approach to apply a distortionterm in pairwise matching, which helps in successfullymatching the reflexive sub-parts of two models distinc-tively. In the end, we demonstrate the applicability ofthe algorithm to match a set of 3D meshes of the SCAPEbenchmark database.

1 Introduction

Finding feature correspondences between two meshes is anew problem in computer vision with various applicationssuch as structure from motion, model registration, shapeanalysis, to name a few. While previous efforts weremostly focused on matching a pair of images, many tasksrequire to find correspondences across multiple meshes infact. A typical example is nonrigid structure from motion[4], where one can hardly reconstruct a nonrigid shapefrom two frames. Furthermore, recent work has shownthat leveraging multi-way information can dramaticallyimprove matching results compared to pairwise match-

ing. The most important constraint for joint matchingis the cycle consistency, i.e., the composition of matchesalong a loop of meshes should be identity, as illustratedin Figure 1. Given pairwise matches, one can possiblyidentify true or false matches by checking all cycles inthe mesh collection. But there are many difficulties forthis approach. For example, the input pairwise matchesare often very noisy Figure 1. An illustration of consis-tent multi-mesh matching. With many false matches andmissing matches, and the features detected from differentmeshes may only have a partial overlap even if the samefeature detector is applied. Therefore, it is likely that veryfew consistent cycles can be found. Moreover, how tosample cycles is not straightforward due to the huge num-ber of possibilities. Recent work on joint matching hasshown that, if all feature correspondences within multi-ple meshes are denoted by a large binary matrix, the cy-cle consistency can be translated into the fact that sucha matrix should be positive semi-definite and low-rank .Based on this observation, convex optimization-based al-gorithms were proposed, which achieved the state-of-the-art performances with theoretical guarantees . But thesealgorithms rely on semi-definite programming (SDP) [9],which is not computationally efficient to handle imagematching problems in practice.

In this paper, we make use of MatchALS [16], a novelalgorithm for multi image matching, and use it for 3Dmodel meshes. The inputs to our algorithm are orig-inal similarities between feature descriptors which weused Heat Kernel Signature as. The outputs are fea-ture correspondences between all pairs of images. Un-like many previous methods starting from quantized pair-wise matches , we postpone the decision until we opti-mize for both pairwise affinities and multi-mesh consis-tency. Instead of using Semi Definite Programming (SDP)relaxation, we formulate the problem as a low-rank ma-

1

Figure 1: Illustration of Consistent Multi-Image Match-ing

trix recovery problem and employ the nuclear-norm re-laxation for rank minimization. We show that the posi-tive semi-definiteness of a desired solution can be spon-taneously fulfilled. Moreover, we derive a fast alternat-ing minimization algorithm to globally solve the problemin the low-dimensional variable space . Besides validat-ing our method on both simulated and real benchmarkdata sets, we also demonstrate the applicability of the pro-posed method combined with Re-weighted Random Walk(RRWM) [7] graph matching to match meshes with dif-ferent models. We introduce a random walk view on theproblem and propose a robust graph matching algorithmagainst outliers and deformation. Matching between twographs is formulated as node selection on an associationgraph whose nodes represent candidate correspondencesbetween the two graphs. The solution is obtained by sim-ulating random walks with re-weighting jumps enforcingthe matching constraints on the association graph. Thisalgorithm achieves noise-robust graph matching by iter-atively updating and exploiting the confidences of candi-date correspondences. In a practical sense, this RRWMDistortion term is of particular importance since the real-world matching problem is made difficult by the presenceof noise and outliers. Extensive and comparative experi-ments demonstrate that it outperforms the state-of-the-artgraph matching algorithms especially in the presence ofoutliers and deformation.

2 Preliminaries and notation

Suppose we have n models and pi features from eachmodel i. The objective is to find feature correspondencesbetween all pairs of models. Before introducing the pro-

posed method, we first give a brief introduction to pair-wise matching techniques and the definition of cycle con-sistency.

2.1 Pairwise matchingTo match an image pair (i,j), one can compute similaritiesfor all pairs of feature points from two images and storethem in a matrix Sij ∈ Rpi×pj . We represent the featurecorrespondences for mesh pair (i,j) by a partial permuta-tion matrix Xij ∈ 0, 1pi×pj , which satisfies the doublystochastic constraints:

0 ≤ Xij1 ≤ 1, 0 ≤ XTij1 ≤ 1 (1)

To find Xij , we can maximize the inner product be-tween Xij and Sij subject to the constraints in (1) result-ing in a linear assignment problem, which has been wellstudied and can be efficiently solved by the Hungarian al-gorithm. In image matching, spatial rigidity is usuallypreferred, i.e., the relative location between two featuresin an mesh should be similar to that between their cor-respondences in the other mesh. This problem is wellknown as graph matching and formulated as a quadraticassignment problem (QAP). While QAP is NP-hard [2],many efficient algorithms have been proposed to solve itapproximately, e.g. Those solvers basically relax the bi-nary constraint on the permutation matrix, solve the opti-mization, and output the confidence of a candidate matchbeing correct. We refer readers to the related literaturefor details. Here we aim to emphasize that the outputsof graph matching solvers are basically optimized affinityscores of candidate matches, which consider both featuresimilarity and spatial rigidity. We will use these scores(saved in Sij) as our input in some cases.

2.2 Cycle ConsistencyFor any three models (i,j,z), cycle consistency can be de-scribed as

Xij = XizXzj , (2)

and can be extended to the case with more than threemodels.The cycle consistency can be described more con-cisely by introducing a virtual universe that is defined asthe set of unique features that appear in the image collec-tion [13]. Each point in the universe may be observed by

2

several images and the corresponding image points shouldbe matched. In this way, consistent matching should sat-isfy Xij = AiA

Tj , where Ai ∈ {0, 1}pi×k denotes the

map from Image i to the universe, k is the number ofpoints in the universe, and k ≥ pi for all i.

Suppose the correspondences for all m =∑ni=1 pi

features in the image collection is denoted by X ∈{0, 1}m×m and all Ais are concatenated as rows in a ma-trix A ∈ {0, 1}m×k. Then one can write X as

X =

X11 X12 . . . X1n

X21 X22 . . . X2n

......

. . ....

Xn1 Xn2 . . . Xnn

, (3)

X = AAT (4)

From (4), it is clear to see that desiredX should be bothpositive semi-definite and low rank, which effectively im-pose cycle consistency without checking all cycles of pair-wise matches.

X � 0, rank(X) ≤ k (5)

3 Joint Matching via Rank Mini-mization

Given the affinity scores {Sij |1 ≤ i, j ≤ n} for all mod-els, we aim to find globally consistent matches X . Affin-ity scores can be computed from either feature similaritiesor graph matching solvers according to specific scenarios,For our experiments we use Heat Kernel Signature (HKS)feature descriptor to define the affinity score matrices.

3.1 Cost FunctionTo formulate the cost function, we treat the problem asa low rank matrix recovery problem. The inner productbetween Xij and Sij is maximized for all i, j as multiplelinear assignment problems. Alongside, to enforce cycleconsistency, the rank of X is minimized. We make fol-lowing two relaxations on X to optimize the solution:

• X is treated as a real matrix X ∈ [0, 1]m×m in-

stead of a binary matrix. Experimental results show

that the solution values can be stably quantized by athreshold of 0.5.

• Rank of X is replaced by nuclear norm ||X||∗ (sumof singular values), which is a tight convex relaxationproven to be very effective in various low-rank prob-lems such as matrix completion [6].

The estimated X should be sparse since at most onevalue in each row of Xij can be nonzero. To induce spar-sity, we minimize the sum of values in X . Combining allthree terms, we obtain the following cost function:

f(X) = −n∑i=1

n∑j=1

〈Sij , Xij〉+ α 〈1, X〉+ λ||X||∗,

= −〈S − α1, X〉+ λ||X||∗, (6)

where 〈., .〉 denotes the inner product, S ∈ Rm×m is acollection of Sijs, α is is the weight of sparsity, λ controlsthe weight of the nuclear norm.

Besides the doubly stochastic constraints in (1), addi-tional constraints shall be imposed on X after relaxation:

Xii = Ipi , 1 ≤ i ≤ n, (7)

Xij = XTji, 1 ≤ i, j ≤ n, i 6= j, (8)

0 ≤ X ≤ 1, (9)

Finally, we obtain the following optimization problem:

minX〈W,X〉+ λ||X||∗,

s.t.X ∈ C, (10)

where W = α1 − S and C denotes a set of matricessatisfying the constraints given in (1), (7), (8), and (9).

4 Fast Alternating MinimizationThe state-of-the-art methods to solve the convex nuclearnorm minimizatiion in (10) are proximal method [14] orADMM [3] based on iterative single value thresholding[5]. However, singular value decomposition (SVD) needsto be performed in each iteration, which is extremely ex-pensive even for a mediumsized problem.

3

Similar low-rank optimization was solved in [16] for2D images more efficiently via a change of variablesX = ABT , where A,B ∈ Rm×k are new variables witha smaller dimension k < m.

We use the following lowrank factorization-based for-mulation implemented for 2D images in [16] and extendit to 3D mesh based models:

minA,B

⟨W,ABT

⟩+ λ||ABT ||∗,

s.t.ABT ∈ C, (11)

Now rewriting the nuclear norm minimization problemas SDP [15],

||X||∗ = minA,B:ABT=X

1

2

(||A||2F + ||B||2F

)(12)

Finally the following formulation is obtained:

minA,B

⟨W,ABT

⟩+λ

2||A||2F +

λ

2||B||2F ,

s.t.ABT ∈ C, (13)

The selection of matrix dimension k is critical to thesuccess of change of variables, while it directly affectsthe computational complexity.

4.1 Algorithm

We extend the following algorithm described in [16] for2D Images to 3D models. The algorithm applies ADMMto solve (13). The augmented Lagrangian of (13) is asfollows:

£µ(X,A,B, Y ) = 〈W,X〉+ λ

2||A||2F +

λ

2||B||2F

+⟨Y X −ABT

⟩+µ

2||X −ABT ||2F , (14)

where Y is the dual variable and µ is a parameter control-ling the step size in optimization. The ADMM alternatelyupdates each primal variable by minimizing £µ and up-dates the dual variable via gradient ascent while fixing allother variables.

Result: Globally consistent matches Xrandomly initialize A and B, Y = 0 ;W = α1− S ;while not converged do

A←(X + 1

µY)B(BTB + λ

µI)τ

;

B ←(X + 1

µY)A(ATA+ λ

µI)τ

;

X ← Pc

(ABT − 1

µ (W + Y ))

;

Y ← Y k + µ(X −ABT

);

endquantize X with a threshold equal to 0.5;

Algorithm 1: Matching via Alternating Least Squares(MatchALS)

5 RRWM

Introduction to problem: Graph matching is an essen-tial problem in computer vision and machine learning. Inthis paper, we implemented a random walk view on theproblem and use a robust graph matching algorithm [7]against outliers and deformation. Matching between twographs is formulated as node selection on an associationgraph whose nodes represent candidate correspondencesbetween the two graphs. The solution is obtained by sim-ulating random walks with re-weighting jumps enforcingthe matching constraints on the association graph. Ouralgorithm achieves noise-robust graph matching by iter-atively updating and exploiting the confidences of candi-date correspondences. In a practical sense, our work is ofparticular importance since the real-world matching prob-lem is made difficult by the presence of noise and outliers.Extensive and comparative experiments demonstrate thatit outperforms the state-of-the-art graph matching algo-rithms especially in the presence of outliers and deforma-tion.

Recent resurgence of combinatorial optimization ap-proaches to feature matching has changed the situationand firmly settled graph matching formulations based onInteger Quadratic Programming (IQP), which is a gener-alization of the classical graph matching problems. IQPexplicitly takes into consideration both unary and pair-wise terms reflecting the compatibilities in local appear-ance as well as the pair-wise geometric relationships be-tween the matching features. Since IQP is known to be

4

NP-hard, approximate solutions are required. This pro-vides a novel interpretation of graph matching in a ran-dom walk view and relates it to the IQP formulation. In-troducing an association graph constructed with nodes ascandidate correspondences and edges as pair-wise com-patibilities between candidate correspondences, we showthat the search for correspondences between the giventwo graphs can be cast as a node ranking [11] and selec-tion problem in the association graph. For this ranking,we introduce an affinity-preserving random walk and de-rive a ranking based on its quasi-stationary distribution,and prove its equivalence to the spectral relaxation forthe IQP formulation. Then, in this random walk view,we adopt the personalization strategy of Web ranking al-gorithms [8] and propose the re-weighted random walkalgorithm by re-weighting jumps for the graph matchingconstraints. It achieves noise-robust graph matching by si-multaneously updating and exploiting the confidences ofcandidate correspondences. In a practical sense, our workis of particular importance since the real-world matchingproblem is made difficult by the presence of deformationand outliers.

5.1 Problem Formulation

The objective of graph matching is to determine the cor-rect correspondences between two attributed graphsGp =(Vp, Ep, Ap) and Gq = (Vq, Eq, Aq) where V representsa set of nodes, E, edges, and A, attributes. Each nodevPi ∈ V P or edge ePi j ∈ EP has an associated attributevector aPi ∈ AP or aPi j ∈ AP . In feature correspondenceproblems, a node attribute aPi usually describes a local ap-pearance of feature i in model P , and an edge attributeaPi j represents the geometric relationship between fea-tures i and j in the image P . For each pair of edges ePi j ∈EP and eQa b ∈ EQ, there is an affinity or compatibilityWia; jb = f

(aPi , a

Pj , a

Pi j, a

Qa , a

Qb , a

Qa b)

that measuresthe mutual consistency of attributes between the pairsof candidate correspondences

(vPi , v

Qa

)and

(vPj , v

Qb

).

Thus, using a matrix form W , a non-diagonal elementWia; jb contains a pair-wise affinity between two corre-spondences

(vPi , v

Qa

)and

(vPj , v

Qb

), and a diagonal term

Wia; ia represents a uanry affinity of a correspondence(vPi , v

Qa

). Representing the correspondence with an as-

signment or permutation matrix X ∈ {0, 1}nPnQ

is com-mon, such thatXia = 1 implies that node vPi correspondsto node vQa , e.g., feature i in the image P is matched tofeature a in the image Q, and Xia = 0 otherwise. Inthis paper, we denote x ∈ {0, 1}n

PnQ

as a column-wisevectorized replica of X . The graph matching problemcan be formulated as an integer quadratic program (IQP),that is, finding the indicator vector x∗ that maximizes thequadratic score function as follows.

x∗ = argmax(xTWx) (15)

s.t.x ∈ {0, 1}nPnQ

,∀inq∑a=1

xiq ≤ 1,∀anp∑i=1

xia ≤ 1

where the two-way constraints refer to the one-to-onematching from GP to GQ. In general, no efficient al-gorithm exists that can guarantee the optimality boundssince the IQP is NP-hard, thus approximate solutions arerequired.

Figure 2: Association Graphs for Graph Matching byRandom Walks

5.2 AlgorithnmIn the previous affinity-preserving random walks, thematching constraints are ignored and not reflected in therandom walk process. Inducing the matching constraintsonly as a post-processing discretization step like leadsto a weak local optimum. How then can we reflect thetwo-way matching constraints in the affinity-preservingrandom walk? We adopts the personalization approachwidely used in Web ranking methods, which strength-ens the effects of reliable nodes in random walks. Thisis achieved by adopting a jump or teleport in the ran-dom walk: the random walker moves by traversing anedge with probability alpha(α) or by performing a jump

5

to some constrained nodes with probability 1 - α. α rep-resents the bias between the two possible actions, i.e., fol-lowing an edge or jumping. To address the lack of person-alization or users focus, Web ranking algorithms adoptedthis approach in topic-sensitive or query dependent vari-ants. In our formulation, adopting the personalized jump,the probability distribution is updated using the followingequation:

(x(n+1)Tx

(n+1)abs

)= α

(x(n)Tx

(n)abs

)P + (1− α) rT

(16)where a re-weighting jump vector r is added to the

affinity-preserving random walk of. In this approach, weuse the jumps for generating a biased random walk to thematching constraints.

6 Experimental ResultsWe evaluate the performance of the proposed method us-ing SCAPE benchmark data [1]. We use the blended in-trisic maps described in [10] to evaluate the performanceof MatchALS for 3D meshes with RRWM Distortion termadded and compare the matching errors for different mod-els. The size of universe is fixed as 100 points and in eachmesh a random sample of the points are observed with aprobability denoted by ρo. The number of meshes is de-noted by n, which is generic in our code but for this sim-ulation purposes it was kept to 4. Then, the ground-truthpairwise matches are established, and random corruptionsare simulated by removing some true matches and addingsome false matches to achieve an error rate of ρe. Fi-nally, the corrupted permutation matrix is fed into our Al-gorithm as the input affinity scores.

Figure 3 shows the results we obtained before we ap-plied the RRWM distortion term. We observed 1-to-manyfeature correspondences prior to RRWM application, andthis can bee seen by the same color of Figure 3 meshes atthe limbs which give false matches sometimes due to theinherent geometrical symmetry of the object.

Next we demonstrate the results obtained from run-ning the MatchALS for 3D Meshes in Figure 4 using theRRWM Distortion term, and the results can be clearlyseen to weed out the false positive matches in Figure 4.

Given the weight matrix W , the reweight factor α,and the inflation factor β ;Prevent conflicting walks by setting Wia; jb = 0forall conflicting match pairs ;Set the maximum degreedmax = maxia

∑jbWia;jb ;

Initialize the transition matrix P =W/dmax, thestarting probability x as uniform ;while x not converged do

( Affinity-preserving random walking by edges ) ;xT = xTP ;( Reweighting with two-way constraints ) ;yT = exp (βx/maxx) ;while y not converged do

normalize across rows byyai = yai/

∑Ii=1 yai;

normalize across columns byyai = yai/

∑Aq=1 yai;

endy = y/

∑yai ;

( Affinity-preserving random walking withreweighted jumps) ;xT = αxT + (1− α) yT ;x = x/

∑xai ;

endDiscretize x by the matching constraints;

Algorithm 2: Reweighted Random Walk Graph Match-ing

Preciously colored the same, the limbs are now of dif-ferent colors, which signifies that RRWM allows one-to-many matches that appear due to inherent symmetry ofmodels, to be filtered out in a very precise manner.

Figure 5 next shows the 1-to-1 correspondences wehave achieved between one pair of meshes we matchedand the run time for the simulation was 4.53 seconds

Next we demonstrate the matching errors between the4 meshes we obtained. For each pair of possible match-ing the evaluation was performed by calculating the av-erage Geodesic Error. For the SCAPE Data-set [12] pro-vides the ground truth vertex-to-vertex correspondence.To evaluate performance of our algorithm we map everyvertex that has a ground truth correspondence. A geodesicdistance between the predicted correspondence and the

6

(a) Mesh1 matched to Mesh3 (b) Mesh2 matched to Mesh3 (c) Mesh3 matched to Mesh4

Figure 4: Results of MatchALS combined with RRWM

Figure 3: Results of MatchALS performed on 4 Meshesfrom SCAPE Benchmark Dataset

true correspondence is recorded. Then the error per sam-ple is averaged to calculate the Average Geodesic Errorof a mapping between a pair of meshes. Table 1 showsthe evaluation results of the pairwise matching of the 4meshes we used to demonstrate our results.

Mesh1 Mesh2 Mesh3 Mesh4Mesh1 0 1.904% 2.637% 13.909%Mesh2 1.904% 0 6.824% 18.363%Mesh3 2.637% 6.824% 0 8.236%Mesh4 13.909% 18.636% 8.236% 0

Table 1: Average Geodesic Error for matching a pair ofMeshes using MatchALS and RRWM

Figure 5: Illustration of 1 to 1 Mapping of feature corre-spondences in 2 Meshes

7 Conclusion

In this paper, we extended the joint matching algorithmMatchALS for 2D images to 3D mesh matching. Weuse pairwise feature similarities, the heat kernel signaturefeature descriptor, as inputs and obtain accurate matchesby an efficient algorithm (MatchALS) that globally op-timizes for both feature affinities and cycle consistencyof matches. The experiments not only validate the effec-tiveness of the proposed method but also demonstrate thatjoint matching is a promising approach to matching a col-lection of meshes obtained from crowd sourced 3D modelcollections MatchALS is a powerful algorithm that can be

7

extended to 3-D mesh models and the performance gen-erated by the algorithm is almost the same as the state-of-the art algorithm but with exponential orders of incre-ment in speed, as we ourselves observed in our simula-tion. Coupling the MatchALS with Re-weighted RandomWalk Matching is something that has not been done be-fore and our novel implementation of stitching the twoalgorithms has given very fruitful and promise-bearing re-sults applicable to real world 3-D model matching appli-cations .The results reveal that the matching accuracy inthe challenging situations largely depends on the effectiveexploitation of the matching constraints.

References[1] D. Anguelov, P. Srinivasan, D. Koller, S. Thrun,

J. Rodgers, and J. Davis. Scape: shape completionand animation of people. In ACM Transactions onGraphics (TOG), volume 24, pages 408–416. ACM,2005. 6

[2] A. C. Berg, T. L. Berg, and J. Malik. Shape matchingand object recognition using low distortion corre-spondences. In Computer Vision and Pattern Recog-nition, 2005. CVPR 2005. IEEE Computer SocietyConference on, volume 1, pages 26–33. IEEE, 2005.2

[3] S. Boyd, N. Parikh, E. Chu, B. Peleato, and J. Eck-stein. Distributed optimization and statistical learn-ing via the alternating direction method of multipli-ers. Foundations and Trends in Machine Learning,3(1):1–122, 2011. 3

[4] C. Bregler, A. Hertzmann, and H. Biermann. Re-covering non-rigid 3d shape from image streams.In Computer Vision and Pattern Recognition, 2000.Proceedings. IEEE Conference on, volume 2, pages690–696. IEEE, 2000. 1

[5] J.-F. Cai, E. J. Candes, and Z. Shen. A singular valuethresholding algorithm for matrix completion. SIAMJournal on Optimization, 20(4):1956–1982, 2010. 3

[6] E. J. Candes and B. Recht. Exact matrix completionvia convex optimization. Foundations of Computa-tional mathematics, 9(6):717–772, 2009. 3

[7] M. Cho, J. Lee, and K. M. Lee. Reweighted ran-dom walks for graph matching. In Computer Vision–ECCV 2010, pages 492–505. Springer, 2010. 2, 4

[8] T. H. Haveliwala. Topic-sensitive pagerank. InProceedings of the 11th international conference onWorld Wide Web, pages 517–526. ACM, 2002. 5

[9] Q.-X. Huang and L. Guibas. Consistent shape mapsvia semidefinite programming. In Computer Graph-ics Forum, volume 32, pages 177–186. Wiley OnlineLibrary, 2013. 1

[10] V. G. Kim, Y. Lipman, and T. Funkhouser. BlendedIntrinsic Maps. Transactions on Graphics (Proc. ofSIGGRAPH), 30(4), 2011. 6

[11] J. M. Kleinberg. Authoritative sources in a hyper-linked environment. Journal of the ACM (JACM),46(5):604–632, 1999. 5

[12] Y. Lipman and T. Funkhouser. Mobius voting forsurface correspondence. In ACM Transactions onGraphics (TOG), volume 28, page 72. ACM, 2009.6

[13] D. Pachauri, R. Kondor, and V. Singh. Solving themulti-way matching problem by permutation syn-chronization. In Advances in neural informationprocessing systems, pages 1860–1868, 2013. 3

[14] N. Parikh and S. Boyd. Proximal algorithms. Foun-dations and Trends in optimization, 1(3):123–231,2013. 3

[15] B. Recht, M. Fazel, and P. A. Parrilo. Guaran-teed minimum-rank solutions of linear matrix equa-tions via nuclear norm minimization. SIAM review,52(3):471–501, 2010. 4

[16] X. Zhou, M. Zhu, and K. Daniilidis. Multi-imagematching via fast alternating minimization. arXivpreprint arXiv:1505.04845, 2015. 1, 4

8