Computer Vision GroupUniversity of California Berkeley

Shape Matching and Object Recognition using Shape Contexts

Jitendra Malik

U.C. Berkeley

(joint work with S. Belongie, J. Puzicha, G. Mori)

Computer Vision GroupUniversity of California Berkeley

Outline

• Shape matching and isolated object recognition

• Scaling up to general object recognition

Computer Vision GroupUniversity of California Berkeley

Biological Shape

• D’Arcy Thompson: On Growth and Form, 1917– studied transformations between shapes of organisms

Computer Vision GroupUniversity of California Berkeley

Deformable Templates: Related Work

• Fischler & Elschlager (1973)

• Grenander et al. (1991)

• Yuille (1991)

• von der Malsburg (1993)

Computer Vision GroupUniversity of California Berkeley

Matching Framework

• Find correspondences between points on shape

• Estimate transformation

• Measure similarity

model target

...

Computer Vision GroupUniversity of California Berkeley

Comparing Pointsets

Computer Vision GroupUniversity of California Berkeley

Shape ContextCount the number of points inside each bin, e.g.:

Count = 4

Count = 10

...

Compact representation of distribution of points relative to each point

Computer Vision GroupUniversity of California Berkeley

Shape Context

Computer Vision GroupUniversity of California Berkeley

Shape Contexts

• Invariant under translation and scale

• Can be made invariant to rotation by using local tangent orientation frame

• Tolerant to small affine distortion– Log-polar bins make spatial blur proportional to r

Cf. Spin Images (Johnson & Hebert) - range image registration

Computer Vision GroupUniversity of California Berkeley

Comparing Shape Contexts

Compute matching costs using Chi Squared distance:

Recover correspondences by solving linear assignment problem with costs Cij

[Jonker & Volgenant 1987]

Computer Vision GroupUniversity of California Berkeley

Matching Framework

• Find correspondences between points on shape

• Estimate transformation

• Measure similarity

model target

...

Computer Vision GroupUniversity of California Berkeley

• 2D counterpart to cubic spline:

• Minimizes bending energy:

• Solve by inverting linear system

• Can be regularized when data is inexact

Thin Plate Spline Model

Duchon (1977), Meinguet (1979), Wahba (1991)

Computer Vision GroupUniversity of California Berkeley

MatchingExample

model target

Computer Vision GroupUniversity of California Berkeley

Outlier Test Example

Computer Vision GroupUniversity of California Berkeley

Synthetic Test Results

Fish - deformation + noise Fish - deformation + outliers

ICP Shape Context Chui & Rangarajan

Computer Vision GroupUniversity of California Berkeley

Matching Framework

• Find correspondences between points on shape

• Estimate transformation

• Measure similarity

model target

...

Computer Vision GroupUniversity of California Berkeley

Terms in Similarity Score• Shape Context difference

• Local Image appearance difference– orientation– gray-level correlation in Gaussian window– … (many more possible)

• Bending energy

Computer Vision GroupUniversity of California Berkeley

Object Recognition Experiments

• Kimia silhouette dataset

• Handwritten digits

• COIL 3D objects (Nayar-Murase)

• Human body configurations

• Trademarks

Computer Vision GroupUniversity of California Berkeley

Shape Similarity: Kimia dataset

Computer Vision GroupUniversity of California Berkeley

Quantitative Comparison

rank

Num

ber

corr

ect

Computer Vision GroupUniversity of California Berkeley

Handwritten Digit Recognition

• MNIST 60 000: – linear: 12.0%

– 40 PCA+ quad: 3.3%

– 1000 RBF +linear: 3.6%

– K-NN: 5%

– K-NN (deskewed): 2.4%

– K-NN (tangent dist.): 1.1%

– SVM: 1.1%

– LeNet 5: 0.95%

• MNIST 600 000 (distortions): – LeNet 5: 0.8%– SVM: 0.8%– Boosted LeNet 4: 0.7%

• MNIST 20 000: – K-NN, Shape Context

matching: 0.63%

Computer Vision GroupUniversity of California Berkeley

Computer Vision GroupUniversity of California Berkeley

Results: Digit Recognition

1-NN classifier using:Shape context + 0.3 * bending + 1.6 * image appearance

Computer Vision GroupUniversity of California Berkeley

COIL Object Database

Computer Vision GroupUniversity of California Berkeley

Error vs. Number of Views

Computer Vision GroupUniversity of California Berkeley

Prototypes Selected for 2 Categories

Details in Belongie, Malik & Puzicha (NIPS2000)

Computer Vision GroupUniversity of California Berkeley

Editing: K-medoids

• Input: similarity matrix

• Select: K prototypes

• Minimize: mean distance to nearest prototype

• Algorithm: – iterative– split cluster with most errors

• Result: Adaptive distribution of resources (cfr. aspect graphs)

Computer Vision GroupUniversity of California Berkeley

Error vs. Number of Views

Computer Vision GroupUniversity of California Berkeley

Human body configurations

Computer Vision GroupUniversity of California Berkeley

Automatically Locating Keypoints

• User marks keypoints on exemplars

• Find correspondence with test shape

• Transfer keypoint position from exemplar to the test shape.

Computer Vision GroupUniversity of California Berkeley

Results

Computer Vision GroupUniversity of California Berkeley

Trademark Similarity

Computer Vision GroupUniversity of California Berkeley

Outline

• Shape matching and isolated object recognition

• Scaling up to general object recognition– Many objects (Mori, Belongie & Malik, CVPR 01)– Gray scale matching (Berg & Malik, CVPR 01)– Objects in scenes (scanning or segmentation)

Computer Vision GroupUniversity of California Berkeley

Mori, Belongie, Malik (CVPR 01)

• Fast Pruning– Given a query shape, quickly return a shortlist of

candidate matches– Database of known objects will be large: ~30000

• Detailed Matching– Perform computationally expensive comparisons

on only the few shapes in the shortlist

Computer Vision GroupUniversity of California Berkeley

Representative Shape Contexts

• Match using only a few shape contexts– Don’t need to

compare every one

),(minarg

),(),(

2*

*

1

2

ui

jqueryui

ij

query

r

jiquery

SCSCSC

SCSCSSdist

Computer Vision GroupUniversity of California Berkeley

Snodgrass Results

Computer Vision GroupUniversity of California Berkeley

Results

Computer Vision GroupUniversity of California Berkeley

Conclusion

• Introduced new matching algorithm matching based on shape contexts and TPS

• Robust to outliers & noise

• Forms basis of object recognition technique that performs well in a variety of domains using exactly the same algorithm