shape descriptors i

Shape Descriptors IShape Descriptors I

Thomas Funkhouser

CS597D, Fall 2003Princeton University

Thomas Funkhouser


3D Representations

What properties are required for analysis and retrieval?

Intuitive specification Yes No No NoGuaranteed continuity Yes No No NoGuaranteed validity Yes No No NoEfficient boolean operations Yes No No NoEfficient rendering Yes Yes No NoAccurate Yes Yes ? ?Concise ? ? ? YesStructure Yes Yes Yes Yes

Edi

ting

Dis

play

Ana

lysi

s

Ret

riev

al

Property

Shape Analysis Problems

Examples:• Feature detection• Segmentation• Labeling• Registration• MatchingRetrieval• Recognition• Classification• Clustering

“How can we find 3D models best matching a query?”“How can we find 3D models best matching a query?”

1)

2)

3)

4)

Query

Ranked Matches

Shape

Definition from Merriam-Webster’s Dictionary:• a : the visible makeup characteristic of a

particular item or kind of item b : spatial form or contour

Shape

Shape is independent of similarity transformation

(rotation, scale, translation, mirror)

=

Shape Similarity

Need a shape distance function d(A,B) that:• matches our intuitive notion of shape similarity• can be computed robustly and efficiently

Perhaps, shape distance function should be a metric:• Non-negative: d(A,B) 0 for all A and B• Identity: d(A,B) = 0 if and only if

A=B• Symmetry: d(A,B) = d(B,A) for all A

and B• Triangle inequality: d(A,B) + d(B,C) d(A,C)

Example Distance Functions

Lp norm:

Hausdorff distance:

Others (Fréchet, etc.)

pp

ii baBAd1

),(

),(~

),,(~

max),(

minmax),(~

ABdBAdBAd

baBAd iiBbAa

Shape Matching

Compute shape distance function for pair of 3D models• Can matching two objects• Can find most similar object among a small set

Are these the same chair?

Shape Retrieval

Find 3D models with shape most similar to query• Searching large database must take less than O(n)

Is this blue chair in the database?

Shape Retrieval

Build searchable shape index

ShapeRetrieval

SimilarObjects

ShapeIndex

ShapeDescriptor

ShapeAnalysis

ShapeAnalysis

Databaseof

3D Models

GeometricQuery

Shape Retrieval

Find 3D models with shape similar to query

3D Query

3D Database

Best Matches

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to match• Discriminating

3D Query ShapeDescriptor

3D Database

BestMatches

Challenge

Need shape descriptor that is:Concise to store• Quick to compute• Efficient to match• Discriminating

3D Database


BestMatches

Challenge

Need shape descriptor that is:• Concise to storeQuick to compute• Efficient to match• Discriminating

3D Database


BestMatches

Challenge

Need shape descriptor that is:• Concise to store• Quick to computeEfficient to match• Discriminating

3D Database


BestMatches

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to matchDiscriminating

3D Database


BestMatches

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to match• Discriminating Invariant to transformations• Insensitive to noise• Insensitive to topology• Robust to degeneracies

Different Transformations(translation, scale, rotation, mirror)

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to match• Discriminating• Invariant to transformations Insensitive to noise• Insensitive to topology• Robust to degeneracies

Scanned Surface

Image courtesy ofRamamoorthi et al.

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to match• Discriminating• Invariant to transformations• Insensitive to noise Insensitive to topology• Robust to degeneracies

Images courtesy of Viewpoint & Stanford

Different Tessellations

Different Genus

Challenge

Need shape descriptor that is:• Concise to store• Quick to compute• Efficient to match• Discriminating• Invariant to transformations• Insensitive to noise• Insensitive to topologyRobust to degeneracies

Images courtesy of Utah & De Espona

No Bottom!

&*Q?@#A%!

Taxonomy of Shape Descriptors

Structural representations• Skeletons• Part-based methods• Feature-based methods

Statistical representations• Voxels, moments, wavelets, …• Attributes, histograms, ...• Point descriptors




Images courtesy of Amenta & Osada




Image courtesy of De Espona

?




?

Statistical Shape Descriptors

Alignment-dependent• Voxels• Wavelets• Moments• Extended Gaussian

Image• Spherical Extent

Function• Spherical Attribute

Image

Alignment-independent• Shape histograms• Harmonic descriptor• Shape distributions

Feature Vectors

Map shape onto point in multi-dimensional space• Similarity measure is distance in feature space

Feature 2

Fea

ture

1

File cabinets

Tables

Desks

Image courtesy ofMao Chen

Feature Vectors

Cluster, classify, recognize, and retrieve similarfeature vectors using standard methods

Feature 2

Fea

ture

1

File cabinets

Tables

Desks

Image courtesy ofMao Chen

What feature vectors?

Voxels

Use voxel values as feature vector (shape descriptor)• Feature space has N3 dimensions

(one dimension for each voxel)

• d(A,B) = ||A-B||N

Example:

( )d =,

NA B A-B

Voxels

Can store distance transform (DT) in voxels

• ||A-DT(B)||1 represents sum of distances from every point on surface of A to closest point on surface of B

Distance TransformSurface

Image courtesy ofMisha Kazhdan

Voxels

Can build hierarchical search structure• e.g., interior nodes store MIV and MSV

Image courtesy ofDaniel Keim, SIGMOD 1999

Voxel Retrieval Experiment

Test database is Viewpoint household collection1,890 models, 85 classes

153 dining chairs 25 livingroom chairs 16 beds 12 dining tables

8 chests 28 bottles 39 vases 36 end tables

Evaluation Metric

Precision-recall curves• Precision = retrieved_in_class / total_retrieved• Recall = retrieved_in_class / total_in_class

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

Evaluation Metric

Precision-recall curves• Precision = 0 / 0• Recall = 0 / 5

44 55 66

77

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

11 22 33

9988

Ranked Matches

Query

Evaluation Metric


44 55 66

77

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

11 22 33

9988

Ranked Matches

Query

Voxel Retrieval Experiment




Voxel Retrieval Results

0 0.2 0.4 0.6 0.8

0

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

Voxels

Random

Voxels

PropertiesDiscriminating Insensitive to noise Insensitive to topologyRobust to degeneraciesQuick to compute• Efficient to match?X Concise to storeX Invariant to transforms

Wavelets

Define shape with wavelet coefficients

16,000 coefficients 400 coefficients 100 coefficients 20 coefficients

Image courtesy ofJacobs, Finkelstein, & Salesin

Wavelets

Descriptor 1:• Given an NxNxN grid, generate an NxNxN array of

the wavelet coefficients for the standard Haar basis functions

Jacobs, Finkelstein, & SalesinSIGGRAPH 95

Wavelets

Descriptor 1:• Given an NxNxN grid, generate an NxNxN array of

the wavelet coefficients for the standard Haar basis functions

Descriptor 2:• Truncate: Find the m largest coefficients and set

all others equal to zero• Quantize: Set the non-zero coefficients to +1 or –1

depending on their sign


Jackie Chan Example

Original Image (256x256)

Truncated And Quantized to 5000

Truncated 100

Truncated 50

Truncated 10

Torus Example

Torus Truncated to 5000

Wavelets

Distance Function 1:• The query metric is defined by:

where A[i,j,k] and B[i,j,k] are the truncated and quantized coefficients and wi,j,k are weights, fine tuned to the database.

kji

kji kjiBkjiAwBAd,,

,, ,,,,),(


Wavelets

Distance Function 2:• The query metric can be approximated by:

to enable efficient indexing and search.

0),,(:,,

,, ),,,,(),(kjiAkji

kji kjiBkjiAwBAd


Wavelets

Properties Insensitive to noise Insensitive to topologyRobust to degeneraciesQuick to computeEfficient to matchConcise to store• Discriminating?X Invariant to transforms


Moments

Define shape by moments of inertia:

surface

rqppqr dxdydzzyxm

Moments Retrieval Experiment




Moments Retrieval Results

0 0.2 0.4 0.6 0.8

0

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

Voxels

Moments [Elad et al.]

Random

Moments

Properties Insensitive to topologyRobust to degeneraciesQuick to computeEfficient to matchConcise to storeX Insensitive to noiseX Invariant to transformsX Discriminating

Extended Gaussian Image

Define shape with histogram of normal directions• Invertible for convex objects• Spherical function

3D Model EGI

EGI Retrieval Experiment




EGI Retrieval Results

0 0.2 0.4 0.6 0.8

0

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

on

1

Voxels


EGI [Horn 84]

Random

Extended Gaussian Images

Properties Insensitive to topologyQuick to computeEfficient to matchConcise to storeX Insensitve to noiseX Robust to degeneraciesX Invariant to transformsX Discriminating

Other Rotation-Dependent Descriptors

Spherical Extent Functions(Vranic & Saupe, 2000)

Shape Histograms (sectors)(Ankherst, 1999)

Shape Descriptors IIShape Descriptors II

Thomas Funkhouser


Thomas Funkhouser






Image

Alignment-independent• Shape histograms• Harmonic descriptor• Shape distributions

Alignment

Translation (Center of Mass)

Scale (Radial Deviation)

n

iip

nc

1

1

n

iip

ns

1

21

Alignment

Rotation (PCA)• Principal axes are eigenvectors associated with

largest eigenvalues of 2nd order moments covariance matrix

PCAComputation

Principal Axis Alignment

Alignment

Rotation (PCA)• Principal axes are eigenvectors associated with

largest eigenvalues of 2nd order moments covariance matrix

Not very robust!

Alignment

Mirror• PCA does not give directions for principal axes

Need heuristics to determine positive axes!

Alignment-Independent Descriptors

Observation: it is difficult to normalize for differences in rotation and mirroring

Motivation: build a shape descriptor that is invariant to rotations and mirrors and as discriminating as possible

Three mugs aligned automatically with PCA

Shape Histograms

Shape descriptor stores histogram of how much surface resides at different radii from center of mass

Image courtesy of Ankerst et al, 1999

Shape Histograms (shells)(Ankherst, 1999)

Radius

Shape Histograms

Shape descriptor stores histogram of how much surface resides at different radii from center of mass

Image courtesy of Misha Kazhdan

ShapeDescriptor

3D Model SphericalDecomposition

0.7

0.3

0.1

Shape Histogram Experiment




Shape Histogram Retrieval Results

Precision-recall curves (mean for all queries)

0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

onShape Histogram [Ankerst et al.]

EGI [Horn]


Random

1

Shape Histograms

Properties Insensitive to noise Insensitive to topologyRobust to degeneraciesQuick to computeEfficient to matchConcise to store Invariant to rotations• Discriminating?

Harmonic Shape Descriptor

Key idea:• Decompose each sphere into irreducible

set of rotation independent components• Store “how much” of the model resides

in each component

3D Model ShapeDescriptor

HarmonicDecompositions

Step 1: Normalization

Normalize for translation and scale

3D Model

Step 2: Voxelization

Rasterize polygon surfaces into 3D voxel grid

3D Voxel Grid

Step 3: Spherical Decomposition

Intersect with concentric spheres

Spherical Functions

Step 4: Frequency Decomposition

Represent each spherical function as a sum of harmonic frequencies (orders)

Spherical Functions



SphericalFunctionSphericalFunction

Spherical Functions



+ + += …SphericalFunction

Harmonic Decomposition



=

+ + +

+ + +

Constant 1st Order 2nd Order

= …

…

SphericalFunction



=

+ + +

+ + +

Frequency Decomposition

= …

…

SphericalFunction

Amplitudes are invariant to rotation

Step 5: Amplitude Computation

Store “how much” (L2-norm) of the shape resides in each harmonic frequency of each sphere

Frequency Radius


Matching Harmonic Descriptors

Define similarity as L2-distance between descriptors• Enables nearest neighbor indexing and fast search

• Provides lower bound for L2-distance between models

, = -

-

-

-

Sim


PropertiesConcise to store?• Quick to compute?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Invariant to transforms?• Efficient to match?• Discriminating?

Frequency Radius

2048 bytes per model(16 frequencies x 32 radii x 4 bytes)


PropertiesConcise to storeQuick to compute?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Invariant to transforms?• Efficient to match?• Discriminating?

1.6

seco

nd

s (o

n

avera

ge)

Polygons

Voxels

SphericalDecomposition

FrequencyDecomposition

HarmonicShapeDescriptorfrequency radius


PropertiesConcise to storeQuick to compute Insensitive to noise Insensitive to topologyRobust to degeneracies• Invariant to transforms?• Efficient to match?• Discriminating?

Rasterize polygon surfaces(no solid reconstruction)


PropertiesConcise to storeQuick to compute Insensitive to noise Insensitive to topologyRobust to degeneracies Invariant to transforms• Efficient to match?• Discriminating?

RotationMirrorTranslation (w/ normalization)Scale (w/ normalization){


PropertiesConcise to storeQuick to compute Insensitive to noise Insensitive to topologyRobust to degeneracies Invariant to transformsEfficient to match?• Discriminating? 0.0

0.5

1.0

1.5

2.0

0 5000 10000 15000 20000

Database size (models)

Se

arc

h t

ime

(s

ec

s)

IndexedNot In

dexed

0.23 secondsto search

17,500 models


PropertiesConcise to storeQuick to compute Insensitive to noise Insensitive to topologyRobust to degeneracies Invariant to transformsEfficient to match?Discriminating?

Harmonic Matching Results




Harmonic Retrieval Results


0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

onHarmonic Shape Descriptor

Shape Histogram [Ankerst et al.]

EGI [Horn]


Random

1





Image

Alignment-independent• Shape histograms• Harmonic descriptorShape distributions

Shape Distributions

Motivation: general approach to finding a common parameterization for matching

3D SurfaceAudio

2D Contour 3D Volume

Shape Distributions

Key idea: map 3D surfaces to common parameterization

by randomly sampling shape function

3D Models D2 Shape Distributions

Randomlysampleshape

function

SimilarityMeasure

Distance

Distance

Pro

babili

tyPro

babili

ty

Which Shape Function?

Implementation: simple shape functions based on

angles, distances, areas, and volumes

A3(angle)

D1(distance)

[Ankerst 99]

D2(distance)

D3(area)

D4(volume)

D2 Shape Distribution

Properties• Concise to store?• Quick to compute?• Invariant to transforms?• Efficient to match?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Discriminating?


PropertiesConcise to store?Quick to compute?• Invariant to transforms?• Efficient to match?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Discriminating? 512 bytes (64 values)

0.5 seconds (106 samples)

Distance

Pro

babili

ty

Skateboard


PropertiesConcise to storeQuick to compute Invariant to transforms?• Efficient to match?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Discriminating?

TranslationRotationMirror{

Normalized Means

Scale (w/ normalization)

Skateboard Porsche

Distance

Pro

babili

ty

Skateboard


PropertiesConcise to storeQuick to compute Invariant to transformsEfficient to match?• Insensitive to noise?• Insensitive to topology?• Robust to degeneracies?• Discriminating?

Porsche


PropertiesConcise to storeQuick to compute Invariant to transformsEfficient to match Insensitive to noise? Insensitive to topology?Robust to degeneracies?• Discriminating?

1% Noise


PropertiesConcise to storeQuick to compute Invariant to transformsEfficient to match Insensitive to noise Insensitive to topologyRobust to degeneraciesDiscriminating?

D2 Shape Distribution Results

Question• How discriminating are

D2 shape distributions?

Test database• 133 polygonal models• 25 classes

4 Mugs

6 Cars

3 Boats


D2 distributions are different across classes

D2 shape distributions for 15 classes of objects


D2 distributions for 5 tanks (gray) and 6 cars (black)

Distance

Pro

babili

ty


Similarity Matrix• Darkness

representssimilarity

Blocks• Tanks, cars• Airplanes• Humans• Helicopters

al bl btbp bt cr cr cw hr hn lp lg me mg ok pn pe pe re sd sa sp sb te tk

animal

ball

beltblimp

boat

car

chair

claw

helicopter

human

lamp

lightning

missle

mug

openbook

pen

phone

plane

rifle

skateboard

sofa

spaceship

sub

table

tank

al bl btbp bt cr cr cw hr hn lp lg me mg ok pn pe pe re sd sa sp sb te tk

animal

ball

beltblimp

boat

car

chair

claw

helicopter

human

lamp

lightning

missle

mug

openbook

pen

phone

plane

rifle

skateboard

sofa

spaceship

sub

table

tank

D2 Retrieval Experiment




D2 Retrieval Results


0 0.2 0.4 0.6 0.80

0.2

0.4

0.6

0.8

1

Recall

Pre

cisi

onHarmonic Shape Descriptor

D2 Shape Distribution [Osada et al.]

Shape Histogram [Ankerst et al.]

EGI [Horn]


Random

1

Shape Distributions

Next steps:• Better shape functions• Better comparsion methods• Analysis apps


D2 shape distributions for 15 classes of objectsLine Segment

Recognizing gross shapes with D2 distributions




Circle



D2 shape distributions for 15 classes of objectsCylinder




Sphere



D2 shape distributions for 15 classes of objectsTwo Spheres



Statistical representations• Voxels, moments, wavelets, …• Attributes, histograms, ...Point descriptors Next Time!

shape descriptors i

Documents

shape similarityneed

shape retrievalfind

b db

d modelscan

hausdorff distance

kind of item b

similar object

particular item