Part-based representation for the retrieval of 3D graphical models

Alexander G. Agathos

PhD Defense

Department of Product and Systems Design Engineering, University of The Aegean Institute of Informatics and Telecommunications, NCSR “Demokritos”

Contents

• 3D Object Retrieval• Retrieval of 3D Articulated Objects using a graph-based

representation• Segmentation Methodologies

1. Mesh segmentation using feature point and core extraction (Katz et. al. 2005)

2. Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007)

3. A new shape decomposition scheme for graph-based representation (Kim et. al. 2005)

4. A polygonal mesh partitioning algorithm based on protrusion conquest for perceptual 3D shape description (Valette et. al. 2005)

• Recognition by Components • Proposed Segmentation algorithm – Results• Graph-based 3D Retrieval – Results• Innovations and Future work

3D Object Retrieval3D Object Retrieval

3D object retrieval is the process which retrieves from a database of 3D objects those that match best a 3D object query using a measure of similarity

Query

Database

Query result

Retrieval of 3D Articulated Objects using a graph-based representation

Segmentation Graph

Representation

Meaningful partsQuery Model

Retrieval : Matching of the query ARG with the ARGs stored in the Database

Attributed RelationalGraph (ARG)

Segmentation Methodologies

•Region Growing : The segmentation regions are generated with the expansion of seed elements•Watersheds : The segmentation regions are generated by the simulation of flood filling of geographic surfaces•Reeb Graph Method : The segmentation regions are generated with the use of Reeb Graphs•Model based Method : The segmentation regions are generated with the use of a model to simulate the concavities of the mesh•Skeleton based Method : The segmentation regions are generated by the use of the skeleton of the model •Clustering Method : The segmentation regions are extracted using usually the k-means algorithm on the faces of the mesh•Spectral Analysis : The segmentation regions are extracted using spectral analysis on the faces of the mesh•Explicit Boundary Extraction Methods : The segmentation regions are extracted indirectly by finding

first the segmentation boundaries of the mesh•Volumetric Methods : The segmentation regions are extracted using a volumetric function or

volumetric methods•Critical Points : The segmentation regions are extracted using the salient points of the mesh

More meaningful results : The critical points utilize the human ability to distinguish the main particulars of an object

Mesh segmentation using feature point and core extraction (Katz et. al. 2005) (Critical Points Method)

MDS: Transform a mesh into a pose invariant representation

2

2

( )ij iji j

iji j

f d

d

Extraction of Salient Points: ( , ) ( , )i i

i i nS S

Spherical mirroring of the MDS transformed object

2mirror

CR C

C

Part extraction using the minimum cut algorithm

( )(1 )ij ij

ijangW edge

ang W edgew

AVG AVG

A new shape decomposition scheme for graph-based representation (Kim et. al. 2005) (Volumetric Method)

Voxelization

Application of the opening morphological operation using a ball-shaped shape element

This morphologicaloperator clears the

object from protrusions

Body ClassOpening

Branch Class

Complement

A new shape decomposition scheme for graph-based representation

The radius of the sphere is determined by maximizing the

weighted convexity :

A part is further split when its CMD yields at least two parts and each of them has only one adjacent part, it is split.

1

( ( )) ( ( ))

( ) ( ( ( )))

kIi i

wi i

N M k N M kC

N M N M k

H

Initial Decomposition

Recursive Decomposition

A polygonal mesh partitioning algorithm based on protrusion conquest for perceptual 3D shape description

(Valette et. al. 2005)

| ( )C p S p x

Shape Diameter Function : . It is defined as the distance to the antipodal surface using an inward normal direction. In a discrete surface :

Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007)

(Volumetric Method)

f

Recognition by Components

Recognition of a 3D object is achieved by understanding its structure. According to the Recognition by Components (RBC) theory of Biederman human perception understands the structure of the 3D object by breaking it into parts and assigning to them basic volumetric primitives

Specifically when an image of an object is painted on the retina, RBC assumes that a representation of the image is segmented-or parsed-into regions of deep concavity, particularly at cusps where there are discontinuities in curvature. Each segmented region is then approximated by one of a possible set of simple components called geons (for “geometricalions”) that can be modeled by generalized cones.

Proposed Segmentation algorithm

The proposed segmentation methodology is based on the premise that the 3D object consists of a main body (core) and its protrusible parts

Input: A mesh Input: A mesh representing a representing a 3D manifold3D manifold

Salient point Salient point extraction stageextraction stage

Salient points Salient points groupinggrouping

Core Core ApproximationApproximation

Partitioning Partitioning boundary boundary

approximation for approximation for each salient each salient

representativerepresentative

Partitioning Partitioning boundary boundary

refinementrefinement

Until all salient representatives have been addressed

Segmentation Flow Chart (Inspired from the general framework of Lin et. al.

Visual salience-guided mesh decomposition)

Segmentation-Salient Point Extraction Stage

Protrusion function (Hilaga et. al.) : ( ) ( , )p S

pf g p dS

The protrusion function receives low values at the center of the object and

high values at its protrusions

Salient Points : Reside at the extrema of the 3D object. They are extracted by finding the local maxima of the protrusion function

N: Geodesic neighborhood with radius 35·10 · ( )area S

( ) ( ) N

is a salient point

( ) normalized in [0,1]

i i

prot

pf pf

pf t pf

Segmentation-Salient Point Grouping

The computed salient points often belong to sub-parts of the 3D object

Grouping : The salient points that are required to be grouped are those which are close to each other in terms of geodesic distance.

1

1 1

( , )

( 1)

S SN N

i ji j i

SS S

g s s

TN N

: , ( , )i k i j SC s S s C g s s T

Representative salient points : ( ) : ( ) ( ),i i k kRep C s C pf s pf s s C

Segmentation-Core Approximation

Core (main body) approximation : An algorithm which approximates the main body of the object is the one that can acquire all the elements (vertices or faces) of the mesh except those that belong to the protrusions of the mesh.

In Our algorithm : The core approximation is extracted by using the minimum cost paths between the representative salient points.

The core approximation Algorithm expands a set of vertices in ascending order of protrusion function value until the expanded set contains a fixed percentage of all elements of the minimum cost paths (15%).

Segmentation-Core Approximation

Basic philosophy of the core approximation algorithm

All the points of the approximation are kept in a list named CoreListAll the points of the Mesh are inserted in a priority queueA minimum cost path remains active if it contains less than 15% of the points of the corethat belongs to itA salient representative remains active if there exist a minimum cost path to all other representatives that is activeA point can be added to the CoreList if the minimum cost path from the nearest representativesalient point is active

Extract a point from the priority queue

Check if it can be added

Check if a path becomes non active

Check if a salient representative becomes non active

Until all representative salient points become non active

Segmentation-Core Approximation

The Minimum Cost Paths span the protrusible parts. The selection of a percentage of them provides a high confidence that the core points will cover areas of the protrusible parts or being very close to the neighboring areas in which the real boundary is situated.

Partitioning Boundary Detection

The partitioning boundary is the boundary between a protrusion and the main body of the mesh.

Construction of closed boundaries which span the area containing the partitioning and are defined by a distance function D

Partitioning Boundary Detection

Construction of closed boundaries : Iso-contours generated by setting a constant value Dc on a distance function D computed with cost:

( , ) ( , )( , ) (1 )

_ _

length u v prot u vcost u v

avg length avg prot

(1 )int u

( )

( ) ( )cD D u

D D u

Partitioning Boundary Detection

The core approximation has its boundaries near the actual boundaries of the distinct parts of the model. Taking advantage of this an area containing the partitioning boundary can be created.

Construction of closed boundaries in the Arithmetic interval :

1 coremin 2 coremin(1 d ) D ,(1 d ) D

1 2 coremind d DWidth

1 2 coremind d D

per

el

Boundary detection :

1

1

, 1, ,

ii+1 i

i

i per

ii+1 i

i

perif per per

per

r i l

perif per per

per

k maxr rPartitioning boundary approximation :

Partitioning Boundary Detection

The selection of the representative of the group may lead to skewed closed boundaries :

minsminsthe point of the mesh of minimal distance to the salient points of the group

mincthe point of the core with the minimum distance from

min min mind ( , )d s c

thresp the first point p of the minimum cost path from smin to cmin mesh where d(smin,p) > 0.3dmin

threspC the isocontour generated by D passing from pthres

r̂s the point of the constrained mesh with the minimum protrusion function

Partitioning Boundary Refinement

The partitioning boundary approximation is not constrained to the concavities where the true partitioning boundary passes, thus there is a need to refine the partitioning boundary approximation so that it passes also through the concave regions of the 3D object.

[0.9· AvgGeodDist,1.1· AvgGeodDist]Region C=

Partitioning Boundary Refinement

VCthe faces of the dual graph,C the edges of the dual graph

CA CBV ,Vthe faces of C that share a common edge with A, B

V V V V {s,t}

E E (s, ), V (t, ), V E: V , {V V }

uvv v v v e u v

C CA CB

C CA CB C CA CB

1if , s,t

_ ( )1Cap( , )

avg( _ )

otherwise

uv

u vAng Dist

u vAng Dist

_ ( ) (1 cos )uv uvAng Dist n a

Segmentation-Results

Results

Consistent mesh partitioning and skeletonisation using the shape diameter function (Shapira et. al. 2007)

Mesh segmentation using feature point and core extraction (Katz et. al. 2005)

Initial Core

Extended Core

A new shape decomposition scheme for graph-based representation

A polygonal mesh partitioning algorithm based on protrusion conquest for perceptual 3D shape description

(Valette et. al. 2005)

Graph-based 3D Retrieval

Matching

Attributed RelationalGraph (ARG)

Unary attributes : Assigned to the nodes of the graph and express the parts geometrical characteristics.

Binary attributes : Assigned to the edges of the graph and express the relationship between the parts

Matching between two ARGs : Accomplished using the EMD similarity measure

Matching

1

,nv

i i iv w

The EMD measure is used to efficiently express the similarity of two signatures belonging to two different distributions in a feature spaceSignatures : ,

1,

muj j j

u w

The set of weights can be considered as the piles of earth that needs to

be transferred to the holes that the other set of weights create in the feature space

1

nvi i

w

Each unit of earth is transferred from pile i to hole j with cost :( , )i jd v u

Total amount of earth transferred from pile i to hole j :( , )i jfThe EMD measures the minimum amount of work required to transfer the piles of earth to the holes.

( , )

1 1

( , )

1 1

( , )n m

i ji j

i jn m

i j

i j

f d v u

fEMD =

Matching

The two ARGs are considered as the two signatures that need to be matched

Matching

( , , , )G V E UB ˆ ˆ ˆ ˆ ˆ( , , , )G V E UB

i j

i j

i

j

ˆ( , )ˆ3 if v ,v normal

ˆ1 ( , )

ˆ( , )ˆ3 if v ,v fixed

ˆ1 ( , )ˆ( , )

v normal,ˆ( , )5 if

ˆˆ v delete0.1 ( , )

otherwise

normal i j

normal i j

fixed i j

fixed i ji j

delete i j

delete i j

D v v

D v v

D v v

D v vd v v

D v v

D v v

Let and be the two graphs to be matched

Ground distance definition :

2 2

ˆ( , ) j jnormal i j i iD v v u u b b

2

ˆ, ( ) jfixed i j iD v v u u

2

ˆ ˆ( , ) jdelete i j i dD v v u u

Matching-Attribute assignment

The following unary and binary attributes will be used :

1 2[ , , , ]c e eUnary attribute :

Binary attribute : 1 2[ , , ]l a a

[0,1,1,1]Delete node :

Unary attributes defined by Papadakis et. al. descriptor.

In this case no binary attributes are used.

Delete node : Vector with zero entries

Spherical harmonic descriptor of Papadakis et. al.

(2007)

Kim et. al. Descriptor (MPEG7)(2004)

Matching-Papadakis et. al. Descriptor

Retrieval-Experimental Results

The evaluation is based on the McGill Database of articulated objects which contains 255 models consisting of the following classes:

‘Ants’ ‘Ctabs’ ‘Spectacles’ ‘Hands’ ‘Humans’

‘Octopuses’ ‘Pliers’ ‘Snakes’ ‘Spiders’ ‘Teddy bears’

Retrieval-Experimental Results

The goal of the experiments:

• To demonstrate the superior performance of the proposed retrieval methodology against Kim et. al. [2004] and Papadakis et. al. [2008]• Second, the proposed retrieval methodology will be used in order to refine Papadakis et. al. (2008) retrieval results •To demonstrate the efficiency of the proposed mesh segmentation against Kim et. al. [2004] segmentation in terms of retrieval accuracy

Abbreviations

•EMD-PPPT : The proposed retrieval methodology using the attributes of Papadakis et. al. (2007) •EMD-MPEG7 : The proposed retrieval methodology using the attributes of Kim et. al. (2004)•SMPEG7 : Kim et. al. (2004) retrieval methodology using the proposed segmentation algorithm•Hybrid : Papadakis et. al. (2008) retrieval methodology which uses a global descriptor •H-EMD-KIM-R : The refined retrieval methodology which uses Papadakis et. al. (2008) global descriptor and the proposed retrieval methodology using Kim et. al. (2004) attributes•H-EMD-PPPT-R : The refined retrieval methodology which uses Papadakis et. al. (2008) global descriptor and the proposed retrieval methodology using Papadakis et. al. (2008) attributes •MPEG7 : Kim et. al. (2004) retrieval methodology

Retrieval-Experimental Results

Evaluation is done using Precision-Recall (PR) Diagrams and the quantitative measures:• Nearest Neighbor (NN) • First Tier (FT)• Second Tier (ST)• Discounted Cumulative Gain (DCG)

Nearest Neighbor : The percentage of queries where the closest match belongs to the query’s class

First Tier : It measures the proportion of the first |C|-1 retrieved models that Belong to class C

Second Tier : It measures the proportion of the first 2(|C|-1) retrieved models that Belong to class C

Discounted Cumulative Gain : A statistic that weights correct results near the front of the list more than correct results later in the ranked list

Retrieval-Experimental Results

Retrieval-Experimental Results

Retrieval-Experimental Results

Retrieval-Experimental Results

1. The best precision results are those of EMD-PPPT

2. EMD-MPEG7 is the second in precision performance

value3. H-EMD-KIM-R and H-EMD-PPPT-R compared to

Hybrid has been improved by 20% and 24%

approximately respectively in the precision performance

value 4. SMPEG7 retrieval curve is better than the

MPEG7 retrieval curve by an average increase in the precision

recall level of the order of 33%

RETRIEVAL ALGORITHMS

RECALL LEVELS

40% 60%

EMD-PPPT 0.88 0.83

EMD-MPEG7 0.84 0.77

SMPEG7 0.84 0.76

Hybrid 0.71 0.59

H-EMD-KIM-R 0.86 0.76

H-EMD-PPPT-R 0.89 0.80

MPEG7 0.61 0.48

Retrieval-Experimental ResultsClass Method NN(%) FT(%) ST(%) DCG(%)

Complete McGill DB

EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

97.693.391.892.594.1

97.3 97.3

74.169.265.255.770.769.947.5

91.188.978.369.882.975.863.2

93.390.889.185.090.290.579.2

Ants EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

96.796.780.010096.7

96.7 90.0

54.958.557.173.663.4

58.3 62.1

79.779.975.689.283.2

81.5 75.5

88.487.586.794.888.989.287.1

Crabs EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

100100100100100

100 90.0

98.289.872.955.287.5

92.6 45.9

99.898.290.371.892.9

94.3 65.5

99.999.295.988.798.098.682.2

Spectacles EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

10096.096.096.096.0

96.0 84.0

70.363.755.853.574.0

73.8 37.8

99.894.363.763.380.0

80.0 50.8

94.089.282.785.990.591.573.6

Hands EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

95.095.095.090.095.0

95.0 60.0

83.979.778.743.477.4

79.7 30.0

88.988.287.957.683.7

83.9 41.3

95.293.493.077.892.394.063.1

Retrieval-Experimental ResultsHumans EMD-PPPT

EMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

96.696.696.610096.6

96.6 79.3

93.586.884.547.079.6

82.0 40.5

96.499.398.063.885.2

84.7 59.1

98.197.497.383.194.394.677.9

Octopuses EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

88.080.084.056.076.0

88.0 72.0

58.845.242.029.545.7

57.8 46.8

81.873.263.045.071.2

80.3 76.2

88.179.180.568.978.187.077.8

Pliers EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

100100100100100

100 95.0

10085.586.171.692.4

99.7 65.5

10010095.587.999.7

99.7 77.9

10098.697.894.699.099.989.5

Snakes EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

10080.080.080.088.0

96.0 76.0

43.246.244.223.742.3

43.7 36.8

95.285.848.028.747.3

47.3 40.7

84.783.476.662.475.775.469.3

Spiders EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

10010096.8100100

100 90.3

87.285.774.871.585.7

87.3 37.3

10097.386.691.096.9

99.0 61.8

98.497.593.993.797.698.377.8

Retrieval-Experimental Results

Teddy-bears EMD-PPPTEMD-MPEG7SMPEG7HybridH-EMD-KIM-RH-EMD-PPPT-RMPEG7

10085.090.010090.0

100 100

45.342.655.890.354.7

52.6 79.2

63.266.370.898.487.4

87.4 84.5

83.978.884.699.185.589.193.4

Retrieval-Experimental Results

Retrieval-Experimental Results

Innovations and Future Enhancements

• A new segmentation algorithm has been proposedNew algorithm for grouping of pointsNew algorithm for core extractionNew algorithm for partitioning boundary extraction

• A new retrieval algorithm has been proposedNew metric and its use in the EMD algorithm

Future EnhancementsThe arithmetic interval is dependant of Dcoremin.

1 2 coremin(d +d )D

coremincoremin

δ(d1+d2)D =δ d=

2D

The segmentation algorithm can become hierarachical and used also for matching purposes

Publications

Agathos, I. Pratikakis, S. Perantonis, N. Sapidis, and P. Azariadis. 3D mesh segmentation methodologies for CAD applications. Computer-Aided Design and Applications, 4(6):827–841, 2007

A. Agathos, I. Pratikakis, P. Papadakis, S. Perantonis, P. Azariadis, and N. S. Sapidis. Retrieval of 3D articulated objects using a graph-based representation. In Eurographics Workshop on 3D Object Retrieval, pages 29–36, 2009

A. Agathos, I. Pratikakis, S. Perantonis, and N. Sapidis. A protrusion-oriented 3D mesh segmentation. Visual Computer. DOI http://dx.doi.org/10.1007/s00371-009-0383-8

A. Agathos, I. Pratikakis, P. Papadakis, S. Perantonis, P. Azariadis, and N. S. Sapidis. 3D Articulated Object Retrieval using a graph-based representation. Visual Computer, submitted.