ARTICLE IN PRESS

0097-8493/$ - se

doi:10.1016/j.ca

�CorrespondE-mail addr

Please cite thi

j.cag.2007.04.

Computers & Graphics ] (]]]]) ]]]–]]]

www.elsevier.com/locate/cag

Calligraphic Interfaces

Classifier combination for sketch-based 3D part retrieval

Suyu Houa, Karthik Ramania,b,�

aPurdue Research and Education Center of Information Sciences in Engineering (PRECISE), School of Mechanical Engineering,

Purdue University, West Lafayette, USAbSchool of Electrical and Computer Engineering (by Courtesy), Purdue University, West Lafayette, USA

Abstract

In this paper, we present a search method with multi-class probability estimates for sketch-based 3D engineering part retrieval. The

purpose of using probabilistic output from classification is to support high-quality part retrieval by motivating user relevance feedback

from a ranked list of top categorical choices. Given a free-hand user sketch, we use an ensemble of classifiers to estimate the likelihood of

the sketch belonging to each predefined category by exploring the strengths of various individual classifiers. Complementary shape

descriptors are used to generate classifiers with probabilistic output using support vector machines (SVM). A weighted linear

combination rule, called adapted minimum classification error (AMCE), is developed to concurrently minimize the classification errors

and the log likelihood errors. Experiments are conducted using our Engineering Shape Benchmark database to evaluate the proposed

combination rule. User studies show that users can easily identify the desired classes and then the parts under the proposed method and

algorithms. Compared with the best individual classifier, the classification accuracy using AMCE increased by 7% for 3D models, and

the average best rank improved by 11.6% for sketches.

r 2007 Elsevier Ltd. All rights reserved.

Keywords: Sketch; Classifier combination; Probability estimate; Relevance feedback; Shape search; Engineering parts; SVM

1. Introduction

Sketch-based 3D model retrieval has received increasingattention in recent years [1,2]. A common method is torepresent the sketch and the projected views of the 3Dmodel by a set of shape descriptors. The system thencomputes the similarity metric between the query and the3D model based on a predefined cost function.

Several limitations exist in the current systems of sketch-based 3D model retrieval. As a common problem incontent-based search (CBS) systems, the retrieval oftenconsists of mixed classes of models that do not share asimilar meaning with the query, thus causing a semanticgap with user expectations. In addition, there is a bigvariance in shape descriptions for different sketches of thesame object because the sketches may be different fromeach other. This is reflected in the observation that manydiverse models are within the top retrievals for query-

e front matter r 2007 Elsevier Ltd. All rights reserved.

g.2007.04.005

ing author. Tel.: +1765 494 5725; fax: +1 765 494 0539.

ess: ramani@purdue.edu (K. Ramani).

s article as: Hou S, Ramani K. Classifier combination for sketc

005

by-sketch. Moreover, the traditional way of relevancefeedback only allows the user to guide the search by givingfeedback through one or several retrieved parts. It is quitepossible that only a few or even none of the retrievalsare wanted by the user given the limited space to showthe thumbnail pictures of the retrievals. In some cases, theretrievals presented to the user are too diverse to allowthe user to discern the desired ones. This is especiallyproblematic when the query is highly dependent on the userinteraction such as query-by-sketch.In this paper, we propose to motivate the user to provide

relevance feedback for a list of part categories obtained bysketch classification. The categorical selections will inturn facilitate user-designated shape similarity retrieval.A strategy to combine the probability estimates frommulti-class classifiers is employed in order to boost theperformance of sketch classification.We use classifiers with probabilistic outputs because 3D

engineering models have a complex scenario. There is nounique criterion to classify engineering models. Even oneengineering model can sometimes be classified into different

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 1. Similar engineering model from different classes.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]2

classes by various standards [3]. For example, the four partsin Fig. 1 have different functions; thus, they belong todifferent part families. However, they can be incorrectlyclassified into the same class by their appearance. Therefore,a traditional binary classifier is not applicable to engineeringmodel classifications. The classifier ought to be able topredict the classification with high accuracy and with noexclusion of possible choices. In addition, sketches arehighly dependent on different user interactions. By enablingthe user to disambiguate the intermediate results driven bythe probabilistic output, the system actively reaches aconsent from the user. Hence, the search engine getsmaximal use of users’ limited interactions for the next stepof searching. Furthermore, we use the probability estimatebecause it is directly applicable to post processing such ascombined estimation. We therefore take advantage of thisproperty to attain a combined estimate so as to improve theconfidence in decision making. The combined classifiertheoretically avoids the worst case and supposedly outper-forms individual classifiers. The notion of classifier combi-nation also implies that the system does not require as muchtraining data as a monolithic classifier in order to reach thesame performance. This is especially useful when thedatabase does not have enough training data, as is oftenseen in real situations.

The contributions of this paper are as follows. First, wepropose and implement a two-stage search method withclassifier combination. The notion of enabling the user tobrowse part classes and then retrieve similar models usingmulti-class probability estimates is applicable to any two-stage search framework. Second, a new combination rulecalled adapted minimum classification error (AMCE) isproposed in Section 4.2. We further define a similaritymeasure for part retrieval by incorporating shape similaritywith probability estimate in Section 3.5. Besides, weaddress the problem of inconsistent view correspondencefor sketch inputs in [4] by introducing a normalizedcertainty measure in Section 3.6. The rest of this paper isorganized as follows. Section 2 introduces the related work.Section 3 presents different modules of work in the systemwork flow. Section 4 gives a detailed description ofclassifier combination. Section 5 discusses the experimentalresults, and the paper concludes in Section 6.

2. Related work

Recent progress in pattern recognition has madeautomatic sketch recognition possible by mapping the

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

shape of the sketch to predefined classes or templates. Oneway to recognize the sketch is to compare it with the shapepatterns embedded in predefined classes through super-vised learning and classification. One of the advantages ofthis method is its simple mathematical interpretation of thepattern from the large set of data extracted from the shape.As a result, each class has a general shape pattern encodedin a mathematical configuration with a limited number ofparameters. However, the supervised learning methodrequires a large set of training examples. In [5], threedifferent approaches were implemented to compute theshape similarity between sketches for sketch recognition:rule-based method, support vector machine (SVM)-basedmethod, and Artificial Neural Network-based method.By viewing sketching as an interactive process, theauthor in [6] suggested using the Hidden Markov Modelto model and recognize freehand sketches. In [7], a SVMclassifier was used to recognize sketched symbols usingZernike moments (ZM). An online scribble recognizercalled CALI was developed in [8,9] using Fuzzy Logic andgeometric features, combined with an extensible set ofheuristics rules. In [10], classifier combination was appliedto sketch symbol recognition using user-defined trainingexamples from the sketch. This method can reach higherclassification accuracy because the sketch query is con-sistent with the training data. However, the idea is notapplicable to 3D engineering parts because engineeringparts are difficult to sketch formally for training purposes.A toolkit called GRANDMA in [11] was developed as atrainable single-stroke recognizer. Other than sketchrecognition, pattern recognition has also been widelyapplied to recognize 2D images and 3D models inparticular. A weighted K-nearest neighbor (KNN) methodwas employed in [12] for engineering part classification.The same group further applied SVM to classify thesame database and demonstrate that SVM has a betterperformance than KNN for their classification problem in[13]. In [14], active learning was employed to facilitatehuman labeling in annotating the 3D models automati-cally. In [15], Bayesian network was used for hierarchicalclassification of 3D models. However, limited effortshave been made to enable the user to disambiguate theclassification results.Classifier combination has recently become a popular

method in various applications of content-based classifica-tion [16–19]. Recent studies in combining multiple classi-fiers for the classification problem have proved that thestrategy of taking advantage of various resources outper-forms traditional monolithic classifiers [20,21]. Among thevarious approaches, research on combining classifierswhich output measurement to represent the likelihood ofthe query belonging to each class has gained attention.Given the measurement output, two groups of combina-tion rules can be divided based on whether an extra processof information extraction is involved. In the first case,output from each classifier is simply combined based onsome rules, such as median or majority vote, product rule,

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 2. System architecture.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]] 3

minimum or maximum [21], Borda count [22], simpleaverage, and the oracle [23]. Their common characteristic isthat the combined measurement is from direct manipula-tion of the individual measurements. In the second case,extra factors are involved in the combination stage besidesthe output from the individual classifiers. Among thevarious approaches, linear combination of classifiers thatoutput measurements [24] has been used often. Linearcombination is simple to implement and has shown asuperior performance over other methods such as majorityvote, product rule, and Borda count from many experi-mental and theoretical studies [21–23,25]. Linear combina-tion rules can be divided into two categories [24]: simpleaverage (SA), which is a linear combination with equalweights; and weighted average (WA), which includes rulessuch as minimum square error (MSE) [26,27] and mini-mum classification error (MCE) [28] to estimate theoptimal weights for the linear combination model. How-ever, it was pointed out in [28] that MSE was derived fromthe regression context rather than from classificationconsiderations. In addition to linear combination, non-linear combination rules such as the minimum entropyapproach [29], Bayesian approach [22], and fusion ofcorrelated probabilities (FCP) [30] consider the correlationof different resources for classifier combination.Thedifference between the nonlinear combination rules in[22,29,30] and the linear combination rules in [26–28] isthat the latter ignores the conditional dependency relation-ship among different classifiers in the process of combinedestimation. However, even though the linear combinationrules ignore the correlations among different resources,they can still reach plausible results at a lower computa-tional cost [21]. These studies in classifier combinationfocus on optimizing the classification accuracy. However,the magnitude of the measurement for the likelihood of thedata being classified to the right class has seldom beenconsidered.

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

3. Sketch-based 3D Part class browsing and retrieval

3.1. System workflow

We use a 3D part retrieval system, ShapeLab [2], and theEngineering Shape Benchmark database (ESB) [3] as thetest bed for this study. The classification criteria of ESB arefrom the methodology developed by Swift and Booker forthe purpose of cost estimation and process planning [31],thus reflecting the engineering semantics in this work. Theidea of classifying a 2D query sketch into a 3D partcategory comes from the notion that (i) engineers usuallyexpress their concept of a 3D shape with three 2Dorthogonal views without losing much information [2];and (ii) the consistency exists between the user sketch of theorthogonal views of the 3D models and the viewsautomatically generated from the 3D models by virtualcontact area (VCA) [2].In this paper, instead of using a conventional single

classifier, we synthesize outputs from individual classifiers,thus improving classification performance and avoidinga biased decision. Fig. 2 presents the system workflow. At Stage I, training data from ESB are used tofinalize the individual classifiers as shown inside the dashedwindow of Fig. 2. Each shape descriptor correspondsto a classifier. Different classifiers which output probabilityestimates of data being classified into a particular classare developed separately using supervised learning.Meanwhile, we exploit the classification output from thetraining data to estimate the optimal weights for theweighted linear combination model. The main idea is thatgiven a classifier, its contribution to the combinedprediction of the testing data is dependent on itsperformance with the training data. A classifier with betterclassification accuracy is considered to have better pre-dication capability and will be given more weight in thecombination model. The process of how to determine the

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 3. GUI of part class browsing and retrieval.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]4

weights using the proposed linear combination rulewill be discussed in Section 4.2. At Stage II, testing datafrom ESB are employed to assess these combinationrules before the rule is applied to the sketch input.The testing data and the sketch pass through the sameprocesses of shape descriptor extraction and the prob-ability estimation before reaching the combination stage.The difference is that the query sketch at Stage III utilizesthe combination rule selected by the testing data at StageII. During the search process at Stage III, the combinedestimate serves two purposes. First, the system presentsa ranked list of top choices of classes for the user tochoose from based on the estimate. Second, the systemincorporates the probability estimate for shape matching inSection 3.5.

3.2. GUI design

Given a query in the form of sketches, we allow the userto browse the possible classes to which the query sketchmay belong driven by the combined probability estimate.In this implementation, the system provides the top 20 outof 42 classes of ESB for the user to browse and select from.Fig. 3 shows the GUI of the implementation. On the left-hand side of the GUI is the ranked list of class images thatprompts the users to choose from. Models of the selectedclass are shown in the order of similarity defined in Section3.5 on the right-hand side. The default images of themodels are the ones belonging to the class that has thehighest probability. The system performs the shapematching within the groups that the user prefers. Theuser is then able to browse groups of similar modelsbased on the selections that he/she thinks to be the rightclasses.

3.3. Sketch acquisition and representation

The sketch acquisition module records users’ searchintent from the sketch. Users can employ the pen or mousein sketching. In our design, the sketches are drawn througha sequence of strokes. Fig. 4 shows the visual appearanceand the architecture of the sketch editor. During thesketching, the system monitors the action of the mouse orpen. Once the mouse cursor is moving and the left button ispressed down, it is assumed that the user has started thesketching. The moving path of the cursor is recorded in realtime, with the end of the stroke indicated by the release ofthe left button.

Each track of stroke S is composed of a sequence ofsmall line segments: S ¼ fððxi; yiÞ; ðxiþ1; yiþ1Þ; tiÞj0pipng,where n is the total number of line segments included insingle stroke S. ðxi; yiÞ and ðxiþ1; yiþ1Þ are the two endpoints of a small line segment at time ti. Consequently,sketching activity A is formed by a sequence of strokeA ¼ fSjj0pjpmg, where m is the number of strokes. In theend, shape descriptors are extracted from A.

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

3.4. Shape descriptor selection

We rely on shape descriptors as the feature vectors forthe classification and the similarity retrieval. Two criteriaare necessary to be met in selecting shape descriptors. First,the shape descriptor has to be applicable both to 2D viewsand the sketch. Second, the shape descriptor has to beinvariant to rotation, translation, and scaling so that theuser has more freedom to sketch.Three shape descriptors are chosen to represent the

shape in this context: 2.5D Spherical Harmonics (2.5D SH)from the contours [32], Fourier Transform (FT) from theouter boundary [33], and the ZM from the region inside theouter boundary [34]. Our experience with sketching hasshown that users prefer to draw the model at a higher levelof abstraction, thus closer to the contour level of the viewgenerated from a 3D model [35]. These three shapedescriptors have been shown empirically to perform wellfor shape matching. Our method is independent of theshape descriptor selected as long as it is in the formof feature vector and satisfies the criteria. However, wechose 2.5D SH, FT and ZM because they complementshape description by perceiving shape from differentperspectives using dissimilar methods. It is expected thata combination of more distinctive classifiers can achieve abetter performance.

3.5. Definition of similarity

The definition of similarity integrates the combinedprobability estimate and the shape similarity in a singleformulation.

Similarityðx;x0Þ ¼ f fShapeðx;x0Þ; pðx 2 oijx;x0 2 oiÞg, (1)

where pðx 2 oijx;x0 2 oiÞ ¼ pðx 2 oijx;x0 2 oiÞ=PJ

j¼1pðx

2 ojjx;x0 2 ojÞ and Shapeðx;x0Þ ¼PK

k¼1Dkðx;x0Þ. In this

equation, x represents the query sketch and x0 representsthe model from selected groups in the database; J is thenumber of groups that the user selects; pðx 2 oijx;x0 2 oiÞ

is the likelihood of x being in the same class as x0; and Dk is

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 4. Sketch acquisition module: (left) sketch editor, (right) information flow.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]] 5

the normalized Euclidean distance between x and x0 usingthe kth designated shape descriptor, which can be any of2.5D SH, FT, or ZM in this case.

There are reasons to further support our choice ofclassifiers with the probabilistic output. The inevitablemisclassification by binary classifier causes a higher risk fora misled search. Shape similarity search will only retrievemodels from the wrong group if the query is misclassified bya binary classifier. The probabilistic output mitigates theundesirable consequences of misclassification. Even thoughthe right class does not have the highest probability, it stillhas a chance. For a good classifier, the probability of theright class is higher than that of most of the incorrect classes.Therefore, shape search still retrieves the desired models as aresult. By associating the probability with the shapesimilarity search, it is equivalent to smoothing the zero/oneloss function from the binary decision, thus avoiding theconsequential risk of a misled search caused by misclassifica-tion. Moreover, we incorporate the probability estimate tothe similarity measure even with user selections. This isbecause a good classifier most likely gives the right choice ahigher probability estimate, thus exerting more share ofsearch to the right group of models. At the same time,selected classes may not have equal preference in the user’smind either. It is possible to let the user rank his/her choices.However, our purpose is to optimize the system performancewith minimal user interactions. In summary, the proposedmethod not only improves the search performance, but alsoenhances the effectiveness of user interaction by involvingonly a limited number of highly possible choices.

3.6. Adaptive view permutation for sketch classification

For our classification problem, each training datum is aconcatenation of the shape descriptors from three separateviews of the 3D model in a certain order. This is because

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

the VCA algorithm generates the views of a 3D model in aconsistent manner. Therefore, data from the same groupshare the same pattern not only in their shape representa-tions but also in their internal data organization. However,when shape generation for a 3D model retrieval isindependent of VCA, such as query by sketch or queryby 2D drawing, the consistency with the training data isnot guaranteed. In our previous work [4], one importantobservation was that the similarity retrieval showed betterperformance when sketches of a 3D model had the samecorrespondence as the views of this model generated byVCA. In order to overcome the inconsistency between thequery sketch and the training data, we propose a methodcalled adaptive view permutation (AVP) to adjust thequery data to a consistent view correspondence with thetraining data.The use of probability estimate implies uncertainty.

Therefore, it is desirable to have a mathematical entity thattakes a probability estimate and returns a value indicatingthe uncertainty of the estimate. In this paper, we employthe classical Shannon entropy in Eq. (2) to measure theuncertainty.

HðPiðxÞÞ ¼ �XC

j¼1

pijðxÞ log2ðpijðxÞÞ; i ¼ 1; . . .L, (2)

where PiðxÞ is the probability estimate for x by the ithclassifier; pijðxÞ is the probability of x being classified in thejth class by the ith classifier; C is the number of classes, andL is the number of classifiers. The uncertainty obtained byEq. (2) has the following characteristics:

(1)

h-ba

HðPiðxÞÞ reaches the maximum when the estimate isuniform, that is when pijðxÞ ¼ 1=C, j ¼ 1; . . .C. In thiscase, we call it total uncertainty, and its certainty factorequals zero.

sed 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESSS. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]6

(ii)

Ple

j.c

HðPiðxÞÞ reaches the minimum of zero when pikðxÞ ¼ 1,and pijðxÞ ¼ 0, j ¼ 1; . . .C, jak. In this case, we call ittotal certainty, and its uncertainty factor is zero.

We normalize the entropy value based on Eq. (3). For eachclassifier, the system keeps a record of rank-of-certainty ofall permutations by evaluating the uncertainty of eachestimate. For each permutation, the system then adds therank-of-certainty from all the classifiers involved. The viewpermutation with the least count entails the highest rank,thus implying that the classifiers have an overall moreconfidence over their estimations. The corresponding datafrom the most promising view permutation is then fed tothe combination stage.

uncertaintyðPiðxÞÞ ¼�PC

j¼1pijðxÞ log2 pijðxÞ

�PC

j¼11=C log2ð1=CÞ,

certaintyðPiðxÞÞ ¼ 1� uncertaintyðPiðxÞÞ. ð3Þ

4. Classifier combination

4.1. Linear combination model

The combination rule applied in this paper linearlyintegrates the probabilistic output from individual classi-fiers. Given a classification scheme O ¼ fo1;o2; . . .oCg of C

classes, and a set of labeled training examples X ¼ fðxi; yiÞ;i ¼ 1; . . .Ng with xi 2 Rn and yi 2 O from a database, acommon goal is to classify a unique example x into aparticular class oi. A simple method is to assign the query tothe class that has the largest confidence measurement fromthe classification prediction. Unlike the traditional binaryclassifier, which hardens confidence measurement into abinary decision, the classifier in this paper normalizes theconfidence measurement into a probabilistic output P ¼

fp1; . . . pj ; . . . pCg withPC

j¼1pj ¼ 1 and pj 2 ½0; 1�.Define D ¼ fD1;D2; . . .DLg to be an ensemble of classi-

fiers with probabilistic output. Each classifier is obtained bysome training data. Let PlðxÞ denote the probabilistic outputfrom the lth classifier Dl for x: PlðxÞ ¼ ðpl1ðxÞ; . . . ; pljðxÞ;. . . ; plCðxÞÞ, where plj ¼ plðy 2 ojjxÞ is the likelihood of x

being classified to class oj by the lth classifier Dl . Let us usethe decision profile (DP) defined in [36] for query x:DPðxÞL�C ¼ ðP1ðxÞ; . . . ;PLðxÞÞ

T where T denotes transpose.In this paper, we use a weighted linear combination modelunder Pcomðx;wÞ ¼ wTDPðxÞ where Pcomðx;wÞ ¼ ðpcom;1ðxÞ;. . . pcom;jðxÞ; . . . ; pcom;CðxÞÞ and pcom;jðxÞ is the probability of x

being classified to the jth class by combining the outputsfrom all classifiers. w ¼ ðw1; . . . ;wLÞ

T is a set of weightsassociated with each classifier. The constraint of

PLl¼1wl ¼ 1

with wl 2 ð0; 1Þ makes the combined estimate satisfy thecondition that

PCj¼1pcom;jðxÞ ¼ 1. Generally, the weights

are estimated by some predefined rules using the trainingoutput. In the next section, the proposed rule to determinethe optimal weights for the linear combination model isexplained.

ase cite this article as: Hou S, Ramani K. Classifier combination for sketc

ag.2007.04.005

4.2. Weight estimation by adapting MCE

Let bðxiÞ ¼ ðb1ðxiÞ; . . . ; bCðxiÞÞ be a C-dimensional classindex vector representing the ideal output for the ithtraining data xi with bkðxiÞ ¼ 1 and bjðxiÞ ¼ 0, whereyi 2 ok, jak, j ¼ 1; . . .C, and i ¼ 1; . . .N. In this paper,we use the k-fold cross-validation to obtain the data forweight estimation. The cross-validation divides the datainto k groups. It uses k � 1 groups for training and theremaining group to test the classifier obtained from thetraining data. This procedure repeats k times for eachconfiguration of training and testing data. The output ofthe testing result at each run is collected for weightestimation.The combination rule in [28] incorporates the MCE

criterion for weight estimation. The discriminant functionof MCE in Eq. (4) measures how likely x is to bemisclassified as another class under the combination rule.From Eq. (4), f ðx;wÞp0 implies a correct decision andf ðx;wÞ40 indicates a misclassification. The system thenobtains the optimal weights by minimizing the overallobjective function derived from the discriminant functionin Eq. (4).

f ðx;wÞ ¼ �f kðxÞ þmaxjak

f jðxÞ; y 2 ok. (4)

The MCE rule is targeted to minimize the overallclassification errors from the training data. However, itmaximizes the measurement of the right class andminimizes the maximal measurement of the wrong classin a manner of equal importance. The minimization of theclassification errors does not necessarily lead to an increasein the measurement of the right class. In reality, it isdesirable that the measurement for the right class is as largeas possible because it is tightly related to the error betweenthe ideal output and the real estimate. Besides, it decidesthe performance of the similarity search in Section 3.6.Therefore, minimization of the log likelihood error is alsonecessary for the optimal weight estimation.

lðx;wÞ ¼ log bkðxÞ � log pcom;kðxÞ ¼ � log pcom;kðxÞ, (5)

where log bkðxÞ ¼ 0 when y 2 ok. Unlike MSE, whichminimizes the summation of errors from the estimates of allclasses, the log likelihood error only pays attention to theerror corresponding to the right class, thus avoiding thedrawback of regression in this context. Obviously, mini-mization of the log likelihood errors has no conflict withthe MCE discriminant function. Therefore, these twoobjectives can be linearly integrated in the same objectivefunction. To unify the target functions, we define the MCEdiscriminant function in our case as in Eqs. (6) and (7)

f kðxÞ ¼ log pcom;kðxÞ, (6)

f ðx;wÞ ¼ � log pcom;kðxÞ þmaxjak

log pcom;jðxÞ, (7)

when y 2 ok. Eq. (7) shares the same property as Eq. (4).Both discriminant functions monotonically decrease as the

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Table 1

Classification accuracy using testing data from ESB

Individual classifiers Combination rules

2.5D SH

contour (%)

Fourier

transform (%)

Zernike

moments (%)

Majority

vote (%)

Product rule

(%)

Simple

average (%)

WA MSE

(%)

WA MCE

(%)

WA AMCE

(%)

68.69 66.38 63.35 71.80 74.50 75.00 73.31 75.00 75.97

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]] 7

measurement for the right class increases. The finaldiscriminant function obtained by integrating the twotargets (AMCE) is presented as

gðxi;wÞ ¼ �ð1þ dÞ log pcom;kðxiÞ þ log maxjak

pcom;jðxiÞ, (8)

where d is a user-defined parameter to leverage thepreference between minimizing the log likelihood errorsand minimizing the classification errors. It is obvious thatthe discriminant function gives more preference to increas-ing the measurement of the right class while preserving theability to minimize the classification error. The sigmoidloss function in Eq. (9) is used to smooth the function inEq. (8). In this paper, x is set to 1.

lðxi;wÞ ¼1

1þ e�xgðxi ;wÞðx40Þ. (9)

Finally, the overall loss from all the training data is used asthe objective function to estimate the weights for the linearcombination model.

w ¼ arg minw

XN

i¼1

XC

k¼1

lkðxi;wÞlðxi 2 okÞ, (10)

withPL

j¼1wj ¼ 1, and wj 2 ð0; 1Þ.

5. Experimental results

5.1. Evaluation of classification performance

We compared the proposed combination rule withselected classifiers using real data from ESB. There are atotal of 856 models in ESB with 55 out of 856 models whichare miscellaneous and do not belong to any of the 42classes. Therefore, there are a total of 801 models groupedin 42 classes in ESB. The size of each group varies. Themaximum size of a group in ESB is 58, while the minimumsize of a group is only 4. Half of the data from each groupare randomly selected as the training data. The averagetraining size from the 42 groups of training data withdifferent sizes is 19.6, with a standard deviation of 14.6,which indicates the complexity of our classificationproblem. The training data from half of the ESB are usedto recognize the classifier first and then to estimate theweights for the linear combination model using AMCE,MSE, or MCE. Testing data from the remainder of thedatabase is then employed to evaluate the quality of the

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

combination rules using the probability estimates fromindividual classifiers.The results from Table 1 show that each combined

classifier using the combination rule outperformed theindividual classifiers. Even the worst combined classifierusing majority vote had over a 3% accuracy increase over thebest individual classifier. SA, WA using MCE, and WAusing AMCE had more competitive performance than othercombination rules across our testing data. Among them, theproposed combination rule AMCE showed the best perfor-mance on our ESB testing data. The product rule alsoshowed good performance in this experiment. However, therisk associated with this method, when one classifier has alarge estimate discrepancy from others, rules it out from ourselection. The fact that the combined classifier outperformseach individual classifier indicates that (1) the system usingthe combined classifier does not require as much trainingdata as a monolithic classifier in order to reach the sameperformance; and (2) the strategy of classifier combinationhelps the system to obtain a better performance when noprior knowledge of the individual classifiers is available.Fig. 5 shows relaxed classification accuracy (RCA) for

the selected classifiers up to the rank of 10 using the testingdata from ESB. RCA for the rank of K is defined asRCAðKÞ ¼ ð1=NÞ

PKi¼1nðiÞ, where nðiÞ is the number of

correct classifications at the ith rank, and N is the totalnumber of the testing data. Solid lines represent RCAunder different combination rules, while the dotted linecomes from individual classifiers. AMCE rule reachesabout 93% at the rank of 5. After the rank of 8, RCAsfrom combined classifiers become stable and approximateto 97% at the rank of 10. At this stage, there is not muchdifference among different combination rules. The outputshows some promising results given the classificationcomplexity in this problem: 42 classes with non-uniformtraining size. The result is consistent with the result from[37], where ASME is employed in example-based partretrieval using multilevel detail (MLD) description. Lowerclassification accuracy may arise when the classifier isapplied to the sketch input. However, the overall perfor-mance boosts our confidence in using the proposed methodfor sketch-based 3D part retrieval.

5.2. User studies

The purpose of this section is to appraise the proposed idea with respect to the system performance for

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 5. Relaxed classification accuracy (RCA) comparisons.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]8

query-by-sketch. Besides, we formally quantify how wellthe proposed combination rule can improve the perfor-mance against single classifiers and other combinationrules. The results assist us in understanding the differencebetween the sketches and the views generated from the 3Dmodel. Besides, the idea of AVP described in Section 3.6can be examined in this section. To obtain an objectiveevaluation, we conducted a user study consisting of twoindependent experiments. In the first experiment, Case I,users were given examples of models from our ESB [38] tosketch. In the second experiment, Case II, engineeringCAD models outside ESB were provided for the user tosketch.

In this experiment, people with no background in theShapeLab system were chosen to participate in thestudy. Users were allowed to take some time to acquaintthemselves with the hardware and the system. During thepractice, users did not encounter any problem. Typicallythey spent several minutes before the real tests began.There was no instruction on how to sketch the 3D objectin particular, for example, the view definition for theorthogonal views (e.g., front, side, or the top view), orthe amount of detail to sketch (e.g., whether to sketch theexternal contour alone or the complete drawing withhidden lines).

Five different users were asked to create a freehandsketch for each example. For each sketch, two groups ofclassifiers were tested based on whether AVP was involvedin the implementation. Each group included three kinds ofclassifiers. The first one was the best individual classifierwhich employs 2.5D SH as the feature vector for trainingand testing. The others were the combined classifiers usingSA and AMCE. Each sketch went through a total of sixclassifiers separately. The users were then asked to give

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

their evaluation to the rank of the right class, as shown bythe class images on the left-hand side of the window. Wethen recorded the ranks for this example from the results ofthese five sketches. In this paper, we use average best rank(ABR), average rank (AR), average worst rank (AWR),and the standard deviation (SD) to evaluate the perfor-mance of each classifier. The ABR means the average of thebest rank given by five user sketches of each example fromall examples involved: averageJ

j¼1ðmin5i¼1rijÞ where rij is therank given to the ith user sketch for the jth example, and J

is the number of examples. AR represents the average of allthe ranks. AWR is the average of the worst rank given byfive user sketches of each example from all examplesinvolved: averageJ

j¼1ðmax5i¼1rijÞ. SD is the standard devia-tion from all the ranks given by the users. A smaller valueof each index indicates a better performance of theclassifier. The overall performance of the selected classifierscan then be evaluated based on the results from sketches ofall the examples.

5.2.1. Sketch examples from ESB

There were a total of 60 user sketches from 12 examplesinvolved in this test. The examples contained a wide varietyof engineering shapes. They were randomly selected fromthe testing data set of ESB. Besides, the sketch inputs weredifferent from the views generated from the trainingexamples. Therefore, it is fair to say that this experimentcan objectively reflect the performance of the system.However, it was expected that the result would be differentfrom the results of the second experiment since theseexamples came from the same database and belong to oneof the 42 classes.Table 2 shows the results using examples from ESB

under Case I and the results using examples outside ESB

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Table 2

Sketch classification under different classifiers with AVP and without AVP using data from ESB (Case I) and outside ESB (Case II)

Overall AR

by example

With AVP by sketch Without AVP by sketch

ABR AR AWR SD ABR AR AWR SD

Case I 2.5D SH 1.67 3.33 4.35 6.17 2.59 3.5 6.72 10.42 3.29

SA 1.5 3 4.35 5.58 2.50 3.25 6.25 9.5 3.14

AMCE 1.5 3.08 4.35 5.58 2.24 3.25 6.17 9.42 3.01

Case II 2.5D SH N/A 7.29 9.69 12.14 4.80 7.71 10.77 14 4.09

SA N/A 6.29 9 11.42 4.79 6.57 10.14 13.14 4.19

AMCE N/A 6.14 9.06 11.28 4.73 6.43 10.11 13.14 4.18

The performance is evaluated by average rank (AR), average best rank (ABR), average worst rank (AWR) and standard deviation (SD).

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]] 9

under Case II. From the results, there is a certain differencebetween the classification of the views generated from 3Dmodels and the classification of the sketches of the models.This is reflected by the difference between the overall ARby example and the ABR by sketch. The reason is becauseof the different creation processes between the trainingviews and the sketches. The strategy of classifier combina-tion helps the system to obtain a better performance forboth cases. The proposed combination rule improves therank over the best individual classifier by 7.5%. However,it is not fully certain if WA with AMCE is better than SAwhen used for sketch classification, although it shows aslightly higher rank regarding some indexes. The results areconsistent with the experimental studies in [24], whichconcludes that SA is as good as WA sometimes in reality,although the author also claims that WA is better than SAtheoretically. We further examined the results and foundthat those sketches that have good classification resultsshare approximately similar shapes to the views of the realexamples generated by VCA, while those having difficultyin having higher ranks are very different from the views ofthe real examples. This in fact explains why the testing datafrom ESB have better classification accuracy. Therefore, ifthe implementation of weight estimation is concerned, it ispreferable to use SA for sketch classification because highquality and dissimilar classifiers can be further inserted intothe combination without making modifications to thecurrent system. However, weighted linear combination hasmore chances to have a better performance based on thetheoretical analysis from [24].

5.2.2. Sketch examples from outside ESB

The purpose of this section is to find out how flexible androbust our system is when the data are outside the range ofour database. Half of these examples did not conceptuallybelong to any of our ESB classes. Some of them were evenhard to sketch based on user experiences. There were atotal of seven examples and therefore 35 user sketches forevaluating the performance of the classifiers. Since therewas no information as to which classes these examplesbelong to, we let the user decide the rank based on thesimilarity between the query and the class images shown on

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

the left-hand side of the window. It is possible that thesame query may belong to multiple classes. Therefore, therank evaluated by the user was the highest rank from thepossible choices given by the system. We calculated theoverall performance for each example following the sameprocedure as in the first experiment. The results in Table 2shows the performance of the selected classifiers underCase II. The results are not as good as those of the firstexperiment, as expected. We further investigated the resultsand found that those examples not belonging to any of theclasses have lower rank evaluations. However, the userscan still find promising categories as they browse the classimages. Similarly, the combination rules improve theclassification performance. AMCE has a 15.5% improve-ment in ABR over the best individual classifier using 2.5DSH. AMCE is slightly better than SA in some indexes.

5.2.3. Sketch classification with AVP

A typical example in Fig. 6 shows the results of threesketches of the same object from different users. Fig. 6(a)was the only case that shared the same correspondence asthe views from the training data generated by VCA.The other two cases had very similar results, as shown inFig. 6(b) and (c), when AVP was used to preprocess thequery. Fig. 6(d) shows the output of the same data as inFig. 6(c) while having no procedure of AVP involved. Theexample illustrates that the system can output consistentresults even though users may have different perspectiveson view permutations. The differences from Fig. 6(a) to6(c) are within expectation because there are alwaysvariations from one sketch to another sketch.Ideally, there should be only one ABR for each kind of

classifier. This is because the best rank comes from the casewhere the sketch has a consistent view correspondence;therefore, no AVP would be involved in such a case.However, there are two different ABRs in Table 2. Thereare several exceptions explaining the differences. First,several examples from outside ESB do not have apparentESB classes to which they belong. Therefore, user opinionmay have effects on the evaluation. Second, the three viewssketched by the user may not share the same viewsgenerated by VCA besides the factor of view correspon-

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 6. Examples of retrievals under different sketches of the same object. (a)–(c) Examples with adaptive view permutation. The right group has ranks of

1, 2, 2, respectively; (d) An example from the same data as Fig. 6(c) without view permutation. The right group is not included in top 4.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]10

dence. This is because the user may not have knowledge oforthographic view generation, which is the central ideabehind VCA. Third, AVP is only a quantitative measure ofconfidence for the probability estimate. It is subject to thequalities of the sketch and the classifiers. We use it only tohelp the system discern the best view correspondenceautomatically. Depending on various conditions involved,it may give a false positive result, which may not have agood classification result. Nevertheless, the use of AVPimproves the performance as shown in Table 2. In additionto the overall improvement over the AR from Case II toCase I, the rank of the worst case is improved significantlyby AVP from all classifiers. We therefore can automaticallyavoid the worst scenario by adding AVP on top of theprobability estimation. The variations of the ranksindicated by SD, with the AVP procedure are smaller thanwithout under Case I. However, the variations of the ranks

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

with the AVP procedure are bigger than without underCase II which may be caused by more user guesses in thissecond experiment.Both experiments demonstrate that the sketch has more

uncertainties compared to the real examples. In fact, query-by-sketch commonly does not have as good retrievals as query-by-example. The goal for our work is to improve the end retrievalby orienteering user feedback at the first stage. Therefore, if wesuccessfully obtain the user feedback for the second stage ofsearch, we still achieve the goal. Fig. 7 shows two examplescomparing the results of sketch-based shape retrieval usingMLD [35] and the proposed method. Both methods allow userrelevance feedback. However, it is apparent that the proposedmethod has more organized retrievals. The experimentalresults support our proposition of providing the user withthe desired choices within a certain range, especially after AVPis added to our latest implementation.

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESS

Fig. 7. Two examples of sketch-based shape retrieval. Left: sketch examples; middle: shape retrieval by the MLD [35]; right: shape retrieval by the

proposed method, the right class is at the first place from the list of choices.

S. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]] 11

6. Conclusions

This paper explored the idea of embedding a multi-classprobability estimate in a 3D part search framework inorder to support fast and effective retrieval. We presented aclassifier combination rule called AMCE to improve boththe classification accuracy and the probability estimate forthe right choice. The use of a classifier with probabilisticoutput and its merits in classifier combination can beapplied to any type of two-stage content-based searchsystem. In addition, an adaptive view permutationprocedure was added to automatically choose the viewpermutation that yields the most confident estimation. Wethen conducted experiments to evaluate the robustness ofthe proposed work using data from ESB and user sketches.Different kinds of classification engines were involved inboth experiments. From the results, it is certain that the useof an ensemble of classifiers improved the classificationaccuracy both for sketches and 3D models. AMCE had a7% greater classification accuracy than the best individualclassifier and 1% greater classification accuracy over thebest combination rules when used for real data from ESB.Besides, AMCE had an overall average 11.6% improve-ment at the best AR over the best individual classifier andshowed a competitive performance over other goodcombination rules when used for sketches. The experi-mental results showed that the system can output the rightclass within a good range, thus enabling the user to choosethe classes for shape matching. The AVP procedure helpedto avoid the worst cases of classification due to inconsistentview correspondence with the training data. In the future,sketch beautification will be integrated into the current

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

system in order to minimize the impact of sketch variationsfor better sketch classification.

Acknowledgments

This material is partly based upon work supported bythe National Science Foundation under Grant IIS No.0535156. Any opinions, findings, and conclusions orrecommendations expressed in this material are those ofthe author(s) and do not necessarily reflect the views of theNational Science Foundation.

References

[1] Funkhouser T, Min P, Kazhdan M, Chen J, Halderman A, Dobkin

D, et al. A search engine for 3D models. Transactions on Graphics

2003;22(1):83–105.

[2] Pu J, Ramani K. A 3D model retrieval method using 2D freehand

sketches. In: Lecture notes in computer science, vol. 3515, 2005.

p. 343–7.

[3] Jayanti S, Iyer N, Kalyanaraman Y, Ramani K. Developing an

engineering shape benchmark for CAD models, Special issue on

shape similarity detection and search for CAD/CAE applications.

Computer Aided Design 2006;38(9):939–53.

[4] Hou S, Ramani K. Sketch-based 3D engineering part class browsing

and retrieval. In: EuroGraphics symposium proceedings on sketch-

based interfaces and modeling, 2006. p. 131–8.

[5] Liu W, Qian W, Xiao R, Jin X. Smart sketchpad an on-line graphics

recognition system. In: Proceedings of sixth international conference

on document analysis and recognition, 2001. p. 1050–4.

[6] Sezgin TM, Davis R. HMM-based efficient sketch recognition. In:

Proceedings of the 10th international conference on intelligent user

interfaces. San Diego, USA, 2001. p. 281–283.

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/

ARTICLE IN PRESSS. Hou, K. Ramani / Computers & Graphics ] (]]]]) ]]]–]]]12

[7] Hse H, Newton AR. Robust sketched symbol recognition using

Zernike moments. In: International conference on pattern recogni-

tion. Cambridge, UK, 2005. p. 367–370.

[8] Fonseca MJ, Pimentel C, Jorge JA. CALI: an online scribble recognizer

for calligraphic interfaces. In: AAAI spring symposium on sketch

understanding, AAAI Technical Report SS-02-08, 2002. p. 51–8.

[9] Fonseca MJ, Jorge JA. Using fuzzy logic to recognize geometric

shapes interactively. In: Proceedings of the 9th IEEE conference on

fuzzy systems, vol. 1, 2000. p. 291–6.

[10] Kara LB, Stahovich TF. An image-based trainable symbol recognizer

for sketch-based interfaces. Computers and Graphics 2005;29(4):

501–17.

[11] Rubine D. Specifying gestures by example. ACM Transaction on

Computer Graphics 1991;25(4):329–37.

[12] Ip CY, Regli WC. Manufacturing processes recognition of machined

mechanical parts using SVMs. In: Proceedings of AAAI 2005, 2005.

p. 1608–9.

[13] Ip CY, Regli WC. Content-based classification of CAD models with

supervised learning. Computer Aided Design and Application

2005;2(5):609–18.

[14] Zhang C, Chen T. A new active learning approach for content-based

information retrieval, IEEE Transactions on Multimedia. Special

Issue on Multimedia Database 2002;4(2):260–8.

[15] Barutcuoglu Z, DeCoro C. Hierarchical shape classification using

Bayesian aggregation. In: Shape modeling international, 2006, p. 44.

[16] Giacinto G, Roli F. Ensembles of neural networks for soft

classification of remote sensing images. In: European symposium

on intelligent techniques, 1997, p. 166–70.

[17] Yan R, Liu Y, Jin R, Hauptmann A. On predicting rare class with

SVM ensemble in scene classification. In: IEEE international

conference on acoustics, speech and signal processing, 2003. p. 21–4.

[18] Senior A. A combination fingerprint classifier. IEEE Transactions on

Pattern Analysis and Machine Intelligence 2001;23(10):1165–74.

[19] Xu L, Krzyzak A, Suen CY. Methods of combining multiple

classifiers and their applications to handwriting recognition. IEEE

Transactions on Systems, Man and Cybernetics 1992;22(3):418–35.

[20] Roli F, Kittler J, Windeatt T., editors. Multiple classifier systems,

Lecture notes in computer science, vol. 3077, 2004.

[21] Kittler J, Hatef M, Duin RP, Matas J. On combining classifiers.

IEEE Transactions on Pattern Analysis and Machine Intelligence

1998;20(3):226–39.

[22] Verikas A, Lipnickas A, Malmqvist K, Bacauskiene M, Gelzinis A.

Soft combination of neural classifiers: a comparative study. Pattern

Recognition Letters 1999;20(4):429–44.

Please cite this article as: Hou S, Ramani K. Classifier combination for sketc

j.cag.2007.04.005

[23] Kuncheva LI. A theoretical study on six classifier fusion strategies.

IEEE Transactions on Pattern Analysis and Machine Intelligence

2002;24(2):281–6.

[24] Famera G, Roli F. A theoretical and experimental analysis of linear

combiners for multiple classifier systems. IEEE Transactions on

Pattern Analysis and Machine Intelligence 2005;27(6):942–56.

[25] Tax DM, van Breukelen M, Duin RP, Kittler J. Combining multiple

classifiers by averaging or multiplying? Pattern Recognition 2000;

33(9):1475–85.

[26] Benediktsson JA, Sveinsson JR, Ersoy OK, Swain PH. Parallel

consensual neural networks. IEEE Transactions on Neural Networks

1997;8(1):54–64.

[27] Hashem S. Optimal linear combination of neural networks. Neural

Networks 1997;10(4):599–614.

[28] Ueda N. Optimal linear combination of neural networks for

improving classification performance. IEEE Transactions on Pattern

Analysis and Machine Intelligence 2000;22(2):207–15.

[29] F. Fouss, M. Saerens, A maximum entropy approach to multiple

classifiers combination. Technical Report, IAG, Universite Catholi-

que de Louvain; 2004.

[30] O’Brien J. Correlated probability fusion for multiple class discrimi-

nation. In: Proceedings of information decision and control.

Adelaide, Australia, 1999. p. 571–7.

[31] Swift K, Booker J. Process selection: from design to manufacture.

New York: Wiley; 1997.

[32] Pu J, Ramani K. On visual similarity based 2D drawing retrieval.

Computer Aided Design 2006;38(3):249–59.

[33] Zhang D, Lu G. Content-based shape retrieval using different shape

descriptors: a comparative study. In: IEEE international conference

on multimedia and expo, 2001. p. 1139–42.

[34] Khotanzad A, Hong YH. Invariant image recognition by Zernike

moments. IEEE Transactions on Pattern Analysis and Machine

Intelligence 1990;12(5):489–97.

[35] Pu J, Jayanti S, Hou S, Ramani K. Similar 3D model retrieval based

on multiple level of detail. In: Pacific conference on computer

graphics and applications, 2006. p. 103–12.

[36] Kuncheva LI. Combining pattern classifiers: methods and algo-

rithms. NJ: Wiley; 2004.

[37] Hou S, Ramani K. A probability-based unified 3D shape search.

In: International conference on semantic and digital media technol-

ogies. Lecture notes in computer science, Berlin: Springer; 2006.

p. 124–37.

[38] Engineering shape benchmark database web demo, hhttp://

shapelab.ecn.purdue.edui.

h-based 3D part retrieval. Computers and Graphics (2007), doi:10.1016/