June 28, 2005 10:41 WSPC/164-IJIG 00188

International Journal of Image and GraphicsVol. 5, No. 3 (2005) 573–593c© World Scientific Publishing Company

3D FACE RECOGNITION FROM RANGE DATA

GANG PAN and ZHAOHUI WU

Department of Computer Science, Zhejiang UniversityHangzhou, Zhejiang 310027, P.R. China

gpan@cs.zju.edu.cn

Received 8 March 2003Revised 24 December 2003Accepted 10 March 2004

3D facial data has a great potential in overcoming the problems of illumination andpose variation in face recognition. In this paper, we investigate face recognition fromrange data by facial profiles and surface. An efficient symmetry plane detection methodfor facial range data is presented to help extract facial profile. A global profile matchingmethod is then exploited to align and compare the two profiles without detecting fiducial

points that is often unreliable. The central profile and two kinds of horizontal profiles —nose-crossing profile and forehead-crossing profile — are employed in recognition. Foreach individual, a statistical model is built to represent the distinct discriminative capa-bility of the different regions on the facial surface. It is then incorporated into a weighteddistance function to measure for the similarity of surfaces. The comparable experimentalresults are achieved on a facial range data database with 120 individuals.

Keywords: 3D face recognition; surface matching; facial profile; symmetry planedetection.

1. Introduction

The automatic face recognition based on 2D images has been actively researchedduring the past three decades, and various techniques have been presented,1,2 suchas Eigenface,3 Fisherface,4 elastic bunch graph matching5 and Kernel method.6

Great strides have been made in recent years, and the existing methods usually workvery well under well-controlled condition. However, variations in illumination, facialexpression and pose may cause serious degradation in performance for most existingsystems. Despite much effort made for these problems, e.g., modeling illumination,7

SFS based view synthesis8 and employing 3D morphable model to correct the pose,9

robust face recognition is still an uphill task.Recent advances in 3D modeling and digitizing techniques have made the acqui-

sition of 3D human face data much easier and cheaper.10,11 The advantages of 3Drange data are: (i) Explicit representation of 3D shape, (ii) representation of faceshape with real size. Recognition using 3D data has the potential to overcome theseproblems.

573

June 28, 2005 10:41 WSPC/164-IJIG 00188

574 G. Pan & Z. Wu

In this work, we focus on automatic face recognition from range data by facialprofiles and surface. For facial surface, three kinds of profiles (central profile, nose-crossing profile and forehead-crossing profile) and their fusion are explored for recog-nition. The experiments are carried out on a database with 120 individuals, andthe competitive results are obtained.

In the following section we will review the previous work. Section 3 gives therecognition from range data via multiple profiles, and Sec. 4 presents the sur-face matching method based on statistical discriminative model. The experimentalresults are reported in Sec. 5 and conclusions are drawn in Sec. 6.

2. Related Work

2.1. Recognition based on 3D data

The activity to exploit 3D data to improve the accuracy and robustness of facerecognition system is still weakly addressed. Only a few works on the use of 3D datahave been reported. These methods can be categorized into four groups: Methodsbased on curvature analysis, methods by shape representation, methods by modelfitting and image synthesis, and other methods.

Many of the early studies concentrate on curvature analysis.12–16 The seminalwork by Gordon13,14 presents a template-based recognition system using descrip-tors derived from range image. The sensed surface regions are classified as convex,concave and saddle by calculating the minimum and maximum principal curvature,then the locations of nose, eyes, mouth and other features are determined, whichare used for depth template comparison. Lee et al.12 propose a method to detectcorresponding regions in two range images by graph matching based on ExtendedGaussian Image (EGI). An approach to label the components of human faces isproposed by Yacoob et al.15 Its preprocessing stage employs a multistage diffu-sion process to identify convexity and concavity points. These points are groupedinto components. Qualitative reasoning about possible interpretations of the compo-nents is performed, followed by consistency of hypothesized interpretations. Tanakaet al.16 also use the Extended Gaussian Image. For each face, two EGIs are con-structed from maximum principal curvature and minimum principal curvature. TheEGI similarity is measured by Fisher’s spherical correlation. However, because theyare involved in computing curvature, all these techniques require high resolutionof the range data, otherwise the computation of curvature will be inaccurate andunreliable.

References 17–19 attempt to use a shape representation to analyze the 3D facialdata. Chua et al.17 describes a technique for 3D face recognition based on PointSignature — a representation for free-form surfaces, which is also highly dependenton the quality of facial range data. In the method, the rigid parts of the face ofone person are simply extracted to deal with different facial expressions. Theirsubsequent work18 combines Point Signature on 3D range data and Gabor filterresponse on 2D grayscale image for facial feature detection and recognition. Pan

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 575

presents a novel signature Curgram for pose-invariant detection of facial featurefrom range data.19

The third kind of approach is to use model fitting or image synthesis to copewith the influence of illumination and pose. For example, Blanz et al.9,20 utilize a3D morphable model21 to fit the input facial image to tackle variation of pose andillumination. For this approach, the shape and texture fitting procedure is hugelytime-consuming. In Ref. 22, Lee et al. employ an edge model and a color regionmodel to analyze face image, and a wireframe model to synthesize the face imagein virtual view for recognition. And Zhao et al. present a method to synthesizethe virtual image with SFS-based 3D shape recovery.8 For this kind of approach,the input is a 2D face image but not 3D data. Because an image is essentially theprojection from 3D space to 2D space — due to the nature of 2D image, theredifficulties in accurate recognition across pose and illumination.

Other 3D face recognition approaches include those mentioned in Refs. 23–25.Beumier et al.23 propose two 3D comparison methods based on surface and profilesmatching respectively. In Ref. 24, authors do the task of face recognition via featurevector that is generated from depth information of the area in some contour line.Recently Bronstein et al. propose a novel 3D face recognition method.25 It convertsfacial shape and texture to the special images by a bending-invariant mappingscheme, then perform eigenface decomposition on the special images to do therecognition task.

2.2. Recognition from facial profile

For recognition using facial profile, many methods have been performed. Many ofthe previous work carried out depends on fiducial points extracted by heuristicrules.26–31 In Ref. 26 by Harmon et al., the outlines of 256 profile photos are man-ually drawn. Nine fiducial points are selected, and a set of 11 features is derivedfrom these fiducial points. Then two profiles are aligned to be matched by twoselected fiducial marks, and the matching is achieved by measuring the Euclideandistance of the feature vectors derived from the outlines. In Ref. 27 they extendtheir work, where eleven features are reduced to ten. Set partitioning technique isused to reduce the number of candidates to be included in the Euclidean distancemeasure. In their subsequent work,28 11 fiducial points are defined to generate 17-Dfeature vector, and searching space is pruned by thresholding windows.

Kaufman et al.29 develop a face recognition system based on profile silhouettes.The feature vector is a set of normalized autocorrelations expressed in polar coordi-nates, and is classified by a distance weighted k-nearest neighbor rule. Wu et al.30

report a face profile recognition system based on 6 fiducial points. They use a B-spline to extract turning points on the outline curve, and six fiducial points and 24features are derived from these points. The stored features are obtained in a train-ing process that use three profiles per person. Yu et al.31 give a tuning method toget more precise position of the fiducial points. They define a number of small steps

June 28, 2005 10:41 WSPC/164-IJIG 00188

576 G. Pan & Z. Wu

around the determined positions of the fiducial points. For each combination of thenew positions of the fiducial points, the matching is performed. An attributed stringmatching method for profile recognition recently is proposed to tackle the inconsis-tency problem of feature point detection, in which a quadratic penalty function isproposed to prohibit large angle changes and over-merging.32

3. Facial Profile Matching

As described above, most current methods for profile recognition are based onfiducial points, which involved in inconsistency problem of feature point definitionand detection. One solution to this issue is to match the whole profile withoutdetection of fiducial points. In this section we present a robust symmetry planedetection method to extract profile from range data and a global matching approachto align and compare two profiles.

3.1. Detection of the bilateral symmetry plane

Given range data for facial profile recognition, we should first extract the centralprofile from range data. Assume that the 3D facial surface is bilateral symmetri-cal and continuous. Extraction of central profile can be achieved by detecting thebilateral symmetry plane.

Detecting symmetry is a well studied problem in the computer vision area.33–36

However, Shen’s method35 only works with 2D image, Kazhdan’s symmetrydescriptor36 is largely time-cosuming, accurate point correspondences are requiredby Zabrodsky et al.,33 and EGI (Extended Gaussian Image)-based method34 needshigh quality of range data. Cartoux et al.37 proposed an approach which extractsthe profile by looking for the vertical symmetry axis of Gaussian curvature values ofthe facial surface, while computation of Gaussian curvature also needs sufficientlyaccurate range data.

Actually, for a practical 3D face recognition system, in most cases, althoughthe head may turn left, right, upward, downward during acquisition of range data,the rotation is slight and its angles corresponding to three axes are usually lessthan 30◦. Under this condition, we present a simple but effective method usingalignment to robustly extract symmetry plane from the symmetrical object, whichdoes not need computation of curvature.

The idea of our symmetry plane detection algorithm is simple. Suppose thatan initial but inaccurate symmetry plane of facial surface is given, in Fig. 1(a).Its mirrored surface can be easily obtained, as shown in Fig. 1(b), where point A′

in Fig. 1(b) is reflectively symmetrical to point A in Fig. 1(a) with respect to theinitial symmetry plane. Once we align two surfaces accurately, segment AA′ mustbe vertical to the true symmetry plane and its central point must lie on the truesymmetry plane. That is the segment AA′ determines the true symmetry plane.

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 577

(a) (b) (c)

Fig. 1. Finding symmetry plane by alignment. (a) The original facial surface, (b) The mirroredfacial surface with respect to an initial symmetry plane, (c) Alignment of both surfaces.

The bilateral symmetry plane can be formalized as

�n · �x + k = 0, (1)

where �n is the normal vector of the plane and k is a constant. After the alignmentof two surfaces, for point A denoted by �a (ax, ay, az) and point A′ denoted by�a′ (a′

x, a′y, a

′z), the true symmetry plane can be obtained by solving:{

�n = (�a − �a′)/(‖�a − �a′‖)�n · (�a + �a′)/2 + k = 0

. (2)

Considering error from range data acquisition and non-exact symmetry of facialsurface, we solve Eq. (2) for each point in facial surface and perform least squaresmethod38 to get the final solution.

The algorithm we have used for alignment in our system is a variant of ICP(Iterated Closest Point).39 For our needs, we are interested in an algorithm thatoffers the highest possible performance. After some experimentation, we employa scheme similar to that in Ref. 11, which can attain complete alignment in onesecond. ICP is attractive because of its simplicity and its good performance.

3.2. Horizontal profiles

If the facial symmetry plane is available, the central profile can be easily extracted.To utilize more discriminative information in range data, we consider extractingmore “profiles” for recognition. These profiles should be robust and less sensitiveto the facial expression. We choose two horizontal profiles located on nose and fore-head, which we call nose-crossing profile and forehead-crossing profile respectively.

The positions on which these profiles are extracted from different range data ofsame individual should be extremely similar, otherwise the horizontal profiles would

June 28, 2005 10:41 WSPC/164-IJIG 00188

578 G. Pan & Z. Wu

be quite different for same individual. We achieved this by setting nose-crossing pro-file 1.5 cm above nose tip and setting forehead-crossing profile 4.2 cm above nosetip after facial data registration described in Sec. 4.1, since profiles in these posi-tions are less sensitive to facial expression and are easy to determined. Figure 2(a)illustrates the positions of two kinds of horizontal profile. Figures 2(b) and (c) showsamples of extracted nose-crossing profile and forehead-crossing profile.

(a) (b)

(c)

Fig. 2. Horizontal profiles. (a) The positions of two kinds of horizontal profiles, (b) nose-crossingprofile, (c) forehead-crossing profile.

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 579

3.3. Similarity metric for profiles

To be tolerant towards the noise, the function measuring the difference between twoprofiles should be robust enough. It should be insensitive to the small difference andshould better measure the global difference between the profiles.

Hausdorff distance is a distance function between two point sets. Regarding theprofile as a point set, Hausdorff distance between two profiles gives a measure oftheir difference.

Given two sets of points A = {a1, . . . , am} and B = {b1, . . . , bn}, the Hausdorffdistance is defined as

H(A, B) = max(h(A, B), h(B, A)), (3)

where

h(A, B) = maxa∈A

minb∈B

‖a− b‖. (4)

But the Hausdorff distance is very sensitive to even a single “outlying” point of A

or B.40 A generalization of the Hausdorff distance is formalized as the following,which is insensitive to small perturbations of the point sets and allows for smallpositional errors in point sets:

hk(A, B) = ktha∈A

minb∈B

‖a − b‖, (5)

where kth denotes the kth ranked value (or equivalently the percentile of m). Ifuser specifies the fraction f , 0 ≤ f ≤ 1, k can be determined by k = �fm�.

In terms of set containment, hk(A, B) ≤ δ if and only if there is some Ak ⊆A such that Ak ⊆ B, where Ak contains k points of A. Thus we can think ofhk(A, B) as partitioning A into two sets, Ak which is “close to” (within δ of) B andthe “outliers” A − Ak. This is in practice an important aspect of the generalizedHausdorff measure: It separately accounts for perturbations (by distance δ) and foroutliers (by the rank k).40

3.4. Matching by optimization

For alignment of profiles, the dimension of transformation space is three, theyare rotation angle θ and translation vector (tx, ty). Putting the parameters intoa parameter vector �a = (θ, tx, ty) together, given a point x in a profile, the trans-formation is:

T2d(�a; x) = T (θ, tx, ty; x)

=(

cos θ sin θ

− sin θ cos θ

)x +

(txty

). (6)

Thus, the alignment of profile L1 = pi by L0 = qj can be written as:

argmin�a

Hlk(L0, T2d(�a; L1)). (7)

During the optimization process, local minimum may occur in the state space.As a result, many conventional local optimization methods like Newton algorithm

June 28, 2005 10:41 WSPC/164-IJIG 00188

580 G. Pan & Z. Wu

usually cannot converge at the global minimum of the function. Here we use thesimulated annealing method41 to solve this optimization problem.

A good choice of the initial parameter can notably reduce the number of itera-tions needed for the convergence of the simulated annealing. To obtain the initialposition, we use a straight line to fit the profile curve, followed by rotation andtranslation of the profiles so that their centroid and the two lines coincide witheach other.

4. Facial Surface Matching

4.1. Facial data registration

Surface matching based method can be split up into two critical parts: Data align-ment and data comparison. The accuracy of alignment will greatly impact on theresult of following comparison. Although Blanz21 gives a nice solution to register3D facial data, huge time cost makes it hard to be incorporated into a practicalrecognition system.

Assuming that facial range data is not subject to projective scaling, registrationof facial data is an optimization problem in a 3D rigid transformation parameterspace consisting of three degrees for translation and three degrees for rotation.Given two sets of facial range data, probe data S = {s1, . . . , sn} and gallery dataM = {m1, . . . , mk}, the task of 3D registration is to find the rigid transformationwhich will optimally align the regions of S with those of M . The transformationT3d include a rotation around the axes X, Y, Z with angles φ, γ and θ respectively,and a translation t3d. So the result of this transformation of 3D point si is

T3d(si) = RφRγRθsi + t3d. (8)

Alignment error can be measured by a function ε2(|x|), for which, a typicalchoice is to define:

ε2(|x|) = ‖x‖2 = ‖T3d(si) − mψ(i)‖, (9)

where mψ(i) is the corresponding model point for si. Thus, the estimate of theoptimal registration is given by minimizing the error:

T̂3d = argminT3d

∑i

‖T3d(si) − mψ(i)‖

= argminT3d

∑i

‖RφRγRθsi + t3d − mψ(i)‖. (10)

Our system solves this linear numerical problem using ICP. To speed up reg-istration procedure, the following steps are performed to obtain an appropriateinitial position, in which a priori knowledge of the human face and facial featuresis exploited.

(i) A plane is fitted to probe S, and frontal view and back view are detected onthe basis of point distribution on both sides of the plane

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 581

(ii) Approximately estimate the location of nose tip(iii) Translate, rotate according to the parameters obtained by steps 1–2.

When computing the point correspondence for models using nearest neighborrule, we reject the worst 10% pairs based on point-to-point distance. Its purposeis to eliminate outliers which may have a large effect when performing the leastsquares minimization.

4.2. The statistical discriminative model

For recognition using surface matching, different region on facial surface usually hasdifferent discriminative distribution for classification, which is not explicit. Varyingfrom one person to other person, those most discriminative regions are difficult toheuristically define and evaluate. For this reason, we present a point-based statis-tical discriminative model for each subject to describe discriminative capability ofeach point.

Given models (or range data) labelled subject A:

{Ai, i = 0, . . . , m}, Ai = {aik, k = 1, . . . , Na}And models labelled non-subject A:

{Bj, j = 1, . . . , n}, Bj = {bjk}.We assume that {Ai} and {Bj} have to be registered by A0 where each point

aik or bjk corresponds to the point a0k with the same index. Actually, the corre-spondence between each pair of model can be built by the nearest neighbor rule. Wedefine the within-class scatter Sw and between-class scatter Sb for each point in A0:

Skw =

1m + 1

m∑i=0

‖aik − m1k‖, (11)

Skb = ‖m1k − m2k‖, (12)

where mik is the mean given by

m1k =1

m + 1

m∑i=0

aik, (13)

m2k =1n

n∑j=1

bjk. (14)

Therefore, the statistical discriminative model (SDM) for A0 is defined as follows

SDM(A0) ={(

Skw, Sk

b

), k = 1, . . . , Na

}(15)

In which the discriminative capability of point a0k is described by within-classscatter Sk

w and between-class scatter Skb together.

June 28, 2005 10:41 WSPC/164-IJIG 00188

582 G. Pan & Z. Wu

4.3. Similarity measurement for surface

Given two pieces of range data A = {ai}, B = {bi} where each point bj correspondsto the point aj with the same index, and the statistical discriminative model ofSDM(A) =

{(Sk

w, Skb

)}, the weighted directed distance is defined as follows, to

measure similarity from B to A:

Simd(A, B) =1

Na

∑i

Sib

Siw

minb∈B

‖ai − b‖. (16)

Note that the worst 10% pairs are rejected when the correspondences are built.

5. Experimental Evaluation

In this section, we measure the system performance by EER (Equal Error Rate),which means at this location the false acceptance rate and the false rejection rate areequal. Since EER is derived from the ROC curve (Receiver Operator Characteristiccurve), it is not exactly multiples of 1/n, where n is the number of probe samples.

5.1. Facial range database

Our experiments were carried out on the facial range database 3D RMA,23 whichis the biggest 3D face database publicly available.

Each face in 3D RMA is represented by scattered 3D point cloud, obtainedby a 3D acquisition system based on structured light. The database consists of120 individuals and two sessions (session1: Nov 97 and session2: Jan 98). For eachsession, three instances were taken, corresponding to three poses — neutral, limitedup/down and left/right. During acquisition, people sometimes wore their spectacles,and some people smiled. Beards and moustaches also were of presence.

In 3D RMA, two databases were built up from the two sessions: (i) auto-matic DB, reconstructed automatically, 120 individuals; and (ii) manual DB, recon-structed interactively, only the first 30 individuals in alphabetical name order. Wedenote these four datasets as DBs1a, DBs2a, DBs1m and DBs2m, shown in Table 1.

The facial range data in 3D RMA database are of limited quality. Its resolutionis relatively low, since there are only about 3000 points in each face model, com-pared with the experimental data in other literature, e.g. still having 75972 points

Table 1. The four datasets in 3D RMA.

Session 1 Session 2(3 instances/person) (3 instances/person)

Automatic DBreconstructed automatically DBs1a DBs2a(120 persons)

Manual DBreconstructed manually DBs1m DBs2m(the first 30 persons out of 120)

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 583

Fig. 3. Four samples of facial range data for our experiment, two views for each model.

after model simplification in Ref. 9, 128 × 512 range image in Ref. 18, 640 × 480range image in Ref. 25. Additionally, some facial features are often incomplete, likenose, eye.

In our implementation, the points below the chin are cropped manually. Figure 3shows several examples, two views for each.

5.2. Time cost

We implemented the proposed system on the Pentium IV 2.0GHz. The mostlytime-consuming stages are symmetry plane detection, facial profile matching andfacial surface registration. Their average computational time is shown in Table 2.

Table 2. Average computational time for three stages.

Stage Average computational time

Symmetry plane detection 0.96 sMatching two profiles 2.17 sFacial surface registration 0.93 s

June 28, 2005 10:41 WSPC/164-IJIG 00188

584 G. Pan & Z. Wu

Fig. 4. Some results of symmetry plane detection carried out on 3D RMA database. Models inthe first row are the initial position for symmetry plane detection and models in the second roware the corresponding detection results. The symmetry plane is marked by the white line.

5.3. Results of symmetry plane detection

Figure 4 shows some results of the symmetry plane detection on 3D RMA database.The top row illustrates the models with the initial position for symmetry planedetection, and the bottom row exhibits their detection results, where symmetryplane is marked by the white line. Some models are obviously incomplete, yet thealgorithm found the bilateral symmetry plane correctly.

5.4. Results by profile matching

Examples of the extracted central profiles are shown in Fig. 5. The facial data inthe first row in Fig. 5(a) is seriously incomplete around eye regions. The second rowin Fig. 5(a) is a sample whose data on the right side obviously less than that on theleft side. The third row in Fig. 5(a) shows a sample in depth rotation. In these cases,the central profiles are all successfully extracted by our symmetry planes detectionalgorithm.

Figure 6 demonstrates the initial position and the final position converged dur-ing profile matching for the three kinds of profiles. ROC curves by central profilematching carried out on DBs1m and DBs1a are shown in Fig. 7. To evaluate theeffect of reconstruction error, we use the first 30 persons in DBs1a, where the indi-viduals are similar to DBs1m. The equal error rate on DBs1m is 2.22%, while thaton DBs1a is 5.56%.

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 585

(a) (b) (c) (d)

Fig. 5. Profiles extracted by the symmetry plane detection algorithm. (a) the input model aftertriangle-based linear interpolation of the original range data, and its initial symmetry planedenoted by a gray line, (b) the mirrored model after aligning, (c) the symmetry plane detected,(d) the profile extracted.

To determine the percentile parameter k in the partial Hausdorff distance, exper-iment with different k was conducted. Figure 8 suggests that the best result isachieved around k = 0.8.

EER by our global profile matching on different data sets of 3D RMA are demon-strated in Table 3. Results of the robust profile recognition method presented byYu31 are also reported.

June 28, 2005 10:41 WSPC/164-IJIG 00188

586 G. Pan & Z. Wu

Fig. 6. Profiles at the initial position and the matching result.

0 10 20 30 40 500

2.22

5.56

10

20

30

40

50

False Acceptance Rate(%)

Fal

se R

ejec

tion

Rat

e(%

)

Receiver Operating Characteristic

Manual DBAutomatic DB

Fig. 7. ROC curves of central profile matching for DBs1m and DBs1a (30 persons).

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 587

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 10

2

4

6

8

10

12

14

16

18

20

k

EE

R (

%)

DBs1aDBs1mDBs2aDBs2m

Fig. 8. EERs of central profile with different percentile of k-Hausdorff distance performed on thefour data sets (only 30 persons).

5.5. Results by 2D methods

Evaluation of the recognition performances from range data by the appearance-based face recognition methods is given here. The 2D appearance-based methodscannot be directly used in 3D data. A simple scheme is to convert the range datato 2D depth image in frontal view as the input. Since the range data in 3D RMAis described by point cloud, generation of 2D depth image needs interpolation.

Delaunay triangle-based linear interpolation is employed to convert the facialrange data to the 2D depth image. Firstly, the irregular range data is Delaunaytriangulated. Then, for each regular grid point, the closest triangle is selected, andthe depth value of the regular point are computed by triangle-based linear interpo-lation of the depth values at the three vertices.42 Several samples after interpolationare shown in Fig. 9.

To make a comparison, we implement three appearance-based face recognitionalgorithms, Eigenface (reduced to 60 principal components),3 Fisherface (reducedto 29 dimension for manual DB and to 60 dimension for automatic DB),4 KernelFisherface (reduced to 29 dimensions for manual DB and to 60 dimension for auto-matic DB, polynomial kernel).6 The distance metric in reduced subspace is L2.Results are shown in Table 3.

5.6. Results by surface matching

Results by surface matching are shown in Table 3. The best EER performance is3.33% obtained on DBs1m. When carried out on 120-person data set DBs1a and

June 28, 2005 10:41 WSPC/164-IJIG 00188

588 G. Pan & Z. Wu

Table 3. Error rates on 3D RMA.

Data set DBs1m (30 × 3) DBs2m (30 × 3) DBs1m + s2m (30 × 6)(persons × instances/person) DBs1a (120 × 3) DBs2a (120 × 3) DBs1a + s2a (120 × 6)

Central profile by Yu31 6.67% 8.89% 13.33%14.62% 15.04% 18.95%

Proposed central 2.22% 4.44% 6.67%profile matching 9.83% 10.83% 13.76%

Nose-crossing profile 8.89% 11.11% 12.78%12.96% 15.72% 16.71%

Forehead-crossing profile 13.33% 13.33% 15.0%16.67% 18.05% 20.38%

Eigenface (60 PCs)3 5.56% 6.67% 7.78%17.47% 19.31% 22.03%

Fisherface (29/60)4 4.44% 6.67% 7.78%18.26% 19.54% 15.83%

Kernel Fisherface6 4.44% 5.56% 7.78%(29/60, polynomial kernel) 16.11% 17.97% 15.09%

Gordon91,13 but facial 13.33% 15.56% 19.44%landmarks labelled manually — — —

Beumier0023 8.0% 7.0% 9.5%— — —

Our SDM-based method 3.33% 5.56% 6.67%6.73% 6.94% 8.79%

Fusion of profile and surface 1.11% 2.22% 4.44%5.64% 6.15% 7.93%

Fig. 9. 2D depth images by triangle-based linear interpolation.

DBs2a, the proposed method still has a low EER, exceeding all other methods listedin Table 3.

Experimental result by seminal approach of Gordon13 is conducted and reportedin Table 3. Because of the low resolution of the range data and in the presence ofnoise, Gordon’s method cannot correctly detect facial features like eyelid, eyeball,corners of eye. We have to manually label these facial features for this method. Onlythe data in manual DB is labelled. The EER on manual DB has already shown itsinferiority.

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 589

The performance of Beumier’s surface matching method23 is also listed here,which is from Beumier’s paper. Result on DBs1a and DBs2a is not reported inliterature.23

5.7. Results for fusion

The main motivation of the profile analysis is to access the complementary informa-tion in range data to be combined with the surface analysis in order to improve therecognition performance and robustness of the system. The performance of variousdecision level fusion schemes will depend on the nature of input data. We conductedthe experiment with several commonly used fusion schemes, including rules of Max,Min, Sum, Product, Median and Majority Vote,43 to make fusion of the four experts(three profile experts and one surface expert). The SUM rule achieves the best per-formance, and is chosen in our approach. The result by SUM rule is demonstratedin Table 3. Error rates on 3D RMA are obviously decreased after fusion. It alsoshows that there is compensation in discrimination between surface — the globalmeasure of shape, and profile — the partial sampling of shape.

6. Conclusions

This paper has investigated face recognition from range data by facial profile andsurface matching. A simple but effective symmetry plane detection method is pre-sented to extract the central profile. And a robust global profile matching methodis developed, discarding detection of feature points on the profile. The authors alsohave explored the two horizontal profiles for recognition. To describe discriminativecapability of different regions in facial surface, a statistical discriminative model isproposed for surface-based recognition. Their effectiveness has been demonstratedby comparable experimental results.

The proposed methods are carried out on the 3D RMA, a facial range datadatabase with limited quality. The experimental results show that the proposedsurface matching and profile matching are competitive, and outperform severalexcellent appearance-based face recognition methods. The fusion result gives aexperimental proof for the presence of discriminative compensation between surfaceand profile.

Recognizing faces across pose and illumination still a hard problem. The poseand illumination variation are two challenges in 2D image-based face recognition.44

Recognizing from 3D data is likely to solve the two problems. Despite variant ofhead pose, the viewpoint is more easily recovered from range data rather thanfrom 2D images. Moreover, if acquisition of range data is not considered, lightingcondition has no effect on our approach. 3D Face recognition from range data ispromising.

In the future, experiments on a more accurate database are expected. It isalso important to characterize the noise in range data produced by different 3Dacquisition systems like laser range scanners and photometric stereo techniques,

June 28, 2005 10:41 WSPC/164-IJIG 00188

590 G. Pan & Z. Wu

and to study the effect of the noise on the face recognition algorithms. Tacklingfacial expression problem is another work in the future.

Acknowledgments

The authors are grateful for the grants from the National Science Foundation ofChina (60273059), National 863 High-Tech Programme (2001AA4180), and Zhe-jiang Provincial Natural Science Foundation for Outstanding Young Scientist ofChina (RC01058). They would like to thank Signal and Image Center at Royal Mil-itary Academy of Belgium and Dr. Charles Beumier for providing 3D RMA facialrange database, and we thank the anonymous reviewers for their valuable commentsin the improvement of this manuscript.

References

1. R. Chellappa, C. L. Wilson and S. Sirohey, “Human and machine recognition of faces:A survey,” Proc. IEEE 83(5), 705–740 (1995).

2. W. Zhao and R. Chellappa, “Face recognition: A literature survey,” University ofMaryland, Tech. Rep. CS-TR4167 (2002).

3. M. Turk and A. Pentland, “Eigenfaces for recognition,” Journal of Cognitive Neuro-science 3(1), 71–86 (1991).

4. P. N. Belhumeur, J. P. Hespanha and D. J. Kriegman, “Eigenfaces vs. fisher-faces: Recognition using class specific linear projection,” IEEE Trans. Pattern Anal.Machine Intell. 19(7), 711–720 (1997).

5. L. Wiskott, J.-M. Fellous and N. Kruger, “Face recognition by elastic bunch graphmatching,” IEEE Trans. Pattern Anal. Machine Intell. 19(7), 775–779 (1997).

6. M.-H. Yang, “Kernel eigenfaces vs. kernel fisherfaces: Face recognition using kernelmethods,” in Proc. IEEE International Conference on Automatic Face and GestureRecognition, pp. 215–220 (2002).

7. A. S. Georghiades, P. N. Belhumeur and D. J. Kriegman, “From few to many: Illu-mination cone models for face recognition under variable lighting and pose,” IEEETrans. Pattern Anal. Machine Intell. 23(6), 643–660 (2001).

8. W. Zhao and R. Chellappa, “Illumination-insensitive face recognition using symmetricshape-from-shading,” in Proc. IEEE International Conference on Computer Vision,pp. 1286–1293 (2000).

9. V. Blanz, S. Romdhani and T. Vetter, “Face identification across different poses andillumination with a 3D morphable model,” in Proc. IEEE International Conferenceon Automatic Face and Gesture Recognition, pp. 202–207 (2002).

10. O. Hall-Holt and S. Rusinkiewics, “Stripe boundary codes for real-time structured-light range scanning of moving objects,” in Proc. IEEE International Conference onComputer Vision (2001).

11. S. Rusinkiewicz, O. Hall-Holt and M. Levoy, “Real-time 3D model acquisition,”in Computer Graphics Proceedings SIGGRAPH 2002, pp. 438–446.

12. J. C. Lee and E. Milios, “Matching range images of human faces,” in Proc. IEEEInternational Conference on Computer Vision, pp. 722–726 (1990).

13. G. G. Gordon, “Face recognition based on depth maps and surface curvature,” inGeometric Methods in Computer Vision, SPIE Proceedings, 1570, 234–247 (1991).

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 591

14. G. G. Gordon, “Face recognition based on depth and curvature features,” in Proc.IEEE Computer Society Conference on Computer Vision and Pattern Recognition,pp. 808–810 (June 1992).

15. Y. Yacoob and L. S. Davis, “Labeling of human face components from range data,”CVGIP: Image Understanding 60(2), 168–178 (September 1994).

16. H. T. Tanaka, M. Ikeda and H. Chiaki, “Curvature-based face surface recognitionusing spherical correlation — principal direcions for curved object recognition,” inProceedings of IEEE International Conference on Automatic Face and Gesture Recog-nition, pp. 372–377 (1998).

17. C. S. Chua, F. Han and Y. K. Ho, “3D human face recognition using point signature,”in Proc. IEEE International Conference on Automatic Face and Gesture Recognition,pp. 233–238 (March 2000).

18. Y. Wang, C.-S. Chua and Y.-K. Ho, “Facial feature detection and face recognitionfrom 2D and 3D images,” Pattern Recognition Letters 23, 1191–1202 (2002).

19. G. Pan, Y. Wang and Z. Wu, “Pose-invariant detection of facial features from rangedata,” in Proceedings of IEEE International Conference on Systems, Man, and Cyber-netics, Hyatt Regency, Washington D.C., pp. 4171–4175 (2003).

20. V. Blanz and T. Vetter, “Face recognition based on fitting a 3D morphable model,”IEEE Transactions on Pattern Analysis and Machine Intelligence 25(9), 1063–1074(2003).

21. V. Blanz and T. Vetter, “A morphable model for the synthesis of 3D faces,” in Com-puter Graphics Proceedings SIGGRAPH’99, pp. 187–194 (1999).

22. M. W. Lee and S. Ranganath, “Pose-invariant face recognition using a 3D deformablemodel,” Pattern Recognition 36(8), 1835–1846 (2003).

23. C. Beumier and M. Acheroy, “Automatic 3D face authentication,” Image and VisionComputing 18(4), 315–321 (2000).

24. Y. Lee, K. Park, J. Shim and T. Yi, “3D face recognition using statistical multiplefeatures for the local depth information,” in Proc. 16th International Conference onVision Interface, Halifax, Canada (2003).

25. A. M. Bronstein, M. M. Bronstein and R. Kimmel, “Expression-invariant 3D facerecognition,” in Proceedings of International Conference on Audio- and Video-basedBiometric Person Authentication, Lecture Notes in Computer Science 2688, 62–70(2003).

26. L. D. Harmon and W. F. Hunt, “Automatic recognition of human face profiles,”Computer Graphics Image Processing 6 (1977).

27. L. D. Harmon, S. C. Kuo, P. F. Raming and U. Raudivi, “Identification of humanface profiles by computer,” Pattern Recognition 10, 301–312 (1978).

28. L. D. Harmon, M. K. Khan, R. Larsch and P. Raming, “Machine identification ofhuman faces,” Pattern Recognition 13, 97–110 (1981).

29. G. J. K. Jr and K. J. Breeding, “The automatic recognition of human faces fromprofile silhouettes,” IEEE Trans. Syst., Man, Cybern., pp. 113–121 (1976).

30. C. J. Wu and J. S. Huang, “Human face profile recognition by computer,” PatternRecognition 23, 255–259 (1990).

31. K. Yu, X. Y. Jiang and H. Bunke, “Robust facial profile recognition,” in Proc. IEEEInternational Conference on Image Processing 3, 491–494 (1996).

32. Y. Gao and M. K. Leung, “Human face profile recognition using attributed string,”Pattern Recognition 35, 353–360 (2002).

33. H. Zabrodsky, S. Peleg and D. Avnir, “Symmetry as a continuous feature,” IEEETrans. Pattern Anal. Machine Intell. 17(12), 1154–1165 (1995).

June 28, 2005 10:41 WSPC/164-IJIG 00188

592 G. Pan & Z. Wu

34. C. Sun and J. Sherrah, “3D symmetry detection using the extended gaussian image,”IEEE Trans. Pattern Anal. Machine Intell. 19(2), 164–169 (1997).

35. D. Shen, H. H. S. Ip, K. K. T. Cheung and E. K. Teoh, “Symmetry detection bygeneralized complex (gc) moments: A close-form solution,” IEEE Trans. Pattern Anal.Machine Intell. 21(5), 466–476 (1999).

36. M. Kazhdan, B. Chazelle, D. Dobkin, A. Finkelstein and T. Funkhouser, “A reflectivesymmetry descriptor,” in Proc. European Conference on Computer Vision 2, 642–656(2002).

37. J. Cartoux, J. T. Lapreste and M. Richetin, “Face authentification or recognition byprofile extraction from range images,” in Workshop on Interpretation of 3D Scenes,pp. 194–199 (November 1989).

38. W. H. Press, S. A. Teukolsky, W. T. Vetterling and B. P. Flannery, Numerical Recipesin C, 2nd ed., Cambridge University Press (1992).

39. P. J. Best and N. D. Mckay, “A method for registration of 3D shapes,” IEEE Trans.Pattern Anal. Machine Intell. 14(2), 239–256 (1992).

40. D. P. Huttenlocher, G. A. Klanderman and W. J. Rucklidge, “Comparing images usingthe Hausdorff distance,” IEEE Trans. Pattern Anal. Machine Intell. 15(9), 850–863(September 1993).

41. S. Kirkpatrick, C. D. Gelatt and M. P. Vecchi, “Optimization by simulated annealing,”Science 220, 671–680 (1983).

42. D. F. Watson and G. M. Phillip, “Triangle based interpolation,” Mathematical Geology8(16), 779–795 (1984).

43. J. Kittler, M. Hatef, R. P. W. Duin and J. Matas, “On combining classifiers,” IEEETransactions on Pattern Analysis and Machine Intelligence 20(3), 226–239 (1998).

44. P. J. Phillips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi and M. Bone,“Face recognition vendor test 2002: Evaluation report,” Tech. Rep. (March 2003).

Gang Pan received the BS and PhD degrees in computer sci-ence from Zhejiang University in 1998 and 2003, respectively.He is a recipient of Microsoft Fellowship Award 2000. FromSeptember to December 2000, he was a visiting student atMicrosoft Research Asia. He has published over ten papers oncomputer vision and image processing.

Currently, he is an assistant professor of computer scienceat Zhejiang University. His research interests include computer

vision, image processing, statistical pattern recognition and computational learn-ing theory. Dr. Pan is a member of the IEEE. His homepage is: http://www.cs.zju.edu.cn/∼gpan.

Zhaohui Wu received the PhD degree in computer science fromZhejiang University in 1993. From 1991 to 1993, he was with TheGerman Research Center for Artificial Intelligence (DFKI) as ajoint PhD student, where he was working in the area of knowl-edge representation and expert system. He joined the ComputerScience Department at Zhejiang University in 1993, and is cur-rently a professor and the vice-director of the College of Com-puter Science and Technology.

June 28, 2005 10:41 WSPC/164-IJIG 00188

3D Face Recognition from Range Data 593

He served as the PC member of various international conferences and is on theeditorial board of several journals. He also is the author of more than 100 refereedpapers. His current research interests mainly include biometrics, knowledge gridcomputing, artificial intelligence and pattern recognition. He is a member of theIEEE and the IEEE Computer Society.