[ieee 2009 24th international conference image and vision computing new zealand (ivcnz) -...

Boosted Dynamic Active Shape ModelYu Chen

School of Computer Science andEngineering

The University Of New South WalesSydney, Australia

Email: [email protected]

Xiongcai CaiSchool of Computer Science and

EngineeringThe University Of New South Wales

Sydney, AustraliaEmail: [email protected]

Arcot SowmyaSchool of Computer Science and

EngineeringThe University of New South Wales

Sydney, AustraliaEmail: [email protected]

Abstract—We present a novel optimization scheme in thepopular Active Shape Model(ASM) framework, which increasesthe accuracy and robustness of searching for a hypothesisshape. The determininistic fitting scheme in traditional ASM issubstituted by a probabilistic estimation approach in our work.A set of weighted particles is used to represent each salientfeature point to form a shape density, the particle densitiesare evolved by the observation model and used to update theshape parameter. The displacement between iterations providesthe particles with dynamic information to locate a new startingshape density for the next iteration. Furthermore, the likelihoodfunction of the observation model is trained with a Gentleboostregression algorithm and results in the proposal distribution. Theproposed algorithm is much more robust in nonlinear and noisyenvironments and not affected by initialisation conditions, asare current ASM optimization algorithms. Finally, the developedapproach provides higher accuracy and increases the convergencerange in face segmentation on the BioID public test data set.

I. INTRODUCTION

There has been significant research into model-based ap-proaches for the interpretation of face images due to theircapability to represent large variations and different expres-sions. In particular, Active Shape Model(ASM) [1] has beensuccessfully applied to model human faces in many computervision tasks such as face tracking and facial motion recog-nition. It restricts the global shape model to the sum of thestatistical mean shape and a weighted eigen matrix that coversthe majority of variations, that are learned from a set of labeledtraining samples. The weight of the eigen matrix is updatedby iteratively matching the points around each salient featurepoint such as eyes, nose etc, with a predefined local appearancemodel and optimizing their locations consistently with theglobal shape model.

The conventional ASM algorithm fits a local appearancemodel either by searching on the gradient profile using alocal eigen model [1] or by searching in a sliding windowwith boosted feature classifiers [2] to find the local maximaof feature points and updating the parameters according tothe displacement. Nevertheless, both the algorithms assume aGaussian distribution and initialise close to the true shape.If the sliding window is distorted by noise or the startingshape is initialised too far from the true location, the ASMlocal search would fail due to the common local minimaproblem. Additionally, the low accuracy of classifiers alsoaffects the final shape hypothesis. Therefore, a probabilistic

representation appears promising in order to model the shapestate in every iteration.

We introduce a Boosted Dynamic Active Shape Model(BDASM), an extension of ASM, to effectively localise humanface images by incorporating a dynamic optimization schemeinto the conventional ASM. The state of the dynamic modelis represented as a set of weighted particles for the locationof each feature point. In the optimization phase, we randomlysearch around feature points and focus on the region that hashigher probability to match the appearance model rather thansearching in a limited range. The dynamic search around eachsalient feature point ensures avoidance of local minima and in-creases the speed of particle search. It utilises the informationon the previous trajectories to predict possible new locationsof particles in the next iteration. Although the hypothesisedshape is an estimated rather than an optimal shape, the shaperegistration in ASM results in a better match. The particlesare propagated when searching around the feature point andmoving towards higher probabilities. Therefore the search areaof each feature point is not limited to a certain range suchas gradient profile or sliding window, which usually lead toinaccurate initialisation.

The paper is organized as follows: related work and back-ground are reviewed in Sections II and III, the proposedBDASM algorithm is described in Section IV, experimentalresults are presented in Section V, with conclusion following.

II. RELATED WORK

The ASM proposed by Cootes et al. [1] is one of themost popular methods for wrapping an initialised shape aroundimage features. It models the statistical shape and its variationsover a labeled training set containing a set of salient featurepoints such as eyes, nose and mouth in the case of faces. Thereare other deformable models such as snakes [3], but unlikesnakes, which have little prior knowledge incorporated, ASMuses an explicit shape model to place global constraints on thegenerated shape. The Active Appearance Model (AAM) pro-posed later by Cootes et al. [4] integrates a global appearancemodel into the shape model.

In recent years, many modifications of the general ASMscheme have been proposed. Traditional ASM searches thegradient profile along each salient feature point to find thepoint that best matches the appearance model. A mixture of

978-1-4244-4698-8/09/$25.00 ©2009 IEEE

24th International Conference Image and Vision Computing New Zealand (IVCNZ 2009)

- 215 -

Gaussian models is employed to represent the shape variationin contrast to assuming a single Gaussian distribution in theclassical ASM [5]. Romdhani et al. [6] use Kernel Princi-pal Components Analysis and a Support Vector Machine tomodel nonlinear changes in 3-D pose variations. Rogers andGraham [7] apply robust least-squares techniques and randomsampling to estimate the shape parameters. Cristinacce andCootes [2] trained a Gentleboost regression model [8] on ahistogram response of Haar-wavelet functions to predict theresidual of each feature point. All these approaches sufferfrom the Gaussian assumption and poor accuracy of classifier,such as the 70% - 80% truth positive rate obtained usingadaboost classifier [9], as well as strong dependence on shapeinitialisation. Zhou et al. [10] estimate shape and poseparameters using Bayesian inference after projecting shapesinto a tangent space. Wang et al. [11] extend Zhou’s workby introducing a resampling scheme to incorporate a pose-dependent appearance model for multi-view face alignment.However, the estimation of pose parameters depends only onthe observation model due to its local Markov property andalso assumes that the observation noise is isotropic Gaussian.

Unlike previous work, we integrate a dynamic model inthe core of ASM and develop a particle filtering optimizationframework for efficient fitting. We will discuss how the shapemodel interacts with the particle filter. Experimental resultsare provided and compared with the segmentation accuracyof Boosted Regression Active Shape Model (BRASM) [2]and Constrained Local Model [12] on the same data set undersimilar conditions.

III. PRELIMINARIES

A. Active Shape Model

The Active Shape Model (ASM) consists of statistical shapemodeling and an optimization process which deforms theshape model to best fit the image data. For general facialanalysis, the shape model is trained with a set of labeled faceimages and aligned from one shape to another with a similar-ity transform containing three pose parameters - translation,scaling and rotation. The average of all the aligned imagesis the mean shape of the training set. Principal ComponentAnalysis (PCA) is used to model shape variation from thecovariance matrix of those aligned images. Any shape can beapproximated using:

x = x̄ + Pb (1)

where x is a vector of model points in shape space, x̄ is themean shape of the training samples, P is a matrix of orthogonaleigenvectors and b is the shape parameter containing a vectorof weights corresponding to the eigenvectors in P. A shapevector X in image space is a euclidean transformation of theshape model x.

X = M(s, θ)[x] + t (2)

where s, θ and t are the pose parameters of scaling, in-planerotation and translation respectively. ASM iteratively updatesthe pose and shape parameters to find a shape vector X that

best matches the image information to the local appearancemodel. Classical ASM matches the local appearance modelby minimising the Mahalanobis distance of the local eigenpatches.

ASM begins from initialising a start shape Xt at iteration teither manually or by using some global detector to determinethe face location. It then repeats the following three steps untilconvergence.

1) Find a hypothesis shape X ′ by matching the appearancemodel for each feature point and its nearby points to findthe minimum mahalanobis distance.

2) The hypothesis shape X ′ is aligned with the shape modelx using registration techniques and the residual dx iscalculated by definition (3) below.

3) The residual dx is used to approximate the variation ofshape parameters db in definition (4) below.

4) Update the pose and shape parameters s, θ, t, b. Usingthese parameters a suggested shape X may be found.

dx = M(s−1, θ−1)[X ′] − t − x (3)

db ≈ dx/P (4)

where s, T, θ refer to the euclidean transformations oftranslation, rotation and scaling respectively. The algorithmconverges when X ′−X is less than a predefined threshold orafter a certain number of iterations.

B. Particle Filter

Particle Filter [13] is commonly used to estimate the poste-rior distribution in Bayesian models based on the propagationof a sample set with a stochastic motion model. It representsthe probability density by a set of weighted particles {qi

t, ωit}

which are drawn from the posterior distribution in the previousstep, qi

t is the state vector and ωit are nonnegative weights

called importance factors, and the propagation proceduresconsist of two parts:

1) Initialisation: sample N particles qi0 from a prior uniform

distribution p(q0).

qi0 ∼ p(q0), i = 1, · · ·N

2) Prediction: sample qit ∼ p(qi

t|qit−1) where p(qi

t|qit−1) is

the probabilistic representation of the dynamic model.3) Correction: update the importance weights:

ωit =

ωit−1 × p(oi

t|qit)

c(5)

where ot is the observation at the current state and p(oit|qi

t)is the maximum likelihood function that indicates the probabil-ity of the particles to be in state qi

t while given the observationis oi

t. The distribution {qit−1, ω

it−1} which is sampled from the

posterior is called the proposal distribution. The predicted stateand updated weight {qi

t, ωit} represents the optimal proposal

distribution, which is resampled to avoid the problem ofdegeneracy and results in the posterior distribution which isalso the prior distribution for the next iteration.


- 216 -

(a) Left eye feature point

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

(b) Confidence map around left eyepoint

(c) Left eyebrow corner

5 10 15 20 25 30 35 40

5

10

15

20

25

30

35

40

(d) Confidence map around lefteyebrow corner point

Fig. 1. Two salient feature points with their confidence maps indicating theaccuracy of the corresponding classifier

IV. BOOSTED DYNAMIC ACTIVE SHAPE MODEL

Most learned classifiers in classification-based ASM achieveabout 80% accuracy in detecting feature points. The left eyepoint and eyebrow point and their confidence maps are shownin Fig.1. The confidence map is generated from a boosted func-tion where a subset of Haar-like features of points in a 20×20patch around the salient feature point is the input; the outputis the probability of these points, and the points with highervalue are more likely to be the true feature points than theothers. It is clear that the eye classifier is more accurate thanthe eyebrow’s, as in Fig 1(d) the distribution is multi-modalrather than Gaussian, as in Fig 1(b). Therefore, probabilisticbased ASM becomes an option as it can estimate both thesefeatures rather than find an optimal solution, especially forthe particle filter that is able to approach a Bayesian optimalestimate on complex distributions. However, if the particlesevolved only depend on the observation model, they may beimpacted by the noisy data.

The proposed BDASM addresses the above problem byintegrating a dynamic model to find the hypothesis shape ineach iteration. More specifically, a particle based shape modelis introduced to replace the shape vector, and the particlesare evolved to find the hypothesis shape in ASM. We utilisethe displacement between two iterations as dynamic informa-tion and update the importance weight with an observationmodel trained by Gentleboost on Haar-wavelet functions.The interaction between the particle filter and shape modelgreatly improves the accuracy and robustness of locating thehypothesis shape. The flow of BDASM is shown in Fig. 2.

A. Particle Based ASM

To form a probabilistic shape space, we need to representeach feature point by a set of N particles {xi

n, yin} associated

Fig. 2. Illustration of BDASM algorithm

with weights ωi, where x,y are the coordinates of each featurepoint. The weights are sampled from a uniform distributionwith ωi = 1/N where N is the number of particles. Thehypothesis shape is the average location of those particles.

B. Dynamic Shape Model

Before searching for the hypothesis shape, the locations ofparticles are predicted according to the previous two iterations.We use a standard second order autoregression model insteadof a Hidden Markov Model as the dynamic model:

qit = αqi

t−1 + β(Xt−1 − Xt−2) + δ (6)

where α and β are the autoregression coefficients, Xt isthe suggested shape vector at iteration t and δ is somerandom noise. In every iteration the suggested shape vectorX is generated from the transformed shape model x, andthe dynamic information drives the particles towards a globalshape model rather than depending only on the observationmodel.

C. Boosted Observation Model

The predicted particles are confirmed with an observationmodel to update their importance weights and form theproposal distribution. The posterior distribution is obtainedfrom a resampling scheme on the proposal distribution. Itis useful to integrate boosting into the observation modelas it increases the robustness compared to minimising themahalanobis distance between the features and model. Theobservation model updates the weight ωi of each particle


- 217 -

with definition (5). The likelihood function is trained from aGentleboost logistic regression on the Haar-wavelet response.We extract the Haar-like features for every feature point andclassify it as a positive sample. We also randomly sampleseveral points around the feature point and classify themas negetive samples. By appling these training samples inAlgorithm 1, the most sufficient Haar-like features are selectedand an output function predicts the likelihood of each particle.Then the weight ωi is updated using definition (5) so asto approximate the proposal distribution {qi

t, ωit}. Finally,

a resampling algorithm is used to eliminate particles withsmall probabilities whilst emphasising those with strong largeprobabilities.

Algorithm 1 GENTLEBOOST TRAINING ALGORITHM

1) Start with weight ωi = 1/N, i = 1, 2, 3, ...., N ,F (x) =0 and yi = 1 for positive examples and yi = −1 fornegative examples.

2) Repeat for i = 1, 2, 3, ...,M

a) Fit the regression function fm(x) by least squaresof yi to xi.

b) Select the fm(x) with least error∑N

i=0(yi −fm(xi)2)

c) Update F (x) ← F (x) + αfm(x)d) Update the residual target value yi ← yi − fm

3) Output the regression function F (x) =∑n

i=1 fm(x)

D. BDASM

The details of integrating the particle filter to search thehypothesis shape are in Algorithm 2 including iterativelyupdating the parameters in ASM. The initial shape vector X0

with N feature points is a Euclidean tranformation of meanshape into the bounding box detected by the face detector; b0

is initialised as a zero vector, s0,Θ0 and t0 are the projec-tion parameters between the mean shape and the detector’sbounding box. For each feature point in the shape vector,we separately sample N particles around this point, withrandom noise added. The particles are evolved by the dynamicfunction, and the observations of these new particles updatethe weights. The hypothesis shape is calculated by the meanlocation of particles around every point, followed by updatingthe shape and pose parameters. The new parameters generatea suggested shape, and the distance between the suggested andprevious shapes provides the dynamic information to guide theparticles. Finally, we run 20 iterations and output the suggestedshape as the final shape hypothesis.

V. EXPERIMENTS

For evaluation of the proposed method, we conductedexperiments on the BioID public test data set [14] whichcontains 1522 labeled face images. We randomly selected 1019images for training the regression function and 493 images fortesting. We adopted the same test criterion as in [2]:

me =1ns

Σd (7)

Algorithm 2 BDASM ALGORITHM

1) Initialising start shape X0 by a global face detector withthe average point projected into the bounding box

X0 ← M [x̄ + Pb0](s0,Θ0) + t0

2) Randomly sample N particles {qi1,2,···N , ωi

1,2,···N}around each salient feature point i and set all the weightsωi

j = 1/N .3) Optimization

repeatfor i = 1 to k do

for j = 1 to N doPredict a new location for every parti-cles with dynamic model by (6).Update weight ωj

i for each particleaccording to the boosted observationsby (5).

end forResample the proposal distribution.Calculate the hypothesis point by the meanof the posterior distribution.

end forUpdate pose and shape parameters with thehypothesis shape as in original ASM (see sec-tion III-A).Align the model from the updated pose and shapeparameters to get the suggested shape Xt+1.Save the displacement between the two iterationsXt+1 −Xt for evolving the particles in the nextiteration.

until Converged

4) output NewX

where d is the point to point error for each individual pointfeature location, s the groundtruth inter-ocular distance, andn=17 includes only the internal feature locations around theeyes, nose and mouth which were used in the experiments,while other feature points were eliminated as they could behighly variable. There are also some parameters to be set. Inthe training phase, a 15 × 15 patch was chosen to extract theHaar-like features and we trained 400 rounds in the boostingpart; in the searching phase, we used 200 particles for eachfeature point and α and β in the dynamic model are set to 1and 0.5 respectively. An example of BDASM search is shownin Figure 3. The initial shape is the groundtruth shape replacedby mean shape and displaced by 40% of the interoculardistance. The results are compared with those of Cristinacceand Cootes [2] and [12]. It is hard to make straightforwardcomparisons, as different training and testing images wereselected in different works, Cristinacce and Cootes used 22feature points for searching and 17 points for evaluation whilewe use 17 points for both searching and evaluation. However,these are the most up-to-date published results on BioID dataset availiable. The accuracy of the previous methods here are


- 218 -

TABLE ITHE PERCENTAGE OF IMAGES WITHIN CERTAIN RANGE OF me

me < 0.05 0.10 0.15CLM [12] 55% 90% 95%BRASM [2] 15% 95% 95%BDASM 60% 96.5% 98%

TABLE IIRANG OF CONVERGENCE IN DIFFERENT DISPLACEMENT

Displacement 10% 20% 30% 40% 50%Det-ASM [2] 99% 97% 70% 30% 8%BRASM [2] 98% 97% 88% 47% 13%BDASM 98% 97% 73% 60% 45%

from the authors’ statements or deduced from the graphs oftheir experiments.

A. Full Search Results

The fully automatic search utilizes the SNoW face detector[15] for each test image to locate a bounding box of the face.It projects the mean shape into the bounding box as the startshape, then BDASM iteratively fits the model to the shapeuntil it converges. The results are shown in Fig. 4, where thex-axis is the point to point error measurement and the y-axisrepresents the percentage of images with me < x. The resultsfor BDASM in Table I outperform the previous method greatlyfor both low and high values of me. Comparing with CLM,BRASM has better results for me < 0.1, but it is not likely toconverge as accurately as in CLM. However, BDASM showsan improved rate of 5% compared to CLM for me < 0.05 and1.5% compared to BRASM for me < 0.01.

B. Displacement Results

In order to demonstrate that the algorithm is not affectedby the initialisation, the ground truth points were replaced bythe mean shape and displaced in eight compass directions by10%, 20%, 30%, 40% and 50% of inter-ocular distance.Therange of convergence within the error limit of me < 0.15 isshown in Table II comparing with BRASM and Det-ASM[2].

C. Discussion

The BDASM optimization is more stable and accurate as theparticles focus on a small range around each salient featurepoint, even when far away or corrupted with noise. However,the dynamic model could affect the performance. If the particlemoves a lot, more noise would be introduced into the searchspace; otherwise it needs more iterations to converge. In TableII, BDASM shows compatible results within 30% of inter-ocular displacement, but the ratio greatly improves if the shapeis displaced more than 30%. However, at 30% displacement inTable II, BDASM is 15% less accurate than BRASM becausethe hypothesised shape is an estimated shape rather than anoptimized one.

(a) Initial shape at 40% of inter-ocular distance me =0.4014

(b) After 5 iterations me = 0.3339

(c) After 10 iterations me = 0.1867

(d) After 15 iterations me = 0.0932

(e) Final iteration (20th iteration) me = 0.0731

Fig. 3. Example of Iterative BDASM search on BioID image


- 219 -

0 0.05 0.1 0.15 0.2 0.25 0.30

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

me

Per

cent

age

Of C

onve

rgen

ce

BDASMFace Detector

Fig. 4. cumulative distribution of point to point measure on BioID test setswhen using face detection to initialise the local search

VI. CONCLUSION

In this paper, we propose BDASM for dynamic featurepoint localisation in Active Shape Models based on particlefilter searching. The implemented method is more stable androbust and provides more accurate results. It do not need aGaussian distribution assumption of local appearance modelused by previous method. In addition, we have shown thatBDASM is efficient for improper accuracy from the groundtruth. Although BDASM has shown promising results onthe BioID dataset, there are still a number of issues to beaddressed. The parameters of the dynamic model must betaken into account as they can affect the performance ofBDASM. The serching takes long time as for each points about20000 features needs to be extracted, therefore the number ofparticles and iterations should be adjusted and vary the featurespace of the observation model to reduce the computationalcomplexity. Some preprocessing to compute the confidencemap could also be useful for increasing the speed of search.Pose changes should be incorporated and this will requireextension to the likelihood function and a more extensivedatabase.

REFERENCES

[1] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham, “Active shapemodels - their training and application,” Comput. Vis. Image Underst.,vol. 61, no. 1, pp. 38–59, January 1995.

[2] D. Cristinacce and T. Cootes, “Boosted regression active shape models,”in 18th British Machine Vision Conference, Warwick, UK, 2007, pp.880–889.

[3] M. Kass, A. Witkin, and D. Terzopoulos, “Snakes: Active contourmodels,” International Journal of Computer Vision, vol. V1, no. 4, pp.321–331, January 1988.

[4] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearancemodels,” Proceedings of the European Conference on Computer Vision,vol. 2, pp. 484–498, 1998.

[5] T. F. Cootes and C. J. Taylor, “A mixture model for representing shapevariation,” in Image and Vision Computing. BMVA Press, 1997, pp.110–119.

[6] S. Romdhani, S. Gong, A. Psarrou, and R. Psarrou, “A multi-viewnonlinear active shape model using kernel pca,” in In British MachineVision Conference. BMVA Press, 1999, pp. 483–492.

[7] M. Rogers and J. Graham, “Robust active shape model search,” in InProceedings of the European Conference on Computer Vision. Springer,2002, pp. 517–530.

[8] J. Friedman, T. Hastie, and R. Tibshirani, “Additive logistic regression:a statistical view of boosting,” Annals of Statistics, vol. 28, p. 2000,1998.

[9] S. C. David, D. Cristinacce, and T. Cootes, “Facial feature detectionusing adaboost with,” in In Proc. BMVC03, 2003, pp. 231–240.

[10] Y. Zhou, L. Gu, and H.-J. Zhang, “Bayesian tangent shape model:estimating shape and pose parameters via bayesian inference,” in Com-puter Vision and Pattern Recognition, 2003. Proceedings. 2003 IEEEComputer Society Conference on, vol. 1, June 2003, pp. I–109–I–116vol.1.

[11] Z. Wang, X. Xu, and B. Li, “Bayesian tactile face,” in Computer Visionand Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, June2008, pp. 1–8.

[12] D. Cristinacce and T. Cootes, “Feature detection and tracking withconstrained local models,” in 17th British Machine Vision Conference,Edinburgh, UK, 2006, pp. 929–938.

[13] S. Maskell and N. Gordon, “A tutorial on particle filters for on-linenonlinear/non-gaussian bayesian tracking,” IEEE Transactions on SignalProcessing, vol. 50, pp. 174–188, 2001.

[14] O. Jesorsky, K. J. Kirchberg, and R. W. Frischholz, “Robust facedetection using the hausdorff distance.” Springer, 2001, pp. 90–95.

[15] M. H. Yang, D. Roth, and N. Ahuja, “A snow-based face detector,” inAdvances in Neural Information Processing Systems 12. MIT Press,2000, pp. 855–861.


- 220 -

[ieee 2009 24th international conference image and vision computing new zealand (ivcnz) -...

Documents