satya mahesh muddamsetty supervisor: tommaso gritti video processing & analysis group examiner:...

Satya mahesh MuddamsettySupervisor: Tommaso Gritti

Video processing & Analysis groupExaminer: Mikael Nilsson, Department of Signal processing, BTHSeptember 30, 2009

Face alignment using Boosted Appearance Model (BAM)

Confidential Divison, MMMM dd, yyyy, Reference 2

Outlines

• Brief summary of Previous methods• Shape model learning in BAM• Appearance model learning in BAM• Alignment using BAM• Experiments & Results• Conclusion

• Introduction

Confidential Divison, MMMM dd, yyyy, Reference

Introduction: Image alignment/fitting

It is the process of moving and deforming a template to minimize the distance between template and an image.Alignment is done in 3 stepsstep1: model choice template, ASM,AAM

step2 : distance metrics MSE (Mean Square Error)

step3: optimization Gradient descent methods


Introduction: applications

• Face fitting [Baker & Matthews’04IJCV]

• Tracking [Hager & Belhumeur’98PAMI]

• Medical Image Interpretation [Mitchell et al.’02TMI]

• Industrial Inspection


Outlines

• Introduction • Brief summary of previous methods• Shape model learning in BAM

• Appearance model learning in BAM• Alignment using BAM

• Experiments & Results• Conclusion


Brief summary of previous methods

• Point distribution model (PDM) [Cootes et al.’92BMVC]

• Active shape model (ASM) [Cootes & Taylor’92BMVC]

• Active appearance model (AAM) [Cootes et al.’01PAMI]

• Inverse compositional (IC) and simultaneous inverse compositional (SIC) AAM fitting [Baker & Matthews’04IJCV]



• Active shape model (ASM) [Cootes & Taylor’92BMVC]

It uses the shape model (PDM) as a template

k

iiip

1

0ss

k1 ppp ,...., 2pMean shape

Shape parameters

Eigen vectors

Drawbacks:

• Only the local appearance information around each landmarks is learned, which might not be the effective way of modeling

It seeks to minimize the distance between model points and the corresponding points found in the image.


AAM template:

• Inverse compositional (IC) and Simultaneous Inverse compositional (SIC) AAM fitting [Baker & Matthews’04IJCV]



• Inverse compositional (IC) and SimultaneousInverse compositional (SIC) AAM fitting [Baker & Matthews’04IJCV]

Image observation

Warping function

Image coordinate

Shape parameter

Appearance parameter

Appearance basis

Mean shape

Mean appearance

*

* Gross et al.’05IVC

AAM-distance



• All these AAM methods know to have a generalization problem, they degrades quickly when the they trained on large dataset.

• And the performance is poor on the unseen data

• These models are generative models

How to solve them?

• New method known as Boosted Appearance Model (BAM)

• It is an discriminative model.

• It has shape model, appearance model and an specific alignment method.

Problems of Previous methods


• Introduction • Breif summary of previous methods

• Appearance model learning in BAM• Alignment using BAM• Experiments & Results• Conclusion

• Shape model learning in BAM

Outlines


• The shape model learned by applying principle component

analysis (PCA) to the set of shape vectors S , where Nii ,...2,1; sS

The shape data consists of points in nD-spacesi = (xi0,xi1,xi2,…,xin,yi0,yi1,…,yin )T with observation i = {1,..,N}Steps:

1. Compute the mean of the training data :

N

iiN 1

1ss0

2. Compute the covariance of the data: (nd x nd matrix)

N

N 1i

Tii -- ))((

1

100 ssssC

3. Compute the Eigen vectors Φi and the corresponding Eigen values of C

i

Sample shape from training set

• Same shape model (PDM) used in previous methods

Shape model learning in BAM


Eigenvectors: fi =(fi,1,fi,2,…,fi,nd-1,fi,nd)(which means (x0,x1,x2,…,xn,y0,y1,…,yn ) )

Eigen values I, 2 ,…, nd

• The eigenvector with highest eigen value is the most dominant shape variation in the training set

• The eigenvectors are therefore ordered in magnitude of eigen value.

• Matrix of eigenvectors: F =[f1T

, f2T,…, fk

T ]

• So finally our parametric shape model s can be expresses as a mean shape plus a linear combination of k eigen vectors fk

0s

k

iiip

1

0ss

Sample shape from training set

Shape model learning in BAMShape model learning in BAM


learned Shape variationsBy varying the shape parameters with

respect to mean shape 0s

iip 0ss

i=1 i=2 i=3 i=4 i=5,..k

kppp ,....., 21pwhere

Shape model learning in BAM


• Introduction • Breif summary of Previous methods• Shape model learning in BAM

• Alignment using BAM• Experiments & Results• Conclusion

• Appearance model learning in BAM

Outlines


• Similar to AAM our appearance model is defined on the warped Image I(W(x;p)

• In BAM, appearance model is a set of weak classifiers which learns the decision boundary between correct alignment (positive class) and incorrect alignment (negative class).

M

mmfF

1

pp

It is a function of warped image I(W(x;p) warped with the shape parameters p

M- number of weak classifiers

Appearance Model learning in BAM


Training samples Positive samples: compute the shape parameters:

where

Negative samples:• Perturb the each element of

original shape parameters:randomly and each element should

be uniformly distributed between [-1,1].

- Is the shape vector - Is the matrix of Eigenvectors

.', vpp ini

n – number of perturbed shape per each original shapev – is the k – dimensional vector with each element uniformly distributed from [-1,1] randomly - is a vector with k- Eigen values

T

0T ssp ii ki ppp ,...., 21p

is



);((1 pxWI

Positive samples

Negative samples

N- original shapes

Nq - perturbed shapes

N- original warped images

Nq- perturbed warped images

);((2 pxWI

);(( pxWIN

);(( )1,1('

1 pxWI

);(( )2,1('

1 pxWI

);(( )3,1('

1 pxWI

);(( )4,1('

1 pxWI

);(( )1,2('

2 pxWI

);(( )2,2('

2 pxWI

);(( )3,2('

2 pxWI

);(( )4,2('

2 pxWI



Label = 1

Label = -1

• Computing the Rectangular Haar features on the warped images via integral image (Viola and Jones)


Boosting:


Num

ber o

f orig

inal im

ages ‘N

’

Number of features ‘K’

Num

ber of p

ertu

rbed im

ages

‘Nq’

Number of features ‘K’Gentle

boosting

Positive samples

Negative samples

11, tHm

MMMm gtH ,,

1g

Haar Features selected by Gentle boost


Boosting:

Confidential Divison, MMMM dd, yyyy, Reference-40 -20 0 20 40-1

0

1

Weak classifier design:

mHmm tgfm

pxWIp ;arctan2

m = 1,2,…..M

Selected Haar feature by gentle boost

threshold

Selected feature by gentle boost computed on any warped imagem=1

1 1-1


m=2 m=3 m=4 m=5 m=6,.M


m=1 m=2 m=3 m=4 m=5 m=6

m=1 to 100

Final Weak classifiers



• Introduction • Breif summary of Previous methods• Shape model & learning in BAM• Appearance model & learning in BAM

• Experiments & Results• Conclusion

• Alignment using BAM

Outlines


M

mmfF

1

ppTakes Input: as warped image pxWI ; Output: SCORE

How to do alignment using BAM ?

Use the classification score from the trained strong classifier as a distance metrics .

This score indicates the quality of alignment

How this score is computed?

Training samples postivesamples400(red color) negative sample 4000(blue color)

Alignment using BAM


• Alignment in the sense given an initial parameters will have negative score, trying to look new parameters will have maximum positive score.

0p

alignedp

pp Fmax

But finding this new parameters is a non-linear optimization problem

To solve this iteratively we are using gradient ascent method

Alignment using BAM


Alignment/fitting via Gradient Ascent method

Solving the correct parameters iteratively

M

mmfF

1

pp

RM

SE

pp Fmax

where

Alignment using BAM

ppp

d

dF


Gradient Ascent SolutionOur trained two- class strong classifier

M

mmfF

1

pp

mHm

M

m

tgFm

pxWIp ;arctan2

1

M

m mHm

Hm

tg

g

d

dF

m

m

12;1

;2

pxWI

pW

pxWI

p

Gradient of is

pF

ppp

d

dF

Alignment using BAM


Inputs: Input Image I

, Initial shape parameters ,warped jacobian

0p

BAM;Shape model {mean shape , Eigen vectors }Appearance model { ;m=1,2,…..M}

0s Tmc

Step 0: Compute the gradient of the image ,

I

p

W

repeat 1. Warp I with, to compute pxW ; pxWI ;

2. Compute the selected feature for the each weak classifier on the warped input image :

mH mHmm tge

m pxWI ;

3. Warp the gradient image with the pxW ;I

4. Compute the steepest descent image p

WpxWI

;SD

5. Compute the integral images for each Colum of SD and obtain the rectangular features for each weak classifier:

SDgb mm

6. Compute using p

M

m m

m

e

b

121

2p

7. Update ppp until

M

mii

1

p

Alignment using BAM: summary


• Introduction • Breif summary of Previous methods• Shape model learning in BAM• Appearance model learning in BAM• Alignment using BAM

• Experiments & Results

• Conclusion

Outlines


• We used challenging FERET DATASET which contains frontal images, with different variations, pose, race, illuminations, expressions.

• samples images are here

Experiments & Results


Training

We trained shape model (PDM) with 1636 image annotations

Shape model

Appearance model

sets No of images

No of positive samples

No of negative samples

TRAINset1 400 400 4000 800 800 8000TRAINset2

Created Negative samples for different perturbation ranges {.8,1,1.2} for TRAINset1 and TRAINset2



Alignment example

RM

SE



performance:

Test Sets No of images Details

TESTset1

TESTset2

300

300

From train data

unseen data

TEST DATA



Experiments & Resultsperformance:

Train perturbation

.8 1 1.2

Test perturbation .4

84% 58% 54%

Test perturbation .6 78% 34% 37%


Test perturbation 1 56% 26% 28%

Test perturbation 1.2

50% 17% 21%


35% 15% 12%

Train perturbation

.8 1 1.2






57% 22% 23%


45% 19% 15%

TESTset1 TESTset2(unseen data)

On TRAINset1



Train perturbation

.8 1 1.2

Test perturbation .4

68% 82% 37%





32% 33% 9%


24% 22% 5%

Train perturbation

.8 1 1.2






30% 41% 11%


21% 24% 7%

performance:

TESTset1 TESTset2(unseen data)

On TRAINset2


• Test on different illumination image database YALE DATABASE

• Collected 30 images • Generated 5 initializations randomly per each image finally

150 trails

Test perturbations

.4 .6 .8 1 1.2 1.4

converged 65% 45% 30% 16% 11% 9%



• Introduction • Breif summary of Previous methods• Shape model & learning in BAM• Appearance model & learning in BAM• Alignment using BAM• Experiments & Results

• Conclusion

Outlines


Conclusions

• Idea of discriminative method in AAM seems like a powerful extension to classical methods

• Computational complexity still quite high

• Influence of amount of perturbation on training set is never mentioned in literature, but very strong

• integration of procrustes analysis not mentioned in the papers, even if it could help in building better shape models

Future work• compare results with classical AAM

implementations

• test with very large training database

satya mahesh muddamsetty supervisor: tommaso gritti video processing & analysis group examiner:...

Documents

bam appearance model

confidential divison

mmmm dd

shape model pdm

model points

aam methods

discriminative model

reference aam template