1 m. bronstein | expression-invariant representation of faces and its applications for face...
Post on 22-Dec-2015
229 Views
Preview:
TRANSCRIPT
1M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Expression-invariant representation of facesand its applications for 3D face recognition
Michael M. Bronstein
Department of Computer ScienceTechnion – Israel Institute of Technology
M.Sc. Seminar 2 November 2004
2M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Biometrics in the age of patriarchs
And Jacob went near unto Isaac
his father; and he felt him, and
said,
‘The voice is Jacob’s voice, but
the hands are the hands of Esau’.
And he recognized him not,
because his hands were hairy, as
his brother Esau’s hands.
Genesis XXVII:22
3M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Biometrics today
IRIS
FINGERPRINT
PALM
VOICE
RETINA
DNASIGNATURE
FACE
4M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
ILLUMINATION+ EXPRESSION
ILLUMINATION+ POSE
ILLUMINATION+ EXPRESSION
+ POSE
A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear
Problems of 2D face recognition
5M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Approaches in 2D face recognition
Few feature points can be reliably extracted
Such features are usually insufficient for good recognition
Appropriate model is a problem
No good model for facial expressions
INVARIANT REPRESENTATION GENERATIVE APPROACH
6M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Intrinsic problem of 2D face recognition
A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear
7M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
2D vs 3D in face recognition
Simple acquisition – legacy hardware and databases
Sensitive to everything (lighting, pose, makeup, expressions)
Insensitive to lighting, pose, make up
Requires special hardware
Sensitive to expressions
Sensitive to aging and plastic surgery (?)
8M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Face is a smooth, connected, compact 2D Riemannian manifold
Parametrization
Metric tensor
Riemannian geometry basics
2 1 1 2 3 1 2 3: , ,..., ,x x x R R
Geodesic distances
ij i jg x x
infCd C
x1
2
S
S
1x
3x
2x
x 1 x
2 x
9M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
ISOMETRIC NON-ISOMETRIC
Isometry is a transformation that preserves geodesic distances
Isometric model
A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear
Isometric model: Facial expressions = isometries of some initial facial surface (neutral expression)
10M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
The open mouth problem
WITHOUTH TOPOLOGICAL CONSTRIANT
Open mouth is not an isometry
Isometric model is true for expressions with closed or open mouth
Extension: topologically-constrained isometric model
WITH TOPOLOGICAL CONSTRIANT
M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
11M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
RIEMANNIAN EUCLIDEAN
Geodesic distances are an invariant description of the surface…
…but are inconvenient to work with
A. Elad and R. Kimmel, CVPR 2001A. Elad and R. Kimmel, IEEE Trans. PAMI, 2003
[Elad & Kimmel 2001]: embed isometric surfaces into a low-dimensional Euclidean space and treat them as rigid objects
Canonical forms
i
jix
jx
,ij i j 2ij i jd x x
12M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
- (m > 2)-dimensional manifold (embedding space)
Isometric embedding
1 1: ,..., , ,..., ,Nm
Nx x D SS
, 1,..., .ij ij id j N
mS
- NN matrix of original geodesic distances ij
- NN matrix of distances in the embedding space ijdD
A mapping between finite metric spaces
such that
13M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Embedding problem in cartography
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV
The globe cannot be embedded into a plane according to Gauss Theorema Egregium.
GLOBE (HEMISPHERE) PLANAR MAP
14M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Does an isometric embedding always exist ?
N. Linial
Example of 4 points on a sphere that cannot be isometrically embedded into an Euclidean space of ay finite dimension.
4 POINTS ON A SPHERE A NEAR-ISOMETRIC EMBEDDING
15M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
- Nm matrix of canonical form parametric coordinates
Embedding error
2
rawi
ijijj
d
X X
2
norm 2
iji j
ij
ij
ji
d
d
X
XX
1,..., NX x x
RAW STRESS:
NORMALIZED STRESS:
I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997
16M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Multidimensional scaling
Nm optimization variables
I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997
min X X
Multidimensional scaling (MDS) problem:
Optimum defined up to an isometry in mS
Non-convex optimization problem
Optimum = canonical form
17M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Euclidean embedding – LS MDS
where
1 if
1 if ij
i ju
N i j
1 if and 0
0 if and 0
if ij
ij
ij iij j
j i
ij
d d
d
i j
b i j
b i j
X
Choose 3m RS
I. Borg and P. Grönen, Modern multidimensional scaling, Springer, 1997
2 2raw
T T
2
trace 2trace
ij ii j i j i j
ji j
A
j id d
A
X X
XUX XB
X
X X
raw 2 2 0 X UX X X X
For require rawmin X X
Use
18M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Alignment
Eliminate first-order moments
Reorder the axes
Eliminate mixed second-order moments
Force the sign
100 010 001 0
200 110 101 200
110 020 011 020 200 020 002
101 011 002 002
;
1
sign 0, 1,...,3N
i
ki k
where 1 2 2
1
N p q
i i i
r
pqri
x x x
19M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Canonical forms of faces (closed mouth)
FACIAL SURFACES
CANONICAL FORMS
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV
20M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
FACIAL SURFACES
CANONICAL FORMS
Canonical forms of faces (open mouth)
M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
21M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Spherical embedding
A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3”, subm. ScaleSpace
Choose as a sphere immersed into
3 4 : 1 x xS R
Geodesic distances (arcs of great circles)
31, cos ,i j i jd x x x x
S
3S 4RmS
ix
jx
Minimize the normalized stress w.r.t. the parametric coordinates
2
2normi j
i
ijj
jj
i
i
d
d
X
X
Alignment performed using Euclidean moments in 4R
22M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Spherical embedding example
Maximum-variance projection onto R3 of a facial surface embedded into S3 with different radii
R = 5 cm
A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3”, subm. ScaleSpace
R = 7.5 cm R = 15 cm
23M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Spherical embedding vs Euclidean embedding
A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3”, subm. ScaleSpace
SPHERE RADIUS R [cm]
EM
BE
DD
ING
ER
RO
R
24M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
3DFACE system – hardware
PROJECTOR
CAMERA
MONITOR
CARDMOUNTING
25M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
3DFACE system – user interface
26M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
3DFACE system – pipeline
27M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Isometric model validation
133 toothpaste markers placed on the face as invariant points
Track how geodesic / Euclidean distances change due to expressions
Lips cropped
16 different expressions
M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
28M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
-60 -40 -20 0 20 40 600
0.2
0.4
0.6
0.8
1
-150 -100 -50 0 50 100 1500
0.2
0.4
0.6
0.8
1
Isometric model validation (cont.)
ABSOLUTE CHANGE mm RELATIVE CHANGE %
EUCLIDEANGEODESIC
M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
29M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Sensitivity to facial expressions
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCVM. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
30M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Sensitivity to facial expressions (closed mouth)
Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database.
-1000 -800 -600 -400 -200 0 200 400 600 800 1000-1000
-800
-600
-400
-200
0
200
400
600
800
1000
-1000 -800 -600 -400 -200 0 200 400 600 800 1000-1000
-800
-600
-400
-200
0
200
400
600
800
1000
RIGID SURFACES CANONICAL FORMS
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV
31M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Sensitivity to facial expressions (open mouth)
Visualization of the distances between faces in the facial expressions experiment. Each point represents a subject in the database.
RIGID SURFACES TOPOL. CONSTR. CANONICAL FORMS-1000 -800 -600 -400 -200 0 200 400 600
-100
-50
0
50
100
150
-1000 -500 0 500 1000
-400
-300
-200
-100
0
100
200
300
400
M. Bronstein, A. Bronstein and R. Kimmel, “Expression-invariant representation for human faces”, submitted
32M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Benchmarks
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV
Benchmark of face recognition algorithms on a database containing 220 instances of 30 faces with extreme facial expressions.
RIGID
CANONICAL
EIGENFACES
FALSE ACCEPTANCE RATE %
FA
LS
E R
EJE
CT
ION
RA
TE
%
RECOGNITION RANKR
EC
OG
NIT
ION
AC
CU
RA
CY
%
33M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Recognition example
A. Bronstein, M. Bronstein and R. Kimmel, “Three-dimensional face recognition”, submitted to IJCV
PROBE EIGENFACES RIGID SURFACE CANONICAL FORM
MORAN 129 ORI 188 SUSY 276 MORAN 114
MICHAEL 17 ALEX 40 ALEX 39 MICHAEL 2
34M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Identical twins – find the differences
The difference between Michael and Alex obtained by comparing the canonical forms reveals a slight difference in the geometry of their nose.
MICHAEL ALEXDIFFERENCE MAP
35M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Bibliography
A. Elad and R. Kimmel, On bending invariant signatures for surfaces, Trans. IEEE PAMI 25(10): 1285-1295, 2003
A. Bronstein, M. Bronstein and R. Kimmel, Expression-invariant 3D face recognition, Proc. AVBPA 2003, LNCS 2688, 62-69, Springer
A. Bronstein, M. Bronstein, R. Kimmel and A. Spira, Face recognition from facial surface metric, Proc. ECCV 2004, 225-237
A. Bronstein, M. Bronstein, R. Kimmel and E. Gordon, Fusion of 2D and 3D information in 3D face recognition, Proc. ICIP 2004
M. Bronstein, A. Bronstein and R. Kimmel, Three-dimensional face recognition, CIS Tech. Report 04, 2004, submitted to IJCV
M. Bronstein, A. Bronstein and R. Kimmel, Expression-invariant representation of faces, submitted to PNAS
A. Bronstein, M. Bronstein and R. Kimmel, On isometric embedding of facial surfaces into S3, submitted to ScaleSpace
36M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Isometric model for facial expressions
NEUTRAL AVERAGE MIN. MAX
The uncertainty region around the face in the presence of facial expressions is so large that many other faces can fit in.
A. Bronstein, M. Bronstein and R. Kimmel, “Expression-invariant 3D face recognition”, chapter in Face processing: advanced modeling and methods, Chellapa and Zhao (Eds.) to appear
37M. Bronstein | Expression-invariant representation of faces and its applications for face recognition
Parametrization of S3
1 1 2 3
2 1 2 3
3 1 3
4 3
cos cos cos ;
cos sin cos ;
sin cos ;
sin ;
0, 0,2 0,
x
x
x
x
A. Bronstein, M. Bronstein and R. Kimmel, “On isometric embedding of facial surfaces into S 3”, subm. ScaleSpace
3 4 : 1 x xS R
Geodesic distances
31
1 1 3 1 3 2 2
3 3 1 1 1 3
, cos ,
cos [cos cos cos cos cos
cos cos sin sin sin sin ]
i j i j
i i j j i j
i j i j i j
d
x xS
top related