an illumination invariant face recognition system for access control using video
DESCRIPTION
An Illumination Invariant Face Recognition System for Access Control using Video. Ognjen Arandjelovi ć Roberto Cipolla. Funded by Toshiba Corp. and Trinity College, Cambridge. Eigenfaces. 3D Morphable Models. Wavelet methods. Face Recognition. - PowerPoint PPT PresentationTRANSCRIPT
An Illumination Invariant Face An Illumination Invariant Face Recognition System for Access Control Recognition System for Access Control using Videousing Video
Ognjen ArandjelovićRoberto Cipolla
Funded by Toshiba Corp. and Trinity College, Cambridge
Face RecognitionFace Recognition
• Single-shot recognition – a popular area of research since 1970s
• Many methods have been developed
• Bad performance in presence of:
– Illumination variation
– Pose variation
– Facial expression
– Occlusions (glasses, hair etc.)
Eigenfaces
Wavelet methods3D MorphableModels
Face Recognition from VideoFace Recognition from Video
• Face motion helps resolve ambiguities of single shot recognition – implicit 3D
• Video information often available (surveillance, authentication etc.)
Recognition setup Training stream Novel stream
Face ManifoldsFace Manifolds• Face patterns describe manifolds which are:
– Highly nonlinear, and
– Noisy, but
– Smooth
Facial features Face pattern manifold Face region
Limitations of Previous WorkLimitations of Previous Work
• In this work we address 3 fundamental questions:
– How to model nonlinear manifolds of face motion
– How to achieve illumination and pose robustness
– How to choose the distance measure
?
Face Motion Manifolds: RevisitedFace Motion Manifolds: Revisited
Unchanging identity, changing illumination
Changing identity, unchanging illumination
• Motivation: How can we use the prior knowledge on the shape of the manifolds?
Pose ClustersPose Clusters• Face motion manifolds are nonlinear, but:
– Low-dimensional (c.f. registration for the reduction of the dimensionality), and
– Key observation: can be described well using only 3 linear pose clusters
Colour-coded pose clusters for 3 manifolds
Determining Pose ClustersDetermining Pose Clusters• Pose clusters are semantic clusters:
– K-means and similar algorithms are unsuitable
– We are using a simple method based on the motion parallax
– Membership decided based on Maximum Likelihood
0.5 reye leye rnostril lnostril
reye leye
x x x xx x
Pupils
Discrepancy η
Image plane
Yaw measure
Distribution for 3 clusters
Pose Clusters: ExamplePose Clusters: Example
Input manifold and colour-coded pose clusters
Sample frames fromthe 3 pose clusters
Illumination compensationIllumination compensation• Performed in two stages:
– Coarse illumination compensation (exploiting face smoothness)
– Fine illumination compensation (exploiting low dimensionality of the face illumination subspace)
RGIC Optimization
Reference Cluster
Illumination Subspace
Input Output
Region-based GICRegion-based GIC
*
2*
,
*
arg min ( , ) ( , )
( , )
Cx yI x y I x y
I I x y
Gamma Intensity Correction (GIC)
Canonical image
• Region-based GIC (RGIC): faces are (roughly) divided into regions with smoothly varying surface normal
Solved by 1D non-linear optimization
1 2
3 4
Face regions
Varying Gamma
Region-based GIC: ArtefactsRegion-based GIC: Artefacts
• Region-based GIC suffers from artefacts at region boundaries
Mean face γ value map Smoothed γ map
Input face RGIC face Our method
Boundary artefacts
Artefactsremoved
Illumination SubspaceIllumination Subspace• Each input frame corrected for a linear Pose Illumination Subspace
component to match the reference distribution of the same pose
– Illumination subspace is high-dimensional
– Constrained to expected variations by Mahalanobis distance
Input manifold
Reference manifold
* *
*
Subject to:
arg min
Where ... is the Mahanalobis distance
in the reference Gaussian
I
I M
M
I I B a
a I B a
Illumination Subspace
Illumination Compensation ResultsIllumination Compensation Results
Original/input frames
Illumination-correctedframes
Reference frames
Strong side lighting
And in face pattern space…
Comparing Pose ClustersComparing Pose Clusters
• “Distribution-based” distances (Kullback-Leibler divergence, Resistor Average Distance etc.) unsuitable
• We use the simple Euclidean distance between cluster centres
Reference cluster
Novel cluster
Reduced spread
Clustercentres
Unified Manifold SimilarityUnified Manifold Similarity• Recognition based based on the likelihood ratio:
1,2,3
1,2,3
( | )( | )P D sP D s
Manifolds belong to the same person
Distances between pose clusters• Learn likelihoods from ground truth training data
Likelihoodhistogram
Undefined value regions
RBF-interpolatedlikelihood
Two-pose interpolatedlikelihood
Likelihood nowmonotonically decreasing
Face Video Database RevisitedFace Video Database Revisited
• Testing performed under extreme, varying illuminations
10 illumination conditions used (random 5 for training, others for testing)
RegistrationRegistration• Linear operations on images are highly nonlinear in the pattern space
• Translation/rotation and weak perspective can be easily corrected for directly from point correspondences
– We use the locations of pupils and nostrils to robustly estimate the optimal affine registration parameters
Translationmanifold
Skewmanifold
Rotationmanifold
Registration Method UsedRegistration Method Used• Feature localization based on the combination of shape and pattern matching (Fukui et al.
1998)
Detect features
Crop & affineregister faces
ResultsResults
• Very high recognition rates attainted (96% average) under extreme variations in illumination
• Other methods showed little to no illumination invariance
Results, continuedResults, continued• The method was shown to give promising results for authentication uses:
– Good separability of inter- and intra- class manifold distances was found
– It can provide a secure system with only 0.1% false positive rate and 8% false negative rate
Cumulative distributionsof inter- and intra- class
manifold distances
The ROC curve forthe proposed method
Future ResearchFuture Research
• Non-constant illumination within a single sequence causes problems
• Illumination compensation is still not perfect – pose illumination subspaces have unnecessarily high dimensions
• Pose estimation is too primitive – outliers cause problems in estimation of linear subspaces
• Complete pose invariance is still not achieved (what if there are no corresponding pose clusters?)
For suggestions, questions etc. please contact me at: [email protected]