manifold models for video-based face recognition - · pdf file ·...
TRANSCRIPT
![Page 1: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/1.jpg)
1
Manifold models for video-based face recognition
Pavan Turaga Rama Chellappa
![Page 2: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/2.jpg)
2
Manifold models of appearance and motion and its application in Video-based FR
Illumination
Shape and Reflectance
Motion
Image Space
![Page 3: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/3.jpg)
3
Constraints as Manifolds
Examples in high-dimensional
spaces
1. Images 2. Patches
Features And Models
1. Subpaces 2. Linear Dynamic
Models
Manifold Learning Riemannian/Differential Geometry
![Page 4: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/4.jpg)
How manifolds can help?
Part 1 • Appearance Models for Detection and Tracking
– Probabilistic Manifold models of faces – Shape-illumination manifold
Part 2 • Joint appearance and motion models
– Image-set modeling using subspaces – Video modeling using dynamical models
4
![Page 5: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/5.jpg)
Manifold learning: Formal tools to address questions of intrinsic dimensionality
5
Pose-Illumination manifold Pose-expression manifold
J. B. Tenenbaum, V. de Silva and J. C. Langford, A Global Geometric Framework for Nonlinear Dimensionality Reduction Science 290 (5500): 2319-2323, 22 December 2000 . Sam Roweis & Lawrence Saul, Nonlinear dimensionality reduction by locally linear embedding. Science v.290 no.5500, Dec.22, 2000. pp.2323--2326 He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3): 328-340 (2005)
![Page 6: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/6.jpg)
6
Riemannian and Differential manifolds
These mappings are called charts, and the set of charts together is called an Atlas
Topological Space: A set X with a collection T of subsets of X with some properties. Haussdorff: Given 2 points, there exist non-overlapping neighborhoods Second Countable: The whole space can be spanned by a system of ‘bases’. e.g. open intervals for the real-line Each neighborhood is homeomorphic with a Euclidean space Topological spaces with these properties are called manifolds
![Page 7: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/7.jpg)
Two strands of research: Similarities and Differences
7
Concept Manifold Learning Differential Geometry
Charts Estimate from a dense sampling of the manifold.
Start with analytically defined charts e.g. stereographic projections of a sphere.
Distance on manifold Shortest path-length on a graph.
Shortest ‘geodesic’ connecting two points.
Point-to-manifold distance
Hard to compute in general.
Hard to compute in general.
Statistics Classical statistics in the charts.
Karcher mean, Karcher variance etc.
![Page 8: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/8.jpg)
Part 1 Example-based manifold models for detection, tracking, and recognition
8
![Page 9: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/9.jpg)
Appearance Manifolds, Murase and Nayar
9
Hiroshi Murase, Shree K. Nayar: Visual learning and recognition of 3-d objects from appearance. International Journal of Computer Vision 14(1): 5-24 (1995) Introduced the idea that high dimensional image data potentially has fewer degrees of freedom, and showed how to exploit it for recognition.
![Page 10: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/10.jpg)
The Lambertian reflectance model and Illumination Cone (1998)
10 What Is the Set of Images of an Object under All Possible Illumination Conditions? P. N. Belhumeur, and D. J. Kriegman, IJCV 1998.
![Page 11: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/11.jpg)
Manifolds as appearance models for detection/tracking
• Appearance model: A description of what is a ‘face’. • Should encode allowable variations: pose, expression,
illumination. • Simpler appearance models
– A static image – A few images with poses, expressions, etc.
• What do manifolds afford – A joint representation of pose, illumination, expression – A formal way of parameterizing observed variations
11
![Page 12: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/12.jpg)
How to create a manifold appearance model?
12
Estimate person-specific manifold
Manifold of Person A
Manifold of Person B Test video
![Page 13: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/13.jpg)
Tools from manifold learning
13
Image courtesy Wikipedia
Many tools now available LLE, Isomaps, Laplacian Eigenmaps, Diffusion maps etc Each method preserves different Properties e.g. global distances, local structures, etc.
J. B. Tenenbaum, V. de Silva and J. C. Langford, A Global Geometric Framework for Nonlinear Dimensionality Reduction Science 290 (5500): 2319-2323, 22 December 2000 . Sam Roweis & Lawrence Saul, Nonlinear dimensionality reduction by locally linear embedding. Science v.290 no.5500, Dec.22, 2000. pp.2323--2326 He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces. IEEE Trans. Pattern Anal. Mach. Intell. 27(3): 328-340 (2005)
![Page 14: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/14.jpg)
Tracking as manifold-based search
14
Person-specific manifold
Sub-image of larger image
Kuang-Chih Lee, Jeffrey Ho, Ming-Hsuan Yang, David J. Kriegman: Visual tracking and recognition using probabilistic appearance manifolds. Computer Vision and Image Understanding 99(3): 303-331 (2005)
![Page 15: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/15.jpg)
Problems that arise in face detection and tracking
15
What is the distance of this new point to each manifold?
• Out of sample extension: Find parameters of a new point.
• Reconstruct ambient data point from low dimensional embedding.
These are relatively new research directions. A few related papers: Yoshua Bengio, Jean-François Paiement, Pascal Vincent, Olivier Delalleau, Nicolas Le Roux, Marie Ouimet: Out-of-Sample Extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. NIPS 2003. Chen Haojun, Jorge Silva, David Dunson and Lawrence Carin. 2010. "Hierarchical Bayesian Embeddings for Analysis and Synthesis of Dynamic Data." 2010 AAAI Fall Symposium Series. S. Gerber, T. Tasdizen, and R. Whitaker. Dimensionality reduction and principal surfaces via kernel map ma nifolds. In Computer Vision, 2009 IEEE 12th International Conference on, page 529--536, 2009.
![Page 16: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/16.jpg)
Recognition as ‘closest manifold’ search
• Do not know the person specific manifold explicitly • Populate with examples • Find closest manifold on a recursive fashion. • A hard optimization problem • Split into two simpler problems
16
Detection sub-problem
Recognition sub-problem
![Page 17: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/17.jpg)
Core problem: Distance of a point to a manifold
17
How to solve this, when only a finite, coarse sampling of the manifold is available?
![Page 18: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/18.jpg)
Probabilistic interpretation: Frame-by-frame recognition
18
Computing the distance of an image patch to the kth manifold using ‘average’ distance to manifold
Using the distance to manifold to obtain a posterior pdf over identities
![Page 19: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/19.jpg)
Simplification using union-of-subspaces
19
Difficult to evaluate
Easy to evaluate (distance of point to plane)
Evaluate using temporal coherence
![Page 20: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/20.jpg)
Imposing temporal coherence
20
Learn subspace-to-subspace transition matrix from training data for each subject
![Page 21: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/21.jpg)
Recognition and tracking
21
Detection sub-problem Solve using distance-to manifold + particle-filter approach
Recognition sub-problem Solve using collection-of-subspace And temporal coherence model
![Page 22: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/22.jpg)
Sample Results
22
![Page 23: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/23.jpg)
Is there a generic constraint on human faces?
• Given a specific identity and pose (ensured by the re-illumination algorithm), the log-difference images encode variations due to shape and illumination.
• Hypothesis: This set of log-difference images live on a smooth manifold, and is ‘generic’ to human faces !
• Named the ‘Shape-Illumination Manifold’ (SIM). • Motivated by ‘generic-shape different-albedo’ assumptions from
literature.
23
Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)
![Page 24: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/24.jpg)
Consider the same identity and pose
24
Image Albedo Shape + Illumination
Only a function of shape+illumination Log-Difference Images
![Page 25: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/25.jpg)
How to use this hypothesis in practice?
25
Data in the image-space encodes variations from pose, illumination, shape etc. Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)
![Page 26: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/26.jpg)
Matching points and manifold neighborhoods
26
Motion Manifold 1 (Video 1)
Motion Manifold 2 (Video 2)
Identify correspondences by taking into account nearest manifold matches.
![Page 27: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/27.jpg)
How to use manifold neighborhoods to regularize correspondences ?
27
![Page 28: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/28.jpg)
Face re-illumination using pose correspondences
• Given two video sequences of the same subject in different illuminations, can one re-illuminate the first video using the second?
• Yes, if you can find good correspondences in terms of pose.
28
Original Face
Re-illuminated face Ognjen Arandjelovic, Roberto Cipolla: Face Recognition from Video Using the Generic Shape-Illumination Manifold. ECCV (4) 2006: 27-40 Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)
![Page 29: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/29.jpg)
Details of Cost Functions
• Edge-based distance maps used to measure pose-error.
• Estimating geodesic distances using Floyd’s algorithm.
• Genetic algorithm to optimize correspondences.
29
![Page 30: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/30.jpg)
Sample Re-illumination results
30
(a) Original images from a novel video sequence and (b) the result of re-illumination using the proposed genetic algorithm with nearest neighbour-based reconstruction.
![Page 31: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/31.jpg)
Generic manifold constraint for identity matching
• Input: Several generic videos of same person in multiple illumination conditions.
• For each video, re-illuminate with video of potential match. • This ensures pose-correspondence as well. • Use re-illuminated and pose-corresponding images to obtain log-
difference images. • Estimate a manifold from these log-difference images.
31
![Page 32: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/32.jpg)
Video matching with the generic SIM model
• Video 1 from Gallery, Video 2 from person to be recognized. • Assuming they are from the same person, apply re-illumination
algorithm. • Use obtained pose-correspondences, to obtain log-likelihood
images. • If the two videos are from the same person, this set of images
will ‘lie close’ to the generic shape-illumination manifold. • Question: How to measure closeness of points to a manifold?
32
![Page 33: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/33.jpg)
How close are a given set of points to a manifold?
• Technical problem: Measure point-to-manifold distance. • No easy solution, very hard as already seen before. • Possible solutions: Manifold learning + out-of-sample extension. • Proposed solution: Fit GMM to Shape-illumination manifold,
and evaluate likelihood of the new points under the GMM.
33
![Page 34: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/34.jpg)
Algorithms for training and recognition
34
![Page 35: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/35.jpg)
Illustrative Results
35
![Page 36: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/36.jpg)
Part 2 Differential geometric approach for joint
appearance and motion modeling
36
![Page 37: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/37.jpg)
37
An analytic model for face appearance variation
• Facial dynamics modeled as LTI system
),0(~)(),()()1(),0(~)(),()()(QNtvtvtAztzRNtwtwtCztf
+=+
+=
A,C,Q,R )0(z ),2(),1(),0( fff
Gaurav Aggarwal, Amit K. Roy Chowdhury, Rama Chellappa: A System Identification Approach for Video- based Face Recognition. ICPR (4) 2004: 175-178. Pavan K. Turaga, Ashok Veeraraghavan, Anuj Srivastava, Rama Chellappa: Statistical Computations on Grassmann and Stiefel Manifolds for Image and Video-Based Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(11): 2273-2286 (2011)
![Page 38: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/38.jpg)
38
What do these matrices represent
dxdd Atz R,R)( ∈∈
nxdn Ctf R,R)( ∈∈
![Page 39: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/39.jpg)
39
Model Fitting
• Learn the (A,C,Q,R) given a sequence of outputs • Asymptotically Optimal Solution: EM • Other approaches include N4SID • Both are computationally expensive for large feature spaces • Sub-optimal approach : PCA-ID
![Page 40: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/40.jpg)
40
Distance measures on models
• Comparing two dynamical models can be done by comparing the column space of the extended observability matrix.
),0(~)(),()()1(),0(~)(),()()(QNtvtvtAztzRNtwtwtCztf
+=+
+=
)0(
.
...)0()0()0(
.
.)]2([)]1([)]0([
.
.)2()2()1()1()0()0(
.
.)2()1()0(
22 zCACAC
zCACAzCz
zCEzCEzCE
wCzwCzwCz
Efff
E
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
=
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
+
+
+
=
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
![Page 41: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/41.jpg)
41
Recognition using dynamic models: Compare column spans of observability matrix
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
.
.
211
11
1
ACACC
),0(~)(),()()1(),0(~)(),()()(
11
11
QNtvtvtzAtzRNtwtwtzCtf
+=+
+=),0(~)(),()()1(),0(~)(),()()(
22
22
QNtvtvtzAtzRNtwtwtzCtf
+=+
+=
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
.
.
222
22
2
ACACC
Compare column span
Key concept is distance between subspaces
![Page 42: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/42.jpg)
Subspace models of face recognition is a special case
42
![Page 43: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/43.jpg)
43
The Grassmann Manifold
• Grassmann Manifold: The space of m dimensional subspaces in RD
Image courtesy Jihun Hamm
![Page 44: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/44.jpg)
44
Distance using Subspace angles
P subspace
p basis lorthonorma
angle principal|)(|cos),( 211
21 == − ppPPd T
T
k
T VUYY⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
=
σ
σ
σ
2
1
21
Cosines of principal angles
![Page 45: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/45.jpg)
45
Formal definitions
• Topological Space: A set X with a collection T of subsets of X with some properties.
• Haussdorff: Given 2 points, there exist non-overlapping neighborhoods • Second Countable: The whole space can be spanned by a system of ‘bases’. e.g. open
intervals for the real-line • Each neighborhood is homeomorphic with the euclidean space • Topological spaces with these properties are called manifolds
![Page 46: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/46.jpg)
46
Differential manifolds: Smooth Mappings
![Page 47: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/47.jpg)
47
Simplest case
• Lets consider Rn.
• Take two open intervals U1 and U2 and consider the chart-changing
• In fact all vector-spaces are manifolds.
nnxx RR →= :,)( φφ
xxFF nn
=
→= −
)(:2
11 RRφφ
![Page 48: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/48.jpg)
48
The case of the sphere
ζξ
φφ
/1)(:2
11
=
→= −
FF CC
![Page 49: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/49.jpg)
49
Tangent Spaces
f
γ
)]0(')[(
))((: 0
γ
γξ
xfD
tfdtdf tx
=
= =
![Page 50: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/50.jpg)
50
Submanifolds
• Consider a subset P of a points of a manifold M, • P = {X in M| f(X) = 0}. • If the function f() satisfies certain conditions, then the set P is a
submanifold of M. • Conditions: f() is ‘full-rank’. i.e. its Jacobian is full rank. • Then, dim(P) = dim(M) – rank(J).
• Example, consider F:Rn,k → Sym(k), given by F(X) = XTX – I. • Then, Stiefel = F-1(0) • Can prove that F is full rank i.e. there exists a Y, for every Z
such that DF(X)[Y] = Z. • Thus, Stiefel is a submanifold of Rn,k
![Page 51: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/51.jpg)
51
Quotient Spaces
• Define an equivalence relation on a manifold M i..e ‘~’. • We say, X~Y if X is equivalent to Y. • Then, P = M/~ is called a Quotient. • P is a manifold if ~ satisfies some properties. • To each ~ we define a projection. • Example, Stiefel(n,k) = O(n)/O(n-k). • Example, Grassmann(n,k) = Stiefel(n,k)/O(k)
![Page 52: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/52.jpg)
52
What can we do once we are on a manifold
• Optimization • Classification and Regression • Interpolation • Pretty much anything you can do on vector spaces.
• But…. need some tools.
![Page 53: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/53.jpg)
53
Retractions
![Page 54: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/54.jpg)
54
Example
VPVPVRP +
+=)(
![Page 55: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/55.jpg)
55
Exponential and Logarithmic Maps
• Special form of retraction that maps to geodesics )sin()cos()( V
VVVPVExpP +=
![Page 56: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/56.jpg)
56
Intrinsic Statistics
• Intrinsic mean
• Sample based Karcher mean – maximum likelihood estimate for the intrinsic mean
for weighted samples
2argmin ( , ) ( )dM M
d X p X Xψ
µ ψ∈
= ∫
![Page 57: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/57.jpg)
57
Geometry of Grassmann manifold
• Best studied in terms of SO(n) i.e. Special Orthogonal Group
• The exp map for SO(n) is the standard matrix exponential.
matrices symmetric-skew are at )( totangents0tangenttangent
0)0()0(
0at t Evaluate0)()()()(
)()(I Q(0) with Q(t)by given SO(n)in curve aConsider
InSO
tQtQtQtQItQtQ
T
T
TT
T
⇒
=+=>
=+⇒
=
=+
=
=
![Page 58: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/58.jpg)
58
Grassmann and Stiefel as quotient of SO(n)
• SO(n): nxn orthonormal matrix
• Stiefel: Quotient space of SO(n)
Q =
Y =[Q]k =
First k
![Page 59: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/59.jpg)
59
Grassmann as Quotient of SO(n)
• Grassmann is a quotient of Stiefel, and thereby also of SO(n).
Y =[Q]k =
First k
G =[Y]= span(Y)
[ ]
[ ]
Stiefel(k) k)-O(n)/O(n
equiv. are)(
*|,
.|)(
,
,,,,
,,
=
⎥⎦
⎤⎢⎣
⎡
−
=
−
−−
−
knYYSuppose
YYn
kkn
knkkkknnkn
knnkn
OZZI
O
![Page 60: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/60.jpg)
60
Tangents of Quotient Spaces
• Any direction which takes you to a different equivalence class is a valid tangent to the quotient space.
How to compute exp and log maps for Grassmann manifold ? K. Gallivan, A. Srivastava, X. Liu, and P. VanDooren, “Efficient Algorithms for Inferences on Grassmann Manifolds,” In 12th IEEE Workshop Statistical Signal Processing, St. Louis, USA, October 2003.
t!Oexp(tA)J
![Page 61: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/61.jpg)
61
Statistics on manifolds: Karcher Mean
Imposing pdfs on manifolds • Choose a ‘Pole’ • Unwrap the points on manifold about the pole. • Estimate pdf on the tangent plane. • Wrap back the tangent plane pdf onto the manifold.
![Page 62: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/62.jpg)
62
Procedure for estimating pdfs
• Choose a ‘Pole’ • Unwrap the points on
manifold about the pole. • Estimate pdf on the
tangent plane. • Wrap back the tangent
plane pdf onto the manifold.
• Issues: Which pole to choose ?
• The ‘Karcher’ mean is the most popular choice.
n = 2, d = 1
![Page 63: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/63.jpg)
63
Extrinsic Metric: Procrustes Distance
)()( min),( 212121 pppptrPPd T ααα
−−=
1 and where,P subspace
=ℜ∈
=
pp
α
α
P subspace
p basis
![Page 64: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/64.jpg)
64
Procrustes Metric
)()(min),( 2121212 RXXRXXtrXXd T
R−−=
. where),(),( 21212 XXAAAItrXXd TT
k =−=
• The Procrustes distance on Stiefel manifold can be derived from the Procrustes representation.
• When R varies over all kxk real matrices, the distance is given by
![Page 65: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/65.jpg)
65
Extrinsic estimate of pdf
• Given several points per class, kernel methods on Grassmann
manifold can be used to estimate class conditional densities with extrinsic divergence measures.
∑=
−− −=n
ii
TTik MXXXXIMKMC
nMXf
1
2/12/1 ])([)(1);(
![Page 66: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/66.jpg)
66
Video-based FR: MBGC experiment
![Page 67: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/67.jpg)
Pose-specific linear subspaces
Pose specific linear subspace
![Page 68: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/68.jpg)
68
Illumination invariant face recognition
![Page 69: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/69.jpg)
Manifold constraints have been exploited in many more biometrics applications
• Face Recognition • Yui Man Lui, J. Ross Beveridge: Grassmann Registration Manifolds for Face
Recognition. ECCV (2) 2008: 44-57 • Mehrtash Tafazzoli Harandi, Conrad Sanderson, Sareh Abolahrari Shirazi, Brian C.
Lovell: Graph embedding discriminant analysis on Grassmannian manifolds for improved image set matching. CVPR 2011: 2705-2712
• Jihun Ham, Daniel D. Lee: Grassmann discriminant analysis: a unifying view on subspace-based learning. ICML 2008: 376-383
• We present a few more examples illustrating the benefits in other biometric applications next.
![Page 70: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/70.jpg)
70
Landmark based Facial Geometric Modeling for Age Estimation and Verification
Age estimation as regression on Grassmann shape manifold
Verification as two-class classification of tangent-vectors of shape manifold
![Page 71: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/71.jpg)
71
Landmarks and their representation
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
nn yx
yxyx
..22
11 ⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
2221
121122
11
.. aaaa
yx
yxyx
nn
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
2221
121122
11
.. aaaa
yx
yxyx
nn
Same Column Space
Pose and Camera Variations approximated
By Affine transforms New
Landmarks
Single point on Grassmann
manifold ⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
2221
121122
11
.. aaaa
yx
yxyx
nn
Landmarks Base Face
![Page 72: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/72.jpg)
72
Results
Cumulative Error on age estimation on FG-Net data
Verification results on FG-Net dataset.
T. Wu, P. Turaga, R. Chellappa, Age Estimation and Face Verification Across Aging using Landmarks, accepted at IEEE Transactions on Information Forensics and Security. 2012.
![Page 73: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/73.jpg)
Blur-invariant Face Recognition
R. Gopalan, S. Taheri, P. Turaga, R. Chellappa, “A Blur-robust Descriptor with Applications to Face Recognition”, PAMI, vol. 34, no. 6, 2012.
Blurred image
Clean image Unknown blur kernel
Noise 2-D Pixel location
Image formation model under blur
Face images under no blur, medium blur, and extreme blur
),(),)(*(),(' 212121 nnnnkynny η+=
![Page 74: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/74.jpg)
Space of blurred images ? Y=[(y*φ1)v (y*φ2)v . . . (y*φN)v]
Y’=[(y’*φ1)v (y’*φ2)v . . . (y’*φN)v]
span(Y) = span(Y’)
Skyy *'=
φ̂̂YY = φ̂ˆˆ' SKYY =
Y’=[(y*kS*φ1)v (y*kS*φ2)v . . . (y*kS*φN)v]
![Page 75: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/75.jpg)
Results on FERET dataset
![Page 76: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/76.jpg)
The Image Deblurring Problem
76
y Hx γ= +
22argmin || || ( )optimal
xx y Hx E xλ= − +
Data Prior
![Page 77: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/77.jpg)
Manifold prior for deblurring
77
Patches available in abundance ! Size of ‘patch-space’ is far smaller than image-space. Far easier to populate.
![Page 78: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/78.jpg)
Regularization with Manifold Model
• Let be a patch at location q. • Let be the projection of patch onto the
patch manifold M. • Then the optimization problem is recast as
78
( )qp x( ) Pr ( ( ))M qc q oj p x=
2
* * 2 2
, [0,1]
( , ) argmin || || || ( ) ( ) ||qx c
x c y Hx p x c q dqλ= − + −∫
))(()( 1 cAveryHIHHx TT λλ ++= −
Reconstruction of image by patch averaging.
Peyre 2009.
Jie Ni, Pavan K. Turaga, Vishal M. Patel, Rama Chellappa: Example-Driven Manifold Priors for Image Deconvolution. IEEE Transactions on Image Processing 20(11): 3086-3096 (2011)
![Page 79: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/79.jpg)
Results: UMD MURI data
79
A Gaussian blur kernel with standard deviation of 2 is used.
![Page 80: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/80.jpg)
Conclusion
• The ‘manifold’ benefits in detection, tracking, recognition, regularizing inverse problems etc.
• Technical challenges are many, but potential pay-offs are high. • Current solutions have only scratched the surface with simple
solutions to difficult problems. • Huge scope for generalization and application to other problems. • Closer collaboration between biometrics, machine learning,
applied mathematics will prove beneficial.
80
![Page 81: Manifold models for video-based face recognition - · PDF file · 2012-09-25Manifold models for video-based face ... He, Yan, Hu, Niyogi, Zhang: Face Recognition Using Laplacianfaces](https://reader030.vdocuments.mx/reader030/viewer/2022021509/5a9e6ab47f8b9a0d158b53ca/html5/thumbnails/81.jpg)
References
• Kuang-Chih Lee, Jeffrey Ho, Ming-Hsuan Yang, David J. Kriegman: Visual tracking and recognition using probabilistic appearance manifolds. Computer Vision and Image Understanding 99(3): 303-331 (2005)
• Arandjelovic, O. and Cipolla, R.. Face recognition from video using the generic shape-illumination manifold. In: Proc. 9th European Conference on Computer Vision, Graz (Austria) volume LNCS 3954, pages 27-40, Springer, 2006.
• H. Murase and S.K. Nayar, "Visual Learning and Recognition of 3D Objects from Appearance,",International Journal on Computer Vision,Vol.14, No.1, pp.5-24, Jan, 1995.
• Ognjen Arandjelovic: Computationally efficient application of the generic shape-illumination invariant to face recognition from video. Pattern Recognition 45(1): 92-103 (2012)
• Gaurav Aggarwal, Amit K. Roy Chowdhury, Rama Chellappa: A System Identification Approach for Video-based Face Recognition. ICPR (4) 2004: 175-178.
• Pavan K. Turaga, Ashok Veeraraghavan, Anuj Srivastava, Rama Chellappa: Statistical Computations on Grassmann and Stiefel Manifolds for Image and Video-Based Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 33(11): 2273-2286 (2011)
Related Applications • Yui Man Lui, J. Ross Beveridge: Grassmann Registration Manifolds for Face Recognition. ECCV (2) 2008: 44-57 • Mehrtash Tafazzoli Harandi, Conrad Sanderson, Sareh Abolahrari Shirazi, Brian C. Lovell: Graph embedding discriminant analysis
on Grassmannian manifolds for improved image set matching. CVPR 2011: 2705-2712
• Jihun Ham, Daniel D. Lee: Grassmann discriminant analysis: a unifying view on subspace-based learning. ICML 2008: 376-383
81