partial face recognition
DESCRIPTION
Partial Face Recognition. S. Liao, A. K. Jain, and S. Z. Li, "Partial Face Recognition: Alignment-Free Approach", IEEE Transactions on Pattern Analysis and Machine Intelligence , Vol. 35, No. 5, pp. 1193-1205, May 2013, doi : 10.1109/TPAMI.2012.191. Cooperative Face Recognition. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/1.jpg)
Partial Face Recognition
S. Liao, A. K. Jain, and S. Z. Li, "Partial Face Recognition: Alignment-Free Approach", IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, No. 5, pp. 1193-1205, May 2013, doi: 10.1109/TPAMI.2012.191
![Page 2: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/2.jpg)
Cooperative Face Recognition
• People stand in front of a camera with good illumination conditions.
• Border pass, access control, attendance, etc.
![Page 3: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/3.jpg)
Unconstrained Face Recognition
• Images are captured with less user cooperation, in more challenging conditions
• Video surveillance, hand held system, etc.
![Page 4: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/4.jpg)
Partial Faces in Unconstrained Environments
![Page 5: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/5.jpg)
Face Recognition and the London RiotsSummer 2011
Widespread looting and rioting:
Extensive CCTV Network:
FR lead to many arrests:
Yet, many suspects still unable to be identified by COTS FRS:
![Page 6: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/6.jpg)
Face Detection in a Crowd
Normalized Pixel Difference (NPD) Face DetectorOpenCV Viola-Jones Face DetectorPittPatt-5 Face Detector
![Page 7: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/7.jpg)
Unconstrained Face Recognition• Problem:– Recognize an arbitrary face image captured in unconstrained
environment• Possible areas for improvement:– Face detection?– Alignment?– Feature representation?– Classification?
• Importance:– Recognize a suspect in crowd– Identify a face from its partial image
![Page 8: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/8.jpg)
Alignment Free Partial Face Recognition (PFR)
• Proposed alignment-free method: MKD-SRC
![Page 9: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/9.jpg)
Alignment Free Partial Face Recognition (PFR)
• Multi Keypoint Descriptors (MKD)– Each image is described by a set of
keypoints and descriptors (e.g. SIFT):• Keypoints: p1, p2, …, pk• Descriptors: d1, d2, …, dk
– The number of descriptors, k, may be different from image to image
![Page 10: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/10.jpg)
Alignment Free Partial Face Recognition (PFR)
![Page 11: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/11.jpg)
Sparse Representation Classification (SRC) based on MKD
• Descriptors from the same class c can be viewed as a sub-dictionary:
• Combining sub-dictionaries: • For each descriptor yi of
in a probe image, solve
• Determine the identity of the probe image by SRC:
![Page 12: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/12.jpg)
Sparse Representation Classification (SRC) based on MKD
![Page 13: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/13.jpg)
An Example Solution
• MKD-SRC is more discriminant for PFR
• The horizontal axis represents the index of the gallery keypoint descriptors
• The vertical axis denotes the coefficient strength, as computed by
Morgan Freeman
Quincy Delight Jones
![Page 14: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/14.jpg)
Large Scale Partial Face Recognition
• In the dictionary, the number of atoms, K, can be of the order of millions
• Fast atom filtering: (*)
For each yi, we filter out only T (T<<K) atoms according to the top T largest values in ci, resulting in a small sub-dictionary.
• The computation of Eq. (*) is linear w.r.t. K, the selection of the largest T values can be done in O(K), thus the proposed fast atom filtering scales linearly w.r.t. K, while the remaining computation of l1 minimization takes a constant time.
![Page 15: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/15.jpg)
Effects of the Fast Atom Filtering
• A subset of FRGCv2, with 1,398 gallery images and 466 probe images, resulting in K=111,643 for the dictionary.
![Page 16: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/16.jpg)
Keypoint Descriptors• Scale Invariant Feature Transform (SIFT)– Advantage: promising results, efficient to compute– Disadvantage: limited number of keypoints (~80), not affine
invariant• Gabor Ternary Pattern (GTP) descriptor– Adopts edge based affine invariant keypoint detector called
CanAff, which provides sufficient number of keypoints (~800) for PFR
– Robust to illumination variations and noises– Even with fast atom filtering, run time is O(n2) with keypoints
per image• 10 times more keypoints, 100 times slower
![Page 17: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/17.jpg)
Keypoint Descriptors
SIFT(37)
GTP(first 150 of 571)
![Page 18: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/18.jpg)
GTP Descriptor
![Page 19: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/19.jpg)
• Normalize the detected region to 40x40 pixels• Clipped Z-Score normalization:
– Normalize the pixel values to [0,1]– Reduce the influence of illumination variation– Reduce the influence of extreme pixel values
Keypoint Region Normalization
![Page 20: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/20.jpg)
Gabor Filters
• Odd Gabor filters with small scale, 4 orientations– Imaginary part of Gabor filters, sensitive to edges and
their locations. – Scale 0, 5x5 support area, 0º, 45º, 90º, 135º
![Page 21: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/21.jpg)
• Encode the responses of the 4 Gabor filters– Local structure about the responses of Gabor
filters in 4 orientations
– Examples of some local structures encoded
4 orientations
Local Ternary Pattern
2201 2011 0222
![Page 22: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/22.jpg)
Building the descriptor
• Calculate the histogram of local ternary patterns (34 bins) over each grid cell, and concatenate them to form a 1,296 element vector
• Transform by a sigmoid function ( tanh(20x) )– Reduce the influence of extreme values
• Reduce the dimension to 128 by PCA
![Page 23: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/23.jpg)
GTP Descriptor
Local patch of 40x40 pixels4x4 grid cells34 bins for each cell
1296 bins in total PCA to 128 dims
![Page 24: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/24.jpg)
Labeled Faces in the Wild (LFW)1
• Real faces from the internet, most with non-frontal views or occlusion
• 13,233 images of 5,749 subjects
1 http://vis-www.cs.umass.edu/lfw/
![Page 25: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/25.jpg)
Experiments on LFW• MKD-SRC performs better than FaceVACS, but is not as
good as PittPatt• Fusion of MKD-SRC & PittPatt improves performance
![Page 26: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/26.jpg)
Experiments on LFW
Face image pairs that can be correctly recognized by MKD-SRC but not by PittPatt at FAR=1%
![Page 27: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/27.jpg)
Experiment on PubFig Database2
• Large-scale open-set identification• Gallery: 5,083 full frontal faces• Probe:– 817 partial faces (belong to gallery) with large pose
variation or occlusion– 7,210 faces as impostors (do not belong to gallery)
2 http://www.cs.columbia.edu/CAVE/databases/pubfig/
![Page 28: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/28.jpg)
Experiment on PubFig Database
• Proposed MKD-SRC method is better than two commercial SDKs, FaceVACS and PittPatt
![Page 29: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/29.jpg)
Synthetic Partial Face Image Generation
5/28/2013 29
Rotate images; degree of rotation randomly drawn from a normal distribution (mean 0, std. dev. 10º)
Sample width and height for the patch, drawn from a uniform distribution from 50-100% of original size
Sample a starting position for the patch
Randomly rescale the patch
Rotated (size reduced for
display)
Original size patch
Rescaled patch
Original(size reduced for
display)
![Page 30: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/30.jpg)
FRGC+ Dataset
• Open set recognition• FRGC dataset• Gallery:
– 466 FRGC Images– 10,000 PCSO Images
• Probe– A. 15,562 FRGC partial faces
(matching the FRGC subjects in gallery)
– B. 10,000 PCSO partial faces (not matching any gallery subjects)
• Average time per probe image ~1 second vs. 10,466 image gallery
• Pittpatt 5.2 fails to enroll ~50% of the partial faces
![Page 31: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/31.jpg)
Experiment on MOBIO database3
• Videos captured by mobile phone from six universities/institutes in Europe
• 4,880 videos of 61 subjects for verification
Gallery (top) and probe (bottom)3 http://www.idiap.ch/dataset/mobio
![Page 32: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/32.jpg)
32
Experiment on MOBIO database
A. Female B. Male
![Page 33: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/33.jpg)
Experiment on the Mobile dataset
• Unconstrained face images with a mobile phone– Pose, illumination, expression, occlusion or invisible parts
• Gallery images of 14 subjects plus additional 1,000 background subjects; one image/subject
• Probe: 168 mobile phone images of 14 subjects, with additional 1,000 impostors
• Open-set (watch-list) identification experiment
![Page 34: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/34.jpg)
PittPatt cannot be applied because the probe faces cannot be aligned
Experiment on the Mobile dataset
![Page 35: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/35.jpg)
Other Keypoint Matching Methods
• Keypoint based representations are naturally variable size
• The previously discussed method reconstructs each probe keypoint from the gallery using SRC
• Other options:– Bag of words methods – fixed sized representation
over a dictionary – Modified Hausdorff Distance – apply a general
distance metric to sets of points
![Page 36: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/36.jpg)
Modified Hausdorff Distance
• Given a distance metric d, and 2 sets of keypoints A and B find:– D(A,B) = mean(mina in A(d(a,B)))• Compute the min distance from each keypoint in A to a
keypoint in B, average the results over all keypoints in A• D(A,B) ≠ D(B,A)
– MHD(A,B) = max(D(A,B), D(B,A))• We calculate all probe to gallery keypoint
distances for the atom filtering step, so computing MHD is not costly
![Page 37: Partial Face Recognition](https://reader036.vdocuments.mx/reader036/viewer/2022062501/56816909550346895de01cb3/html5/thumbnails/37.jpg)
Summary
• Face recognition based on applying SRC to local keypoint descriptors
• Outperformed by other methods for mugshot style images, but can be used even when faces cannot be aligned – E.g. only part of the face is available, or face/eye
detection fail