ieee transactions on information forensics and …rossarun/pubs/dantchevaeliarosssoftbio… ·...

IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 11, NO. 3, MARCH 2016 441

What Else Does Your Biometric Data Reveal?A Survey on Soft Biometrics

Antitza Dantcheva, Petros Elia, and Arun Ross

Abstract— Recent research has explored the possibility ofextracting ancillary information from primary biometric traitsviz., face, fingerprints, hand geometry, and iris. This ancillaryinformation includes personal attributes, such as gender, age,ethnicity, hair color, height, weight, and so on. Such attributesare known as soft biometrics and have applications in surveil-lance and indexing biometric databases. These attributes can beused in a fusion framework to improve the matching accuracyof a primary biometric system (e.g., fusing face with genderinformation), or can be used to generate qualitative descriptionsof an individual (e.g., young Asian female with dark eyes andbrown hair). The latter is particularly useful in bridging thesemantic gap between human and machine descriptions of thebiometric data. In this paper, we provide an overview of softbiometrics and discuss some of the techniques that have beenproposed to extract them from the image and the video data.We also introduce a taxonomy for organizing and classifying softbiometric attributes, and enumerate the strengths and limitationsof these attributes in the context of an operational biometricsystem. Finally, we discuss open research problems in this field.This survey is intended for researchers and practitioners in thefield of biometrics.

Index Terms— Soft biometrics, biometrics, computer vision,gender, age, ethnicity, race, cosmetics, privacy, semantics, visualattributes.

I. INTRODUCTION

A. Biometrics

B IOMETRICS is the science of recognizing individualsbased on their physical, behavioral, and physiological

attributes such as fingerprint, face, iris, gait and voice [1].A classical biometric system acquires biometric data from anindividual (e.g., a fingerprint image), extracts a set of featuresfrom the data, and compares this feature set with templatesin the database in order to verify a claimed identity or todetermine an identity.

Manuscript received February 6, 2015; revised May 11, 2015; acceptedMay 17, 2015. Date of publication September 22, 2015; date of currentversion December 24, 2015. The work of A. Dantcheva was supportedby a Postdoctoral Fellowship from West Virginia University and MichiganState University, and by an ERCIM “Alain Bensoussan” Fellowship fundedby the European Union Seventh Framework Programme (FP7/2007-2013)under Grant 246016. The work of A. Ross was supported by U.S. NationalScience Foundation CAREER Award IIS 0642554. The associate editorcoordinating the review of this manuscript and approving it for publicationwas Dr. Sebastien Marcel.

A. Dantcheva is with the STARS Team, Institut National de Rechercheen Informatique et en Automatique, Sophia Antipolis 06902, France (e-mail:[email protected]).

P. Elia is with the Mobile Communications Department, Eurecom,Biot 06410, France (e-mail: [email protected]).

A. Ross is with the Department of Computer Science andEngineering, Michigan State University, East Lansing, MI 48824 USA(e-mail: [email protected]).

Color versions of one or more of the figures in this paper are availableonline at http://ieeexplore.ieee.org.

Digital Object Identifier 10.1109/TIFS.2015.2480381

Fig. 1. Anthropometry card of Alphonse Bertillon, who originated thecriminal identification system based on profile and full-face photos, and keybody measurements (1892). These key measurements include body height,body weight, build, complexion, head length, head width, cheek width,measurements of right ear and left foot, as well as “peculiar marks” suchas birthmarks, scars, and tattoos.

While biometric data is typically used to recognizeindividuals, it is possible to deduce other types of attributesof an individual from the same data. For example, attributessuch as age, gender, ethnicity, height, hair color and eye colorcan be deduced from data collected for biometric recognitionpurposes. Recent work [2] has established the possibility ofcomputing the body mass index (BMI) from face images,thereby suggesting the possibility of assessing health frombiometric data.

B. Soft Biometrics

These additionally deduced attributes, while not necessarilyunique to an individual, can be used in a variety of applica-tions. Further, they can be used in conjunction with primarybiometric traits in order to improve or expedite recognitionperformance.

It is perhaps this latter application that has led to theseattributes being referred to as soft biometrics [3]–[5] or lightbiometrics [6]. In this context, soft biometrics can be tracedback to Bertillon [7] (see Figure 1), who brought to the fore theidea of using anatomical, morphological and anthropometricalcharacteristics for person identification. These attributes havealso been referred to as semantics [8], [9], in referenceto their semantic interpretation (e.g., describing a face as“young male”).

1) Scope and Benefits: Various researchers have attemptedto define the scope of soft biometrics. Jain et al. [3] defined

1556-6013 © 2015 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

442 IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, VOL. 11, NO. 3, MARCH 2016

Fig. 2. Importance of soft biometrics. Typical video surveillancescenario: when faces are of low resolution, appear in different poses, andare either occluded or not visible, other attributes such as age, gender, haircolor and style, height, body type, clothes and accessories can be used foridentification and re-identification. Image obtained from PETS 2007 [12].

soft biometrics to be the set of characteristics thatprovide some information for recognizing individuals, butthat are not capable of distinguishing between individuals,mainly due to their lack of distinctiveness and permanence.Samangooei et al. [8], as well as Reid and Nixon [10], furtherassociated soft biometrics with labels which people use todescribe each other: an association that nicely bridges the gapbetween human and machine descriptions of biometric data.

Combining the above with the ideas in Dantcheva et al. [11],and keeping in mind that such soft traits can go beyond personrecognition, one could define soft biometrics as follows. Softbiometric traits are physical, behavioral, or material acces-sories, which are associated with an individual, and which canbe useful for recognizing an individual. These attributes aretypically gleaned from primary biometric data, are classifiablein pre-defined human understandable categories, and can beextracted in an automated manner.

a) Benefits of soft biometrics: Soft biometrics are oftendescriptive and have a semantic representation. In addition - asnoted by Jain et al. [4] - they can be inexpensive to compute,discerned at a distance in a crowded environment, and requireless or no cooperation of the observed subjects. To elaborate,we note the following benefits.

Human understandable interpretation: Soft biometricattributes have a semantic interpretation, in the sense that thatthey can provide a description that can be readily understoodby humans; for example the description “young, tall, female”.This makes them particularly useful in applications such asvideo surveillance, where they are directly compatible withhow humans perceive their surroundings [13]–[20]. In otherwords, when a human attempts to verbally describe a per-son, obvious characteristics regarding the person’s appearancesuch as gender, age, height and clothes color are often used(e.g., in police reports). This allows soft biometrics to beused in applications where traditional biometrics may beinsufficient, as is argued, for example, by Klontz and Jain [21]in the case of the 2013 Boston bombings.

Robustness to low data quality: Some soft biometricattributes can be deduced from low-quality biometric data(see Figure 2. In this context, such attributes can be extracted,when primary biometric data is not conclusive, due to poor

Fig. 3. Ancillary information, referred to as soft biometrics, can be gleanedfrom the following biometric modalities: face, iris, fingerprint, gait, body,hand, voice.

TABLE I

SOFT BIOMETRIC TAXONOMY WITH FOUR GROUPS: I) DEMOGRAPHIC,

II) ANTHROPOMETRIC AND GEOMETRIC, III) MEDICAL,

IV) MATERIAL AND BEHAVIORAL

acquisition quality. For example, if the input iris image isof poor quality, one could utilize the surrounding periocularinformation to perform recognition, rather than relying on theiris itself.

Consent-free acquisition: Soft biometrics can often becaptured without the consent and cooperation of the observedsubject. For example, information about a person’s height orgender can be deduced from a distance.

Privacy: Since soft biometric traits are not distinctive, theyonly provide a partial description of a person (such as “female,tall, young”). This limitation has positive privacy ramificationswhen it comes to extracting and storing such soft biometricdata.

2) Taxonomy: With the aforementioned scope and benefitsin mind, it is worth identifying a taxonomy that can facil-itate organization and categorization of soft-biometric traits.This taxonomy is based on utility, and it considers fourgroups of attributes: demographic, anthropometric, medical,and material and behavioral attributes (see Table I). Thiscategorization - and the more refined sub-categorization basedon the modalities of face, iris, body, gait, fingerprint and hand(Figure 3) - will also help us structure the exposition of thestate-of-art in the rest of this survey paper.

The above taxonomy need not necessarily result in disjointgroups and it is certainly not a unique taxonomy. For example,taxonomies in Jain et al. [4] and Dantcheva et al. [11]

DANTCHEVA et al.: SURVEY ON SOFT BIOMETRICS 443

(see also [22]) have different definitions that categorize traitsbased on their ability to distinguish between individuals aswell as their variability over time.

C. Domains of Application

Automated soft biometric extraction has a number of appli-cations: in the area of security where algorithms can locatea person-of-interest based on a specific set of soft biometricattributes; in image-tagging and video indexing where photoor video album management can be performed based, forexample, on age, gender, and clothing; in human-computer-interaction where data and personalized avatars can be auto-matically designed according to the user’s external appearance(e.g. hair- and skin-color, age and gender); in forensics whereartists can amend sketches of the suspect or the victim basedon old pictures; and in surveillance where suspects can belocated based on semantic descriptions. Other applicationsinclude age-specific access control where, for example, chil-dren can be prevented from watching certain movies, accessingcertain web sites, or entering bars or liquor-stores. There areindustrial systems1 that extract demographic information ofcustomers for customizing advertisements or for collectingaggregate data about consuming habits (e.g., based on age,gender, ethnicity). In addition, Electronic Customer Rela-tionship Management (ECRM) can use soft biometrics-basedcategorization for effectively managing customers by offeringcustomized products and services. For example, age or genderspecific advertisement can be presented for consumer goodssuch as mobile phones, fashion, and food. In cosmetology, itis of interest to estimate the rejuvenating effect of decorativecosmetics and cosmetic surgery by computing the perceivedage of an individual from their face image.

In video retrieval systems [18], [23]–[25], soft biometrictraits can be used to locate specific individuals in a videostream either by verbal descriptions (e.g., “individual witha red shirt”) or by automatically extracting soft biometricfeatures from an input image and using these features to locatea matching individual in the video stream.

Finally, in health monitoring, soft biometrics are envisionedto play a major role in early diagnosis of illness, sicknessprevention and health maintenance. Such traits include bodyweight/body mass index, skin abnormalities, and wrinkles.We will expand on this possibly later on in the paper.

Below, we describe the various contexts in which softbiometric traits can be used.

1) Uni-Modal System: Often applications might require theextraction of a single soft biometric trait (e.g. gender in agender-personalized advertising campaign), in a so called uni-modal soft biometric system. Such a system generally containsthe “preprocessing”, “feature extraction” and “classification”modules, with the main focus being on the choice ofrepresentation (feature extraction).

2) Fusion With Primary Biometric Traits: Here, the goalis to improve the recognition accuracy of a biometric system.Such an approach was proposed by Jain et al. [4], who consid-ered a hybrid system that combined fingerprint identification

1http://www.quividi.com/

with soft biometric attributes such as age, gender and height,to improve the overall matching accuracy.

Let W = {w1, w2, . . . , wn} be the set of n subjects enrolledin the database, and let x be the feature vector correspondingto the primary biometric system. The output of the primarybiometric system is of the form P(wi |x), i = 1, 2, . . . , n,where P(wi |x) is the probability that the input data belongs tosubject wi given the feature vector x. Let y = {y1, y2, . . . , ym}be the soft biometric feature vector. Then the updated prob-ability P(wi |x, y) that the subject in question is wi , can becalculated using the Bayes rule to be

P(wi |x, y) = p(y|wi )P(wi |x)∑n

j=1 p(y|w j )P(w j |x)(1)

where, p(y|wi ), i = 1, 2, . . . , n represents the conditionalprobability of the random vector y given subject wi .

Other notable research on fusing soft biometrics andclassical biometrics, include the works in [3] and [26]–[29].

3) Search Space Reduction: Soft biometrics can also beused to expedite the search in large biometric databases byfiltering out subjects. A number of attributes such as age,gender, hair and skin color have been proposed for efficientfiltering of face databases [30]–[32]. Furthermore, an analysisof the filtering-gain versus filtering-reliability tradeoff in usingsoft biometric traits to prune large databases was presentedin [33].

D. Visual Attributes

The computer vision community refers to describable visualattributes as any visual and contextual information that ishelpful in representing an image (see Scheirer et al. [26]).In this approach, semantically meaningful labels areemployed towards image retrieval and object categorization.In the context of human recognition, this semanticinformation can describe gender [34], ethnicity [26],accessories [35], clothing style [36], and facial-feature-shapes [34]. Related work include fusion of attributes byScheirer et al. [37], pruning of large-scale datasets byRussakovsky and Fei-Fei [38], as well as studies on similar-ities between faces or objects based on relative attributes byParikh and Grauman [39] and Zhou et al. [40]. Other pertinentliterature include [35] and [41]–[44]. Of specific interest arethe “zero-shot” learning approaches, where previously unseenobjects are described using attributes of objects encounteredin the training set (see Parikh and Grauman [39]).

E. Structure of Paper

The survey provides a review of salient techniques forextracting soft biometrics from modalities such as face, body,fingerprint, iris, and voice. While an exhaustive survey of allsoft biometric traits is not possible due to the richness ofthe field (for example, we do not expand on traits relating tothe ear, or to saccadic movements), we try to offer a holisticview of most of these traits. In this way, this survey paperis significantly different from other introductory overviews(see [11], [13], [14], [45], [46] and Table II) that have focusedon specific soft biometric traits such as gender, age or ethnicity.


TABLE II

EXISTING INTRODUCTORY OVERVIEWS ON GENDER

AND AGE ESTIMATION TECHNIQUES

The structure of this survey is based on the aforementionedtaxonomy of soft biometrics. We discuss soft biometric traitsthat are heavily used as demographic attributes (Section II),as anthropometric (geometric) attributes (Section III),medical attributes (Section IV), and as miscellaneous materialand behavioral attributes (Section V). Finally in Section VIwe discuss open research problems that are currently beingaddressed in the field of soft biometrics.

II. DEMOGRAPHIC ATTRIBUTES

The term demographics, in addition to referring to thequantifiable statistics of a given population, refers to attributessuch as age, gender, ethnicity and race that are widely usedin common population statistics. Since the early publicationsin [56] and [57], research on this class of soft biometrics hasbeen embraced by the computer vision community.

A. Gender EstimationThe traditional definition of sex refers to the biological

characteristics that differentiate men and women, as opposedto gender, which is related to the social and cultural dis-tinctions between the sexes. However, very often, the terms“sex” and “gender” have been used interchangeably in thebiometrics literature. Consequently, we do not make anyexplicit distinction between the two terms in this article.

Gender estimation remains a challenging task, whichis inherently associated with different biometric modalitiesincluding fingerprint, face, iris, voice, body shape, gait, signa-ture, DNA, as well as clothing, hair, jewelry and even bodytemperature (see [58]). The forensic literature [59] suggeststhat the skull, specifically the chin and the jawbone, as wellas the pelvis, are the most significant indicators of the genderof a person; in juveniles, these shape-based features have beenrecorded to provide classification accuracy of 91% − 99%.It has been argued (see for example the work byLoth and Iscan [59]) that there is no single skeletal featurethat definitely reveals the evidence of sexual dimorphism, andthat there is in fact a cross-gender metric overlap of up to 85%,which can be attributed to environmental influences and patho-logic conditions, such as diet and occupational stress. In spite

of this, forensic experts argue [60] that near 100% genderdetermination accuracy can be attained by visual examinationof the entirety of the skeleton.

Humans are generally quite good at gender recognition, asthey have been programmed - from an evolutionary stand-point - to classify gender from early on in their lives [61].As pointed out by Edelman et al. [62], humans performface image-based gender classification with an error rate ofabout 11%, which is commensurate to that of a neural networkalgorithm performing the same task (at that point in time).

Despite this, automated gender recognition from biometricdata remains to be a challenge and is impacted by other softbiometrics, for example, age and race; gender dimorphism isaccentuated only in adults, and varies across different races.

1) Gender From Face: In gender recognition from face,feature-based approaches extract and analyze a specific set ofdiscriminative facial features (patches) in order to identify thegender of a person. This is a particularly challenging problem,as is implied from the fact that female and male average facialshapes are generally found to be very similar.

One of the primary challenges in face-based genderrecognition is the step of feature selection, where one mustjudicially select the type of considered features in order toimprove gender recognition rates. Towards understanding thisfeature selection process, different types of strategies havebeen attempted, such as the work in Sun et al. [63] thatemployed genetic algorithms for eigen-feature selection. Otherapproaches focus on specific facial features, such as theapproach by Zhang et al. [64] that focused on the eye browand the jaw region.

Another challenge comes in unconstrained settings wherethe face image is affected by changes in illumination, pose,etc. While in more constrained settings face-based genderestimation has been reported to achieve classification rates ofup to 99.3% (see Table III), this performance significantlydecreases in more realistic and unconstrained settings.

The majority of gender classification methods containtwo steps succeeding face detection, namely feature extractionand pattern classification.

Feature extraction: Notable efforts include the earlywork by Moghaddam and Yang [65] and the work byBaluja and Rowley [66] who used raw pixel intensities asinputs to SVM and Adaboost classifiers, in order to achievea 96% success rate on low resolution images. Interesting workcan also be found in Cao et al. [67] who investigated facialmetrology for pertinent gender traits, which resulted in errorrates that were observed to be between 3.8% and 5.7% lowerthan that of appearance-based methods. Other feature extrac-tion approaches are found in the work of Saatci and Townin [68], who presented an active appearance model (AAM)based geometric-approach for extracting gender and expres-sion (using a SVM classifier with a radial basis kernel), aswell as recent approaches that use SIFT [69], LBP [51], [70],semi-supervised discriminant analysis (SDA) [71] or combi-nations of different features [72], [73].

Classification: A number of classification methods havebeen used for gender estimation, and a useful comparativeguide of these classification methods can be found in


TABLE III

OVERVIEW OF FACE-BASED GENDER CLASSIFICATION ALGORITHMS. ABBREVIATIONS USED: PRINCIPAL COMPONENT ANALYSIS (PCA),

INDEPENDENT COMPONENT ANALYSIS (ICA), SUPPORT VECTOR MACHINES (SVM), GAUSSIAN PROCESS CLASSIFIERS (GPC),

ACTIVE APPEARANCE MODEL (AAM), LOCAL BINARY PATTERN (LBP), ACTIVE SHAPE MODEL (ASM),

DISCRETE COSINE TRANSFORM (DCT), SEMI-SUPERVISED DISCRIMINANT ANALYSIS (SDA)

91.9%96.0%91.1%

96.6%99.3%88.7%95.75%97.0%94.8%96.32%

93.46% ± 1.65%93.57% ± 1.39%93.57% ± 1.39%

93.74%86.54%93%

92.89%83.7%94.44%89.28%95.0%99.3%93.59%86.83%90.1%88.04%94.81%75.10%96.86%

Mäkinen and Raisamo [51]. One interesting conclusion oftheir work was that image size did not greatly influence theclassification rates. This same work also revealed that manualalignment affected the classification rates positively, and thatthe best classification rates were achieved by SVM.

The area of gender estimation has also received numerousother contributions such as those that go beyond using static2D visible spectrum face-images. Related publications includethe work of Han et al. [89] that explored the use of 3D faceimages and SVMs, achieving gender classification with anaverage error rate of 17.44% on the GavabDB database thatcontained multiple facial surface images of 45 males and16 females. Chen and Ross [90] and Ross and Chen [84]used near-infrared (NIR) and thermal face images for genderclassification. Their work demonstrated that the local binarypattern histogram (LBPH)-descriptor and SVM classifier offerthe best accuracy results, which were reported to reach 93.59%for NIR images; an accuracy of 90.66% was achieved forthermal images using LBP+PCA+SVM. Satta et al. [91]used - in addition to facial features (represented by

LBPH-Pyr (local) and LDA (global) features) - othercontextual features such as hair (captured by HSV histograms,fuzzy color and texture histogram (FCTH), and edge histogramdescriptor (EHD)) in an AdaBoost framework to obtain anaccuracy of about 75%.

2) Gender From Fingerprint: Fingerprint-based genderclassification has received attention in forensic anthropologyas a pruning tool that can reduce the fingerprint search spaceby offering a likelihood that a specific fingerprint belongsto a male or a female. This approach exploits the fact thatthere appear to be differences in male and female fingerprints.Such differences include the ridge-thickness to valley-thicknessratio (RTVTR), the ridge count (the average ridge count isslightly higher in males than in females), and the count ofwhite lines [92].

Gupta and Rao [93] used wavelet transformation and backpropagation artificial neural networks to achieve an overallclassification rate of 91.45% on a private database of550 fingerprints (275 male, 275 female). Similar results wereobtained by Badawi et al. [94], who employed Fuzzy Cognitive


Fig. 4. Fingerprint patterns obtained from female ((a)-(c)) and male ((d)-(f))subjects [99]. Fingerprint-based gender classification in forensic anthropologyconsiders ridge thickness to valleythicknessratio (RTVTR), ridge count andcount of white lines [92]. We note that variabilities due to gender cannot bevisually deduced from fingerprint images.

Maps (FCM) and neural networks to achieve a fingerprint-based gender classification rate of 88% (see Figure 4).Additionally Tom et al. [95] used 2D wavelet transform andPCA to obtain 70% accuracy on a 547 subject-database,while Gnanasivam and Muttan [96] fused fingerprint featuresobtained by discrete wavelet transform (DWT) and singularvalue decomposition (SVD) to achieve an overall classificationrate of 87.52%. Marasco et al. [97] used a combination ofimage quality and texture features for gender determinationwith overall classification rates of up to 88.7% on a dataset of494 subjects. However, their experimental protocol does notindicate if subjects in the training and test sets were mutuallyexclusive. Recently, Rattani et al. [98] explored the use ofclassical texture descriptors - Local Binary Pattern (LBP),Local Phase Quantization (LPQ), Binarized Statistical ImageFeatures (BSIF) and Local Ternary Pattern (LTP) - to estimategender from fingerprint images and tested them on the WVUfingerprint dataset consisting of 237 users. In their experi-ments, the subjects used in the training and test sets weremutually exclusive thus resulting in a statistically unbiasedevaluation. Fusion of descriptors and different fingers providedoverall classification rates of up to 81.4%.

3) Gender From Iris: Thomas et al. [100] were the first topredict gender from near-infrared iris images using automatedmethods. They used a combination of texture features andseven geometric features, including the horizontal, vertical,and Euclidean distances between the pupil center and theiris center, to perform gender classification. Classification wasperformed by bagging 100 C4.5 decision trees, which resultedin an accuracy of 75% on a dataset of 28,000 iris-images, andan improved 80% accuracy when considering only Caucasiansubjects.

Furthermore, Lagree and Bowyer [101] continued thisline of research by exploring additional texture features(extracted by texture filters such as “spot detectors” and “linedetectors”), and without any geometric features. The authorsrecorded an accuracy of 62% on a 600-image datasetof 60 different subjects. The reduced performance,

Fig. 5. Gender from body: A human silhouette is extracted by subtracting thebackground and, subsequently, an average silhouette is computed, constitutingthe gait energy image, GEI. The image depicts male ((a)-(c)) and female((d)-(f)) GEIs [104]. Specifically head and chest regions were observed toprovide pertinent gender cues.

compared to [100], is attributed to the use of a sequentialminimal optimization (SMO) support vector classifier, as wellas on the difference in the size of the datasets (a 50-folddifference) that resulted in a much smaller training set.Lagree and Bowyer [101] considered both gender and raceclassification. An interesting observation in their work wasthat gender classification from iris is a more challenging taskthan race prediction, and race prediction is harder on femalesthan on males. A related work can be found in [102] whereBansal et al. combined statistical features and texture featuresusing wavelets, and designed a gender prediction model usingSVM that achieved an accuracy of 85.68%.

4) Gender From Body: Some biometric traits such as gaitrequire the collection of full body images. Body-based classifi-cation of gender has received much attention since the humanbody has a number of cues for distinguishing gender, as wellas the fact that body shape and gait have the potential to revealthe human gender from a longer distance. These cues includebody sway, waist-hip ratio, and shoulder-hip ratio (see [56]);for example, females have a distinct waist-to-hip ratio andswing their hips more, whereas males have broader shouldersand swing their shoulders more.

At the same time, extracting gender from body is negativelyimpacted by several influential factors which - in the case ofside profile images - include shoes, background and clothes.

In addition, while the gait cycle contains plenty of infor-mation, it also introduces a high feature dimensionality,increasing computational complexity. To reduce dimension-ality (a critical step for obtaining better gender estimationaccuracy [47]), often, the human silhouette is extracted and astatistical feature - namely an average silhouette - is computed.This is referred to as the gait energy image (GEI). TheGEI has been reported to be a good feature for gait andgender recognition because it is robust to silhouette errors andnoise (see Figure 5 and also [103], [104]). To again simplifyanalysis, the body is divided into different regions, such as


TABLE IV

GENDER CLASSIFICATION BASED ON BODY FEATURES. ABBREVIATIONS USED: GAIT ENERGY IMAGE (GEI)(EQUIVALENT TO

AVERAGE SILHOUETTE), WEIGHTED AVERAGE SILHOUETTE COMPONENTS (WASS), CORRECT CLASSIFICATION RATE (CCR),

CANONICAL CORRELATION ANALYSIS (CCA), HISTOGRAM OF ORIENTED GRADIENTS (HOG), MALE (M), FEMALE (F)

legs, waist, buttocks, back, chest and head, and these areanalyzed separately for gender cues. Results have shown thatthe head and chest regions contribute significantly to gendercues.

From a computer vision point of view, human gait analysiscan be appearance-based or it can be model-based, wherecharacteristics such as height, step-frequency, or angle betweentwo thighs are extracted. Given that extracting such individualcharacteristics can be a challenging task, most gait recognitionalgorithms are appearance-based. Table IV gives an overviewof body-based gender classification algorithms with the bestreported performance being 97.2% by Shan et al. [105], whofused GEI and facial features. The GEI-based SVM algorithmobtained a classification rate of 93%.

5) Gender From Hand: Extracting gender information fromthe hand dates back to forensics and archaeological effortsthat deduced gender from skeletons with damaged skeletalregions that are traditionally good indicators of gender, viz.,the pelvis, skull-region, and larger long bones (for singlepopulations). In the biometrics context, hand-based genderclassification is useful as hand images are typically acquiredunder controlled position, orientation and illumination, result-ing in lesser variability than, for example, facial images whichare known to be affected by many factors such as facialexpression changes.

In terms of suitability of analyzing hands for gender classi-fication, it has been noted that hand breadth measurements aswell as the index to ring finger ratio offer a strong insighton the gender of a person, but at the same time, thesecan vary between populations and are further influenced bygenetics, environment and social conditions. Good studieson this can be found in Kanchan and Krishan [110] andScheuer and Elkington [111]. Falsetti [112] and Lazenby [113]used measurements of metacarpals of the human hand andobtained gender classification rates of up to 92%. Anotherapproach to determine the gender by Krishan et al. [114],employed the dimensions of hand and feet. Their studyemployed a test database containing 123 men and 123 womenaged 17-20 years old from North India. By analyzing handlength, hand breadth, foot length and foot breadth to classifygender, the study concluded that left foot breadth provided thehighest accuracy, which was recorded to be 86.9%.

Fig. 6. Hand-based gender classification [115]. A female hand (left), a malehand (middle), and a segmented hand (right).

Interestingly, while most studies on the specific topic are ofanthropological and psychological nature, Amayeh et al. [115]adopted a computer-vision approach, in their effort to findgender specific traits in the shape of a hand. They segmentedthe hand into 6 parts (hand palm and fingers) (see Figure 6),and each part was represented by Zernike moments [116] andFourier descriptors. Different fusion methods (feature-level,score-level and decision-level) were used to combine thesedescriptors, and in a very small database of 40 subjects, themethod achieved an accuracy of 99%.

6) Gender From Speech: In the effort to classify gender, itis only natural to employ speech, since there are substantialperceived differences between typical male and female voices.Towards achieving audio based-automated gender recogni-tion (AGR), speech signals can provide reliable traces, espe-cially in cases of adults, while less reliable traces can be foundin infants and elderly individuals (Madry-Pronobis [117]).Audio-based gender traces can be of voice source-character(such as pitch frequency), or can be of vocal tract-character(such as the first four formants with their respective fre-quency, amplitude and bandwidth [118]). For voice-basedapproaches, the gender classification problem remains open.This is because, while the 170-275Hz female frequency rangeis disjoint and nicely differentiable from the male equivalentof 112-146Hz ( [119]), in real world conditions consisting ofbackground noise, low recording quality and variabilities inthe physical and emotional state of a subject, human voicescan be obfuscated, turning audio-based AGR into a significantchallenge [120]. In clean speech data, Childers and Wu [121]analyzed ten vowels to predict the gender. Using a data-base of 52 talkers (27 male and 25 female), the methodwas recorded to have 100% gender classification accuracy.Sorokin and Makarov [122] found that features such as theinstant of maximum glottis area, the maximum derivative of


TABLE V

EXAMPLES OF DATABASES THAT HAVE BEEN USED FOR EVALUATING AUTOMATED FACE AND BODY GENDER ESTIMATION METHODS

the area, the slope of the spectrum of the glottal airflow volumevelocity pulse, the amplitude ratios of harmonics of thisspectrum, and the frequency pitch, can result in an accuracyof 94.7% for male voice recognition and 95.9% for femalevoice recognition. A comparative study of AGR approachescan be found in the work of Walawalkar et al. [123], wheredifferent classifiers such as nearest neighbor classifier andSVMs were evaluated.

7) Hybrid Approaches: In addition to the aforementionedpurely-image or purely-audio based gender classificationapproaches, recent work has sought to further the gains of suchapproaches by combining them with video sources, resultingin audio-visual cues that offer a more comprehensive genderanalysis, as well as higher resilience to degradation of any ofthese sources. Interesting work that employed this approachcan be found in Pronobis and Magimai-Doss [124] whichexplored different audio and visual features, and proceededto fuse both modalities at the classifier level. Experimentsconducted on the BANCA corpus, verified the usefulnessof this hybrid approach by showing that the integration ofaudio-visual cues indeed yielded higher resilience and bet-ter performance in noisy conditions. El Shafey et al. [125]employed Total Variability (i-vectors) and Inter-Session Vari-ability (ISV) modeling techniques for audio-visual genderestimation, achieving accuracy rates of up to 98% on theMOBIO dataset2 containing 100 male and 52 female subjects.

Similar to the aforementioned audio-visual approaches,other hybrid approaches seek to combine different biometricmodalities in order to increase gender classificationperformance. Notable related work includes the work ofShan et al. [126] which combines gait and facial features.Specifically, this work employed Adaboost and SVMs toclassify human GEIs along with facial features that wererelated by canonical correlation analysis (CCA) on theCASIA Gait Database (Dataset B) [127] consisting of124 subjects (93 male, 31 female), and achieved an overallclassification rate of up to 99.0% ± 1.3% for males and92.0% ± 4.6% for females.

8) Databases: One distinct advantage of gender classifica-tion is that classical databases such as FERET,3 MORPH-II,

2https://www.idiap.ch/dataset/mobio3http://www.nist.gov/itl/iad/ig/colorferet.cfm

CAS-PEAL and LFW already contain gender annotatedinformation, while other face-based databases can be easilyannotated with gender information by visual examination,as opposed to, for example, age information. Additionally,it is often the case that baseline-performances for genderclassification are provided for databases such as FERET.

In Table V we list different databases that have beenused to evaluate gender classification algorithms and can thusserve as test-beds for developing novel gender-classificationalgorithms.

B. Age Estimation

The ability to estimate age is considered to be an importanttrait among humans, for evolutionary reasons. Specificallyhumans learn early on to reliably estimate age of their peersbased on holistic face features (e.g. outline of the face, faceshape, skin texture), local face features (e.g. areas of theeyes, nose and forehead), as well as related configuration(e.g. symmetry) [130], [131] (see Fig. 7). From this categoryof facial features, age can be quantified from features such asthe craniofacial index (see Ramanathan and Chellappa [132]),wrinkles, skin quality [133], facial hair and chin line(Alberta et al. [134]). Age can also be estimated using otherbody features such as hands, gait, clavicle (collarbone), thepubic bone, and the teeth [135].

A main challenge in estimating age - whether this estimationis performed by humans (human-estimated visually perceivedappearance age) or by machines (algorithmically estimatedage) - comes from the fact that the aforementioned featuresare a function of many unknowns, including genetics, healthcondition, living style, working environment, and sociality(Alley [136], Rhodes [130]), all of which vary across people,thus impacting age progression differently from person toperson. This variability motivates research in algorithmicage estimation, which - as suggested by a recent study byHan et al. [137] - can often outperform humans at the sametask.

Below, we will describe different methods of automated ageestimation, focusing on the challenging module of age imagerepresentation. As in the case of gender classification, the taskof automated age estimation can be separated into two modularsubtasks; the aforementioned part of age image representationduring which features are extracted and which will receive


TABLE VI

AGE ESTIMATION TECHNOLOGIES BASED ON CLASSIFICATION. ABBREVIATIONS USED: ACTIVE APPEARANCE MODEL (AAM),

APPEARANCE MODEL (APM), AGE MANIFOLD (AMF), BIO INSPIRED FEATURES (BIF), DISTANCE METRIC LEARNING (DML),

CUMULATIVE ATTRIBUTES (CA), CONVOLUTIONAL NEURAL NETWORK (CNN), DEEP LEARNED AGING PATTERN (DLA),

GRAPH-BASED COMPOSITIONAL AND DYNAMIC MODEL (GB), ORDINAL DISCRIMINATIVE AGING (PLO),

MALE (M), FEMALE (F)

88%83% 82%

100%

80%3.82 − 5.58

5.8150% 43%57.3% 54.7%3.0 4.4

≈ 70%8.0 7.85.4 6.33

6.0 5.578711 7.42

89.7%88% 85%

84%5.218.6

82.7%77%88%

4.675.884.774.26

Fig. 7. The human face over a span of time: images obtained from FG-NET.The images (from left to right) represent a subject in FG-NET at theages of 18, 30, 35, 48, 52 and 61 years old. Facial features that have changedwith age include the facial shape, the chin line, and skin quality.

most of our focus here, and the modular part of automatedclassification or regression where the extracted features lead toeither a classification decision (corresponding to say a binarydecision on whether a person is ‘young’ or ‘old’), or wherethe extracted features are used in a regression framework tooutput a more refined decision (corresponding to, for example,an estimate of a person’s age). Table VI offers an overview ofrecent age estimation works and their reported performance.It is here worth mentioning the study by Fu et al. [54] thatfocuses on automated analysis of age progression, discussingautomated age synthesis and also age estimation techniques.

1) Age From Face:a) Geometric based approaches: Anthropometric

(geometric based) approaches are mainly based on thecranio-facial development theory (see Alley [136]), where -in this case - the growth of the human head is modeled basedon a number of anthropometric measurements over a span oftime, from infancy to adulthood (see for example the caseexhibited in Farkas [138] corresponding to 57 landmarksor fiducial points). Often, what provides more meaningfulconclusions on age are ratios between different anthropometricmeasurements. This ratio-based approach was employed byRamanathan and Chellapa [132] who used 8 such distance-measure ratios to model age progression. Their model wastested on a database partly adopted from FG-NET,4 and wasshown to provide 8-15% improvement in face recognitionperformance, on ages up to 18 years. In terms of drawbacks,anthropometric-based models tend to be better for onlydifferentiating between infants and adults. Furthermore, the

4www.fgnet.rsunit.com


corresponding measurements and their associated ratios canonly be reliably determined from frontal or 3D images.An early 3D approach by Xia et al. [139], which was testedon the FRGCv2 dataset with 466 scans, was shown to achievea mean absolute error (MAE) of 3.29 years for 4 age classes.

b) Approaches based on appearance models: Anotherimage representation approach relates to appearance models,and considers texture and shape features (and thus indirectlyanthropometry and wrinkles), for the purpose of determiningthe human age. Early approaches were presented byKwon and Lobo [140], [141], analyzing wrinkles in facialregions such as the forehead and in the periocular region.This analysis determined the classification of young and senioradults. A recent related study by Guo et al. [142] obtainedbest reported results (MAEs = 2.6 years, tested on the YGAdatabase). These results were obtained by simultaneouslyconsidering age and gender automatic estimation, biologically-inspired features and manifold learning techniques.

Approaches based on active appearance models (AAMs)(see [143]) form a special class under the aforementionedappearance-based models. Interesting work can be foundin [144] (Lanitis et al.) which extended the AAMs by propos-ing an aging function to describe the variation in age. In thisaging function, the input is a vector containing 50 raw modelparameters learned from the AAMs. The proposed functiondefines the relationship between the age of the individuals andthe parametric description of the face images. Based on age-simulation (using a ‘Weighed Person Specific Age Simulation’method), the age in all images was normalized to elimi-nate the effects of aging. This approach led to an improve-ment of face recognition performance from 63% to 71%and from 51% to 66%, on two different datasets. Otherinteresting AAM-based approaches have been employed byGuo et al. [145] and by Lakshmiprabha et al. [146].

c) Approaches based on aging pattern subspaces: Agingpattern subspaces are formed from a sequence of imagesper person representing the chronological progress of aging,i.e., representing an aging pattern. Specifically, each pointin the subspace corresponds to one aging pattern. The ageof an unseen face image is determined by the projection inthe subspace that can best reconstruct the face image. Theposition of the face in the subspace indicates the age of theassociated face. In this approach, which was introduced byGeng et al. [147], [148], this sequence of images forms whatis specifically known as the AGing pattErn Subspace (AGES),which is then properly represented - for each person, at eachgiven age - by AAMs (missing ages are synthesized using anEM-like iteration algorithm). In the testing phase, an image ofa face is matched to the faces of the subspace (i.e., of AGES),and the degree of the matching is then translated into anestimate of the age. This approach was tested [147], [148]on the FG-NET face database, resulting in a reported MAE of6.77 years. While promising, this approach has the drawbackthat it requires multiple images per subject in order to learnage progression. Additionally, wrinkles are - in general - notsufficiently represented by AAMs, and are thus downplayedin the process of estimation. A related approach can be foundin the recent work by Guo et al. [142], [149] which draws

from the feed-forward path theory of the cortex for visualprocessing. In this approach, after sequentially filtering animage with a Gabor filter and a standard deviation basedfilter, the obtained features are subjected to PCA resultingin a lower-dimensional bio-inspired-features (BIF). Recently,deep-learning aging pattern approaches were presented byWang et al. [150] and Tian and Chen [151].

d) Approaches based on age manifolds: First presentedby Fu et al. [152], the main idea behind this approach is thatfunctions that estimate age can be of reduced dimensionality,i.e., will take as input only a reduced number of character-istic parameters. Such functions are used in conjunction withmanifold learning algorithms which reveal these characteristicparameters and find a low-dimensional representation of thedata (see Seung and Lee [153]). The resulting manifold can belinear (Yan et al. [154]) or non-linear (Tenenbaum et al. [155]).Furthermore Guo et al. [145] explored learning manifoldsbased on projections that preserve orthogonal locality. Thisapproach was reported to provide an MAE of 5.3 years onFG-NET, if the size of the training data is large enough. Therelated linear manifold embedding method is referred to asorthogonal locality preserving projections (OLPP). Enhancedage estimation performance was recorded by Li et al. [156]who performed age-manifold learning under locality preserv-ing requirements as well as under ordinal requirements.

e) Automated age classification or regression: Aspreviously mentioned, following the feature extraction inthe age-image-representation part, one can proceed with ageestimation in the form of automated classification or auto-mated regression. A useful exposition of classification andregression approaches can be found in Guo et al. [157]which places emphasis on techniques based on support vectorregression (SVR) and support vector machines (SVM). Inbrief, the authors conclude that comparable results can beobtained using both approaches. Furthermore, the work ofGuo [158] discusses the application of SVMs in learning anestimator or recognizer for the purpose of age extraction.Finally, it is worth noting that classification and regressionapproaches can indeed be combined to form hybrid approaches(see Guo et al. [159]).

2) Age From Body: In the context of age estimation,Lu and Tan [173] recently analyzed gait. In this setting -where, as the authors state, gait appearances vary betweenmales and females even within the same age group - theapproach was based on learning a multilabel-guided subspaceto better characterize and correlate the age and gender informa-tion of a person for estimating human age. The authors extractGabor features including both Gabor magnitude and Gaborphase information of a gait sequence, and perform multiplefeature fusion to enhance the age estimation performance. Therelated experimental results present an MAE of 3.02 years onthe USF database [174] consisting of 1870 gait sequences from122 subjects (85 male and 37 female subjects of age between19 and 59 years old).

3) Age From Hand: One characteristic work in this settingis that by Shamir [175] which presented a new approach ofautomatic age classification based on the features that appearon the back of the human hand. Estimating age from the hand,


TABLE VII

EXAMPLES OF DATABASES THAT HAVE BEEN USED FOR EVALUATING AUTOMATED AGE ESTIMATION METHODS

enjoys - as opposed to age estimation from the face - theadvantage of being privacy preserving, and the advantage ofbeing invariant to facial makeup and plastic surgery. Exper-imental results in the same work show that the method candifferentiate between older and younger individuals with anaccuracy of 88%. The hand photos were taken from BaltimoreLongitudinal Study of Aging5 (BLSA). The dataset includes106 subjects, and 212 images (two from each subject).

4) Databases: An increasing number of databases featurebiometric data related to different ages. We report some ofthese in Table VII.

C. Ethnicity and Race Classification

The traditional definition of race is related to biologicalfactors and often refers to a person’s physical appearancecorresponding to traits such as skin color, eye color, haircolor, bone/jaw structure, face and body shape, and other traits,while the traditional definition of ethnicity is more relatedto sociological factors and it relates primarily to culturalidentifiers such as nationality, culture, ancestry, language aswell as beliefs. The terminology overlaps and is often usedinterchangeably in the biometric literature, and hence we willalso adopt the same loose convention here.

a) Limitations and challenges in race classification:Before addressing some of the progress in automated raceclassification, it is worth touching upon some points whichsuggest that ethnicity- or race-categorization is a rather chal-lenging problem. The first important point - as indicatedin the book of Mordini and Tzovaras [177] - is thatintra-class variation can often be significantly larger than theinter-class variation between races. Similarly, recent findingsby geneticists (see [178], [179]) show that there is oftenno clear relation between races on one hand and the fre-quency of particular genetic variants on the other. In the samevein, Rosenberg et al. [180] reported that within-populationdifferences among individuals account for 93%-95% of geneticvariations, whereas major ethnic groups differentiate onlyby 3%-5%. At the same time, however, the work in [180]identified six main genetic clusters, five of which corresponded

5http://www.blsa.nih.gov/

to major geographic regions, and sub-clusters that oftencorresponded to individual populations.

Related to the challenging nature of race-categorizationis the “other race effect”, where humans charged with aface recognition task are observed to perform significantlybetter when having to recognize the face of a person fromtheir own race (see O’Toole et al. [181]). Interestingly, suchan effect has also been witnessed in automated face reco-gnition [181], [182]. Specifically, this “other race effect”was observed in an international face recognition competitionwhere a fusion of algorithms from Western countries per-formed better in recognizing Caucasian faces, while a fusionof algorithms from East Asian countries performed better inrecognizing East Asian faces.

1) Race From Face: Before presenting some research high-lights on this topic, we note that as with gender and ageestimation, race estimation typically comprises of a featurerepresentation module, where the features are extracted, anda subsequent classification module, where the extracted fea-tures are categorized into a discrete number of race classes.As before, approaches for classification include geometricbased approaches [138], holistic/appearance-based approaches(involving color, texture and shape-based algorithms), as wellas approaches based on local feature descriptors. While itis generally agreed that humans perform race classificationby employing a feature based approach [183], many of thepromising computer vision solutions that we will see later onemploy approaches based on holistic and local features.

We describe some of these approaches below. Table VIIIsummarizes these approaches. The interested reader can alsoexplore a recent survey by Fu et al. [55] which provides acomprehensive study on human and automated analysis of raceclassification.

a) Approaches based on chromaticity: Chromaticity, orskin tone, has long been employed as the primary featurefor race classification; see for example Xie et al. [184],Roomi et al. [185], and Tariq et al. [186]. The latter work usedhuman silhouette images, and achieved classification ratesof 80.37% for ‘East and Southeast Asian’ subjects, 78.67%for ‘White Caucasian’ subjects, 58.21% for ‘Black’ subjects,and 33.33% for ‘South Asian’ subjects. We note here that


TABLE VIII

ETHNICITY CLASSIFICATION. ABBREVIATIONS USED: DECISION TREES (DT), MULTI-SCALE MULTI-RATIO LBP (MM-LBP), KERNEL

CLASS-DEPENDENT FEATURE ANALYSIS (KCFA), ORIENTED GRADIENT MAPS (OGMs), GATE ENERGY IMAGE (GEI),

SUPPORT VECTOR MACHINES (SVM), CATEGORIES (CaT)

>

2%

>

chromaticity-based race classification has the limitation ofbeing strongly sensitive to illumination changes.

b) Approaches based on global features: Global featurerepresentation, or holistic representation, has been found topreserve configural face information (interrelations betweenfacial regions), which is helpful towards race classification.Classical methods include the work of O’Toole et al. basedon PCA of raw pixel intensities [182], [187], achieving accu-racies of about 80% for distinguishing between Japanese andCaucasian subjects. The work by Lu and Jain [188] employedan LDA-based algorithm to classify 263 subjects into2 categories (Asian and Non Asian) obtaining a success rate ofabout 96%.

c) Approaches based on local feature descriptor repre-sentation: These include methods proposed by Lin et al. [189],Klare and Jain [190], Heo and Savvides [191], andHosoi et al. [192]. The latter employed Gabor Wavelet Trans-formation and retina sampling, along with a SVM classifier.On a database containing 771 Asian, 774 European and446 African subjects, an accuracy of about 94% was obtained.Yang and Ai [70] examined 11680 Asian and 1016 non-Asiansubjects, and employed LBP in an AdaBoost classificationto separately determine their ethnicity, gender, and age.For ethnicity classification, the resulting EER was 2.98%.Another approach was presented by Fu et al. [193] whoused topographic independent component analysis (TICA) toform a hierarchical multi-level cortex-like mechanism, whichachieved a classification rate of 82.5%.

Fig. 8. Iris images corresponding to different ethnic groups, (a) a subjectwith Caucasian ethnicity, (b) a subject with Asian ethnicity [101].

Finally, one can also encounter early hybrid approaches thataim to combine local and global features. One such hybridapproach was presented by Ding et al. [194] that boosted localtexture and global shape features, resulting in accuracies of upto 98.3%.

2) Race From Iris: Recent research has explored the pos-sibility of deducing race from iris images (see Figure 8). Forexample, Qiu et al. [200] showed that geometric characteristicsof the iris exhibit significant differences across races. This hasbeen further confirmed in [101], [201], and [202]. Buildingon this property, Qiu et al. [200] developed a model topredict ethnicity from iris texture. Their work considereda binary classification problem (Asian vs. non Asian), andemployed the CASIA database with 2400 images for the Asiansubjects, while for the non Asian subjects it employed theUPOL database6 with 384 images of 64 subjects and the

6http://phoenix.inf.upol.cz/iris/


TABLE IX

EXAMPLES OF DATABASES THAT HAVE BEEN USED FOR EVALUATING AUTOMATED RACE OR ETHNICITY CLASSIFICATION METHODS

UBIRIS database [203] with 1198 images of 240 subjects.Using AdaBoost, the work recorded an accuracy of 85.95%.The images in the UPOL and UBIRIS databases were acquiredusing visible light, while the images in CASIA were acquiredin near-infrared illumination. The authors conducted an exper-iment to establish whether illumination affects the proposedmethod for ethnicity classification. The experiment consistedof selecting irises of Asian subjects from UPOL, UBIRIS andCASIA, and attempting to classify images into those capturedunder visible light versus those captured in the presence ofnear-infrared illumination. The related results showed a near-random classification (of illumination), suggesting that thepresented ethnicity classification is due to inherent character-istics of iris texture, rather than illumination. An extensionof this work by the same authors [201] computed texturefeatures using 40 Gabor filters formed by eight orientationsand five scales. The work used 200 images of Asian sub-jects from the CASIA database, and 200 images from theBioSecure7 database for the non-Asian subjects. The testdataset contained 2,400 additional images of 60 subjects(30 from the CASIA and 30 from the BioSecure database).A K-means algorithm was then used to form 64 clustersrepresenting textons that are commonly occurring fundamentaltexture elements. An image was then analyzed in terms of64-texton histograms, where each pixel was classified into oneof 64 textons, so that an image is represented by a 64-elementfeature vector. With the above in place, an SVM classifierwas then applied, achieving a correct classification rate of upto 88.3%. Lagree and Bowyer [101] focused on analyzingdifferent features and texture filters. Using SMO supportvector algorithms with Weka’s default parameter settings,8 thework reported an accuracy of 90.58% (the employed datasetconsisted of 60 subjects and 600 images).

3) Databases: In terms of classical databases - such as theFERET database - the advantage of having annotated ethnic-ity/race information is diminished by the fact that this infor-mation is often statistically unbalanced, simply because suchdatabases were not created with race-classification in mind.

7http://biosecure.it-sudparis.eu/AB/8http://www.cs.waikato.ac.nz/ml/weka

This often forced researchers in multi-race classification tocombine different databases to ensure a racially balanced anddiverse information set. As a consequence of this, one facesthe problem of employing different datasets that were createdunder different acquisition conditions. As a result, recentefforts have been made to create racially diverse databases.We list some of these in Table IX.

D. Relation Between Age, Gender and Race

It is often the case that a single facial feature can carryinformation about different soft biometric traits. That is whytraits such as age, gender and ethnicity are often treatedand categorized simultaneously (see [189], [207]–[209]) - anapproach that is in line with the perceived correlation of thesesoft biometric traits. In addition to the correlation, exploringthis intertwined nature of traits can carry substantial advan-tages; for example, from a genetic point of view, understandingthe interaction of race with aging allows for conclusionson race-based differences in longevity and aging-associateddiseases, as well as the possible role of genetic factors in suchdifferences [210].

This perceived correlation between these traits has moti-vated additional work such as that in [211], which investi-gated if ethnicity-based gender classification can improve theaccuracy of three gender estimation algorithms, namely pixel-based, HOG and LBP. The work concluded that joint treatmentis not beneficial and that gender and ethnicity can be estimatedseparately, due to the fact that features used to estimate genderare shared by all ethnic groups and features used for ethnicityclassification are present in both female and male faces.

III. ANTHROPOMETRIC ATTRIBUTES

We next focus on soft biometric traits that have been his-torically used to quantify the geometry and shape of the face,body and skeleton (i.e., of classical anthropometric measures).These traits have become pivotal in the context of geometryextraction - i.e., the localization of facial landmarks related toeyes, mouth, nose, chin, etc. - which is a key step in a numberof applications ranging from human identification to gender,ethnicity, and age estimation to emotion and expressionrecognition [212].


While a number of geometric estimation methods have beendeveloped, these methods have to account for several differentfactors. Firstly, some of the geometric measurements can becorrelated (Farkas [138]); thus, the model used for assessingcorrelation can impact the accuracy of estimation. Secondly,the estimation methods have to account for variations in sensorand calibration; this is necessary to ensure that the esti-mated 3D affine measurements are consistent across differentsystems [213], [214]. Thirdly, if there are multiple camerasfocusing on the same scene, then it may be necessary tojudiciously combine the complementary information providedby these cameras during the estimation process.

“Forensic anthropometry” [215] studies somatometry(measurements of the body) [216]–[218], cephalometry (mea-surements of the head) [219], craniometry (measurements ofthe skull) [220] and osteometry (measurements of the bonesof the skeleton) [221], and is used for person identification inthe case of unknown human remains [222].

A. Anthropometric Features From Face

A number of efforts have been directed to suitably captureand represent different geometric anthropometric measures ofthe face. As mentioned earlier, the main reason that such facialgeometric anthropometric measures are of importance, has todo with the fact that localization of facial landmarks (related toeyes, mouth, nose, chin), is often a key step towards precisegeometry-extraction, which is in turn crucial for humanidentification and a class of other recognition systems [212]that generally employ these traits as trackers. In this context,algorithms such as AAM, active shape model (ASM),and deformable shape model have been used (seeDing and Martinez [223]) to obtain such facial landmarks.

Further motivation comes from the fact that - as noted byNixon in [224] - a judicious combination of these facial fea-tures can result in a relatively high degree of distinctiveness forperson recognition. In addition, face-based geometric humanmetrology is an important tool in forensic anthropology [110].

Another line of interesting work is the design ofalgorithms that robustly recognize people in the pres-ence of occluded and disguised faces. In this setting,Ramanathan and Wechsler [225] combine appearance basedapproaches (PCA + LDA), and anthropometric/geometricmeasurements (19 manually extracted geometric measure-ments of the head and shoulders) via decision-level fusion(neural networks) and feature-level fusion (AdaBoost),to design an algorithm that is robust to occluded anddisguised faces. Related work can also be found inEl Kissi Ghalleb et al. [226] where facial measurements -ultimately used for human recognition - are extracted basedon geometry in the detected face (SMQT and split up SNOWclassifier). This resulted in a soft biometric system, which wasfused with different classical biometric systems, to achieve animproved matching accuracy.

B. Anthropometric Features From the Body

Among the many geometric traits, body height isthe most prominent. In extracting this measure, dif-ferent challenges remain, including that of the human

pose which can serve as a primary biasing factor.Therefore, BenAbdelkader and Davis [227] resorted toaveraging different body measurements over different posesin order to reduce the pose bias. Other methods that resultin robust height estimation can be found in the work byMadden and Piccardi [228] where a person’s height wasestimated from surveillance video, thus allowing for session-based biometric matching using two disjoint camera views.

Body height can often be estimated together with othergeometric measures. For example, the aforementioned workby BenAbdelkader and Davis [227] considered shoulderbreadth (bideltoid breadth) in addition to body height, andthe two measures were jointly averaged for the purposeof improving multi-target tracking across multiple cameras.In a similar spirit, but in the setting of person identification,Johnson and Bobick [229] extracted four static parameters(body height, torso length, leg length, and step length),achieving a typical error of about 30cm for anthropometricmeasure estimation. Furthermore, BenAbdelkader et al. [230]used height and stride (extracted from gait videos) for personrecognition, achieving true recognition rates of 47% forfronto-parallel sequences of 41 people, and 65% fornon-fronto-parallel sequences of 17 people.

Recently, body shape has been estimated in millimeter waveimages (acquired at 94 GHz) [231]. Contour coordinates anddynamic time warping were used resulting in an EER of 1.33%on a dataset of 50 individuals (BIOGIGA [232]).

a) 3D techniques in geometric anthropometric measure-ments: Recently, 3D techniques have been used to obtaingeometric anthropometric measurements. Some of these worksinclude the study by Carnický and Chorvát [233] that focusedon the acquisition of 3D measurements with structured lightillumination, and the work by Allen et al. [234] that developeda method for fitting high-resolution template meshes to 250detailed human body range scans from the 3D CAESARdatabase9 with sparse 3D markers to model the variation ofdifferent bodies.

Additionally, in the work of Adjeroh et al. [235] spanningface and body metrology, the 1D-measurements in the3D CAESAR database were used to show a high degree ofcorrelation between certain face and body measurements.Utilizing the correlation structure between measurements,the researchers showed that gender could be estimatedwith a success rate of 100% (this corresponded to 44measurements), and body weight could be estimated witha success rate of 92.5%. Based on the idea that someanthropometric measurements are highly correlated andthat missing values can thus be predicted, an iteratedclosest point (ICP) registration-model and Laplacianmesh was presented by Hasler et al. [236], wherea detailed 3D human body shape in the presence ofclothes was modeled based on a space of human shapes,learned from a large database of registered body scans.Similarly, by using the visual hull, a method for estimatingdetailed 3D body shape under clothes was presented byBalan and Black [237]. This work used the

9http://www.hec.afrl.af.mil/cardlab/CAESAR/index.html


HumanEva10 dataset. Furthermore, Criminisi et al. [213]computed 3D affine measurements from single perspectiveviews, where affine scene structure could be determinedfrom the image, without knowledge of the camera’s internalcalibration (e.g. focal length), nor of the explicit humanpose. Interesting work can also be found in Godil andRessler [238] who performed similarity-based retrievalin the 3D CAESAR anthropometric database (3D scans),where the technique was based on body and head shaperepresentation. Finally geometric/anthropometric featureshave also been exploited to enhance gait recognition accuracy(see Moustakas et al. [239]).

C. Databases

While some datasets include the annotation of pertinentfacial landmarks such as eyes, nose and mouth, these aremainly used for image alignment. Often, more detailed land-mark annotation in common datasets is performed manually,in order to capture the necessary measurement ratios, whichare often more valuable than the measurements themselves.Notable exceptions include the two paid-for datasets11 ofthe Civilian American and European Surface AnthropometryResource Project (CAESAR).

CAESAR 3D Anthropometric Database (North AmericanEdition) includes 40 anthropometric/geometric measurementsper person, from a North American population sample of2,400 male and female subjects, aged 18-65. It also includesdemographic information and 3D model scans. There areseveral poses of each person.

CAESAR 3D Anthropometric Database (European Edition)was recorded in the same project as the North American Edi-tion and also features 40 anthropometric/geometric measure-ments from a European population sample of 2,000 Europeanmale and female subjects, aged 18-65. Demographic informa-tion as well as 3D scans are provided under several poses.

IV. MEDICAL ATTRIBUTES

A. Image-Based Automated Self Diagnostic Methods

Image-based automated self diagnostics is a novel andpromising method for enhancing classical medical care. Suchmethods, when applied properly, can provide a broad range ofmedical benefits, including early detection and prevention ofillnesses, as well as efficient treatments when possible. Suchmethods are needed now more than ever, due to the everincreasing burden on traditional medical care systems. Withthis need in place, we are already seeing different medicalsectors employ computer vision schemes, such as in the crucialarea of monitoring the health of elderly people to improvetheir safety, autonomy and quality of life [240]–[242]. Suchpromising applications, and many others, have introduced theneed for algorithms and systems that can reliably monitorthe state of different human traits, which in turn accentuatesthe importance of being able to properly characterize softbiometric traits such as body weight, body mass index (BMI =

body weight(body height)2 ), or even skin color and quality.

10http://vision.cs.brown.edu/humaneva/11http://store.sae.org/caesar/#3dna

B. Health and Weight

Again in the context of health, body weight and BMI havereceived increased attention in recent years. This boost inattention is related to health concerns that have been expressedin several countries. For example, it has been stated that morethan two-thirds of adults in the United States are now over-weight or obese [243], and obesity - which is now recognizedas a disease by the American Medical Association12 - accountsfor nearly one out of every 10 American deaths. Specifically,an increased body weight and thus BMI,13 is often associatedwith many health problems such as diabetes, heart disease,certain cancers, strokes, osteoarthritis and gallstones, all ofwhich cost the United States 223 billion per year.14 This hasmotivated the use of automated techniques for estimating bodyweight and BMI - that are derived from body weight and bodyheight - which further has the advantage of being contactless,low-cost, unobtrusive, and unconstrained.15

Recently, BMI was estimated from face images byWen and Guo [2]. In their work, the face was detected,normalized, and an active shape model was fitted, based onwhich, geometry and ratio features were extracted (cheekboneto jaw width, width to upper facial height ratio, perime-ter to area ratio, eye size, lower face to face height ratio,face width to lower face height ratio and mean of eyebrowheight), normalized and finally subjected to support vectorregression. When training and testing using the MORPH-IIdatabase - which contains 55,000 images, with variations inage and ethnicity - the method obtained MAEs ranging from2 kg/m2 (‘normal weight’) to 9.5 kg/m2 (‘obese’), wherethe higher errors generally related to medically atypical casessuch as ‘under weight’ and ‘obese’ subjects. Prior to that,Velardo and Dugelay [244] had devised a method to estimateweight from manually measured body-anthropometric data, byconstructing a weight prediction metric using multiple regres-sion analysis. When training and testing using the NHANESdataset16 - which contains data of 28,000 subjects - the resultsshowed that weight could be estimated in 93% of the test datawith an error of less than 10%. Velardo et al. [245] presented apractical study with a weight estimation error of 4% on a selfrecorded Kinect RGBD dataset of 15 subjects, where bodyanthropometric measurements were extracted automatically.This work proceeded also to evaluate - based on the samemeasurements - the subjects’ height and gender, using real-time reduced-complexity 3D model fitting. A related workfor weight estimation by Labati et al. [246] studied framesequences representing walking subjects, where different walk-ing directions and lighting conditions were used to challengethe algorithms. Features such as body height, body volume(estimated by ellipses passing through the silhouettes), bodyshape and walking direction were extracted to train a neural

12www.ama-assn.org/go/obesity13The use of BMI as a sole measure of obesity has been challenged in the

medical literature.14http://www.health.harvard.edu/newsletters/harvard_mens_health_watch/

2012/February15Extracting BMI or body weight also offers other advantages, e.g., in

forensics, as these measures can be sensed at a distance.16http://www.cdc.gov/nchs/nhanes.htm


Fig. 9. Examples of skin diseases. The human skin often exhibits symptomsof internal and external diseases. (a) Parvovirus. (b) Rosacea. (c) ContactDermatitis.

network towards body weight estimation. The mean errorsranged from −2.38 kg to 0.02 kg on a small proprietary datasetof 20 subjects.

C. Health and Skin Lesions

In addition to geometric body characteristics, another classof soft biometrics that relates to medical conditions is thehuman skin (see Figure 9). The main motivation here isclear; early detection of malignant skin lesions is very criticaltowards initiating early and effective cure methods. For caseslike skin cancer - which is the most common cancer type,with 3.5 million cases of basal and squamous cell skincancer diagnosed in the United States each year17 - earlydetection can allow for a suitable cure and even prevent death.While such automated systems cannot fully replace medicalprofessionals, it is often noted that visual examination byan expert can be difficult (deeper subsurface inspection isrequired), while biopsies can be invasive and time consuming.As a result there have been substantial efforts to develop novelnoninvasive optical imaging techniques as a way to detectand analyze skin lesions [247], [248], with recent advancesinvolving smart hand-held devices [249], [250]. In addition toskin cancer, image-based methods can apply to other commonskin diseases like chronic inflammatory psoriasis [251], [252],pressure ulcers [253] and hyper-pigmentation evolution [254].

b) Health and wrinkles: In addition to skin lesions,wrinkles can also reveal stress states that relate to the medicalcondition of an individual. Naturally wrinkles also carrydiscriminative clues, as was analyzed by Batool et al. [133],where a set of facial wrinkles extracted from images weretreated as curve patterns towards finding similarity betweensubjects. In their work, different metrics based on Hausdorffdistance and curve-to-curve correspondences were presentedin an effort to quantify this similarity, and bipartite graphmatching was employed for finding correspondences betweencurves from two patterns. The resulting recognition ratesexceeded 65% at rank 1 and 90% at rank 4, suggesting that aset of wrinkle patterns may be unique to an individual.

D. Databases

There is currently a limited number of publicly availablebiometric datasets with annotated health-information. This ispartly due to the sensitive nature of medical data which raisesseveral ethical and privacy concerns especially when the sub-ject identity is divulged. Hence, studies on related topics ofteninvolve private medical datasets, accessible only by associates.

17http://www.cancer.org/cancer/skincancer/index

The MORPH-II and the CAESAR datasets are exceptions,providing images which include weight information.

V. MATERIAL, BEHAVIORAL, AND OTHER

SOFT BIOMETRIC ATTRIBUTES

We next consider objects that may be associated with aperson (e.g. accessories such as hats and scarves), traits thathave a strong behavioral utility, as well as other soft biometrictraits that have received substantial attention lately. Some ofthese attributes - related to eye lenses, eye glasses, hats, scarfs,clothes, etc. - may not be immediately associated with bio-metrics in the classical sense; nonetheless they remain of highinterest in identification and security applications, often due totheir conspicuous nature. Some of these objects and attributesmay naturally be associated with different utilities at once.Take for example the color of the clothes. This attribute, whileheavily related to subject occlusion, can also assist in humanrecognition, as shown in the work by Denman et al. [20]who presented a video surveillance and tracking system thatis based on the color of clothes, where specifically colorhistograms of the head, torso and legs were processed in orderto re-identify individuals. The same attribute was used also byD’Angelo and Dugelay in [255] for analyzing the color ofclothes in a sports-soccer scenario, with the aim of preventingfights between fans of opposing teams (shirt color can be astrong indicator of the team which a subject supports).

A. Material Characteristics

Unlike most other attributes, this class consists of char-acteristics that can in fact hinder the recognition process.These include face occluding characteristics, such as scarfs,caps, hats and eye glasses. Such occluding accessories havereceived attention primarily for their ability to hinder personrecognition or even human detection, from both machine andhuman perspectives. The consideration of such attributes -in the context of face recognition, occlusion and occlusiondetection - can be found in papers by Yoon and Kee [256]and Wen et al. [257]. In their work, occlusion detection isperformed for specific facial features such as the mouth region,while some other works such as that by Lin and Liu [258],observe the face holistically, classifying it as occluded or not,by employing PCA and SVM. Below we review some of thesematerial characteristics.

Scarfs are of particular importance because they are some-times used by individuals in a crime spree. Min et al. [259]presented a scarf detection algorithm based on PCA and SVM,and reported a detection rate of about 99%, on the ARFD data-base ( [260]) which features 300 scarf-occluded and 300 not-occluded faces. This work was later used by Min et al. [261]towards face recognition, where Gabor wavelets, PCA andSVM were employed for occluded faces, while non-occludedfacial parts were computed by block-based LBP. This workused the AR face database,18 and reported a recognition rateof 94.83% for non-occluded faces, and 92.08% for occludedfaces.

18http://www-sipl.technion.ac.il/new/DataBases/Aleix%20Face%20Database.htm


In contrast to scarfs, which are mainly used to cover thelower part of the face, headgear can occlude the upper part ofthe face. Du et al. [262] studied the occlusion of hard hats inconstruction sites, and studied face detection using the ViolaJones algorithm, also analyzing the motion and color in theupper face region in a dataset of 5 video sequences. Similarly,cap detection was performed by Min and Dugelay [263]who utilized dynamic time warping (DTW) and agglomerativehierarchical clustering, to achieve an accuracy of 87.5% on aprivate dataset of 10 recorded subjects with and without capsin 40 videos.

Similarly, eye glasses and contact lenses can also have anoccluding effect - albeit lesser than that of the previous twocharacteristics - and can interfere with the performance ofiris and periocular-based recognition systems. In this context,the impact of contact lenses on iris recognition was exam-ined by Baker et al. [264]. Their experiments suggested thateven non-cosmetic prescription contact lenses can impact thematching accuracy of iris recognition. Subsequent research byErdogan and Ross [265] established the possibility of auto-matically detecting such lenses in iris images. Glass detectionin face images was studied by Jiang et al. in [266]–[268], withbest results obtained using thermal images [269].

Makeup can also be viewed as an occluding characteristic,and has the potential to substantially alter the appearance ofa face. Furthermore, makeup and cosmetics can be used tosuccessfully alter or camouflage wrinkles, birth moles, scarsand tattoos. The impact of makeup on face recognition hasbeen studied by Dantcheva et al. [270], while Chen et al. [271]performed makeup detection, achieving a detection accuracyof approximately 91%.

B. Behavioral Soft Biometric Traits

Behavioral attributes are a relatively new class of softbiometric traits that can assist in detecting different humanbehaviors. For example, the accent of a person can begleaned from human speech which can also convey ancillaryinformation such as the gender of the speaker. The associ-ated gender estimation and accent recognition can be fusedwith classical speech or speaker recognition methods inorder to improve recognition performance. In this context,Deshpande et al. [272] showed promising results, where,focusing on American and Indian accents in the Englishlanguage, they analyzed the second and third formant frequen-cies of specific accent markers and classified those based onGaussian mixture models.

Some other behavioral traits that can be automaticallygleaned from video and images include facial expressionswhich also carry supplementary biometric information.Interesting work can be found in related surveypapers [212], [273], [274]. In the specific context of softbiometrics, Kashyap et al. [275] studied video segments ofindividuals encoded by a facial action coding system (FACS)for facial expression quantification. Related experimentsstudied the intensity of a number of facial units (each facialunit represents movement of a specific facial muscle), andconfirmed the suitability of facial asymmetry, as well as ofaction unit combinations, towards person identification.

Fig. 10. Examples of different tattoos [277].

C. Scars, Marks, Tattoos

Scars, marks and tattoos are increasingly employedtowards person recognition in forensics and law enforcement(see Spaun [276]). Such traits can be particularly usefulin describing wanted or missing people, or even unidenti-fied bodies, and the traits are often bundled together dueto the high degree of distinctiveness they can jointly pro-vide. Scars and marks are mainly defined by the loca-tion in which they occur, while tatoos carry information intheir position, color, content (such as human forms, ani-mals, flags, plants, words, and other objects, see Fig. 10),as well as the way in which they are ‘imprinted’ (chem-ical, branded, cut) into the skin (see ANSI/NIST standardreport.19) Despite the rich information that is embeddedin tatoos and marks, there is substantial ambiguity associ-ated with their classification. To overcome this ambiguity,Lee et al. [277] proposed a content-based image retrievalsystem that uses SIFT features, reporting rank-20 retrievalaccuracies of 98.6% on high resolution images (Web-DB),and 77.2% accuracy on an operational database (MI-DB).Furthermore, Jain and Park [27] and Park [278] used AAM tolocate and segment primary facial features (e.g., eyes, nose,and mouth) and employed Laplacian-of-Gaussian (LoG) andmorphological operators in order to detect facial marks. Theiralgorithm was tested on a subset of the FERET databaseconsisting of 426 images of 213 subjects, and showed amarginal improvement in rank-1 identification accuracy overthe commercial FaceVACS face recognition system. A similarevaluation was conducted on the Mugshot-dataset having1,225 images of 671 subjects, achieving an improvement inaccuracy from 91.88% to 93.14%. Along the same lines,Lin and Tang [279] used SIFT-based extraction and fusion ofskin irregularities - such as facial marks - to improve facematching performance by about 5% on the XM2VTS andHRDB database.

Additionally, Pierrard and Vetter [280], in the setting of faceidentification, proposed a method to extract moles using nor-malized cross correlation (NCC) matching and a morphablemodel. Experiments were conducted on the FERET database,and the authors claimed that their method was pose- andlighting- invariant (albeit computationally expensive) due tothe underlying 3D morphable model. Srinivas et al. [281]used facial marks to differentiate between identical twins,while Nurhudatiana et al. [282] studied relatively permanent

19http://www.nist.gov/itl/ansi/upload/sp500-245-a16.pdf


pigmented or vascular skin marks (RPPVSM) found on thebacks of 269 male subjects, and employed this soft biometrictrait in forensics.

1) Databases:

XM2VTS database.20 This is a multi-modal face data-base, corresponding to 295 subjects, where for each sub-ject there are four recordings taken over a period offour months, and where each recording contains a speakinghead shot and a rotating head shot. Sets of data takenfrom this database are available, including high quality colorimages, 32 KHz 16-bit sound files, video sequences and a 3DModel. XM2VTS has been used for skin irregularity detection(e.g. facial marks).

Mugshot [283] database. This database contains face imagesof 1500 subjects. Each subject has two images: a frontal viewand a profile view. The photographs are provided from archivesof the FBI. The Mugshot dataset has been used in facial marksstudies.

Web-DB database.21 The Web-downloaded Tattoo ImageDatabase contains 4,323 tattoo images downloaded from theWeb. Each image is of size 90 × 90.

MI-DB database. The Michigan State Police Tattoo Data-base contains tattoo images from suspects and convicts,recorded over a period of ten years, by the Michiganpolice department. Tattoos are photographed when a sus-pect/convict is booked, and assigned a keyword as definedin the ANSI/NIST standard and then stored in the databasealong with other demographic information. There are 69,507operational tattoo images, each of size 640 × 480.

VI. OPEN RESEARCH PROBLEMS

There are a number of open research problems in thisexpanding field.

A. Correlation Between Soft Biometric TraitsThe issue of correlation is not well studied in the soft

biometric literature. While Adjeroh et al. [235] investigatedthe correlation between geometric attributes of the humanbody, such a study has not been undertaken for other traits.Understanding the correlation structure between these traitswill allow for the design of effective fusion schemes forcombining them.

B. Defining the Number of Categories for a Soft BiometricTrait

A main issue also relates to finding an efficient and robustway to create the corresponding categories for a particulartrait. While for some traits, this categorization is straightforward, for other traits the division into categories can becomplex and not immediately clear. For example genderrecognition is easily viewed as a binary classification problem(male vs. female), while other traits such as age or race canbe continuous and ambiguous in nature. The challenge is thefact that this classification must (at least partly) adhere to thesemantic interpretation of traits (recall human compliance), aswell as must account for the significant differences between

20http://www.ee.surrey.ac.uk/CVSSP/xm2vtsdb/21http://www.tattoodesign.com/gallery/

actual traits and the human perceived version of these sametraits.

In addition to the aforementioned challenges relating tothe perceptual idiosyncracies of humans, there are also chal-lenges relating to the limitations of algorithmic and hardwareresources. In short, a soft biometric system and its classifica-tion must take into consideration the resources of the system.For instance, from an image processing point of view, traitcategorization can be limited by the employed estimationalgorithms as well as the quality of the input images. As aresult, while a certain camera-and-algorithm system may beable to robustly differentiate between many categories, anothermay come short of that, hence introducing the need for carefulconsideration of resources on the degree of refinement ofcategories.

C. Automatic Quality Assessment of Extracted Soft BiometricTraits

On a similar note, another challenge is to reliably extractthese traits. For instance, from an image-processing pointof view, the challenge of reliable feature extraction is oftenrelated to ambient factors such as lighting. Illumination varia-tion is a significant problem which, in many cases - especiallyin surveillance videos - can result in massive variations incolor. At the same time, soft biometrics offer substantialflexibility as far as acquisition is concerned. For example, traitssuch as hair color, skin color, height, weight, gait and differentfacial measurements, can be sensed (and estimated) from bothfrontal as well as side views of a subject. So the challenge is tobalance the flexibility in acquisition afforded by soft biometrictraits (e.g., under different pose and expression changes) withvariations due to sensing (e.g., illumination) by optimizingthe number of categories that can be used for a particular softbiometric trait (e.g., number of discrete colors for clothing).Automatic assessment of data quality may be necessary forthis reason.

D. Subjective Perception by Humans

One wide open challenge is to design soft biometricmatching systems that account for human variability indescribing different traits. The human compliance attributedto soft biometrics, while useful in many ways, also intro-duces problems relating to the broad variability in syntaxand language when it comes to describing different softbiometric traits (e.g., hair can be described as “red”, “auburn”to “brown” - see [284] for an interesting experiment on this).Furthermore, humans can be inaccurate when describing mea-surements [285], where descriptions can depend on the per-son’s own attributes and their own perception of populationaverages and variation [286]. Early literature to overcomethis include methods that apply reference examples, or thatapply comparative/pairwise ranking (see [286]). For example,determining whether a person is taller than someone else iseasier and more accurate than an absolute estimation of theperson’s height.

E. Fusion of Soft Biometrics With Primary Traits

Another topic is the incorporation of soft biometric traitsinto primary biometric systems. One could imagine a soft


biometric system that quickly classifies and filters a database,and where this pruning is followed by a more robust butslower classical biometric search. In this setting, the challengeis to design and fuse the component systems in a way thatsatisfies the specific speed and reliability requirements of theoverall system.

F. Statistical Modeling and MathematicalAnalysis of Soft Biometric Systems

It is necessary to provide a mathematical analysis of thelimits and capabilities of soft biometric systems. This needis accentuated by the fact that such systems are bound to getbigger and more complex (by considering numerous biometricand soft biometric traits), as well as more demanding. Systemsmay be required to process an abundance of informationpertaining to large populations (e.g., over 1.2 Billion subjectsin the UIDAI biometrics project.22) Analysis of complexsoft biometric systems is necessary; while simulations andempirical studies can often provide some insight into the limitsof such systems, empirical approaches may not be able toaccurately characterize the performance of large-scale systemswith increased complexity.

One related problem that has remained largely unexploredis that of estimating the reliability of person recognition usingsoft biometrics. Such analysis would have to consider the effectof the population statistics, as well as have access to mathe-matical models for the sensing and algorithmic capabilities ofthe system. This can help in automatically establishing thenumber of categories that are necessary for a soft biometrictrait in order to guarantee a certain degree of performance.Further, it may be instructive to derive theoretical boundson the discriminative and classification capabilities of a softbiometric system. Very preliminary work on this can be foundin [287].

Along the same lines, it would be worthwhile to predict theexpected improvement in matching accuracy and reduction incomputational complexity when utilizing soft biometric traitsas filters on large databases. Such analysis would be beneficialin applications such as time-constrained human identificationin video surveillance systems. The analysis should take intoaccount the implicit errors in soft biometric classification,since these errors would affect the overall identification per-formance. A very early analysis can be found in the work byDantcheva et al. [33].

G. Spoofing of Soft Biometrics

The potential of using soft biometrics in several sensitivecommercial and security applications, has to be balanced withunderstanding and minimizing the vulnerability of such traitsto obfuscation (where an individual camouflages their softbiometric traits) and spoofing (where an individual attempts tolook like another person in terms of their soft biometric traits).This is particularly important because the very nature of softbiometrics allows for easier spoofing than classical biometrictraits; indeed one can easily imagine how gender, age andethnicity can be disguised simply by styling and cosmetics.

22http://uidai.gov.in/

Fig. 11. Example images for gender spoofing [288]: female-to-malesubset: female subjects apply makeup to look like males (a)→(e), (b)→(f);male-to-female subset: male subjects apply makeup to look likefemales (c)→(g), (d)→(h). The images in the upper row are beforethe application of makeup and the ones below are the corresponding imagesafter makeup.

Similarly, eye color can be spoofed with color lenses, hair canbe dyed, and skin color - and the overall facial appearance -can be modified by makeup. In this regard, the use of cosmeticproducts towards soft biometric spoofing would be a realisticthreat. Early work on this can be found in [288], where theimpact of makeup on gender- and ageestimation algorithmshas been studied. Specifically the authors in this work discussgender spoofing (see Figure 11) where male subjects attemptto look like females and vice versa, as well as discuss agealteration where female subjects attempt to look youngeror older than they actually are. Experimental results withseveral gender- and age-estimation algorithms, suggested thatgender and age estimation systems can be impacted by theapplication of facial makeup. Additional proof of the impactof makeup on age has been reported in [289], thus furtheraccentuating the open challenge of designing algorithms thatare less susceptible to spoofing and obfuscation techniques.

H. Ethical Issues

In addition to the mainly algorithmic challenges mentionedabove, there exist other challenges in the area of soft biomet-rics, including ethical challenges related to the use of skincolor, ethnicity and gender in recognition systems (see [177]).Ethical issues might also rise as a result of the fact that softbiometrics can be covertly extracted from data collected forother purposes. At the same time, soft biometrics do a betterjob at preserving privacy, given that the stored signatures areoften less descriptive than, say, a high resolution image of aniris. The challenge is to properly traverse this thin line betweenprivacy enhancement and privacy infringement.

I. Privacy Implications

In the context of biometrics, privacy refers to the assurancethat the biometric data collected from an individual is not usedto deduce any type of information about the individual [290],i.e., it should be used only for matching purposes. However,as discussed in this article, biometric data offer additionalinformation about an individual which can be automaticallydeduced. This can be viewed as privacy leakage since an entitycan learn additional information about a person (or population)from the stored data, without receiving authorization from


Fig. 12. Perturbing gender information in face images byOthman and Ross [293]. The soft biometric attribute of a face (e.g., gender)is suppressed, while preserving the ability of the face matcher to recognizethe individual. This ensures that the stored biometric data is not used forpurposes beyond what was expressed during the time of data collection.

the person for such a disclosure. Therefore, it is necessaryto ensure that biometric data stored in a system are usedonly for the intended purpose and not for purposes that mayresult in a ‘function creep’. This has heightened the need todevelop differential privacy constructs, where the biometricdata can only reveal certain attributes (e.g., gender) whileother attributes (e.g., ethnicity) are suppressed. The term“de-identification” has often been used in this context.

De-identification has become an essential keyword, sincea large amount of video surveillance systems have beenemployed in public spaces. Such surveillance unintention-ally invades the privacy of individuals captured in videos.To circumvent such a privacy invasion, subjects in the videocan be de-identified. De-identification refers to the obfusca-tion of the identity of an individual without obscuring theperformed action, see [291]. For maintaining the purpose ofvideo surveillance, limited information related to subject andaction has to be kept. Gender, ethnicity, body height, color ofclothes and color of hair may adequately serve as remainingevidence. In [292] the authors present a privacy preservingvideo surveillance system, where subjects are de-identified andthe remaining information includes the subject height, weightand color of clothes. Specifically, this approach uses bodyheight, weight and clothes color to track a person inside abuilding under surveillance, and to display the position of thisperson on a monitor, in the form of a square placed within themap of the building. The square displayed in the monitor isproperly colored; the upper part of the square has the color ofthe clothing of the upper body of the subject, while the lowerpart of the square is colored to match that of the clothes onthe lower part of the body.

On the other hand, de-identification of soft biometrics, whileretaining the facial identity [293] (see Fig. 12) can be usefulfor extending different levels of privacy to a face image in acentral database.

J. Standardization

ANSI/NIST-ITL 1-2011 (Data Format for the Interchangeof Fingerprint Facial, & Other Biometric Information -Part 1) suggests “additional descriptive information” such aseye patch, clear glasses, dark glasses, head covering, scarf,moustache, beard, eye color, hair color. Further, the standard

“American National Standard for Information Systems- DataFormat for the Interchange of Fingerprint, Facial, & ScarMark & Tattoo (SMT) Information” widely covers the use andclassification of scars, marks and tattoos. Further standardiza-tion will be essential to balance privacy and utility.

VII. CONCLUSIONS

A review of the biometric literature suggests that research insoft biometrics is on the rise. This expansion is due to the largenumber of applications that can benefit from the extraction ofsoft biometric traits. Examples of such applications includesecurity, surveillance, retrieval, and health care. In this article,we reviewed some of the methods that have been developedfor extracting soft biometric attributes from image, video andaudio data. We also introduced a taxonomy to organize thevarious soft biometric traits that have been presented in thebiometric literature. Further, we discussed the benefits andlimitations of different soft biometric approaches. Finally,we discussed some of the open research problems in softbiometrics research.

We believe that the main advantage of soft biometrictraits lies in their ability to describe people from a humanperspective. Therefore, it bridges the gap between machineand human descriptions of a person. We also believe that theperformance of soft biometric systems needs to be consideredcarefully, as it can be affected significantly by several factorssuch as sensing and feature extraction. By carefully inves-tigating the accuracy, reliability and distribution of differentsoft biometric traits, it is possible to judiciously use them inlarge-scale biometric systems. However, it is necessary to becognizant of the privacy implications of using soft biometrictraits. By balancing privacy with performance, it is likely thatsoft biometric traits will have a critical role to play in nextgeneration identification systems.

REFERENCES

[1] A. K. Jain, P. J. Flynn, and A. A. Ross, Eds., Handbook of Biometrics.New York, NY, USA: Springer-Verlag, 2007.

[2] L. Wen and G.-D. Guo, “A computational approach to body massindex prediction from face images,” Image Vis. Comput., vol. 31, no. 5,pp. 392–400, 2013.

[3] A. K. Jain, S. C. Dass, and K. Nandakumar, “Can soft biometrictraits assist user recognition?” Proc. SPIE, vol. 5404, pp. 561–572,Aug. 2004.

[4] A. K. Jain, S. C. Dass, and K. Nandakumar, “Soft biometric traitsfor personal recognition systems,” in Proc. 1st Int. Conf. BiometricAuthentication, 2004, pp. 731–738.

[5] M. S. Nixon, P. L. Correia, K. Nasrollahi, T. B. Moeslund, A. Hadid,and M. Tistarelli, “On soft biometrics,” Pattern Recognit. Lett.,2015.

[6] H. Ailisto, M. Lindholm, S.-M. Mäkelä, and E. Vildjiounaite, “Unob-trusive user identification with light biometrics,” in Proc. 3rd NordiCHI,2004, pp. 327–330.

[7] H. T. F. Rhodes, Alphonse Bertillon: Father of Scientific Detection.New York, NY, USA: Abelard-Schuman, 1956.

[8] S. Samangooei, B. Guo, and M. S. Nixon, “The use of semantichuman description as a soft biometric,” in Proc. 2nd IEEE Int. Conf.Biometrics, Theory, Appl. Syst., Sep./Oct. 2008, pp. 1–7.

[9] D. A. Reid and M. S. Nixon, “Imputing human descriptions in semanticbiometrics,” in Proc. 2nd ACM Workshop Multimedia Forensics, Secur.Intell., 2010, pp. 25–30.

[10] D. A. Reid and M. S. Nixon, “Using comparative human descrip-tions for soft biometrics,” in Proc. Int. Joint Conf. Biometrics, 2011,pp. 1–6.


[11] A. Dantcheva, C. Velardo, A. D’Angelo, and J.-L. Dugelay, “Bag ofsoft biometrics for person identification: New trends and challenges,”Multimedia Tools Appl., vol. 51, no. 2, pp. 739–777, 2011.

[12] J. Ferryman, “PETS 2007 video database,” in Proc. Privacy EnhancingTechnol. Symp., 2007, pp. 1–103.

[13] D. A. Reid, S. Samangooei, C. Chen, M. S. Nixon, and A. Ross,“Soft biometrics for surveillance: An overview,” in Handbook ofStatistics, vol. 31, C. R. Rao and V. Govindaraju, Eds. Amsterdam,The Netherlands: Elsevier, 2013.

[14] Y. Fu, G. Guo, and T. S. Huang, “Soft biometrics for video sur-veillance,” in Intelligent Video Surveillance: Systems and Technology.Cleveland, OH, USA: CRC Press, 2009, pp. 407–432.

[15] D. De Angelis et al., “New method for height estimation of subjectsrepresented in photograms taken from video surveillance systems,”J. Forensic Legal Med., vol. 121, no. 6, pp. 489–492, 2007.

[16] Y. Ran, Q. Zheng, R. Chellappa, and T. M. Strat, “Applicationsof a simple characterization of human gait in surveillance,” IEEETrans. Syst., Man, Cybern. B, Cybern., vol. 40, no. 4, pp. 1009–1020,Aug. 2010.

[17] S. Denman, C. Fookes, A. Bialkowski, and S. Sridharan, “Soft-biometrics: Unconstrained authentication in a surveillance environ-ment,” in Proc. Int. Conf. Digit. Image Comput., Techn. Appl., 2009,pp. 196–203.

[18] D. A. Vaquero, R. S. Feris, D. Tran, L. Brown, A. Hampapur, andM. Turk, “Attribute-based people search in surveillance environments,”in Proc. IEEE Workshop Appl. Comput. Vis., Dec. 2009, pp. 1–8.

[19] M. Demirkus, K. Garg, and S. Guler, “Automated person categorizationfor video surveillance using soft biometrics,” Proc. SPIE, vol. 7667,p. 76670P, Apr. 2010.

[20] S. Denman, A. Bialkowski, C. Fookes, and S. Sridharan, “Determiningoperational measures from multi-camera surveillance systems usingsoft biometrics,” in Proc. IEEE Int. Conf. Adv. Video Signal BasedSurveill., Aug./Sep. 2011, pp. 462–467.

[21] J. C. Klontz and A. K. Jain, “A case study on unconstrained facialrecognition using the Boston Marathon bombings suspects,” MSU, EastLansing, MI, USA, Tech. Rep. MSU-CSE-13-4, 2013.

[22] A. Dantcheva, J.-L. Dugelay, and P. Elia, “Person recognition using abag of facial soft biometrics (BoFSB),” in Proc. IEEE Int. WorkshopMultimedia Signal Process., Oct. 2010, pp. 511–516.

[23] M. Halstead, S. Denman, S. Sridharan, and C. Fookes, “Locatingpeople in video from semantic descriptions: A new database andapproach,” in Proc. 22nd Int. Conf. Pattern Recognit., Aug. 2014,pp. 4501–4506.

[24] J. Thornton, J. Baran-Gale, D. Butler, M. Chan, and H. Zwahlen,“Person attribute search for large-area video surveillance,” in Proc.IEEE Symp. Technol. Homeland Secur., Nov. 2011, pp. 55–61.

[25] U. Park, A. K. Jain, I. Kitahara, K. Kogure, and N. Hagita, “ViSE:Visual search engine using multiple networked cameras,” in Proc. 18thInt. Conf. Pattern Recognit., vol. 3. 2006, pp. 1204–1207.

[26] W. J. Scheirer, N. Kumar, K. Ricanek, P. N. Belhumeur, and T. E. Boult,“Fusing with context: A Bayesian approach to combining descriptiveattributes,” in Proc. Int. Joint Conf. Biometrics, 2011, pp. 1–8.

[27] A. K. Jain and U. Park, “Facial marks: Soft biometric for facerecognition,” in Proc. 16th IEEE Int. Conf. Image Process., vol. 1.Nov. 2009, pp. 37–40.

[28] M. C. Da Costa Abreu and M. Fairhurst, “Enhancing identity predictionusing a novel approach to combining hard- and soft-biometric infor-mation,” IEEE Trans. Syst., Man, Cybern. C, Appl. Rev., vol. 41, no. 5,pp. 599–607, Sep. 2011.

[29] U. Park and A. K. Jain, “Face matching and retrieval using softbiometrics,” IEEE Trans. Inf. Forensics Security, vol. 5, no. 3,pp. 406–415, Sep. 2010.

[30] N. Kumar, P. N. Belhumeur, and S. K. Nayar, “Facetracer: A searchengine for large collections of images with faces,” in Proc. 10th Eur.Conf. Comput. Vis., 2008, pp. 340–353.

[31] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, “Attributeand simile classifiers for face verification,” in Proc. IEEE 12th Int.Conf. Comput. Vis., Sep./Oct. 2009, pp. 365–372.

[32] L. Hong, A. K. Jain, and S. Pankanti, “Can multibiometrics improveperformance?” in Proc. IEEE AutoID, vol. 99. Oct. 1999, pp. 59–64.

[33] A. Dantcheva, A. Singh, P. Elia, and J.-L. Dugelay, “Search pruningin video surveillance systems: Efficiency-reliability tradeoff,” in Proc.IEEE Int. Conf. Comput. Vis. Workshops, Nov. 2011, pp. 1356–1363.

[34] W. J. Scheirer, N. Kumar, V. N. Iyer, P. N. Belhumeur, andT. E. Boult, “How reliable are your visual attributes?” Proc. SPIE,vol. 8712, p. 87120Q, May 2013.

[35] L. Bourdev, S. Maji, and J. Malik, “Describing people: A poselet-basedapproach to attribute classification,” in Proc. IEEE Int. Conf. Comput.Vis., Nov. 2011, pp. 1543–1550.

[36] Z. Song, M. Wang, X.-S. Hua, and S. Yan, “Predicting occupation viahuman clothing and contexts,” in Proc. IEEE Int. Conf. Comput. Vis.,Nov. 2011, pp. 1084–1091.

[37] W. J. Scheirer, N. Kumar, P. N. Belhumeur, and T. E. Boult, “Multi-attribute spaces: Calibration for attribute fusion and similarity search,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2012,pp. 2933–2940.

[38] O. Russakovsky and L. Fei-Fei, “Attribute learning in large-scaledatasets,” in Trends and Topics in Computer Vision. Berlin, Germany:Springer-Verlag, 2012, pp. 1–14.

[39] D. Parikh and K. Grauman, “Relative attributes,” in Proc. IEEE Int.Conf. Comput. Vis., Nov. 2011, pp. 503–510.

[40] Z. Zhou, Y. Wang, and E. K. Teoh, “People re-identification basedon bags of semantic features,” in Proc. Asian Conf. Comput. Vis.Workshops, 2014, pp. 574–586.

[41] N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar, “Describablevisual attributes for face verification and image search,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 33, no. 10, pp. 1962–1977, Oct. 2011.

[42] M. Douze, A. Ramisa, and C. Schmid, “Combining attributes and fishervectors for efficient image retrieval,” in Proc. IEEE Conf. Comput. Vis.Pattern Recognit., Jun. 2011, pp. 745–752.

[43] C. H. Lampert, H. Nickisch, and S. Harmeling, “Learning to detectunseen object classes by between-class attribute transfer,” in Proc.IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2009, pp. 951–958.

[44] D. Mahajan, S. Sellamanickam, and V. Nair, “A joint learning frame-work for attribute models and object descriptions,” in Proc. IEEE Int.Conf. Comput. Vis., Nov. 2011, pp. 1227–1234.

[45] K. Ricanek and B. Barbour, “What are soft biometrics and how canthey be used?” Computer, vol. 44, no. 9, pp. 106–108, Sep. 2011.

[46] M. Ichino and Y. Yamazaki, “Soft biometrics and its application tosecurity and business,” in Proc. Int. Conf. Biometrics Kansei Eng.,2013, pp. 314–319.

[47] C. B. Ng, Y. H. Tay, and B.-M. Goi, “Recognizing human gender incomputer vision: A survey,” in Proc. 12th Pacific Rim Int. Conf. Artif.Intell., vol. 7458. 2012, pp. 335–346.

[48] S. A. Khan, M. Nazir, S. Akram, and N. Riaz, “Gender classificationusing image processing techniques: A survey,” in Proc. IEEE 14thINMIC, Dec. 2011, pp. 25–30.

[49] J. Bekios-Calfa, J. M. Buenaposada, and L. Baumela, “Revisiting lineardiscriminant techniques in gender recognition,” IEEE Trans. PatternAnal. Mach. Intell., vol. 33, no. 4, pp. 858–864, Apr. 2011.

[50] N. Ramanathan, R. Chellappa, and S. Biswas, “Age progression inhuman faces: A survey,” J. Vis. Lang. Comput., vol. 20, no. 3,pp. 131–144, 2009.

[51] E. Mäkinen and R. Raisamo, “Evaluation of gender classificationmethods with automatically detected and aligned faces,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 30, no. 3, pp. 541–547, Mar. 2008.

[52] E. Mäkinen and R. Raisamo, “An experimental comparison of gen-der classification methods,” Pattern Recognit. Lett., vol. 29, no. 10,pp. 1544–1556, 2008.

[53] G. Guo, “Human age estimation and sex classification,” Video Analyticsfor Business Intelligence. Berlin, Germany: Springer-Verlag, 2012,pp. 101–131.

[54] Y. Fu, G. Guo, and T. S. Huang, “Age synthesis and estimation viafaces: A survey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32,no. 11, pp. 1955–1976, Nov. 2010.

[55] S. Fu, H. He, and Z.-G. Hou, “Learning race from face: A sur-vey,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 12,pp. 2483–2509, Dec. 2014.

[56] G. Mather and L. Murdoch, “Gender discrimination in biologicalmotion displays based on dynamic cues,” in Biol. Sci. B, vol. 258,no. 1353, pp. 273–279, 1994.

[57] K. Sudo, J. Yamato, and A. Tomono, “Determining gender of walkingpeople using multiple sensors,” in Proc. IEEE/SICE/RSJ Int. Conf.Multisensor Fusion Integr. Intell. Syst., Dec. 2011, pp. 641–646.

[58] J. C. McLachlan and H. Storey, “Hot male: Can sex in humansbe modified by temperature?” J. Theoretical Biol., vol. 222, no. 1,pp. 71–72, 2003.

[59] S. R. Loth and M. Iscan, Sex Determination, Encyclopedia of ForensicSciences, vol. 1. San Diego, CA, USA: Academic, 2000.

[60] W. M. Krogman and M. Y. Iscan, Human Skeleton in ForensicMedicine. Springfield, IL, USA: Charles C. Thomas, 1986.


[61] A. J. O’Toole, A. Peterson, and K. A. Deffenbacher, “An ‘other-raceeffect’ for categorizing faces by sex,” Perception, vol. 25, pp. 669–676,Feb. 1996.

[62] B. Edelman, D. Valentin, and H. Abdi, “Sex classification of face areas:How well can a linear neural network predict human performance?”J. Biol. Syst., vol. 6, no. 3, p. 241, 1998.

[63] Z. Sun, G. Bebis, X. Yuan, and S. J. Louis, “Genetic feature subsetselection for gender classification: A comparison study,” in Proc. 6thIEEE Workshop Appl. Comput. Vis., Dec. 2002, pp. 165–170.

[64] D. Zhang, Q. Zhao, and F. Chen, “Quantitative analysis of human facialbeauty using geometric features,” Pattern Recognit., vol. 44, no. 4,pp. 940–950, 2011.

[65] B. Moghaddam and M.-H. Yang, “Learning gender with support faces,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 24, no. 5, pp. 707–711,May 2002.

[66] S. Baluja and H. A. Rowley, “Boosting sex identification performance,”Int. J. Comput. Vis., vol. 71, no. 1, pp. 111–119, 2006.

[67] D. Cao, C. Chen, M. Piccirilli, D. Adjeroh, T. Bourlai, and A. Ross,“Can facial metrology predict gender?” in Proc. Int. Joint Conf.Biometrics, 2011, pp. 1–8.

[68] Y. Saatci and C. Town, “Cascaded classification of gender and facialexpression using active appearance models,” in Proc. IEEE 7th Int.Conf. Autom. Face Gesture Recognit., Apr. 2006, pp. 393–400.

[69] M. Toews and T. Arbel, “Detection, localization, and sex classificationof faces from arbitrary viewpoints and under occlusion,” IEEE Trans.Pattern Anal. Mach. Intell., vol. 31, no. 9, pp. 1567–1581, Sep. 2009.

[70] Z. Yang and H. Ai, “Demographic classification with local binarypatterns,” in Proc. IAPR Int. Conf. Biometrics, 2007, pp. 464–473.

[71] J. Bekios-Calfa, J. M. Buenaposada, and L. Baumela, “Robust gen-der recognition by exploiting facial attributes dependencies,” PatternRecognit. Lett., vol. 36, pp. 228–234, Jan. 2014.

[72] G. Guo, C. R. Dyer, Y. Fu, and T. S. Huang, “Is gender recognitionaffected by age?” in Proc. IEEE 12th Int. Conf. Comput. Vis. Work-shops, Sep./Oct. 2009, pp. 2032–2039.

[73] J.-G. Wang, J. Li, W.-Y. Yau, and E. Sung, “Boosting dense SIFTdescriptors and shape contexts of face images for gender recognition,”in Proc. IEEE Comput. Conf. Comput. Vis. Pattern Recognit. Work-shops, Jun. 2010, pp. 96–102.

[74] B. A. Golomb, D. T. Lawrence, and T. J. Sejnowski, “SexNet: A neuralnetwork identifies sex from human faces,” in Proc. Adv. Neural Inf.Process. Syst., 1990, pp. 572–577.

[75] S. Gutta, P. J. Phillips, and H. Wechsler, “Gender and ethnic classifica-tion of face images,” in Proc. 3rd IEEE Int. Conf. Autom. Face GestureRecognit., Apr. 1998, pp. 194–199.

[76] A. Jain and J. Huang, “Integrating independent components and lineardiscriminant analysis for gender classification,” in Proc. 6th IEEE Int.Conf. Autom. Face Gesture Recognit., May 2004, pp. 159–163.

[77] A. Khan, A. Majid, and A. M. Mirza, “Combination and optimizationof classifiers in gender classification using genetic programming,” Int.J. Knowl.-Based Intell. Eng. Syst., vol. 9, no. 1, pp. 1–11, 2005.

[78] N. Sun, W. Zheng, C. Sun, C. Zou, and L. Zhao, “Gender classificationbased on boosting local binary pattern,” in Proc. 3rd Int. Symp. Adv.Neural Netw., 2006, pp. 194–201.

[79] H.-C. Kim, D. Kim, Z. Ghahramani, and S. Y. Bang, “Appearance-based gender classification with Gaussian processes,” Pattern Recognit.Lett., vol. 27, no. 6, pp. 618–626, 2006.

[80] B. Xia, H. Sun, and B.-L. Lu, “Multi-view gender classification basedon local Gabor binary mapping pattern and support vector machines,”in Proc. Int. Joint Conf. Neural Netw., 2008, pp. 3388–3395.

[81] W. Gao and H. Ai, “Face gender classification on consumer images ina multiethnic environment,” in Proc. 3rd Int. Conf. Biometrics, 2009,pp. 169–178.

[82] C. Shan, “Gender classification on real-life faces,” Advanced Conceptsfor Intelligent Vision Systems. Berlin, Germany: Springer-Verlag, 2010,pp. 323–331.

[83] M. Nazhir, M. Ishaiq, A. Batool, M. A. Jaffar, and A. M. Mirza,“Feature selection for efficient gender classification,” in Proc. 11thWSEAS Int. Conf. Neural Netw., 11th WSEAS Int. Conf. Evol. Comput.,11th WSEAS Int. Conf. Fuzzy Syst., 2010, pp. 70–75.

[84] A. Ross and C. Chen, “Can gender be predicted from near-infraredface images?” in Proc. 8th Int. Conf. Image Anal. Recognit., 2011,pp. 120–129.

[85] S. Y. D. Hu, B. Jou, A. Jaech, and M. Savvides, “Fusion of region-based representations for gender identification,” in Proc. Int. Joint Conf.Biometrics, 2011, pp. 1–7.

[86] C. Shan, “Learning local binary patterns for gender classificationon real-world face images,” Pattern Recognit. Lett., vol. 33, no. 4,pp. 431–437, 2012.

[87] E. Ramón-Balmaseda, J. Lorenzo-Navarro, and M. Castrillón-Santana,“Gender classification in large databases,” in Progress in PatternRecognition, Image Analysis, Computer Vision, and Applications.Berlin, Germany: Springer-Verlag, 2012, pp. 74–81.

[88] S. Jia and N. Cristianini, “Learning to classify gender from four millionimages,” Pattern Recognit. Lett., vol. 58, pp. 35–41, Jun. 2015.

[89] X. Han, H. Ugail, and I. Palmer, “Gender classification based on 3Dface geometry features using SVM,” in Proc. Int. Conf. CyberWorlds,2009, pp. 114–118.

[90] C. Chen and A. Ross, “Evaluation of gender classification methodson thermal and near-infrared face images,” in Proc. Int. Joint Conf.Biometrics, 2011, pp. 1–8.

[91] R. Satta, J. Galbally, and L. Beslay, “Children gender recognition underunconstrained conditions based on contextual information,” in Proc.22nd Int. Conf. Pattern Recognit., 2014, pp. 357–362.

[92] E. O. Omidiora, O. Ojo, N. A. Yekini, and T. O. Tubi, “Analysis,design and implementation of human fingerprint patterns system.Towards age & gender determination, ridge thickness to valley thick-ness ratio (RTVTR) & ridge count on gender detection,” Int. J. Adv.Res. Artif. Intell., vol. 1, no. 2, pp. 57–63, 2012.

[93] S. Gupta and A. P. Rao, “Fingerprint based gender classification usingdiscrete wavelet transform & artificial neural network,” Int. J. Comput.Sci. Mobile Comput., vol. 3, no. 4, pp. 1289–1296, 2014.

[94] A. Badawi, M. Mahfouz, R. Tadross, and R. Jantz, “Fingerprint-basedgender classification,” in Proc. Int. Conf. Image Process., Comput. Vis.Pattern Recognit., 2006, pp. 41–46.

[95] R. J. Tom, T. Arulkumaran, and M. E. Scholar, “Fingerprint basedgender classification using 2D discrete wavelet transforms and principalcomponent analysis,” Int. J. Eng. Trends Technol., vol. 4, no. 2,pp. 199–203, 2013.

[96] P. Gnanasivam and S. Muttan. (2012). “Fingerprint gender classificationusing wavelet transform and singular value decomposition.” [Online].Available: http://arxiv.org/abs/1205.6745

[97] E. Marasco, L. Lugini, and B. Cukic, “Exploiting quality and texturefeatures to estimate age and gender from fingerprints,” Proc. SPIE,vol. 9075, p. 90750F, May 2014.

[98] A. Rattani, C. Chen, and A. Ross, “Evaluation of texture descriptorsfor automated gender estimation from fingerprints,” in Proc. Eur. Conf.Comput. Vis. Workshops, 2014, pp. 764–777.

[99] S. Crihalmeanu, A. Ross, S. Schuckers, and L. Hornak, “A proto-col for multibiometric data acquisition, storage and dissemination,”Dept. Comput. Sci. Elect. Eng., WVU, Morgantown, WV, USA,Tech. Rep. 1095, 2007.

[100] V. Thomas, N. V. Chawla, K. W. Bowyer, and P. J. Flynn, “Learning topredict gender from iris images,” in Proc. IEEE Int. Conf. Biometrics,Theory, Appl. Syst., Sep. 2007, pp. 1–5.

[101] S. Lagree and K. W. Bowyer, “Predicting ethnicity and gender from iristexture,” in Proc. IEEE Int. Conf. Technol. Homeland Secur., Nov. 2011,pp. 440–445.

[102] A. Bansal, R. Agarwal, and R. K. Sharma, “Predicting gender usingiris images,” Res. J. Recent Sci., vol. 3, no. 4, pp. 20–26, Apr. 2014.

[103] S. Yu, T. Tan, K. Huang, K. Jia, and X. Wu, “A study on gait-basedgender classification,” IEEE Trans. Image Process., vol. 18, no. 8,pp. 1905–1910, Aug. 2009.

[104] B. DeCann and A. Ross, “Gait curves for human recognition, backpackdetection, and silhouette correction in a nighttime environment,” Proc.SPIE, vol. 7667, p. 76670Q, Apr. 2010.

[105] C. Shan, S. Gong, and P. W. McOwan, “Fusing gait and face cuesfor human gender recognition,” Neurocomputing, vol. 71, nos. 10–12,pp. 1931–1938, 2008.

[106] L. Cao, M. Dikmen, Y. Fu, and T. S. Huang, “Gender recognition frombody,” in Proc. 16th ACM Int. Conf. Multimedia, 2008, pp. 725–728.

[107] L. Chen, Y. Wang, and Y. Wang, “Gender classification based on fusionof weighted multi-view gait component distance,” in Proc. IEEE Chin.Conf. Pattern Recognit., Nov. 2009, pp. 1–5.

[108] X. Li, S. J. Maybank, S. Yan, D. Tao, and D. Xu, “Gait componentsand their application to gender recognition,” IEEE Trans. Syst., Man,Cybern. C, Appl. Rev., vol. 38, no. 2, pp. 145–155, Mar. 2008.

[109] J. Yoo, D. Hwang, and M. S. Nixon, “Gender classification in humangait using support vector machine,” in Advanced Concepts for Intel-ligent Vision Systems, vol. 3708. Berlin, Germany: Springer-Verlag,2005, pp. 138–145.


[110] T. Kanchan and K. Krishan, “Anthropometry of hand in sex determi-nation of dismembered remains—A review of literature,” J. ForensicLegal Med., vol. 18, no. 1, pp. 14–17, 2011.

[111] J. L. Scheuer and N. M. Elkington, “Sex determination frommetacarpals and the first proximal phalanx,” J. Forensic Sci., vol. 38,no. 4, pp. 769–778, 1993.

[112] A. B. Falsetti, “Sex assessment from metacarpals of the human hand,”J. Forensic Sci., vol. 40, no. 5, pp. 774–776, 1995.

[113] R. A. Lazenby, “Identification of sex from metacarpals: Effect of sideasymmetry,” J. Forensic Sci., vol. 39, no. 5, pp. 1188–1194, 1994.

[114] K. Krishan, T. Kanchan, and A. Sharma, “Sex determination from handand foot dimensions in a North Indian population,” J. Forensic Sci.,vol. 56, no. 2, pp. 453–459, 2011.

[115] G. Amayeh, G. Bebis, and M. Nicolescu, “Gender classification fromhand shape,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.Workshops, Jun. 2008, pp. 1–7.

[116] A. Khotanzad and Y. H. Hong, “Invariant image recognition by Zernikemoments,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 12, no. 5,pp. 489–498, May 1990.

[117] M. Madry-Pronobis, “Automatic gender recognition based on audio-visual cues,” M.S. thesis, KTH, Stockholm, Sweden, 2009.

[118] J. A. Bachorowski and M. J. Owren, “Acoustic correlates of talkersex and individual talker identity are present in a short vowel segmentproduced in running speech,” J. Acoust. Soc. Amer., vol. 106, no. 2,pp. 1054–1063, 1999.

[119] K. Wu and D. G. Childers, “Gender recognition from speech. PartI: Coarse analysis,” J. Acoust. Soc. Amer., vol. 90, pp. 1828–1840,Oct. 1991.

[120] M. Pronobis and M. Magimai-Doss, “Analysis of F0 and cepstralfeatures for robust automatic gender recognition,” IDIAP, Martigny,Switzerland, Tech. Rep. IDIAP-RR-30-2009, 2009.

[121] D. G. Childers and K. Wu, “Gender recognition from speech. Part II:Fine analysis,” J. Acoust. Soc. Amer., vol. 90, pp. 1841–1856,Oct. 1991.

[122] V. N. Sorokin and I. S. Makarov, “Gender recognition from vocalsource,” Acoust. Phys., vol. 54, no. 4, pp. 571–578, 2008.

[123] L. Walawalkar, M. Yeasin, A. M. Narasimhamurthy, and R. Sharma,“Support vector learning for gender classification using audio andvisual cues: A comparison,” in Proc. SVM, 2002, pp. 144–159.

[124] M. Pronobis and M. Magimai-Doss, “Integrating audio and vision forrobust automatic gender recognition,” IDIAP, Martigny, Switzerland,Tech. Rep. IDIAP-RR-73-2008, 2008.

[125] L. El Shafey, E. Khoury, and S. Marcel, “Audio-visual gender recog-nition in uncontrolled environment using variability modeling tech-niques,” in Proc. Int. Joint Conf. Biometrics, 2014, pp. 1–8.

[126] C. Shan, S. Gong, and P. McOwan, “Learning gender from human gaitsand faces,” in Proc. IEEE Int. Conf. Adv. Video Signal Based Surveill.,Sep. 2007, pp. 505–510.

[127] S. Yu, D. Tan, and T. Tan, “A framework for evaluating the effect ofview angle, clothing and carrying condition on gait recognition,” inProc. 18th Int. Conf. Pattern Recognit., 2006, pp. 441–444.

[128] K. Ricanek and T. Tesafaye, “MORPH: A longitudinal image databaseof normal adult age-progression,” in Proc. IEEE 7th Int. Conf. Autom.Face Gesture Recognit., Apr. 2006, pp. 341–345.

[129] G. Guerra-Filho and A. Biswas, “The human motion database:A cognitive and parametric sampling of human motion,” Image Vis.Comput., vol. 30, no. 3, pp. 251–261, 2012.

[130] M. G. Rhodes, “Age estimation of faces: A review,” Appl. Cognit.Psychol., vol. 23, no. 1, pp. 1–12, 2009.

[131] C. S. Morrison, B. Z. Phillips, J. T. Chang, S. R. Sullivan, andH. O. B. Taylor. (2011). “The relationship between age and facialasymmetry.” [Online]. Available: http://meeting.nesps.org/2011/80.cgi

[132] N. Ramanathan and R. Chellappa, “Modeling age progression in youngfaces,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2006,pp. 387–394.

[133] N. Batool, S. Taheri, and R. Chellappa, “Assessment of facial wrinklesas a soft biometrics,” in Proc. 10th IEEE Int. Conf. Autom. Face GestureRecognit., Apr. 2013, pp. 1–7.

[134] A. M. Albert, K. Ricanek, Jr., and E. Patterson, “A review of theliterature on the aging adult skull and face: Implications for forensicscience research and applications,” Forensic Sci. Int., vol. 172, no. 1,pp. 1–9, 2007.

[135] A. M. Mughal, N. Hassan, and A. Ahmed, “Bone age assessmentmethods: A critical review,” Pakistan J. Med. Sci., vol. 30, no. 1,pp. 211–215, 2014.

[136] T. R. Alley, Ed., Social and Applied Aspects of Perceiving Faces.Hillsdale, NJ, USA: Lawrence Erlbaum Association, 1988.

[137] H. Han, C. Otto, and A. K. Jain, “Age estimation from face images:Human vs. machine performance,” in Proc. Int. Conf. Biometrics, 2013,pp. 1–8.

[138] L. G. Farkas, Ed., Anthropometry of the Head and Face. New York,NY, USA: Raven, 1994.

[139] B. Xia, B. B. Amor, M. Daoudi, and H. Drira, “Can 3D shape of theface reveal your age?” in Proc. Int. Conf. Comput. Vis. Theory Appl.,2014, pp. 1–9.

[140] Y. Kwon and N. Lobo, “Age classification from facial images,” in Proc.IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 1994, pp. 762–767.

[141] Y. Kwon and N. da Vitoria Lobo, “Age classification from facialimages,” Comput. Vis. Image Understand., vol. 74, no. 1, pp. 1–21,1999.

[142] G. Guo, G. Mu, Y. Fu, C. Dyer, and T. S. Huang, “A study on automaticage estimation using a large database,” in Proc. IEEE 12th Int. Conf.Comput. Vis., Sep./Oct. 2009, pp. 1986–1991.

[143] T. F. Cootes, G. J. Edwards, and C. J. Taylor, “Active appearancemodels,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 23, no. 6,pp. 681–685, Jun. 2001.

[144] A. Lanitis, C. J. Taylor, and T. J. Cootes, “Toward automatic simulationof aging effects on face images,” IEEE Trans. Pattern Anal. Mach.Intell., vol. 24, no. 4, pp. 442–455, Apr. 2002.

[145] G. Guo, Y. Fu, C. R. Dyer, and T. S. Huang, “Image-based human ageestimation by manifold learning and locally adjusted robust regression,”IEEE Trans. Image Process., vol. 17, no. 7, pp. 1178–1188, Jul. 2008.

[146] N. S. Lakshmiprabha, J. Bhattacharya, and S. Majumder, “Age estima-tion using gender information,” in Computer Networks and IntelligentComputing. Berlin, Germany: Springer-Verlag, 2011, pp. 211–216.

[147] X. Geng, Z.-H. Zhou, and K. Smith-Miles, “Automatic age estimationbased on facial aging patterns,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 29, no. 12, pp. 2234–2240, Dec. 2007.

[148] X. Geng, Z.-H. Zhou, Y. Zhang, G. Li, and H. Dai, “Learning fromfacial aging patterns for automatic age estimation,” in Proc. 14th Annu.ACM Int. Conf. Multimedia, 2006, pp. 307–316.

[149] G. Guo, G. Mu, Y. Fu, and T. S. Huang, “Human age estimationusing bio-inspired features,” in Proc. IEEE Conf. Comput. Vis. PatternRecognit., Jun. 2009, pp. 112–119.

[150] X. Wang, R. Guo, and C. Kambhamettu, “Deeply-learned featurefor age estimation,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.,Jan. 2015, pp. 534–541.

[151] Q. Tian and S. Chen, “Cumulative attribute relation regulariza-tion learning for human age estimation,” Neurocomputing, vol. 165,pp. 456–467, Oct. 2015.

[152] Y. Fu, Y. Xu, and T. S. Huang, “Estimating human age by manifoldanalysis of face pictures and regression on aging features,” in Proc.IEEE Int. Conf. Multimedia Expo, Jul. 2007, pp. 1383–1386.

[153] H. S. Seung and D. D. Lee, “The manifold ways of perception,”Science, vol. 290, no. 5500, pp. 2268–2269, 2000.

[154] S. Yan, D. Xu, B. Zhang, H.-J. Zhang, Q. Yang, and S. Lin, “Graphembedding and extensions: A general framework for dimensionalityreduction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 29, no. 1,pp. 40–51, Jan. 2007.

[155] J. B. Tenenbaum, V. de Silva, and J. C. Langford, “A global geometricframework for nonlinear dimensionality reduction,” Science, vol. 290,no. 5500, pp. 2319–2323, 2000.

[156] C. Li, Q. Liu, J. Liu, and H. Lu, “Learning ordinal discriminativefeatures for age estimation,” in Proc. IEEE Conf. Comput. Vis. PatternRecognit., Jun. 2012, pp. 2570–2577.

[157] G. Guo, Y. Fu, T. S. Huang, and C. R. Dyer, “Locally adjusted robustregression for human age estimation,” in Proc. IEEE Workshop Appl.Comput. Vis., Jan. 2008, pp. 1–6.

[158] G. Guo, “Soft biometrics from face images using support vectormachines,” in Support Vector Machines Applications, Y. Ma andG. Guo, Eds. New York, NY, USA: Springer-Verlag, 2014,pp. 269–302.

[159] G. Guo, Y. Fu, C. R. Dyer, and T. S. Huang, “A probabilistic fusionapproach to human age prediction,” in Proc. IEEE Comput. Soc. Conf.Comput. Vis. Pattern Recognit. Workshops, Jun. 2008, pp. 1–6.

[160] T. Kanno, M. Akiba, Y. Teramachi, H. Nagahashi, and T. Agui,“Classification of age group based on facial images of young malesby using neural networks,” IEICE Trans. Inf. Syst., vol. E84-D, no. 8,pp. 1094–1101, 2001.

[161] A. Lanitis, C. Draganova, and C. Christodoulou, “Comparing differentclassifiers for automatic age estimation,” IEEE Trans. Syst., Man,Cybern. B, Cybern., vol. 34, no. 1, pp. 621–628, Feb. 2004.


[162] S. K. Zhou, B. Georgescu, X. S. Zhou, and D. Comaniciu, “Imagebased regression using boosting method,” in Proc. 10th IEEE Int. Conf.Comput. Vis., Oct. 2005, pp. 541–548.

[163] K. Ueki, T. Hayashida, and T. Kobayashi, “Subspace-based age-groupclassification using facial images under various lighting conditions,” inProc. IEEE 7th Int. Conf. Autom. Face Gesture Recognit., Apr. 2006,pp. 43–48.

[164] H. Takimoto, Y. Mitsukura, M. Fukumi, and N. Akamatsu, “A designof gender and age estimation system based on facial knowledge,” inProc. Int. Joint Conf. SICE-ICASE, Oct. 2006, pp. 3883–3886.

[165] H. Takimoto, T. Kuwano, Y. Mitsukura, H. Fukai, and M. Fukumi,“Appearance-age feature extraction from facial image based on ageperception,” in Proc. Annu. Conf. SICE, Sep. 2007, pp. 2813–2818.

[166] Y. Fu and T. S. Huang, “Human age estimation with regression ondiscriminative aging manifold,” IEEE Trans. Multimedia, vol. 10, no. 4,pp. 578–584, Jun. 2008.

[167] X. Zhuang, X. Zhou, M. Hasegawa-Johnson, and T. Huang, “Face ageestimation using patch-based hidden Markov model supervectors,” inProc. 19th Int. Conf. Pattern Recognit., Dec. 2008, pp. 1–4.

[168] B. Ni, Z. Song, and S. Yan, “Web image mining towards universal ageestimator,” in Proc. 17th ACM Int. Conf. Multimedia, 2009, pp. 85–94.

[169] B. Xiao, X. Yang, Y. Xu, and H. Zha, “Learning distance metric forregression by semidefinite programming with application to humanage estimation,” in Proc. 17th ACM Int. Conf. Multimedia, 2009,pp. 451–460.

[170] S. Yan, H. Wang, Y. Fu, J. Yan, X. Tang, and T. S. Huang, “Synchro-nized submanifold embedding for person-independent pose estimationand beyond,” IEEE Trans. Image Process., vol. 18, no. 1, pp. 202–210,Jan. 2009.

[171] J. Suo, S.-C. Zhu, S. Shan, and X. Chen, “A compositional and dynamicmodel for face aging,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 32,no. 3, pp. 385–401, Mar. 2010.

[172] K. Chen, S. Gong, T. Xiang, and C. C. Loy, “Cumulative attribute spacefor age and crowd density estimation,” in Proc. IEEE Conf. Comput.Vis. Pattern Recognit. (CVPR), Jun. 2013, pp. 2467–2474.

[173] J. Lu and Y.-P. Tan, “Gait-based human age estimation,” IEEE Trans.Inf. Forensics Security, vol. 5, no. 4, pp. 761–770, Dec. 2010.

[174] S. Sarkar, P. J. Phillips, Z. Liu, I. R. Vega, P. Grother, andK. W. Bowyer, “The humanID gait challenge problem: Data sets,performance, and analysis,” IEEE Trans. Pattern Anal. Mach. Intell.,vol. 27, no. 2, pp. 162–177, Feb. 2005.

[175] L. Shamir, “Automatic age estimation by hand photos,” Comput. Sci.Lett., vol. 3, no. 1, 2011, pp. 1–6.

[176] K. Scherbaum, M. Sunkel, H.-P. Seidel, and V. Blanz, “Predictionof individual non-linear aging trajectories of faces,” Comput. Graph.Forum, vol. 26, no. 3, pp. 285–294, 2007.

[177] E. Mordini and D. Tzovaras, Second Generation Biometrics:The Ethical, Legal and Social Context. New York, NY, USA:Springer-Verlag, 2012.

[178] E. Marshall, “DNA studies challenge the meaning of race,” Science,vol. 282, no. 5389, pp. 654–655, 1998.

[179] N. Angier, Do Races Differ? Not Really, Genes Show. New York, NY,USA: The New York Times, Aug. 2000.

[180] N. A. Rosenberg et al., “Genetic structure of human populations,”Science, vol. 298, no. 5602, pp. 2381–2385, 2002.

[181] A. O’Toole, K. A. Deffenbacher, D. Valentin, and H. Abdi, “Structuralaspects of face recognition and the other-race effect,” Memory Cognit.,vol. 22, no. 2, pp. 208–224, 1994.

[182] P. J. Phillips, F. Jiang, A. Narvekar, J. Ayyad, and A. J. O’Toole,“An other-race effect for face recognition algorithms,” ACM Trans.Appl. Perception, vol. 8, no. 2, 2011, Art. ID 14.

[183] D. A. Wilbraham, J. C. Christensen, A. M. Martinez, and J. T. Todd,“Can low level image differences account for the ability of humanobservers to discriminate facial identity?” J. Vis., vol. 8, no. 15, p. 5,2008.

[184] Y. Xie, K. Luu, and M. Savvides, “A robust approach to facial ethnicityclassification on large scale face databases,” in Proc. IEEE 5th Int.Conf. BTAS, Sep. 2012, pp. 143–149.

[185] S. M. M. Roomi, S. L. Virasundarii, S. Selvamegala, S. Jeevanandham,and D. Hariharasudhan, “Race classification based on facial features,”in Proc. IEEE 3rd Nat. Conf. Comput. Vis., Pattern Recognit., ImageProcess., Graph., Dec. 2011, pp. 54–57.

[186] U. Tariq, Y. Hu, and T. S. Huang, “Gender and ethnicity identificationfrom silhouetted face profiles,” in Proc. 16th IEEE Int. Conf. ImageProcess., Nov. 2009, pp. 2441–2444.

[187] V. Natu, D. Raboy, and A. J. O’Toole, “Neural correlates of own- andother-race face perception: Spatial and temporal response differences,”NeuroImage, vol. 54, no. 3, pp. 2547–2555, 2011.

[188] X. Lu and A. K. Jain, “Ethnicity identification from face images,”Proc. SPIE, Biometric Technol. Human Identificat., vol. 5404, p. 114,Aug. 2004.

[189] H. Lin, H. Lu, and L. Zhang, “A new automatic recognition system ofgender, age and ethnicity,” in Proc. 6th World Congr. Intell. ControlAutom., vol. 2. 2006, pp. 9988–9991.

[190] B. Klare and A. K. Jain, “On a taxonomy of facial features,” inProc. 4th IEEE Int. Conf. Biometrics, Theory, Appl., Syst., Sep. 2010,pp. 1–8.

[191] J. Heo and M. Savvides, “Gender and ethnicity specific generic elasticmodels from a single 2D image for novel 2D pose face synthesis andrecognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 34, no. 12,pp. 2341–2350, Dec. 2012.

[192] S. Hosoi, E. Takikawa, and M. Kawade, “Ethnicity estimation withfacial images,” in Proc. 6th IEEE Int. Conf. Autom. Face GestureRecognit., May 2004, pp. 195–200.

[193] S.-Y. Fu, G.-S. Yang, and X.-K. Kuai, “A spiking neural networkbased cortex-like mechanism and application to facial expressionrecognition,” Comput. Intell. Neurosci., vol. 2012, Jul. 2012,Art. ID 946589.

[194] H. Ding, D. Huang, Y. Wang, and L. Chen, “Facial ethnicity clas-sification based on boosted local texture and shape descriptions,” inProc. 10th IEEE Int. Conf. Autom. Face Gesture Recognit., Apr. 2013,pp. 1–6.

[195] G. Shakhnarovich, P. A. Viola, and B. Moghaddam, “A unified learningframework for real time face detection and classification,” in Proc. 5thIEEE Int. Conf. Autom. Face Gesture Recognit., May 2002, pp. 14–21.

[196] X. Lu, H. Chen, and A. K. Jain, “Multimodal facial gender and ethnic-ity identification,” in Proc. Int. Conf. Biometrics, 2006, pp. 554–561.

[197] G. Zhang and Y. Wang, “Multimodal 2D and 3D facial ethnicityclassification,” in Proc. 5th Int. Conf. Image Graph., Sep. 2009,pp. 928–932.

[198] D. Zhang, Y. Wang, Z. Zhang, and M. Hu, “Ethnicity classificationbased on fusion of face and gait,” in Proc. 5th IAPR Int. Conf.Biometrics, Mar./Apr. 2012, pp. 384–389.

[199] D. Zhang, Y. Wang, and Z. Zhang, “Ethnicity classification basedon a hierarchical fusion,” in Biometric Recognition (Lecture Notes inComputer Science), vol. 7701, W.-S. Zheng, Z. Sun, Y. Wang, X. Chen,P. C. Yuen, and J. Lai, Eds. Berlin, Germany: Springer-Verlag, 2012,pp. 300–307.

[200] X. Qiu, Z. Sun, and T. Tan, “Global texture analysis of iris images forethnic classification,” in Proc. IAPR Int. Conf. Adv. Biometrics, 2006,pp. 411–418.

[201] X. Qiu, Z. Sun, and T. Tan, “Learning appearance primitives of irisimages for ethnic classification,” in Proc. IEEE Int. Conf. ImageProcess., Sep./Oct. 2007, pp. II-405–II-408.

[202] H. Zhang, Z. Sun, T. Tan, and J. Wang, “Ethnic classification basedon iris images,” in Biometric Recognition (Lecture Notes in ComputerScience), vol. 7098, Z. Sun, J. Lai, X. Chen, and T. Tan, Eds. Berlin,Germany: Springer-Verlag, 2011, pp. 82–90.

[203] H. Proenca, S. Filipe, R. Santos, J. Oliveira, and L. A. Alexandre,“The UBIRIS.v2: A database of visible wavelength iris images capturedon-the-move and at-a-distance,” IEEE Trans. Pattern Anal. Mach.Intell., vol. 32, no. 8, pp. 1529–1535, Aug. 2009.

[204] P. Lucey, J. F. Cohn, T. Kanade, J. Saragih, Z. Ambadar, andI. Matthews, “The extended Cohn–Kanade dataset (CK+): A completedataset for action unit and emotion-specified expression,” in Proc.IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Workshops,Jun. 2010, pp. 94–101.

[205] D. Riccio, G. Tortora, M. De Marsico, and H. Wechsler,“EGA—Ethnicity, gender and age, a pre-annotated face database,”in Proc. IEEE Workshop Biometric Meas. Syst. Secur. Med. Appl.,Sep. 2012, pp. 1–8.

[206] H. Han, C. Otto, X. Liu, and A. K. Jain, “Demographic estimation fromface images: Human vs. machine performance,” IEEE Trans. PatternAnal. Mach. Intell., vol. 37, no. 6, pp. 1148–1161, Jun. 2015.

[207] G. Guo and G. Mu, “A study of large-scale ethnicity estimation withgender and age variations,” in Proc. IEEE Comput. Soc. Conf. Comput.Vis. Pattern Recognit. Workshops, Jun. 2010, pp. 79–86.

[208] F. S. Manesh, M. Ghahramani, and Y. P. Tan, “Facial part displacementeffect on template-based gender and ethnicity classification,” inProc. 11th Int. Conf. Control, Autom., Robot., Vis., Dec. 2010,pp. 1644–1649.


[209] H. Han and A. K. Jain, “Age, gender and race estimation fromunconstrained face images,” Dept. Comput. Sci. Eng., Michigan StateUniv., East Lansing, MI, USA, Tech. Rep. MSU-CSE-14-5, 2014.

[210] N. S. Gavrilova and L. A. Gavrilov, “Genetics: Ethnicity,” inEncyclopedia of Aging, vol. 2. New York, NY, USA: Macmillan, 2002,pp. 538–539.

[211] G. Farinella and J.-L. Dugelay, “Demographic classification: Do genderand ethnicity affect each other?” in Proc. Int. Conf. Informat., Electron.,Vis., May 2012, pp. 383–390.

[212] Z. Zeng, M. Pantic, G. I. Roisman, and T. S. Huang, “A survey of affectrecognition methods: Audio, visual, and spontaneous expressions,”IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 1, pp. 39–58,Jan. 2009.

[213] A. Criminisi, I. Reid, and A. Zisserman, “Single view metrology,” Int.J. Comput. Vis., vol. 40, no. 2, pp. 123–148, 2000.

[214] Z. Zhang, “A flexible new technique for camera calibration,” IEEETrans. Pattern Anal. Mach. Intell., vol. 22, no. 11, pp. 1330–1334,Nov. 2000.

[215] K. Krishan, “Anthropometry in forensic medicine and forensicscience-‘forensic anthropometry,”’ Internet J. Forensic Sci., vol. 2,no. 1, pp. 95–97, 2007.

[216] M. H. Raxter, B. M. Auerbach, and C. B. Ruff, “Revision of the fullytechnique for estimating statures,” Amer. J. Phys. Anthropol., vol. 130,no. 3, pp. 374–384, 2006.

[217] C. Pelin, I. Duyar, E. M. Kayahan, R. Zagyapan, A. M. Agildere, andA. Erar, “Body height estimation based on dimensions of sacral andcoccygeal vertebrae,” J. Forensic Sci., vol. 50, no. 2, pp. 294–297,2005.

[218] R. Hauser, J. Smolinski, and T. Gos, “The estimation of stature onthe basis of measurements of the femur,” Forensic Sci. Int., vol. 147,nos. 2–3, pp. 185–190, 2005.

[219] F. Introna, Jr., G. Di Vella, and S. Petrachi, “Determination of heightin life using multiple regression of skull parameters,” Bull. Italian Soc.Experim. Biol., vol. 69, no. 3, pp. 153–160, 1993.

[220] P. Dias et al., “A new approach for 3D craniometric measurementsusing 3D skull models,” in Proc. 17th Inf. Conf. Visualisat. (IV),Jul. 2013, pp. 462–467.

[221] M. Steyn, P. J. Becker, E. N. L’Abbé, Y. Scholtz, and J. Myburgh,“An assessment of the repeatability of pubic and ischial measurements,”Forensic Sci. Int., vol. 214, nos. 1–3, pp. 210.e1–210.e4, 2012.

[222] B. M. Auerbach, “Methods for estimating missing human skeletalelement osteometric dimensions employed in the revised fullytechnique for estimating stature,” Amer. J. Phys. Anthropol., vol. 145,no. 1, pp. 67–80, 2011.

[223] L. Ding and A. M. Martinez, “Precise detailed detection of faces andfacial features,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.,Jun. 2008, pp. 1–7.

[224] M. Nixon, “Eye spacing measurement for facial recognition,” Proc.SPIE, Appl. Digit. Image Process. VIII, vol. 575, pp. 279–285,Dec. 1985.

[225] V. Ramanathan and H. Wechsler, “Robust human authentication usingappearance and holistic anthropometric features,” Pattern Recognit.Lett., vol. 31, no. 15, pp. 2425–2435, 2010.

[226] A. El Kissi Ghalleb, S. Sghaier, and N. E. Ben Amara, “Face recogni-tion improvement using soft biometrics,” in Proc. 10th Int. Multi-Conf.SSD, Mar. 2013, pp. 1–6.

[227] C. BenAbdelkader and L. Davis, “Estimation of anthropomeasuresfrom a single calibrated camera,” in Proc. IEEE 7th Int. Conf. Autom.Face Gesture Recognit., Apr. 2006, pp. 499–504.

[228] C. Madden and M. Piccardi, “Height measurement as a session-basedbiometric for people matching across disjoint camera views,” in Proc.Image Vis. Comput. New Zealand, 2005, pp. 282–286.

[229] A. Y. Johnson and A. F. Bobick, “Gait recognition using static,activity-specific parameters,” in Proc. IEEE Comput. Soc. Conf.Comput. Vis. Pattern Recognit., 2001, pp. I-423–I-430.

[230] C. BenAbdelkader, R. Cutler, and L. Davis, “View-invariant estimationof height and stride for gait recognition,” in Proc. Int. Conf. BiometricAuthentication, vol. 2359. 2002, pp. 155–167.

[231] E. Gonzalez-Sosa, R. Vera-Rodriguez, J. Fierrez, and J. Ortega-Garcia,“Comparison of body shape descriptors for biometric recognition usingMMW images,” in Proc. 22nd Int. Conf. Pattern Recognit., Aug. 2014,pp. 124–129.

[232] M. Moreno-Moreno, J. Fierrez, R. Vera-Rodriguez, and J. Parron,“Distance-based feature extraction for biometric recognition ofmillimeter wave body images,” in Proc. IEEE Int. Carnahan Conf.Secur. Technol., Oct. 2011, pp. 1–6.

[233] J. Carnický and D. Chorvát, Jr., “Three-dimensional measurement ofhuman face with structured-light illumination,” Meas. Sci. Rev., vol. 6,no. 1, pp. 1–4, 2006.

[234] B. Allen, B. Curless, and Z. Popovic, “The space of human bodyshapes: Reconstruction and parameterization from range scans,” ACMTrans. Graph., vol. 22, no. 3, pp. 587–594, 2003.

[235] D. Adjeroh, D. Cao, M. Piccirilli, and A. Ross, “Predictability andcorrelation in human metrology,” in Proc. IEEE Int. Workshop Inf.Forensics Secur., Dec. 2010, pp. 1–6.

[236] N. Hasler, C. Stoll, B. Rosenhahn, T. Thormählen, and H.-P. Seidel,“Estimating body shape of dressed humans,” Comput. Graph., vol. 33,no. 3, pp. 211–216, 2009.

[237] A. O. Balan and M. J. Black, “The naked truth: Estimating bodyshape under clothing,” in Proc. 10th Eur. Conf. Comput. Vis., 2008,pp. 15–29.

[238] A. Godil and S. Ressler. (May 2011). “Retrieval and clustering froma 3D human database based on body and head shape.” [Online].Available: http://arxiv.org/abs/1105.2800

[239] K. Moustakas, D. Tzovaras, and G. Stavropoulos, “Gait recognitionusing geometric features and soft biometrics,” IEEE Signal Process.Lett., vol. 17, no. 4, pp. 367–370, Apr. 2010.

[240] A. Rodrigues, J. Sa Silva, and F. Boavida, “iSenior—A support systemfor elderly citizens,” IEEE Trans. Emerg. Topics Comput., vol. 1, no. 2,pp. 207–217, Dec. 2013.

[241] K. Ozcan, A. K. Mahabalagiri, M. Casares, and S. Velipasalar,“Automatic fall detection and activity classification by a wearableembedded smart camera,” IEEE J. Emerg. Sel. Topics Circuits Syst.,vol. 3, no. 2, pp. 125–136, Jun. 2013.

[242] S. Wang, M. Skubic, and Y. Zhu, “Activity density map visualizationand dissimilarity comparison for eldercare monitoring,” IEEE Trans.Inf. Technol. Biomed., vol. 16, no. 4, pp. 607–614, Jul. 2012.

[243] E. A. Yates, A. K. Macpherson, and J. L. Kuk, “Secular trends in thediagnosis and treatment of obesity among US adults in the primarycare setting,” Obesity, vol. 20, no. 9, pp. 1909–1914, 2012.

[244] C. Velardo and J.-L. Dugelay, “Weight estimation from visual bodyappearance,” in Proc. 4th IEEE Int. Conf. Biometrics, Theory, Appl.,Syst., Sep. 2010, pp. 1–6.

[245] C. Velardo, J.-L. Dugelay, M. Paleari, and P. Ariano, “Buildingthe space scale or how to weigh a person with no gravity,” inProc. IEEE Int. Conf. Emerg. Signal Process. Appl., Jan. 2012,pp. 67–70.

[246] R. D. Labati, A. Genovese, V. Piuri, and F. Scotti, “Weight estimationfrom frame sequences using computational intelligence techniques,” inProc. IEEE Int. Conf. Comput. Intell. Meas. Syst. Appl., Jul. 2012,pp. 29–34.

[247] M. Sadeghi, T. K. Lee, D. I. McLean, H. Lui, and M. S. Atkins,“Detection and analysis of irregular streaks in dermoscopic imagesof skin lesions,” IEEE Trans. Med. Imag., vol. 32, no. 5, pp. 849–861,May 2013.

[248] R. Hennessy, S. Bish, J. W. Tunnell, and M. K. Markey, “Segmentationof diffuse reflectance hyperspectral datasets with noise for detectionof melanoma,” in Proc. Annu. Int. Conf. IEEE Eng. Med. Biol. Soc.,Aug./Sep. 2012, pp. 1482–1485.

[249] B. D’Alessandro and A. P. Dhawan, “3-D volume reconstruction of skinlesions for melanin and blood volume estimation and lesion severityanalysis,” IEEE Trans. Med. Imag., vol. 31, no. 11, pp. 2083–2092,Nov. 2012.

[250] T. Wadhawan, N. Situ, H. Rui, K. Lancaster, X. Yuan, andG. Zouridakis, “Implementation of the 7-point checklist for melanomadetection on smart handheld devices,” in Proc. Annu. Int. Conf. IEEEEng. Med. Biol. Soc., Aug./Sep. 2011, pp. 3180–3183.

[251] J. Lu, E. Kazmierczak, J. H. Manton, and R. Sinclair, “Automaticsegmentation of scaling in 2-D psoriasis skin images,” IEEE Trans.Med. Imag., vol. 32, no. 4, pp. 719–730, Apr. 2013.

[252] M. H. A. Fadzil, D. Ihtatho, A. M. Affandi, and S. H. Hussein,“Objective assessment of psoriasis erythema for PASI scoring,” J. Med.Eng. Technol., vol. 33, no. 7, pp. 516–524, 2009.

[253] J. Leachtenauer, S. Kell, B. Turner, C. Newcomer, C. Lyder, andM. Alwan, “A non-contact imaging-based approach to detecting stage Ipressure ulcers,” in Proc. 28th Annu. Int. Conf. IEEE Eng. Med. Biol.Soc., Aug./Sep. 2006, pp. 6380–6383.

[254] S. Prigent, X. Descombes, D. Zugaj, L. Petit, and J. Zerubia,“Multi-scale analysis of skin hyper-pigmentation evolution,” in Proc.IEEE Int. Conf. Image Process., Sep. 2013, pp. 626–629.

[255] A. D’Angelo and J.-L. Dugelay, “People re-identification in cameranetworks based on probabilistic color histograms,” Proc. SPIE, Vis.Inf. Process. Commun. II, vol. 7882, p. 78820K, Jan. 2011.


[256] S. M. Yoon and S.-C. Kee, “Detection of partially occluded face usingsupport vector machines,” in Proc. IAPR MVA, 2002, pp. 546–549.

[257] C.-Y. Wen, S.-H. Chiu, Y.-R. Tseng, and C.-P. Lu, “The mask detectiontechnology for occluded face analysis in the surveillance system,”J. Forensic Sci., vol. 50, no. 3, pp. 593–601, 2005.

[258] D.-T. Lin and M.-J. Liu, “Face occlusion detection for automated tellermachine surveillance,” in Advances in Image and Video Technology(Lecture Notes in Computer Science), vol. 4319. Berlin, Germany:Springer-Verlag, 2006, pp. 641–651.

[259] R. Min, A. D’Angelo, and J.-L. Dugelay, “Efficient scarf detectionprior to face recognition,” in Proc. 18th Eur. Signal Process. Conf.,Aug. 2010, pp. 259–263.

[260] A. M. Martinez and R. Benavente, “The AR face database,” Comput.Vis. Center, Barcelona, Spain, Tech. Rep. 24, Jun. 1998.

[261] R. Min, A. Hadid, and J. L. Dugelay, “Improving the recognition offaces occluded by facial accessories,” in Proc. IEEE Int. Conf. Autom.Face Gesture Recognit., Mar. 2011, pp. 442–447.

[262] S. Du, M. Shehata, and W. Badawy, “Hard hat detection in videosequences based on face features, motion and color information,” inProc. 3rd Int. Conf. Comput. Res. Develop., Mar. 2011, pp. 25–29.

[263] R. Min and J.-L. Dugelay, “Cap detection for moving people inentrance surveillance,” in Proc. 19th ACM Int. Conf. Multimedia, 2011,pp. 1253–1256.

[264] S. E. Baker, A. Hentz, K. W. Bowyer, and P. J. Flynn, “Degrada-tion of iris recognition performance due to non-cosmetic prescriptioncontact lenses,” Comput. Vis. Image Understand., vol. 114, no. 9,pp. 1030–1044, 2010. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S1077314210001360

[265] G. Erdogan and A. Ross, “Automatic detection of non-cosmetic softcontact lenses in ocular images,” Proc. SPIE, vol. 8712, p. 87120C,May 2013.

[266] X. Jiang, M. Binkert, B. Achermann, and H. Bunke, “Towards detectionof glasses in facial images,” Pattern Anal. Appl., vol. 3, no. 1, pp. 9–18,2000.

[267] Y. Xiao and H. Yan, “Extraction of glasses in human face images,” inProc. 1st Int. Conf. Biometric Authentication, 2004, pp. 214–220.

[268] H. Wu, G. Yoshikawa, T. Shioyama, S. Lao, and M. Kawade, “Glassesframe detection with 3D Hough transform,” in Proc. 16th Int. Conf.Pattern Recognit., 2002, pp. 346–349.

[269] J. Heo, S. G. Kong, B. R. Abidi, and M. A. Abidi, “Fusion ofvisual and thermal signatures with eyeglass removal for robust facerecognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit.Workshops, Jun. 2004, p. 122.

[270] A. Dantcheva, C. Chen, and A. Ross, “Can facial cosmetics affect thematching accuracy of face recognition systems?” in Proc. IEEE 5thInt. Conf. BTAS, Sep. 2012, pp. 391–398.

[271] C. Chen, A. Dantcheva, and A. Ross, “Automatic facial makeupdetection with application in face recognition,” in Proc. IAPR Int. Conf.Biometrics, Jun. 2013, pp. 1–8.

[272] S. Deshpande, S. Chikkerur, and V. Govindaraju, “Accent classificationin speech,” in Proc. 4th IEEE AutoID, Oct. 2005, pp. 139–143.

[273] A. Martinez and S. Du, “A model of the perception of facial expressionsof emotion by humans: Research overview and perspectives,” J. Mach.Learn. Res., vol. 13, no. 1, pp. 1589–1608, 2012.

[274] G. Sandbach, S. Zafeiriou, M. Pantic, and L. Yin, “Static and dynamic3D facial expression recognition: A comprehensive survey,” Image Vis.Comput., vol. 30, no. 10, pp. 683–697, 2012.

[275] A. L. Kashyap, S. Tulyakov, and V. Govindaraju, “Facial behavioras a soft biometric,” in Proc. 5th IAPR Int. Conf. Biometrics,Mar./Apr. 2012, pp. 147–151.

[276] N. A. Spaun, “Forensic biometrics from images and video at the FederalBureau of Investigation,” in Proc. 1st IEEE Int. Conf. Biometrics,Theory, Appl., Syst., Sep. 2007, pp. 1–3.

[277] J.-E. Lee, A. K. Jain, and R. Jin, “Scars, marks and tattoos (SMT): Softbiometric for suspect and victim identification,” in Proc. BiometricsSymp., Sep. 2008, pp. 1–8.

[278] U. Park, “Face recognition: Face in video, age invariance, and facialmarks,” Ph.D. dissertation, Dept. Comput. Sci., Michigan State Univ.,East Lansing, MI, USA, 2009.

[279] D. Lin and X. Tang, “Recognize high resolution faces: Frommacrocosm to microcosm,” in Proc. IEEE Comput. Soc. Conf. Comput.Vis. Pattern Recognit., Jun. 2006, pp. 1355–1362.

[280] J.-S. Pierrard and T. Vetter, “Skin detail analysis for face recognition,”in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2007, pp. 1–8.

[281] N. Srinivas, G. Aggarwal, P. J. Flynn, and R. W. Vorder Bruegge,“Analysis of facial marks to distinguish between identical twins,” IEEETrans. Inf. Forensics Security, vol. 7, no. 5, pp. 1536–1550, Oct. 2012.

[282] A. Nurhudatiana et al., “The individuality of relatively permanentpigmented or vascular skin marks (RPPVSM) in independently anduniformly distributed patterns,” IEEE Trans. Inf. Forensics Security,vol. 8, no. 6, pp. 998–1012, Jun. 2013.

[283] C. I. Watson, “Mugshot identification data—Fronts and profiles,” NIST,Gaithersburg, MD, USA, Reference Data NIST Special Database 18,1994.

[284] N. Chhaya and T. Oates, “Joint inference of soft biometric features,”in Proc. 5th IAPR Int. Conf. Biometrics, Mar./Apr. 2012, pp. 466–471.

[285] J. C. Yuille and J. L. Cutshall, “A case study of eyewitness memoryof a crime,” J. Appl. Psychol., vol. 71, no. 2, pp. 291–301, 1986.

[286] D. A. Reid, M. S. Nixon, and S. V. Stevenage, “Soft biometrics; humanidentification using comparative descriptions,” IEEE Trans. PatternAnal. Mach. Intell., vol. 36, no. 6, pp. 1216–1228, Jun. 2013.

[287] A. Dantcheva, J.-L. Dugelay, and P. Elia, “Soft biometrics systems:Reliability and asymptotic bounds,” in Proc. 4th IEEE Int. Conf.Biometrics, Theory, Appl., Syst., Sep. 2010, pp. 1–6.

[288] C. Chen, A. Dantcheva, and A. Ross, “Impact of facial cosmetics onautomatic gender and age estimation algorithms,” in Proc. Int. Conf.Comput. Vis. Theory Appl., 2014, pp. 1–9.

[289] R. Feng and B. Prabhakaran, “Quantifying the makeup effect in femalefaces and its applications for age estimation,” in Proc. IEEE Int. Symp.Multimedia, Dec. 2012, pp. 108–115.

[290] E. J. Kindt, “Privacy and data protection issues of biometric appli-cations,” in A Comparative Legal Analysis (Law, Governance andTechnology Series), vol. 12. New York, NY, USA: Springer-Verlag,2013.

[291] P. Agrawal and P. J. Narayanan, “Person de-identification in videos,”IEEE Trans. Circuits Syst. Video Technol., vol. 21, no. 3, pp. 299–310,Mar. 2011.

[292] C. Velardo, C. Araimo, and J.-L. Dugelay, “Synthetic and privacy-preserving visualization of video sensor network outputs,” in Proc. 5thACM/IEEE Int. Conf. Distrib. Smart Cameras, Aug. 2011, pp. 1–5.

[293] A. Othman and A. Ross, “Privacy of facial soft biometrics: Suppress-ing gender but retaining identity,” in Proc. Eur. Conf. Comput. Vis.Workshops, 2014, pp. 682–696.

[294] D. Freire-Obregon, M. Castrillon-Santana, E. Ramon-Balmaseda,and J. Lorenzo-Navarro, “Automatic clothes segmentation for softbiometrics,” in Proc. IEEE Int. Conf. Image Process., Oct. 2014,pp. 4972–4976.

Antitza Dantcheva received the Ph.D. degree insignal and image processing from Eurecom/TelecomParisTech, France, in 2011. She was a PostdoctoralFellow with Michigan State University andWest Virginia University, USA. She is currentlya Marie Curie Postdoctoral Fellow with theSTARS Team, Institut National de Recherche enInformatique et en Automatique, France. She hasworked on retrieval of soft biometrics from imagesand their corresponding analysis. Her researchinterests are in soft biometrics for security and

commercial applications. She was a recipient of the Best Presentation Awardin ICME 2011, the Best Poster Award in ICB 2013, and the Tabula RasaSpoofing Award in ICB 2013.


Petros Elia received the B.Sc. degree from theIllinois Institute of Technology, and the M.Sc. andPh.D. degrees in electrical engineering from the Uni-versity of Southern California (USC), Los Angeles,in 2001 and 2006, respectively. Since 2008, he hasbeen with Eurecom. He is currently an AssistantProfessor with the Department of Mobile Commu-nications, Eurecom, Sophia Antipolis, France.

His latest research deals with the role of feed-back and complexity in multiuser communications,MIMO, cooperative and multiple access protocols

and transceivers, complexity of communication, and isolation and connectivityin dense networks, queuing theory and cross-layer design, coding theory,information theoretic limits in cooperative communications, and surveillancenetworks. He is a Fulbright Scholar. He was a corecipient of the SPAWC-2011Best Student Paper Award on the topic of reduced complexity bidirectionalcommunication with limited feedback, and the NEWCOM++ DistinguishedAchievement Award 2008–2011 for a sequence of publications on the topicof reduced complexity multimode communications in the presence of little orno feedback.

Arun Ross received the B.E. (Hons.) degree in com-puter science from the Birla Institute of Technologyand Science, Pilani, India, in 1996, and the M.S. andPh.D. degrees in computer science and engineeringfrom Michigan State University, East Lansing, in1999 and 2003, respectively. From 1996 to 1997,he was with the Design and Development Group,Tata Elxsi (India) Ltd., Bangalore, India. He spentthree summers (2000–2002) with the Imaging andVisualization Group, Siemens Corporate Research,Inc., Princeton, NJ, working on fingerprint recogni-

tion algorithms. He is currently an Associate Professor with the Departmentof Computer Science and Engineering, Michigan State University. He hascoauthored the books Introduction to Biometrics: A Textbook and Handbookof Multibiometrics, and coedited the book Handbook of Biometrics. Hisresearch interests include pattern recognition, classifier fusion, computervision, and biometrics. He is a recipient of the NSF’s CAREER Award andwas designated a Kavli Frontier Fellow by the National Academy of Sciencesin 2006. He was an Associate Editor of the IEEE TRANSACTIONS ON IMAGE

PROCESSING and the IEEE TRANSACTIONS ON INFORMATION FORENSICS

AND SECURITY.

ieee transactions on information forensics and …rossarun/pubs/dantchevaeliarosssoftbio… ·...

Documents