sketch to photo matching

MET’S School of EngineeringDepartment of Computer Science and Engineering

Academic Year 2015-2016

Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 1

SKETCH TO PHOTO MATCHING SYSTEM

Submitted by

Chippy Thomas (GWAOCCS004)

S4-M.Tech CSE (2014-16 Batch)

19-03-2016

May 2, 2023 2


Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE

Introduction

System that match sketch with mugshot

For the identification of suspectsUse when eyewitness are only evidence

Sketch/composites-hand-drawn-software generated-surveillance composites

•mugshot19-03-2016 2

May 2, 2023 3



Introduction – contd… So matching sketch and photo

Direct matching is not possibleIe..We don’t match them pixel by pixel

Match by finding high-level features from both the sketch and the photo

Features such as the structural distribution and the shape of the eyes, nose and chin etc.

19-03-2016 3

May 2, 2023 4



Introduction – contd…

19-03-2016 3

May 2, 2023 5



Hand-drawn composite

19-03-2016 4

•Drawn by forensic artist •Based on description provided by one or more witness•Artist need proper training

May 2, 2023 6



Software-generated composites

19-03-2016 5

•Drawn by operator using software•Most used software's are identikit and FACES•Operator need training

May 2, 2023 7



Surveillance composites

19-03-2016 6

•Drawn by forensic artist•Based on poor quality surveillance images

May 2, 2023 8



Example of Verbal Description

The man was about 5ft. 6in. in height, and 34 or 35 years of age, with dark complexion and dark mustache turned up at the ends . . . He had a heavy mustache curled up, and dark eyes and bushy eyebrows. He had no side whiskers, and his chin was clean shaven. He looked like a foreigner."

19-03-2016 7

Objective


Develop a system that can automatically match sketch with mugshots

System that manage every types of sketch such as -forensic sketches -composite sketches -surveillance sketches

Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 19-03-2016 8

• Law enforcement agencies disseminate a sketch to media outlets

• Hope that someone will recognise the suspect

• And provide information leading to an arrest.

19-02-2016 2



Existing system

19-03-2016 9

May 2, 2023 11



Existing system – contd…

Problem description


•Slow and tedious process

•Fails to leverage resources available to law enforcement agencies (mugshot databases)

•So there is a need for a method that can automatically match facial composites to large police mugshot databases.


• Automatically match facial composites against mugshots

• For this 2 algorithms are used• Holistic representation (developed by Klare and Jain)• Component representation (developed by Han et al.)

19-02-2016 2



Proposed system

19-03-2016 12

May 2, 2023 14



Proposed system – contd…


Working •Sketches that is given as input to the system

•From sketch features are extracted based on 2 algorithms

•There will be a database to hold the features of large mugshot

•By applying 2 algorithms it matches with the mugshot

•2 algorithms produce matching scores

•Combining matching scores of 2 algorithm




Figure

19-03-2016 15

Holistic algorithm



•First step is detection of eye

•Alignment and CSDN

•Feature extraction -SIFT -MLBP

•LDA- learn optimal subspaces & projecting into subspaces•PCA

19-03-2016

18

SKETCH EYE DETECTION

ALIGNMENT AND CSDN

PATCHWISE SEGMENTATION

DENSE GRID OF KEYPOINTS

MLBP

SIFT

LDA Patch-wise concatenation & PCA

Sketch Feature Vector

Mugshot Feature Vector

Repeated for all patches



L2 norm

19-03-2016 17

Holistic Algorithm – contd…

Holistic algorithm – contd…



•The same steps are performed for sketch and photo

•Sketch feature vector and mugshot feature vector

•These 2 features are comparing for similarity using L2 norm

•Finding score of matching

19-03-2016

Component-based algorithm



•In holistic algorithm, features are extracted from each patch / grid from images•But component-based algorithm, features are extracted from each component•Per component, features are extracting

19-03-2016

May 2, 2023 21



Component-based algorithm – contd…

Modules



Eye Detection

•Accurate and reliable•Eyes are protected and remain reliably unchanged for decades•Eyes are relatively stable feature

19-03-2016

Modules – contd…


Number: 04 Guide Name: Mr. JOBI V DAS, AP/CSE 2219-02-2016

Alignment and CSDN

•composite is normalized to a fixed height and width•transformed such that right and left eyes are at the same position for every composite.

•The center-surround divisive normalization (CSDN)•filter is then applied to the composite to compensate change in modality between composite and mugshot.

19-03-2016




•SIFT and MLBP Feature Extraction

•SIFT features are extracted from dense grid of keypoints

•Multi-scale local binary pattern (MLBP) features are extracted from a uniform patches across the face,

19-03-2016




LDA

•For both SIFT and MLBP features, optimal subspaces are learned for each patch using linear discriminant analysis (LDA).•projecting the patch-wise features into their respective subspaces

PCAPCA is applied to the feature vector to reduce template size

19-03-2016




L2 norm•It is used as classifier•For finding similarity between mugshot and sketch features

Sum of Score Fusion•Scores from 2 algorithms are fused using sum of score fusion

19-03-2016

Innovative idea



•Modifying existing systems with Enhanced Holistic method and Enhanced Composite method.

•Replacing feature extraction methods and classifier

19-03-2016

Innovative idea



Innovative idea – contd…



•In the proposed scheme it is done by replacing the existing systems algorithms such as SIFT and MLBP with SURF and harlick algorithms.

•Also adding Neural network classifier instead of L2 norm for better accuracy.

19-03-2016

screenshots – contd…



•Input sketch




•Eye detection from sketch




•Aligned sketch




•Features extracted from sketch




•Eye detection from photo




•Feature extraction from photo

Conclusion


3619-02-2016

•Facial composites are used by law enforcement agencies for the identification of suspects involved in criminal activities

•Sketch to photo system effectively make the use of facial composites

•Automatic

•Can be used with every type of composites




•For that 2 algorithms are using here

•Holistic algorithm and component based algorithm

•Match scores from 2 algorithms are combining

•Also filtering based on demographic information

•Along with the matched image some details such as name, age can be displayed.

19-3-2016

Conclusion – contd…

References SKETCH TO PHOTO MATCHING SYSTEM


•Bonnen K. , B. F. Klare, and A. K. Jain, “Component-based representation in automated face recognition,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 239–253, Jan. 2012.

•Best-Rowden L. , H. Han, C. Otto, B. Klare, and A. Jain, “Unconstrained face recognition: Identifying a person of interest from a media collection,” IEEE Trans. Inf. Forensics Security,Dec. 2014

•Han H. , B. F. Klare, K. Bonnen, and A. K. Jain, “Matching composite sketches to face photos: A component-based approach,” IEEE Trans. Inf. Forensics Security, vol. 8, no. 1, pp. 191–204, Jan. 2013

19-03-2016

References – Contd… SKETCH TO PHOTO MATCHING SYSTEM


•Scott J. Klum, Hu Han, Brendan F. Klare Anil K. Jain, ”The FaceSketchID System: Matching Facial Composites to Mugshots” IEEE Trans. Inf. Forensics Security,Dec. 2014 ,Vol. 9, no. 12, Dec 2014

19-03-2016

Quires



May 2, 2023

?



May 2, 2023

Thank you