machine learning & computer vision

RECOGNISING FACESMACHINE LEARNING & COMPUTER VISION: 101

SAKSHAM GAUTAM

08.06.2016 Saksham Gautam

LET’S START WITH A QUICK SHOW OF HANDS

HOW MANY OF YOU…

▸ have heard about machine learning?

▸ have used machine learning in your projects?

▸ have implemented any ML algorithm from scratch?

▸ have done Andrew Ng’s (or other) courses on ML?

▸ understand that deep learning uses neural network?

▸ still remember what the kernel trick is?


SHOW OF HANDS ON YOUR FAMILIARITY WITH COMPUTER VISION

HOW ABOUT…

▸ know how an image can be represented as a matrix?

▸ have used openCV or MATLAB?

▸ understand how convolution can be used to detect edges?

▸ know the role of scale space in computer vision?

▸ remember how eigenvectors can be used for face recognition?


WHAT DO WE WANT TO ACHIEVE?

FACE DETECTION & RECOGNITION

http://docs.opencv.org/master/d7/d8b/tutorial_py_face_detection.html#gsc.tab=0

FACE

MONA LISA

NOT A FACE!


TO RECOGNISE AND LABEL OBJECTS IN AN IMAGE

OBJECT RECOGNITION


BASIC STEPS FOR FACE RECOGNITION

BUT HOW EXACTLY?

1. Capture image

2. Filter out noise

3. Find face in the image

4. Create a similarity metric and a model (Training)

5. Match any given face to one from the database

6. Return the closest match with the probability


FACE RECOGNITION CAN BE BROKEN DOWN INTO SIMPLE STEPS

BUILDING BLOCKS

RAW IMAGE PROCESSED IMAGE FEATURES

MACHINE LEARNING ALGORITHM

Training

Validation

MODEL

RAW IMAGE

DECISION

How can I capture image?

Remove any noise?

What’s the information in the image?

Can we match patterns?


HUMAN SEES BEAUTY BUT COMPUTER SEES NUMBERS

WHAT IS THIS?

=


PIXEL VALUES CAN BE THOUGHT OF AS ‘ELEVATION’ IN IMAGE PLANE

MONALISA IN 3D?


FILTER - I, MEAN FILTER CAN REMOVE RANDOM NOISE

MEAN FILTER


FILTERS - II, MEDIAN FILTER CAN REMOVE SALT & PEPPER NOISE

MEDIAN FILTER


FILTERS-III, ADVANCED FILTERS CAN REMOVE DIFFERENT TYPES OF NOISE

ADVANCED FILTERS

▸ Bilateral filters

▸ Adaptive bilateral filters

▸ …


38 53 38 70 84

5 56 69 82 98

34 87 85 5 40

25 99 43 69 76

11 61 86 94 59

-1 0 1

-1 0 1

-1 0 1

CONVOLUTION CAN BE APPLIED VERY QUICKLY ON AN IMAGE

CONVOLUTION

*

38 0 -38

5 0 -69

34 0 -85

-115

-115 56 16

-69 112 12

-93 3 6


CONVOLUTION CAN BE USED FOR COMPUTING IMAGE GRADIENT

IMAGE GRADIENT

0 0 0

100 100 100

0 50 100

100 50 0

-1 0 1*0

0

100

-100

=


EDGES AND CORNERS ARE FEATURES IN AN IMAGE

SOBEL FILTER FOR DETECTING EDGES

HARRIS CORNER DETECTOR

-1 0 1

-2 0 2

-1 0 1

Gx =-1 -2 -1

0 0 0

1 2 1

Gy =

$python sobel-filter.py

$python harris-corner.py


MORE ROBUST FEATURES CAN BE USED FOR OBJECT RECOGNITION

SIFT, SURF, HOG

▸ More advanced features can be used for scale invariance

▸ Some are robust even under varying lighting conditions

▸ These serve as the starting point for the ML part


CASCADES OF FILTERS ON AN IMAGE CAN BE USED FOR DETECTING FACES

DETECTING FACES

http://siret.ms.mff.cuni.cz/facereco/method/

$python viola-jones.py


FEATURES FRO THE FACE CAN BE FED TO AN ML ALGORITHM

BUILDING BLOCKS



Training

Validation

MODEL

RAW IMAGE

DECISION

PERFORMANCE (P) OF A METHOD FOR A TASK (T) INCREASES WITH EXPERIENCE (E)

Tom Mitchell

BTW, WHO LEARNS? THE MACHINE, REALLY?


PROBABILITY AND STATISTICS CAN HELP ANSWER MANY QUESTIONS

T-SHIRT SIZE FOR THE SUMMITMAYBE I SHOULD HAVE PICKED ‘M’ INSTEAD OF ’S'


EXACT SOLUTION IS NOT ALWAYS POSSIBLE

T-SHIRT SIZE ~ MY BODY’S MEASUREMENTS


MAXIMUM LIKELIHOOD ESTIMATE HELPS ON THE FACE OF UNCERTAINTY

CLASSIFICATION PROBLEM?

Length #2

Length #1

S

MLength #1

Length #2

S


MEMBERSHIP TO A CLUSTER CAN BE USED FOR CLASSIFICATION

CLUSTERING

Length #2

Length #1

S

M


MARGINS CAN BE USED FOR SEPARATING CLASSES

LARGE MARGIN CLASSIFIER

Length #2

Length #1


LOG(DISTANCE FROM THE HYPERPLANE) = PROBABILITY

LOGISTIC REGRESSION

Length #2

Length #1


LARGE MARGIN CLASSIFIER CAN BE USED FOR DETECTING FACES

FEEDING VALUES FROM VIOLA JONES FILTER

X1

X2FACE

NOT A FACE


FEATURES SHOULD MAXIMISE VARIANCE (SCATTER)

EIGENVECTORS

V1

V2

X1

X2


MATRIX DECOMPOSITION ON FACES CAN BE USED TO FIND EIGENFACES

FACIAL FEATURES AS EIGENVECTORS

V1

V2

Projection


ACCURACY ITSELF IS NOT ENOUGH

CROSS VALIDATION AND F SCORE

DUMMY CANCER DETECTOR

Data

No

ACCURACY = 98%


IMAGE CAN BE DIRECTLY FED INTO NEURAL NETWORK

NEURAL NETWORK


GO WITH THE HYPE, BUT WITH CARE

DEEP LEARNING ~ MASSIVE NEURAL NETWORK

▸ Learning algorithm is the same, i.e. back propagation

▸ Has the same problem with overfitting

▸ Can be used for feature extraction and selection

▸ Mathematical foundations for neural network still not “perfect”

▸ Pointer: https://www.tensorflow.org from Google

https://www.tensorflow.org


MACHINE LEARNING PIPELINE

SUMMARY



Training

Validation

MODEL

RAW IMAGE

DECISION

REFERENCES

• OpenCV Documentation. http://docs.opencv.org/3.1.0/#gsc.tab=0

• Andrew Ng. Machine Learning Courser on Coursera. http://www.coursera.org/learn/machine-learning

• Christopher Bishop. Machines that Learn. https://www.youtube.com/watch?v=icaA7gVxqSs

• Video Lecture on Face Detection and Tracking. https://www.youtube.com/watch?v=WfdYYNamHZ8

• Adam Harvey explains Viola-Jones Face Detection. http://www.makematics.com/research/viola-jones/

• Christopher, M. Bishop. "Pattern recognition and machine learning." Company New York 16.4 (2006): 049901.

• Bradski, Gary, and Adrian Kaehler. Learning OpenCV: Computer vision with the OpenCV library. " O'Reilly Media, Inc .", 2008

• Solem, Jan Erik. Programming Computer Vision with Python: Tools and algorithms for analyzing images. " O'Reilly Me dia, Inc.", 2012.

• Hartley, Richard, and Andrew Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.

http://docs.opencv.org/3.1.0/#gsc.tab=0

http://www.coursera.org/learn/machine-learning

https://www.youtube.com/watch?v=icaA7gVxqSs

https://www.youtube.com/watch?v=WfdYYNamHZ8

http://www.makematics.com/research/viola-jones/

BACKUP SLIDES


IMAGE GRADIENT IS THE BASIS OF FEATURE EXTRACTION

GRADIENT IMAGE

-1 0 1gx =-1

0

1

gy =* A * A


EVERY SIGNAL CAN BE DECOMPOSED TO SINES AND COSINES

FOURIER TRANSFORM

▸ Frequency can be thought of as information in the image

▸ Fourier Transform can be used to decompose a signal into these components

▸ Signal can be multiplied with filter in frequency domain

▸ Multiplication in frequency domain is convolution in time domain

machine learning & computer vision

Data & Analytics