support vector machines (part 1)

Face Recognition & Biometric Systsems

Support Vector Machines (part 1)


Plan of the lecture

Problem of classificationSVM for solving linear problems training classification

Application of convolution kernels


Bibliography

Corrina Cortez, Vladimir VapnikSupport-Vector Networks


Classification problem

Aim: classification of an element to one of defined classesTwo stages: training classification of samples

Available solutions: Artificial Neural Networks Support Vector Machines other classifiers



Training set - requirements: classified representative

Training process: aims at finding general rules a risk of overfitting to the training

set (especially when it is not representative)



Classification of samples: must be preceded by the training stage applies rules derived from the training

Number of classes: SVM solves two-class problems it is possible to solve multi-class

problems basing on two-class problems



Linearly separable



Non-linearly separable



Training with error (soft margin)



Margin maximisation


Linear separability

Data set: (y1,x1),...,(yl,xl), yi{-1,1}

Vector w, scalar value b:w • xi + b 1 for yi = 1

w • xi + b -1 for yi = -1

henceyi (w • xi + b) 1

The condition must be fulfilled for the whole data set


SVM – training

SVM solves linear separable two-class problems other cases transformed to the

basic problem

Optimal hyperplane margin between samples of two

classes margin maximisation


SVM – training

Optimal hyperplane:w0 • x + b0 = 0

2D example – hyperplane is a line

Margin width (without b):

||

max||

min}1:{}1:{ w

wx

w

wxw

yxyx,bρ


SVM – training

Optimal width:

Maximisation of , minimisation of w0 • w0

Limitation: yi (w • xi + b) 1

00000

2

||

2

wwww

),bρ(


SVM – training

Margin:Optimal hyperplane:

yi – class identifier i – Lagrange multipliers

A problem: how to find i?

1)( by ii xw

l

iiiiy

1

00 xw


SVM – training

Function maximisation:

1 – unitary vector (l – dimensional)D – l x l matrix:

DΛΛ1ΛΛ TTW2

1)(

),...,( 1 lT Λ

jijiij yyD xx


SVM – training

Optimisation limits:

Optimisation based on the gradient method

0Λ0YΛT

),...,( 1 lT yyY


SVM – training

Lagrange multipliers : non-zero values for support vectors equal zero for other vectors

(majority)

Training set after the training: support vectors (a small subset of

the training set) coefficients for every vector


SVM – classification

Calculate y for a vector which is to be classified:

xr, xs – support vectors from both classes

Classification decision

byfl

iiii

1)( xxx

l

isiriii yb

1)(

2

1xxxx


SVM – limitations

SVM conditions: solves two-class problem linear separability of data

A XOR problem:


SVM – limitations

Possibilities of enhancement: SVM for non-linear data – too

complicated calculations transformation of the data, so that

they are linearly separable

Mapping into higher dimension example of XOR in 2D mapped into

3D


Convolution kernelsFunction:Mapping into higher dimension: x (x)Calculations use scalar product of vectors, not the vectors themselvesKernels of convolution may be used instead of scalar products

No need to find function

Nn RR :

)()(),( vuvu K


Convolution kernels

),( jijiij KyyD xx

Training with convolution kernels


Convolution kernels

bKyfl

iiii

1),()( xxx

l

isiriii KKyb

1)],(),([

2

1xxxx

xr, xs – support vectors from both classes

Classification with convolution kernels


Convolution kernels

Linear

Polynomial

RBF (radial basis functions)

2

2||

),( vu

vu

eK

vuvu ),(K

dK )1(),( vuvu


Summary

ClassifiersBasic problem: two-class linear separable data set solved by the SVM

Enhancement convolution kernels – SVM for non-

linear separable data


Thank you for your attention!

Next week

Support Vector Machines – continued... multi-class cases soft margin training applications to face recognition

support vector machines (part 1)

Documents