classification bayes

March 2006 Alon Slapak 1 of 1

Bayes Classification

A practical approachExample

Discriminant function

Bayes theorem

Bayes discriminant

function

Bibliography


Discriminant functionDefinition: a discriminant function is an n-dimensional hypersurface which divides the n-dimensional feature space into two separate areas contain separate classes.

A 2-dimwnsinaldiscriminant function

A 1-dimwnsinaldiscriminant function

2-dimensional feature space

1-dimensional feature space Example


Bayes theorem

Bayes discriminant

function

Bibliography


Discriminant functionLet h(x) be a discriminant function. A two-category classifier uses the following rule:

Decide 1 if h(x) > 0 and 2 if h(x) < 0

If h(x) = 0 x is assigned to either class.

h(x)=x-6.25h(x) < 0

h(x) < 0

1 2( ) 1.02 2.5h x x x

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Thomas BayesAt the time of his death, Rev. Thomas Bayes (1702 –1761) left behind two unpublished essays attempting to determine the probabilities of causes from observed effects. Forwarded to the British Royal Society, the essays had little impact and were soon forgotten.

When several years later, the French mathematician Laplace independently rediscovered a very similar concept, the English scientists quickly reclaimed the ownership of what is now known as the “Bayes Theorem”.

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Conditional ProbabilityDefinition: Let A and B be events with P(B) > 0.The conditional probability of A given B, denoted by P(A|B), is defined as:

P(A|B) = P(A B)/P(B)

A B

1020 5

Venn Diagram

Given: N(A) = 30

N(A B) = 10

P(B | A) = N(A B)/N(A) = 10/30 = 1/3

Example:Example


Bayes theorem

Bayes discriminant

function

Bibliography


Bayes’ theoremSince P(A | B) = P(A B)/P(B),

we have: P(A | B)P(B) = P(A B)

Symmetrically we have: P(B | A)P(A) = P(B A) = P(A B)

Therefore: P(A | B)P(B) = P(B | A)P(A)

And:

Bayes' theoremP A B P B

P B AP A

where P(A | B) is the conditional probability, P(A) , P(B) are the prior probabilities, P(B | A) is the posterior probability

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Bayes’ theorem in a pattern recognition notation

Given classes i and a pattern x,

( | ) ( )( | )

( )

( ) ( | ) ( )

j jj

j jj

P PP

P

where

P P P

xx

x

x x

prior probability

• The prior probability reflects knowledge of the relative frequency of instances of a class

likelihood

• The likelihood is a measure of the probability that a measurement value occurs in a class.

evidence

• The evidence is a scaling term

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Bayes classifierThe following phrase classify each pattern x to one of two classes:

2

1

2

1

1 2

1 1 2 2

( | ) ( | )

( | ) ( ) ( | ) ( )

( ) ( )

P P

P P P P

P P

x x

x x

x x

or (since P(x) is common to both sides):

2

1

2

1

1 1 2 2

1 2

2 1

( | ) ( ) ( | ) ( )

( | ) ( )

( | ) ( )

P P P P

P Pl

P P

x x

xx

x

Means, decide 1 if P(1|x) > P(2|x)

Likelihood ratio

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Bayes Discriminant functionSince a ratio of probabilities may yield very small values, it is common to use the log of the likelihood ratio:

and the derived Bayes’ discriminant function is:

11 2

2

( )ln ( | ) ln ( | ) ln

( )

Ph P P

P

x x x

2

1

1 2

2 1

( | ) ( )ln ln ln

( | ) ( )

P Pl

P P

xx

x

Remember: Decide 1 if h(x) > 0 and 2 if h(x) < 0If h(x) = 0 x is assigned to either class.

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian Distributions

A multi dimensional Gaussian distribution is:

21

2/ 2 1/ 2

2 1

th

th

1

2 | |

where:

( ) ( )

is the feature vector

is the mean vector of the class

is the covariance matrix of the class

feature space dimension

i

i

d

i n

i

Ti i i

i

i

P e

d

i

i

n

xx

x x m x m

x

m

120 130 140 150 160 170 180 19030

35

40

45

50

55

60

65

70

75

80

height [cm]

wei

ght

[kg]

Females

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian Distributions

A multi dimensional Gausian distribution is:

2 21 2

1 112 2

/ 2 / 21/ 2 1/ 221 2

2 2 11 11 1 2 22 2 2 2

2

2 2 1 111 2 2

2 2

( )1 1ln ln ln

( )2 | | 2 | |

( )1 1ln 2 ln | | ln 2 ln | | ln

2 2 ( )

| | ( )1 1ln ln

2 2 | | ( )

d d

n n

n n

Pe e

P

Pd d

P

d

h

Pd

P

x x

x x

x

x

x

1 1 1 111 1 1 2 2 2 2

2 2

| | ( )1 1( ) ( ) ( ) ( ) ln ln

2 2 | | ( )T T P

P

x m x m x m x m

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian DistributionsAssume two Gaussian distributed classes withAnd

clear allN1 = 150; N2 = 150;E1 = [50 40; 40 50]; E2 = [50 40; 40 50];M1 = [30,55]'; M2 = [60,40]';%-------------------------------------------------------------------------% Classes drawing%-------------------------------------------------------------------------[P1,A1] = eig(E1); [P2,A2] = eig(E2);y1=randn(2,N1); y2=randn(2,N2);for i=1:N1, x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;end;for i=1:N2, x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;end;figure;plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');axis([0 100 0 100]);xlabel('x1')ylabel('x2')

1 2 1 2( ) ( )P P

2 1 1 1 2 2

1

2T T Th x m m x m m m m

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian Distributions%-------------------------------------------------------------------------% Classifier drawing%-------------------------------------------------------------------------ep=1.2;k=1;for i=0:k:100, for j=0:k:100, x=([i;j]); h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2)); if (abs(h)<ep), hold on; plot(i,j,'*k'); hold off; end; end;end; h(x) > 0

h(x) < 0

2 1 1 1 2 2

10

2T T Th x m m x m m m m

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian DistributionsAssume two Gaussian distributed classes withAnd

clear allN1 = 150; N2 = 150;E1 = [50 40; 40 50]; E2 = [50 -40; -50 50];M1 = [30,55]'; M2 = [60,40];'

-------------------------------------------------------------------------% %Classes drawing

-------------------------------------------------------------------------%]P1,A1 = [eig(E1); [P2,A2] = eig(E2);

y1=randn(2,N1); y2=randn(2,N2);for i=1:N1,

x1(:,i) =P1*sqrt(A1)* y1(:,i)+M1;end;for i=1:N2,

x2(:,i) =P2*sqrt(A2)* y2(:,i)+M2;end;figure;plot(x1(1,:),x1(2,:),'^',x2(1,:),x2(2,:),'or');axis([0 100 0 100]);

xlabel('x1')ylabel('x2')

1 2 1 2( ) ( )P P

1 1 111 1 1 2 2 2 2

2

| |1 1( ) ( ) ( ) ( ) ln

2 2 | |T Th

x x m x m x m x m

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Example - Gaussian Distributions%-------------------------------------------------------------------------% Classifier drawing%-------------------------------------------------------------------------ep=1;k=1;for i=0:k:100, for j=0:k:100, x=([i;j]); h=0.5*(x-M1)'*inv(E1)*(x-M1)-0.5*(x-M2)'*inv(E2)*(x-M2)+0.5*log(det(E1)/det(E2)); if (abs(h)<ep), hold on; plot(i,j,'*k'); hold off; end; end;end; h(x) > 0

h(x) < 0

h(x) < 0

11 11 1 1 2 2 2

2

1 1 1( ) ( ) ( ) ( ) ( ) ln 0

2 2 2T Th

x x m x m x m x m

Example


Bayes theorem

Bayes discriminant

function

Bibliography


Exercise• Synthesize two classes with different a-priory probabilities. Show how the probabilities influence the discriminant function.

• Synthesize three classes and plot the discriminant functions. Example


Bayes theorem

Bayes discriminant

function

Bibliography


Summary

Steps for Building a Bayesian Classifier

• Collect class exemplars

• Estimate class a priori probabilities

• Estimate class means

• Form covariance matrices, find the inverse and determinant for each

• Form the discriminant function for each classExample


Bayes theorem

Bayes discriminant

function

Bibliography


Bibliography1. K. Fukunaga, Introduction to Statistical Pattern

Recognition, 2nd ed., Academic Press, San Diego, 1990.

2. L. I. Kuncheva, J. C. Bezdek amd R. P.W. Duin, “Decision Templates for Multiple Classier Fusion: An Experimental Comparison”, Pattern Recognition, 34, (2), pp. 299-314, 2001.

3. R. O. Duda, P. E. Hart and D. G. Stork, Pattern Classification (2nd ed), John Wiley & Sons, 2000.

Example


Bayes theorem

Bayes discriminant

function

Bibliography

classification bayes

Documents

gaussian distributions assume

gaussian distributed classes

gaussian distributions

gaussian distributions

p1 sqrt

classifier drawing

p2 sqrt

y1 randn