1

Use of Autocorrelation Kernels in Kernel Canonical Correlation

Analysis for Texture Classification

Yo Horikawa

Kagawa University , Japan

2

･ Support vector machine (SVM)

･ Kernel canonical correlation analysis (kCCA)

　 with autocorrelation kernels

→ Invariant texture classification

only using raw pixel data

without explicit feature extraction

Compare the performance of the kernel methods.

Discuss the effects of the order of autocorrelation kernels.

3

Support vector machine (SVM) Sample data: xi (1 ≤ i ≤ n), belonging to Class ci {-1, 1} ∊SVM learns a discriminant function for test data x:

d(x) = sgn(∑i=1n’ αiciK(x, xsi) + b)

αi and b are obtained through the quadratic programming problem.

Kernel function: Inner product of nonlinear maps φ(x): K(x

i, xj) = φ(xi) ･ φ(xj)

Support vectors: xsi (1 ≤ i ≤ n’ (≤ n)): a part of sample data

Feature extraction process is implicitly done in SVM through the kernel function and support vectors.

4

Autocorrelation kernelThe kth-order autocorrelation of data xi(t):

rxi(t1, t2, ∙∙∙ , tk-1) = ∫xi(t)xi(t+t1) ･･･ xi(t+tk-1)dt

The inner product between rxi and rxj is calculated with the k-th power of the cross-correlation function (2nd-order):

rxi･ rxj = ∫{∫xi(t)xj(t+t1)dt}k dt1 　

The calculation of explicit values of the autocorrelations is avoided.

→ High-order autocorrelations are tractable with practical computational cost.

　･ Linear autocorrelation kernel: K(xi, xj) = rxi･ rxj

　･ Gaussian autocorrelation kernel: K(xi, xj) = exp(-μ|rxi - rxj|2)

= exp(-μ(rxi･ rxj + rxi･ rxj - 2rxi･ rxj))

5

Calculation of autocorrelation kernels rxi･ rxj for 2-dimensional image data: x(l, m) (1≤ l ≤ L, 1≤ m ≤ M)

･ Calculate the cross-correlations between xi(l, m) and xj(l, m):

rxi, xj (l1, m1) = ∑l=1L-l1∑m=1

M-m1 xi(l, m)xj(l+l1, m+m1)/(LM)

(1 ≤ l1 ≤ L1, 1 ≤ m1 ≤ M1)

･ Sum up the kth-power of the cross-correlations:

rxi･ rxj = ∑l1=0L1-1∑m1=0

M1-1 {rxi, xj (l1, m1)}k

L

M

M1

L1

xi(l, m)xj(l+l1, m+m1)

∑l,m xi(l+m)xj(l+l1, m+m1)

rxi ･ rxj = ∑l1, m1 { ・ }k

6

0 ΦΘ f Φ2+γxI 0 f = λ ΘΦ 0 g 0 Θ2+γyI g

Kernel canonical correlation analysis (kCCA) Pairs of feature vectors of sample objects: (xi, yi) (1 ≤ i ≤ n)

KCCA finds projections (canonical variates) (u, v) that yield maximum correlation between φ(x) and θ(y).

(u, v) = (wφ ･ φ(x), wθ ･ θ(y)) wφ = ∑i=1

n fiφ(xi), wθ = ∑i=1n giθ(yi)

where fT = (f1, ∙∙∙, fn) and gT = (g1, ∙∙∙, gn) are the eigenvectors of the generalized eigenvalue problem:

Φij = φ(xi) ･ φ(xj) Θij = θ(yi) ･ θ(yj) I: Identity matrix 　　　　　　　　　　　　　　　　　　　　　　　　　　　 of n×n

7

Application of KCCA for classification problems

Use an indicator vector as the second feature vector y. y = (y1, ∙∙∙, yC) corresponding to x: yc = 1 if x belongs to class c yc = 0 otherwise (C: the number of classes) Mapping θ of y is not used. C-1 eigenvectors fk = (fk1, …, fkn) (1 ≤ k ≤ C-1) corresponding to non-zero eigenvalues are obtained. Canonical variates uk (1 ≤ k ≤ C-1) for a test object (x, ?) are calculated by

uk = ∑i=1n fiφ(xi) ･ φ(x) = ∑i=1

n fi K(xi, xj)

Classification methods, e.g., the nearest-neighbor method, can be applied in the canonical variate space (u1, …, uC-1).

8

Classification experiment

D4 D84 D5 D92 (a) Brodatz album

Bark.0000 Sand.0000 Brick.0000 Tile.0007 (b) VisTex database

Shift Scaling Rotation Noise SD

Sample [0, 400] 1.0 0 0.0

Test 1 [0, 400] 1.0 0 0.0

Test 2 [0, 400] 1.0 0 0.1

Test 3 [0, 400] [0.5, 1.0] [0, 2π ] 0.0

Test 4 [0, 400] [0.5, 1.0] [0, 2π ] 0.1

Fig. 1. Texture images.

Table 1. Sample and test sets.

4-class classification problems with SVM and kCCA

Original images: 512×512 pixels (256 gray scale) in the VisTex database and the Brodatz album

Sample and test images: 50×50 pixels, chosen in the original images with random shift and scaling, rotation, Gaussian noise (100 images each)

9

Kernel functions K(xi, xj) ( ) Linear kernel: ⅰ xi ･ xj ( ) Gaussian kernel: exp(-μ||ⅱ xi – xj||2) ( ) ⅲ Linear autocorrelation kernel: rxi ･ rxj 　( ) ⅳ Gaussian autocorrelation kernel: = exp(-μ|rxi - rxj|2) = exp(-μ(rxi ･ rxj + rxi ･ rxj - 2rxi ･ rx

j))

Range of correlation lags: L1 = M1 = 10 (in 50×50 pixel images)

The simple nearest-neighbor classifier is used for classification with canonical variates (u1, …, uC-1) in kCCA.

Parameter values are empirically chosen. (Soft margin: C = 100, Gaussian:μ, Regularization:γx, γy)

10

0.0

0.2

0.4

0.6

0.8

1.0

linea

r (ⅰ

)

Gau

ss (ⅱ

)

k=2

(ⅲ)

k=3

(ⅲ)

k=4

(ⅲ)

k=5

(ⅲ)

k=6

(ⅲ)

k=2

(ⅳ)

k=3

(ⅳ)

k=4

(ⅳ)

k=5

(ⅳ)

k=6

(ⅳ)

2nd

cor.

linea

r

3rd

cor.

linea

r

2nd

cor.

Gau

ss

3rd

cor.

Gau

ss

CC

RTest 1Test 2Test 3Test 4

Fig. 2. Correct classification rates (CCR (%)) in SVM.

(a) Brodatz Album

0.0

0.2

0.4

0.6

0.8

1.0

linea

r (ⅰ

)

Gau

ss (ⅱ

)

k=2

(ⅲ)

k=3

(ⅲ)

k=4

(ⅲ)

k=5

(ⅲ)

k=6

(ⅲ)

k=2

(ⅳ)

k=3

(ⅳ)

k=4

(ⅳ)

k=5

(ⅳ)

k=6

(ⅳ)

2nd

cor.

linea

r

3rd

cor.

linea

r

2nd

cor.

Gau

ss

3rd

cor.

Gau

ss

CC

R

Test 1Test 2Test 3Test 4

(b) VisTex database

11

Fig. 3. Correct classification rates (CCR (%)) in kCCA.

0.0

0.2

0.4

0.6

0.8

1.0

linea

r (ⅰ

)

Gau

ss (

ⅱ)

k=2

(ⅲ)

k=3

(ⅲ)

k=4

(ⅲ)

k=5

(ⅲ)

k=6

(ⅲ)

k=10

(ⅲ

)

CC

R

(a) Brodatz Album

0.0

0.2

0.4

0.6

0.8

1.0

linea

r (ⅰ

)

Gau

ss (

ⅱ)

k=2

(ⅲ)

k=3

(ⅲ)

k=4

(ⅲ)

k=5

(ⅲ)

k=6

(ⅲ)

k=10

(ⅲ

)

CC

R

Test 1Test 2Test 3Test 4

(b) VisTex database

12

Comparison of the performances Classification experiments for the Brodatz album texture images

(D4 and D84), (D5 and D92) with various filtering methods and vector quantization learning (Randen and Husøy, 1999)

→ CCRs are about 90%.

SVM and kCCA with autocorrelation kernels show comparable performance.

Test1 0.995 k=2 (ⅳ ) 0.947 k=2 (ⅲ )0.960 k=4 (ⅳ )0.925 k=2 (ⅲ )Test2 0.953 k=2 (ⅳ ) 0.925 k=2 (ⅲ )0.973 k=4 (ⅳ )0.920 k=2 (ⅲ )Test3 0.748 k=4 (ⅳ ) 0.720 k=4 (ⅲ )0.805 k=4 (ⅲ )0.695 k=2 (ⅲ )Test4 0.735 k=2 (ⅳ ) 0.697 k=2 (ⅲ )0.638 k=2 (ⅳ )0.592 k=2 (ⅲ )Mean 0.858 0.822 0.844 0.783

SVM kCCA(a) Brodatz (b) VisTex

SVM kCCA

0.0

0.2

0.4

0.6

0.8

1.0

linea

r(ⅰ

)

Gau

ss(ⅱ

)

k=2

(ⅲ)

k=3

(ⅲ)

k=4

(ⅲ)

k=5

(ⅲ)

k=6

(ⅲ)

k=2

(ⅳ)

k=3

(ⅳ)

k=4

(ⅳ)

k=5

(ⅳ)

k=6

(ⅳ)

2nd

cor.

linea

r

3rd

cor.

linea

r

2nd

cor.

Gau

ss

3rd

cor.

Gau

ss

CC

R

Test 1Test 2Test 3Test 4

Table 2. Highest correct classification rates in SVM and kCCA.

13

Effects of the order of autocorrelation kernels

Experiments of face detection (Popovici and Thiran, 2001)→ CCR increases as the autocorrelation order increases.

The result of this texture classification experiment → the lower-order (k = 2, 3, 4) kernels show better performance.

The best order may depend on the objects.

14

The order k of autocorrelation kernels increases.

→ The generalization ability and robustness are lost.

rxi･ rxj = ∑t1 (rxi, xj (t1))k → δi, j (k → ∞)

For test data x (≠xi), rxi･ rx = 0

In kCCA, Φ= I, Θ: block matrix, eigenvectors:

f = (p1, …, p1, p2, …, p2, … , pC, …, pC) (fi = pc, if xi class c) ∊

For sample data, canonical variates lie on a line through the origin corresponding to its class:

uxi = (rxi･ rxi)pc (pc = (pc,1, ∙∙∙, pc,C-1)) , if xi class c∊For test data: ux ≈ 0

Modification: Use of lp-norm like functions

(rxi･ rxj)1/k = |∑t1 (rxi, xj (t1))k|1/k

15

Fig. 4. Scatter diagram of canonical variates (u1, u2) and (u3, u1) of Test 1 data of texture images in the Brodatz album in kCCA. Plotted are square (■) for D4, cross (×) for D84, circle (●) for D5 and triangle (Δ) for D92.

(a) linear kernel ( ) ⅰ (b) Gaussian kernel ( ) ⅱ

(c) 2nd-order correlation kernel ( ) ⅲ (d) 3rd-order correlation kernel ( ) ⅲ

(e) 4th-order correlation kernel ( ) ⅲ

(f) 10th-order correlation kernel ( ) ⅲ

-0.0

90.

000.

09

- 0.08 0.00 0.08u1

u2

-0.0

70.

000.

07

- 0.08 0.00 0.08u3

u1

-0.0

120.

000

0.01

2

- 0.03 0.00 0.03u1

u2

-0.0

30.

000.

03

- 0.008 0.000 0.008u3

u1

-0.1

0.0

0.1

- 0.13 0.00 0.13u1

u2

-0.1

20.

000.

12

- 0.07 0.00 0.07u3

u1

-0.0

70.

000.

07

- 0.06 0.00 0.06u1

u2

-0.0

60.

000.

06

- 0.03 0.00 0.03u3

u1

-0.0

60.

000.

06

- 0.11 0.00 0.11u1

u2

-0.1

10.

000.

11

- 0.02 0.00 0.02u3

u1

-0.0

30.

000.

03

- 0.02 0.00 0.02u1

u2

-0.0

20.

000.

02

- 0.007 0.000 0.007u3

u1

Most of test data u ≈ 0

16

Summary SVM and kCCA with autocorrelation kernels are applie

d to texture classification.

The performance compete with conventional feature extraction methods and learning.

The Gaussian autocorrelation kernel of the order 2 or 4 gives highest correct classification.

The generalization ability of the autocorrelation kernels decreases as the order of the correlation increases.