multiway generalized canonical correlation analysis (mgcca)€¦ · multiway generalized canonical...

Multiway Generalized Canonical CorrelationAnalysis (MGCCA)

A. Gloaguen 1,2, C. Philippe2, V.Frouin2, L. Le Brusquet1 & A. Tenenhaus 1,3

1 Laboratoire des Signaux et Systemes (L2S, UMR CNRS 8506) CNRS - CentraleSupelec- Universite Paris-Sud - Gif-sur-Yvette

2 NeuroSpin/UNATI - CEA, Universite Paris-Saclay, Universite Paris-Saclay -France3 Brain and Spine Institute, Bioinformatics and Biostatistics platform - Paris

janvier

2/23

Motivation : Raman DataDesign

1 13 volunteers.

2 2 arms : moisturizer/placebo.

3 Raman spectroscopy on each arm.

Goal

Evaluate the efficiency of a moisturizer and identify differences in spectrum betweentreated/placebo.

Data

5 three-way tensors acquired at 0, 2, 4, 8 and 12 weeks.

2/23

A. Gloaguen 1,2, C. Philippe2, V.Frouin2, L. Le Brusquet1 & A. Tenenhaus 1,3 L2S / NeuroSpin Multiway Generalized Canonical Correlation Analysis (MGCCA)

3/23

Motivation : Raman Data

3/23


4/23

Introduction MGCCA Results Conclusion

Outline

1 From PLS to RGCCA

2 Multiway GCCA

3 Results on Raman data

4 Conclusion andPerspectives

4/23


5/23


Notations

For matrices

1 X1, . . . ,XL are L data matrices.

2 Xl ∈ RI×Jl : a block.

3 wl ∈ RJl : a block-weight vector.

4 ξl = Xlwl ∈ RI : a block component.

5/23


6/23


PLS/CCA/R-CCA

Canonical Correlation Analysis(CCA)

maxw1,w2

Var(X1w2)=1Var(X2w2)=1

Cor2 (X1w1, X2w2)

Partial Least Squares (PLS)

maxw1,w2

‖w1‖=‖w2‖=1

Cov2 (X1w1, X2w2)

Regularized-CCA (R-CCA)

maxw1,w2

Cov2 (X1w1, X2w2)

s.t. (1− τl)Var (Xlwl) + τl‖wl‖2 = 1, l = 1, 2.

6/23


6/23


PLS/CCA/R-CCACanonical Correlation Analysis(CCA)

maxw1,w2

Var(X1w2)=1Var(X2w2)=1

Cor2 (X1w1, X2w2)

Partial Least Squares (PLS)

maxw1,w2

‖w1‖=‖w2‖=1

Cov2 (X1w1, X2w2)

Regularized-CCA (R-CCA)

maxw1,w2

Cov2 (X1w1, X2w2)

s.t. (1− τl)Var (Xlwl) + τl‖wl‖2 = 1, l = 1, 2.

6/23


7/23


PLS/CCA/R-CCA with a figure

1 X ∼ N(

(0, 0) ,

(1 0.5

0.5 1

)).

2 y = 0 if x1 < 0, 1 otherwise.

7/23


7/23


PLS/CCA/R-CCA with a figure

1 X ∼ N(

(0, 0) ,

(1 0.5

0.5 1

)).

2 y = 0 if x1 < 0, 1 otherwise.

RCCA

7/23


8/23


Regularized Generalized Canonical CorrelationAnalysis (RGCCA) : Scheme

8/23


9/23


RGCCA : Optimization problem

Optimization problem :

maxw1,...,wL

L∑k,l=1

ckl g (Cov (Xkwk ,Xlwl))

s.t. (1− τl)Var (Xlwl) + τl‖wl‖2 = 1, l = 1, . . . , L.

with g a continuous, convex and derivable function.

9/23


10/23


Content

1 From PLS to RGCCA

2 Multiway GCCA



10/23


11/23


Multiway Generalized Canonical Correlation Analysis(MGCCA)

Χ ..1l

I Χ ..2l

Jl

Χ .. Kl

l

JlJl

Jl

I

K l

I

K 1

Χ 1

Χ l

Χ l

J1 J L

I

KL

ΧL

ξ1 ξL

ξ l

Lateral slice

Frontal slice

Unfolding

c1 L

cl Lc1 l

Χ . j .1

Χ ..kl

11/23


12/23


Kronecker product

a⊗ b =

[a1

a2

]⊗

b1

b2

b3

b4

=

a1

b1

b2

b3

b4

a2

b1

b2

b3

b4

=

a1b1

a1b2

a1b3

a1b4

a2b1

a2b2

a2b3

a2b4

From a vector of length 2 and a vector of length 4, a vector of

length 8 was created → estimation of 8 vs. 6 coefficients.

12/23


13/23


Component via Kronecker product

ξl = Xl(wKl ⊗wJ

l ) = Xl(wKl ⊗ IJl )wJ

l

=

( Kl∑k=1

wKlk Xl

..k

)wJ

l

ξl = Xl(wKl ⊗wJ

l ) = Xl(IKl⊗wJ

l )wKl

=

( Jl∑j=1

wJlj X

l.j .

)wK

l

13/23


14/23


MGCCA : Optimization problem

maxw1,...,wL

L∑k,l=1

ckl g (Cov (Xkwk ,Xlwl))

s.t. (1− τl)Var (Xlwl) + τl‖wl‖2 = 1

and wl = wKl ⊗wJ

l , l = 1, . . . , L

Optimized with a Block Coordinate Ascent approach with at eachupdate the following problem to solve(

vK?

l , vJ?

l

)= argmax

vKl ,vJl

‖vKl ⊗vJl ‖=1

vKl>

QlvJl → SVD of Ql of dimension Jl×Kl .

14/23


15/23


Discussion

Advantages

1 Simple algorithm that monotonically converges toward astationary point.

2 The 3-way structure is taken into account.

3 Less weights to estimate : from Jl × Kl to Jl + Kl for eachblock.

15/23


16/23


Content

1 From PLS to RGCCA

2 Multiway GCCA



16/23


17/23


Motivation : Raman DataDesign

1 13 volunteers.

2 2 arms : moisturizer/placebo.

3 Raman spectroscopy on each arm.

Goal

Evaluate the efficiency of a moisturizer and identify differences in spectrum betweentreated/placebo.

Data

5 three-way tensors acquired at 0, 2, 4, 8 and 12 weeks.

17/23


18/23


Raman MGCCA scheme

To adjust τl , l = 1, . . . , 5

1 10-folds MCCV.

2 LDA on concatenation ofblock components.

3 Itrain = 8/Itest = 5.

Results : In accuracy, 0.72(mean), 0.8 (median), 0.17(std).

18/23


19/23


Raman MGCCA weights

wJl , l = 1, . . . , 5

wKl , l = 1, . . . , 5

19/23


20/23


Content

1 From PLS to RGCCA

2 Multiway GCCA



20/23


21/23


Conclusion

1 Simple algorithm that monotonically converges toward astationary point.

2 The 3-way structure is taken into account.

3 Less weights to estimate : from Jl × Kl to Jl + Kl .

4 Gain in interpretability thanks to vector weights specific toeach dimension.

21/23


22/23


Perspectives

Future work :

Keep 3-way structure for higher components computation.

Apply MGCCA on more data sets : ADNI.

Develop Sparse MGCCA :

maxw1,...,wL

L∑k,l=1

ckl g(

w>k X>k Xlwl

)s.t. w>l Mlwl = 1 and wl = wK

l ⊗wJl , l = 1, . . . , L

‖wKl ‖1 ≤ cKl and/or ‖wJ

l ‖1 ≤ cJl

22/23


23/23


Thank you for your attention !

Q U E S T I O N S ?

23/23


1/9

NotationsFor 3-way tensors

1 X1, . . . ,XL are L 3-way tensors.2 Xl ∈ RI×Jl×Kl : a block.3 Xl

..k ∈ RI×Jl : the kth frontal slice of Xl .4 Xl

.j . ∈ RI×Kl : the j th lateral slice of Xl .5 Xl = [Xl

..1, . . . ,Xl..kl, . . . ,Xl

..Kl] ∈ RI×JlKl : the matricized

version of Xl .

1/9


2/9

MGCCA : Change of variable

Idea : get rid of Ml . For that uses

Pl = I−1/2XlM−1/2l

vl = M1/2l wl

Ml = MKl ⊗MJ

l

vl = M1/2l wl = (MK

l )1/2wKl ⊗ (MJ

l )1/2wJl = vKl ⊗ vJl

Thus, the new optimization problem can be written as

maxv1,...,vL

f (v1, . . . vL) =L∑

k,l=1

clk g(

v>k P>k Plvl)

s.t. v>l vl = 1 and vl = vKl ⊗ vJl , l = 1, . . . , L (1)

2/9


3/9

MGCCA : Idea for the Algorithm (1)We want to find an update vl such thatf (v) ≤ f (v1, ..., vl−1, vl , vl+1, ..., vL). f is a continuouslydifferentiable multi-convex function, thus noting v = (v1, . . . , vL)

f (v1, ..., vl−1, vl , vl+1, . . . , vL) ≥ f (v)+∇l f (v)>(vl−vl) = `l(vl , v)

The solution that maximizes this minorizing function over vl isobtained by considering the following optimization problem :

v?l = rl(v) = argmaxvl=vKl ⊗vJl‖vKl ⊗vJl ‖=1

`l(vl , v)

In the end

f (v) = `l(vl , v) ≤ `l(v?l , v) ≤ f (v1, ..., vl−1, v?l , vl+1, ..., vL).

3/9


4/9

MGCCA : Idea for the Algorithm (2)It is possible to show that

∇l f (v1, ..., vL) = P>l

(L∑

k=1

clkg′(vl>P>l Pkvk)Pkvk

)= P>l zl .

Thus, introducing Ql = [(Pl..1)>zl , . . . , (Pl

..Kl)>zl ]

>, then(vK

?

l , vJ?

l

)= argmax

vKl ,vJl

‖vKl ⊗vJl ‖=1

zl>Pl(vKl ⊗ vJl )

= argmaxvKl ,v

Jl

‖vKl ⊗vJl ‖=1

vKl>

QlvJl

At the heart of MGCCA’s algorithm lies a Singular ValueDecomposition (SVD).

4/9


5/9

MGCCA : Algorithm

Algorithm 1 RGCCA algorithm for three-way data analysis1: Data : Xl , τl , g , ε2: Result : v1, . . . vl3: Initialization : choose random unit norm v0

l , l = 1, . . . , L ;4: s = 0 ;5: repeat6: for l = 1 to L do7:

zsl =

l−1∑k=1

clkg′(vsl>P>l Pkvs+1

k )P>l Pkvs+1k +

L∑k=l

clkg′(vsl>P>l Pkvsk )P>l Pkvsk

8: (vK

?

l , vJ?

l

)= argmax

vKl ,vJl‖vKl ⊗vJl ‖=1

vKl>

Qsl vJl

9: end for10: s = s + 1 ;11: until f (vs+1

1 , . . . , vs+1L )− f (vs1, . . . , v

sL) < ε

5/9


6/9

Discussion : Next components

Deflation is used. Same optimization problem is solved butreplacing each Xl , l = 1, . . . , L by

X(1)l = Xl − ξ

(1)l

((ξ

(1)l )>ξ

(1)l

)−1(ξ

(1)l )>Xl

6/9


7/9

Discussion : Choice of Ml

Ml = (1− τ)1

IX>l Xl + τ I

Same as in RGCCA. However, MJl and MK

l can appears

w>l Mlwl =

(wK

l ⊗wJl

)>[

(1− τ)1

IX>

l Xl + τ I

] (wK

l ⊗wJl

)= wK

l

> (IKl ⊗wJ

l

)>[

(1− τ)1

IX>

l Xl + τ I

] (IKl ⊗wJ

l

)wK

l

= wKl

>

(1− τ)1

I

( Jl∑j=1

w Jlj X

l.j.

)>( Jl∑j=1

w Jlj X

l.j.

)+ τ‖wJ

l ‖2IKl

wKl

= wKl

>MK

l wKl

7/9


8/9

Raman Data

8/9


9/9

Raman results

Table – 10 folds MCCV with ntrain = 16/ntest = 10 pairwised

Method Mean Median Std

Train

MGCCA hierarchical 1 1 0

MGCCA complete 1 1 0

Parafac 0.95 1 0.11

RGCCA hierarchical 1 1 0

Test

MGCCA hierarchical 0.72 0.8 0.17

MGCCA complete 0.74 0.8 0.16

Parafac 0.70 0.8 0.19

RGCCA hierarchical 0.70 0.8 0.199/9


multiway generalized canonical correlation analysis (mgcca)€¦ · multiway generalized canonical...

Documents