generative models of m/eeg: group inversion and meg+eeg+fmri multimodal integration rik henson (with...

Generative Models of M/EEG:

Group inversion and MEG+EEG+fMRI multimodal integration

Rik Henson

(with much input from Karl Friston)

Overview

1. A Generative Model of M/EEG

2. Group inversion (optimising priors across subjects)

3. Multimodal integration:

3.1 Symmetric integration (fusion) of MEG + EEG

3.2 Asymmetric integration of MEG + fMRI

3.3 Full fusion of MEG/EEG + fMRI?

1. A PEB Framework for MEG/EEG(Generative Model)

Phillips et al (2005), Neuroimage

Y = LJ +EY = Data n sensorsJ = Sources p>>n sourcesL = Leadfields n sensors x p sourcesE = Error n sensors

(Linear) Forward Model for MEG/EEG (for one timepoint):

| ( | ) ( )p( ) p pJ Y Y J J

(Gaussian) Likelihood:

( )( | ) ( , )ep NY J LJ C

( ) (0, )jp N ( )J C

C(e) = n x n Sensor (error) covariance

Prior:

C(j) = p x p Source (prior) covariance

Posterior:

( )eiQ

Specifying (co)variance components (priors/regularisation):

1. Sensor components, (error):

i ii

C Q C = Sensor/Source covarianceQ = Covariance componentsλ = Hyper-parameters

( )jiQ2. Source components,

(priors/regularisation):

“IID” (white noise):

# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

“IID” (min norm):

# sources

# s

ou

rce

s

Multiple Sparse Priors (MSP):

( ) ( , )p NX m C

# sources

# s

ou

rce

sFriston et al (2008) Neuroimage


ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei


Friston et al (2008) Neuroimage

Fixed

Variable

Data

( )1jQ ( )

2jQ

1. A PEB Framework for MEG/EEG(Inversion)

Friston et al (2002) Neuroimage

ˆ max ( | ) maxp F

λ Y λ

ˆˆln ( | ) ln ( , | ) ( , )p m p J m dJ F Y Y J λ

ˆˆ max ( | , ) maxj jp F J J Y λ

ln ( | , ) ( | ) lnq

F p p q Y J λ J λ

1. Obtain Restricted Maximum Likelihood (ReML) estimates of the hyperparameters (λ) by maximising the variational “free energy” (F):

2. Obtain Maximum A Posteriori (MAP) estimates of parameters (sources, J):( ) ( ) ( ) -1ˆ ˆ ˆ( )j T j T eC L LC L +C Y cf. Tikhonov

…and an estimate of their posterior covariance (inverse precision):

( ) ( ) ( )ˆ ˆ ˆ ˆˆ -j j j T -1Σ C C L C LC

3. Maximal F approximates Bayesian (log) “model evidence” for a model, m:

(relevant to MEG+EEG integration)

(relevant to MEG+fMRI integration)

( )ˆ ˆ ˆj e T ( )C LC L C

ˆ{ , , }m L Q λ

Summary:

1. A PEB Framework for MEG/EEG

• Automatically “regularises” in principled fashion…

• …allows for multiple constraints (priors)…

• …to the extent that multiple (100’s) of sparse priors possible…

• …(or multiple error components or multiple fMRI priors)…

• …furnishes estimates of source precisions and model evidence

2. Group Inversion

( )eiQ



i ii





# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors


# sources

# s

ou

rce

s


( ) ( , )p NX m C

# sources

# s

ou

rce


( )eiQ



i ii


( )jiQ


# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors

( ) ( , )p NX m C

2. Optimise Multiple Sparse Priors by pooling across participants

2. Group Inversion

Litvak & Friston (2008) Neuroimage

# sources

# s

ou

rce

s

ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei

2. Group Inversion (single subject)(Generative Model)


( )1jQ ( )

2jQ

ΕJ

1Y

iL

( )jC ( )eC

( , )N 0 C ( , )N 0 C

...

( )jk

( )21eQ

( )eik

2. Group Inversion (multiple subjects)(Generative Model)


( )1jQ ( )

2jQ

2Y NY

( )12eQ ( )

1eNQ( )

11eQ ...

)()()( jk

jk

j QC

Ti

eki

eik

ei AQAC )()()(

)(0

)(0

eTj CLCLC

)(

)(2

)(1

)(

00

0

0

00

eN

e

e

e

C

C

C

C

NNNE

E

E

J

L

L

L

Y

Y

Y

2

1

2

1

2

1

~

~

~

iii YAY ~

…projecting data and leadfields to a reference subject (0):

Common source-level priors:

Subject-specific sensor-level priors:

10 )( T

iiTii LLLLA

2. Group Inversion(Generative Model)


2. Group Inversion(Generative Model)


MMN MSP MSP (Group)

fMRI MEG ? (future)Data:

Causes (hidden):

Generative (Forward)Models:

BalloonModel

HeadModel

?

EEG

HeadModel

“Neural”Activity

3. Types of Multimodal Integration

(inversion)

AsymmetricIntegration


Causes (hidden):

Generative (Forward)Models:

BalloonModel

HeadModel

?

EEG

HeadModel


SymmetricIntegration(Fusion)

3. Types of Multimodal Integration

Daunizeau et al (2007), Neuroimage

( )eiQ



i ii





# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors


# sources

# s

ou

rce

s


( ) ( , )p NX m C

# sources

# s

ou

rce


3.1 Fusion of MEG+EEG(Theory)

( )eijQ



Ci(e) = Sensor error covariance for ith modality

Qij = jth component for ith modalityλij = Hyper-parameters




# sources

# s

ou

rce

s


# sources

# s

ou

rce

s

( ) ( ) ( )e e ei ji ij

j

C Q

# sensors

# s

en

sors

# sensors

# s

en

sors

E.g, white noise for 2 modalities:( )11eQ ( )

21eQ

Henson et al (2009) Neuroimage


ΕJ

MEGY

MEGL

( )jC( )eC

( , )N 0 C ( , )N 0 C

( )1jQ ( )

2jQ

( )ji

( )1eQ ( )

2eQ

( )ei

3.1 Fusion of MEG+EEG(Generative Model)


ΕJ

MEGY

MEGL

( )jC( )1eC

( , )N 0 C ( , )N 0 C

( )1jQ ( )

2jQ

( )ji

( )11eQ ( )

12eQ

( )eij

3.1 Fusion of MEG+EEG(Generative Model)

EEGY

EEGL

( )2eC

( )21eQ ( )

22eQ




(1)11 1(1)22 2

(1)dd d

EY L

EY LJ

EY L

• Stack data and leadfields for d modalities:

1 ( )i

ii T

i im

YY

tr YY

• Where data / leadfields scaled to have same average / predicted variance:

)(

)(2

)(1

)(

00

0

0

00

ed

e

e

e

C

C

C

C

mi = Number of spatial modes (e.g, channels)1 ( )

i

ii T

i im

LL

tr L L

(note: common sources and source priors, but separate error components)

ERs from 12 subjects for 3 simultaneously-acquired Neuromag sensor-types:

RM

S f

T/m V

FacesScrambled

fT

3.1 Fusion of MEG+EEG(Application)

Magnetometers (MEG, 102)

(Planar) Gradiometers (MEG, 204)

Electrodes (EEG, 70)


150-190ms

Faces - Scrambled

ms ms ms

MEG mags MEG grads

EEG

FUSED

+31 -51 -15 +19 -48 -6

+43 -67 -11 +44 -64 -4

3.1 Fusion of MEG+EEG


ˆ1/ 59ii ˆ1/ 76ii

IID noise for each modality; common MSP for sources

ˆ1/ 95ii ˆ1/ 127ii

(fixed number of spatial+temporal modes)

Scrambled

150-190msFaces – Scrambled,

Faces

3.1 Fusion of MEG+EEG(Conclusions)


• Fusing magnetometers, gradiometers and EEG increased the conditional precision of the source estimates relative to inverting any one modality alone (when equating number of spatial+temporal modes)

• The maximal sources recovered from fusion were a plausible combination of the ventral temporal sources recovered by MEG and the lateral temporal sources recovered by EEG

• (Simulations show the relative scaling of mags and grads agrees with empty-room data)

3.2 Integration of M/EEG+fMRI

( )eiQ



i ii





# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors


# sources

# s

ou

rce

s


( ) ( , )p NX m C

# sources

# s

ou

rce


Henson et al (in press) Human Brain Mapping

( )eiQ



i ii


( )jiQ


# sensors

# s

en

sors

Empty-room:

# sensors

# s

en

sors


# sources

# s

ou

rce

s

fMRI Priors:

( ) ( , )p NX m C

# sources

# s

ou

rce

s

2. Each suprathreshold fMRI cluster becomes a separate prior

3.2 Integration of M/EEG+fMRI

3.2 Integration of M/EEG+fMRI(Generative Model)

ΕJ

YL

( )jC( )eC

( , )N 0 C ( , )N 0 C

...

( )ji

( )1eQ ( )

2eQ ...

( )ei

( )1jQ ( )

2jQ

3.2 Integration of M/EEG+fMRI(Generative Model)

ΕJ

L

( )jC( )eC

( , )N 0 C ( , )N 0 C

MEGY

fMRIY

( )1jQ ( )

2jQ

( )ji

( )ei

( )1eQ ( )

2eQ( )

3jQ ( )

4jQ


T1-weighted MRI

Anatomical data

{T,F,Z}-SPM

Gray matter segmentation

Cortical surfaceextraction

3D geodesicVoronoï diagram

Functional data

…

1. Thresholding and connected component labelling

…

2. Projection onto the cortical surface using the Voronoï diagram

…

3. Prior covariance components ( )jiQ

3.2 Integration of M/EEG+fMRI (Priors)

3.2 Integration of M/EEG+fMRI (Application)


SPM{F} for faces versus scrambled faces,

15 voxels, p<.05 FWE

5 clusters from SPM of fMRI data from separate group of (18) subjects in MNI space

1 2

3 4 5

3.2 Fusion of MEG+fMRI (Application)


(binarised, variance priors)

Magnetometers (MEG)

* *

* *

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*

Gradiometers (MEG)



(binarised, variance priors)

Magnetometers (MEG)

* *

* *

Gradiometers (MEG)

None Global Local (Valid) Local (Invalid) Valid+Invalid

Electrodes (EEG)

Neg

ativ

e F

ree

Ene

rgy

(a.u

.)(m

odel

evi

denc

e)

*

*

**

*

*

*



None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)



fMRI priors counteract superficial bias of L2-norm

None Global Local (Valid) Local (Invalid)

Magnetometers (MEG)

Gradiometers (MEG)

Electrodes (EEG)

IID sources and IID noise (L2 MNM)



NB: Priors affect variance, not precise timecourse…

R

L

Gradiometers (MEG)

(5 Local Valid Priors)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

Right Posterior Fusiform (rPF) Right Medial Fusiform (rMF) Right Lateral Fusiform (rLF)

Left occipital pole (lOP)

-27 -93 0

+26 -76 -11 +41 -43 -24 +32 -45 -12

-43 -47 -21

Left Lateral Fusiform (lLF)

Diff

eren

tial R

espo

nse

(Fac

es v

s S

cram

bled

)

• Adding a single, global fMRI prior increases model evidence

• Adding multiple valid priors increases model evidence furtherHelpful if some fMRI regions produce no MEG/EEG signal (or arise from neural activity at different times)

• Adding invalid priors rarely increases model evidence, particularly in conjunction with valid priors

• Can counteract superficial bias of, e.g, minimum-norm

• Affects variance but not not precise timecourse

• (Adding fMRI priors to MSP has less effect)

3.2 Fusion of MEG+fMRI (Conclusions)


3.3 Fusion of fMRI and MEG/EEG?


Causes (hidden):

BalloonModel

HeadModel

?

EEG

HeadModel


Fusion of fMRI + MEG/EEG?

Henson (2010) Biomag

J

MEGY

( )sMEGL

( )jC

( )1jQ ( )

2jQ

MEGΕ

( )eC

( )1eQ ( )

2eQ



J

MEGY

( )sMEGL

( )jC

( )1jQ ( )

2jQ

( )tfMRIH

MEGΕ

( )eC

( )1eQ ( )

2eQ

fMRIΕ

fMRIY

( )1tA ( )

2sQ

time (t)?space (s)



Overall Conclusions

1. The PEB (in SPM8) framework is advantageous

2. Group optimisation of MSPs can be advantageous

3. Full fusion of MEG and EEG is advantageous

4. Using fMRI as (spatial) priors on MEG is advantageous

5. Unclear that fusion of fMRI and M/EEG is advantageous

The End

3. Fusion of MEG+EEG


Participant

Grads

Mags

log

(λx10

6 )

Participant

EE

G

Grads

Mags

log

(λx10

6 )

3. Fusion of MEG+EEGH

yper

pa

ram

ete

rs


4. Fusion of MEG+fMRI


fMR

I hyp

erp

ara

me

ters

ln(λ

)+32

ln(λ

)+32

Participant

Participant

Magnetometers (MEG) Gradiometers (MEG) Electrodes (EEG)

Local Valid

Local Invalid

generative models of m/eeg: group inversion and meg+eeg+fmri multimodal integration rik henson (with...

Documents

source components

sensor components

generative forwardmodels

megeeg generative model1

multiple sparse priors

group inversionlitvak

peb framework

generative models of