symmetry breaking clusters when deciphering the neural code september 12, 2005 albert e. parker...

Symmetry breaking clusters when deciphering the neural code

September 12, 2005

Albert E. Parker

Department of Mathematical Sciences

Center for Computational Biology

Montana State University

Collaborators: Tomas Gedeon, Alex Dimitrov, John Miller, and Zane Aldworth

Deciphering the Neural CodeA Clustering ProblemThe Dynamical SystemBifurcationsTheoretical ResultsNumerical Results

Talk Outline

Deciphering the Neural Code:

How does neural activity represent information about environmental stimuli?

“The little fly sitting in the fly’s brain trying to fly the fly”

Inputs: stimuliX outputs: neural response

Y

Looking for the dictionary to the neural code …

decoding

encoding

… but the dictionary is not deterministic!

Given a stimulus, an experimenter observes many different neural responses:

X

Yi| Xi = 1, 2, 3, 4

… but the dictionary is not deterministic!

Given a stimulus, an experimenter observes many different neural responses:

Neural encoding is stochastic!!

X

Yi| Xi = 1, 2, 3, 4

Similarly, neural decoding is stochastic:

Y

Xi|Yi = 1, 2, … , 9

Probability Framework

X Y

environmentalstimuli

neuralresponses

decoder: P(X|Y)

encoder: P(Y|X)

Deciphering the Neural Code=

Determining the encoder P(Y|X) or the decoder P(X|Y)

Common Approaches: parametric estimations, linear methods

Difficulty: There is never enough data.

One Approach: Cluster the responses

X Y

Stimuli Responses

Zq(Z |Y)

Clustered Responses

K objects {yi} N objects {zi}L objects {xi}

p(X,Y)


X Y

Stimuli Responses

Zq(Z |Y)

Clustered Responses


p(X,Y)

P(Y|X)

P(X|Y)


X Y

Stimuli Responses

Zq(Z |Y)

Clustered Responses


p(X,Y)

P(Y|X) P(Z|X)

P(X|Y) P(X|Z)


• q(Z|Y) is a stochastic clustering of the responses • The outputs Y are clustered in Z so that the information that one can learn about X by observing Z , I(X;Z), is as close as possible to the mutual information I(X;Y)

X Y

Stimuli Responses

Zq(Z |Y)


p(X,Y)

Clustered Responses

• Rate Distortion Theory (Shannon 1950’s) Minimal Informative Compression

min I(X,Z) constrained by D(X,Z) D0

• Deterministic Annealing (Rose 1990’s) A Clustering Algorithm

max H(Z|X) constrained by D(X,Z) D0

Relationship between these formulations:

I(X,Z)=H(Z) – H(Z|X)

q

Two optimization problems which use this approach optimizing at a distortion level D(Y,Z) D0

q

• Information Bottleneck Method (Tishby, Pereira, Bialek 1999)

min I(Y,Z) constrained by I(X;Z) I0

max –I(Y,Z) + I(X;Z)

• Information Distortion Method (Dimitrov and Miller 2001)

max H(Z|Y) constrained by I(X;Z) I0

max H(Z|Y) + I(X;Z)

•

q

Examples:

q

q

q

A basic annealing algorithmto solve

maxq(G(q)+D(q))

Let q0 be the maximizer of maxq G(q), and let 0 =0. For k 0, let (qk , k ) be a solution to maxq G(q) + D(q ). Iterate the following steps until K = max for some K.

1. Perform -step: Let k+1 = k + dk where dk>0

2. The initial guess for qk+1 at k+1 is qk+1(0) = qk + for some small

perturbation .

3. Optimization: solve maxq (G(q) + k+1 D(q)) to get the maximizer qk+1 , using initial guess qk+1

(0) .

Application of the annealing method to the Information Distortion problem maxq (H(Z|X) + I(X;Z))

when p(X,Y) is defined by four gaussian blobs

Y, Inputs

X, Outputs

Y X

K=52 outputsL=52 inputs

p(X,Y) X Zq(Z|X)

K=52 outputs N=4 clustered outputs

X, Outputs

Z, C

lust

ered

Ou

tpu

ts

Evolution of the optimal clustering: Observed Bifurcations for the Four Blob problem:

We just saw the optimal clusterings q* at some *= max . What do the clusterings look like for < max ??

I(Y

,Z)

bits

??????

Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations?

What kinds of bifurcations do we expect: pitchfork-like, transcritical, saddle-node, or some other type?

How many bifurcating branches are there?

What do the bifurcating branches look like? Are they subcritical or supercritical ?

What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem?

Are there bifurcations which alter the classes after all of the classes have resolved ?

q*

Conceptual Bifurcation Structure

Observed Bifurcations for the 4 Blob Problem

I(Y

,Z)

bits

A General Problem:To determine the bifurcations of solutions to clustering problems of the form

maxqG(q) constrained by D(q)I0

where

• q is a vector of conditional probabilities in RNK.• G =g(qi) and D=d(qi) are sufficiently smooth on , and q=(q1

T… qNT)T

where qi RK . This implies that:1. G and D have symmetry: they are invariant to re-labeling of the classes of Z; 2. The Hessians d2G and d2D are block diagonal.

• The Hessians d2G and d2D satisfy a set of generic regularity conditions at bifurcation.

X Z

q(Z|X)

K objects N clusters

A similar formulation:Using the method Lagrange multipliers, the goal of determining the bifurcation structure

of solutions of the optimization problem can be rephrased as finding the bifurcation structure of stationary points of the problem

maxq(G(q)+D(q))

where [0,).• q is a vector of conditional probabilities in RNK.• G =g(qi) and D=d(qi) are sufficiently smooth on , and q=(q1

T… qNT)T

where qi RK . • The Hessians d2G and d2D satisfy a set of generic regularity conditions at bifurcation.

X Z

q(Z|X)

K objects N clusters

The Dynamical SystemGoal: To solve maxq (G(q) + D(q)) for each , incremented in

sufficiently small steps, as .

Method: Study the equilibria of the of the gradient flow

• Equilibria of this system are possible solutions of the the maximization problem (satisfy the necessary conditions of constrained optimality)

• If wT d2q (G(q*) + D(q*))w < 0 for every wker J, then q* is a maximizer

of .

• The Jacobian q,L(q*,*) is symmetric, and so only bifurcations of equilibria can occur.

• The first equilibrium is q*(0 = 0) 1/N.

Yy z

yqq yzqqDqGqq

1)|()()(:),,( ,, L

The Symmetries:

To better understand the bifurcation structure, we capitalize on the symmetries of the function G(q)+D(q)

X Zq(Z|X) : a clustering

K objects {xi} N objects {zi}

class 1

class 3

X Zq(Z|X) : a clustering

K objects {xi} N objects {zi}

class 3

class 1

The Symmetries:

To better understand the bifurcation structure, we capitalize on the symmetries of the function G(q)+D(q)

The symmetry group of all permutations on N symbols

is

SN.

Equivariant Branching Lemma:

The subgroups of SN

with 1D fixed point spaces

determine the Bifurcation Structure

4S

3S3S

3S 3S

0

3

vv

v

v

0

3

vv

v

v

0

3vv

v

v

0

3vv

v

v

2S2S 2S2S2S2S2S2S

1

0

2

0

vv

v

2S 2S 2S2S

0

2

0

vv

v

0

2

0

vv

v

0

0

2

vv

v

0

2

0

vv

v

0

2

0

vv

v

0

0

2

v

v

v

0

20v

v

v

0

0

2

v

v

v

0

0

2

v

v

v

0

0

2

v

v

v

0

02v

v

v

A partial subgroup lattice for S4 and the corresponding bifurcating directions

4S

34,12 24,13

23,14

v

v

v

v

v

v

v

v

v

v

v

v

A partial subgroup lattice for S4 and the corresponding bifurcating directions corresponding to subgroups isomorphic to S2 x S2.

Symmetry Breaking Bifurcations

q*

4

11

N

q

41 by fixed is SSq N

N


q*

4

11

N

q

*q


N

31* by fixed is SSq N

4S

3S3S

3S 3S

2S2S 2S2S2S2S2S2S

1

2S 2S 2S2S


q*

4

11

N

q

*q


N


*q


4S

3S3S

3S 3S

2S2S 2S2S2S2S2S2S

1

2S 2S 2S2S


q*

4S

3S3S

3S 3S

2S2S 2S2S2S2S2S2S

1

2S 2S 2S2S


q*

4S

34,12 24,13

23,14


q*

*q

)34(),12(by fixed is 22* SSq

4S

34,12 24,13

23,14

Observed Bifurcation Structure

4S

3S3S

3S 3S

2S2S 2S2S2S2S2S2S

1

2S 2S 2S2S

Group Structure

Group Structure

q*Observed Bifurcation Structure

4S

3S3S

3S 3S

2S2S 2S2S2S2S2S2S

1

2S 2S 2S2S

The Equivariant Branching Lemma shows that the bifurcation structure contains the branches …

Group Structure

q*Observed Bifurcation Structure

4S

34,12 24,13

23,14

The subgroups {S2x S2} give additional structure …

q*

Theorem: There are at exactly K bifurcations on the branch (q1/N , ) whenever G(q1/N) is nonsingular

There are K=52bifurcations on the first

branch

Observed Bifurcation Structure

??????

Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations?


How many bifurcating solutions are there?

What do the bifurcating branches look like? Are they subcritical or supercritical ?

What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem?

Are there bifurcations which alter the classes after all of the classes have resolved ?

q*

Conceptual Bifurcation Structure

Observed Bifurcations for the 4 Blob Problem

??????

Why are there only 3 bifurcations observed? In general, are there only N-1 bifurcations?There are N-1 symmetry breaking bifurcations from SM to SM-1 for M N.


How many bifurcating solutions are there? There are at least N from the first bifurcation (SN SN –1), at least N-1 from the next one (SN -1 SN –2), etc, as well as branches with symmetry breaking from SM Sm x Sn

for all (m,n) where m + n =M.

What do the bifurcating branches look like? They are subcritical or supercritical depending on the sign of the bifurcation discriminator (q*,*,m,n) .

What is the stability of the bifurcating branches? Is there always a bifurcating branch which contains solutions of the optimization problem? Yes for , No for the annealing problem .

Are there bifurcations which alter the classes after all of the classes have resolved ? Generically, no.

Conceptual Bifurcation StructureObserved Bifurcations for the 4 Blob Problem

q*

Hessian d constraine theis ,2 Lqd

DGF

Fd q

Hessian nedunconstrai is2

singular is ,2 Lqd

singular is2 Fd q rnonsingula is2 Fd q

rnonsingula is1

1

MN

iKi MIRB

1M 1M

Symmetry breaking pitchfork-like

bifurcation

Impossible scenario

Saddle-node bifurcation

Impossible scenario

Non-generic

rnonsingula is1

1

MN

iKi MIRB

singular is1

1

MN

iKi MIRB

singular is1

1

MN

iKi MIRB

Continuation techniques provide

numerical confirmation of the theory

q*

I(Y

,Z)

bits

Bifurcating branches with symmetry S2 x S2

= <(12),(34)>

q*

I(Y

,Z)

bits

A closer look …

q*

I(Y

,Z)

bits

Bifurcation from S4 to S3…

q*

I(Y

,Z)

bits

The bifurcation from S4 to S3 is subcritical …

(the theory predicted this since the bifurcation discriminator (q1/4,*,m,n)<0 )

I(Y

,Z)

bits

Theorem: The bifurcation discriminator of the pitchfork-like branch

(q*,*,*) + (tu,0,(t)) with symmetry Sm x Sn is

If (q*,*,m,n) < 0, then the branch is subcritical. If (q*,*,m,n) > 0, then the branch is supercritical.

Ki

KT

iii

MIRBA

vvvfdb

bAnmnm

nmmnIBb

FdBv

qdqgqfF

vvvvfdnmq

1

3

122

2

4**

],,[

)(

ofblock singular the, of vector null theis

))()(()(

],,,[3),,(

Additional structure!!

I(Y

,Z)

bits

I(Y

,Z)

bits

I(Y

,Z)

bits

Conclusions …

We have a complete theoretical picture of how the

clusterings evolve for any problem of the form maxq(G(q)+D(q))

subject to the assumptions stated earlier.

SO WHAT?? There are theoretical consequences for “Rate Distortion Curve”… This yields a new and improved algorithm for solving

the neural coding problem …

A numerical algorithm to solve max(G(q)+D(q))

Let q0 be the maximizer of maxq G(q), 0 =1 and s > 0. For k 0, let (qk , k ) be a solution to maxq G(q) + D(q ). Iterate the following steps until K = max for some K.

1. Perform -step: solve

for and select k+1 = k + dk where dk = (s sgn(cos )) /(||qk ||2 + ||k ||2 +1)1/2.

2. The initial guess for (qk+1,k+1) at k+1 is (qk+1

(0),k+1 (0)) = (qk ,k) + dk ( qk, k) .

3. Optimization: solve maxq (G(q) + k+1 D(q)) using pseudoarclength continuation to get the maximizer qk+1, and the vector of Lagrange multipliers k+1 using initial guess (qk+1

(0),k+1 (0)).

4. Check for bifurcation: compare the sign of the determinant of an identical block of each of q [G(qk) + k D(qk)] and q [G(qk+1) + k+1 D(qk+1)]. If a bifurcation is detected, then set qk+1

(0) = qk + d_k u where u is bifurcating direction and repeat step 3.

),,(),,( ,, kkkqk

kkkkq q

qq

LL

k

kq

q

Application to cricket sensory data

E(Y|Z): stimulusmeans conditioned

on each of the classes

Y: Neural responses

Z:optimal clustering

More about Bifurcations Theorem: All symmetry breaking bifurcations are pitchfork-like.

Outline of proof: ’(0)=0 since 2xx r(0,0) =0.

Theorem: Generically, bifurcations which alter the classes do not occur after all of the classes have resolved. That is, only saddle-node bifurcations are possible, which do not alter class structure due to explicit bifurcating direction.

Theorem: If d2D(q*) is positive definite on ker d2F (q*,*), then the singularity (q*,*,*) is a bifurcation. In particular, if d2G(q*) is negative definite on ker d2F (q*,*), then d2D(q*) is positive definite on ker d2F (q*,*).

Theorem: A symmetry breaking bifurcating direction u is an eigenvector of d2q,L ((q*,*)

+tu,*+ (t)) for small t. If the corresponding eigenvalue is positive, then the branch consists of stationary points which are not solutions of .

Theorem: Subcritical bifurcating branches may be solutions of either or Solutions of need not be solutions of . Solutions of are always solutions of .

Theorem: If there exists a saddle-node bifurcation of solutions to the Information Bottleneck problem at I0 = I*, then RI(I0) is neither concave, nor convex in any neighborhood of I¤. Similarly, the existence of a saddle-node bifurcation of solutions to the Information Distortion problem at I0 = I* implies that RH(I0) is neither concave, nor convex in any neighborhood of I*.

Continuation• A local maximum qk

*(k) of is an equilibrium of the gradient flow .• Initial condition qk+1

(0)(k+1(0)) is sought in tangent direction qk, which is found by solving the matrix system

• The continuation algorithm used to find qk+1*(k+1) is based on Newton’s method.

• Parameter continuation follows the dashed (---) path, pseudoarclength continuation follows the dotted (…) path ),,(),,( ,, kkkq

k

k

kkkq qq

q

LL

k)0(

1k

),( , kkkq

),,( 111 kkkq

),( )0(1

)0(1

)0(1 kkkq

),( 11 kkq

),( kkq

),( q

),( )0(1

)0(1 kkq

The Groups• Let P be the finite group of n ×n “block” permutation matrices which represents the action of SN

on q and F(q,) . For example, if N=3,

permutes q(Z1|y) with q(Z2|y) for every y

• F(q,) is P -invariant means that for every P, F( q,) = F(q,)

• Let be the finite group of (n+K) × (n+K) block permutation matrices

which represents the action of SN on and q, L(q,,):

q, L(q, , ) is -equivariant means that for every q, L(q, , ) = q, L( ,)

q

! |0

0: fixed are sconstraint and smultiplier lagrange the

P

KKnK

Kn

I

q

P

K

K

K

I

I

I

00

00

00

Notation and Definitions• The symmetry of is measured by its isotropy subgroup

• An isotropy subgroup is a maximal isotropy subgroup of if there does not exist an isotropy subgroup of such that .

• At bifurcation , the fixed point subspace of q*,* is

qqq |,

q

),( *

*

*

q

**** ,

***,,

,|),,(ker)(Fix

qqq

wwqw L

Equivariant Branching LemmaOne of the Existence Theorems we use to describe a bifurcation in the

presence of symmetries is the Equivariant Branching Lemma (Vanderbauwhede and Cicogna 1980-1).

Idea: The bifurcation structure of local solutions is described by the isotropy subgroups of which have dim Fix()=1. • System: .

• r(x,) is G-equivariant for some compact Lie Group G• • Fix(G)={0}• Let H be an isotropy subgroup of G such that dim Fix (H) = 1.• Assume r(0,0) 0 (crossing condition).

Then there is a unique smooth solution branch (tx0,(t)) to r = 0 such that x0 Fix (H) and the isotropy subgroup of each solution is H.

mmrxrx :),,(

0)0,0(,0)0,0( rr x

Smoller-Wasserman Theorem

Another Existence Theorem:

Smoller-Wasserman Theorem (1985-6)

For variational problems where

there is a bifurcating solution tangential to Fix(H) for every maximal isotropy subgroup H, not only those with dim Fix(H) = 1.

• dim Fix(H) =1 implies that H is a maximal isotropy subgroup

),(),( xfxr x

symmetry breaking clusters when deciphering the neural code september 12, 2005 albert e. parker...

Documents