formalizing didday’s modelilab.usc.edu/classes/2002cs564/lecture_notes/07w-amari... · 2002. 9....

Michael Arbib & Laurent Itti: CS564 Brain Theory and Artificial Intelligence, USC, Fall 2000. Lect. 5. Didday Prey-Selector 1

Formalizing Didday's model These pages complete the lecture contained in the Powerpoint file: 5. Didday Max-Selector. The following analysis comes from TMB2, Section 4.4, and is based on the paper: Arbib, M.A., and Amari, S.I., (1977) Competition and

Cooperation in Neural Nets, in Systems Neuroscience (J. Metzler, Ed.), New York: Academic Press, pp. 119-165.


Replace the neural fields by a string of cells indexed by i for 1≤i≤n, with the blind spot only one neuron wide: si input "foodness" signal at position i ui(t) membrane potential of the "relative foodness" unit Ei Si(t) membrane potential of the "sameness" cell i


Ei's output is the step function (aka Heaviside function)

f(u) = ��1 u > 00 u ≤ 0

Si's output is the semi-linear function

g(Si) = ��Si Si > 0

0 Si ≤ 0


The Didday model is then given by

τ' dui(t)

dt = -ui(t) + si - c g(Si(t)) - h

and

τ dSidt = -Si + �

i≠j f(uj) - h3

for suitable time constants τ, τ' and constants c, h, and h3. At equilibrium Si = �

i≠j f(uj) - h3 .


Amari and Arbib 1977 offered a variant, the primitive competition model, in which the sameness units are combined into a single inhibitory cell I with membrane potential v(t) while Ei has local recurrent excitation to replace

removal of the blind spot from inhibitory summation.

+ +-

dui(t)

dt = -ui + w1 f(ui) - w2 g(v) - h1+ si

τ dvdt = -v + �

i=1

n f(ui) - h2


To relate this to the previous model, recall that

τ dSidt = - Si + �

i≠j f(uj) - h3 .

Sum these differential equations for 1≤i≤n, then divide by n-1. Then on setting

v(t) = �=

n

iiS

1/(n-1)

we obtain

τ dvdt = -v(t) + �

i=1

n f(uj) -

nn-1 h3

which is just the Amari-Arbib formula

τ dvdt = -v + �

i=1

n f(ui) - h2

if we set h2 = n

n-1 h3.


Now recall that, at equilibrium,

Si = �i≠j

f(uj) - h3 while v = �i=1

n f(ui) - h2 .

Hence, near equilibrium, we may approximate Si by v - f(ui) + h', where h' = h2 - h3 > 0, so that the Diddayesque formula

τ' dui(t)

dt = -ui(t) + si - c g(Si(t)) - h

can be approximated by

τ' dui(t)

dt = -ui(t) + si - c g(v - f(ui) + h') - h

which in turn is approximately = -ui(t) + si + c f(ui) - c g(v) - (h + ch').


Proving Properties of the Amari-Arbib Version of the Didday Model:

It is a maximum selector or winner-take-all circuit: Given a set of inputs, it is to pick the largest, but to do this in a distributed way.

dui(t)

dt = -ui + w1 f(ui) - w2 g(v) - h1+ si (1a)

τ dvdt = -v + �

i=1

n f(ui) - h2 (1b)

where ui's output is the step function

f(u) = ��1 u > 00 u ≤ 0

and v's output is the semi-linear function

g(v) = ��v v > 00 v ≤ 0


Properties of the Model

At equilibrium, dui(t)

dt = 0 = dv/dt, and so

v = �j

f(uj) - h2 (2a)

and thus each ui = w1f(ui) - w2 g(�

jf(uj)-h2 ) - h1 + si (2b)


Observation: When all si equal zero, the state ui = - h1, v = - h2 is an equilibrium state if h1 ≥ 0, h2 ≥ 0.

Proof: If h1 ≥ 0, h2 ≥ 0, then f(-h1) = 0 and g(-h2) = 0, and so (2a) and (2b) are satisfied. Postulate (i): h1 ≥ 0, h2 ≥ 0.


Observation: A quiescent state (all ui < 0) is an equilibrium only if stimulus si < h1 for each i.

Proof: If each ui ≤ 0, then each f(ui) = 0, and so (2b) becomes ui = -h1 + si and this is less than 0 only if si < h1. For this reason, we may call h1 the threshold for (1a).


Let the system start in the quiescent state ui = -h1, v = -h2 and let s1 > 0, while all other si = 0. By continuity, this guarantees a non-zero time interval during which all ui and v remain negative, so that

du1(t)

dt = -(u1 + h1) + s1 ≥ 0 if s1 ≥ h1 + u1

while dui/dt = -(ui + h1) = 0 for i ≠ 1, and τ dv/dt = -(v + h2) = 0. Hence u1 will grow from -h1 to exceed 0 so long as s1 > h1, else u1 will remain below 0. Again we see h1 acting as threshold it is the minimal stimulation that will cause a neuron to fire, when starting from the quiescent state.


Apply s1> h1, and wait till u1 first exceeds 0: Then there is a non-zero interval in which only u1 is positive and the dynamics of v are given by τ dv/dt = - (v + h2) + 1 = - v + (1 - h2) which tends monotonically toward the equilibrium value v = 1 - h2.

There cannot be effective inhibition during this period unless 1 - h2 > 0, i.e., h2 < 1. To ensure that the activity of a single cell can activate the inhibition of other cells, we introduce Postulate (ii): h2 < 1.


Theorem 1: Given Postulates (i) and (ii), the number K of elements excited in equilibrium is less than

(s + w1 - h) w2 -1 + h2 where s is the smallest stimulus for any excited element.

Proof: Assume that K elements are excited in the equilibrium, so that �f(uj) = K.

Let ui be one of the excited elements. Then ui = w1 f(ui) - w2 g(K - h2) - h1 + si But 0 < ui and f(ui) = 1, so 0 < w1 - w2(K - h2) - h1 + si and so w2(K - h2) < si + w1 - h1

i.e., K < (si + w1 - h1) w2 -1 + h2 (3) even if si is the smallest stimulus s applied to an excited element.


K < (si + w1 - h1) w2 -1 + h2 (3) Corollary: If the maximum stimulus applied to the network satisfies smax < h1 - w1 + (L + 1 - h2) w2 then at most L elements can be excited in equilibrium. Proof: smax < h1 - w1 + (L + 1 - h2) w2 means

(smax + w1 - h1) w2 -1 < L + 1 - h2 By (3), we then have

K < (smax + w1 - h1) w2 -1 + h2 < (L + 1 - h2) + h2 = L + 1. an equilibrium state selects one or more elements, the number that can break through their neighbor's inhibition increases with the overall level of stimulation. When L = 1, there is at most one winner: smax < h1 - w1 + (2 - h2) w2


Theorem 2: If the system starts from a state in which all ui's are equal and reaches an equilibrium in which K elements are excited, then they are those receiving the K largest stimuli. Proof: Assume that s1 > s2 and u1(0) = u2(0).

Recall dui(t)

dt = -ui + w1 f(ui) - w2 g(v) - h1+ si (1a)

Then U(t) = u1(t) - u2(t) satisfies dU(t)/dt = -U(t) + w1[f(u1) - f(u2)] + s1 - s2. = a(t) where a(t) = w1[f(u1) - f(u2)] + s1 - s2. Now observe that the solution U(t) of the equation dU(t)/dt = -U(t) + a(t) never becomes negative if U(0) > 0 and a(t) > 0. Since s1 - s2 ≥ 0 while f(u1) - f(u2) > 0 if u1 > u2, we have U(t) > 0, i.e., u1(t) > u2(t), for t >0. Thus the ordering of the ui(t)'s is that of the si's the K elements excited in equilibrium must be those receiving the K largest stimuli.


Hysteresis To reset the network globally Amari-Arbib either temporarily drop the threshold of all cells so far that all cells

are equally active (equivalently, we may provide strong global excitation); or

temporarily raise the threshold so high that all cells are equally inactive (equivalently, we may provide strong global inhibition).

[Recall: Didday used a different mechanism, which is local rather than global, to combat hysteresis. introducing an "N-cell" for each S-cell to monitor temporal

changes in the activity of its region. For a dramatic increase in the region's activity it overrides the inhibition on the S-cell and permits this new level of activity to enter the relative

foodness layer.]


Temporarily drop the threshold: If ui is excited as one of K elements excited at equilibrium, we saw in the proof of Theorem 1 that ui = w1 - w2(K - h2) - h1 + si > 0 (4) If only element 1 is active, u1 = w1 - w2(1 - h2) - h1 + s1 > 0 i.e., s1 > h1 - w, where w= w1 - w2(1 - h2). When h1 changes to a bigger threshold h1' with h1' > h1, element 1 remains excited when s1 > h1' - w and is inactivated when s1 < h1' -w. Once it is inactivated, element 2, which receives bigger stimulus s2, becomes newly excited once the threshold h1' returns to the original h1. Therefore, by this process of changing the threshold, h1→h1'→h1, the model can take the new situation into account provided s1 is smaller than h1' - w. However, if s1 is bigger than h1'-w, no change occurs even if s2 is far bigger.


Temporarily raise the threshold: The other process is the change h1→h1"→h1 (h1" < h1) of the threshold.

Rewriting ui = w1 - w2(K - h2) - h1 + si > 0 (4) with K = 2, w1 - w2(2 - h2) - h1" + si = w" - h1" + si > 0 where w" = w1 - (2 - h2)w2. Thus element 2 is activated only when h1" < s2 + w" i.e., s2 > h1" - w" = sc , say. When element 2 becomes activated, then by increasing the threshold to again equal h1, element 1 becomes inactivated because only one element can be excited at the threshold h1 and s1 < s2. However, even when h1 changes to h1", no change occurs if s2 is (or other si"s are) weaker then h1" - w". We may thus call sc = h1" - w" the challenging value. By this process, the elements which receive a stimulus bigger than sc can challenge the element which has been already excited, and the one which receives the maximum stimulus wins among them.

formalizing didday’s modelilab.usc.edu/classes/2002cs564/lecture_notes/07w-amari... · 2002. 9....

Documents