![Page 1: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/1.jpg)
Neural NetworksChapter 2
Joost N. Kok
Universiteit Leiden
![Page 2: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/2.jpg)
Hopfield Networks
• Network of McCulloch-Pitts neurons • Output is 1 iff and is -1
otherwise i
jjijSw
![Page 3: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/3.jpg)
Hopfield Networks
![Page 4: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/4.jpg)
Hopfield Networks
![Page 5: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/5.jpg)
Hopfield Networks
![Page 6: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/6.jpg)
Hopfield Networks
• Associative Memory Problem:
Store a set of patterns in such a way that when presented with a new pattern, the network responds by producing whichever of the stored patterns most closely resembles the new pattern.
![Page 7: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/7.jpg)
Hopfield Networks
• Resembles = Hamming distance
• Configuration space = all possible states of the network
• Stored patterns should be attractors
• Basins of attractors
![Page 8: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/8.jpg)
Hopfield Networks
• N neurons
• Two states: -1 (silent) and 1 (firing)
• Fully connected
• Symmetric Weights
• Thresholds
![Page 9: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/9.jpg)
Hopfield Networksw13
w16 w57
-1 +1
![Page 10: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/10.jpg)
Hopfield Networks
• State:
• Weights:
• Dynamics:
25,251,25
25,11,1
ww
ww
w
251 ... SSS
25
1sgn:
i jiji SwS
![Page 11: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/11.jpg)
Hopfield Networks
• Hebb’s learning rule: – Make connection stronger if neurons have the
same state– Make connection weaker if the neurons have a
different state
![Page 12: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/12.jpg)
Hopfield Networksneuron 1 synapse neuron 2
![Page 13: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/13.jpg)
)(
1
)(1
j
p
ipijw
Hopfield Networks
• Weight between neuron i and neuron j is given by
![Page 14: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/14.jpg)
Hopfield Networks
• Opposite patterns give the same weights
• This implies that they are also stable points of the network
• Capacity of Hopfield Networks is limited: 0.14 N
![Page 15: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/15.jpg)
Hopfield Networks
• Hopfield defines the energy of a network:
E = - ½ ij SiSjwij + i Sii
• If we pick unit i and the firing rule does not change its Si, it will not change E.
• If we pick unit i and the firing rule does change its Si, it will decrease E.
![Page 16: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/16.jpg)
![Page 17: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/17.jpg)
Hopfield Networks
• Energy function:
• Alternative Form:
• Updates:
ij
jiij SSwH2
1
)(ij
jiij SSwCH
jjiji SwS sgn'
![Page 18: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/18.jpg)
Hopfield Networks
ij
jiijij
jiij SSwSSwHH ''
iij
jijiij
jiji wSwSSwSHH 222'
![Page 19: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/19.jpg)
Hopfield Networks
• Extension: use stochastic fire rule– Si := +1 with probability g(hi)– Si := -1 with probability 1-g(hi)
![Page 20: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/20.jpg)
Hopfield Networks
• Nonlinear function:
x
g(x)
g(x) = 1 + e – x
1
0
![Page 21: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/21.jpg)
Hopfield Networks• Replace the binary threshold units by binary stochastic
units.• Define = 1/T• Use “temperature” T to make it easier to cross energy
barriers.– Start at high temperature where its easy to cross energy barriers.
– Reduce slowly to low temperature where good states are much more probable than bad ones.
A B C
![Page 22: Neural Networks Chapter 2 Joost N. Kok Universiteit Leiden](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d775503460f94a58774/html5/thumbnails/22.jpg)
Hopfield Networks
• Kick the network our of spurious local minima
• Equilibrium: becomes time independent
iS
)..2exp(1
11Pr
iii h
hfS
T
1