1 neural networks 3. 2 hopfield network (hn) model a hopfield network is a form of recurrent...

1

Neural networks 3

2

Hopfield network (HN) model• A Hopfield network is a

form of recurrent artificial neural network invented by John Hopfield in 1982

• The network is entirely interconnected – All neurons are both

input and output• with binary threshold

units1, if x > 0

F(x) = -1, otherwise

These are single layered recurrent networks

3

Hopfield network modelApplications

• Recalling or Reconstructing corrupted patterns

• Large-scale computational intelligence systems

• Handwriting Recognition Software• Hopfield is model used to solve

optimization problems such as traveler salesman, task scheduling, etc.

• Practical applications of HNs are limited because number of training patterns can be at most about 14% the number of nodes in the network.

• If the network is overloaded -- trained with more than the maximum acceptable number of attractors -- then it won't converge to clearly defined attractors.

These are single layered recurrent networks

4

Hopfield network model• Hopfield nets serve as content-

addressable memory systems with binary threshold nodes (associative memory)– The information is distributed and stored in the

network connection.• If one element disappear, the information is not lost• We access to information using memory content (CAM

: Content addressable memory)

• The network build progressively its attractors– In the learning phase neurones adjust there

synapses based on there activities and predefined rules

– Hebb rule– Widrow-Hoff rule (delta rule)

5

Hopfield network modelAssociative memory

Learning

6

Hopfield network model

Learning (Hebb rule): Neurone are symetric:

|P|: number of training patterns (input combinations)

is ith bit in the pattern P

0

1

ii

Pp

pj

pijiij

W

ssP

WW

pis

7

• Lets train this network for following patterns• Pattern 1:- ie Oa(1)=-1,Ob(1)=-1,Oc(1)=1

• Pattern 2:- ie Oa(2)=1,Ob(2)=-1,Oc(3)=-1

• Pattern 3:- ie Oa(3)=-1,Ob(3)=1,Oc(3)=1

w1,1 = 0w1,2 = 1/3( OA(1) × OB(1) + OA(2) × OB(2) + OA(3) × OB(3)) =1/3( (-1) × (-1) + 1 × (-1) + (-1) × 1) = 1/3w1,3 = 1/3( OA(1) × OC(1) + OA(2) × OC(2) + OA(3) × OC(3)) = 1/3((-1) × 1 + 1 × (-1) + (-1) × 1) = 1/3( -3)w2,2 = 0w2,1 = 1/3( OB(1) × OA(1) + OB(2) × OA(2) + OB(3) × OA(3)) = 1/3((-1) × (-1) + (-1) × 1 + 1 × (-1) = 1/3( -1)w2,3 = 1/3( OB(1) × OC(1) + OB(2) × OC(2) + OB(3) × OC(3)) = 1/3((-1) × 1 + (-1) × (-1) + 1 × 1) = 1/3(1)w3,3 = 0w3,1 = 1/3( OC(1) × OA(1) + OC(2) × OA(2) + OC(3) × OA(3)) = 1/3( 1 × (-1) + (-1) × 1 + 1 × (-1)) = 1/3( -3)w3,2 = 1/3( OC(1) × OB(1) + OC(2) × OB(2) + OC(3) × OB(3)) = 1/3( 1 × (-1) + (-1) × (-1) + 1 × 1) = 1/3( 1)


8


Utilization :– A pattern is entered in the network by

setting all nodes (neurones) to a specific value, or by setting only part of the nodes.

– The network is then subject to a number of iterations using asynchronous or synchronous updating.

– This is stopped after a while. The network neurons are then read out to see which pattern is in the network.

9


There are two ways of updating them:• Asynchronous: one picks one neuron,

calculates the weighted input sum and updates immediately. This can be done in a fixed order, or neurons can be picked at random, which is called asynchronous random updating.

• Synchronous: the weighted input sums of all neurons are calculated without updating the neurons. Then all neurons are set to their new value, according to the value of their weighted input sum.

10


Neurone state modification:

Nj

ti

ti

tjij

Nj

ti

tjij

Nj

ti

tjij

sssW

ssW

ssW

11

1

1

then 0. if

1 then 0. if

1 then 0. if

11


Network energy : a stable states have a low energy

This value is called the "energy" because the definition ensures that when units are randomly chosen to update, the energy E will either lower in value or stay the same.

Ni

iiNji

ijij sThresholdssWE *..2

1

,

12

Hopfield network modelShortcomings of HNs

• Training patterns can be at most about 14% the number of nodes in the network.

• If more patterns are used then the stored patterns become unstable; spurious stable states appear (i.e., stable states

which do not correspond with stored patterns).• Can sometimes misinterpret the corrupted

pattern.

13

Learning HNs through example• Moving onto little more complex problem described in Haykin’s

Neural Network Book• They book used N=120 neuron and trained network with 120

pixel images where each pixel was represented by one neuron.• Following 8 patterns were used to train neural network.


14

Flow Chart summarizing overall process


15

Learning HNs through example


16

Learning HNs through example

16Hopfield network model

17

Shortcomings of HNs


18

Project (exam)

1 neural networks 3. 2 hopfield network (hn) model a hopfield network is a form of recurrent...

Documents