hopfield network

HOPFIELD NETWORKPRESENTED BY :Ankita PandeyME ECE - 112604

04/08/2023 PRESENTATION ON HOPFIELD NETWORK 2

CONTENTIntroduction

Properties of Hopfield network

Hopfield network derivation

Hopfield network example

Applications

References


INTRODUCTION

• as a network with associative memory • can be used for different pattern recognition

problems.

Hopfield neural network is proposed by John Hopfield in 1982 can be seen

• Means it has only one layer, with each neuron connected to every other neuron

It is a fully connected, single layer auto associative network

All the neurons act as input and output.


INTRODUCTION

The Hopfield network(model) consists of a set of neurons and

corresponding set of unit delays, forming a multiple loop feedback system as

shown in fig.


INTRODUCTION

The number of feedback loops is equal to the number of neurons.

• no self feedback in the network.

Basically, the output of the neuron is feedback, via a unit delay element, to each of the other neurons in the network.


PROPERTIES OF HOPFIELD NETWORK

1 •A recurrent network with all nodes connected to all other nodes.

2 •Nodes have binary outputs (either 0,1 or -1,1).

3 •Weights between the nodes are symmetric .

4 •No connection from a node to itself is allowed.

5 •Nodes are updated asynchronously ( i.e. nodes are selected at random).

6 •The network has no hidden nodes or layer.


Consider the noiseless, dynamical model of the neuron shown in fig. 1

The synaptic weights represents conductance’s.

The respective inputs represents the potentials, N is number of inputs.

These inputs are applied to a current summing junction characterized as follows:• Low input resistance.• Unity current gain.• High output resistance.

txtxtx n,......, 21

HOPFIELD NETWORK

jnjj www ,......, 21


Σ

id

iy

ie

ADDITIVE MODEL OF A NEURON

NEURAL NETWORK

MODEL


dttdv

CRtv j

jj

j

HOPFIELD NETWORK

i

N

iiji Itxw

1

The total current flowing toward the input node of the nonlinear element(activation function) is:

Total current flowing away from the input node of the nonlinear element as follows:

• Where first term due to leakage resistance • And second term due to leakage

capacitance.


• By applying KCL to the input node of the nonlinearity , we get

………..(1)

• The capacitive term add dynamics to the model of a neuron.• Output of the neuron j determined by using the non linear relation

• The RC model described by the eq. (1) is referred to the additive model

)(tvtx jj

HOPFIELD NETWORK

N

ijiji

j

jjj Itxw

Rtv

dttdv

C1

dttdv

C jj


HOPFIELD NETWORKA feature of Additive model is that the signal xᵢ(t) applied to the neuron j by adjoining neuron i• is a slowly varying function of the time t.

Thus, a recurrent network consisting of an interconnection of N neurons, • each one of which is assumed to have the same

mathematical model described by the equation :

• •

…….(2)

,1

N

ijiji

j

jjj Itxw

Rtv

dttdv

C

Nj ,.....,2,1


HOPFIELD NETWORK

ijji ww

xv i1

Now, we use eq (2) which is based on the additive model of the neuron.

Assumptions:

• The matrix of synaptic weights is symmetric, as shown by:

• for all i and j.• Each neuron has a nonlinear activation of its own,

hence use of in eq.(2)• The inverse of the nonlinear activation function

exists, so we can write•

……….(3)

i


HOPFIELD NETWORK• Let the sigmoid function be defined by the

hyperbolic tangent function

• Which has slope of .• refers as the gain of neuron i.

The inverse I/O relation of eq.(3) may be written as ………..(4)

va

vavavxi

iii

exp1exp1

2tanh

xx

axv

i 11log11

vi

ia2/ia


HOPFIELD NETWORK• Standard form of the inverse I/O relation for a neuron of

unity gain is:

• We can rewrite the eq. (4) in terms of standard relation as x

ax

ii

11 1

xxx

11log1


Plot of (a) Sigmoidal Nonlinearity and (b) its inverse

UNKNOWN

SYSTEMf(.)

Σix

iy

ie

Model Output

Error

(a) (b)


HOPFIELD NETWORK• The energy function of the Hopfield network is defined

by:

• Differentiating E w.r.t. time , we get

• by putting the value in parentheses from eq.2, we get …………..(5)

dtdx

IRv

xwdtdE j

N

j

N

ij

j

jiji

1 1

N

j

N

j

x N

jjjj

jjiji

N

i

j

xIdxxR

xxwE1 1 0 1

1

1

121

dtdx

dtdv

CdtdE j

N

j

jj

1


HOPFIELD NETWORK

N

j j

jjjj

jN

j

jjj

dxxd

dtdx

C

dtdx

dtxd

CdtdE

1

12

1

1

xv i1

jv• The inverse relation that defines in terms of is

• By using above relation in eq. (5), we have

…………..(6)

• From fig. (b) we see that the inverse I/O relation is monotonically increasing function of the output Therefore,

jx


HOPFIELD NETWORK

0dtdE

02

dtdx j

Also, for all . Hence all the factors that make up the sum on R.H.S. of eq(6) are non-negative.Thus the energy function E defined as

jx


HOPFIELD NETWORK

• The energy function E is a Lyapunov funtion of the continuous Hopfield model.

• The model is stable in accordance with Lyapunov’s Theorem 1.

We may make the following

two statements:

• Which seeks the minima of the energy function E and comes to stop at fixed points.

The time evolution of the continuous Hopfield model

described by the system of nonlinear

first order differential equations represents the trajectory in the

state space


HOPFIELD NETWORK

• From eq.(6) the derivative vanishes only if

for all j.

• Thus we can say,

expect at fixed point ………(7)

• The eq.(7) forms the basis for following theorem• The energy function E of a Hopfield network is a

monotonically decreasing function of time.

0

dttdx j

dtdE

0dtdE


HOPFIELD NETWORK EXAMPLE

Neuron 1 (N1)

Neuron 2 (N2)

Neuron 3 (N3)

Neuron 4 (N4)

Neuron 1 (N1)

(N/A) N2->N1

N3->N1

N4->N1

Neuron 2 (N2)

N1->N2 (N/A) N3-

>N2N4->N2

Neuron 3 (N3)

N1->N3

N2->N3 (N/A) N4-

>N3

Neuron 4 (N4)

N1->N4

N2->N4

N3->N4 (N/A)

Connection of Hopfield Neural Network

A Hopfield Neural network:


• The connection weights put into this array, also called a weight matrix, allow the neural network to recall certain patterns when presented. • For example, the values shown in Table below show the correct values to use to recall the patterns 0101 .

HOPFIELD NETWORK EXAMPLE

Neuron 1 (N1)

Neuron 2 (N2)

Neuron 3 (N3)

Neuron 4 (N4)

Neuron 1 (N1) 0 -1 1 -1

Neuron 2 (N2) -1 0 -1 1

Neuron 3 (N3) 1 -1 0 -1

Neuron 4 (N4) -1 1 -1 0

Weight Matrix used to recall 0101.


Calculating The Weight Matrix

Step 1: Convert 0101 to bipolar

• Bipolar is nothing more than a way to represent binary values as –1’s and 1’s rather than zero and 1’s. • To convert 0101 to bipolar we convert all of the zeros to –1’s. This results in:• 0 = -1 1 = 1 0 = -1 1 = 1• The final result is the array (-1, 1, -1, 1)


Calculating The Weight Matrix Step 2: Multiply (-1, 1, -1, 1) by its Inverse

For this step we will consider -1, 1, -1, 1 to be a matrix.

Taking the inverse of this matrix we have.

Now, multiply these two matrices-1 X (-1) = 1 1 X (-1) = -1 -1 X (-1) = 1 1 X (-1) = -1

-1 X 1 = -1 1 X 1 = 1 -1 X 1 = -1 1 X 1 = 1

-1 X (-1) = 1 1 X (-1) = -1 -1 X (-1) = 1 1 X (-1) = -1

-1 X 1 = -1 1 X 1 = 1 -1 X 1 = -1 1 X 1 = 1


Calculating The Weight Matrix• And the matrix is:

Step 3: Set the Northwest diagonal to zero

• The reason behind this is, in Hopfield networks do not have their neurons connected to themselves.

• So positions [1][1], [2][2], [3][3] and [4][4] in our two dimensional array or matrix, get set to zero. This results in the weight matrix for the bit pattern 0101.


Recalling Pattern• To do this we present each input neuron, with the pattern. Each neuron will activate based upon the input pattern. • For example, when neuron 1 is presented with 0101 its activation will be the sum of all weights that have a 1 in input pattern. • The activation of each neuron is:

The final output vector then (-2,1,-2,1)

a b c d a+b+c+d

N1 0 -1 0 -1 -2N2 0 1 0 0 1N3 0 -1 0 -1 -2N4 0 1 0 0 1


Recalling Pattern

So the following neurons would fire.

N1 activation is –2, would not fire (0)N2 activation is 1, would fire (1)N3 activation is –2, would not fire(0)N4 activation is 1 would fire (1).

The threshold usually used for a Hopfield network, is any value greater than zero.

Now, Threshold value determines what range of values will cause the neuron to fire.


Recalling Pattern

An auto associative neural network, such as a Hopfield networkWill echo a pattern back if the pattern is recognized.

The final binary output from the Hopfield network would be 0101.

This is the same as the input pattern.

We assign a binary 1 to all neurons that fired, and a binary 0 to all neurons that do not fire.


APPLICATION

Image Detection and Recognition

Enhancing X-Ray Images

In Medical Image Restoration


References

• Jacek M. Zurada, Introduction To Artificial Neural Systems (10th edition)• Simon Haykin, Neural Networks (2nd edition)• Satish Kumar, Neural Networks; A Classroom Approach (2nd Edition)• http://www.learnartificialneuralnetworks.com/hopfield.html• http://www.heatonresearch.com/articles/2/page6.html• http://www.thebigblob.com/hopfield-network/#associative-memory•http://www.dsi.unive.it/~pelillo/Didattica/RetiNeurali/Introduction_To_ANN_lesson_6.pdf

.• http://en.wikipedia.org/wiki/Hopfield_network.

http://www.learnartificialneuralnetworks.com/hopfield.html

http://www.learnartificialneuralnetworks.com/hopfield.html

http://www.heatonresearch.com/articles/2/page6.html

http://www.thebigblob.com/hopfield-network/

http://www.thebigblob.com/hopfield-network/

http://www.dsi.unive.it/~pelillo/Didattica/RetiNeurali/Introduction_To_ANN_lesson_6.pdf

http://www.dsi.unive.it/~pelillo/Didattica/RetiNeurali/Introduction_To_ANN_lesson_6.pdf

http://en.wikipedia.org/wiki/Hopfield_network


THANK

YOU.

hopfield network

Education

neural networks

10312012 presentation

energy function

nonlinear

additive model

term due

2nd edition

hopfield network