MODULE-V

Hopfield Nets.

Module 5

Hopfield nets: Recurrent networks - stability - Associative

memory-applications - Thermo dynamic systems -

Statistical Hopfield networks -Bidirectional associative

memories - Continuous BAM - Adaptive resonance theory -

Architecture classification - implementation.

•Feed forward networkthere is no feed back for feed forward network.eg:-Perceptron ,Back Propagation networks etc.

Recurrent network

Recurrent network is a network with feedback.

Some of its outputs are connected to its input.

Its a discrete time network.

Delay Block

The delay output is the input delayed by one step.

a(t) =u(t-1)

The initial condition is always taken through another

connection

Basic Blocks Used to build Recurrent Networks

Intergrator Block

)0()()(0

aduta

Recurrent Networks

In recurrent network its output fed back to inputs.

After applying a new input out put is again recalculated

and fed back to input.

The process is repeated

For stable network output change become

smaller and smaller for successive iteration.

For unstable network that lead to chaotic

condition.

Recurrent Network Configuration

Recurrent Network Configuration

Check the stability and output of the pattern recognition problem of apple and orange using Hopfield networks. Take weights and bias as

9.0

0

9.0

2.000

02.10

002.0

b

w

Assume the sensor output is

1

1

1

a

1

1

1

)3(,

1

1

1

)2(

1

1

7.0

)1(,

1

1

1

)0(

aa

aa

Binary Hopfield Network

Activation function used is a simple threshold functionOutput is 1 if the weighted sum is greater than a threshold otherwise 0.

.

ji

jij INwijOUTNET

OUTj=1 if NETj >Tj

OUTj=0 if NETj <TjIt remains unchanged if NETj=Tj

State of the network•The set of current value of OUT.

Geometrical Visualization of State

If there are two neurons in the output layer, it can be represented as a vertex of a square.

(0 1)

(0,0) (1 0)

(1 1)

For a three neuron system it can be represented as vertex of a cube.

(0 0 0)

(0 0 1)(0 1 0)

For an n neuron setup it can be represented by an n dimensional hypercube.

When an input is applied the output changes from vertex to

vertex until it stabilizes.

The stable vertex is determined by the network weights,

current value of inputs , threshold value.

If the input vector is partially incorrect or incomplete, the

output settle to the vertex closest to the one desired.

Hopfield Network Stability

Cohen and Grossberg shown that the recurrent networks are

stable stable when matrix is symmetrical with zeros on its

main diagonal; i.e. wij=wji,wii=0 for all i.

Hopfield proposed a function ,called Energy Function ,based on Lyapunov stability function.

i j i

iijiij ssswsV 2

1)(

i j i

iijiij ssswsV 2

1)(

Wij weight between I and j

Si Output from neuron II Threshold value

i

jji j j

jjjiij OUTIOUTTOUTOUTwsE2

1)(

The change in Energy due to a change in the state of neuron j is

The energy function for hopefield neuron can be stated as

jjj

jjjji

iij

OUTTNET

OUTTIOUTwE

][

])([

If NET >Threshold the out put of neuron j changes

in the +ve direction. Hence Energy Change is –Ve or

No change.

If NET <Threshold the out put of neuron j changes

in the -ve direction or zero. Hence Energy Change is –

Ve or No change.

If NET =Threshold the out put of neuron j doesn't

change.. Hence Energy Change is zero.

According to Hopfield energy Criterion (Lyapnov) If any

change in the state of neuron will either reduce the energy

or remain unchanged ,because of the continuous downward

trend ,eventually it will find a minimum and stop and

become stable.

ASSOCIATIVE MEMORY

Human memory operates in an associative manner.

A hint, a few related sensory experience will reproduce

the entire stored memory.

A recurrent network forms an associative memory.

A portion of the desired data is supplied and full data

in the memory is returned.

To make an associative memory using recurrent

network, weight must be selected to produce energy

minima at the desired vertex of the unit hypercube.

Task of an associative memory is to learn Q pair of

prototype input/output vectors.

{p1,t1},{p2,t2},{p3,q3}……….{pQ,tQ}

I.e. if the network receives an input p=pq,then it should

produce an output a= tq .,for q= 1 2 3…….Q.

In addition to this if the input is changed slightly (ie

P=Pq+ε),then the output must also change by a slight

quantity.(a=tp+δ)

Associative memory is called content Addressable

Memory for Hopfield net.

Its because it retrieves stored memories on the basis of a

part of the content.

This is in contrast to the standard computer memories,

where we retrieve data based on their address.

•Hopfield developed an associative memory in which the

output are continuous.

•The memories are encoded as binary vectors and are

stored in the weights according to the formula

''1

)*(mtod

jdid OUTOUTwij

Where m=no.of desired memories.d=no. of desired memory elements.,OUTij ijth element.

Hopfield-Boltzmann Thermodynamic System(Simulated Annealing of Hopfield Net/Statistical Hopfield Net)

Hopfield net is also a minimum seeking Network

Hopfield network also suffers the problem of local

minima and may trapped there rather than reaching global

minima.

This problem can be solved by combing Hopfield net with

Boltzmann machine.

The probability of weight change is calculated by the

amount by which the NET output of the neuron exceeds the

threshold.

Ek=NETk-Tk

)]exp(1[

1

T

EP

kk

•Initially set T at high value•Calculate the probability •Set the state of neuron to one if probability is equal to pk, otherwise to zero.•Reduce T gradually ,repeat iterations till reaching the required minimum

Applications

Analog to Digital Converter [Hopfield and Tank,1985]

•Hopfield developed a recurrent network to produce a four-

bit analog to digital converter.

•Amplifiers serving as neurons.

•Resistors represents the weight.

•To satisfy the stability constraint no resistor is connects the

a neurons output to its own.

•The out put weights are made symmetrical.

•The application assumes a threshold function as activation

function.

•The outputs are changed at the beginning of every epoch.

•The output is set one/zero according to the net function at

the beginning of the epoch.

•The object of this ANN solution is to select the weights

(Resistors)

The energy function is defined as

jjj

j

jj

j OUTOUTOUTXE )]1()[2(]2[2

1 122

The first term in bracket is minimized when output is close to input.Similarly second term will reach a minimum value when outputs are ones or zeros.

Now compare this equation with Hopfield- Lyapunov energy equation and find the weights required.

Example1

An ADC output is represented by the equation

2

1

)1(2i

i

iiay y the input given,ai is the output

Apply Hopfield technique to find the weights of the recurrent network.

Answer

Energy equation of the ADC set up is

)]1(2[2

1]2[

2

1 )1(2)1( ii

i

i

ii aaayE

)1(2)1(22)1(

)1(2)1(22)1(

22)]1(2

222]2[

ii

iiii

i

iji

ii

i

ii

aaaa

aaayyay

Substitute these functions in energy equation.

)1(2)1(22)1(2)1(2 22]222[2

1 ii

ii

iji

ii aaaaayyE

abWaaaV TT 2

1)(

Now arrange E to the following form

222

1

02

20

y

yb

w

Bidirectional Associative Memories (BAM)

Human memory is of associative in nature.

Hopfield net was an auto associative memory.

The bidirectional associative memory have

heteroassociative.

BAM has all the properties of Hopfield associative memory.Its more closer to human brain characteristic.

BAM STRUCTURE

BAM stores a a set of pattern associations by summing bipolar correlation matrices.

BAM

Bipolar BAM Binary BAM Continuous BAM

BAM has two layer of neurons.

The net iterates by sending signals back and forth

between two layers until all neuron reach equilibrium.

The weight matrix for the signal from one layer to the

other layer is the transpose of the other.

From X to Y W

From Y to X WT

Y1

Y2

Y3

X1

X2

X3

W11

W12

Structure of BAM

WWT

Layer 1 Layer 2

Setting Weights

The weight matrix to store a set of input and target vectors S(p):t(p),p=1,2,3……..P

S(P)=(s1(p), s2(p), s3(p),………, sn(p))t(p)= (t1(p),t2(p),t3(p)………..tm(p))

Can be determined by Hebbs rule. for Binary network

P

ij Psjpsiw1

)]1)(2)(1)(2[(

For bipolar input vectors

P

jiij Ptpsw1

)](*)([(

Activation FunctionAppropriate Threshold functions are used as activation function for discrete type BAM

0

00

01

netifyj

netif

netif

y j

For Binary Function

0

00

01

netifx

netif

netif

x

j

i

Activation Function

netify

netif

netif

y

netifx

netif

netif

x

j

i

j

i

1

1

1

1For bipolar function

Training Algorithm

•Initialize weights to store a set of P vectors.

•Present input pattern to x to X layer and y to Y layer.

•If activation not converged, update activations (outputs)

•Ie compute ynet=Σxiwij.Find y=f(ynet).

• compute xnet=Σyiwij.Find x=f(xnet).

•If X,Y reached equilibrium stop,otherwise continue.

Continuous BAM

A continuous BAM transforms Output continuously into output in the rage [0,1] using sigmoid function as activation function.

P

jiij Ptpsw1

)]1)(2)(1)(2[(

Activation function is )exp1(

1)( NET

Example:Use BAM to store the following three pair of vectors X1=(1 -1 -1 ) Y1=(1 -1) X2=(-1 1 -1 ) Y2=(-1 1) X1=(-1 -1 1 ) Y3=(1 -1)Examine the network with the vector(-1 -1 -1)

]11[

11

33

11

]111[

)111(

11

33

11

11

1

1

1

11

1

1

1

11

1

1

1332211

XWY

XIF

W

YXYXYXW TTT

The weight vector

Adaptive Resonance Theory

The human brain extract vital information from flood

of information

Human memory retain the stability of existing data,

while storing the new.

Conventional ANN failed to overcome in this temporal

instability problem.

Carpenter and Grossberg [in 1987 & 1990] introduced a

Adaptive Resonance Theory Network to solve this

problem

ART ARCHITECTURE

RecognitionLayer

Comparison Layer

Gain 2

Gain 1 ResetVigilance

ART network maintain the plasticity required to learn the new patterns.

RecognitionLayer

Comparison Layer

Gain 2

Gain 1 Reset

ART STRUCTURE

Vigilance

X

RC

-

+

-+

+

+

Comparison Layer

G

X

CR

T

The comparison layer receives the binary input vector X

Initially it is passed through it as C.

Later the binary vector R produced by recognition layer

is compared and C is modified.

Each neuron in the comparison layer receives 3 inputs

A component xi from input vector X.

The feed back signal pi from recognition layer.

A gain signal from Gain 1.

To output a one at least two input out of three must be

high (Two third Policy).

Initially gain 1 is set to one and all components of R are

set to zero.

Hence vector C is identical to input vector X initially.

Recognition Layer

The recognition layer helps to classify the input vectors.

In this layer the neuron with a weight vector best

matching with the input vector fires.; all others are

inhibited. Each recognition layer neuron compute a dot

product between its weight and incoming C vector.

The neuron with largest output wins while inhibiting the

others.

GAIN 2G2 is set to 1 if one of the components of the input vector is one.Ie G2 is the logical OR of input vector components,xi

Gain 1

The output of gain 1 is set to1 if any components of X is one., but , if any component of R is 1 then it is forced to zero.

Reset module

It measures the similarity between the vectors X and C.If it differ more than the vigilance parameter ,a reset signal is send to diasable the firing neuron in the recognition layer.