Neural Network

Hopfield model

Kim, Il Joong

Contents

1. Neural network: Introduction① Definition & Application② Network architectures③ Learning processes (Training)

2. Hopfield model① Summary of model② Example③ Limitations

3. Hopfield pattern recognition on a scale-free neural network

Definition of Neural Network A massively parallel system made up of simple process

ing units and dense interconnections, which has a natural propensity for storing experien-tial knowledge and making it available for use.

Interconnection strengths, known as synaptic weights, are used to store the acquired knowledge.

=> Learning process.

Application of Neural Network Patterns-pattern mapping, pattern

completion, pattern classification

Image Analysis Speech Analysis & Generation Financial Analysis Diagnosis Automated Control

Network architectures Single-layer feedforward network

Network architectures Multilayer feedforward network

Network architectures Recurrent network

Learning processes (training)

Error-correction learning Memory-based learning Hebbian learning Competitive learning Boltzmann learning

Hebbian learning process If two neurons on either side of a synapse connection are

activated simultaneously, then the strength of that synapse is increased. If two neurons on either side of a synapse are activated

asynchronously, then the strength of that synapse is weakened or

eliminated.

Hopfield model

N processing units (binary) Fully(Infinitely) connected : N(N-1) connections Single-layer(no hidden layer) Recurrent(feedback) network : No self-feedback loof

Network architecture

Hopfield model Learning process

Let denote a known set of N-dim. memories.

M ,,,, 321

)(1

1

MN

WM

T

Hopfield model Inputting and updating

Let denote an unknown N-dimensional input vector.probe

Update asynchronously (i.e., randomly and one at a time) according to the rule

Hopfield model Convergence and Outputting

Repeat updating until the state vector remains unchanged. Let denote the fixed point (stable state).

fixedX

fixedXY Associated memories

Memory vectors are states that corresponds to minimum E.

Any input vector converges to the stored memory vector that is most similar or most accessible to the input.

j

jii

ijji xxE 2

1

ji

iijijjjj xxnEnEE

2

1)()1(

M ,,,, 321

Hopfield model N=3 example

Let (1,-1,1), (-1,1,-1) denote the stored memories. (M=2)

022

202

220

3

1W

Limitations of Hopfield model① The stored memories are not always stable.

② There may be stable states that were not the stored memories. (Spurious states)

The signal-to-noise ratio:

for large M.

The quality of memory recall breaks down at M=0.14N

M

N

Limitations of Hopfield model③ Stable state may not be the state that is most

similar to the input state.

On a scale-free neural network Network architecture: the BA scale-free network

A small core of m nodes. (fully connected) N (≫m) nodes are added.

Total N + m processing units. Total Nm connections. (for 1≪m≪N)

On a scale-free neural network Hopfield pattern recognition

Stored P different patterns: Input pattern: 10% reversal of ( =0.8) Output pattern: The quality of recognition: overlap

),,2,1( Pi

1i

iS

i

iiSN11

On a scale-free neural network Small m : N=10000, m=2,3,5

On a scale-free neural network Large m : N+m=10000, P=10,100,1000

On a scale-free neural network Comparison with a fully connected network (m=N)

For small m, low quality of recognition. For 1≪m≪N, good quality of recognition. Gain a factor N/m>>1 in the computer memory and time. A gradual decrease of quality of recognition.

References

D. Stauffer et al., http://xxx.lanl.gov/abs/cond-mat/0212601 (2002)

A. S. Mikhailov, Foundations of Synergetics 1, Springer-Verlag Berlin Heidelberg (1990)

John Hertz et al., Introduction to the theory of neural computation, Addison-Wesley (1991)

Judith E. Dayhoff, Neural Network Architectures, Van Nostrand Reinhold (1990)

S. Haykin, Neural Networks, Prentice-Hall (1999)