UNIT-IINTRODUCTION TO
ARTIFICIAL NEURAL NETWORK
EC0054 NEURAL NETWORK AND FUZZY LOGIC
ELEMENTARY NEURO- PHYSIOLOGY
Neuron:A neuron nerve cell is an electricallyexcitable cell that processes and transmits information by electrical and chemical signaling. Chemical signaling occurs via synapses, specialized connections with other cells. Neurons connect to each other to form networks.
PARTS OF THE NEURON Cell Body
Contains the nucleus Dendrites
Receptive regions; transmit impulse to cell body
Short, often highly branched May be modified to form receptors
Axons Transmit impulses away from cell body Axon hillock; trigger zone
Where action potentials first develop Presynaptic terminals (terminal
boutons) Contain neurotransmitter substance (NT) Release of NT stimulates impulse in
next neuronBundles of axons form nerves
ELECTRICAL SIGNALS Neurons produce electrical signals called
action potentials ( = nerve impulse) Nerve impulses transfer information from
one part of body to anothere.g., receptor to CNS or CNS to effector
Electrical properties result from ionic concentration differences across plasma
membrane permeability of membrane
Single Neuron Physiology
Resting Potential
Inhibitory & Exitatory Action Potential
Synapse or synaptic junction
axon of presynapticneuron
dendrite ofpostsynapticneuron
bipolar.about.com/library
Neural circuits and computation Networks either converge or diverge. Divergences is sending nerve impulse,
converging is receiving inputs. Some have feedback – excitatory is positive
feedback and inhibitory is negative feedback.
The working and interconnection of these simple networks, depicts the function the complex human brain – was modeled as a computational unit by McColloch and Pitts in 1943.
Assumptions of the McColloch – Pitts model –
1. The activity of a neuron is all or none process.2. certain number of synapses (>1) must be
excited within a latent period , for a neuron to be excited.
3. The only delay is the synaptic delay.4. An active inhibitory synapse prevents
excitation.5. The structure of the network does not change
over time.
Mc-Pitts Model of Neural Networks
Fig. A and Fig. B. are concepts of divergence and convergence.
Fig. B, C and D illustrate concepts of feedback.
According to assumptions of McColloch-Pitts, a neuron has binary behaviour (activity is all or none) – either ON or OFF.
This can be represented in propositional logic as - a predicate Ni(t) that denotes the assertion that ith neuron fires at time t.
Conversely, ~Ni(t) denotes that ith neuron does not fire at time t.
MCCULLOCH-PITTS NEURON (1943) Inputs - (+1) or (-1) Activation function
Multiply inputs with corresponding weights & sum it Output – (+1) if output is positive else (-1)
EA C461 Artificial Intelligence 16
MCCULLOCH-PITTS NEURON (1943)
FEATURES OF MCCULLOCH-PITTS MODEL Allows binary 0,1 states only Operates under a discrete-time
assumption Weights and the neurons’ thresholds are
fixed in the model and no interaction among network neurons
Just a primitive model NOT ACCURATE.
HEBBIAN LEARNING Biological neurons – learn over time –
supervised learning. Hebb – in his theory of Organizational
behaviour – states that – when an axon of a cell A is near enough to excite a cell B and repeatedly or persistently takes part in firing it, some growth process or metabolic change takes place in one or both of the cells such that A’s efficiency (as one of the cells firing B) is increased.
Based on behavior of dogs during lunch timing.
Dogs were initially trained to see the food to cause salivation.
Later bell was rung along with the sight of food to cause salivation.
Finally, sight of food was removed and only bell was rung – dogs still salivated…
So sound sense (second trigger) overruled sight sense (first trgieer) to cause salivation..
How does neuron learn ?
ESSENCE OF Hebbian learning (1949) Speculated that the learning occur by the
modification of synapsesRepeated firing across a synapse increase it’s
sensitivity & hence the future likelihood of firing If a particular stimulus repeatedly causes activity in
a group of cells, those become strongly associated In future similar stimuli would tend to excite the same
neural pathways recognition of the stimuli
Feedforward unsupervised learning
FROM NEURONS TO ARTIFICIAL NEURAL
NETWORKS (ANS)
Structure of Processing Element
WEIGHTS Each neuron is connected to every other
neuron by means of directed links Links are associated with weights Weights contain information about the
input signal and is represented as a matrix
Weight matrix also called connection matrix
WEIGHT MATRIXW=
1
2
3
.
.
.
.
.
T
T
T
T
n
www
w
=
11 12 13 1
21 22 23 2
1 2 3
...
...
..................
...................
...
m
m
n n n nm
w w w ww w w w
w w w w
The output is a function of activation value.
xi = f(ai) The activation value is fn. of
activation value at previous time instant and the net of the PE at that instant.
ai = F(ai(t-1), neti) =~ F(neti) So, xi = F(neti)
Is referred to activation function. Domain is set of activation values net.
Scalar product of weight and input vector
Neuron as a processing node performs the operation of summation of its weighted input – as a function, which is the learning law.
TYPES OF LEARNING LAWS OR LEARNING RULES
Perceptron Learning rule. Delta Learning Rule Widrow-Hoff Winner-Take-All
THE PERCEPTRON - PART 1 The first ANS developed by Frank
Rosenblatt in the late 1950s. to "illustrate some of the fundamental
properties of intelligent systems”. The photoperceptron is a device
that responds to optical patterns. Has - Sensory (S) area , Association (A)
area and Response (R) area
light impinges on the sensory (S) points of the retina structure.
Each S point responds in an all-or-nothing manner to the incoming light.
connections may be either excitatory or inhibitory.
Impulses generated by the S points are transmitted to the associator (A) units in the association layer.
an A unit becomes active if the sum of its inputs exceeds some threshold value.
It then produces an output.
Associator units are connected to response (R) units in the response layer.
The pattern of connectivity is again random.
R unit inhibits the A units in the complement.
each R unit inhibits the other. These factors aid in the establishment of a
single, winning R unit for each stimulus pattern appearing on the retina.
The R units give an output value of +1; otherwise, the output is —1.
Such a system classifies patterns appearing on the retina into categories, according to the number of response units in the system.
Patterns that are sufficiently similar should excite the same R unit.
So, a perceptron has to successfully distinguish between different pattern classes.
The perceptron was a learning device.
A pattern was applied to the retina, and the stimulus was propagated through the layers until a response unit was activated.
If the correct response unit was active, the output of the contributing A units was increased.
If the incorrect R unit was active, the output of the contributing A units was decreased.
Rosenblatt claims that the perceptron could classify patterns successfully in what he termed a differentiated environment.
PERCEPTRON - PART 2: by Marvin Minsky and Seymour Papert. Analysis of the perceptron in terms of its
capabilities and limitations. Perceptrons can differentiate patterns
only if the patterns are linearly separable.
Instead of probabilistic approach by Rosenblatt,they used predicate calculus in their analysis.
PERCEPTRON 2.
Echo suppression in Telephone networks
Frequency response characteristics of different Filters
NEURAL NETWORK ARCHITECTURES• Several NN have been proposed & investigated in
recent years
• Supervised versus unsupervised• Architectures (feedforward vs. recurrent)• Implementation (software vs. hardware)• Operations (biologically inspired vs. psychologically
inspired)
• In this chapter, we will focus on modeling problems with desired input-output data set, so the resulting networks must have adjustable parameters that are updated by a supervised learning rule
SAMPLE FEED FORWARD NETWORK (NO LOOPS) FFN
WeightsWeights
Weights
WjiVik
F( S wji xj
LMS LEARNING RULE 1. Apply input to Adaline input 2. Find the square error of current input Errsq(k) = (d(k) - W x(k))**2 3. Approximate Grad(ErrorSquare) by differentiating Errsq approximating average Errsq by Errsq(k) obtain -2Errsq(k)x(k) Update W: W(new) = W(old) +
2mErrsq(k)X(k) Repeat steps 1 to 4.
Structure of ADALINE
Structure of ALC(Adaptive Linear Combiner)
ALC as a transversal Filter
Use of ADALINE in solving XOR problems
MDALINE Architecture
BPN Architecture
IMAGE TO ASCII CONVERSION USING NEURAL NETWORK
Image to ASCII Conversion using Neural Network
Image to ASCII Conversion using Neural Network (Cont.d)
REVIEW QUESTIONS
What is Processing Element. How would you relate the PEs with real neurons
Define Resting Potential. What is the average refractory period of a neuron. Is it limited to a particular value. If Yes mention How?
Differentiate Resting potential and action potential
State Hebbs Learning Rule. Draw a sample memory mapping diagram by your own.
How would you factor out the weight vector from the exception value terms
What is the use of signal processing techniques in neural networks
REVIEW QUESTIONS (CONTD..)
REFERENCES
J. A. Freeman and D. M. Skapura, Neural Networks- Algorithms, Applications and Programming Techniques, Pearson Education( singapore) Pvt. Ltd., 1991.
(Chapters 1 &2) psychology.about.com/od/biopsychology/f/
neuron01.htm www.cell.com/neuron www.neurophys.com faculty.washington.edu/chudler/chnt1.html