neural networks biological analogy introduction to artificial neural networks typical architectures
TRANSCRIPT
NEURAL NETWORKS
• Biological analogy• Introduction to Artificial Neural Networks• Typical architectures
João Sousa, José Borges XI.2
Biological neuron
• Soma: body of the neuron.• Dendrites: receptors (inputs) of the neuron.• Axon: output of neuron; connected to dendrites of other
neurons via synapses.• Synapses: transfer of information between neurons
(electrical-chemical-electrical).
João Sousa, José Borges XI.3
Neural networks
• Biological neural networks• Neuron switching time: 0.001 second• Number of neurons: 1010
• Connections per neuron (synapses): 104,5
• Recognition time: 0.1 s
para
llel c
ompu
tati
on
• Artificial neural networks• Weighted connections amongst units• Highly parallel, distributed process• Emphasis on tuning weights automatically
João Sousa, José Borges XI.4
Artificial Neural Networks
• Artificial Neuron
Threshold function
Piece-wise Linear Sigmoidal function
João Sousa, José Borges XI.5
Use of Artificial Neural Networks
• Input is high-dimensional • Output is multidimensional• Mathematical form of system is unknown• Interpretability of identified model is unimportant
Biological neural network
Artificial neural network
Soma Neuron
Dendrite Input
Axon Output
Synapse Weight
• Applications• Pattern recognition
• Classification
• Prediction
• Modeling
João Sousa, José Borges XI.6
Architectures of typical ANN
Out
puts
igna
ls
• Feedforward ANN
João Sousa, José Borges XI.7
Architectures of typical ANN
• Recurrent ANN
ADAPTIVE NETWORKS
• Adaptive ANN• Network Classification• Backpropagation
João Sousa, José Borges XI.9
Adaptive (neural) networks
• Massively connected computational units inspired by the working of the human brain
• Provide a mathematical model for biological neural networks (brains)
• Characteristics:• learning from examples• adaptive and fault tolerant• robust for fulfilling complex tasks
João Sousa, José Borges XI.10
Network classification
• Learning methods: supervised, unsupervised
• Architectures: feedforward, recurrent
• Output types: binary, continuous
• Node types: uniform, hybrid
• Implementations: software, hardware
• Connection weights: adjustable, hard-wired
• Inspirations: biological, psychological
João Sousa, José Borges XI.11
Adaptive network
• Nodes can be static or parametric• Network can consist of heterogeneous nodes• Links do not have parameters associated• Node functions are differentiable except at a finite number
of points
adaptive nodes
x1
x2
4
5
36
7 9
8
Input layer Layer 1 Layer 2 Output layer
x8
x9
fixed nodes
João Sousa, José Borges XI.12
Calculating with a network
),( axfy
x f y
x
a
f y
),(),,( ayhvaxgu
x g u
y h va
a
y h v
x g u
João Sousa, José Borges XI.13
Backpropagation learning rule
• Simple gradient descent applied to layered networks• An overall error measure is minimized for P data
points and L layers
2
,11 1
LNPP
p k L kpp k
E E d x
change inparameter
change inoutputs of nodes
containing
change innetwork'soutputs
change inerror measure
• Derivative information propagated by the use of chain rule,
João Sousa, José Borges XI.14
Ordered vs. partial derivatives
y
x
f
g z )(
),(
xfy
yxgz
( , )z g x y
x x
x
xf
y
yxg
x
yxg
x
xfxg
x
z
xfyxfy
)(),(),(
))(,(
)()(
partial derivative
ordered derivative
João Sousa, José Borges XI.15
BP for feedforward networks
• Define an error signal at each node
iL
p
iL
piL x
E
x
E
,,,
output node
il
mlN
mml
il
mlN
m ml
p
iL
piL x
f
x
f
x
E
x
E ll
,
,1
1,1
,
,1
1 ,1,,
11
hidden layer node
il
ilil
il
pp ff
x
EE ,,
,
,
P
p
pEE
1
João Sousa, José Borges XI.16
Error propagation network
x1
x2
4
5
36
7 9
8 x8
x9
1
24
5
36
7 9
8 8
9
w83
w97w52 w75
w31
6
99
6
88
6
9
96
8
866 x
f
x
f
x
f
x
E
x
f
x
E
x
E ppp