neural network architectures and learning algorithms author : bogdan m. wilamowski source : ieee...
TRANSCRIPT
1
Neural network architectures and learning algorithms
Author : Bogdan M. WilamowskiSource : IEEE INDUSTRIAL ELECTRONICS MAGAZINEDate : 2011/11/22Presenter : 林哲緯
2
Outline
• Neural Architectures• Parity-N Problem• Suitable Architectures• Use Minimum Network Size• Conclusion
3
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
4
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
5
Neural Architectures
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
6
error back propagation(EBP) algorithm
• error back propagation(EBP) algorithm– multilayer perceptron (MLP)
Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)
7
multilayer perceptron (MLP)
Neural network architectures and learning algorithms, Wilamowski, B.M.
MLP-type architecture 3-3-4-1(without connections across layers)
8
neuron by neuron(NBN) algorithm
• neuron by neuron(NBN) algorithm– bridged multilayer perceptron (BMLP)– fully connected cascade (FCC)
Neural network architectures and learning algorithms, Wilamowski, B.M.
arbitrarily connected network
9
neuron by neuron(NBN) algorithm
• Levenberg–Marquardt(LM) algorithm– Improve nonlinear function of least square– Forward & Backward Computation
• Jacobian Matrix– Forward-Only Computation
10
bridged multilayer perceptron (BMLP)
Neural network architectures and learning algorithms, Wilamowski, B.M.
BMLP architecture 3=3=4=1(with connections across layers marked by dotted lines)
11
fully connected cascade (FCC)
Neural network architectures and learning algorithms, Wilamowski, B.M.
Bipolar neural network for parity-8 problem in a FCC architecture
12
Outline
• Neural Architectures• Parity-N Problem• Suitable Architectures• Use Minimum Network Size• Conclusion
13
parity-8 problem
MLP 8*9 + 9 = 81 weights BMLP 4*9 + 8 + 4 + 1 = 49 weightsNeural network architectures and learning algorithms, Wilamowski, B.M.
14
parity-8 problem
9 + 10 + 11 + 12 = 42 weights
Neural network architectures and learning algorithms, Wilamowski, B.M.
15
parity-17 problem
• MLP architecture needs 18 neurons• BMLP architecture with connections across
hidden layers needs 9 neurons• FCC architecture needs only 5 neurons
16
parity-N problem
• MLP architectures
• BMLP architectures
• FCC architectures
nn = neuronsnw = weights
Neural network architectures and learning algorithms, Wilamowski, B.M.
17
Outline
• Neural Architectures• Parity-N Problem• Suitable Architectures• Use Minimum Network Size• Conclusion
18
suitable architectures
• For a limited number of neurons, FCC neural networks are the most powerful architectures, but this does not mean that they are the only suitable architectures
19
suitable architectures
• if the two weights marked by red dotted lines– signal has to be propagated by fewer layers
Neural network architectures and learning algorithms, Wilamowski, B.M.
20
Outline
• Neural Architectures• Parity-N Problem• Suitable Architectures• Use Minimum Network Size• Conclusion
21
Use Minimum Network Size
• receive a close-to-optimum answer for all patterns that were never used in training
• generalization abilities
22
Case Study
Neural network architectures and learning algorithms, Wilamowski, B.M.
TSK fuzzy controller:(a) Required control surface(b) 8*6 = 48 defuzzification rules
TSK fuzzy controller:(a) Trapezoidal membership functions(b) Triangular membership functions
23
Case Study
Neural network architectures and learning algorithms, Wilamowski, B.M.
(a) 3 neurons in cascade (12 weights), training error = 0.21049(b) 4 neurons in cascade (18 weights), training error = 0.049061
(a) 5 neurons in cascade (25 weights), training error = 0.023973(b) 8 neurons in cascade (52 weights), training error = 1.118E-005
24
time complexity
NBN algorithm can train neural networks 1,000 times faster than the EBP algorithm.
Neural network architectures and learning algorithms, Wilamowski, B.M.
(a) EBP algorithm, average solution time of 4.2s, and average 4188.3 iterations(b) NBN algorithm, average solution time of 2.4ms , and average 5.73 iterations
25
two-spiral problem
Neural network architectures and learning algorithms, Wilamowski, B.M.
NBN algorithm using FCC architecture244 iterations and 0.913s
EBP algorithm using FCC architecture30,8225 iterations and 342.7s
26
Outline
• Neural Architectures• Parity-N Problem• Suitable Architectures• Use Minimum Network Size• Conclusion