soft computing lecture 7 multi-layer perceptrons
TRANSCRIPT
![Page 1: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/1.jpg)
Soft computing
Lecture 7
Multi-Layer perceptrons
![Page 2: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/2.jpg)
Why hidden layer is needed
![Page 3: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/3.jpg)
![Page 4: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/4.jpg)
Problem of XOR for simple perceptron
X2
X1
(0,1)
(0,0)(1,0)
(1,1)Class 1
Class 1
Class 2
Class 2
In this case it is not possible to draw descriminant line
![Page 5: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/5.jpg)
![Page 6: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/6.jpg)
![Page 7: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/7.jpg)
Minimization of error
![Page 8: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/8.jpg)
![Page 9: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/9.jpg)
Main algorithm of training
![Page 10: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/10.jpg)
Kinds of sigmoid used in perceptrons
Exponential
Rational
Hyperbolic tangent
![Page 11: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/11.jpg)
Formulas for error back propagation algorithm
Modification of weights of synapsesof jth neuron connected with ith ones,xj – state of jth neuron (output)
For output layer
For hidden layersk – number of neuron in next layerconnected with jth neuron
(1)
(2)
(3)
![Page 12: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/12.jpg)
(2), (1)
(1)
(3), (1)
(1)
![Page 13: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/13.jpg)
Example of implementation TNN=Class(TObject) public State:integer; N,NR,NOut,NH:integer; a:real; Step:real; NL:integer; // ъюы-тю шЄхЁрЎшщ яЁш юсєўхэшш S1:array[1..10000] of integer; S2:array[1..200] of real; S3:array[1..5] of real; G3:array[1..5] of real; LX,LY:array[1..10000] of integer; W1:array[1..10000,1..200] of real; W2:array[1..200,1..5] of real; W1n:array[1..10000,1..200] of real; W2n:array[1..200,1..5] of real; SymOut:array[1..5] of string[32]; procedure FormStr; procedure Learn; procedure Work; procedure Neuron(i,j:integer); end;
![Page 14: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/14.jpg)
Procedure of simulation of neuron; procedure TNN.Neuron(i,j:integer);var k:integer; Sum:real;begincase i of 1: begin if Form1.PaintBox1.Canvas.Pixels[LX[j],LY[j]]= clRed then S1[j]:=1 else S1[j]:=0; end; 2: begin Sum:=0.0; for k:=1 to NR do Sum:=Sum + S1[k]*W1[k,j]; if Sum> 0 then S2[j]:=Sum/(abs(Sum)+Net.a) else S2[j]:=0;end; 3: begin Sum:=0.0; for k:=1 to NH do Sum:=Sum + S2[k]*W2[k,j]; if Sum> 0 then S3[j]:=Sum/(abs(Sum)+Net.a) else S3[j]:=0; end; end;end;
![Page 15: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/15.jpg)
Fragment of procedure of learning For i:=1 to NR do for j:=1 to NH do begin S:=0; for k:=1 to NOut do begin if (S3[k]>0) and (S3[k]<1) then D:=S3[k]*(1-S3[k]) else D:=1; W2n[j,k]:=W2[j,k]+Step*S2[j]*(G3[k]-S3[k])*D; S:=S+D*(G3[k]-S3[k])*W2[j,k] end; if (S2[j]>0) and (S2[j]<1) then D:=S2[j]*(1-S2[j]) else D:=1; S:=S*D; W1n[i,j]:=W1[i,j]+Step*S*S1[i]; end;end;
![Page 16: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/16.jpg)
Generalization
![Page 17: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/17.jpg)
Some of the test data are now misclassified. The problem is that the network, with two hidden units, now has too much freedom and has fitted a decision surface to the training data which follows its intricacies in pattern space without extracting the underlying trends.
![Page 18: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/18.jpg)
Overfitting
![Page 19: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/19.jpg)
Local minima
![Page 20: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/20.jpg)
Two tasks solved by MLP
• Classification (recognition)– Usually binary outputs
• Regression (approximation)– Analog outputs
![Page 21: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/21.jpg)
Theorem of Kolmogorov
“Any continuous function from input to output can be implemented in a three-layer net, given sufficient number of hidden units nH, proper nonlinearities, and weights.”
![Page 22: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/22.jpg)
Advantages and disadvantages of MLP with back propagation
• Advantages:– Guarantee of possibility of solving of tasks
• Disadvantages:– Low speed of learning– Possibility of overfitting– Impossible to relearning– Selection of structure needed for solving of
concrete task is unknown
![Page 23: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/23.jpg)
Increase of speed of learning
• Preliminary processing of features before getting to inputs of percepton
• Dynamical step of learning (in begin one is large, than one is decreasing)
• Using of second derivative in formulas for modification of weights
• Using hardware implementation
![Page 24: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/24.jpg)
Fight against of overfitting
• Don’t select too small error for learning or too large number of iteration
![Page 25: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/25.jpg)
Choice of structure
• Using of constructive learning algorithms– Deleting of nodes (neurons) and links
corresponding to one– Appending new neurons if it is needed
• Using of genetic algorithms for selection of suboptimal structure
![Page 26: Soft computing Lecture 7 Multi-Layer perceptrons](https://reader035.vdocuments.mx/reader035/viewer/2022062221/56649f575503460f94c7cc4e/html5/thumbnails/26.jpg)
Impossible to relearning
• Using of constructive learning algorithms– Deleting of nodes (neurons) and links
corresponding to one– Appending new neurons if it is needed