1 the tiling algorithm learning in feedforward layered networks: the tiling algorithm writed by marc...
TRANSCRIPT
![Page 1: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/1.jpg)
1
The tiling algorithm
“Learning in feedforward layered networks: the tiling algorithm”
writed by Marc Mézard and Jean-Pierre Nadal
![Page 2: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/2.jpg)
2
Outline
Introduction The tiling algorithm Simulations Concluding remarks
![Page 3: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/3.jpg)
3
Introduction
The feedforward layered system The drawbacks of back propagation
The structure of the network has to be guessed.
The error is not guaranteed to converge to an absolute minimum with zero error.
Units are added like tiles whenever they are needed.
![Page 4: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/4.jpg)
4
Introduction
![Page 5: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/5.jpg)
5
The tiling algorithm
Basic notions and notation Theorem for convergence Generation the master unit Building the ancillary units: divide and
conquer
![Page 6: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/6.jpg)
6
Basic notions and notation
We consider layered nets, made of binary units which can be in a plus or minus state.
A unit i in the Lth layer is connected to the NL-1 units and has state
1
0
)1(,
)( sgnLN
j
Lj
Lji
Li SwS
![Page 7: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/7.jpg)
7
Basic notions and notation
For a given set of p0 (distinct) pattern of N0 binary units, we want to learn a given mapping
(1 -1 1 1 -1 -1)
![Page 8: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/8.jpg)
8
Theorem for convergence
To each input pattern (1 -1 1 1 -1 ) there corresponds a set of values of the neurons in the Lth layer as the internal representation of pattern in the layer L.
![Page 9: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/9.jpg)
9
Theorem for convergence
We say that two patterns belong to the same class (for the layer L) if they have the same internal representation, which we call the prototype of the class.
The problem becomes to map these prototypes onto the desired output.
![Page 10: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/10.jpg)
10
Theorem for convergence master unit
the first unit in each layer ancillary unit
all the other units in each layer, use to fulfil the faithfulness condition.
faithful two input patterns having different output
should have different internal representations.
![Page 11: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/11.jpg)
11
Theorem for convergence
Theorem: Suppose that all the classes in layer L-1
are faithful, and that the number of errors of the master unit, eL-1, is non-zero. Then there exists at least one set of weights w connecting the L-1 layer to the master unit such that . Furthermore, one can construct explicitly one such set of weights u.
11 LL ee
![Page 12: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/12.jpg)
12
Theorem for convergence
Proof: Let be the prototypes in layer L-1. be t
he desired output(1 or –1). If the master unit of the Lth layer is connect
ed to the L-1th layer with the weight w(w1 = 1, wj = 0 for j ~= 1), then eL=eL-1.
s
![Page 13: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/13.jpg)
13
Theorem for convergence
Let be one of the patterns for which, and let u be u1=1 and
then
then
000
1 s
00 jj su
)sgn(1
00
0,11
LN
jjjjsm
1/100 LNifsm
![Page 14: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/14.jpg)
14
Theorem for convergence
Consider other patterns,
can be -NL-1, -NL-1+2, …, NL-1
Because the representations in the L-1 layer are faithful, -NL-1 can never be obtained.
Thus one can choose
so the patterns for which
still remain .
1
00
jjjss
s1
)1/(1 1 LN
)( sm
![Page 15: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/15.jpg)
15
Theorem for convergence
Hence u is one particular solution which, if used to define the master unit of layer L, will give
111 0 LLL eVee
![Page 16: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/16.jpg)
16
Generation the master unit
Using ‘pocket algorithm’ If the particular set u of the previous
section is taken as initial set in the pocket algorithm, the output set w will always satisfy .11 LL ee
![Page 17: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/17.jpg)
17
Building the ancillary units
The master unit is not equal to the desired output unit means that at least one of the two classes is unfaithful.
We pick one unfaithful class and add a new unit to learn the mapping
for the patterns μ belonging to this class only.
Repeat above process until all classes are faithful.
s
![Page 18: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/18.jpg)
18
Simulations
Exhaustive learning (use the full set of the 2N patterns) Parity task Random Boolean function
Generalization Quality of convergence Comments
![Page 19: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/19.jpg)
19
Parity task
In the parity task for N0 Boolean units the output should be 1 of the number of units in state +1 is even, and –1 otherwise.
![Page 20: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/20.jpg)
20
Parity task
Hidden layerUnit
number
Threshold Coupling from the input layer to the hidden unit i
1 -55 +11 +11 +11 -11 +11 -11
2 +33 -11 -11 -11 +11 -11 +11
3 -11 +11 +11 +11 -11 +11 -11
4 -11 -11 -11 -11 +11 -11 +11
5 +33 +11 +11 +11 -11 +11 -11
6 -55 -11 -11 -11 +11 -11 +11
![Page 21: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/21.jpg)
21
Parity task
Output unit
Threshold Couplings from the hidden layer to the output unit
+11 +11 +13 +13 +13 +13 +11
![Page 22: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/22.jpg)
22
Random Boolean function
A random Boolean function is obtained by drawing at random the output (±1 with equal probability) for each input configuration.
The numbers of layers and of hidden units increase rapidly with N0.
![Page 23: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/23.jpg)
23
Generalization
The number of training patterns is smaller than 2N.
The N0 input neurons are organized in a one-dimensional chain, and the problem is to find out whether the number of domain walls is greater or smaller than three.
![Page 24: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/24.jpg)
24
Generalization
domain wall The presence of two neighboring neurons
pointing in opposite directions
When the average of domain walls are three in training patterns, the problem is harder than other numbers.
![Page 25: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/25.jpg)
25
See figure 1 in page 2199Learning in feedforward layered networks: the tiling algorithm
![Page 26: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/26.jpg)
26
Quality of convergence
To quantify the quality of convergence one might think of at least two parameters. eL
pL : the number of distinct internal representations in each layer L.
![Page 27: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/27.jpg)
27
See figure 2 in page 2200Learning in feedforward layered networks: the tiling algorithm
![Page 28: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/28.jpg)
28
Comments
There is a lot of freedom in the choice of the unfaithful classes to be learnt.
How to choose the maximum number of iterations which are allowed before one decides hat the perceptron algorithm has not converged?
![Page 29: 1 The tiling algorithm Learning in feedforward layered networks: the tiling algorithm writed by Marc M é zard and Jean-Pierre Nadal](https://reader034.vdocuments.mx/reader034/viewer/2022051819/5517901655034645368b55e7/html5/thumbnails/29.jpg)
29
Concluding remarks
presented a new strategy for building a feedforward layered network
Identified some possible roles of the hidden units: the master units and the ancillary units
continuous inputs and binary outputs conflicting data more than one output units