neural networks part 4 dan simon cleveland state university 1
Post on 15-Jan-2016
214 views
TRANSCRIPT
Neural NetworksPart 4
Dan SimonCleveland State University
1
Outline
1. Learning Vector Quantization (LVQ)2. The Optimal Interpolative Net (OINet)
2
Learning Vector Quantization (LVQ)
Invented by Tuevo Kohonen in 1981Same architecture as the Kohonen Self Organizing MapSupervised learning
x1
xi
xn
y1
yk
ym
w11
w1kw1m
wi1wik
wim
wn1 wnk
wnm
3
LVQ Notation:
x = [x1, …, xn] = training vectorT(x) = target; class or category to which x belongswk = weight vector of k-th output unit = [w1k, …, wnk]a = learning rate
LVQ Algorithm:Initialize reference vectors (that is, vectors which represent prototype inputs for each class)while not (termination criterion)
for each training vector xk0 = argmink || x – wk ||if k0 = T(x) then
wk0 wk0 + a(x – wk0)else
wk0 wk0 – a(x – wk0)end if
end forend while
4
w1
w2
w3
x
x–w2
We have three input classes.Training input x is closest to w2.
If x class 2, then w2 w2 + a(x – w2)that is, move w2 towards x.
If x class 2, then w2 w2 – a(x – w2) that is, move w2 away from x.
LVQ Example
LVQ reference vector initialization:
1.Use a random selection of training vectors, one from each class.2.Use randomly-generated weight vectors.3.Use a clustering method (e.g., the Kohonen SOM). 5
LVQ Example: LVQ1.m
(1, 1, 0, 0) Class 1(0, 0, 0, 1) Class 2(0, 0, 1, 1) Class 2(1, 0, 0, 0) Class 1(0, 1, 1, 0) Class 2
Final weight vectors:(1.04, 0.57, 0.04, 0.00)(0.00, 0.30, 0.62, 0.70)
6
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1Training Data
0 0.2 0.4 0.6 0.8 10
0.2
0.4
0.6
0.8
1Final Result
Final classification results on the training data, and final weight vectors. 14
classification errors after 20 iterations.
LVQ Example: LVQ2.m
Training data from Fausett, p. 190.
Four initial weight vectors are at the corners of the training data.
7
8
-0.2 0 0.2 0.4 0.6 0.8-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Training Data
-0.2 0 0.2 0.4 0.6 0.8-0.2
0
0.2
0.4
0.6
0.8
1
1.2
Final Result
LVQ Example: LVQ3.m
Final classification results on the training data, and final weight vectors. Four
classification errors after 600 iterations.
Training data from Fausett, p. 190.20 initial weight vectors are randomly
chosen with random classes.
In practice it would be better to use our training data to assign the classes of the initial weight vectors.
LVQ Extensions:
9
w1
w2
w3
xx–w2
The graphical illustration of LVQ gives us some ideas for algorithmic modifications.•Always move the correct vector towards x, and move the closest p vectors that are incorrect away from x.•Move incorrect vectors away from x only if they are within a distance threshold.•Popular modifications are called LVQ2, LVQ2.1, and LVQ3 (not to be confused with the names of our Matlab programs).
10
LVQ Applications to Control Systems:
•Most LVQ applications involve classification•Any classification algorithm can be adapted for control•Switching control – switch between control algorithms based on the system features (input type, system parameters, objectives, failure type, …)•Training rules for a fuzzy controller – if input 1 is Ai and input 2 is Bk, then output is Cik – LVQ can be used to classify x and y•User intent recognition – for example, a brain machine interface (BMI) can recognize what the user is trying to do
11
The optimal interpolative net (OINet) – March 1992Pattern classification; M classes; if x class k, then yi = ki
The network grows during training, but only as large as needed.
x1
x2
xN
y1
y2
yM
w11
w12w1m
w21
w22
w2m
wq1wq2
wqM
q hidden neurons
v11
v12v1q
v21
v22v2q
vN1vN2
vNq
vi = weight vector to i-th hidden neuron; vi is N-dimensionalvi = prototype; { vi } { xi }
12
1 11
1
(|| ||)
(|| ||
1) ( )( 1( )
T T
q q
y W W
M
v x
v x
M q q
Suppose we have q training samples: y(xi) = yi, for i {1, …, q}. Then:
11 1
1
1
1
]
( ) ( )
[
( )
qT
q
q qq
T
T
y W
Y W G
M q M q q
y
q
W G Y
Note if xi = xk for some i k,then G is singular
13
The OINet works by selecting a set of {xi} to use as input weights.These are the prototype vectors {vi}, i = 1, …, p.
Choose a set of {xi} to optimize the output weights W.These are the subprototypes {zi}, i = 1, …, l.
Include xi in {zi} only if needed to correctly classify xi.
Include xi in {vi} only if G is not ill-conditioned, and only if it decreasesthe total classification error.
1
1
1
more exp
(
licit notation:
) ( )( )
(
(
)
) ( ) ( )
T
T T T
T
l T l T lp l p p
G
M l M p p l
W Y G GG
G R
Y
Y W
Y
W G R
Use l inputs for training.Use p hidden neurons.Gik = (||vizk||)
14
Suppose we have trained the network for a certain l and p.All training inputs considered so far have been correctly classified.
We then consider xi, the next input in the training set.Is xi correctly classified with the existing OINet? If so, everything is fine, and we move on to the next training input.
If not, then we need to add xi to the subprototype set {zi} and obtaina new set of output weights W. We also consider adding xi to the prototypeset {vi}, but only if it does not make G ill-conditioned, and only if it reducesthe error by enough.
1 1 1
1
1
(|| ||) (|| ||)
(|| ( ||) (|| ||)
( )
) ( )(( )
lT
l
p p l
l T ll p p
y
v z v z
y
v z v z
W G
l M p p l
p l
W
Y
M
15
Suppose we have trained the network for a certain l and p.
Suppose xi, the next input in the training set, is not correctly classified. We need to add xi to the subprototype set {zi} and retrain W.
1, , We have , and ( )l l l lp p p pRW G R
1
1 1 1 1 11
1 1
1 1
1 11
1 1 1 11
(|| ||) (|| ||) (|| ||)
)
(|| ( ||) (|| ||) (|| ||)
( )
( 1)) ( )( ( 1))
(
) ( )( ) (
(
l i
l ll T
l l
p
p
p p l l
l T ll p p
l T l T lp l p p
x
v z v z v z
y
v z v z v z
W G
l M p p
z
y y W
Y
M l
Y GW R
This is going to get expensive if we have lots of data, and if we have to perform a new matrix inversion every time we add a subprototype.Note: Equation numbers from here on refer to those in Sin & deFigueiredo
1 (Eq. 11)lpk
16
1 1 1 1 1 1 1
Matrix inversion lemma (prove for homewo
)
rk):
( ( )C A A B D CA B CAA BD
1 1 1 1 1 11
1 1
1 1 1 1 1
11 1 1 1 1 1 1 1
1
([ ( (
(
(
( ) ( )
( ) ( ) ( ( ) ( ( )
(
)( ) ] ) )
)
)
( )
) )
( )
l Tpl l l T l l l l T l l T
p p p p p p p p pl Tp
l l l Tp p p
l l l l Tp p p p
l l l l T l l l T lp p p p p p p p
lp
G G k k kG
R G G G
R
R R
R
k
k k
k k
k I k kR R
RR
kR
1 1 1 1
1 1 1
) ( ( )(Eq. 17)
( ( )
)
1 )
l l l T lp p p p
l T l lp p p
k k
k k
R
R
We only need scalar division to compute the new inverse, because we already know the old inverse!
(Eq. 10)
17
We can implement this equation, but we can also find a recursive solution.
1 1 1 11
1 1 1 11
) ( ) ( )(
( )
l T l T lp l p p
l l lp p p
Tl
Y G R
R G Y
W
W
1 1 1 11 1 1
1 1 11
1 1 1 11 1 1
1 11 1 1
( ) ( ( )( )
( ( )
( ) ( ( )( ) ( ) ( )
(
)
1 )
)
( )
(
1 )
l l l T l Tp p p pl l l l l
p p p pl T l l Tp p p l
l l l T lp p p pl l T l T l T l T
p p l p l p l p ll T l lp p p
lp
k kG
k k
k kG G
R R YW R k
R y
R RR Y k y Y k y
R
kR
W
k
1 1 111 1
1 1 1 1
)
1 )
( ( ( ) )) (Eq. 20)
( ( )
l T l l l Tp p p p ll l T
p p l l T l lp p p
k k
k k
W R yk y
R
18
1 1 1
1 1 1 1 1 11 1 1
1
1 1 1
1 1
1 1
11
11 11
( )
( ( ) ( ( ( ) )
( ( )
( ) ( ) ( )
) )
1 )
)
( )
( ( )
l l l lp p p p
T l T l l T l T l l l Tl p p p l p p p p l
l T l lp p p
l l lp p p
T l T l l T l T l lp p p l p
l lp p l
Tpl pk
W W R k
y R y
R k y k R k y
W R y
R
W R k
y
k k k k
k k
Wk
1 1 1
111 1
1 1 1
1
( ( )
(( ) (Eq. 20)
(
1 )
( )
)
1 )
l T l lp p p
T l T ll p pl l l
p p p l T l lp
T
p p
k k
k
k k
R
y WW R k
R
We have a recursive equation for the new weight matrix.
19
Now we have to decide if we should add xi to the prototypeset {vi}. We will do this if it does not make G ill-conditioned, and if it reducesthe error by enough.
1
1 1 1 1 1
11 1 1
1 1
1 1 1 1
1 11 1 1
1
(|| ||) (|| ||) (|| ||)
)(|| ( ||) (|| ||) (|| ||)
(|| ( ||) (|| ||) (|| ||)
(
( 1)
(
(
)
p
p i
l l
l Tl l p
p p l l
p p l l
l T ll p
p
p
x
v z v z v z
yv z v z v z
v z v z v z
W G
v
y y W
Y
M l
111
1 1
1 1 1 1 1 1 1 11 1 1 1 1 1 1 1
) ( ( 1))(( 1) ( 1))
) ( ) ] ( ) (
(Eqs. 22-24)
( [ ( ) )
l ll p pp T
l l
l T l T l l T l T lp l p p p l p p
G k
M p p l
r
W
G
Y G G G Y G R
I wonder if we can think of something clever to avoid the new matrix inversion …
20
1 1 11 1 1
11
111 1
1 11 1
1 21 1 1 1 1
1 11
( )
( )
(Eq. 26)( ) ( )
'( )
' '
( )( )
'
l l l Tp p p
l Tl lp lp pl TTp ll l
l l lp p l l p
T l T l T Tl p l p l l l
lp
R
G rG k
kr
R G r k A
r G k
G G
B
r r C D
AR
D
B
C
1
1
1 1 1 1
1 1 1 1 1
Another matrix inversion lemma (prove for homework):
' '- First define
' '
'
' ' '
BA B A B
E D CAC D C D
A A
B A C E D
A BE CA
BE CA E
21
1 1 1 1
12 1 1 1 1
1 1
11 1
11 1
1 11
1 1
1
1
1
1 1 1 1 1
( )
( ( )
' ( (
(
(Eq. 29
(
( ) )
) )
)
ˆ
) ( ) )
ˆ
(
)]
( )
[
T T l T l T l l ll l l l p l p p p l
l lp p
lp
T
l lp l
l p
l p
T l T l Tl p l p
l l lp l l p p
Tl l
G r k
r G k
G r
A R R
r r r G k R G
G r
r k
r
k
R
r
1 1 1
1 1 1 1 1 1 1 1
1 1
1
1
1
11 1
1 1 1
(
(Eq. 30)
' ( ( ( (
ˆ) (Eq. 28)
ˆ ˆ) ) ) ) ( /
(
( /( (E
ˆ ˆ ˆ ˆ( ) ( )
q. 27
)
' ) /
) /) )
1//
l lp p
l l
T T l T l lp
l l T lp p p p p
l Tp
l Tl pp T
p pr r r G
R G
A R R G G R
R uu
R uu uR
R G
u r
r
u
r
r
A
Homework: Derive this
22
We already have everything on the right side, so we can derive the newinverse with only scalar division (no additional matrix inversion).
Small ill conditioningSo don’t use xi as a prototype if < 1 (threshold)
Even if > 1 , don’t use xi as a prototype if the error only decreases by a small amount, because it won’t be worth the extra network complexity.
1 1 11
1 1 11 1 1 1
1 1 2
,
1 1 21 1
,
1 11
( ) ( 1) matrix (Eq. 35)
( ) ( 1)
)
)
(Eq. 38)
matrix
(
(
l l T lp l p p
l l T lp l p p
l lp p ij
i j
l lp p ij
i j
l lp p
Y W G l
Y W G l
e e
E M
E M
e E
e E
e
Before we check e, let’s see if we can find a recursive formula for W …
23
1 1 1 11 1 1 1
11 1
1 1 1
1 1 1 1 1 11 1
11 1
( /
1/
( / ( /
( )
) /
/
) /
/
)
/
/
/ /
l l l Tp p p l
l l Tl Tp p lpT TTl l l
l l T l T l l T lp p p l p p p l
T l T T lp l p l
W
G k YR uu u
r yu
R G uu G ur R k uu k u
u G r u k
R G Y
1
1
2
1
1 1
1 1 1 1 11
1 1 1
1
1 1 11
111
11
1
/
/
( ) (
ˆˆ( )
/
/
ˆ /
/ (
)
)
/
T l Tp l
T l
l l T l l Tp p l p p l
lp
TlTl
l l l T T lp
Tp l
T l Tp
T Tl l
Tl l
T Tl l
p p l p
ur Y
u y
ur
uu G Y
uu k y
u
R G
Y
y
w
w
w
R G rrW Y
Y R k y
W G YY u
R
1 1 1 1 11
1 1 1 1 1 1 1 11 1
1 1 1 1 1 11
1 1 1 1 11
ˆˆ ( ) ( )
ˆˆ( ) ( ) ( )
ˆˆ( ) ( )
ˆ ˆ/ (Eq. 34) ( )
/
/
/
l T l T l l Tp p p p l
l l l T T l T l l Tp p p l p p p l
l l l T T l T lp p p l p p
l l l lp p p p
G rr G R G Y
R G rr Y G R G Y
R G rr Y G
E r w R G r
W
W W
W
(Eq. 33)T
24
2
11 1
1 1 1 11 1
1 1
1
1
1 11
1
1
11 1
ˆ/ /
ˆ
/ /
) ]ˆ[ ( /
) ] /
) )] /
( )
ˆ ˆ[ (
ˆ[ ((
ˆ ( )
/
/
/T l T T T
p l l
T l T l l T T Tp p p l l
T l T l T Tp p l
T l T l
T l T T l Tp l
Tp p l
T l T
T T Tl l p l
p
l lr Y yuw
u G Y r Y
r G R G Y r Y
r G W r Y
r G W Y
r
G Y u k y
E
1 1 1 1
11
1
ˆ
(Eq.
( )
125)
T
l l l Tl p p pp T
lp
T
R G r
MW
M
WW
pW
We have a recursive equation for the new weight matrix.
25
Back to computing e (see Eq. 38):Suppose Ax = b, and dim(x) < dim(b), so system is over-determinedLeast squares:
1
2 1 21
1 2
1 1
1 1 1 1
1
( )
|| ( (
( ( (
|| || || ( ) ||
) ) ||
) ) ) )
) ) ) ) )
(
( ( ( ( (
(( ) )
T T
T T
T T
T T T T T T
T T T T T T T T T
T T T
Ax b A A A A b b
A A I b
A A I A A I
x A A A b
e
A A
b A A A A
A A A A A A A A A I
A
b
b A A A A A A A b
I AA bb A
Now suppose that we add another column to A and another element to x.
ˆ
ˆ [ a]
ˆ ˆ [ a]=T T T
T
T T T
Ax b
A A
A A A A aA A A
a a A a a
We have more degrees of freedom in x, so the approximation error should decrease.
26
Matrix inversion lemma:
11
1 1
1
12
1 11 2
) /ˆ ˆ)/
( ) ( ( ) )
( )
ˆ ˆ ˆ ˆ( ( ) )
ˆ ˆ ˆ ˆ( )
( /(
1
)
/
(
T TT
T
T T T T T T
T T
T T
T
T
T T T T T T
A ggA A
a a a A A A A a a I A A A A a
g A A a
e b I A A A A b
e e e b A A A A
A
b b A A A
g
A b
g
A
But notice:
11
1
1 2
) /ˆ ˆ ˆ ˆ( ) [/
( ) / / / /
/
( /]
1/
/ / /
T T TT T T T
T T
T T T T T T T T T
T T T T T T T
A gg AA A A A b b A b
a
b A A A A Agg A ag A Aga aa b
e e e b Agg A ag A A
A gb
ga a
g
a b
a
27
2
21
21
/
)
2
/
( ) /
( ( ) /
T T T T T T T T
T T
T T T T
T T T
e b b b ag A b b aa b
b Ag b a
b A A A A a b a
b I A A A
Agg A
aA
1 2
21
1 1 11 1 1 1
1 2
ˆ) [ ] (error
ˆ has one more column than
Now consider (error , and )
(error ,
. Solve for .
( ) /
This is like going from
(G ) ) (G )to
T T T T
l T l T l Tp p l p p
AX A a X B e
A A X
e
AX B e
e
e e B B A A A A a
W Y W
1 21 (error )l T
l eY
28
1 2
21 1 1 1
1 1
21 1
1
21
2
ˆ( ) ( )
ˆ( )
ˆ /
|| ||
/
/
l T l ll l p p p
l T ll p p
lp
Y
Y
e e e
Y G R G r
W G r
E r
We have a very simple formula for the error decrease due to adding a prototype. Don’t add the prototype unless (e / e1) > 2 (threshold)
29
The OINet Algorithm
Training data {xi} {yi}, i {1, …, q}Initialize prototype set: V = {x1}, and # of prototypes = p = |V| = 1Initialize subprototype set: Z = {x1}, and # of subprototypes = l = |Z| = 1
1 11
1 11
211
1 211
/
1
( )
) /(
l l lp p p
l Tp
l l l Tp p p
lp
W W G
yW
R G G
y
R
x1
x2
xN
y1
y2
yM
w11
w12
w1m
1 hidden neuron
v11
v21
vN1
What are the dimensionsof these quantities?
30
Begin outer loop: Loop until all training patterns are correctly classified.n = q – 1 Re-index {x2 … xq} & {y2 … yq} from 1 to nFor i = 1 to n (training data loop)
Send xi through the network. If correctly classified, continue loop.Begin subprototype addition:
1
11 1 1
1 1
1
1 1
1 1 1 1
(Eq. 11)
(R computation (Eq. 17)
computation (Eq. 20)
Consider using as a prototype (note
(|| ||) (|| ||
)
)
(||
:
(Eq. ||) (|| || 2
l i
Tlp l p l
lp
lp
i p l
T
l l l l
z x
v z v z
z
z z z
k
x
z
W
v
r
1 1 1
1
1
3)
(Eq. 2(|| ||)
ˆ
4)
(Eq. 29)
l l l
l
l
r
z z
r
31
1 1 1 1 1
1 1 1 1
1 1 11
1 2
11 2
ˆ)
ˆ ˆ ˆ
(Eq. 10), ( (Eq. 28)
) ( (Eq. 30)
E ) (Eq. 35)
(Eq. 34), (Eq. 40)
If th
ˆ( )
(
ˆ / || ||
en a und /
l l l l lp p p p p
T T l T l lp p p
l l T lp l p p
lp
lp
G r
r r r r
Y
E r e
e
G k u R G
G R G
W G
E
111
1 1
1 11
1 11
1 1 1 11
se as a prototype
(Eq. 22)
Update (Eq. 25)
Update (Eq. 26)
Update (
End
(
1
End prototype additi
) ) (Eq. 27)
subprototype add
on
1
i
ll pp T
l l
l lp p
l lp p
l lp p
x
GG
r
W
R
R
p
W
R
R
p
ll
End training dat
End
it
ou
a l
ter
i
oo
o
p
n
loop
32
Homework:
•Implement the OINet using FPGA technology for classifying subatomic particles using experimental data. You may need to build your own particle accelerator to collect data. Be careful not to create any black holes.•Find the typo in Sin and deFigueiredo’s original OINet paper.
References
•L. Fausett, Fundamentals of Neural Networks, Prentice Hall, 1994•S. Sin and R. deFigueiredo, An evolution-oriented learning algorithm for the optimal intepolative net, IEEE Transactions on Neural Networks, vol. 3, no. 2, pp. 315–323, March 1992
33