aplicación de redes neuronales

7/26/2019 Aplicacin de Redes Neuronales

1/222

ContentsWhat is SOINN

Why SOINNDetail algorithm of SOINN

SOINN for machine learningSOINN for associative memory

References

Self-organizing incremental neural network and its

application

F. Shen1 O. Hasegawa2

1National Key Laboratory for Novel Software Technology, Nanjing University

2Imaging Science and Engineering Lab, Tokyo Institute of Technology

June 12, 2009

F. Shen, O. Hasegawa Self-organizing incremental neural network and its application
http://find/


2/222




References

Contents of this tutorial

1 What is SOINN

2 Why SOINN

3 Detail algorithm of SOINN

4 SOINN for machine learning

5 SOINN for associative memory

6 References

http://find/


3/222




References

What is SOINN

1 What is SOINN

2 Why SOINN




6 References

http://find/http://goback/


4/222




References

What is SOINN

What is SOINN

SOINN: Self-organizing incremental neural network

Represent the topological structure of the input data

Realize online incremental learning


C
http://goforward/http://find/http://goback/


5/222




References

What is SOINN

What is SOINN





C t t
http://find/


6/222




References

What is SOINN

What is SOINN





Contents
http://find/


7/222




References

What is SOINN

What is SOINN





Contents
http://find/


8/222




References

BackgroundCharacteristics of SOINN

1 What is SOINN

2 Why SOINN




6 References


Contents
http://find/


9/222




References


Background: Networks for topology representation

SOM(Self-Organizing Map): predefine structure and size ofthe network

NG(Neural Gas): predefine the network size

GNG(Growing Neural Gas): predefine the network size;

constant learning rate leads to non-stationary result.


Contents
http://find/


10/222




References








Contents
http://find/


11/222

Co t tsWhat is SOINN



References








Contents
http://find/


12/222

What is SOINNWhy SOINN

Detail algorithm of SOINNSOINN for machine learning

SOINN for associative memoryReferences








Contents
http://find/


13/222





Background: Networks for incremental learning

Incremental learning: Learning new knowledge without destroy

of old learned knowledge (Stability-Plasticity Dilemma)ART(Adaptive Resonance Theory): Need a user definedthreshold.

Multilayer Perceptrons: To learn new knowledge will destroy

old knowledgeSub-network methods: Need plenty of storage


Contents
http://find/


14/222











ContentsWh i SOINN
http://find/


15/222











ContentsWh t i SOINN
http://find/


16/222











http://find/


17/222











http://find/


18/222





Characteristics of SOINN

Neurons are self-organized with no predefined network

structure and sizeAdaptively find suitable number of neurons for the network

Realize online incremental learning without any prioricondition

Find typical prototypes for large-scale data set.Robust to noise


http://find/


19/222











http://find/


20/222











http://find/


21/222



References








http://find/


22/222



References








http://find/


23/222



References









Architecture of SOINNTraining process of SOINN
http://find/


24/222



References

Training process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results

1 What is SOINN

2 Why SOINN




6 References



Wh SOINN

http://find/


25/222



References

Training process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results

Structure: Two-layer competitive network

Two-layer competitivenetwork

First layer: Competitivefor input data

Second layer: Competitivefor output of first-layer

Output topology structureand weight vector ofsecond layer



Wh SOINN

http://find/


26/222



References

g pSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results








Why SOINN

http://find/


27/222



References

g pSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results








Why SOINN

http://find/


28/222



References

Similarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results








Why SOINN

Architecture of SOINNTraining process of SOINNSi il i h h ld f j d i i d
http://find/


29/222



References









Why SOINN

Architecture of SOINNTraining process of SOINNSi il it th h ld f j d i i t d t
http://find/


30/222



References


Training flowchart of SOINN

Adaptively updatedthreshold

Between-classinsertion

Update weight ofnodes

Within-classinsertion

Remove noise nodes



Why SOINN

Architecture of SOINNTraining process of SOINNSimilarit threshold for j dging inp t data
http://find/


31/222

yDetail algorithm of SOINN


References







Remove noise nodes



Why SOINN

Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input data
http://find/


32/222

yDetail algorithm of SOINN


References







Remove noise nodes



Why SOINN

http://find/


33/222









Remove noise nodes



Why SOINND il l i h f SOINN

http://find/


34/222









Remove noise nodes



Why SOINND t il l ith f SOINN

http://find/


35/222









Remove noise nodes




http://find/


36/222



y j g g pLearning rateSimple version of SOINNSimulation results






Remove noise nodes


ContentsWhat is SOINNWhy SOINN

Detail algorithm of SOINN

http://find/


37/222



Learning rateSimple version of SOINNSimulation results

First layer: adaptively updating threshold Ti

Basic idea: within-class distance Tbetween-class distance

1 Initialize: Ti = +when node i is a new node.2 When iis winner or second winner, update Ti by

If ihas neighbors, Tiis updated as the maximum distancebetween iand all of its neighbors.

Ti= maxcNi

||Wi Wc|| (1)

If ihas no neighbors, Tiis updated as the minimum distanceof iand all other nodes in network A.

Ti= mincA\{i}

||Wi Wc|| (2)




http://find/


38/222








Ti= maxcNi

||Wi Wc|| (1)


Ti= mincA\{i}

||Wi Wc|| (2)




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataL i
http://find/


39/222








Ti= maxcNi

||Wi Wc|| (1)


Ti= mincA\{i}

||Wi Wc|| (2)




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataL i t
http://find/


40/222








Ti= maxcNi

||Wi Wc|| (1)


Ti= mincA\{i}

||Wi Wc|| (2)




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rate
http://find/


41/222

gSOINN for machine learning







Ti= maxcNi

||Wi Wc|| (1)


Ti= mincA\{i}

||Wi Wc|| (2)




http://find/


42/222


References






Ti= maxcNi

||Wi Wc|| (1)


Ti= mincA\{i}

||Wi Wc|| (2)




http://find/


43/222


References


Second layer: constant threshold Tc

Basic idea 1: within-class distance Tbetween-classdistance

Basic idea 2: we already have some knowledge of input data

from results of first-layer.Within-class distance:

dw = 1

NC

(i,j)C

||Wi Wj|| (3)

Between-class distance of two class Ci and Cj:

db(Ci, Cj) = miniCi,jCj

||Wi Wj|| (4)



Detail algorithm of SOINNSOINN f hi l i

http://find/


44/222


References






dw = 1

NC

(i,j)C

||Wi Wj|| (3)



||Wi Wj|| (4)



Detail algorithm of SOINNSOINN f hi l i

http://find/


45/222


References

e g teSimple version of SOINNSimulation results





dw = 1

NC

(i,j)C

||Wi Wj|| (3)



||Wi Wj|| (4)




http://find/


46/222


References

gSimple version of SOINNSimulation results





dw = 1

NC

(i,j)C

||Wi Wj|| (3)



||Wi Wj|| (4)




http://find/


47/222


References

Simple version of SOINNSimulation results





dw = 1

NC

(i,j)C

||Wi Wj|| (3)



||Wi Wj|| (4)




SOINN for machine learning

http://find/


48/222


References


Second layer: constant threshold Tc (continue)

1 Set Tcas the minimum between-cluster distance.

Tc=db(Ci1 , Cj1 ) = mink,l=1,...,Q,k=l

db(Ck, Cl) (5)

2 Set Tcas the minimum between-class distance.


db(Ck, Cl) (6)

3 IfTc is less than within-class distance dw, set Tcas the next

minimum between-cluster distance.

Tc=db(Ci2 , Cj2 ) = mink,l=1,...,Q,k=l,k=i1,l=j1

db(Ck, Cl) (7)

4 Go to step 2 to update Tc until Tc is greaterthan dw.F. Shen, O. Hasegawa Self-organizing incremental neural network and its application




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSi l i f SOINN
http://find/


49/222


References





db(Ck, Cl) (5)



db(Ck, Cl) (6)




db(Ck, Cl) (7)





Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSi l i f SOINN
http://find/


50/222

S gSOINN for associative memory

References





db(Ck, Cl) (5)



db(Ck, Cl) (6)




db(Ck, Cl) (7)





Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINN
http://find/


51/222

gSOINN for associative memory

References





db(Ck, Cl) (5)



db(Ck, Cl) (6)




db(Ck, Cl) (7)





http://find/


52/222






db(Ck, Cl) (5)



db(Ck, Cl) (6)




db(Ck, Cl) (7)





http://find/


53/222



Updating learning rate 1(t) and 2(t)

Update of weight vector

Ws1 = 1(t)( Ws1 ) (8)

Wi = 2(t)( Wi) (iNs1 ) (9)

After the size of network becomes stable, fine tune the network

stochastic approximation: a number of adaptation steps witha strength (t) decaying slowly but not too slowly, i.e.,t=1(t) =, and

t=1

2(t)


54/222





Ws1 = 1(t)( Ws1 ) (8)

Wi = 2(t)( Wi) (iNs1 ) (9)



t=1

2(t)


55/222


pSimulation results



Ws1 = 1(t)( Ws1 ) (8)

Wi = 2(t)( Wi) (iNs1 ) (9)



t=1

2(t)


56/222


pSimulation results



Ws1 = 1(t)( Ws1 ) (8)

Wi = 2(t)( Wi) (iNs1 ) (9)



t=1

2(t)


57/222


Simulation results

Single-layer SOINN

For topologyrepresentation,first-layer is enough

Within-classinsertion slightlyhappened infirst-layer

Using subclass anddensity to judge ifconnection isneeded.


Contents



SOINN for associative memory

Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNS
http://find/


58/222


Simulation results

Single-layer SOINN





Contents




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSi l i l
http://find/


59/222


Simulation results

Single-layer SOINN





Contents




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSi l ti lt
http://find/


60/222

yReferences

Simulation results

Single-layer SOINN





Contents




Architecture of SOINNTraining process of SOINNSimilarity threshold for judging input dataLearning rateSimple version of SOINNSimulation results
http://find/


61/222

yReferences

Simulation results

Single-layer SOINN





Contents




http://find/


62/222

ReferencesSimulation results

Single-layer SOINN





Contents



SOINN for associative memoryf

http://find/


63/222


Artificial data set: topology representation

Stationary and non-stationary

Stationary: all training data obey same distributionNon-stationary: next training sample maybe obey differentdistribution from previous one.

Original data Stationary Non-stationary


Contents



SOINN for associative memoryR f

http://find/


64/222







Contents




http://find/


65/222

References






Contents




http://find/


66/222

References






Contents




http://find/


67/222

References






Contents




http://find/


68/222

References






Contents




http://find/


69/222

References






Contents




http://find/


70/222

Artificial data set: topology representation (continue)

Original data Two-layer SOINN Single-layer SOINN

Conclusion of experiments: SOINN is able to

Represent topology structure of input data.

Realize incremental learning.

Automatically learn number of nodes de-noise etcF. Shen, O. Hasegawa Self-organizing incremental neural network and its application Contents




http://find/


71/222










http://find/


72/222












73/222










http://find/


74/222












75/222






Automatically learn number ofnodes, de-noise, etc.F. Shen, O. Hasegawa Self-organizing incremental neural network and its application

Contents




http://find/


76/222







Contents




http://find/


77/222







Contents




Unsupervised learningSupervised learningSemi-supervised learningActive learning
http://find/


78/222

1 What is SOINN

2 Why SOINN




6 References


Contents

What is SOINNWhy SOINNDetail algorithm of SOINN


References

http://find/


79/222

Some objectives of unsupervised learning

Automatically learn number of classes of input data

Clustering with no priori knowledge

Topology representation

Realize real-time incremental learning

Separate classes with low density overlapped area


Contents



References

http://find/


80/222








Contents



References

http://find/


81/222








Contents



References


S bj i f i d l i
http://find/


82/222







F. Shen, O. Hasegawa Self-organizing incremental neural network and its application Contents



References


S bj i f i d l i
http://find/


83/222










References


S bj i f i d l i
http://find/


84/222










References


SOINN f i d l i If t d t d
http://find/


85/222

SOINN for unsupervised learning: If two nodes connected

with one path, the nodes belong to one class

1 Do SOINN for input data, output topology representation ofnodes

2 Initialize all nodes as unclassified.3 Randomly choose one unclassified node i from node set A.

Mark node ias classified and label it as class Ci.

4 Search A to find all unclassified nodes that are connected to

node iwith a path. Mark these nodes as classified and labelthem as the same class as node i.

5 Go to Step3 to continue the classification process until allnodes are classified.




References


SOINN f i d l i If t d t d
http://find/


86/222









F Shen O Hasegawa Self-organizing incremental neural network and its application Contents



References


http://find/


87/222












References


http://find/


88/222












References


http://find/


89/222














http://find/


90/222














Artificial data set: 5 classes with 10% noise
http://find/


91/222


Original data Clustering result

Conclusion of experiments

Automatically reports number of classes.

Perfectly clustering data with different shape and distribution.

Find typical prototypes; incremental learning; de-noise; etc.F Shen O Hasegawa Self organizing incremental neural network and its application




References


http://find/


92/222










References




93/222










References


http://find/


94/222










References


http://find/


95/222










References


http://find/


96/222










References


Face recognition: AT&T face data set
http://find/


97/222

g

Experiment results

Automatically reports there are 10 classes.

Prototypes of every classes are reported.With such prototypes, recognition ratio (1-NN rule) is 90%.

F Shen O Hasegawa Self organizing incremental neural network and its application Contents





http://find/


98/222

g

Experiment results







References


http://find/


99/222

g

Experiment results







References




100/222

g

Experiment results







References




101/222

Experiment results







References


Prototype-based classifier: based on 1-NN or k-NN rule


102/222

Nearest Neighbor Classifier (NNC): all training data asprototypesNearest Mean Classifier (NMC): mean of each class asprototypes

k-means classifier (KMC), Learning Vector Quantization(LVQ), and others: predefine number of prototypes for everyclass.

Main difficulty

1 How to find enough prototypes without overfitting2 How to realize Incremental learning

Incremental of new data inside one class (non-stationary orconcept drift);Incremental of new classes.





References




103/222



Main difficulty

1

How to find enough prototypes without overfitting2 How to realize Incremental learning






References


http://find/


104/222



Main difficulty

1







References




105/222



Main difficulty

1







References




106/222



Main difficulty

1







References


http://find/


107/222



Main difficulty

1







References


http://find/


108/222



Main difficulty

1







References




109/222



Main difficulty

1







References


http://find/


110/222



Main difficulty

1







References




111/222



Main difficulty

1







References


SOINN for supervised learning: Targets
http://find/


112/222

Automatically learn the number of prototypes needed torepresent every class

Only the prototypes used to determine the decision boundarywill be remained

Realize both types of incremental learning

Robust to noise





References


http://find/


113/222




Robust to noise





References


http://find/


114/222




Robust to noise





References


http://find/


115/222




Robust to noise





References




116/222




Robust to noise





References


Adjusted SOINN Classifier (ASC)
http://find/

aplicación de redes neuronales

Documents