A New Classifier Based on Associative Memories

Israel Román-Godínez, Itzamá López-Yáñez, Cornelio Yáñez-MárquezCentro de Investigación en Computación

Juan de Dios Bátiz s/n esq. Miguel Othón de MendizábalUnidad Profesional Adolfo López MateosDel. Gustavo A. Madero, México, D. F.

Méxicoiromanb05@sagitario.cic.ipn.mx,

ilopezb05@sagitario.cic.ipn.mx, cyanez@cic.ipn.mx

Abstract

The Lernmatrix, which is the first known model of as-sociative memory, is an heteroassociative memory, but itcan also act as a binary pattern classifier depending on thechoice of the output patterns. However, this model sufferstwo great problems: saturation and imperfect recall of someof the associations, even in the fundamental set, dependingon the associations. In this work, a modification to the orig-inal Lernmatrix recall phase algorithm is presented. Thismodification improves the recalling capacity of the originalmodel. Experimental results show this improvement

1 Introduction

Karl Steinbuch was one of the first researchers to developa method to code information in square arrays known ascrossbars [1,2]. The trascendence of the Lernmatrix [3] isevidenced by an affirmation by Kohonen in [4] where hepoints that correlation matrices, foundation of his innova-tive work, substitute Steinbuch’s Lernmatrix.The Lernmatrix, which is the first known model of as-

sociative memory, is an heteroassociative memory, but canalso act as a binary pattern classifier, depending on thechoice of associations. It is an input-output system that re-ceives a binary input pattern and produces the associatedclass, codified with a simple method: one-hot [5]. How-ever, this model suffers two great problems: the phenom-enon of saturation, and imperfect recall of some of the as-sociations, even in the fundamental set. In both cases, thereason for recall to fail is that there are more than one 1’s inthe components of the output pattern. There are two rea-sons for this to happen. First, when there are more thanone input pattern associated with one output pattern (sat-

uration), and second, when the characteristic set (the set ofcomponents in the pattern which have the value of 1, as in-troduced in [17-19] and further discussed in section 2) ofone or more learned patterns is a subset of the character-istic set of another learned pattern. In this work, a mod-ification to the original Lernmatrix recall phase algorithmis presented in order to avoid, as best as possible, the lat-est problem. Through the use of this modification, morepatterns are perfectly recalled than when using the originalLernmatrix algorithm.The rest of the paper is organized as follows: section 2

presents some background on the original Lernmatrix andrelated theretical advances presented by Sánchez-Garfias,et. al., while in section 3 the proposed modification is pre-sented. Section 4 contains some experimental results of theapplication of the proposed method and section 5 presentsconclusions and future work.

2 The Steinbuch’s Lernmatrix

Basic concepts about associative memories were estab-lished three decades ago in [13-15], nonetheless here we usethe concepts, results and notation introduced in the Yáñez-Márquez’s PhD Thesis [16]. An associative memoryM isa system that relates input patterns, and outputs patterns,as follows: x −→ M −→ y with x and y the inputand output pattern vectors, respectively. Each input vec-tor forms an association with a corresponding output vec-tor. For k integer and positive, the corresponding asso-ciation will be denoted as

¡xk,yk

¢. Associative memory

M is represented by a matrix whose ij-th component ismij . Memory M is generated from an a priori finite setof known associations, known as the fundamental set of as-sociations. If µ is an index, the fundamental set is repre-sented as: {(xµ,yµ) | µ = 1, 2, ..., p} with p the cardinal-

Proceedings of the 15th International Conference on Computing (CIC'06)0-7695-2708-6/06 $20.00 © 2006

ity of the set. The patterns that form the fundamental setare called fundamental patterns. If it holds that xµ = yµ

∀µ ∈ {1, 2, ..., p}, then M is autoassociative, otherwiseit is heteroassociative. In this latter case it is possible toestablish that ∃µ ∈ {1, 2, ..., p} for which xµ 6= yµ. Adistorted version of a pattern xµ to be recalled will be de-noted as x̃µ. If when feeding a distorted version of xω withω ∈ {1, 2, ..., p} to an associative memoryM, it happensthat the output corresponds exactly to the associated patternyω, we say that recall is perfect.The Lernmatrix is an heteroassociative memory, but it

can act as a binary pattern classifier depending on the choiceof the output patterns; it is an input-output system that gets abinary input pattern xµ ∈ An, A = {0, 1} and n ∈ Z+ andproduces the class (from q different classes) codified witha simple method: assigning for the output binary patternyµ the following values: yµk = 1, and yµj = 0 for j =1, 2, . . . , k − 1, k + 1, . . . , q where k ∈ {1, 2, . . . , q}In the learning phase, each component mij of the Lern-

matrixM is initialized to zero, and it is updated accordingto the following rule: mij = mij +∆mij , where:

∆mij =

+ if yµi = 1 = xµj−ε if yµi = 1 and x

µj = 0

0 otherwise

where is any positive constant previously chosen.The recalling phase consists of finding the class to which

an input pattern xω ∈ An belongs. Finding the class meansobtaining the components of the vector yω ∈ Ah whichcorrespond to xω, according to the construction method ofall yµ; the class should be obtained without ambiguity.The i-th coordinate yωi of the class vector is obtained

according to the next expression, where ∨ is the maximumoperator:

yωi =

(1 if

Pnj=1mijxωj =

Wqh=1

hPnj=1mhjxωj

i0 otherwise

However, there is a problem called saturation. This hap-pens when yω has a 1 in more than one component, andtherefore we could not consider this a correct association.Saturation could be caused due to two diferents reason: 1)when there is more than one input pattern associated to oneoutput pattern, and 2) during the learning phase, when thecharacteristic set of one or more learned patterns is a subsetof the characteristic set of another learned pattern, the per-fect recall is not guaranteed neither. Therefore we have awrong classification.

In order to get a better understanding of this phenomenonit is important to include the concept of characteristic setintroduced by Sánchez-Garfias in [17-19].

Definition 1 Let A = {0, 1} and xω ∈ An be a pat-tern. We call characteristic set of xω to the index setTω = {i|xωi = 1} and its cardinality is denoted by |T |.Definition 2 Let A = {0, 1} and xα, xβ ∈ An be two pat-terns then xα ≤ xβ if and only if ∀i ∈ {1, 2...n} it holdsthat xαi = 1→ xβi = 1.

The first step will be to establish a relationship betweenthis characteristic set, the usual order relationship and theLernmatrix. We will do this using the Lemma presented in[17-19].

Lemma 1 Let A = {0, 1} and let xα, xβ ∈ An be twopatterns which may belong or not to the fundamental set,then xα ≤ xβ ⇔ Tα ⊆ T β.

The proof of this lemma appears in [19]. The lemmameans that an order relation between patterns implies an or-der relation between their characteristic set and vice versa.

3 Our Proposed Modification to the Lernma-trix Algorithm

In this section we show how a new algorithm imple-mented on a very well known hetereoassociative memoryas Steinbuch’s Lernmatrix, can improve its performance inthe recalling phase, trying to reduce the problem originatedby subsets among pattern’s characteristic set, on this kindof associative memory. Beside, in the following section wewill show how this algorithm allows us to use this new het-ereoassociative memory as a pattern classifier.Once we have finished with the Lernmatrix’s normal

process, we have obtain a y which represent the class as-sociated with a especific pattern x . If the class vector doesnot present the saturation problem, then the correct classvector has been found. Otherwise, we need to create an adi-tional colum vector s ∈ Aq of dimension q, which will beuseful in the recalling phase in our proposed modified Lern-matrix. This vector will contain in its i-th component, thesum of the i-th row value of theM matrix.

si =nXj=0

mij

Once we have created the s vector, the next step is totake the output class of the normal Lernmatrix process andcreate a new one based on the algorithm presented here:Let z ∈ Aq be the resulting class from the normal Lern-

matrix process, and y ∈ Aq be the resulting class from ourproposal. Each component in our new class colum vector yis given as:

yi =

½1 zi = 1 and si = ∧qk=1sk0 otherwise

Proceedings of the 15th International Conference on Computing (CIC'06)0-7695-2708-6/06 $20.00 © 2006

After this process, the new class vector y representsthe correct association created in the Lernmatrix’s lerningphase, if the input pattern belongs to the fundamental set.In the case of a pattern x̃, which is an altered version ofone of the fundamental patterns, it will be associated withthe pattern y to which T̃⊆T and the cardinality of thecharacteristic set of x is the lowest. However, it couldhappen that there are more than one x which fulfill thecondition. If this latter situation occurs, the resulting classvector y will be ambiguous, presenting more than one com-ponent with value 1, thus presenting imperfect recall.Bellow, some numerical examples using our proposed

modified Lernmatrix are shown.Let p = 3, n = 5 and ε = 1 be the three first association

as presented below:

x1 =

10101

, x2 =11001

, x3 =00001

y1 =

100

, y2 = 010

, y3 = 001

In this case the characteristics sets are denoted by T 1 =

{1, 3, 5} , T 2 = {1, 2, 5} , T 3 = {5} and their order rela-tion is T 3 ⊆ T 1 and T 3 ⊆ T 2 and if we use the normalLernmatrix process it will present the problem of imperfectrecall.The lerning matrixM is shown below:

M =

ε −ε ε −ε εε ε −ε −ε ε−ε −ε −ε −ε ε

Recalling pattern x3 from the fundamental patterns with

the normal Lernmatrix recalling phase will yield:

M · x3 =

ε −ε ε −ε εε ε −ε −ε ε−ε −ε −ε −ε ε

·00001

=

εεε

→ 111

but with the new algorithm the class vector will be as

follows:

STEP 1: Create the s vector as proposed:

s =

3ε3εε

STEP 2: Apply the new algorithm to find the new columnclass vector:

M · x3 =

ε −ε ε −ε εε ε −ε −ε ε−ε −ε −ε −ε ε

·00001

=

εεε

→ 111

= z→ 001

= y3

Now we have obtained the new class vector without am-biguity: we have perfectly recalled y3. Is important to em-phasize that our proposed modification offers an improve-ment to the Lernmatrix recalling capacity by partially solv-ing the problem if imperfect recall related to subsets of char-acteristic sets among fundamental patterns. This is, a per-fect recall will be obtained if we have only created associa-tions of one input-one output.patterns, which is, by the way,one somewhat naiveway to solve the problem of saturation.

4 Experimental Results

In this section the proposed algorithm is tested with theIris Plant Database from UCI. This database contains threeclasses of fifty instances each one, where each class rep-resent one type of iris plant.One class is linearly separablefrom the other two, which are not linearly separable fromeach other. Each element of a class is represented by a dataset of four dimensional real vector, which latter will be con-verted to his binary representation in order to be used withthe Lernmatrix. To perform the experiments a software wasmade.Each of the four real values from the data set was con-

verted to his binary representation and a new vector x wascreated by concatening each of them. With these new vec-tors, which is now a binary pattern, the fundamental setof dimension p was created by taking randomly k patternsfrom each class. The test set are created with the patternsthat were not selected for the fundamental set and listed inorder where the first 50 − k patterns correspond to the irissetosa, the second 50− k patterns to the iris versicolor andthe last to the iris virginica. The test set was created thisway in order to know which patterns correspond to whichclass.The experiments (the randomly patterns selection, thelearning and recalling phase) were repeated one thousandtimes in order to obtain the one hundred best results.

LEARNING PHASE

Once we have both sets, the normal lerning phase of thelearnmatrix was applied to the fundamental set to create the

Proceedings of the 15th International Conference on Computing (CIC'06)0-7695-2708-6/06 $20.00 © 2006

memory. This memory will be used to classify the totalityof the patterns, both the fundamental set as well as the testset created before.

RECALLING PHASE

For the recalling phase, we applied the proposed mod-ification. The system presents to the learnmatrix everypattern, then the class vector produced by the system ischecked automatically to determine to which class corre-spond, in the case of the fundamental set the patterns areperfectly recalled, without ambiguity. However, in the testset some of the patterns presents imperfect recall. In orderto determine to which class correspond each pattern, the 1´sthat belong to a specific class -the first k positions on theclass vector for the fisrt class, the second k positions for thesecond class, and the third k positions for the third class-are added, then the one that has the greatest number of 1´sis selected as the class.Once the system has determined a class for each pattern,

it verifies if they were assigned correctly by verifying if inthe first 50 − k patterns from the test set are only iris se-tosa patterns and in the same way for the other two classes.Then the total of errors produced per class (considering anerror when the recalled class from a specific pattern differsfrom the correct one) are calculated. Then, the system au-tomatically selects the one hundred results that has the lesernumber of errors and those are presented on a list of threecolumns where each of this column represent the number oferrors produced per class. With this data set we can obtainthe accuracy percentage from each class in all the iterations.Two different experiments were done. In the first one

(whose 100 best results are shown in figure 1), 20 patternsfrom each of the three classes were taken to form the fun-damental set (60 patterns in total), leaving 30 patterns perclass for testing purposes. For the second experiment (re-ported in figure 2), the fundamental set was made up bythirty patterns from each class, while the test set contained20 patterns from each class. All patterns were selected ran-domly.In table 1 the best, worst, and average performance of

both experiments are shown.

Table 1. Best, worst, and avarage performancein experiments 1 and 2

Performance 20 TrainingPatterns

30 TrainingPatterns

Best 96% 98%Worst 83.3% 90%Average 90.42% 94.36%

In all the cases the patterns of the fundamental set wereperfectly recalled. When the patterns did not belong to the

767880828486889092949698

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97

Experiments

Percenatge

Figure 1. Experiment 1: 100 best results outof 1000 repetitions with 20 patterns per classin the fundamental set and 30 patterns perclass in the test set

86

88

90

92

94

96

98

100

1 7 13 19 25 31 37 43 49 55 61 67 73 79 85 91 97

Experiments

Percentage

Figure 2. Experiment 2: 100 best results outof 1000 repetitions with 30 patterns per classin the fundamental set and 20 patterns perclass in the test set

Proceedings of the 15th International Conference on Computing (CIC'06)0-7695-2708-6/06 $20.00 © 2006

fundamental set, the algorithm searches for the pattern bywhich it is the closer subset then a class is asigned to thepattern.

5 Conclusions and Future Work

In this paper we have presented a modification to theoriginal Lernmatrix recalling phase. This modification letus obtain a perfect recall when there are only one input-oneoutput patterns associations and only fundamental patternsare presented to the memory. Furthermore, we have beenable to apply the proposed model to pattern classificationwith very positive results. The former can be seen in theexperiments shown, where the proposed model was used onthe Iris Plant Database, a very famous database in MachineLearning. The modified recalling phase allows us to per-fectly recall all fundamental patterns, and more test patternsthan the original Lernmatrix, even achieveing percentagesof perfect classification above 95% (although the averageof the 100 best results out of 1000 repetitions is between 90and 95%).As future work, it remains to test the proposed model

with other databases. Also, the mathematical foundationof this new model is being developed, although no definiteconclusions have been reached yet.

Acknowledgements. The authors would like to thank theInstituto Politécnico Nacional (Secretaría Académica,COFAA, SIP, and CIC), the CONACyT, and SNI fortheir economical support to develop this work.

References

[1] Yáñez-Márquez, C. & Díaz-de-León Santiago, J.L.(2001). Lernmatrix de Steinbuch, IT-48, Serie Verde,CIC-IPN, México

[2] Steinbuch, K. (1961). Die Lernmatrix, Kybernetik, 1,1, 36-45

[3] Steinbuch, K. & Frank, H. (1961). Nichtdigitale Lern-matrizen als Perzeptoren, Kybernetik, 1, 3, 117-124

[4] Kohonen, T. (1972). Correlation Matrix Memories,IEEE Transactions on Computers, C-21, 4, 353-359

[5] Chren, W.A., "One-hot residue coding for high-speednon-uniform pseudo-random test pattern generation,"Circuits and Systems, 1995

[6] Duda,R.O., Hart, P.E. & Stork, D.G.(2001). PatternClassification, USA: Jhon Wiley and Sons

[7] Marqués de Sá, J.P.(2001). Pattern Recognition, Con-cepts, Methods and Applications, Germany: Springer

[8] Schürmann, J. (1996) Pattern Classification, A unifiedview of statistical and neural approaches, USA: JohnWiley

[9] Kuncheva, L.I. (2002). A theoretical Study on SixClassifiers Fusion Strategies, IEEE Transations onPattern Analisis and Machine Intellegence, vol. 24, no2, 281-286

[10] Shalkoff, R.(1992). Pattern Recognition, Statical,Structural and Neural Approaches, USA: John Wiley

[11] Dasarathy, B. V.(1991). Nearest Neighbor(NN)Norms: NN Pattern Classification Techniques, USA:IEEE Computer Society Pres

[12] Pal, S. (1999) Neuro-Fuzzy, Pattern Recognition:Methods in Soft Computing. USA: JohnWiley & Sons

[13] Kohonen, T.: Self-Organization and AssociativeMemory. Springer-Verlag, Berlin Heidel-berg NewYork (1989)

[14] Hassoun, M. H.: Associative Neural Memories. Ox-ford University Press, New York (1993)

[15] Kohonen, T.: Correlation Matrix Memories. IEEETransactions on Computers. 21(4). (1972) 353-359

[16] Yáñez-Márquez, C.: Associative Memories Based onOrder Relations and Binary Operators (In Spanish).PhD Thesis. Center for Computing Research, México(2002)

[17] Sánchez Garfias, F.A., Díaz-de-León Santiago, J.L.& Yáñez Márquez, C. (2004). New Results on theLernmatrix Properties, Research on Computing Sci-ence Series, Vol. 10, IPN, México, pp. 91-102. ISSN1665-9899

[18] Sánchez Garfias, F.A., Díaz-de-León Santiago, J.L.& Yáñez Márquez, C. "Lernmatrix de Steinbuch:condiciones necesarias y suficientes para recuperaciónperfecta de patrones", en Díaz-de-León Santiago,J.L. & Yáñez Márquez, C. (Eds.) "Reconocimientode Patrones. Avances y Perspectivas", ColecciónRESEARCH ON COMPUTING SCIENCE, Vol. 1,ISBN 970189476-6, CIC-IPN,México, 2002, pp. 437-448

[19] Sánchez-Garfias, F. A., Díaz-de-León, J. L. & Yáñez,C. (2004). Lernmatrix de Steinbuch: Avances Teóri-cos , Computación y Sistemas, Vol. 7, No..3, pp. 175-189.ISSN 1405-5546

Proceedings of the 15th International Conference on Computing (CIC'06)0-7695-2708-6/06 $20.00 © 2006