Kohonen’s Feature Maps Applied to Ordered Clustering Applications Arati B. Baruah, Boeing Computer Services, P.O. Box 24346, MS 7L-22, Seattle, WA 98124

Les E. Atlas, Alistair D.C. Holden, Dept. of Electrical Engineering, FT-10, University of Wash- ington, Seattle, Wa 98195

Abstract A novel application to a computer aided design data base using Kohonen’s self-organizing feature maps is discussed. An input representation is chosen to decode the input geometric drawings into a vector of 12 elements for each drawing. The input vectors are than mapped to one-dimensional Kohonen’s feature maps. Upon convergence the output nodes organized themselves in ascending or descending order, resulting in a sorted output of the clusters of similar geometric figures. Introduction Kohonen’s self-organizing feature maps [ 101 have the important characteristic of significant spa- tial organization across neurons. For almost all other techniques of clustering there is no topolog- ical relationship between clusters. In Kohonen’s feature mapping the output units are arranged in a one or higher dimensional array in a geometrical way. The weight update is made to the winner plus the neighboring losers towards the average input direction so that a unit eventually succeeds in the competition. Reference [l] introduces the idea of ordered mapping to vector quantization (VQ). VQ [7,11] is a lossy compression technique and is a multi-dimensional generalization of scalar quantization (SQ). In SQ the labeling of codebook indices is ordinal. For VQ, the concept of ordered labeling is not straightforward. However, the output nodes of Kohonen’s feature map are able to organize an order on the input vectors. In this paper we use the ordering property to compensate for input order-dependency for cluster- ing a geometric data set. In [3] and [4] we showed computer simulations of a geometric data set clustering using ART1, where if we scramble the order of presentations we get a different cluster- ing. When Kohonen’s method is used, once the weights are trained, changes in input order will not result in changes in the codebook number. In this application, the method also provides “con- tinuity of representation” where small perceived changes in the input objects lead to small changes in the output. Many representations, unfortunately, do not have this very desirable prop- erty. Here we use this ordering property to find the similarities (or dissimilarities) of objects in dif- ferent clusters. Consider a large Computer Aided Design (CAD) data base of two or three dimensional geometric figures[l3]. To illustrate the concepts the small set of ten figures shown in Fig. 1 will be used. Our goal is to easily and quickly cluster the similar drawings in coarse and fine groupings. An impor- tant auxiliary goal is to have clusters ordered to allow easy browsing through drawings. If we use a competitive network [8,9, 121 we need to normalize the input vectors. We can get a good clus- tering of the input using a competitive net (one winner), however, we cannot easily order or change the number of the clusters. Using Kohonen’s self-organizing feature map we can use a non-normalized input. The next two sections describe Kohonen’s self-organizing feature map algorithm and our input representation for a geometric shape.

The Algorithm For this work we employ the standard Kohonen's self-organizing feature maps. The algorithm employs an array of N neurons receiving a random sequence of input samples from a space V to be mapped onto A. Each input v E V is represented as a vector of activities VI, v2, ..., vd, where d is input dimension of space K Each neuron is labeled by its position r E A and completely con- nected with input with weight vectors w,. Let the neuron s be the one which is most vigorously excited by the input. This neuron is determined by

11 w, - v I I = minll w,- V I ] (1)

aW = Eh,,(v-wW,) (2)

Then the weights in the neighborhood of the neuron s is updated as,

E, h,, each are a number between 0 and 1 and is a prespecified adjustment function of the distance r-s; E is zero outside the neighborhood. The algorithm is summarized as follows: Assign suitable (random, if no a priori information is available) initial values to the weights. 1. Present an input pattern. 2. Determine the location s for which the distance given in (2) is minimal. 3. Update weights in the neighborhood of s w y w = Wold , + ~ h , , ( v - w ; ' ~ )

and continue with step 1.

Input Representation We used the same kind of input feature set for e ch object th

(3)

Burr [2] used for spoken nd h nd written text recognition, adapted to our class of objects. He describes a 7-segment and a 13-seg- ment decoder for decoding the input letters. In our case we found a 12-segment decoder as shown in fig.1 to be capable of picking up the features we are interested in. Here we hand coded the 12- segments for our ten inputs as shown in fig.2. However, for a large number of input objects a sim- ulation computer program would be useful. Thus we have a 12 element vector for each input data- object where each feature is an element. Even though the encoding is non-invertible, the general shapes of an object can be visualized from the feature vectors. Reference [ll deals with raw image pixel data which leads to a large amount of data to represent each object. In this paper we used a much smaller feature set. The method here is illustrated for only a few objects but is extendable to a large data base. Feature Code Used Twelve features, fi, were used as follows:

fl and f3: f l is the length of the projection of the part of the object in region A (Fig. 3a) on to side ac, as shown in fig.3b. In the same figure f3 is the projection on to the side ge of the part enclosed in region B.

597

f2 and f4: f2 is the length of the projections of the part of the object enclosed in region D (fig. 4a) on to side ce as shown in fig. 4b. Similarly, f4 is on to the side ag from the region C.

fs: Projection of the part enclosed in the triangle boc to the diagonal oc (fig. 5a and 5b).

fs: Projection of the part enclosed in the mangle cod to the diagonal oc (fig. 6a and 6b).

The other six remaining triangles similarly form the projections f7-fl2.

The Results Figure 7 shows the clustering for a codebook of six. The six clusters of ten input vectors are arranged in ascending order. Note that, if the index goes one up or one down we can Observe slight changes in the objects. Figure 8 shows the output groupings for a codebook of four (four clusters). As seen in the equation (3), Kohonen’s algorithm has two parameters to vary, E, h,,, the neighborhood size and the gain term. In our computer simulation, the neighborhood size used are 6 ,4 ,2 ,0 for codebook six, and 4,2,0 for codebook four. In both cases h is linearly reduced from 0.9 to 0.001. At least 200 iterations are made for each case. In figures 7 and 8, we see how the objects form a cluster and also produce an order between clusters. This order, of course, will depend on the input representation. The better the representation, the better will be the meaning of the order. Thus, when we perceive small changes in the objects the index changes only by a lit- tle. This is the “continuity of representation” referred to before. It is interesting to observe that for a codebook of six (fine grouping, as opposed to a codebook of four), the index 1 contains no objects, thus indicating that the index zero and two are quite different, which is actually the case. The choice of initial weights for the codebook vectors may make slight variations in clustering. We tried both random weights and then codebook vectors initialized to first few input vectors. However, the change of index did not vary by more than one index. Thus, in a large data base for similar figures one should browse into two or three consecutive codebook indices. Discussion Since we analyzed ten cases here the problem of convergence is minimal. We can easily watch the output and perform enough iterations to obtain convergence to proper topological order. It has been shown that it takes about N3 steps to get convergence, where N is the number of output nodes [6]. Thus if we have 2000 objects in the data base and we want a codebook of 200 we need to perform 2OO3 iterations. This would take an excessive amount of computational time. How- ever, for our database both the shape and the size of the objects are important. Thus, we can par- tition the data base in groups of sizes of desired ranges and perform Kohonen’s maps on each group separately. It is very useful to get sorted clusters of similar shapes as has been done here. Advantages of ordered output for progressive bit rate data transmissions have already been shown in [l]. We plan to extend this work to full three dimensional CAD data base.

Acknowledgment The authors thank Professor Jenq-Neng Hwang for his input on input representation. Arati Baruah enjoyed the discussions of convergences, specially for the 256 code book case, with Shyh-

598

Rong Lay and is thankful of his contribution. References [ 11 Riskin, E.A, Atlas, L.E., Lay S., “Ordered Neural Maps and Their Applications to Data Com- pression’’, Proceedings of First IEEE-SP Workshop on Neural Networks for Signal Processing”, Sept. 29-Oct. 2, 1991, Princeton, NJ. [2] Burr, David J., “Experiments on Neural Net Recognition of Spoken and Written Text”, IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 36, No. 7, July 1988. [3] Baruah, Arati B, Welti, Russell C., “Adaptive Resonance Theory and the Classical Leader Algorithm”, IJCNN-91 (International Joint Conference on Neural Networks), Seattle, July 8- 12, 1991 [4] Baruah, A.B., Holden A.D.C., “Adaptive Resonance Theory and The Classical Leader Algo- rithms, Similarities and Additions”, Submitted to ANNIE-91 (Artificial Neural Networks in Engi- neering), Nov. 10-12, Rolla, Missouri, [5] Carpenter, Gail A., Grossberg, Stephen, “A Massively Parallel Architecture for a Self-organiz- ing Neural Pattern Recognition Machine”, Computer Vision, Graphics, and Image Processing,

[6] Geszti, T, “Physical Models of Neural Networks”, Chapter 10, World Scientific, 1990. [7] Gray, R.M., “Vector Quantization”, IEEE ASSP Magazine, 1:4-29, April 1984 [8] Grossberg, Stephen, “Competitive Learning: from Interactive Activation to Adaptive Reso- nance”, Cognitive Science, 1987, No. 11, pp. 23-63. [9] Hertz, J., Krogh A., Palmer R.G., “Introduction to the Theory of Neural Computation”, Addi- tion-Wesley Publishing Company, Redwood City, CA 94065, 1991. [ 101 T. Kohonen, Self-Organization and Associative Memory, 2nd Edition, Springer-Verlag, Ber- lin, 1988 111 John Makhoul, Salim Roucos, Herbert Gish, “Vector Quantization in Speech Coding”, Pro- ceedings of the IEEE, vol. 73, No. 11, November 1985 1121 Richard P. Lippmann, “An Introduction to Computing with Neural Nets”, IEEE ASSP Maga- zine, April 1987 [131 Caudell, Thomas P, Smith, Scott D.G., Johnson, G. Craig, Wunsch, Donald C., “An Applica- tion of Neural Networks to Group Technology”, Proceedings of SPIE, May 1991, Orlando, Flori- da.

1987, NO. 37, pp. 54-115.

599

Fig. 1 Input data set

f2

..................................................... I Fig. 2 Twelve Segment Decoder

"l