a tabu search-based algorithm for the fuzzy clustering problem

Pergamon

Pattern Recognition, Vol. 30, No. 12, pp. 2023-2030, 1997 © 1997 Pattern Recognition Society. Pubfished by Elsevier Science Ltd

Printed in Great Britain. All rights reserved 0031-3203/97 $17.00+.00

PII: S0031-3203(97)00020-4

A TABU SEARCH-BASED ALGORITHM FOR THE FUZZY CLUSTERING PROBLEM

KHALED S. AL-SULTAN ~ and CHAWKI A. FEDJKI

Department of Systems Engineering, King Fahd University of Petroleum and Minerals, Dhahran 31261, Saudi Arabia

(Received 9 July 1996; in revised form 27 November 1996)

Abstract--The Fuzzy Clustering Problem (FCP) is a mathematical program which is difficult to solve since it is nonconvex, which implies possession of many local minima. The fuzzy C-means heuristic is the widely known approach to this problem, but it is guaranteed only to yield local minima. In this paper, we propose a new approach to this problem which is based on tabu search technique, and alms at finding a global solution of FCE We compare the performance of the algorithm with the fuzzy C-means algorithm. © 1997 Pattern Recognition Society. Published by Elsevier Science Ltd.

Fuzzy clustering Fuzzy C-means algorithm Tabu search technique Global optimization

1. INTRODUCTION

The Fuzzy Clustering Problem (FCP) can be stated as follows: given a number of patterns with several features, it is required to classify them into a number of clusters such that each pattern is allowed to belong to more than one cluster with different degrees of association. This is in contrast to the hard clustering problem in which patterns are to be classified into clusters such that each pattern belongs to one and only one cluster.

To get mathematical representations of FCP, one can represent patterns in s dimensional space, where s is the number of features of patterns, then the FCP can be cast as a nonlinear program as follows:

Minimize

subject to

, iw, z l : w:ll z, ll 2 i=1 j = l

w i j = l , l < i < n , j 1

(1)

(2)

wij >_ O, 1 < i < n, 1 < j <_ c, (3)

where n is the number of patterns (data points) to be clustered; c a fixed and known number of clusters, 2 _ < c < n ; m a scalar, m > 1; x i C R s, 1 < i < n , are data points in the feature space RS; zj C R s, 1 <_ j < c, are (unknown) cluster centers; 11" I[ the Euclidean norm xi with cluster z/; Z - [ z l , z 2 , . . . ,zc] is an s x c matrix; W = {wij} is an n x c matrix.

If the matrix W is known, the problem reduces to finding the cluster centers Z. The solution of the latter problem is straightforward and its global min imum is always attained. If the cluster centers Z are known, then a

* E-mail: [email protected]

problem in W is obtained whose global solution can easily be obtained.

As will be explained in the next section, the fuzzy C- means algorithm (FCMA) is one of the best known approaches to FCR The FCM Algorithm is initiated by selecting a value for W, then the algorithm iterates between computing cluster centers, Z, given W and computing W, given Z. The algorithm terminates when two successive values of Wor Z are equal. It is proven that the algorithm converges. However, it may stop at a local min imum of Problem FCP or even at a saddle point. This is because the function J(w,z) is nonconvex, while J(u;-) and J(.,z) are both convex (the dot is used to replace the known argument of the function). Stopping at a local minimum implies that the solution obtained can be improved further. The proposed tabu search algorithm attempts to obtain a global solution of the FCP which the FCM Algorithm may fail to reach.

The paper is organized as follows. In Section 2, we briefly review the literature for the fuzzy clustering problem. A brief description of the tabu search technique is given in Section 3. Statement of the proposed algo- ri thm is given in Section 4, followed by the parameters of the proposed algorithm in Section 5. Computational experience is presented in Section 6. Finally, our conclusion is given in Section 7.

2. LITERATURE REVIEW

The fuzzy clustering problem has been studied by many researchers. Dunn (1) was the first to formulate this problem as minimizing a function subject to conditions on membership functions. Bezdek (2l suggested a generalized formulation of the problem, and proposed an algo- ri thm for this problem that is analogous to the well known k-means algorithm to the hard clustering problem. This algorithm is the fuzzy C-means algorithm (FCMA). This algorithm is heuristic in nature, and therefore only converges to a partial optimal solution of the problem (i.e.

2023

2024 K.S. AL-SULTAN and C. A. FEDJKI

not to a local optimal solution, let alone global). Inspite of that, it is the most widely known approach to FCP, which might be attributed to its simplicity. Studies of the convergence of the fuzzy C-means algorithm are reported in references (3-5). Efficient implementation of the FCMA is reported in reference (6). Kamel and Selim (7"8) proposed two algorithms for FCP which converge to partial optimal solutions (i.e. like the FCMA) but they claim that these algorithms are faster than the FCMA. Progress in the theory and applications of the FCMA are reported in reference (9). A1-Sultan and Selim ~m~ have proposed a simulated annealing algorithm for globally solving FCP, and provided a limited computational experience. A1-Sultan (11) proposed a tabu search algorithm for the hard clustering problem. M-Sultan and Khan (12)

reported on computational comparisons among the k- means, the simulated annealing, the tabu search, and genetic algorithms for the hard clustering problem. In Section 3, we briefly describe the tabu search algorithm.

3. THE TABU SEARCH TECHNIQUE

Tabu search is a metaheuristic that guides local heuristic search procedures to explore the solution space beyond local optimality. It was introduced by Glo- ver (13-15) specifically for combinatorial problems. Its basic ideas have also been proposed by Hansen (16~ and Hansen and Janmard ~17~ with another name "steepest ascent mildest descent". Since then, tabu search has successfully been applied to flow shop scheduling, (18'199 generalized assignment problems, (2°) architectural design, (21~ time-tabling problems, (22~ interacting hub facilities location, (23) among others.

The tabu search requires the following basic elements to be defined:

• Conf igurat ion is an assignment of values to variables (i.e. a solution).

• A move is a specific procedure for getting a trial solution which is a feasible solution to the optimization problem that is related to the current solution [i.e. a move is a procedure by which a new solution (a neighbor) is generated from the current one by some minor random perturbation of the current solution]. This procedure is usually straightforward for the case of combinatorial optimization, but a lot harder to design for the case of continuous optimization.

• Se t o f candidate moves is the set of all possible moves out of a current configuration. This set is usually countable for the case of combinatorial optimization, but could be large, in which case one could operate with a subset of this set. For continuous optimization problems, the set of possible moves is uncountable and one has to be more creative in defining them.

• Tabu restrictions: This is the most distinctive feature of tabu search. It is what differentiates tabu search from simulated annealing algorithm. The tabu restrictions play a memory role for the search in the sense that recent moves are not allowed to be reversed by making them forbidden (tabu). This is done by forming a list called tabu list, of all of these tabu moves.

• Aspira t ion criteria: These are rules that override tabu restrictions, i.e. i f a certain move is forbidden by tabu restrictions then the aspiration criteria, when satisfied, can make this move allowable (e.g. if a tabu move is better than the best obtained so far by the search, then one can take this move even though it is tabu, or override the tabu restriction).

Given the above basic elements, the tabu search scheme can be described as follows: start with an initial configuration, label this configuration as the current one, evaluate the objective (criterion) function for that configuration. Then, using the current configuration and a procedure for getting trial solutions, generate a certain set of candidate moves and evaluate their corresponding objective function values. If the best of these moves (in terms of the objective function) is not tabu or if the best is tabu, but satisfies the aspiration criteria, then pick that move and consider it to be the new current configuration, otherwise pick the first nontabu move and consider it to be the new current configuration. Repeat the above procedure for a certain number of iterations. On termination, the best solution obtained so far is the solution obtained by the algorithm.

Note that the move that is picked at a certain iteration is put in the tabu list so that it is not allowed to be reversed in the next few iterations. The tabu list has a certain size (which is treated as a parameter), and when the length of the tabu list reaches that size and a new move enters that list, then the first move (i.e. the oldest tabu move) in the tabu list is freed from being tabu and the process con- tinues (i.e. the tabu list is circular). The tabu list size controls the tabu search to either emphasize exploration

or intensification. If the tabu list is small, then intensification is emphasized, i.e. local search around the current point is intensified. If the tabu list is large, then exploration (different regions of the solution space are explored) is emphasized, i.e. points that are far from the current point are examined. The aspiration criteria could reflect the value of the criterion (objective) function, or, if the tabu move results in a value of the criterion function that is better than the best known so far, then the aspirat ion criteria is satisfied and this overrides the tabu restriction.

In Section 4, we develop a new algorithm for solving the FCP based on the above tabu search technique.

4. THE NEW ALGORITHM

In this section, we present our tabu search-based algorithm for the FCP. However, before we state our algorithm, we need to introduce some notation:

Given a set of zj, j = 1 , 2 , . . . , c, one can compute globally the optimal wq, i = 1 , 2 , . . . ,n, j = 1,2 . . . . . c using the following formula:

1 =~{[jx, zA/'l[ . . . . I[} 2/ml

W/j :

Zq~Xi~ q = l , 2 , . . . ,c,

Zj : Xi,

Zj = Xt, t 7~ i.

(4)

A tabu search-based algorithm for the fuzzy clustering problem 2025

The above formula come from solving the set of equations which one obtained by setting the partial derivatives of J(w,z) with respect to W to be zero. The fuzzy C-means algorithm uses equation (4) to get the values of the weights given the values of the centers. Given the weights wij and the centers zj, one can compute the objective function J(w,z) as

z, ii i=1 j 1

Thus, from the above, it is clear that for any given set of centers zj, there corresponds a specific value for the objective function which, for simplicity, is denoted by J. Therefore, we will use our tabu search algorithm for generating the centers zj and consequently it will map into a value for the objective function J. We will operate with three types of centers and their corresponding objective function values.

Remark. Given the weights wij, one can compute the optimal centers zj, j = 1, . . . . using the following formula:

~inl (Wij)mxj j = 1 , . . . ,C. (6) Zj - - n rn 2i=l(Wij)

The above formula come from solving the set of equations which are obtained by setting the partial derivatives of J(w,z) with respect to Z to be zero. When re=l, (i.e. hard clustering) equation (6) represent the centroids of the clusters, and therefore these equations are similar to getting centroids in hard clustering. The fuzzy C-means algorithm uses equation (6) to get centers of clusters given the values of the weights. A scheme for generating trial solutions using wij may be found in reference (10). Therefore, one may choose to operate with wij using tabu search technique, and calculate zjs using equation (6). We have tried this approach, but we empirically found that our original scheme (i.e. to operate with zjs) is more efficient than this latter approach (i.e. to operate with wijs).

Let Z,, Z,, Zb denote the trial, current, and best arrays and J,, J~ and Jb denote the corresponding trial, current, and best objective function values respectively.

We will always operate with centers which we call the current centers Z, and then through moves which we discuss below, we generate trial centers Zt. As the algorithm proceeds, we also save the best centers found so far which are denoted by Zb. Corresponding to these arrays, we also operate with the objective function values &, Jt and Jb respectively.

Now, we discuss how to make moves for the centers, i.e. a move from Z, to Zt. Since we are dealing with tabu search, it is necessary to discretize moves in order to be able to store them in the tabu list. Therefore, we define our moves as follows:

g = d + c~d, (7)

where d is the j th trial center, z j is the j th current center, and d c R s is picked randomly from the directions

d = (di), d / = 0, - 1 , or +1. It was observed that as we get close to the solution, we need very small steps, so we decided to use the following scheme for the direction

multiplier c~. We are now ready to state our algorithm.

S t a t e m e n t o f t h e n e w a l g o r i t h m

Step 1. Initialization. Let Zu be arbitrary centers and J,, the corresponding objective function value. Let Zb = Z,, and Jb = Ju- Select values for NTLM (tabu list size), P (probability threshold), NH (number of trial solutions) and IMAX (the maximum number of iterations for each center), a = l (the direction multipliers) and ,3=0.75 (the iteration reducer). Let k = l , NTL=0, l e t / = 1 and go to step 2.

Step 2. Using Zu, fix all centers and move center zlu by generating NH neighbors (for simplifying notation call it z i) z l , z 2 , . . . , z 5~, and evaluate their corresponding objective function values J ¢ . . . j,Nft [use equation (7) to generate neighbors]. Go to step 3.

Step 3.

1. Sort aT,/, i = 1 , . . . , NH in a nondecreasing order and denote them j~] . . . . ,a[~ NH]. Clearly j } l ]< j[2] . . .~

j}NH]. I f j i l l -~ J b replace k by k+ l .

2. I f thedirect ionusedto generate z [1] i sno t t abuor i f i t i s tabu but j~l] < Jb then let .I = z[ll and J , = j~lI and .c u

go to step 4. Otherwise generate u ~ U(0, 1) [where U(0,1) is a uniform density function between 0 and 1 ], if Jb < a[! 1] < J,, and u > P then let Ju = a@ and zt, = z [1] and go to step 4, otherwise, go to (4) of step

3. 3. Repeat (2) of step 3 for z121,..., zl yH] until a neighbor

is selected, then go to step 4. 4. If k > IMAX go to step 5, otherwise; select a new set

of neighbors by going to step 2.

Step 4. Insert the direction that is used to generate zl, (if not already in the tabu list) at the bottom of the tabu list, and let NTL = N T L + I . If NTL = N T L M + I , delete the top of the list and let NTL = N T L - 1 . If Jb > J , , let Jb -- J,, and Zb = Z,,. Go to step 2.

Step 5. If l < c (number of clusters) replace 1 by l+ 1, and go to step 2. Otherwise set IMAX=/3(IMAX). If IMAX > 1 then let 1=1 and go to step 2; otherwise, stop (Zb represents the best centers, and Jb is the corresponding best solution).

5. PARAMETERS OF THE ALGORITHM

In this section, we discuss the parameters of the algorithm, and their expected effects on the performance. These parameters are described below:

• Tabu list size (NTLM). The tabu list contains the history of the search, and it is the most distinctive feature of tabu search technique. It assumes that the last NTL moves will not be reversed, and hence the solutions in the immediate past will not be revisited.

2026 K.S . AL-SULTAN and C. A. FEDJKI

N T L M is the m a x i m u m n u m b e r o f m o v e s that

one can store in the list, and hence the larger the

va lue o f N T L M , the s t ronger the m e m o r y o f the search

and hence the a lgor i thm encourages more diversifica- tion. On the other hand, the smal le r the va lue o f

N T L M , the less m e m o r y the search has, and hence

the a lgor i thm e m p h a s i z e s m o r e intensification in the search.

• Probability Threshold (P). The probabi l i ty th reshold P

is u sed to al low m o v e s that are t abu and better than the

p resen t current solut ion (a l though worse than the bes t

solut ion found so far) to be e x a m i n e d as this m a y lead

to a bet ter solution.

• Number of trial solutions (NH). In tabu search, one

inves t iga tes N H trial solut ions for dec id ing on the nex t

m o v e to take. Hence the larger the value o f N H the

more ne ighbors are e x a m i n e d and the m o r e the search

is intensif ied, whi le smal le r va lues o f N i l imp ly fewer

ne ighbors to e x a m i n e and hence the search emphas i ze s

diversif icat ion.

• The maximum numberofnonimproving movesforeach center (IMAX). This pa rame te r decides on how m a n y

n o n i m p r o v i n g m o v e s are a l lowed for each center

before go ing to the nex t one. One wou ld expec t that

for more coupled p rob lems I M A X should be smaller. It

was observed that as one m o v e s toward the solution,

the t ime needed to e x a m i n e a g iven center is reduced,

so it was decided to use a var iable pa ramete r I M A X

ins tead o f f ix ing it. Hence one starts by a certain value

I M A X unti l no i m p r o v e m e n t is poss ib le for each

Table 1. Results for the fuzzy C-means, and tabu search algorithms for problems with: Number of Patterns--10, Dimension-2, Number of Clusters=3

*Pb *C-means *Tabu *Iter *Dev. C-means Tabu Iter Dev.

m-- l .2 m=1.5 1 139.82 30.66 555 0.7807 100.56 28.87 595 0.7129 2 142.99 32.38 667 0.7736 102.84 31.43 603 0.6944 3 92.90 21.65 633 0.7670 66.82 19.65 574 0.7059 4 125.01 26.26 587 0.7899 89.91 25.71 611 0.7140 5 122.71 31.28 598 0.7451 88.26 30.56 662 0.6538 6 115.04 22.97 578 0.8003 82.74 22.78 741 0.7247 7 104.88 43.65 634 0.5838 75.43 40.08 615 0.4686 8 79.58 39.19 647 0.5075 57.24 26.43 674 0.5383 9 128.00 36.39 530 0.7157 92.06 31.18 566 0.6613 10 99.77 45.58 568 0.5431 71.76 41.20 545 0.4259 Av 599 0.7007 618 0.6300

m=2 m=3 1 58.06 23.40 540 0.5970 19.35 10.84 631 0.4398 2 59.38 25.37 613 0.5728 19.79 11.41 609 0.4234 3 38.58 14.99 631 0.6115 12.86 6.76 604 0.4743 4 51.91 20.80 603 0.5993 17.30 9.45 598 0.4538 5 50.95 24.48 571 0.5195 16.98 10.72 562 0.3687 6 47.77 19.75 583 0.5866 15.92 9.65 558 0.3938 7 43.55 28.95 651 0.3352 14.52 11.77 583 0.1894 8 33.05 20.35 68t 0.3843 11.02 7.86 628 0.2868 9 53.15 23.38 570 0.5601 17.72 10.87 562 0.3866 10 41.43 30.07 619 0.2742 13.81 11.61 624 0.1593 *Av 606 0.5040 595 0.3576

m--5 1 2.15 1.45 561 0.3256 2 2.20 1.90 535 0.1364 3 1.43 0.93 612 0.3497 4 1.92 1.27 658 0.3385 5 1.89 1.42 590 0.2487 6 1.77 1.38 682 0.2203 7 1.61 1.45 565 0.0994 8 1.22 0.98 739 0.1967 9 1.97 1.36 546 0.3096 10 1.53 1.36 531 0.1111 Av 601 0.2336

*Definitions: Pb: problem number; Iter: number of iterations to get the tabu search solution; C-means: Objective function value obtained by fuzzy C-means; Tabu: Objective function obtained by the proposed tabu search algorithm; Dev: Relative deviation of the tabu search objective function value from the fuzzy C-means' one; Av: Average number of iterations and average relative deviation.

A tabu search-based algorithm for the fuzzy clustering problem 2027

center, t hen reduce it by a fac tor /3 (see defini t ion of /3

below) unt i l it goes be low 1, w h i c h cor responds to our

s topping criteria.

• The reduction factor for the maximum number of nonimproving moves for each center (/3). As we m e n - t ioned earlier, i f I M A X n o n i m p r o v i n g m o v e s are

pe r fo rmed , then the nex t center is considered. W h e n

all centers are cons idered , then I M A X is r educed by a

factor /3, where 0 < / 3 < 1. Clearly, the smal le r the

va lue of /3 , the fas ter I M A X goes be low 1 and hence

the fewer pa s se s t h rough the centers the a lgor i thm

makes , bu t this could be at the expense o f the solut ion

quality. • The direction multiplier (cO. Th i s is a scalar wh i ch

represents the step l eng th a long a certain direct ion for

f ind ing a trial solut ion (a ne ighbor) f r om the current

one. c~ m u s t be posit ive, and the h ighe r the va lue o f c~,

the far ther the ne ighbor f r om the current solution. One

shou ld r e m e m b e r that c~ can be r educed w h e n one gets

c loser to the solut ion.

5.1. Best values of the parameters

We have tes ted wi th several p rob lems to get the bes t

va lues o f the above parameters , and we have found the

fo l lowing va lues for the test problems:

N T L M

P N H

oz

I M A X

/3

~5" total n u m b e r o f poss ib le directions, or

(3 n - 1 ) /15 (n be ing the n u m b e r o f patterns).

0.97. 10% of the total n u m b e r o f poss ib le directions.

Each t ime a center is considered, init ial ize c~ to

1, and reduce it every t ime by a factor o f 0.8.

Ini t ia l ized at 40 then reduced by a factor o f

0.75 after one pass t h rough all centers .

O.75.

6. COMPUTATIONAL EXPERIENCE

The a lgor i thm presen ted in Sect ion 4 has been coded

in F O R T R A N , and tes ted on an I B M 3090 m a c h i n e

Table 2. Results for the fuzzy C-means, and tabu search algorithms for problems with: Number of Patterns=10 Dimension-2 Number of Clusters=4

Pb C-means Tabu Iter Dev. C-means Tabu Iter Dev.

m--1.2 m--1.5 1 132.00 20.30 773 0.8462 87.09 17.26 805 0.8018 2 135.00 19.67 850 0.8543 89.06 18.41 869 0.7933 3 87.70 13.51 768 0.8460 57.86 13.00 732 0.7753 4 118.02 19.91 859 0.8313 77.86 19.31 793 0.7520 5 115.85 21.47 826 0.8147 76.43 21.29 821 0.7214 6 108.61 17.38 864 0.8400 71.66 17.13 850 0.7610 7 99.02 33.88 832 0.6578 65.33 25.02 814 0.6170 8 75.13 15.83 803 0.7893 49.57 15.00 792 0.6974 9 120.84 17.63 769 0.8541 79.72 17.43 752 0.7814 10 94.19 27.18 702 0.7114 62.14 25.25 683 0.5937 Av 804 0.8045 791 0.7294

m=2 m=3 2 44.53 13.49 805 0.6971 11.13 4.94 803 0.5562 3 28.93 9.02 817 0.6882 7.23 3.44 841 0.5242 4 38.93 11.25 826 0.7110 9.73 4.30 783 0.5581 5 38.22 14.98 861 0.6081 9.55 5.38 797 0.4366 6 35.83 13.70 873 0.6176 8.96 3.51 975 0.6083 7 32.66 18.48 824 0.4342 8.17 5.74 814 0.2974 8 24.79 10.34 890 0.5829 6.20 3.37 831 0.4565 9 39.86 14.43 753 0.6380 9.97 5.09 860 0.4895 10 31.07 17.49 754 0.4371 7.77 5.46 799 0.2973 Av 829 0.6121 829 0.4784

m=5 1 0.68 0.39 815 0.4265 2 0.70 0.38 759 0.4571 3 0.45 0.28 874 0.3778 4 0.61 0.37 762 0.3934 5 0.60 0.39 719 0.3500 6 0.56 0.29 893 0.4821 7 0.51 0.41 877 0.1961 8 0.39 0.29 869 0.2564 9 0.62 0.39 778 0.3710 10 0.49 0.40 813 0.1837 Av 815 0.3494


Table 3. Summaries of the average deviation of the tabu search objective function value from the fuzzy 10 problems for each set)

C-means' one (average for

*Set *N *S *C **1.2 **1.5 *'2.0 *x3.0 *'5.0

1 10 2 3 0.7007 0.6300 0.5040 0.3576 0.2336 2 10 2 4 0.8045 0.7294 0.6121 0.4748 0.3494 3 10 3 3 0.5755 0.4718 0.3225 0.1737 0.0844 4 10 3 4 0.7033 0.5914 0.4136 0.2457 0.1478 5 10 5 3 0.4290 0.2869 0.1446 0.0437 0.0086 6 10 5 4 0.5932 0.4258 0.2348 0.0829 0.0217

7 50 2 3 0.5693 0.4519 0.3058 0.1617 0.0725 8 50 2 4 0.6957 0.5842 0.4151 0.2317 0.1067 9 50 3 3 0.3686 0.2340 0.1084 0.0238 0.0019 10 50 3 4 0.4858 0.3233 0.1543 0.0346 0.0020 11 50 5 3 0.2150 0.0838 0.0092 0.0000 0.0000 12 50 5 5 0.2916 0.1210 0.0132 0.0086 0.0000

13 100 2 3 0.5478 0.4363 0.2891 0.1479 0.0630 14 100 2 4 0.6875 0.5668 0.3937 0.2025 0.0793 15 100 3 3 0.3333 0.1998 0.0836 0.0114 0.0000 16 100 3 4 0.4565 0.2990 0.1289 0.0165 0.0000 17 100 5 3 0.1666 0.0500 0.0010 -0.0001 -0.0001 18 100 5 5 0.2281 0.0760 0.0013 0.0000 -0.0001

Definitions: S: dimension of the problems; C: number of clusters considered; N: number of patterns for the problems; Set: the problem set (each set consisting of 10 problems); **m: degree of membership fuzziness (m=l corresponds to the hard clustering problem).

0.8

g "g 0.6

~0 .4

0.2

average deviation as a function of m for N=IO S=2 C=3 i t i k i i

I P J r

2 2; 3 3; 4 415 m

Fig. 1. Average deviation as a function of m for N--10, S=2, C=3.

0.75 average deviation as a function of N for m=1.2 S=2 C=3

i ~ i i i i E i

o ~ 0.7

"~ 0.65

g 0.6

> 0.55

0"~0 2; 3; 4; 5; 6; 7~0 8qO 90 100

Fig. 2. Average deviation as a function of N for m = 1.2, S=2, C=3.

runn ing the Un ix Opera t ing S y s t e m on several test p roblems. We have used the fo l lowing set o f test pro-

b lems: genera te xl r a n d o m l y in the range [ - 5 , 5 ] ,

i = 1 , . . . , n. W e have tes ted the a lgor i thm for var ious

combina t ions o f m, n, c and s. T hese combina t ions inc lude va lues for m = 1.2, 1.5, 2.0, 3.0, 5.0, va lues for

n = 10, 50 and 100, va lues for c = 3 , 4 and values for s = 2 , 3 and 5. A sample o f these resul ts is

shown in Tables 1 and 2, bu t the who le resul ts are

s u m m a r i z e d in Table 3. One can clearly see that in mos t o f the cases our a lgor i thm is better than the fuzzy

C - m e a n s a lgor i thm, and by a s igni f icant factor. The

0.8

._o_. 0.7

-u 0.6 &

~>o.5

0.4 2

A tabu search-based algorithm for the fuzzy clustering problem

average deviation as a function of S for N=IO m=1.2 C=3

l I t ~ I

2.5 3 3,5 4 4.5 S

Fig. 3. Average deviation as a function of S for N=10, m=l.2, C=3.

2029

0.85

._o = 0.8 "~

>~0.75

O~

0.7

average deviation as a function of C for N=IO m=1.2 S=2 i + i i i L i i t

J

J J

J J

' i i i ' ' i 3.1 3.2 3 3 34 3 5 3.6 3.7 3.8 3 9 C

Fig. 4. Average deviation as a function of C for N--10, m=1.2, S=2.

improvement becomes more pronounced as m, the number of patterns, and their dimension get smaller and the number of clusters gets larger. This is illustrated in Figs 1-4. For some cases, we get the same solution as the fuzzy C-means algorithm. Only in very few cases was the fuzzy C-means insignificantly better, but we feel that if the tabu search algorithm is run longer, it will yield the same results as the fuzzy C-means algorithm (we con- jecture that the solutions obtained by the two algorithms are the global solutions of these test problems for the last

cases).

7. CONCLUSION

In this paper, we have proposed a tabu search algorithm for solving the fuzzy clustering problem. The algorithm aims at solving the problem globally. The algorithm has been implemented and tested on several test problems. The proposed algorithm has been found to outperform the fuzzy C-means algorithm considerably in most of the cases. One might implement other features of the tabu search algorithm, such as long term memory,

among others.

R E F E R E N C E S

1. J. C. Dunn, A fuzzy relative of the ISODATA process and its use in detecting compact well-separated clusters, J. Cybernet. 3(3), 32-57 (1974).

2. J. C. Bezdek, Fuzzy mathematics in pattern classification, Ph.D. Dissertation, Department of Applied Mathematics, Cornell University, Ithaca, New York (1973).

3. M. A. Ismail and S. Z. Selim, Fuzzy C-means: Optimality of solutions and effective termination of the algorithm, Pattern Recognition 19(6), 651-663 (1986).

4. S. Z. Selim and M. A. Ismail, On the local optimality of the fuzzy isodata clustering algorithm, IEEE Trans. Pattern Analysis Mach. lntell. PAMI-8(2), 284-288 (1986).

5. J. C. Bezdek, R. Hathaway, M. Sabin and W. Tucker, Convergence theory for fuzzy c-means: Counterexamples and repairs, IEEE Trans. System Man Cybernet. 17(5), 873-877 (1987).

6. R. L. Canon, J. C. Bezdek and J. V. Dave, Efficient implementation of the fuzzy C-means clustering algorithm, IEEE Trans. Pattern Recognition Mach. Intell. 8(2), 248- 255 (1986).

7. M. Kamel and S. Z. Selim, New algorithms for solving the fuzzy clustering problem, Pattern Recognition 27(3), 421- 428 (1994).

8. M. Kamel and S. Z. Selim, A relaxation approach to the fuzzy clustering problem, Fuzz3" Sets Systems 61, 177-188 (1994).

9. J. C. Bezdek, Analysis of Fuzzy Information, Vol. 3. CRC Press, Boca Raton, Florida (1987).

10. K. S. A1-Sultan and S. Z. Selim, A global algorithm for the fuzzy clustering problem, Pattern Recognition 26, 1357- 1371 (1993).

11. K. S. M-Sultan, A tabu search approach to the clustering problem, Pattern Recognition 28(9), 1443-1451 (1995).

12. K. S. A1-Sultan and M. M. Khan, Computational experience on four algorithms for the hard clustering problem, Pattern Recognition Lett. 17, 295-308 (1996).

13. F. Glover, Future paths for integer programming and linkage to artificial intelligence, Comput. Operations Res. 13. 533-549 (1986).

14. E Glover, Tabu search--Part I, ORSA Z Comput. 1(3), 190-206 (1989).

15. E Glover, Tabu search--Part IX, ORSA J. Comput. 2(1), 4- 32 (1990).

16. P. Hansen, The steepest ascent mildest descent heuristic for combinatorial programming, Congress on Numerical


Methods in Combinatorial Optimization, Capri, Italy (1986).

17. E Hansen and B. Jumard, Algorithm for the maximum satisfiability problem, RUTCOR Research Report RR 43- 87, Rutgers, New Brunswick, New Jersey (1987).

18. E. Nowicki and C. Smutnicki, A fast taboo search algorithm for the flow shop, Report ICT PRE 8/94, Technical University of Wroclaw, Eur. J. Open Res. 91(1), 160-175 (1996).

19. E. Talllard, Some efficient heuristic methods for the flow shop sequencing problem, Eur. J. Oper. Res. 47, 65-74 (1990).

20. M. Laguna, J. E Kelly, J. L. Gonzalez-Velarde and F. Glover, Tabu search for the multilevel generalized assignment problem, Eur. J. Open Res. 82, 176-189 (1995).

21. J. A. Bland and G. R Dawson, Tabu search and design optimization, Computer-aided Design 23(3), 195-201 (1991).

22. A. Hertz, Tabu search for large scale time tabling problems, Eur. J. Oper. Res. 51, 39~47 (1991).

23. D. Skorin-Kapov and J. Skorin-Kapov, On tabu search for the location of interacting hub facilities, Eur. J. (?per. Res. 73, 502-509 (1994).

About the A u t h o r - - K . S. AL-SULTAN was born in Algaseem, Saudi Arabia, in 1963. He received B.Sc and M.S. degrees in Systems Engineering from King Fahd University of Petroleum and Minerals (KFUPM), in 1985 and 1987, respectively. M.S. degree in Mathematics and Ph.D. degree in Industrial and Operations Engineering both from the University of Michigan, Ann Arbor, in 1990. Since 1990, he has been a faculty member in the Department of Systems Engineering at KFUPM, Dhahran, Sandi Al'abia, where he is now an Associate Professor. He has served as a Chairman of that Department for the period 1993-1996. At present, Dr A1-Sultan is the Dean of the College of Computer Sciences and Engineering at KFUPM. Dr A1-Snitan is a registered professional engineer in the State of Michigan, U.S.A., and is a member of the Institute of Industrial Engineers, the Institute for Operation Research and Management Sciences, and the Mathematical Programming Society. Dr A1-Sultan's research interests are in the areas of mathematical Programming, design and analysis of algorithms, cluster analysis, quality control, and maintenance management.

About the A u t h o r - - C . A. FEDJKI was born in Constantine, Algeria, in 1962. He received an "Ingeniorat d'etat" in Computer Science from the University of Constantine in 1985, and a Master of Science in Operations Research from Carnegie-Mellon University in 1989. From 1989 to 1992 he was a Lecturer at the University of Serif (Algeria) and from 1992 to 1994 a Lecturer at the University of Biskra (Algeria). He is currently a Ph.D. Student at King Fahd University of Petroleum and Minerals (KFUPM). He is a member of the Institute of Operations Research and Management Science (INFORMS). Mr FEDJKI's research interests are in mathematical programming, combinatorial optimization and cluster analysis.

a tabu search-based algorithm for the fuzzy clustering problem

Documents