Download - A Hybrid Method to Categorical Clustering
A Hybrid Method to Categorical Clustering
學生:吳宗和Advisor : Prof. J. Hsiang
Date : 7/4/2006
Outlines
Introduction Motivation Related Work Research Goal / Notation Our Method Experiments Conclusion Future Work
Introduction
Data clustering is to partition a set of data elements into groups such that elements in the same groups are similar, while elements in different groups are dissimilar. [1] The similarity must be well-defined by developer.
Data elements can be Recorded as Numerical values. (Degree: 0o~360o ) Recorded as Categorical values. (Sex : { Male, Female})
Find the latent structure in the dataset. Clustering Methods purposed for numerical data
can’t fit for categorical data, because lack of a proper measure of similarity[21].
Example:
An instance of the movie database. Attributes
InstancesDirector Actor Genre
Partition
1
Partition
2X-man III Brett Ratner Hugh Jackma
n Action
AdventureC1 G1
Superman returns
Bryan Singer Brandon Routh ActionAdventure
C2 G1
X-man II Bryan Singer Hugh Jackman
ActionAdventure
C1 G1
Spiderman III Sam Raimi Tobey Maguire Science Fiction Fantasy
C3 G2
Van Helsing Stephen Sommers
Hugh Jackman
ActionAdventure
C1 G1
{Brett Ratner, Bryan Singer,
Sam Raimi,
Stephen Sommers}
{Hugh Jackman,
Brandon Routh,
Tobey Maguire}
{ActionAdventure,
Science Fiction Fantasy}
Outlines
Introduction Motivation Related Work Research Goal / Notation Our Method Experiments Conclusion Future Work
Motivation
Needs, For database, much of the data in databases are
categorical (described by people). For web navigation, web documents can be categorical
data after feature selection. For knowledge discovery, find latent structure in data.
The difficulties to solve clustering categorical data, data elements with categorical values they can take are
not ordered. no intuitive measure of distance.[21] methods specified for clustering categorical data are not
easy to use.
Motivation (cont’)
Problems in Related work Hard to understand or use. Parameters are data sensitive. Most of them CANNOT decide the number of groups when cluste
ring.[3,13,17,18] Our gauss/assumption.
Need All new method? Reuse the methods proposed for numerical data.
( measure of distance can’t fit the categorical data ) Or only NEED a good measure of similarity / distance between t
wo groups. REUSE the framework for clustering numerical data. ( intuitive ap
proach, fewer parameters, and less sensitive for data ) Get better clustering result.
Related Work
Partition Based K-modes [Z. Huang. (1998)] Monte-Carlo algorithm [Tao Li, et. al 2003] COOLCAT [Barbara, D., Couto, J., & Li, Y. (200
2)] Agglomerative Based
ROCK [Guha, S., Rastogi, R.,& Shim, K. (2000)] CACTUS [ Ganti, V., Genhrke, J., & Ramarkrish
nan, R. (1999)] Best-K ( ACE )[Keke Chen, Ling Liu (2005)]
Summarization of relate work
Local
MinimalDecide
ParametersDecide the number of
groups
K-Modes Yes No Human Monte-Carlo
algorithmNo No Human
COOLCAT Yes No Human
ROCK Yes Hard Human CACTUS Yes Hard Human Best-K ( ACE )
Yes NoMachine(Quality is not good)
Outlines
Introduction Motivation Related Work Research Goal / Notation Our Method Experiments Conclusion Future Work
Research Goal
Propose a method A measure of similarity between two groups. Reuse the gravitation concept.( for intuition,
fewer parameters ) Decide the proper number of groups by
machine. Strength the clustering result.( for better result )
Notation
D: data set of N elements p1, p2, p3, …,pn ,|D| = n.
Element pi is a multidimensional vector of d categorical attributes, i.i.e. pi = <ai
1,ai2,…,ai
d> , where aij Aj, Aj is th
e set of all possible values aij can take, and A
j is a finite set. For related work, K : an integer given by user,
then divide elements into K groups G1,G2,…,GK such that G∪ i=D and i,j , ij, Gi∩Gj =ψ.
Outlines
Introduction Motivation Related Work Research Goal
Our Method A measure of similarity Step 1: Gravitation Model Step 2: Strength the result.
Experiments Conclusion Future Work
Our method
One similarity function, two steps. Define a measure of similarity between two
groups. Use the similarity to be the measure of distance.
Two steps. Step 1.
Use the concept of gravitation model. Find the most suitable group k.
Step 2. Conquer the local optimal. Optimize the result.
The Intuition of similarity
Based on the structure of the group. Each group has its own probability distribution of each attribute
and can be represented as its group structure. Groups are similar if their have very similar probability
distribution of each attribute.
Gi Gj
Similarity(Gi,Gj)
100%
90%
80%
70%
60%
50%
40%
30%
40%
50%
60%
70%
Y軸 Attributes distribution of group Gi
A1=Va11
A2=Va21
A3=Va31
A4=Va41
A5=Va51
A6=Va61
A7=Va71
A8=Va81
A9=Va91
A10=Va10,1
A11=Va11,1
A12=Va12,1
100%
90%
80%
70%
60%
50%
40%
30%
40%
50%
60%
70%
Y軸 Attributes distribution of group Gj
A1=Va11
A2=Va21
A3=Va31
A4=Va41
A5=Va51
A6=Va61
A7=Va71
A8=Va81
A9=Va91
A10=Va10,1
A11=Va11,1
A12=Va12,1
The Similarity function
A group Gi = <A1,A2,A3,…,Ad>, Ar is a random variable for all r = 1…d, p(Ar=v | Gi) is the probability, when Ar=v.
Entropy of Ar in group Gi : entropy(Ar) =
Entropy of Gi E(Gi) =
Sim(Gi,Gj)=
( | ) log( ( | ))r
r i r iv A
p A v G p A v G
1
( )d
rr
entropy A
| | ( ) | | ( ) | | ( )i j i j i i j jG G E G G G E G G E G
To geometric analogy
Use Sim(Gi,Gj) to be our geometric analogy, because Sim(Gi,Gj) 0, ≧ i,j.
Sim(Gi,Gi) = 0, i.
Sim(Gi,Gi) = Sim(Gj,Gi), i,j.
Step 1.
Use the concept of gravitation model.
Mass, Radius, and Time. Mass of a group = the # of elements. Radius of two groups Gi,Gj = similarity(Gi,Gj) Time = ∆T, a constant.
Merging groups Get a clustering tree.
G1 G2
1 21 2 2
1 2
( ) ( )( , )
( , )
GM G M Gfg G G
R G G
1 2( , )R G G
Time T
A Merging Pass M(Gi)= # elements in Gi. R(Gi,Gj)=Sim(Gi,Gj). G is 1.0. fg(Gi,Gj) is the gravitation force. Δt is the time period. Δt = 1.0.
Rnew(Gi,Gj) = R(Gi,Gj) - 1/2(accg
i+accgj)Δt2
If Rnew(Gi,Gj) < 0, Merge Gi and Gj.
Getting closer {Radius = 0} merge. Build a clustering tree
G1 G2
1 21 2 2
1 2
( ) ( )( , )
( , )
GM G M Gfg G G
R G G
1 2( , )R G G
Time T
1
1 2
1
( , )
( )G
fg G Gacc
M G
2
1 2
2
( , )
( )G
fg G Gacc
M G
Time T + ∆t
G1 G2
1 2( , )newR G G
A clustering tree.
p1 p2 p3 pn-1p5 pn
C1
p4
Suitable K.
In the sequence of merging passes, find Suitable number of groups in the partition.
Suppose step1 merges to 1 group, after T iterations. S1, S2, S3,…, ST-1, ST. Si : i-th iteration
Stablei = ΔTime(Si , Si+1) x Radius(Si), Radius(Si) = Min{ Sim
( Gα,Gβ) , Gα,Gβ in Si and αβ }
In the following slides, we use K to denote the proper number of clusters in the partition.
r=0
The suitable K = the # of groups in S , such that
Stable(S ) is maximal, is given by user.
i
l
i r
i
l
Step 2.
Still has Local minimal problem? Yes. Conquer it by randomized algorithm.
Randomized step. If Exchange two elements from two groups[11]
Time consumption. Goal : Enlarge distance/radius of two groups.
For Efficiency Tree-structured data structure digital search
tree. Enlarge the distance/radius of each two groups
in the result obtained from Step 1.
Step 2(con’t)
Why digital search tree (D.S.T)? Locality.
Storing elements storing strings from elements. Deeper nodes are looked like their siblings and
children.
Operations in D.S.T
D.S.T is tree-structured, storing binary strings, and degree of each node 2.≦
Operations cost less ( O(d) ) Search Insert Delete
Search operation. ( O(d) ) Ex:
Operations in D.S.T (cont’)
Insertion (O(d)), Ex: Insert C010 into the tree.
Deletion (O(d)), Ex: delete G110
Randomizing
Transform the result from step 1 into a forest with K digital search trees. Nodes in Tree i ≡ elements in Gi , i.
Randomly select 2 trees i,j, and select an internal node αfrom tree i.
Let gα = { node n | α is the ancestor of node n } ∪{α}
To calculate, , ( ( ) , ) ( , )
, .( , , ) i a j a i jYes if Sim G g G g Sim G Gi j a No OtherwiseGoodExchange G G g
Randomizing (cont’)
For example, K = 2. α= ‘0101’
G10000
00102
00112
0
1
0101
0
G20000
0110
0
1001
1
1010
0
11012
1
1100
0
0001
0
3.248661
Phase 0
0101
1
G20000
0110
1001
1
1010
0
11012
1
1100
0
G10000
00102
00112
0
1
0001
0
1
8.985047
Phase 1
Good Exchange !!!
Outlines
Introduction Motivation Related Work Research Goal / Notation Our Method
Experiments Datasets Compared methods and Criterion
Conclusion Future Work
Experiments
Datasets Real datasets[7].
Congressional Voting Record. Mushroom.
Document datasets[8,9]. Reuters 21578. 20 Newsgroups.
Compared methods & Criterion
Compared methods K-modes ( randomly partition elements into K groups ) Coolcat ( select some elements to be seeds and place se
eds into K groups, then place the remaining)
Criterion (10 times average) Category Utility =
Expected Entropy =
Purity =
1 1
( | ) log( ( | ))r
K d
r k r kk r v A
p A v c p A v c
2 2
1 1
| | ( | ) ( )r
K d
k r k rk r v A
c p A v c p A v
K
1
1( ) , ( ) max ( ) , is the # of elements
of the -th input class that were assigned to the -th cluster.
Kj jk
k k j k kk k
nP c P c n n
n n
i j
Real datasets
Do classify like human. Mushroom dataset
2 classes( edible, poisonous ) 22 attributes. 8124 records.
# Groups Category Utility
Expected Entropy
Purity
K-modes 22 0.60842867 20.2 95.5%
Coolcat 30 0.25583642 35.1 96.2%
Our Method 18 0.74743694 19.4 99.4%
Real datasets
Voting Record dataset 2 classes (democrat, republican), 435 records,
16 attributes (all boolean valued). Some records (10%) have missed value.
# Groups Category Utility
Expected Entropy
Purity
K-modes 22 0.22505624 6.04533891 93.4%
Coolcat 29 0.14361348 7.42124111 94.6%
Our Method 14 0.3177278 6.84627962 92.4%
Document Datasets
For applying. 20 Newsgroups
20 subjects. Each subject contains 1000 articles. 20000 documents (too many). Select documents from 3 subjects to be the dataset
(3000 articles). Use BOW package[4] to do feature selection. 100
features.
# Groups Expected Entropy Purity
K-modes K = 5 3.02019429 92.2%
Coolcat K = 5 3.44101501 90.8%
Our Method K = 5 2.93761945 95.0%
Document Datasets
Reuters 21578 135 subjects. 21578 articles, each subject
contains different # of articles. Select first 10 most articles subjects. Use BOW package[4] to do feature selection. 100
features.
# Groups Expected Entropy Purity
K-modes K = 18 2.30435514 51.8%
Coolcat K = 18 2.68157959 68.3%
Our Method K = 18 0.78800815 66.9%
Conclusion
In this work, we purposed a measure of similarity such that Can reuse the framework used on numerical
data clustering. With FEWER parameters and easy to use. Try to avoid trapping into local minima. Still can obtain same or better clustering result
than methods proposed for categorical data.
Future Work
For our method itself, Still take a long time to calculate, because we
need to fix and satisfy “triangle-inequality law” for our similarity function.
For application. To build a framework to cluster documents
without other packages’ help. Use our method to solve “Binding sets” problem
in Bioinformatics.
Reference
[1] A.K. Jain M.N Murty, and P.J. Flynn, Data Clustering: A Review, ACM Computing Surveys. Vol.31, 1999.
[2] Yen-Jen Oyang, Chien-Yu Chen, Shien-Ching Huang, and Cheng-Fang Lin. Characteristics of a Hierarchical Data Clustering Algorithm Based on Gravity Theory. Technical Report of NTUCSIE 02-01.
[3] S. Guha, R. Rastogi, and K. Shim. ROCK: A robust clustering algorithm for categorical attributes. Proc. of IEEE Intl. Conf. on Data Eng. (ICDE), 1999.
[4] McClallum, A. K. Bow: A tookit for statistical language modeling, text retrieval, classification and clustering. http://www.cs.cmu.edu/mccallum/bow.
[5] DIGITAL LIBRARIES: Metadata Resources. http://www.ifla.org/II/metadata.htm [6] W.E. Wright. A formulization of cluster analysis and gravitational clustering. Doctoral
Dissertation. Washington University, 1972. [7] Newman, D.J. & Hettich, S. & Blake, C.L. & Merz, C.J. (1998). UCI Repository of mac
hine learning databases. [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, Department of Information and Computer Science.
[8] David D. Lewis. Reuters-21578 text categorization test collection. AT&T Labs – Research. 1997.
[9] Ken Lang. 20 Newsgroups. Http://people.csail.mit.edu/jrennie/20Newsgroups/ [10] T. Cover and J. Thomas. Elements of Information Theory. Wiley, 1991.
Reference
[11] T. Li, S. Ma, and M. Ogihara. Entropy-based criterion in categorical clustering. Proc. of Intl. Conf. on Machine Learning (ICML), 2004.
[12] Bock, H.-H. Probabilistic aspects in cluster analysis. In O. Opitz(Ed.), Conceptual and numerical analysis of data, 12-44. Berlin: Springer-verlag. 1989.
[13] Z. Huang. A fast clustering algorithm to cluster very large categorical data sets in data mining. Workshop on Research Issues on Data Mining and Knowledge Discovery, 1997.
[14] MacQueen, J. B. (1967) Some Methods for Classification and Analysis of Multivariate Observations, In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, pp. 281-297.
[15] Richard O. Duda and Peter E. Hard. Pattern Classification and Secen Analysis. A Wiley-Interscience Publication, New York, 1973.
[16] P Andritsos, P Tsaparas, RJ Miller, KC Sevcik. LIMBO: Scalable Clustering of Categorical Data. Hellenic Database Symposium, 2003.
[17] V. Ganti, J. Gehrke, and R. Ramakrishnan. CACTUS-clustering categorical data using summaries. Proc. of ACM SIGKEE Conference, 1999.
[18] D. Barbara, Y. Li, and J. Couto. Coolcat: an entropy-based algorithm for categorical clustering. Proc. of ACM Conf. on Information and Knowledge Mgt. (CIKM), 2002.
[19] R. C.Dubes. How many clusters are best? – an experiment. Pattern Recognition, vol. 20, no 6, pp.645-663, 1987.
[20] Keke Chen and Ling Liu: "The ‘Best K’ for Entropy-based Categorical Clustering ", Proc of Scientific and Statistical Database Management (SSDBM05). Santa Barbara, CA June 2005.
[21] 林正芳,歐陽彥正,”以重力為基礎的二階段階層式資料分群演算法特性之研究”,國立臺灣大學資訊工程學系,碩士論文。民 91。
[24] I.H. Witten, E. Frank. Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations, Morgan Kaufmann Publishers, San Francisco, 2000.
Appendix 1
K-modes Extend the k-means algorithm for clustering large data sets with cate
gorical values. Use “Mode” to represent a group.
1 2 3: { , , ,... }, ( ),1
( , ) ( , ).
i n j j
i i i i
Mode DQ x x x x x S X i K
dist d DQ CountDiffAttributes d DQ
Appendix 2
ROCK Using Jaccard coefficient:
to measure the distance between points Xi,Xk in attribute j Define
Otherwise Neighbor(xi,xk) = false. Define Link(p,Ci ) = the # of Neightbor(p,q) = 1, for every q belon
gs to cluster Ci. Best clusters (Objective function):
For well defined data set , ψ is 0.5, f(ψ) = (1+ψ)/(1-ψ) Threshold ψ affects the result is good or not, and hard to decide. Relaxing the problem to many clusters.
Aj Aij=1 Aij=0
Aik=1 a bAik=0 c d
( , ) ,j i k
aJ x x for attribute j
a b c
( , ) , ( , )i k j i kj
Neighbor x x true if J x x threshold
1 2 ( )
( , )| |
| |i
ii f
p C i
Link p CMaximize C
C
Appendix 3
CACTUS is an agglomerative algorithm. Use attribute values to group data. Define:
Support : Strong connection: S_conn(sij,sjt)=true,
, if Pair(sij,sjt) > threshold ψ. From pairs to sets of attributes. A cluster is defined as a region of attributes that are pai
rwise strongly connected. From the attributes view. Find k regions of attributes. Setting threshold ψ decides the result is good or not.
( , ) #{( ) ( )}ji kt j ji k ktPair s s S s S s
Appendix 4 COOLCAT [2002] Use entropy to measure the uncertainty of the attribu
te j. Si = { Si1, Si2, Si3,…,Sit },
Entropy(Si) = When Si is much clear, Entropy(Si) 0. uncertainty is small.
For a cluster k, the entropy of k is
在Ci中,各屬性 Si的亂度的總和。 For a partition P = {C1,C2,…Ck} The expected Entropy of P is
1
( 1) ( ) log( ( ))t
i ij i ijj
p S s p S s
1
( ) ( ) ,d
k ii
E C Entropy S
| |( ) ( )i
kk
CE P E C
m
Appendix 4 (cont’) For a given partition P, P is best clusters, P satisfies unique val
ue in each attribute for each cluster Ci. COOLCAT’s objective function is . Since
Coolcat picks k points from data set S, such that k are most unlike for each other, when beginning. And assign these points to k clusters.
For each point p in S-{ assigned points }, assign p into cluster i with the minimal increasing entropy.
Very easy to implement, but performance is strongly connected to the initial selection.
Sequence sensitive. Trapped in local minima easily, expected entropy is not the mini
mal.