A Brief History of Data Mining Society

1989 IJCAI Workshop on Knowledge Discovery in Databases Knowledge Discovery in Databases (G. Piatetsky-Shapiro

and W. Frawley, 1991) 1991-1994 Workshops on Knowledge Discovery in Databases

Advances in Knowledge Discovery and Data Mining (U. Fayyad, G. Piatetsky-Shapiro, P. Smyth, and R. Uthurusamy, 1996)

1995-1998 International Conferences on Knowledge Discovery in Databases and Data Mining (KDD’95-98)

Journal of Data Mining and Knowledge Discovery (1997)

A Brief History of Data Mining Society

ACM SIGKDD conferences since 1998 and SIGKDD Explorations

More conferences on data mining PAKDD (1997), PKDD (1997), SIAM-Data

Mining (2001), (IEEE) ICDM (2001), etc. ACM Transactions on KDD starting in

2007

Conferences and Journals on Data Mining

KDD Conferences ACM SIGKDD Int. Conf. on Knowledge

Discovery in Databases and Data Mining (KDD)

SIAM Data Mining Conf. (SDM) (IEEE) Int. Conf. on Data Mining (ICDM) Conf. on Principles and practices of Knowledge

Discovery and Data Mining (PKDD) Pacific-Asia Conf. on Knowledge Discovery and

Data Mining (PAKDD)

Where to Find References? DBLP, CiteSeer, Google

Data mining and KDD (SIGKDD: CDROM) Conferences: ACM-SIGKDD, IEEE-ICDM, SIAM-DM,

PKDD, PAKDD, etc. Journal: Data Mining and Knowledge Discovery, KDD

Explorations, ACM TKDD

Bioinformatics Conferences: RECOMB, CSB, PSB, BIBE, etc Journals: Bioinformatics, BMC Bioinformatics, TCBB,…

Top-10 Algorithm Finally Selected at ICDM’06

#1: Decision Tree (61 votes) #2: K-Means (60 votes) #3: SVM (58 votes) #4: Apriori (52 votes) #5: EM (48 votes) #6: PageRank (46 votes) #7: AdaBoost (45 votes) #8: kNN (45 votes) #9: Naive Bayes (45 votes) #10: CART (34 votes)

Association Rules

Association Rules

Association Rules

support, s, probability that a transaction contains X Y

confidence, c, conditional probability that a transaction having X also contains Y

Association Rules

Let’s have an example

Association Rules T100 1,2,5 T200 2,4 T300 2,3 T400 1,2,4 T500 1,3 T600 2,3 T700 1,3 T800 1,2,3,5 T900 1,2,3

Association Rules

Classification

Classification—A Two-Step Process

Classification classifies data (constructs a model)

based on the training set and the values (class labels) in a classifying attribute and uses it in classifying new data

predicts categorical class labels (discrete or nominal)

Classification

Typical applications Credit approval Target marketing Medical diagnosis Fraud detection And much more

age income student credit_rating buys_computer<=30 high no fair no<=30 high no excellent no31…40 high no fair yes>40 medium no fair yes>40 low yes fair yes>40 low yes excellent no31…40 low yes excellent yes<=30 medium no fair no<=30 low yes fair yes>40 medium yes fair yes<=30 medium yes excellent yes31…40 medium no excellent yes31…40 high yes fair yes>40 medium no excellent no

Decision Tree Decision Tree induction is the learning

of decision trees from class-labeled training tuples

A decision tree is a flowchart-like tree structure, where each internal node denotes a test on an attribute

Each Branch represents an outcome of the test

Each Leaf node holds a class label

Decision Tree Example

Decision Tree Algorithm

Basic algorithm (a greedy algorithm) Tree is constructed in a top-down recursive

divide-and-conquer manner At start, all the training examples are at the

root Attributes are categorical (if continuous-

valued, they are discretized in advance) Test attributes are selected on the basis of a

heuristic or statistical measure (e.g., information gain)

Attribute Selection Measure: Information Gain (ID3/C4.5) Select the attribute with the highest

information gain Let pi be the probability that an

arbitrary tuple in D belongs to class Ci, estimated by |Ci, D|/|D|

Expected information (entropy) needed to

classify a tuple in D:)(log)( 2

1i

m

ii ppDInfo

Attribute Selection Measure: Information Gain (ID3/C4.5) Information needed (after using A

to split D into v partitions) to classify D:

Information gained by branching on attribute A

)(||

||)(

1j

v

j

jA DI

D

DDInfo

(D)InfoInfo(D)Gain(A) A

age income student credit_rating buys_computer<=30 high no fair no<=30 high no excellent no31…40 high no fair yes>40 medium no fair yes>40 low yes fair yes>40 low yes excellent no31…40 low yes excellent yes<=30 medium no fair no<=30 low yes fair yes>40 medium yes fair yes<=30 medium yes excellent yes31…40 medium no excellent yes31…40 high yes fair yes>40 medium no excellent no

Decision Tree

940.0)14

5(log

14

5)

14

9(log

14

9)5,9()( 22 IDInfo

age income student credit_rating buys_computer<=30 high no fair no<=30 high no excellent no31…40 high no fair yes>40 medium no fair yes>40 low yes fair yes>40 low yes excellent no31…40 low yes excellent yes<=30 medium no fair no<=30 low yes fair yes>40 medium yes fair yes<=30 medium yes excellent yes31…40 medium no excellent yes31…40 high yes fair yes>40 medium no excellent no

Decision Tree

694.0)2,3(14

5

)0,4(14

4)3,2(

14

5)(

I

IIDInfoage

age income student credit_rating buys_computer<=30 high no fair no<=30 high no excellent no31…40 high no fair yes>40 medium no fair yes>40 low yes fair yes>40 low yes excellent no31…40 low yes excellent yes<=30 medium no fair no<=30 low yes fair yes>40 medium yes fair yes<=30 medium yes excellent yes31…40 medium no excellent yes31…40 high yes fair yes>40 medium no excellent no

means “age <=30” has 5 out of 14 samples, with 2 yes’s and 3 no’s.

I(2,3) = -2/5 * log(2/5) – 3/5 * log(3/5)

)3,2(14

5I

Decision Tree

Similarily, we can compute Gain(income)=0.029 Gain(student)=0.151 Gain(credit_rating)=0.048

Since “age” obtains highest information gain, we can partition the tree into:

246.0)()()( DInfoDInfoageGain age

Decision Tree

Decision Tree