DECISION TREESAsher Moody, CS 157B

Overview Definition Motivation Algorithms

ID3 Example Entropy Information Gain Applications Conclusion

Decision Tree Decision trees are a fundamental

technique used in data mining.

Decision trees are used for classification, clustering, feature selection, and prediction.

Motivation Decision trees help accurate classify

data

Decision trees help understand the predictive nature of the data by recognizing patterns

Decision trees depict the relationships between input data and target outputs

Algorithms Decision trees algorithms are greedy so

once test has been selected to partition the data other options will not be explored

Popular Algorithms Computer Science: ID3, C4.5, and C5.0 Statistics: Classification and Regression

Trees (CART)

ID3 Algorithm Given: Examples(S); Target attribute (C);

Attributes (R) Initialize Root Function ID3 (S,C,R) Create a Root node for the tree IF S = empty, return a single node with value Failure; IF S = C, return a single node C; IF R = empty, return a single node with most frequent target

attribute (C); ELSE BEGIN… (next slide)

ID3 (cont) BEGIN Let D be the attribute with largest Gain Radio (D, S) among

attributes in R; Let {dj | j = 1, 2, …, n} be the values of attribute D; Let {Sj | j = 1, 2, …, n} be the subsets of S consisting respectively of

records with value dj for attribute D; Return a tree with root labeled D arcs d1, d2, …, dn going

respectively to the trees; For each branch in the tree IF S = empty, add a new branch with most frequent C; ELSE ID3 (S1, C, R – {D}), ID3 (S2, C, R – {D}), …, IDC(Sn, C, R – {D}) END ID3 Return Root

Example 1

Example 2

Entropy Entropy gives us a measure of how uncertain we are

about the data Maximum: The measure should be maximal if all the

outcomes are equally likely (uncertainty is highest when all possible events are equiprobable).

where Pi is the proportion of instances in the dataset that take the ith value of the target attribute

Information Gain Gain calculates the reduction in entropy (gain in

information) that would result from splitting the data at a particular attribute A.

where v is a value of A, |Sv| is the subset of instances of S where A takes the value v, and |S| is the number of instances

Applications Business: to track purchasing patterns Medical: identify potential risks

associated with diseases Banks: identify potential credit risks Governments: to determine features of

potential terrorists Seismology: to predict earthquakes

Conclusion Search through attributes to find the

proportions Calculate the entropy for each possible

data input for a particular attribute Calculate the gain for each attribute Make the attribute with the highest gain

the root node Continue the process until decision tree

is complete

References Berry, M. W. (2006). Lecture Notes in

Data Mining. World Scientific http://www.decisiontrees.net http://en.wikipedia.org/wiki/Entropy http://en.wikipedia.org/wiki/

Information_gain_in_decision_trees