classification and prediction (cont.) pertemuan 09

Classification and Prediction (cont.)

Pertemuan 09

Matakuliah : M0614 / Data Mining & OLAP Tahun : Feb - 2010

Bina Nusantara

Pada akhir pertemuan ini, diharapkan mahasiswa

akan mampu :• Mahasiswa dapat menggunakan teknik analisis

classification by decision tree induction, Bayesian classification, classification by back propagation, dan lazy learners pada data mining. (C3)

Learning Outcomes

3

Bina Nusantara

Acknowledgments

These slides have been adapted from Han, J., Kamber, M., & Pei, Y. Data Mining: Concepts and Technique and Tan, P.-N., Steinbach, M., & Kumar, V. Introduction to Data Mining.

Bina Nusantara

• Rule-based classification• Classification by back propagation• Lazy learners (or learning from your

neighbours)

Outline Materi

5

Rule-Based Classifier• Classify records by using a collection of “if…then…” rules

• Rule: (Condition) y– where

• Condition is a conjunctions of attributes • y is the class label

– LHS: rule antecedent or condition– RHS: rule consequent– Examples of classification rules:

• (Blood Type=Warm) (Lay Eggs=Yes) Birds• (Taxable Income < 50K) (Refund=Yes) Evade=No

Rule-based Classifier (Example)

R1: (Give Birth = no) (Can Fly = yes) BirdsR2: (Give Birth = no) (Live in Water = yes) FishesR3: (Give Birth = yes) (Blood Type = warm) MammalsR4: (Give Birth = no) (Can Fly = no) ReptilesR5: (Live in Water = sometimes) Amphibians

Name Blood Type Give Birth Can Fly Live in Water Classhuman warm yes no no mammalspython cold no no no reptilessalmon cold no no yes fisheswhale warm yes no yes mammalsfrog cold no no sometimes amphibianskomodo cold no no no reptilesbat warm yes yes no mammalspigeon warm no yes no birdscat warm yes no no mammalsleopard shark cold yes no yes fishesturtle cold no no sometimes reptilespenguin warm no no sometimes birdsporcupine warm yes no no mammalseel cold no no yes fishessalamander cold no no sometimes amphibiansgila monster cold no no no reptilesplatypus warm no no no mammalsowl warm no yes no birdsdolphin warm yes no yes mammalseagle warm no yes no birds

April 21, 2023Data Mining: Concepts and

Techniques 8

age?

student? credit rating?

<=30 >40

no yes yes

yes

31..40

no

fairexcellentyesno

• Example: Rule extraction from our buys_computer decision-tree

IF age = young AND student = no THEN buys_computer = no

IF age = young AND student = yes THEN buys_computer = yes

IF age = mid-age THEN buys_computer = yes

IF age = old AND credit_rating = excellent THEN buys_computer = yes

IF age = young AND credit_rating = fair THEN buys_computer = no

Rule Extraction from a Decision Tree

Rules are easier to understand than large trees

One rule is created for each path from the root to a leaf

Each attribute-value pair along a path forms a conjunction: the leaf holds the class prediction

Rules are mutually exclusive and exhaustive

Application of Rule-Based Classifier• A rule r covers an instance x if the attributes of the instance

satisfy the condition of the rule

R1: (Give Birth = no) (Can Fly = yes) Birds

R2: (Give Birth = no) (Live in Water = yes) Fishes

R3: (Give Birth = yes) (Blood Type = warm) Mammals

R4: (Give Birth = no) (Can Fly = no) Reptiles

R5: (Live in Water = sometimes) Amphibians

The rule R1 covers a hawk => Bird

The rule R3 covers the grizzly bear => Mammal

Name Blood Type Give Birth Can Fly Live in Water Classhawk warm no yes no ?grizzly bear warm yes no no ?

How does Rule-based Classifier Work?






A lemur triggers rule R3, so it is classified as a mammal

A turtle triggers both R4 and R5

A dogfish shark triggers none of the rules

Name Blood Type Give Birth Can Fly Live in Water Classlemur warm yes no no ?turtle cold no no sometimes ?dogfish shark cold yes no yes ?

Rule Coverage and Accuracy

• Coverage of a rule:– Fraction of records that

satisfy the antecedent of a rule

• Accuracy of a rule:– Fraction of records that

satisfy both the antecedent and consequent of a rule

Tid Refund Marital Status

Taxable Income Class

1 Yes Single 125K No

2 No Married 100K No

3 No Single 70K No

4 Yes Married 120K No

5 No Divorced 95K Yes

6 No Married 60K No

7 Yes Divorced 220K No

8 No Single 85K Yes

9 No Married 75K No

10 No Single 90K Yes 10

(Status=Single) No

Coverage = 40%, Accuracy = 50%

Characteristics of Rule-Based Classifier

• Mutually exclusive rules– Classifier contains mutually exclusive rules if the rules

are independent of each other– Every record is covered by at most one rule

• Exhaustive rules– Classifier has exhaustive coverage if it accounts for

every possible combination of attribute values– Each record is covered by at least one rule

From Decision Trees To Rules

YESYESNONO

NONO

NONO

Yes No

{Married}{Single,

Divorced}

< 80K > 80K

Taxable Income

Marital Status

Refund Classification Rules

(Refund=Yes) ==> No

(Refund=No, Marital Status={Single,Divorced},Taxable Income<80K) ==> No

(Refund=No, Marital Status={Single,Divorced},Taxable Income>80K) ==> Yes

(Refund=No, Marital Status={Married}) ==> No

Rules are mutually exclusive and exhaustive

Rule set contains as much information as the tree

Rules Can Be Simplified

YESYESNONO

NONO

NONO

Yes No

{Married}{Single,

Divorced}

< 80K > 80K

Taxable Income

Marital Status

RefundTid Refund Marital

Status Taxable Income Cheat



3 No Single 70K No



6 No Married 60K No


8 No Single 85K Yes

9 No Married 75K No

10 No Single 90K Yes 10

Initial Rule: (Refund=No) (Status=Married) No

Simplified Rule: (Status=Married) No

Effect of Rule Simplification

• Rules are no longer mutually exclusive– A record may trigger more than one rule – Solution?

• Ordered rule set• Unordered rule set – use voting schemes

• Rules are no longer exhaustive– A record may not trigger any rules– Solution?

• Use a default class

Ordered Rule Set• Rules are rank ordered according to their priority

– An ordered rule set is known as a decision list• When a test record is presented to the classifier

– It is assigned to the class label of the highest ranked rule it has triggered

– If none of the rules fired, it is assigned to the default class






Name Blood Type Give Birth Can Fly Live in Water Classturtle cold no no sometimes ?

Rule Ordering Schemes

• Rule-based ordering– Individual rules are ranked based on their quality

• Class-based ordering– Rules that belong to the same class appear together

Rule-based Ordering

(Refund=Yes) ==> No




Class-based Ordering

(Refund=Yes) ==> No




Building Classification Rules

• Direct Method: • Extract rules directly from data• e.g.: RIPPER, CN2, Holte’s 1R

• Indirect Method:• Extract rules from other classification models (e.g.

decision trees, neural networks, etc).• e.g: C4.5rules


Techniques 19

Rule Induction: Sequential Covering Method

• Sequential covering algorithm: Extracts rules directly from training data

• Typical sequential covering algorithms: FOIL, AQ, CN2, RIPPER

• Rules are learned sequentially, each for a given class Ci will cover many

tuples of Ci but none (or few) of the tuples of other classes

• Steps:

– Rules are learned one at a time

– Each time a rule is learned, the tuples covered by the rules are removed

– The process repeats on the remaining tuples unless termination condition, e.g., when no more training examples or when the quality of a rule returned is below a user-specified threshold

• Comp. w. decision-tree induction: learning a set of rules simultaneously


Techniques 20

Sequential Covering Algorithm

while (enough target tuples left)generate a ruleremove positive target tuples satisfying this rule

Examples coveredby Rule 3

Examples coveredby Rule 2Examples covered

by Rule 1

Positive examples


Techniques 21

How to Learn-One-Rule?

• Star with the most general rule possible: condition = empty

• Adding new attributes by adopting a greedy depth-first strategy

– Picks the one that most improves the rule quality

• Rule-Quality measures: consider both coverage and accuracy

– Foil-gain (in FOIL & RIPPER): assesses info_gain by extending condition

It favors rules that have high accuracy and cover many positive tuples

• Rule pruning based on an independent set of test tuples

Pos/neg are # of positive/negative tuples covered by R.

If FOIL_Prune is higher for the pruned version of R, prune R

)log''

'(log'_ 22 negpos

pos

negpos

posposGainFOIL

negpos

negposRPruneFOIL

)(_


Techniques 22

Rule Generation• To generate a rule

while(true)find the best predicate pif foil-gain(p) > threshold then add p to current ruleelse break

Positive examples

Negative examples

A3=1A3=1&&A1=2

A3=1&&A1=2&&A8=5

Aspects of Sequential Covering

• Rule Growing

• Instance Elimination

• Rule Evaluation

• Stopping Criterion

• Rule Pruning

Rule Growing

• Two common strategies

Status =Single

Status =Divorced

Status =Married

Income> 80K...

Yes: 3No: 4{ }

Yes: 0No: 3

Refund=No

Yes: 3No: 4

Yes: 2No: 1

Yes: 1No: 0

Yes: 3No: 1

(a) General-to-specific

Refund=No,Status=Single,Income=85K(Class=Yes)

Refund=No,Status=Single,Income=90K(Class=Yes)

Refund=No,Status = Single(Class = Yes)

(b) Specific-to-general

Rule Growing (Examples)• CN2 Algorithm:

– Start from an empty conjunct: {}– Add conjuncts that minimizes the entropy measure: {A}, {A,B}, …– Determine the rule consequent by taking majority class of instances covered by the

rule• RIPPER Algorithm:

– Start from an empty rule: {} => class– Add conjuncts that maximizes FOIL’s information gain measure:

• R0: {} => class (initial rule)• R1: {A} => class (rule after adding conjunct)• Gain(R0, R1) = t [ log (p1/(p1+n1)) – log (p0/(p0 + n0)) ]• where t: number of positive instances covered by both R0 and R1

p0: number of positive instances covered by R0n0: number of negative instances covered by R0p1: number of positive instances covered by R1n1: number of negative instances covered by R1

Rule Evaluation

• Metrics:– Accuracy

– Laplace

– M-estimate

kn

nc

1

kn

kpnc

n : Number of instances covered by rule

nc : Number of instances covered by rule

k : Number of classes

p : Prior probability

n

nc

Stopping Criterion and Rule Pruning

• Stopping criterion– Compute the gain– If gain is not significant, discard the new rule

• Rule Pruning– Similar to post-pruning of decision trees– Reduced Error Pruning:

• Remove one of the conjuncts in the rule • Compare error rate on validation set before and after

pruning• If error improves, prune the conjunct

Summary of Direct Method

• Grow a single rule

• Remove Instances from rule

• Prune the rule (if necessary)

• Add rule to Current Rule Set

• Repeat

Direct Method: RIPPER• For 2-class problem, choose one of the classes as positive class,

and the other as negative class– Learn rules for positive class– Negative class will be default class

• For multi-class problem– Order the classes according to increasing class prevalence

(fraction of instances that belong to a particular class)– Learn the rule set for smallest class first, treat the rest as

negative class– Repeat with next smallest class as positive class

Direct Method: RIPPER• Growing a rule:

– Start from empty rule– Add conjuncts as long as they improve FOIL’s information gain– Stop when rule no longer covers negative examples– Prune the rule immediately using incremental reduced error

pruning– Measure for pruning: v = (p-n)/(p+n)

• p: number of positive examples covered by the rule in the validation set

• n: number of negative examples covered by the rule in the validation set

– Pruning method: delete any final sequence of conditions that maximizes v

Direct Method: RIPPER

• Building a Rule Set:– Use sequential covering algorithm

• Finds the best rule that covers the current set of positive examples

• Eliminate both positive and negative examples covered by the rule

– Each time a rule is added to the rule set, compute the new description length

• stop adding new rules when the new description length is d bits longer than the smallest description length obtained so far

Direct Method: RIPPER

• Optimize the rule set:– For each rule r in the rule set R

• Consider 2 alternative rules:– Replacement rule (r*): grow new rule from scratch– Revised rule(r’): add conjuncts to extend the rule r

• Compare the rule set for r against the rule set for r* and r’

• Choose rule set that minimizes MDL principle– Repeat rule generation and rule optimization for the remaining

positive examples

Indirect Methods

Rule Set

r1: (P=No,Q=No) ==> -r2: (P=No,Q=Yes) ==> +r3: (P=Yes,R=No) ==> +r4: (P=Yes,R=Yes,Q=No) ==> -r5: (P=Yes,R=Yes,Q=Yes) ==> +

P

Q R

Q- + +

- +

No No

No

Yes Yes

Yes

No Yes

Indirect Method: C4.5rules

• Extract rules from an unpruned decision tree• For each rule, r: A y,

– consider an alternative rule r’: A’ y where A’ is obtained by removing one of the conjuncts in A

– Compare the pessimistic error rate for r against all r’s– Prune if one of the r’s has lower pessimistic error rate– Repeat until we can no longer improve generalization

error

Indirect Method: C4.5rules• Instead of ordering the rules, order subsets of rules (class

ordering)– Each subset is a collection of rules with the same rule

consequent (class)– Compute description length of each subset

• Description length = L(error) + g L(model)• g is a parameter that takes into account the

presence of redundant attributes in a rule set (default value = 0.5)

ExampleName Give Birth Lay Eggs Can Fly Live in Water Have Legs Class

human yes no no no yes mammalspython no yes no no no reptilessalmon no yes no yes no fisheswhale yes no no yes no mammalsfrog no yes no sometimes yes amphibianskomodo no yes no no yes reptilesbat yes no yes no yes mammalspigeon no yes yes no yes birdscat yes no no no yes mammalsleopard shark yes no no yes no fishesturtle no yes no sometimes yes reptilespenguin no yes no sometimes yes birdsporcupine yes no no no yes mammalseel no yes no yes no fishessalamander no yes no sometimes yes amphibiansgila monster no yes no no yes reptilesplatypus no yes no no yes mammalsowl no yes yes no yes birdsdolphin yes no no yes no mammalseagle no yes yes no yes birds

C4.5 versus C4.5rules versus RIPPERC4.5rules:

(Give Birth=No, Can Fly=Yes) Birds

(Give Birth=No, Live in Water=Yes) Fishes

(Give Birth=Yes) Mammals

(Give Birth=No, Can Fly=No, Live in Water=No) Reptiles

( ) Amphibians

GiveBirth?

Live InWater?

CanFly?

Mammals

Fishes Amphibians

Birds Reptiles

Yes No

Yes

Sometimes

No

Yes No

RIPPER:

(Live in Water=Yes) Fishes

(Have Legs=No) Reptiles

(Give Birth=No, Can Fly=No, Live In Water=No)

Reptiles

(Can Fly=Yes,Give Birth=No) Birds

() Mammals

C4.5 versus C4.5rules versus RIPPER

PREDICTED CLASS Amphibians Fishes Reptiles Birds MammalsACTUAL Amphibians 0 0 0 0 2CLASS Fishes 0 3 0 0 0

Reptiles 0 0 3 0 1Birds 0 0 1 2 1Mammals 0 2 1 0 4

PREDICTED CLASS Amphibians Fishes Reptiles Birds MammalsACTUAL Amphibians 2 0 0 0 0CLASS Fishes 0 2 0 0 1

Reptiles 1 0 3 0 0Birds 1 0 0 3 0Mammals 0 0 1 0 6

C4.5 and C4.5rules:

RIPPER:

Advantages of Rule-Based Classifiers

• As highly expressive as decision trees• Easy to interpret• Easy to generate• Can classify new instances rapidly• Performance comparable to decision trees


Techniques 40

Classification by Backpropagation

• Backpropagation: A neural network learning algorithm

• Started by psychologists and neurobiologists to develop and test computational analogues of neurons

• A neural network: A set of connected input/output units where each connection has a weight associated with it

• During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class label of the input tuples

• Also referred to as connectionist learning due to the connections between units


Techniques 41

Classification by Backpropagation

• Backpropagation: A neural network learning algorithm

• Started by psychologists and neurobiologists to develop and test computational analogues of neurons

• A neural network: A set of connected input/output units where each connection has a weight associated with it

• During the learning phase, the network learns by adjusting the weights so as to be able to predict the correct class label of the input tuples

• Also referred to as connectionist learning due to the connections between units


Techniques 42

Neural Network as a Classifier

• Weakness– Long training time – Require a number of parameters typically best determined

empirically, e.g., the network topology or “structure.”– Poor interpretability: Difficult to interpret the symbolic meaning

behind the learned weights and of “hidden units” in the network• Strength

– High tolerance to noisy data – Ability to classify untrained patterns – Well-suited for continuous-valued inputs and outputs– Successful on a wide array of real-world data– Algorithms are inherently parallel– Techniques have recently been developed for the extraction of

rules from trained neural networks


Techniques 43

A Neuron (= a perceptron)

• The n-dimensional input vector x is mapped into variable y by means of the scalar product and a nonlinear function mapping

k-

f

weighted sum

Inputvector x

output y

Activationfunction

weightvector w

w0

w1

wn

x0

x1

xn)sign(y

ExampleFor n

0ikiixw


Techniques 44

A Multi-Layer Feed-Forward Neural Network

Output layer

Input layer

Hidden layer

Output vector

Input vector: X

wij

ijkii

kj

kj xyyww )ˆ( )()()1(


Techniques 45

How A Multi-Layer Neural Network Works?

• The inputs to the network correspond to the attributes measured for each training tuple

• Inputs are fed simultaneously into the units making up the input layer

• They are then weighted and fed simultaneously to a hidden layer

• The number of hidden layers is arbitrary, although usually only one

• The weighted outputs of the last hidden layer are input to units making up the output layer, which emits the network's prediction

• The network is feed-forward in that none of the weights cycles back to an input unit or to an output unit of a previous layer

• From a statistical point of view, networks perform nonlinear regression: Given enough hidden units and enough training samples, they can closely approximate any function


Techniques 46

Defining a Network Topology

• First decide the network topology: # of units in the input layer, # of hidden layers (if > 1), # of units in each hidden layer, and # of units in the output layer

• Normalizing the input values for each attribute measured in the training tuples to [0.0—1.0]

• One input unit per domain value, each initialized to 0

• Output, if for classification and more than two classes, one output unit per class is used

• Once a network has been trained and its accuracy is unacceptable, repeat the training process with a different network topology or a different set of initial weights


Techniques 47

Backpropagation• Iteratively process a set of training tuples & compare the network's

prediction with the actual known target value

• For each training tuple, the weights are modified to minimize the mean squared error between the network's prediction and the actual target value

• Modifications are made in the “backwards” direction: from the output layer, through each hidden layer down to the first hidden layer, hence “backpropagation”

• Steps– Initialize weights (to small random #s) and biases in the network– Propagate the inputs forward (by applying activation function) – Backpropagate the error (by updating weights and biases)– Terminating condition (when error is very small, etc.)


Techniques 48

Lazy vs. Eager Learning

• Lazy vs. eager learning– Lazy learning (e.g., instance-based learning): Simply stores

training data (or only minor processing) and waits until it is given a test tuple

– Eager learning (the above discussed methods): Given a set of training set, constructs a classification model before receiving new (e.g., test) data to classify

• Lazy: less time in training but more time in predicting• Accuracy

– Lazy method effectively uses a richer hypothesis space since it uses many local linear functions to form its implicit global approximation to the target function

– Eager: must commit to a single hypothesis that covers the entire instance space


Techniques 49

Lazy Learner: Instance-Based Methods

• Instance-based learning: – Store training examples and delay the processing (“lazy

evaluation”) until a new instance must be classified• Typical approaches

– k-nearest neighbor approach• Instances represented as points in a Euclidean space.

– Locally weighted regression• Constructs local approximation

– Case-based reasoning• Uses symbolic representations and knowledge-based

inference

Nearest Neighbor Classifiers• Basic idea:

– If it walks like a duck, quacks like a duck, then it’s probably a duck

Training Records

Test Record

Compute Distance

Choose k of the “nearest” records

Nearest-Neighbor Classifiers

Requires three things

– The set of stored records

– Distance Metric to compute distance between records

– The value of k, the number of nearest neighbors to retrieve

To classify an unknown record:

– Compute distance to other training records

– Identify k nearest neighbors

– Use class labels of nearest neighbors to determine the class label of unknown record (e.g., by taking majority vote)

Unknown record

Definition of Nearest Neighbor

X X X

(a) 1-nearest neighbor (b) 2-nearest neighbor (c) 3-nearest neighbor

K-nearest neighbors of a record x are data points that have the k smallest distance to x


Techniques 53

The k-Nearest Neighbor Algorithm

• All instances correspond to points in the n-D space• The nearest neighbor are defined in terms of Euclidean

distance, dist(X1, X2)• Target function could be discrete- or real- valued• For discrete-valued, k-NN returns the most common

value among the k training examples nearest to xq

• Vonoroi diagram: the decision surface induced by 1-NN for a typical set of training examples

. _

+_ xq

+

_ _+

_

_

+

.

..

. .

Nearest Neighbor Classification• Compute distance between two points:

– Euclidean distance

• Determine the class from nearest neighbor list– take the majority vote of class labels among the k-nearest

neighbors– Weigh the vote according to distance

• weight factor, w = 1/d2

i ii

qpqpd 2)(),(

Nearest Neighbor Classification…• Choosing the value of k:

– If k is too small, sensitive to noise points– If k is too large, neighborhood may include points from other

classes

X

Nearest Neighbor Classification…• Scaling issues

– Attributes may have to be scaled to prevent distance measures from being dominated by one of the attributes

– Example:• height of a person may vary from 1.5m to 1.8m• weight of a person may vary from 90lb to 300lb• income of a person may vary from $10K to $1M

Nearest neighbor Classification…

• k-NN classifiers are lazy learners – It does not build models explicitly– Unlike eager learners such as decision tree induction

and rule-based systems– Classifying unknown records are relatively expensive

Example: PEBLS

• PEBLS: Parallel Examplar-Based Learning System (Cost & Salzberg)– Works with both continuous and nominal features

• For nominal features, distance between two nominal values is computed using modified value difference metric (MVDM)

– Each record is assigned a weight factor– Number of nearest neighbor, k = 1

Example: PEBLS

Class

Marital Status

Single Married Divorced

Yes 2 0 1

No 2 4 1

i

ii

n

n

n

nVVd

2

2

1

121 ),(

Distance between nominal attribute values:

d(Single,Married)

= | 2/4 – 0/4 | + | 2/4 – 4/4 | = 1

d(Single,Divorced)

= | 2/4 – 1/2 | + | 2/4 – 1/2 | = 0

d(Married,Divorced)

= | 0/4 – 1/2 | + | 4/4 – 1/2 | = 1

d(Refund=Yes,Refund=No)

= | 0/3 – 3/7 | + | 3/3 – 4/7 | = 6/7

Tid Refund MaritalStatus

TaxableIncome Cheat



3 No Single 70K No



6 No Married 60K No


8 No Single 85K Yes

9 No Married 75K No

10 No Single 90K Yes10

ClassRefund

Yes No

Yes 0 3

No 3 4

Example: PEBLS

d

iiiYX YXdwwYX

1

2),(),(

Tid Refund Marital Status

Taxable Income Cheat

X Yes Single 125K No

Y No Married 100K No 10

Distance between record X and record Y:

where:

correctly predicts X timesofNumber

predictionfor used is X timesofNumber Xw

wX 1 if X makes accurate prediction most of the time

wX > 1 if X is not reliable for making predictions

Bina Nusantara

Dilanjutkan ke pert. 10Classification and Prediction (cont.)

classification and prediction (cont.) pertemuan 09

Documents

rule antecedent

rule r1

exhaustiveapplication

birdthe rule r3

rulebased classifier

based classifiera rule

large treesone rule

lemur triggers rule