covering algorithm

Tilani Gunawardena

Covering(Rule-Based) Algorithms

Example: generating a rule

• Possible rule set for class “a”• If x > 1.2 then class = a• If x > 1.2 and y > 2.6 then class = a

• Possible rule set for class “b”:• If x ≤ 1.2 then class = b• If x > 1.2 and y ≤ 2.6 then class = b

• Could add more rules, get “perfect” rule set

• Corresponding decision tree: (produces exactly the same predictions)

• But: rule sets can be more perspicuous when decision trees suffer from replicated subtrees

• Also: in multiclass situations, covering algorithm concentrates on one class at a time whereas decision tree learner takes all classes into account

Rules vs Trees

• Generates a rule by adding tests that maximize rule’s accuracy

• Similar to situation in decision trees: problem of selecting an attribute to split on

• Each new test reduces rule’s coverage:

Simple covering algorithm

• Convert decision tree into a rule set – Straightforward, but rule set overly complex – More effective conversions are not trivial

• Instead, can generate rule set directly – for each class in turn find rule set that covers all

instances in it(excluding instances not in the class)

• Called a covering approach: – at each stage a rule is identified that “covers” some

of the instances

Covering Algorithms

Selecting a test

• Goal: maximize accuracy– t total number of instances covered by rule– p positive examples of the class covered by rule– t – p number of errors made by ruleÞ Select test that maximizes the ratio p/t

• We are finished when p/t = 1 or the set of instances can’t be split any further

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None

Example: Contact lens data• Rule we seek:

• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal

If ? then recommendation = hard

Example: Contact lens data• Rule we seek:

• Possible tests:Age = Young 2/8Age = Pre-presbyopic 1/8Age = Presbyopic 1/8Spectacle prescription = Myope 3/12Spectacle prescription = Hypermetrope 1/12Astigmatism = no 0/12Astigmatism = yes 4/12Tear production rate = Reduced 0/12Tear production rate = Normal 4/12

If ? then recommendation = hard

Modified rule and resulting data• Rule with best test added:

• Instances covered by modified rule:Age Spectacle

prescriptionAstigmatism Tear production

rateRecommended lenses

Young Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal hardPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None

If astigmatism = yes then recommendation = hard

Further refinement

• Current state:

• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeTear production rate = ReducedTear production rate = Normal

If astigmatism = yes and ? then recommendation = hard

Further refinement

• Current state:

• Possible tests:Age = Young 2/4Age = Pre-presbyopic 1/4Age = Presbyopic 1/4Spectacle prescription = Myope 3/6Spectacle prescription = Hypermetrope 1/6Tear production rate = Reduced 0/6Tear production rate = Normal 4/6

If astigmatism = yes and ? then recommendation = hard

Modified rule and resulting data• Rule with best test added:

• Instances covered by modified rule:

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope Yes Normal HardYoung Hypermetrope Yes Normal hardPre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope Yes Normal None

If astigmatism = yes and tear production rate = normal then recommendation = hard

Further refinement• Current state:

• Possible tests:

• Tie between the first and the fourth test– We choose the one with greater coverage

Age = Young 2/2Age = Pre-presbyopic 1/2Age = Presbyopic 1/2Spectacle prescription = Myope 3/3Spectacle prescription = Hypermetrope 1/3

If astigmatism = yes and tear production rate = normal and ?then recommendation = hard

The result

• Final rule:

• Second rule for recommending “hard lenses”:(built from instances not covered by first rule)

If astigmatism = yesand tear production rate = normaland spectacle prescription = myopethen recommendation = hard

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None

Example: Contact lens data• Rule we seek:

• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal

If ? then recommendation = hard

Example: Contact lens data• Rule we seek:

• Possible tests:Age = Young 1/7Age = Pre-presbyopic 0/7Age = Presbyopic 0/7Spectacle prescription = Myope 0/9Spectacle prescription = Hypermetrope 1/12Astigmatism = no 0/12Astigmatism = yes 1/9Tear production rate = Reduced 0/12Tear production rate = Normal 1/9

If ? then recommendation = hard

Modified rule and resulting data• Rule with best test added:

• Instances covered by modified rule:

If age = Young then recommendation = hard

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal Hard

Further refinement

• Current state:

• Possible tests:

If age = Young and ? then recommendation = hard

Spectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal

Further refinement

• Current state:

• Possible tests:

If age = Young and ? then recommendation = hard

Spectacle prescription = Myope 0/3Spectacle prescription = Hypermetrope 1/4Astigmatism = no 0/4Astigmatism = yes 1/3Tear production rate = Reduced 0/4Tear production rate = Normal 1/3

Modified rule and resulting data• Rule with best test added:

• Instances covered by modified rule:

If age = Young and Astigmatism = yesthen recommendation = hard

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope Yes Reduced NoneYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal Hard

Further refinement

• Current state:

• Possible tests:Spectacle prescription = MyopeSpectacle prescription = HypermetropeTear production rate = ReducedTear production rate = Normal

If age = Young and Astigmatism = yesand ?then recommendation = hard

Further refinement

• Current state:

• Possible tests:Spectacle prescription = Myope 0/1Spectacle prescription = Hypermetrope 1/2Tear production rate = Reduced 0/2Tear production rate = Normal 1/1

If age = Young and Astigmatism = yesand ?then recommendation = hard

Final Results

If age = Young and astigmatism = yesand tear production rate=normalthen recommendation = hard

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope Yes Normal HardYoung Hypermetrope Yes Normal Hard

If astigmatism = yesand tear production rate = normaland spectacle prescription = myopethen recommendation = hard

Age Spectacle prescription

Astigmatism Tear production rate

Recommended lenses

Young Myope Yes Normal HardPre-presbyopic Myope Yes Normal HardPresbyopic Myope Yes Normal Hard

Pseudo-code for PRISMFor each class C Initialize E to the instance set While E contains instances in class C Create a rule R with an empty left-hand side that predicts class C Until R is perfect (or there are no more attributes to use) do For each attribute A not mentioned in R, and each value v, Consider adding the condition A = v to the left-hand side of R Select A and v to maximize the accuracy p/t (break ties by choosing the condition with the largest p) Add A = v to R Remove the instances covered by R from E

• PRISM with outer loop removed generates a decision list for one class– Subsequent rules are designed for rules that are

not covered by previous rules– Order doesn’t matter because all rules predict the

same class• Outer loop considers all classes separately– No order dependence implied

Separate and conquer

• Methods like PRISM (for dealing with one class) are separate-and-conquer algorithms:– First, a rule is identified– Then, all instances covered by the rule are

separated out– Finally, the remaining instances are “conquered”

• Difference to divide-and-conquer methods:– Subset covered by rule doesn’t need to be

explored any further