covering algorithm
TRANSCRIPT
Tilani Gunawardena
Covering(Rule-Based) Algorithms
Example: generating a rule
• Possible rule set for class “a”• If x > 1.2 then class = a• If x > 1.2 and y > 2.6 then class = a
• Possible rule set for class “b”:• If x ≤ 1.2 then class = b• If x > 1.2 and y ≤ 2.6 then class = b
• Could add more rules, get “perfect” rule set
• Corresponding decision tree: (produces exactly the same predictions)
• But: rule sets can be more perspicuous when decision trees suffer from replicated subtrees
• Also: in multiclass situations, covering algorithm concentrates on one class at a time whereas decision tree learner takes all classes into account
Rules vs Trees
• Generates a rule by adding tests that maximize rule’s accuracy
• Similar to situation in decision trees: problem of selecting an attribute to split on
• Each new test reduces rule’s coverage:
Simple covering algorithm
• Convert decision tree into a rule set – Straightforward, but rule set overly complex – More effective conversions are not trivial
• Instead, can generate rule set directly – for each class in turn find rule set that covers all
instances in it(excluding instances not in the class)
• Called a covering approach: – at each stage a rule is identified that “covers” some
of the instances
Covering Algorithms
Selecting a test
• Goal: maximize accuracy– t total number of instances covered by rule– p positive examples of the class covered by rule– t – p number of errors made by ruleÞ Select test that maximizes the ratio p/t
• We are finished when p/t = 1 or the set of instances can’t be split any further
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None
Example: Contact lens data• Rule we seek:
• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal
If ? then recommendation = hard
Example: Contact lens data• Rule we seek:
• Possible tests:Age = Young 2/8Age = Pre-presbyopic 1/8Age = Presbyopic 1/8Spectacle prescription = Myope 3/12Spectacle prescription = Hypermetrope 1/12Astigmatism = no 0/12Astigmatism = yes 4/12Tear production rate = Reduced 0/12Tear production rate = Normal 4/12
If ? then recommendation = hard
Modified rule and resulting data• Rule with best test added:
• Instances covered by modified rule:Age Spectacle
prescriptionAstigmatism Tear production
rateRecommended lenses
Young Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal hardPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None
If astigmatism = yes then recommendation = hard
Further refinement
• Current state:
• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeTear production rate = ReducedTear production rate = Normal
If astigmatism = yes and ? then recommendation = hard
Further refinement
• Current state:
• Possible tests:Age = Young 2/4Age = Pre-presbyopic 1/4Age = Presbyopic 1/4Spectacle prescription = Myope 3/6Spectacle prescription = Hypermetrope 1/6Tear production rate = Reduced 0/6Tear production rate = Normal 4/6
If astigmatism = yes and ? then recommendation = hard
Modified rule and resulting data• Rule with best test added:
• Instances covered by modified rule:
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope Yes Normal HardYoung Hypermetrope Yes Normal hardPre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope Yes Normal None
If astigmatism = yes and tear production rate = normal then recommendation = hard
Further refinement• Current state:
• Possible tests:
• Tie between the first and the fourth test– We choose the one with greater coverage
Age = Young 2/2Age = Pre-presbyopic 1/2Age = Presbyopic 1/2Spectacle prescription = Myope 3/3Spectacle prescription = Hypermetrope 1/3
If astigmatism = yes and tear production rate = normal and ?then recommendation = hard
The result
• Final rule:
• Second rule for recommending “hard lenses”:(built from instances not covered by first rule)
If astigmatism = yesand tear production rate = normaland spectacle prescription = myopethen recommendation = hard
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Myope Yes Normal HardYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Myope Yes Normal HardPre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Myope Yes Normal HardPresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal HardPre-presbyopic Myope No Reduced NonePre-presbyopic Myope No Normal SoftPre-presbyopic Myope Yes Reduced NonePre-presbyopic Hypermetrope No Reduced NonePre-presbyopic Hypermetrope No Normal SoftPre-presbyopic Hypermetrope Yes Reduced NonePre-presbyopic Hypermetrope Yes Normal NonePresbyopic Myope No Reduced NonePresbyopic Myope No Normal NonePresbyopic Myope Yes Reduced NonePresbyopic Hypermetrope No Reduced NonePresbyopic Hypermetrope No Normal SoftPresbyopic Hypermetrope Yes Reduced NonePresbyopic Hypermetrope Yes Normal None
Example: Contact lens data• Rule we seek:
• Possible tests:Age = YoungAge = Pre-presbyopicAge = PresbyopicSpectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal
If ? then recommendation = hard
Example: Contact lens data• Rule we seek:
• Possible tests:Age = Young 1/7Age = Pre-presbyopic 0/7Age = Presbyopic 0/7Spectacle prescription = Myope 0/9Spectacle prescription = Hypermetrope 1/12Astigmatism = no 0/12Astigmatism = yes 1/9Tear production rate = Reduced 0/12Tear production rate = Normal 1/9
If ? then recommendation = hard
Modified rule and resulting data• Rule with best test added:
• Instances covered by modified rule:
If age = Young then recommendation = hard
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope No Reduced NoneYoung Myope No Normal SoftYoung Myope Yes Reduced NoneYoung Hypermetrope No Reduced NoneYoung Hypermetrope No Normal SoftYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal Hard
Further refinement
• Current state:
• Possible tests:
If age = Young and ? then recommendation = hard
Spectacle prescription = MyopeSpectacle prescription = HypermetropeAstigmatism = noAstigmatism = yesTear production rate = ReducedTear production rate = Normal
Further refinement
• Current state:
• Possible tests:
If age = Young and ? then recommendation = hard
Spectacle prescription = Myope 0/3Spectacle prescription = Hypermetrope 1/4Astigmatism = no 0/4Astigmatism = yes 1/3Tear production rate = Reduced 0/4Tear production rate = Normal 1/3
Modified rule and resulting data• Rule with best test added:
• Instances covered by modified rule:
If age = Young and Astigmatism = yesthen recommendation = hard
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope Yes Reduced NoneYoung Hypermetrope Yes Reduced NoneYoung Hypermetrope Yes Normal Hard
Further refinement
• Current state:
• Possible tests:Spectacle prescription = MyopeSpectacle prescription = HypermetropeTear production rate = ReducedTear production rate = Normal
If age = Young and Astigmatism = yesand ?then recommendation = hard
Further refinement
• Current state:
• Possible tests:Spectacle prescription = Myope 0/1Spectacle prescription = Hypermetrope 1/2Tear production rate = Reduced 0/2Tear production rate = Normal 1/1
If age = Young and Astigmatism = yesand ?then recommendation = hard
Final Results
If age = Young and astigmatism = yesand tear production rate=normalthen recommendation = hard
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope Yes Normal HardYoung Hypermetrope Yes Normal Hard
If astigmatism = yesand tear production rate = normaland spectacle prescription = myopethen recommendation = hard
Age Spectacle prescription
Astigmatism Tear production rate
Recommended lenses
Young Myope Yes Normal HardPre-presbyopic Myope Yes Normal HardPresbyopic Myope Yes Normal Hard
Pseudo-code for PRISMFor each class C Initialize E to the instance set While E contains instances in class C Create a rule R with an empty left-hand side that predicts class C Until R is perfect (or there are no more attributes to use) do For each attribute A not mentioned in R, and each value v, Consider adding the condition A = v to the left-hand side of R Select A and v to maximize the accuracy p/t (break ties by choosing the condition with the largest p) Add A = v to R Remove the instances covered by R from E
• PRISM with outer loop removed generates a decision list for one class– Subsequent rules are designed for rules that are
not covered by previous rules– Order doesn’t matter because all rules predict the
same class• Outer loop considers all classes separately– No order dependence implied
Separate and conquer
• Methods like PRISM (for dealing with one class) are separate-and-conquer algorithms:– First, a rule is identified– Then, all instances covered by the rule are
separated out– Finally, the remaining instances are “conquered”
• Difference to divide-and-conquer methods:– Subset covered by rule doesn’t need to be
explored any further