Machine Learning

Concept Learning

General-to Specific Ordering

(Inductive Classification)

Note

• Simple approach– Assumption: no noise– Illustrates key concepts

Concept learning task

• Concept learning = classification (categorizatin)• Given examples, learn a general concept (category)

– subset of some general domain– boolean-valued function over the domain (characteristic function)

• Determine– infer a boolean-valued function from samples of it (input, output)

Training Examples for EnjoySport

Sky Temp Humid Wind Water Fore

Cast

Enjoy Sport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change No

• Concept learning: inferring a boolean-valued function from training examples

• Binary classification• Concept to learn: Enjoy Sport

Representing Hypotheses

• Many possible representations• Here, h is conjunction of constraints on attributes• Each constraint can be

– a specific value (e.g., Water=Warm)– don't care (e.g., Water=?)– no value allowed (e.g., Water=Ø)

• For example,

<Sky Temp Humid Wind Water Forecast ><Sunny ? ? Strong ? Same >

Concept learning task

• Given:– A description of an instance, xX, where X is the instance

language or instance space.

– A fixed set of categories: C={c1, c2,…cn}

• Determine:– The category of x: c(x)C, where c(x) is a categorization

function whose domain is X and whose range is C.– If c(x) is a binary function C={0,1} ({true,false}, {positive,

negative}, {yes, no}) then it is called a concept.

Example task

• “Days on which Aldo enjoys his water sports”– day = set of attributes– predict the value of one attribute given values of others

• Representation?– Conjunction of constraints on attributes

Sky Temp Humid Wind Water Fore

Cast

Enjoy Sport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change No

Notation

• X: set of items over which the concept is defined• c: target concept

– function X {0,1}– c(x) = 1: x is a positive example of c– c(x) = 0: x is a negative example of c

• D: set of examples <x, c(x)>• H: hypothesis space

– design choice– members of H are functions X -> 0/1

• Goal of (concept) learning– find h in H such that h(x) = c(x) for all x in X

Prototypical Concept Learning Task

• Given:– Instances X: Possible days, each described by the attributes

Sky, Temp, Humidity, Wind, Water, Forecast– Target function c: EnjoySport: X {0,1}– Hypotheses H: Conjunctions of literals. E.g.

< ?, Cold, High, ?, ?, ?>

– Training examples D: Positive and negative examples of the target function

<x1, c(x1)> , … <xn, c(xn)>

• Determine: A hypothesis h in H such that h(x)=c(x) for all x in D.

Inductive Learning Hypothesis

Any hypothesis found to approximate the target function well over a sufficiently large set of training examples will also approximate the target function well over other unobserved examples.

Notes

• Unique – most general hypothesis– most specific hypothesis

• Concept learning task requires– domain (set of instances)– target function– set of candidate hypotheses– set of (labeled) examples

Induction Learning hypothesis

• All we know of c are the examples• Best we can do is to produce a h consistent with

example data• Hypotheses

– best h regarding unseen instances is the best h regarding seen ones

– fundamental assumption in inductive learning

Concept learning as search

• H = search space– find h best fitting to D

• Selection of representation defines search space– size of space (syntactic/semantic)

• Efficient search in very large (or infinite) spaces for best h

General-to-Specific order

• Useful structure – organize the search process– exists for any concept learning task– search possible without enumerating all members of

H (may be infinite)

• h1 ‘more general than or equal’ h2– for all x: h1(x) = 1 implies h2(x) = 1– independent of c!– defines a partial order on H

Instance, Hypotheses, and More-General-Than

Find-S Algorithm

1. Initialize h to the most specific hypothesis in H

2. For each positive training instance xFor each attribute constraint ai in h

If the constraint ai in h is satisfied by x

Then do nothing

Else replace ai in h by the next more general constraint that is

satisfied by x

3. Output hypothesis h

Find-S Algorithm

Key property of Find-S

• For H defined as conjunction of constraints– finds the most specific h consistent with the positive

examples– consistent with negative ones if no noise and c is in H

Problems of Find-S

• Can not tell whether has learned c– have we converged to c?– if not, how uncertain we are of c?

• Why choose most specific h?• Is training data consistent?

– noise can severely mislead Find-S– detection, overcoming

• ‘most specific’ is not always unique (Depending on H, there might be several!)

Version Spaces and Candidate Elimination

• Another learning approach:

– outputs a description of all members of H that are consistent with D

– possible without enumerating members of H (ordering!)

– suffers from noise– useful framework for introducing many fundamental

issues of ML

Version Space

• Given an hypothesis space, H, and training data, D, the version space (VS) is the complete subset of H that is consistent with D.

• The version space can be naively generated for any finite H by enumerating all hypotheses and eliminating the inconsistent ones.

),(|, DhConsistentHhVS DH

Version Space with S and G• The version space can be represented more compactly

by maintaining two boundary sets of hypotheses, S, the set of most specific consistent hypotheses, and G, the set of most general consistent hypotheses:

• S and G represent the entire version space via its boundaries in the generalization lattice:

)]},([),(|{ DsConsistentssHsDsConsistentHsS )]},([),(|{ DsConsistentggHgDgConsistentHgG

versionspace

G

S

List-Then-Eliminate Algorithm

• VersionSpace a list containing every hypothesis in H

• For each training example, <x, c(x)> remove from VersionSpace any hypothesis hfor which h(x) ≠ c(x)

• Output the list of hypotheses in VersionSpace

Example Version Space

Sky Temp Humid Wind Water Fore

Cast

Enjoy Sport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change No

Representing Version Spaces

• The General boundary, G, of version space VSH,D is the set of its maximally general members

• The Specific boundary, S, of version space VSH,D is the set of its maximally specific members

• Every member of the version space lies between these boundaries

where x ≥ y means x is more general or equal to y

))()((|, shgGgSsHhVS DH

Elimination algorithms

• List-then-Eliminate – something to start with– ineffective

• Candidate elimination– same principle– more compact representation– maintain most specific and most general elements of

VS(H,D)

Candidate Elimination Algorithm• G maximally general hypotheses in H• S maximally specific hypotheses in H• For each training example d, do If d is a positive example

• Remove from G any hypothesis inconsistent with d• For each hypothesis s in S that is not consistent with d

– Remove s from S– Add to S all minimal generalizations h of s such that

1. h is consistent with d, and2. some member of G is more general than h

– Remove from S any hypothesis that is more general than another hypothesis in S

If d is a negative example• Remove from S any hypothesis inconsistent with d• For each hypothesis g in G that is not consistent with d

– Remove g from G– Add to G all minimal specializations h of g such that

1. h is consistent with d, and2. some member of S is more specific than h

– Remove from G any hypothesis that is less general than another hypothesis in G

Example Trace

Next training example?

Sky Temp Humid Wind Water Fore

Cast

Enjoy Sport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change No

Sky Temp Humid Wind Water Fore

Cast

Enjoy Sport

Sunny Warm Normal Strong Warm Same Yes

Sunny Warm High Strong Warm Same Yes

Rainy Cold High Strong Warm Change No

Sunny Warm High Strong Cool Change No

How Should These Be Classified?

What Justifies this Inductive Leap?

Remarks on VS & CE

• Converges to correct h?– no errors, H rich enough: yes– exact: G = S and both singletons

• Effect of errors (noise)– example 2 negative CE removes the correct target

from VS!– detection: VS gets empty– similarly when c can not be represented (e.g.

disjunctions)

What example next?

• Assume learner may ask– analogy: experiments, teacher– query: instance constructed by learner,

classified by teacher

• What to ask in example case?– Is there a good general strategy?– Should attempt to discriminate among

alternatives in VS

What to ask...

• Discriminating example– c(x) = 1: S will generalize– c(x) = 0: G will specialize– optimal choice halves VS– x satisfies half of VS members– not possible to generate one in general

How to use partially learned concepts?

• Classification with ambiguous VS– h(x) = 0/1 for every h in VS: ok– enough to check with G (0) & S (1)– 3rd example: 50% support for both– 4th: 66% support for 0– majority vote + confidence?– Ok if all h are equally likely

Inductive Bias

• What if c is not in H?– Use H that includes ‘everything’?– Will be quite large --> effect on learning

(generalization, # of examples)

• We concentrate us on C-E, but– results apply to any algorithm outputting any

consistent h for D

Biased hypothesis space

• Assure H contains c– make H general enough– what if it’s not?

• Example case– max specific h consistent with x1 & x2 (+) is

too general for x3 (-)– reason: we have biased the learner to

consider only conjunctions