generative and discriminative learning - · pdf filemachine learning fall 2017 generative and...

of 20/20
Machine Learning Fall 2017 Generative and Discriminative Learning 1

Post on 07-Feb-2018

214 views

Category:

Documents

0 download

Embed Size (px)

TRANSCRIPT

  • MachineLearningFall2017

    GenerativeandDiscriminativeLearning

    1

  • Whatwesawmostofthesemester

    Afixed,unknowndistributionDoverX Y X:Instancespace,Y:labelspace(eg:{+1,-1})

    GivenadatasetS={(xi,yi)}

    Learning IdentifyahypothesisspaceH,definealossfunctionL(h,x,y) Minimizeaveragelossovertrainingdata(plusregularization)

    Theguarantee Ifwefindanalgorithmthatminimizeslossontheobserveddata Then,learningtheoryguaranteesgoodfuturebehavior(asafunction

    ofH)

    2

  • Whatwesawmostofthesemester

    Afixed,unknowndistributionDoverX Y X:Instancespace,Y:labelspace(eg:{+1,-1})

    GivenadatasetS={(xi,yi)}

    Learning IdentifyahypothesisspaceH,definealossfunctionL(h,x,y) Minimizeaveragelossovertrainingdata(plusregularization)

    Theguarantee Ifwefindanalgorithmthatminimizeslossontheobserveddata Then,learningtheoryguaranteesgoodfuturebehavior(asafunction

    ofH)

    3

    IsthisdifferentfromassumingadistributionoverXandafixedoraclefunctionf?

  • Whatwesawmostofthesemester

    Afixed,unknowndistributionDoverX Y X:Instancespace,Y:labelspace(eg:{+1,-1})

    GivenadatasetS={(xi,yi)}

    Learning IdentifyahypothesisspaceH,definealossfunctionL(h,x,y) Minimizeaveragelossovertrainingdata(plusregularization)

    Theguarantee Ifwefindanalgorithmthatminimizeslossontheobserveddata Then,learningtheoryguaranteesgoodfuturebehavior(asafunction

    ofH)

    4

    IsthisdifferentfromassumingadistributionoverXandafixedoraclefunctionf?

  • Whatwesawmostofthesemester

    Afixed,unknowndistributionDoverX Y X:Instancespace,Y:labelspace(eg:{+1,-1})

    GivenadatasetS={(xi,yi)}

    Learning IdentifyahypothesisspaceH,definealossfunctionL(h,x,y) Minimizeaveragelossovertrainingdata(plusregularization)

    Theguarantee Ifwefindanalgorithmthatminimizeslossontheobserveddata Then,learningtheoryguaranteesgoodfuturebehavior(asafunction

    ofH)

    5

    IsthisdifferentfromassumingadistributionoverXandafixedoraclefunctionf?

  • Whatwesawmostofthesemester

    Afixed,unknowndistributionDoverX Y X:Instancespace,Y:labelspace(eg:{+1,-1})

    GivenadatasetS={(xi,yi)}

    Learning IdentifyahypothesisspaceH,definealossfunctionL(h,x,y) Minimizeaveragelossovertrainingdata(plusregularization)

    Theguarantee Ifwefindanalgorithmthatminimizeslossontheobserveddata Then,learningtheoryguaranteesgoodfuturebehavior(asafunction

    ofH)

    6

    IsthisdifferentfromassumingadistributionoverXandafixedoraclefunctionf?

  • Discriminativemodels

    Goal:learndirectlyhowtomakepredictions

    Lookatmany(positive/negative)examples

    Discoverregularitiesinthedata

    Usethesetoconstructapredictionpolicy

    AssumptionscomeintheformofthehypothesisclassBottomline:approximatingh:XYis

    estimatingP(Y|X)7

  • Generativemodels

    Explicitlymodelhowinstancesineachcategoryaregenerated

    Thatis,learnP(X|Y) andP(Y)

    WedidthisfornaveBayes NaveBayesisagenerativemodel

    PredictP(Y|X)usingtheBayesrule

    8

  • Example:GenerativestoryofnaveBayes

    9

  • Example:GenerativestoryofnaveBayes

    10

    YP(Y)

    Firstsamplealabel

  • Example:GenerativestoryofnaveBayes

    11

    YP(Y)

    X1

    P(X1 |Y)

    Giventhelabel,samplethefeaturesindependentlyfromtheconditionaldistributions

  • Example:GenerativestoryofnaveBayes

    12

    YP(Y)

    X1

    P(X1 |Y)

    X2

    P(X2 |Y)

    Giventhelabel,samplethefeaturesindependentlyfromtheconditionaldistributions

  • Example:GenerativestoryofnaveBayes

    13

    YP(Y)

    X1

    P(X1 |Y)

    X2

    P(X2 |Y)

    X3

    P(X3 |Y)

    Giventhelabel,samplethefeaturesindependentlyfromtheconditionaldistributions

  • Example:GenerativestoryofnaveBayes

    14

    YP(Y)

    X1

    P(X1 |Y)

    X2

    P(X2 |Y)

    X3

    P(X3 |Y)

    ...

    Giventhelabel,samplethefeaturesindependentlyfromtheconditionaldistributions

  • Example:GenerativestoryofnaveBayes

    15

    YP(Y)

    X1

    P(X1 |Y)

    X2

    P(X2 |Y)

    X3

    P(X3 |Y)

    Xd

    P(Xd |Y)

    ...

    Giventhelabel,samplethefeaturesindependentlyfromtheconditionaldistributions

  • Generativemodels learnP(x,y) Characterizehowthedataisgenerated(bothinputsandoutputs) Eg:NaveBayes,HiddenMarkovModel

    Discriminativemodels learnP(y|x) Directlycharacterizesthedecisionboundaryonly Eg:LogisticRegression,Conditionalmodels(severalnames)

    Generativevs Discriminativemodels

    16

  • Generativemodels learnP(x,y) Characterizehowthedataisgenerated(bothinputsandoutputs) Eg:NaveBayes,HiddenMarkovModel

    Discriminativemodels learnP(y|x) Directlycharacterizesthedecisionboundaryonly Eg:LogisticRegression,Conditionalmodels(severalnames)

    Generativevs Discriminativemodels

    17

  • Generativemodels learnP(x,y) Characterizehowthedataisgenerated(bothinputsandoutputs) Eg:NaveBayes,HiddenMarkovModel

    Discriminativemodels learnP(y|x) Directlycharacterizesthedecisionboundaryonly Eg:LogisticRegression,Conditionalmodels(severalnames)

    Generativevs Discriminativemodels

    18

  • Generativemodels learnP(x,y) Characterizehowthedataisgenerated(bothinputsandoutputs) Eg:NaveBayes,HiddenMarkovModel

    Discriminativemodels learnP(y|x) Directlycharacterizesthedecisionboundaryonly Eg:LogisticRegression,Conditionalmodels(severalnames)

    Generativevs Discriminativemodels

    19

  • Generativemodels learnP(x,y) Characterizehowthedataisgenerated(bothinputsandoutputs) Eg:NaveBayes,HiddenMarkovModel

    Discriminativemodels learnP(y|x) Directlycharacterizesthedecisionboundaryonly Eg:LogisticRegression,Conditionalmodels(severalnames)

    Generativevs Discriminativemodels

    Agenerativemodeltriestocharacterizethedistributionoftheinputs,adiscriminativemodeldoesntcare

    20