lexical semantics wrap-up and midterm reviekathy/nlp/2019/classslides/class13... · 2019-10-16 ·...
TRANSCRIPT
![Page 1: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/1.jpg)
LexicalSemanticsWrap-upandMidtermReview
![Page 2: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/2.jpg)
Howdoweknowwhenawordhasmorethanonesense?• ATISexamples
• Whichflightsservebreakfast?• DoesAmericaWestservePhiladelphia?
• The“zeugma”test:
• ?DoesUnitedservebreakfastandSanJose?
![Page 3: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/3.jpg)
Synonyms• Wordthathavethesamemeaninginsomeorallcontexts.• filbert/hazelnut• couch/sofa• big/large• automobile/car• vomit/throwup• Water/H20
• TwolexemesaresynonymsiftheycanbesuccessfullysubsNtutedforeachotherinallsituaNons• Ifsotheyhavethesameproposi&onalmeaning
![Page 4: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/4.jpg)
Synonyms• Buttherearefew(orno)examplesofperfectsynonymy.• Whyshouldthatbe?• EvenifmanyaspectsofmeaningareidenNcal• SNllmaynotpreservetheacceptabilitybasedonnoNonsofpoliteness,slang,register,genre,etc.
• Example:• WaterandH20
![Page 5: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/5.jpg)
Somemoreterminology• Lemmasandwordforms
• Alexemeisanabstractpairingofmeaningandform• Alemmaorcita&onformisthegrammaNcalformthatisusedtorepresentalexeme.
• Carpetisthelemmaforcarpets• Dormiristhelemmaforduermes.
• Specificsurfaceformscarpets,sung,duermesarecalledwordforms
• Thelemmabankhastwosenses:• Instead,abankcanholdtheinvestmentsinacustodialaccountintheclient’sname
• Butasagricultureburgeonsontheeastbank,theriverwillshrinkevenmore.
• AsenseisadiscreterepresentaNonofoneaspectofthemeaningofaword
![Page 6: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/6.jpg)
Synonymyisarelationbetweensensesratherthanwords
• Considerthewordsbigandlarge• Aretheysynonyms?
• Howbigisthatplane?• WouldIbeflyingonalargeorsmallplane?
• Howabouthere:• MissNelson,forinstance,becameakindofbigsistertoBenjamin.
• ?MissNelson,forinstance,becameakindoflargesistertoBenjamin.
• Why?• bighasasensethatmeansbeingolder,orgrownup• largelacksthissense
![Page 7: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/7.jpg)
Antonyms• Sensesthatareoppositeswithrespecttoonefeatureoftheirmeaning
• Otherwise,theyareverysimilar!• dark/light• short/long• hot/cold• up/down• in/out
• Moreformally:antonymscan• defineabinaryopposiNonoratoppositeendsofascale(long/short,fast/slow)
• Bereversives:rise/fall,up/down
![Page 8: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/8.jpg)
Hyponymy• Onesenseisahyponymofanotherifthefirstsenseismorespecific,denoNngasubclassoftheother• carisahyponymofvehicle• dogisahyponymofanimal• mangoisahyponymoffruit
• Conversely• vehicleisahypernym/superordinateofcar• animalisahypernymofdog• fruitisahypernymofmango
superordinate vehicle fruit furniture mammal
hyponym car mango chair dog
![Page 9: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/9.jpg)
Hypernymymoreformally• Extensional:
• Theclassdenotedbythesuperordinate• extensionallyincludestheclassdenotedbythehyponym
• Entailment:• AsenseAisahyponymofsenseBifbeinganAentailsbeingaB
• HyponymyisusuallytransiNve• (AhypoBandBhypoCentailsAhypoC)
![Page 10: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/10.jpg)
• Whywouldhypernyms/hyponymsbeimportanttoconstrucNngameaningrepresentaNon?
![Page 11: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/11.jpg)
![Page 12: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/12.jpg)
II.WordNet• Ahierarchicallyorganizedlexicaldatabase• On-linethesaurus+aspectsofadicNonary
• Versionsforotherlanguagesareunderdevelopment
Category Unique Forms
Noun 117,097
Verb 11,488
Adjective 22,141
Adverb 4,601
![Page 13: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/13.jpg)
WordNet
• Whereitis:• h_ps://wordnet.princeton.edu/
![Page 14: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/14.jpg)
FormatofWordnetEntries
![Page 15: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/15.jpg)
WordNetNounRelations
![Page 16: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/16.jpg)
WordNetVerbRelations
![Page 17: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/17.jpg)
WordNetHierarchies
![Page 18: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/18.jpg)
Howis“sense”deJinedinWordNet?• Thesetofnear-synonymsforaWordNetsenseiscalledasynset(synonymset);it’stheirversionofasenseoraconcept
• Example:chumpasanountomean• ‘apersonwhoisgullibleandeasytotakeadvantageof’
• Eachofthesesensessharethissamegloss• ThusforWordNet,themeaningofthissenseofchumpisthislist.
![Page 19: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/19.jpg)
Wordnetexample
![Page 20: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/20.jpg)
Questions?
![Page 21: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/21.jpg)
Midterm• Format
• MulNpleChoicequesNons• ShortanswerquesNons• Problemsolving
• Whatwillitcover?• Anythingcoveredinclass• Fromreadingthatsupportsmaterialinclass• Mathasneededforneuralnets,machinelearning,smoothing
![Page 22: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/22.jpg)
Midterm• Closedbook,nonotes,noelectronics
• Willavoidaskingyoutorecallformulas• Thatsaid,youshouldknowhowtocomputetheprobabilityofngrams,ofPOStags,basicsforsmoothing,languagemodeling,howtodocomputaNonforneuralnets.
• Willcoveranythingfrombeginningthroughtoday
• SamplemidtermquesNonsposted
![Page 23: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/23.jpg)
Toptopics• Viterbialgorithm• Dependencyparsing• RNNs
![Page 24: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/24.jpg)
Questions?
![Page 25: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/25.jpg)
ViterbiandPOS
![Page 26: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/26.jpg)
Twokindsofprobabilities(1)• TagtransiNonprobabiliNesp(ti|ti-1)
• Determinerslikelytoprecedeadjsandnouns• That/DTflight/NN• The/DTyellow/JJhat/NN• SoweexpectP(NN|DT)andP(JJ|DT)tobehigh• ButP(DT|JJ)tobelow
• ComputeP(NN|DT)bycounNnginalabeledcorpus:
![Page 27: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/27.jpg)
Twokindsofprobabilities(2)• WordlikelihoodprobabiliNesp(wi|ti)
• VBZ(3sgPresverb)likelytobe“is”• ComputeP(is|VBZ)bycounNnginalabeledcorpus:
10/16/19
27
![Page 28: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/28.jpg)
AnExample:theverb“race”
• Secretariat/NNPis/VBZexpected/VBNto/TOrace/VBtomorrow/NR
• People/NNSconNnue/VBto/TOinquire/VBthe/DTreason/NNfor/INthe/DTrace/NNfor/INouter/JJspace/NN
• Howdowepicktherighttag? 10/16/19
28
![Page 29: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/29.jpg)
Disambiguating“race”
10/16/19
29
![Page 30: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/30.jpg)
Disambiguating“race”
10/16/19
30
![Page 31: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/31.jpg)
Disambiguating“race”
10/16/19
31
![Page 32: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/32.jpg)
Disambiguating“race”
10/16/19
32
![Page 33: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/33.jpg)
• P(NN|TO)=.00047• P(VB|TO)=.83• P(race|NN)=.00057• P(race|VB)=.00012• P(NR|VB)=.0027• P(NR|NN)=.0012• P(VB|TO)P(NR|VB)P(race|VB)=.00000027• P(NN|TO)P(NR|NN)P(race|NN)=.00000000032• Sowe(correctly)choosetheverbreading,
10/16/19
33
![Page 34: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/34.jpg)
HMMS
![Page 35: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/35.jpg)
HiddenMarkovModels• Wedon’tobservePOStags
• Weinferthemfromthewordswesee
• Observedevents
• Hiddenevents
10/16/19
35
![Page 36: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/36.jpg)
HiddenMarkovModel• ForMarkovchains,theoutputsymbolsarethesameasthestates.• Seehotweather:we’reinstatehot
• Butinpart-of-speechtagging(andotherthings)• Theoutputsymbolsarewords• Thehiddenstatesarepart-of-speechtags
• Soweneedanextension!• AHiddenMarkovModelisanextensionofaMarkovchaininwhichtheinputsymbolsarenotthesameasthestates.
• Thismeanswedon’tknowwhichstatewearein.
10/16/19
36
![Page 37: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/37.jpg)
HiddenMarkovModels• StatesQ = q1, q2…qN; • ObservaNonsO= o1, o2…oN;
• EachobservaNonisasymbolfromavocabularyV={v1,v2,…vV}
• TransiNonprobabiliNes• Transition probability matrix A = {aij}
• ObservaNonlikelihoods• Output probability matrix B={bi(k)}
• SpecialiniNalprobabilityvectorπ
€
π i = P(q1 = i) 1≤ i ≤ N
€
aij = P(qt = j |qt−1 = i) 1≤ i, j ≤ N
€
bi(k) = P(Xt = ok |qt = i)
![Page 38: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/38.jpg)
HiddenMarkovModels
• Someconstraints
10/16/19
38
€
π i = P(q1 = i) 1≤ i ≤ N
€
aij =1; 1≤ i ≤ Nj=1
N
∑
€
bi(k) =1k=1
M
∑
€
π j =1j=1
N
∑
![Page 39: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/39.jpg)
Assumptions• Markovassump&on:
• Output-independenceassump&on
10/16/19
39
€
P(qi |q1...qi−1) = P(qi |qi−1)
€
P(ot |O1t−1,q1
t ) = P(ot |qt )
![Page 40: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/40.jpg)
ThreefundamentalProblemsforHMMs
• Likelihood:GivenanHMMλ=(A,B)andanobservaNonsequenceO,determinethelikelihoodP(O,λ).
• Decoding:GivenanobservaNonsequenceOandanHMMλ=(A,B),discoverthebesthiddenstatesequenceQ.
• Learning:GivenanobservaNonsequenceOandthesetofstatesintheHMM,learntheHMMparametersAandB.WhatkindofdatawouldweneedtolearntheHMMparameters?
10/16/19
40
![Page 41: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/41.jpg)
Decoding• Thebesthiddensequence
• Weathersequenceintheicecreamtask• POSsequencegivenaninputsentence
• Wecoulduseargmaxovertheprobabilityofeachpossiblehiddenstatesequence• Whynot?
• Viterbialgorithm• Dynamicprogrammingalgorithm• Usesadynamicprogrammingtrellis
• Eachtrelliscellrepresents,vt(j),representstheprobabilitythattheHMMisinstatejaterseeingthefirsttobservaNonsandpassingthroughthemostlikelystatesequence 10/1
6/19
41
![Page 42: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/42.jpg)
Viterbiintuition:wearelookingforthebest‘path’
10/16/19
42
promised to back the bill
VBD
VBN
TO
VB
JJ
NN
RB
DT
NNP
VB
NN
promised to back the bill
VBD
VBN
TO
VB
JJ
NN
RB
DT
NNP
VB
NN
S1 S2 S4 S3 S5
promised to back the bill
VBD
VBN
TO
VB
JJ
NN
RB
DT
NNP
VB
NN
Slide from Dekang Lin
![Page 43: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/43.jpg)
Intuition• ThevalueineachcelliscomputedbytakingtheMAXoverallpathsthatleadtothiscell.
• AnextensionofapathfromstateiatNmet-1iscomputedbymulNplying:
10/16/19
43
![Page 44: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/44.jpg)
TheViterbiAlgorithm
10/16/19
44
![Page 45: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/45.jpg)
TheAmatrixforthePOSHMM
10/16/19
45
What is P(VB|TO)? What is P(NN|TO)? Why does this make sense? What is P(TO|VB)? What is P(TO|NN)? Why does this make sense?
![Page 46: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/46.jpg)
TheBmatrixforthePOSHMM
10/16/19
46
Look at P(want|VB) and P(want|NN). Give an explanation for the difference in the probabilities.
![Page 47: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/47.jpg)
![Page 48: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/48.jpg)
Problem• Iwanttorace(possiblestates:PPSVBTONN)
![Page 49: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/49.jpg)
Viterbiexample
10/16/19
49
t=1
![Page 50: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/50.jpg)
Viterbiexample
10/16/19
50
t=1
X
![Page 51: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/51.jpg)
Viterbiexample
10/16/19
51
t=1
X
J=NN
I=S
![Page 52: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/52.jpg)
TheAmatrixforthePOSHMM
10/16/19
52
![Page 53: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/53.jpg)
Viterbiexample
10/16/19
53
t=1
X
J=NN
I=S
.041X
![Page 54: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/54.jpg)
TheBmatrixforthePOSHMM
10/16/19
54
Look at P(want|VB) and P(want|NN). Give an explanation for the difference in the probabilities.
![Page 55: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/55.jpg)
Viterbiexample
10/16/19
55
t=1
X
J=NN
I=S
.041X 0 0
![Page 56: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/56.jpg)
Viterbiexample
10/16/19
56
t=1
X
J=NN
I=S
.041X 0 0
0
0
.025
![Page 57: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/57.jpg)
Viterbiexample
10/16/19
57
t=1
J=NN
I=S
0
0
0
.025
Show the 4 formulas you would use to compute the value at this node and the max.
![Page 58: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/58.jpg)
DependencyParsing
![Page 59: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/59.jpg)
Dependencyparsing• AnexamplefromtheNYTimestoday:
Lastweek,onthethirdfloorofasmallbuildinginSanFrancisco’sMissionDistrict,awomanscrambledtheDlesofaRubik’sCube
![Page 60: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/60.jpg)
Dependencyparsing• AnexamplefromtheNYTimestoday:
Lastweek,onthethirdfloorofasmallbuildinginSanFrancisco’sMissionDistrict,awomanscrambledtheDlesofaRubik’sCube
![Page 61: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/61.jpg)
Dependencyparsing• AnexamplefromtheNYTimestoday:
Lastweek,onthethirdfloor,awomanscrambledtheDlesofaRubik’sCube
![Page 62: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/62.jpg)
RNNsandLSTMs
![Page 63: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/63.jpg)
wx
wh
RecurrentNeuralNetworks
x1
h0
h1
ℎ↓𝑡 = 𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
σ
Slide from Radev
![Page 64: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/64.jpg)
wx
wh
RNN
x1
h0
h1
ℎ↓𝑡 = 𝜎(𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 ) 𝑦↓𝑡 =𝑠𝑜𝑓𝑡𝑚𝑎𝑥( 𝑊↓𝑦 ℎ↓𝑡 )
σ
y1soGmax
wy
Slide from Radev
![Page 65: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/65.jpg)
RNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3soGmax
The
cat sat
wy
Slide from Radev
![Page 66: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/66.jpg)
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3soGmax
The
cat sat
Cost
wy
Backpropagation through time
Slide from Radev
![Page 67: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/67.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
I had
in
wx
wh
x3
h3
σ
mind
…
In this overview, w refers to the weights But there are different kinds of weights Let’s be more specific
![Page 68: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/68.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had
in
wx
U
x3
h3
σ
mind
…
σ
![Page 69: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/69.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had
in
wx
U
x3
h3
σ
mind
…
W are the weights: the word embedding matrix multiplication with xt yields the embedding for x U is another weight matrix H0 is often not specified. H is the hidden layer.
σ
![Page 70: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/70.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
U
x1
h0
h1
wx
U
x2
h2
σwx
U
x3
h3
σ
y3sigmoid
I had
in
wx
U
x3
h3
σ
mind
…
ht = σ ( U wxt ) ht-1
σ
![Page 71: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/71.jpg)
RNN–Ihadinmindyourfacts,buddy,nothers.
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
Sigmoid
I had
in
wx
wh
x3
h3
σ
mind
…
Y = positive? Y = negative? Final embedding run through the sigmoid
function -> [0,1] 1 = positive 0= negative Often final h is used as word embedding for the sentence
![Page 72: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/72.jpg)
UpdatingParametersofanRNN
wx
wh
x1
h0
h1
σwx
wh
x2
h2
σwx
wh
x3
h3
σ
y3sigmoid
Cost
wy
Backpropagation through time Gold label = 0 (negative) Adjust weights using gradient Repeat many times with all examples
Slide from Radev
I had
in
![Page 73: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/73.jpg)
TransformingRNNtoLSTM
𝑢↓𝑡 = 𝜎( 𝑊↓ℎ ℎ↓𝑡−1 + 𝑊↓𝑥 𝑥↓𝑡 )
wx
wh
x1
h0
u1
σ
[slides from Catherine Finegan-Dollak]
![Page 74: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/74.jpg)
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
[slides from Catherine Finegan-Dollak]
![Page 75: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/75.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slides from Catherine Finegan-Dollak]
![Page 76: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/76.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slides from Catherine Finegan-Dollak]
![Page 77: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/77.jpg)
TransformingRNNtoLSTM
𝑐↓𝑡 = 𝑓↓𝑡 ⊙ 𝑐↓𝑡−1 + 𝑖↓𝑡 ⊙ 𝑢↓𝑡
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
[slides from Catherine Finegan-Dollak]
![Page 78: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/78.jpg)
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑓↓𝑡 = 𝜎( 𝑊↓ℎ𝑓 ℎ↓𝑡−1 + 𝑊↓𝑥𝑓 𝑥↓𝑡 ) f1
x1
h0
σwhf
wxf
[slides from Catherine Finegan-Dollak]
![Page 79: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/79.jpg)
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1
TransformingRNNtoLSTM
𝑖↓𝑡 = 𝜎( 𝑊↓ℎ𝑖 ℎ↓𝑡−1 + 𝑊↓𝑥𝑖 𝑥↓𝑡 )
x1
h0
σwhi
wxi
i1
[slides from Catherine Finegan-Dollak]
![Page 80: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/80.jpg)
TransformingRNNtoLSTM
wx
wh
x1
h0
u1
σ
c0
+ c1
f1
i1h1
tanh
o1
ℎ↓𝑡 = 𝑜↓𝑡 ⊙ tanh𝑐↓𝑡
[slides from Catherine Finegan-Dollak]
![Page 81: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/81.jpg)
LSTMforSequences
wx
wh
x1
h0
u1
σ
c0
+c1
f1
i1h1
tanh
o1wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2wx
wh
x2
u2
σ
+c2
f2
i2h2
tanh
o2
The cat sat
[slides from Catherine Finegan-Dollak]
![Page 82: Lexical Semantics Wrap-up and Midterm Reviekathy/NLP/2019/ClassSlides/Class13... · 2019-10-16 · • See hot weather: we’re in state hot • But in part-of-speech tagging (and](https://reader033.vdocuments.mx/reader033/viewer/2022041823/5e5f79b01479584990518c48/html5/thumbnails/82.jpg)
Problem10fromsamplemidtermquestions