cs 7650: natural language processing
TRANSCRIPT
CS7650:NaturalLanguageProcessing
WeiXu(many slides from Greg Durrett)
Administrivia
‣ Coursewebsite: hAps://cocoxu.github.io/CS7650_fall2021/
‣ PiazzaandGradescope:‣ linksonthecoursewebsite‣ WewilldoourbesttomakesurequesNonsaboutthehomework,etc.getansweredwithin24hours
‣ TAOfficehours:TBA
CourseRequirements
‣ Priorexposuretomachinelearningveryhelpful
‣ Programming/Pythonexperience
‣ Probability
‣ LinearAlgebra
‣ MulNvariableCalculus
There will be a lot of math and programming!
FreeTextbooks!
‣ 2reallyawesomefreetextbooksavailable
‣ Therewillbeassignedreadingsfromboth
‣ Bothfreelyavailableonline
CourseworkPlan
‣ ProblemSet1(mathreview)willbereleasedlaterthisweekonGradeScope.
‣ 3ProgrammingProjects(40%;fairlysubstanNalimplementaNoneffort)
‣ TextclassificaNon
‣ NamedenNtyrecogniNon(BiLSTM-CNN-CRF)
‣ Neuralchatbot(Seq2SeqwithaAenNon)
‣ 2wriAenassignments(20%)+midtermexam(15%)
‣ MostlymathproblemsrelatedtoML/NLP
‣ Finalproject(20%;detailsoncoursewebsite,willdiscusslater)
{subject to change
ProgrammingProjects‣ ModernNLPmethodsrequirenon-trivialcomputaNon
‣ TrainingneuralnetworkswithmanyparameterscantakealongNme(itisaverygoodideatostartworkingontheassignmentsearly!)
‣ MostprogrammingwillbedonewithPyTorchlibrary(canbetrickytodebug)
‣ YouwillwanttouseaGPU
‣ GoogleColab:freeGPUs(somelimitaNons;proaccountfor$10/month)
‣ TheprogrammingprojectsaredesignedwithColabinmind
What’sthegoalofNLP?
‣ Beabletosolveproblemsthatrequiredeepunderstandingoftext
Siri,what’sthemostvaluableAmerican
company?
Apple
recognizemarketCapisthetargetvalue
recognizepredicate
docomputaNon
WhoisitsCEO?
‣ Example:dialoguesystems
resolvereferences
TimCook
AutomaNcSummarizaNon
…
…
OneofNewAmerica’swriterspostedastatementcriNcalofGoogle.EricSchmidt,Google’sCEO,wasdispleased.
Thewriterandhisteamweredismissed.
providemissingcontext
paraphrasetoprovideclarity
compresstext
MachineTranslaNon
TrumpPopefamilywatchahundredyearsayearintheWhiteHousebalcony
People’sDaily,August30,2017
NLPAnalysisPipeline
SyntacNcparses
CoreferenceresoluNon
EnNtydisambiguaNon
Discourseanalysis
Summarize
ExtractinformaNon
AnswerquesNons
IdenNfysenNment
‣ NLPisaboutbuildingthesepieces!Translate
TextAnalysis Applica/onsText Annota/ons
‣ AllofthesecomponentsaremodeledwithstaNsNcal approachestrainedwithmachinelearning
Howdowerepresentlanguage?Labels
Sequences/tags
Trees
Text
themoviewasgood +Beyoncéhadoneofthebestvideosofall6me subjec/ve
TomCruisestarsinthenewMissionImpossiblefilmPERSON MOVIE
Ieatcakewithicing
PPNP
S
NPVP
VBZ NNflightstoMiami
λx.flight(x)∧dest(x)=Miami
HowdoweusetheserepresentaNons?
Labels
Sequences
Trees
TextAnalysisText
‣MainquesNon:WhatrepresentaNonsdoweneedforlanguage?Whatdowewanttoknowaboutit?
‣ Boilsdownto:whatambiguiNesdoweneedtoresolve?
…
Applica/ons
Treetransducers(formachinetranslaNon)
ExtractsyntacNcfeatures
Tree-structuredneuralnetworks
end-to-endmodels …
Whyislanguagehard? (andhowcanwehandlethat?)
LanguageisAmbiguous!
‣ HectorLevesque(2011):“Winogradschemachallenge”(namedarerTerryWinograd,thecreatorofSHRDLU)
Thecitycouncilrefusedthedemonstratorsapermitbecausethey______violence
theyfeared
theyadvocated
‣ Thisissocomplicatedthatit’sanAIchallengeproblem!(AI-complete)
‣ ReferenNal/semanNcambiguity
LanguageisAmbiguous!
‣ AmbiguousNewsHeadlines:
slidecredit:DanKlein
‣ SyntacNc/semanNcambiguity:parsingneededtoresolvethese,butneedcontexttofigureoutwhichparseiscorrect
‣ TeacherStrikesIdleKids‣ HospitalsSuedby7FootDoctors‣ BanonNudeDancingonGovernor’sDesk‣ IraqiHeadSeeksArms
‣ StolenPainNngFoundbyTree‣ KidsMakeNutriNousSnacks‣ LocalHSDropoutsCutinHalf
LanguageisReallyAmbiguous!
‣ Therearen’tjustoneortwopossibiliNeswhichareresolvedpragmaNcally
‣ CombinatoriallymanypossibiliNes,manyyouwon’tevenregisterasambiguiNes,butsystemssNllhavetoresolvethem
Itisreallyniceout
ilfaitvraimentbeau It’sreallyniceTheweatherisbeauNfulItisreallybeauNfuloutsideHemakestrulybeauNful
ItfactactuallyhandsomeHemakestrulyboyfriend
‣ Lotsofdata!
slidecredit:DanKlein
Whatdoweneedtounderstandlanguage?
Whatdoweneedtounderstandlanguage?
‣ Worldknowledge:haveaccesstoinformaNonbeyondthetrainingdata
DOJgreenlightsDisney-Foxmerger
metaphor;“approves”
DepartmentofJus6ce
‣ Whatisagreenlight?Howdoweunderstandwhat“greenlighNng”does?
‣ Grounding:learnwhatfundamentalconceptsactuallymeaninadata-drivenway
McMahanandStone(2015)Gollandetal.(2010)
Whatdoweneedtounderstandlanguage?
‣ LinguisNcstructure‣ …butcomputersprobablywon’tunderstandlanguagethesamewayhumansdo
‣ However,linguisNcstellsuswhatphenomenaweneedtobeabletodealwithandgivesushintsabouthowlanguageworks
CenteringTheoryGroszetal.(1995)
Whatdoweneedtounderstandlanguage?
Whattechniquesdoweuse?(tocombinedata,knowledge,linguisNcs,etc.)
Unsup:topicmodels,grammarinducNon
Collinsvs.Charniakparsers
Abriefhistoryof(modern)NLP
1980 1990 2000 2010 2018
earlieststatMTworkatIBM
“AIwinter”rule-based,expertsystems
Penntreebank
NP VP
S
Ratnaparkhitagger
NNP VBZ
Sup:SVMs,CRFs,NER,SenNment
Neural
Pretraining
Semi-sup,structuredpredicNon
StructuredPredicNon
‣ SupervisedtechniquesworkwellonveryliAledata
annotaNon(twohours!)
unsupervisedlearning
‣ EvenneuralnetscandopreAywell!
“LearningaPart-of-SpeechTaggerfromTwoHoursofAnnotaNon” GarreAeandBaldridge(2013)
beAersystem!
‣ Allofthesetechniquesaredata-driven!Somedataisnaturallyoccurring,butmayneedtolabel
Bahdanauetal.(2014)DeNeroetal.(2008)
LessManualStructure?
Doesmanualstructurehaveaplace?
‣ Neuralnetsdon’talwaysworkoutofdomain!
MoosaviandStrube(2017)
‣ Coreference:rule-basedsystemsaresNllaboutasgoodasdeeplearningout-of-domain
‣ LORELEI:transiNonpointbelowwhichphrase-basedsystemsarebeAer
‣ Whyisthis?InducNvebias!
‣ CanmulN-tasklearninghelp?
Wikipedia
Newswire
TrumpPopefamilywatchahundredyearsayearintheWhiteHousebalcony
‣ Maybemanualstructurewouldhelp…
Doesmanualstructurehaveaplace?
Wherearewe?
‣ NLPconsistsof:analyzingandbuildingrepresentaNonsfortext,solvingproblemsinvolvingtext
‣ Theseproblemsarehardbecauselanguageisambiguous,requiresdrawingondata,knowledge,andlinguisNcstosolve
‣ Knowingwhichtechniquesuserequiresunderstandingdatasetsize,problemcomplexity,andalotoftricks!
‣ NLPencompassesallofthesethings
NLPvs.ComputaNonalLinguisNcs
‣ NLP:buildsystemsthatdealwithlanguagedata
‣ CL:usecomputaNonaltoolstostudylanguage
Hamiltonetal.(2016)
NLPvs.ComputaNonalLinguisNcs
‣ ComputaNonaltoolsforotherpurposes:literarytheory,poliNcalscience…
Bamman,O’Connor,Smith(2013)
CourseGoals
‣ CoverfundamentalmachinelearningtechniquesusedinNLP
‣ Makeyoua“producer”ratherthana“consumer”ofNLPtools
‣ CovermodernNLPproblemsencounteredintheliterature:whataretheacNveresearchtopicsin2021?
‣ Thethreeassignmentsshouldteachyouwhatyouneedtoknowtounderstandnearlyanysystemintheliterature
‣ UnderstandhowtolookatlanguagedataandapproachlinguisNcphenomena
Assignments
‣ 3ProgrammingAssignments(40%grade)
‣ ImplementaNon-oriented
‣ ~2weeksperassignment,3“slipdays”forautomaNcextensions
Theseprojectsrequireunderstandingoftheconcepts,abilitytowriteperformantcode,andabilitytothinkabouthowtodebugcomplexsystems.Theyarechallenging,sostartearly!
FinalProject
‣ Finalproject(20%grade)‣ Groupsof3-4preferred,1ispossible.‣ Goodideatotalktorunyourprojectideabymeinofficehoursoremail.
‣ 4pagereport+finalprojectpresentaNon.
QuesNons?
�33
Piazza — https://piazza.com/class/ksjq7xenrbp3g5