linguist 180: introduction to computational linguistics
DESCRIPTION
LINGUIST 180: Introduction to Computational Linguistics. Dan Jurafsky, Marie-Catherine de Marneffe Lecture 9: Grammar and Parsing (I). Thanks to Jim Martin for many of these slides!. Outline for Grammar/Parsing Week. Context-Free Grammars and Constituency - PowerPoint PPT PresentationTRANSCRIPT
1 LING 180 Autumn 2007
LINGUIST 180: Introduction to Computational Linguistics
Dan Jurafsky, Marie-Catherine de MarneffeLecture 9: Grammar and Parsing (I)
Thanks to Jim Martin for many of these slides!
2 LING 180 Autumn 2007
Outline for Grammar/Parsing Week
Context-Free Grammars and ConstituencySome common CFG phenomena for English
Sentence-level constructionsNP, PP, VPCoordinationSubcategorization
Top-down and Bottom-up ParsingDynamic Programming ParsingQuick sketch of probabilistic parsing
3 LING 180 Autumn 2007
Review
Parts of SpeechBasic syntactic/morphological categories that words belong to
Part of Speech taggingAssigning parts of speech to all the words in a sentence
4 LING 180 Autumn 2007
Syntax
Syntax: from Greek syntaxis “setting out together, arrangement’’Refers to the way words are arranged together, and the relationship between them.Distinction:
Prescriptive grammar: how people ought to talkDescriptive grammar: how they do talk
Goal of syntax is to model the knowledge of that people unconsciously have about the grammar of their native language
5 LING 180 Autumn 2007
Syntax
Why should we care?Grammar checkersQuestion answering Information extractionMachine translation
6 LING 180 Autumn 2007
key ideas of syntax
Constituency (we’ll spend most of our time on this)
SubcategorizationGrammatical relations
Plus one part we won’t have time for:Movement/long-distance dependency
7 LING 180 Autumn 2007
Context-Free Grammars (CFG)
Capture constituency and orderingOrdering:
– What are the rules that govern the ordering of words and bigger units in the language?
Constituency:How words group into units and how the various kinds of units behave
8 LING 180 Autumn 2007
Constituency
E.g., Noun phrases (NPs)
– Three parties from Brooklyn– A high-class spot such as Mindy’s– The Broadway coppers– They– Harry the Horse– The reason he comes into the Hot Box
How do we know these form a constituent?
9 LING 180 Autumn 2007
Constituency (II)
They can all appear before a verb:– Three parties from Brooklyn arrive…– A high-class spot such as Mindy’s attracts…– The Broadway coppers love…– They sit
But individual words can’t always appear before verbs:– *from arrive…– *as attracts…– *the is– *spot is…
Must be able to state generalizations like:– Noun phrases occur before verbs
10 LING 180 Autumn 2007
Constituency (III)
Preposing and postposing:On September 17th, I’d like to fly from Atlanta to DenverI’d like to fly on September 17th from Atlanta to DenverI’d like to fly from Atlanta to Denver on September 17th.
But not:*On September, I’d like to fly 17th from Atlanta to Denver*On I’d like to fly September 17th from Atlanta to Denver
11 LING 180 Autumn 2007
CFG example
S -> NP VPNP -> Det NOMINALNOMINAL -> NounVP -> VerbDet -> aNoun -> flightVerb -> left
12 LING 180 Autumn 2007
CFGs: set of rules
S -> NP VPThis says that there are units called S, NP, and VP in this languageThat an S consists of an NP followed immediately by a VPDoesn’t say that that’s the only kind of SNor does it say that this is the only place that NPs and VPs occur
13 LING 180 Autumn 2007
Generativity
As with FSAs you can view these rules as either analysis or synthesis machines
Generate strings in the languageReject strings not in the languageImpose structures (trees) on strings in the language
How can we define grammatical vs. ungrammatical sentences?
14 LING 180 Autumn 2007
Derivations
A derivation is a sequence of rules applied to a string that accounts for that string
Covers all the elements in the stringCovers only the elements in the string
15 LING 180 Autumn 2007
Derivations as Trees
S
NP VP
NP
VerbPro
Nom
Det NounNoun
I prefer morninga flight
16 LING 180 Autumn 2007
CFGs more formally
A context-free grammar has 4 parameters (“is a 4-tuple”)1) A set of non-terminal symbols (“variables”) N
2) A set of terminal symbols (disjoint from N)
3) A set of productions P, each of the form– A -> – Where A is a non-terminal and is a string of symbols
from the infinite set of strings ( N)*
4) A designated start symbol S
17 LING 180 Autumn 2007
Defining a CF language via derivation
A string A derives a string B if A can be rewritten as B via some series of rule applications
More formally:If A -> is a production of P and are any strings in the set ( N)*Then we say that A directly derives or A
Derivation is a generalization of direct derivationLet 1, 2, … m be strings in ( N)*, m>= 1, s.t. 1 2, 2 3… m-1 m
– We say that 1derives m or 1* m
We then formally define language LG generated by grammar G– A set of strings composed of terminal symbols derived from S
– LG = {w | w is in * and S * w}
18 LING 180 Autumn 2007
Parsing
Parsing is the process of taking a string and a grammar and returning a (many?) parse tree(s) for that string
19 LING 180 Autumn 2007
Context?
The notion of context in CFGs has nothing to do with the ordinary meaning of the word context in language
All it really means is that the non-terminal on the left-hand side of a rule is out there all by itself (free of context)
A -> B CMeans that I can rewrite an A as a B followed by a C
regardless of the context in which A is found
20 LING 180 Autumn 2007
Key Constituents (English)
SentencesNoun phrasesVerb phrasesPrepositional phrases
21 LING 180 Autumn 2007
Sentence-Types
Declaratives: A plane leftS -> NP VP
Imperatives: Leave!S -> VP
Yes-No Questions: Did the plane leave?S -> Aux NP VP
WH Questions: When did the plane leave?S -> WH Aux NP VP
22 LING 180 Autumn 2007
NPs
NP -> PronounI came, you saw it, they conquered
NP -> Proper-NounLos Angeles is west of TexasJohn Hennessy is the president of Stanford
NP -> Det NounThe president
NP -> Nominal
Nominal -> Noun NounA morning flight to Denver
23 LING 180 Autumn 2007
PPs
PP -> Preposition NPFrom LATo the storeOn Tuesday morningWith lunch
24 LING 180 Autumn 2007
Recursion
We’ll have to deal with rules such as the following where the non-terminal on the left also appears somewhere on the right (directly)
NP -> NP PP [[The flight] [to Boston]]VP -> VP PP [[departed Miami] [at noon]]
25 LING 180 Autumn 2007
Recursion
Of course, this is what makes syntax interesting
Flights from DenverFlights from Denver to MiamiFlights from Denver to Miami in FebruaryFlights from Denver to Miami in February on a FridayFlights from Denver to Miami in February on a Friday under $300Flights from Denver to Miami in February on a Friday under $300 with
lunch
26 LING 180 Autumn 2007
Recursion
[[Flights] [from Denver]][[[Flights] [from Denver]] [to Miami]][[[[Flights] [from Denver]] [to Miami]] [in February]][[[[[Flights] [from Denver]] [to Miami]] [in February]] [on a
Friday]]Etc.
NP -> NP PP
27 LING 180 Autumn 2007
Implications of recursion and context-freeness
If you have a rule likeVP -> V NP
It only cares that the thing after the verb is an NPIt doesn’t have to know about the internal affairs of that NP
28 LING 180 Autumn 2007
The point
VP -> V NP(I) hateflights from Denverflights from Denver to Miamiflights from Denver to Miami in Februaryflights from Denver to Miami in February on a Fridayflights from Denver to Miami in February on a Friday under $300flights from Denver to Miami in February on a Friday under $300 with
lunch
29 LING 180 Autumn 2007
Bracketed Notation
[S [NP [PRO I]] [VP [V prefer] [NP [Det a] [Nom [N morning] [N flight] ] ] ] ] S
NP VP
NP
VerbPro
Nom
Det NounNoun
I prefer morninga flight
30 LING 180 Autumn 2007
Coordination Constructions
S -> S and SJohn went to NY and Mary followed him
NP -> NP and NPVP -> VP and VP…In fact the right rule for English isX -> X and X (Metarule)
However we can say “He was longwinded and a bully.”
31 LING 180 Autumn 2007
Problems
AgreementSubcategorizationMovement (for want of a better term)
32 LING 180 Autumn 2007
Agreement
This dogThose dogs
This dog eatsThose dogs eat
*This dogs*Those dog
*This dog eat*Those dogs eats
33 LING 180 Autumn 2007
Possible CFG Solution
S -> NP VPNP -> Det NominalVP -> V NP…
SgS -> SgNP SgVPPlS -> PlNp PlVPSgNP -> SgDet SgNomPlNP -> PlDet PlNomPlVP -> PlV NPSgVP ->SgV Np…
34 LING 180 Autumn 2007
CFG Solution for Agreement
It works and stays within the power of CFGsBut it’s uglyAnd it doesn’t scale all that well
35 LING 180 Autumn 2007
Subcategorization
Sneeze: John sneezed*John sneezed the book
Say: You said [United has a flight]S
Prefer: I prefer [to leave earlier]TO-VP
*I prefer United has a flight
Give: Give [me]NP[a cheaper fare]NP
Help: Can you help [me]NP[with a flight]PP
*Give with a flight
36 LING 180 Autumn 2007
Subcategorization
Subcat expresses the constraints that a predicate (verb for now) places on the number and syntactic types of arguments it wants to take (occur with).
37 LING 180 Autumn 2007
So?
So the various rules for VPs overgenerate
They permit the presence of strings containing verbs and arguments that don’t go together
For example:VP -> V NP thereforeSneezed the book is a VP since “sneeze” is a verb and “the book” is a valid NP
38 LING 180 Autumn 2007
Possible CFG Solution
VP -> VVP -> V NPVP -> V NP PP…
VP -> IntransVVP -> TransV NPVP -> TransVwPP NP PP…
39 LING 180 Autumn 2007
Forward Pointer
It turns out that verb subcategorization facts will provide a key element for semantic analysis (determining who did what to who in an event).
40 LING 180 Autumn 2007
Movement
Core exampleMy travel agent booked the flight
[[My travel agent]NP [booked [the flight]NP]VP]S
i.e. “book” is a straightforward transitive verb. It expects a single NP arg within the VP as an argument, and a single NP arg as the subject.
41 LING 180 Autumn 2007
Movement
What about?Which flight do you want me to have the travel agent book?
The direct object argument to “book” isn’t appearing in the right place. It is in fact a long way from where its supposed to appear.And note that it’s separated from its verb by 2 other verbs.
42 LING 180 Autumn 2007
CFGs: a summary
CFGs appear to be just about what we need to account for a lot of basic syntactic structure in English.
But there are problemsThat can be dealt with adequately, although not elegantly, by staying within the CFG framework.
There are simpler, more elegant, solutions that take us out of the CFG framework (beyond its formal power).Syntactic theories: HPSG, LFG, CCG, Minimalism, etc.
43 LING 180 Autumn 2007
Other syntactic stuff
Grammatical relationsSubject
– I booked a flight to New York– The flight was booked by my agent
Object– I booked a flight to New York
Complement– I said that I wanted to leave
44 LING 180 Autumn 2007
Dependency parsing
Word to word links instead of constituencyBased on the European rather than American traditionsBut dates back to the GreeksThe original notions of Subject, Object and the progenitor of subcategorization (called ‘valence’) came out of Dependency theory.Dependency parsing is quite popular as a computational modelsince relationships between words are quite useful
45 LING 180 Autumn 2007
Dependency parsing
Bills on ports and immigration were submitted by Senator Brownback
NP
S
NP
NNP NNP
PP
IN
VP
VP
VBN
VBD
NNCCNNS
NPIN
NP PP
NNS
submitted
Bills were Brownback
Senator
nsubjpass auxpass agent
nnprep_on
ports
immigration
conj_and
Parse tree:
Nesting of multi-word constituents
Typed dep parse:
Grammatical relations between individual words
46 LING 180 Autumn 2007
Why are dependency parses useful?
Example: multi-document summarization
Need to identify sentences from different documents that each say roughly the same thing:
phrase structure trees of paraphrasing sentences which differ in word order can be significantly differentbut dependency representations will be very similar
47 LING 180 Autumn 2007
Parsing
Parsing: assigning correct trees to input stringsCorrect tree: a tree that covers all and only the elements of the input and has an S at the topFor now: enumerate all possible trees
A further task: disambiguation: means choosing the correct tree from among all the possible trees.
48 LING 180 Autumn 2007
Treebanks
Parsed corpora in the form of treesThe Penn Treebank
The Brown corpusThe WSJ corpus
Tgrep http://www.ldc.upenn.edu/ldc/online/treebank/
Tregexhttp://www-nlp.stanford.edu/nlp/javadoc/javanlp/
49 LING 180 Autumn 2007
Parsing involves search
As with everything of interest, parsing involves a search which involves the making of choicesWe’ll start with some basic (meaning bad) methods before moving on to the one or two that you need to know
50 LING 180 Autumn 2007
For Now
Assume…You have all the words already in some bufferThe input isn’t pos taggedWe won’t worry about morphological analysisAll the words are known
51 LING 180 Autumn 2007
Top-Down Parsing
Since we’re trying to find trees rooted with an S (Sentences) start with the rules that give us an S.Then work your way down from there to the words.
52 LING 180 Autumn 2007
Top Down Space
S
NP VP
S
VP
S
NP VPAux
NP VP
S
NP VP
S
NP VPAux
S
VPNPAux
S
NP
VP
S
VP
S
53 LING 180 Autumn 2007
Bottom-Up Parsing
Of course, we also want trees that cover the input words. So start with trees that link up with the words in the right way.Then work your way up from there.
54 LING 180 Autumn 2007
Bottom-Up Space
that flightBook
flightthatBook flightthatBook
flightthatBook flightthatBook
flightthatBook flightthatBook
55 LING 180 Autumn 2007
Control
Of course, in both cases we left out how to keep track of the search space and how to make choices
Which node to try to expand nextWhich grammar rule to use to expand a node
56 LING 180 Autumn 2007
Top-Down, Depth-First, Left-to-Right Search
57 LING 180 Autumn 2007
Example
58 LING 180 Autumn 2007
Example
[flight] [flight]
59 LING 180 Autumn 2007
Example
flight flight
60 LING 180 Autumn 2007
Top-Down and Bottom-Up
Top-downOnly searches for trees that can be answers (i.e. S’s)But also suggests trees that are not consistent with the words
Bottom-upOnly forms trees consistent with the wordsSuggest trees that make no sense globally
61 LING 180 Autumn 2007
So Combine Them
There are a million ways to combine top-down expectations with bottom-up data to get more efficient searchesMost use one kind as the control and the other as a filter
As in top-down parsing with bottom-up filtering
62 LING 180 Autumn 2007
Adding Bottom-Up Filtering
63 LING 180 Autumn 2007
3 problems with TDDFLtR Parser
Left-RecursionAmbiguityInefficient reparsing of subtrees
64 LING 180 Autumn 2007
Left-Recursion
What happens in the following situationS -> NP VPS -> Aux NP VPNP -> NP PPNP -> Det Nominal…With the sentence starting with
– Did the flight…
65 LING 180 Autumn 2007
Ambiguity
“One morning I shot an elephant in my pajamas. How he got into my pajamas I don’t know.”
Groucho Marx
66 LING 180 Autumn 2007
Lots of ambiguity
VP -> VP PPNP -> NP PP
Show me the meal on flight 286 from SF to Denver
14 parses!
67 LING 180 Autumn 2007
Lots of ambiguity
Church and Patil (1982)Number of parses for such sentences grows at rate of number of parenthesizations of arithmetic expressionsWhich grow with Catalan numbers
€
C(n) =1
n +1
2n
n
⎛
⎝ ⎜
⎞
⎠ ⎟
PPs Parses1 22 53 144 1325 4696 1430
68 LING 180 Autumn 2007
Avoiding Repeated Work
Parsing is hard, and slow. It’s wasteful to redo stuff over and over and over.
Consider an attempt to top-down parse the following as an NP:
A flight from Indianapolis to Houston on TWA
69 LING 180 Autumn 2007
flight
70 LING 180 Autumn 2007
flight
flight
71 LING 180 Autumn 2007
flight
flight
72 LING 180 Autumn 2007
flight
flight
73 LING 180 Autumn 2007
Grammars and Parsing
Context-Free Grammars and ConstituencySome common CFG phenomena for EnglishBaby parsers: Top-down and Bottom-up ParsingThursday: Real parsers:: Dynamic Programming parsing
CKY
Probabilistic parsingOptional section: the Earley algorithm