competitive grammar writing

1

Jason Eisner Noah A. SmithJohns Hopkins Carnegie Mellon

Competitive Grammar Writing

VP

Tree structure

N = Noun V = Verb P = Preposition D = Determiner R = Adverb

The girl with the newt pin hates peas quite violently N V N R R N D P N D

Tree structure

N = Noun V = Verb P = Preposition D = Determiner R = Adverb

NP = Noun phrase VP = Verb phrase PP = Prepositional

phrase S = Sentence

The girl with the newt pin hates peas quite violently

N

NP

NP

NP

PP

VP

VP

RP

S

N V N R R N D P N D

V N R R

VP RP

N N

N

D

NP

P N D

Generative Story: PCFG


NP

PP

NP

VP

S

Given a set of symbols (phrase types) Start with S at the root Each symbol randomly generates 2 child symbols, or 1 word Our job (maybe): Learn these probabilities

p(NP VP | S)

V N R R

VP RP

P

Context-Freeness of Model

with N N

N

D

NP

the newt pin hates peas quite violently N D

The girl

NP

PP

NP

VP

S

In a PCFG, the string generated under NP doesn’t depend on the context of the NP.

All NPs are interchangeable.

Inside vs. Outside


This NP is good because the “inside” string looks like a NP

S

NP

Inside vs. Outside


S

NP

This NP is good because the “inside” string looks like a NP and because the “outside” context looks like it expects a

NP. These work together in global inference, and could help

train each other during learning (cf. Cucerzan & Yarowsky 2002).

Inside vs. Outside


N

NP

N N D




Inside vs. Outside


NP

NP

NP

PP

VP

VP

RP

S

V N R R P N D




1. Welcome to the lab exercise! Please form teams of ~3 people … Programmers, get a linguist on your team

And vice-versa Undergrads, get a grad student on your team

And vice-versa

11

2. Okay, team, please log in

The 3 of you should use adjacent workstations Log in as individuals Your secret team directory:

cd …/03-turbulent-kiwi You can all edit files there Publicly readable & writeable No one else knows the secret directory name

12

3. Now write a grammar of English You have 2 hours.

13

3. Now write a grammar of English

What’s a grammar?

1 S1 NP VP .

1 VP VerbT NP

20 NP Det N’ 1 NP Proper

20 N’ Noun 1 N’ N’ PP

1 PP Prep NP

Here’s one to start with. You have 2 hours.

14

3. Now write a grammar of English

1 Noun castle 1 Noun king … 1 Proper Arthur 1 Proper Guinevere

… 1 Det a 1 Det every

… 1 VerbT covers 1 VerbT rides

… 1 Misc that 1 Misc bloodier 1 Misc does

…

Plus initial terminal rules.

1 S1 NP VP .

1 VP VerbT NP



1 PP Prep NP

Here’s one to start with.

15

3. Now write a grammar of English Here’s one to start with.

S1 1

NP VP .

1 S1 NP VP .

1 VP VerbT NP



1 PP Prep NP

16


S1

NP VP .

Det N’20/21

1/21

1 S1 NP VP .

1 VP VerbT NP



1 PP Prep NP

17


S1

NP VP .

Det N’

Nounevery

castle

drinks [[Arthur [across the [coconut in the castle]]] [above another chalice]]

1 S1 NP VP .

1 VP VerbT NP



1 PP Prep NP

18

4. Okay – go!

How will we be tested

on this?

19

4. Okay – go!

How will we be tested

on this?

5. Evaluation procedure We’ll sample 20 random sentences

from your PCFG. Human judges will vote on whether

each sentence is grammatical. By the way, y’all will be the judges

(double-blind).

You probably want to use the sampling script to keep testing your grammar along the way.

20

Ok, we’re done!All our sentences

are already grammatical.

We’ll sample 20 random sentences from your PCFG.

Human judges will vote on whether each sentence is grammatical.

You’re right: This only tests precision.

How about recall?

5. Evaluation procedure

1 S1 NP VP .

1 VP VerbT NP



1 PP Prep NP

21

questions, movement,(free) relatives, clefts,agreement, subcat frames, conjunctions, auxiliaries, gerunds, sentential subjects, appositives …

Development setYou might want your grammar to generate …

Arthur is the king . Arthur rides the horse near the castle . riding to Camelot is hard . do coconuts speak ? what does Arthur ride ? who does Arthur suggest she carry ? why does England have a king ? are they suggesting Arthur ride to Camelot ? five strangers are at the Round Table . Guinevere might have known . Guinevere should be riding with Patsy . it is Sir Lancelot who knows Zoot ! either Arthur knows or Patsy does . neither Sir Lancelot nor Guinevere will speak of it .

We provide a fileof 27 sample sentencesillustrating a range ofgrammatical phenomena

covered by initial grammar

22

questions, movement,(free) relatives, clefts,agreement, subcat frames, conjunctions, auxiliaries, gerunds, sentential subjects, appositives …

Development setYou might want your grammar to generate …

the Holy Grail was covered by a yellow fruit . Zoot might have been carried by a swallow . Arthur rode to Camelot and drank from his chalice . they migrate precisely because they know they will grow . do not speak ! Arthur will have been riding for eight nights . Arthur , sixty inches , is a tiny king . Arthur knows Patsy , the trusty servant . Arthur and Guinevere migrate frequently . he knows what they are covering with that story . Arthur suggested that the castle be carried . the king drank to the castle that was his home . when the king drinks , Patsy drinks .

23

What we could have done: Cross-entropy on a similar, held-out test set

5’. Evaluation of recall (= productivity!

!)

How should we parse sentences with OOV words?

No OOVs allowedin the test set.

Fixed vocabulary.

every coconut of his that the swallow dropped sounded like a horse .

24

What we could have done: Cross-entropy on a similar, held-out test set


!)

You should try togenerate sentences thatyour opponentscan’t parse.

What we’ll actually do, to heighten competition & creativity:Test set comes from the participants!

In Boggle, you getpoints for findingwords that youropponents don’t find.

Use the fixed vocabulary creatively.

25

1 Noun castle 1 Noun king … 1 Proper Arthur 1 Proper Guinevere

… 1 Det a 1 Det every

… 1 VerbT covers 1 VerbT rides

… 1 Misc that 1 Misc bloodier 1 Misc does

…

Initial terminal rules


The initial grammar sticksto 3rd-person singular transitive present-tense forms. All grammatical.

But we provide 183 Misc words (not accessible from initial grammar) that you’re free to work into your grammar …

26

1 Misc that 1 Misc bloodier 1 Misc does

…

Initial terminal rules


The initial grammar sticksto 3rd-person singular transitive present-tense forms. All grammatical.

But we provide 183 Misc words (not accessible from initial grammar) that you’re free to work into your grammar …

pronouns (various cases),plurals,

various verb forms,non-transitive verbs,

adjectives (various forms),adverbs & negation,

conjunctions & punctuation,wh-words,

…

27

In Boggle, you getpoints for findingwords that youropponents don’t find.


!)


What we could have done (good for your class?):Cross-entropy on a similar, held-out test set

What we actually did, to heighten competition & creativity:Test set comes from the participants!

28


!)


We’ll score your cross-entropywhen you try to parse the sentences

that the other teams generate.

(Only the ones judged grammatical.)

What we could have done (good for your class?):Cross-entropy on a similar, held-out test set


You probably want to use the parsing script to keep testing your grammar along the way.

29



!)What we could have done (you could too):

Cross-entropy on a similar, held-out test set

We’ll score your cross-entropywhen you try to parse the sentences

that the other teams generate.

(Only the ones judged grammatical.)

What if my grammar can’t parse

one of the testsentences?

0 probability??You get the

infinite penalty.

So don’t do that.

30

S2 S2 _Noun S2 _Misc _Noun Noun _Noun Noun _Noun _Noun Noun _Misc _Misc Misc _Misc Misc _Noun _Misc Misc _Misc

(etc.)

Use a backoff grammarInitial backoff grammar

: Bigram POS HMM

_Verb

Verb _Misc

Misc _Punc

Punc _Noun

Noun

S2

i.e., something that starts with a Verb

rides

‘s

!

swallow

i.e., something that starts with a Misc . . .

_Verb

Verb _Misc

Misc

31


(etc.)

S1 NP VP .

VP VerbT NP

NP Det N’ NP Proper

N’ Noun N’ N’ PP

PP Prep NP

Use a backoff grammarInit. linguistic grammar Initial backoff grammar

: Bigram POS HMM

32


(etc.)

S1 NP VP .

VP VerbT NP

NP Det N’ NP Proper

N’ Noun N’ N’ PP

PP Prep NP

Use a backoff grammar

Init. linguistic grammar Initial backoff grammar

: Bigram POS HMM

START S1 START S2

Initial master grammar

Choose these weights wisely!

Mixturemodel

33

6. Discussion What did you do? How? Was CFG expressive enough?

How would you improve the formalism? Would it work for other languages?

How should one pick the weights? And how could you build a better backoff grammar? Is grammaticality well-defined? How is it related to probability?

What if you had 36 person-months to do it right? What other tools or data do you need? What would the resulting grammar be good for? What evaluation metrics are most important?

features, gapping

34

7. Winners announced

35

7. Winners announced Of course, no one finishes their ambitious plans.

Alternative: Allow 2 weeks (see paper) …

Anyway, a lot of work!

Helps to favor

backoff grammar

yay

unreachable

36

What did past teams do? More fine-grained parts of speech do-support for questions & negation Movement using gapped categories X-bar categories (following the initial grammar) Singular/plural features Pronoun case Verb forms Verb subcategorization; selectional restrictions (“location”) Comparative vs. superlative adjectives Appositives (must avoid double comma) A bit of experimentation with weights One successful attempt to game scoring system (ok with

us!)

37

Good opening activity

Why do we recommend this lesson? Good opening activity Introduces many topics – touchstone for later teaching

Grammaticality Grammaticality judgments, formal grammars, parsers Specific linguistic phenomena Desperate need for features, morphology, gap-passing

Generative probability models: PCFGs and HMMs Backoff, inside probability, random sampling, … Recovering latent variables: Parse trees and POS taggings

Evaluation (sort of) Annotation, precision, recall, cross-entropy, … Manual parameter tuning

Why learning would be valuable, alongside expert knowledge

http://www.clsp.jhu.edu/grammar-writing

38

A final thought The CS curriculum starts with programming

Accessible and hands-on Necessary to motivate or understand much of CS

In CL, the equivalent is grammar writing It was the traditional (pre-statistical) introduction

Our contributions: competitive game, statistics, finite-state backoff, reusable instructional materials

Much of CL work still centers around grammar formalisms We design expressive formalisms for linguistic data Solve linguistic problems within these formalisms Enrich them with probabilities Process them with algorithms Learn them from data Connect them to other modules in the pipeline

Akin toprogramming

languages

competitive grammar writing

Documents