incremental semantic dependencyparsing · 2020. 12. 9. · introduction: motivation the person who...

Incremental Semantic Dependency Parsing

Marten van Schijndel William SchulerDepartment of LinguisticsThe Ohio State University

http://ling.osu.edu/∼vanschm

August 8, 2013

van Schijndel, Schuler Incremental Dependencies August 8, 2013 1 / 35

Introduction: Motivation

The person who officials say stole millions . . .




Goal: Incrementally obtain correct parse

• Filler-gap is hard for computers[Rimell et al., 2009, Nguyen et al., 2012]




Goal: Test human processing claims

• Filler-gap is hard for humans? [Chomsky and Miller, 1963]





• Filler-gap is hard for humans [Gibson, 2000, Chen et al., 2005]





• Filler-gap is hard for humans [Gibson, 2000, Chen et al., 2005]

• Embeddings speed processing [Pynte et al., 2008]

• Finishing embeddings = Fast[Wu et al., 2010, van Schijndel and Schuler, 2013]

Center embedding or filler-gap?


Overview

Contribution

Introduce an incremental semantic parser

• Fits reading times better than syntax parsing

• Replicate previous findings sans surface confounds


Overview

Contribution

Introduce an incremental semantic parser

• Fits reading times better than syntax parsing

• Replicate previous findings sans surface confounds

1 Generalized Categorial Grammar

2 Incremental Semantic Parser

3 Eye-tracking evaluation

4 Results


Generalized Categorial Grammar

Reannotate WSJ [Nguyen et al., 2012]

S

VP

NP

N

rights

D

the

V

bought

NP

N

studio

D

the

V

V-aN

N

N-aD

rights

D

the

V-aN-bN

bought

N

N-aD

studio

D

the


Generalized Categorial Grammar

We can also keep the WSJ traces around.

NP

RP

S

VP

T

ti

V

bought

NP

N

studio

D

the

R

that

NP

N

rights

D

the

N

V-rN

V-gN

tV-aN-gN

bought

N

N-aD

studio

D

the

N-rN

that

N

N-aD

rights

D

the


Interpretation: Reannotation Rules

g :d h: c-ad

(fc-ad g h): c

g :dψ h: c-ad

λk (fc-ad (g k) h): cψ

g :d h: c-adψ

λk (fc-ad g (h k)): cψ

g :dψ h: c-adψ

λk (fc-ad (g k) (h k)): cψ

g : c-bd h:d

(fc-bd g h): c

g : c-bdψ h:d

λk (fc-bd (g k) h): cψ

g : c-bd h:dψ

λk (fc-bd g (h k)): cψ

g : c-bdψ h:dψ

λk (fc-bd (g k) (h k)): cψ(Aa–h)

g : u-ad h:c

(fIM g h):c

g : u-adψ h:c

λk (fIM (g k) h):cψ

g : u-ad h:cψ

λk (fIM g (h k)):cψ

g : u-adψ h:cψ

λk (fIM (g k) (h k)):cψ

g :c h: u-ad

(fFM g h):c

g :cψ h: u-ad

λk (fFM (g k) h):cψ

g :c h: u-adψ

λk (fFM g (h k)):cψ

g :cψ h: u-adψ

λk (fFM (g k) (h k)):cψ(Ma–h)

g : c-ad

λk (fc-ad {k} g): c-gd

g : c-bd


g :c

λk (fIM {k} g):c-gd(Ga–c)

g :e h: c-gd

λi ∃j (g i) ∧ (h i j):e

g :d-re h: c-gd

λk j ∃i (g k i) ∧ (h i j): c-re

g :d-ie h: c-gd

λk j ∃i (g k i) ∧ (h i j): c-ie(Fa–c)

g :e h:c-rd

λi ∃j (g i) ∧ (h i j):e(R)



g :d h: c-ad

(fc-ad g h): c

g :dψ h: c-ad


g :d h: c-adψ


g :dψ h: c-adψ


g : c-bd h:d

(fc-bd g h): c

g : c-bdψ h:d


g : c-bd h:dψ


g : c-bdψ h:dψ


g : u-ad h:c

(fIM g h):c

g : u-adψ h:c


g : u-ad h:cψ


g : u-adψ h:cψ


g :c h: u-ad

(fFM g h):c

g :cψ h: u-ad


g :c h: u-adψ


g :cψ h: u-adψ


g : c-ad


g : c-bd


g :c


g :e h: c-gd

λi ∃j (g i) ∧ (h i j):e

g :d-re h: c-gd


g :d-ie h: c-gd


g :e h:c-rd

λi ∃j (g i) ∧ (h i j):e(R)


Interpretation

Connected Components

S

VP

NP

—–NP

ND

the

V

bought

NP

N

studio

D

the

S/NP

}

NP/N


Interpretation

Reannotated Connected Components

V

V-aN

N

—–N

N-aDD

the

V-aN-bN

bought

N

N-aD

studio

D

the

S/N

}

N/N-aD


Connected Component Parsing

NP

ND

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

NP/N



S

VPNP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S/VP



S

VP

NPV

bought

NP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S/NP



S

VP

NP

—–NP

ND

the

V

bought

NP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S/NP

NP/N



S

VP

NP

—–D

GNP

N

author

D

the

V

bought

NP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S/NP

D/G



S

VP

NP

ND

G

’s

NP

N

author

D

the

V

bought

NP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S/N



S

VP

NP

N

rights

D

G

’s

NP

N

author

D

the

V

bought

NP

N

studio

D

the

NP/NP

NP/NP

NP/NP

NP/NP

NP/NP

WorkingMemory:

S


Interpretation: Referent States

i5

i4 i6

i3i2 i7

i1

1 2

1 21

1

say

officials stole

who

person millions

the

0

0 0

0

0 0

0


Interpretation: Fa/LaFirst or Last element of a CC

∃i1j1.. iℓ jℓ ... ∧ (gℓ:c/d {jℓ} iℓ) xt

∃i1j1.. iℓ ... ∧ ((gℓf ):c iℓ)

xt 7→M f :d (–Fa)

∃i1j1.. iℓjℓ ... ∧ (gℓ:c/d {jℓ} iℓ) xt

∃i1j1.. iℓjℓ iℓ+1 ... ∧ (gℓ:c/d {jℓ} iℓ) ∧ (f :e iℓ+1)

xt 7→M f :e (+Fa)

∃i1j1.. iℓ−1jℓ−1 iℓ ... ∧ (gℓ:d iℓ)

∃i1j1.. iℓ jℓ ... ∧ ((fgℓ):c/e {jℓ} iℓ)

g :d h:e ⇒ (f g h):c

g :d h:e ⇒ λk(f (g k) h):c

g :d h:e ⇒ λk(f g (h k)):c

g :d h:e ⇒ λk(f (g k) (h k)):c

(–La)

∃i1j1.. iℓ−1jℓ−1iℓ ... ∧ (gℓ−1:a/c {jℓ−1} iℓ−1) ∧ (gℓ:d iℓ)

∃i1j1.. iℓ−1jℓ−1 ... ∧ (gℓ−1 ◦ (f gℓ):a/e {jℓ−1} iℓ−1)

g :d h:e ⇒ (f g h):c

g :d h:e ⇒ λk(f (g k) h):c

g :d h:e ⇒ λk(f g (h k)):c

g :d h:e ⇒ λk(f (g k) (h k)):c

(+La)


Interpretation


Interpretation: Fb/Lb

ψ ∈ {-r,-i}×C

∃i1j1.. in jn.. iℓjℓ ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ (gℓ:c/d {jℓ} iℓ) xt

∃i1j1.. in jn.. iℓ ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ ((gℓ(f ′{jn}f )):c iℓ)

xt 7→M λk(f′{k} f ):d (–Fb)

∃i1j1.. in jn.. iℓjℓ ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ (gℓ:c/d {jℓ} iℓ) xt

∃i1j1.. in jn.. iℓjℓ iℓ+1 ... ∧ (gn :y/zψ {jn} in) ∧ ... ∧ (gℓ:c/d {jℓ} iℓ) ∧ ((f ′{jn}f ):e iℓ+1)

xt 7→M λk(f′{k} f ):e (+Fb)

∃i1j1.. in jn.. iℓ−1jℓ−1iℓ ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ (gℓ:d iℓ)

∃i1j1.. in jn.. iℓjℓ ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ ((fgℓ) ◦ (f ′{jn}):cψ/e {jℓ} iℓ)

g :d h:e ⇒ λk (fg (f′{k} h)):cψ (–Lb)

∃i1j1.. in jn.. iℓ−1jℓ−1iℓ ... ∧ (gn :y/zψ {jn} in) ∧ ... ∧ (gℓ−1:a/cψ {jℓ−1} iℓ−1) ∧ (gℓ:d iℓ)

∃i1j1.. in jn.. iℓ−1jℓ−1 ... ∧ (gn:y/zψ {jn} in) ∧ ... ∧ (gℓ−1 ◦ (fgℓ) ◦ (f ′{jn}):a/e {jℓ−1} iℓ−1)

g :d h:e ⇒ λk(fg (f′{k} h)):cψ (+Lb)


Interpretation: Lc/N

∃i1j1.. iℓ−1jℓ−1iℓ ... ∧ (gℓ:d iℓ)

∃i1j1.. iℓ jℓ ... ∧ ((fgℓ) ◦ (λh k i (h k)):a/eψ {jℓ} iℓ)

g :d h:eψ ⇒ (fg h):c

(–Lc)

∃i1j1.. iℓ−1jℓ−1iℓ ... ∧ (gℓ−1:a/c {jℓ−1} iℓ−1) ∧ (gℓ:d iℓ)

∃i1j1.. iℓ−1jℓ−1 ... ∧ (gℓ−1 ◦ (fgℓ) ◦ (λh k i (h k)):a/eψ {jℓ−1} iℓ−1)

g :d h:eψ ⇒ (fg h):c

(+Lc)

∃i1j1.. iℓ jℓ ... ∧ (gℓ−1:c/dψ {jℓ−1} iℓ−1) ∧ (gℓ:dψ/e {jℓ} iℓ)

∃i1j1.. iℓ−1jℓ−1 ... ∧ (gℓ−1 ◦ (λh i∃j (h j)) ◦ gℓ:c/e {jℓ−1} iℓ−1)

(+N)

All of these rules may be made probabilistic


Interpretation


Evaluation: Syntactic vs Semantic

Syntactic Parser [van Schijndel et al., 2013]

• Only Fa/La

• Trained on WSJ 02-21

• Split-merged ×5 [Petrov et al., 2006]

Semantic Parser

• Trained on Reannotated WSJ 02-21

• Split-merged ×3



Test Corpus: Dundee

• Log-transformed go-past durations

• Omit:• first and last of each line (wrap-up)• < 5 times in WSJ (accuracy) [Fossum and Levy, 2012]• saccade length > 4 (track loss) [Demberg and Keller, 2008]


Eye Tracking

Go-past durations:

John went to the shop today

Cumulative factors are summed over the go-past regionNon-cumulative factors are based on the initial word in a region (shop)


Eye Tracking

Go-past durations:

John went to the shopto the shop today

X = Go-past region

Cumulative factors are summed over the go-past regionNon-cumulative factors are based on the initial word in a region (shop)



Fitting a linear mixed effects model (lmer in R)

Fixed Effects

• Word length

• Sentence position

• Prev, Next word fixated?

• Unigram and bigram probs

• Surprisal [Hale, 2001]

• Region length

• Cum. surprisal

• Cum. entropy reduction [Hale, 2003]

• Joint interactions

• Spillover predictors

By-subject random slopes

• Region length

• Prev word fixated?• Cumulative surprisal

Subject and Item random intercepts



Model log-likelihood AIC

syntactic -64175 128619semantic -64169 128609

Goodness-of-fitsRelative likelihood: 0.0009 (n = 151,331)


Evaluation: Semantic Factors

Factor coeff std. err. t-score p-value

F+L– (encoding) 0.014 0.005 2.665 0.02F–L+ (integration) -0.021 0.005 -4.109 0.001F–L+|N+ -0.021 0.005 -4.109 ?

F–L– – – – .50F+L+ – – – –

Significance of residualized factors on reading time.Positive t-score: inhibitionNegative t-score: facilitation



Corpus: Reannotated WSJ

• Remove sentences with modifier embeddings [Pynte et al., 2008]

For example:The CEO sold [[the shares] of the company]



Model coeff std err t-score

Canonical -0.040 0.010 -4.05Other -0.017 0.004 -4.20

Significance of residualized factors on reading time.Positive t-score: inhibitionNegative t-score: facilitation

To achieve convergence, residualization was used


Conclusion

Results

• Described incremental semantic dependency parser

• General metrics are not hurt by semantic calculation

• Semantic metrics predict reading times better than syntactic

• Replicated negative integration cost without FG confound

• Failed to find support for maintenance cost


Fin

Thanks to Elliot Schumacher (and viewers like you)!Questions?


Extras 1: Probabilistic Formulae

Pφℓ(‘–’ rF | 〈i , c〉〈j , d〉)

def∝ Eγ∗

ℓ(c

0→ d ...) ·

∑

x Pγ(d → x) · JrF = 〈i , ‘id’, j〉K (1a)

Pφℓ(‘+’ rF | 〈i , c〉〈j , d〉)

def∝ Eγ∗

ℓ(c

+→ d ...) ·

∑

x Pγ(d → x) · JrF = 〈‘–’, ‘–’, ‘–’〉K

(1b)

Pλℓ(‘+’ | 〈i , c〉〈j , d〉)def∝

∑

c′,e Eγ∗ℓ(c

0→ c′ ...) · PγB,ℓ (c

′ → d e) (2a)

Pλℓ(‘–’ | 〈i , c〉〈j , d〉)def∝

∑

c′,e Eγ∗ℓ(c

+→ c′ ...) · PγA,ℓ (c

′ → d e) (2b)

Pνℓ(‘+’ | 〈i , c〉〈j , d〉〈j′, d ′〉〈k, e〉)

def= Jc, d, d ′∈C×{-g}×C ∧ e /∈C×{-g}×CK (3a)

Pνℓ(‘–’ | 〈i , c〉〈j , d〉〈j′, d ′〉〈k, e〉)

def= Jc, d, d ′ /∈C×{-g}×C ∨ e∈C×{-g}×CK (3b)



Pαℓ (〈i′, c′〉 rA | l 〈i , c〉〈j , d〉)

def∝

{

if l=‘+’ :∑

e Eγ∗ℓ(c

+→ c′ ...) · PγA,ℓ (c

′ → d e)

if l=‘–’ : Jc′ = dK

·

{

if l=‘+’ ∨ [d ..⇒ c′] ∈ Ae–h,Me–h : Ji ′ = jKif l=‘–’ ∧ [d ..⇒ c′] ∈ Aa–d,Ma–d : Ji ′ = iZ+1K

·

{

if l=‘+’ : JrA = 〈i ′, ‘id’, j〉Kif l=‘–’ : JrA = 〈‘–’, ‘–’, ‘–’〉K

(1)

Pβs,ℓ(〈k, e〉 rB | 〈i , c〉〈j , d〉)

def∝ Pγs,ℓ(c → d e) ·

{

if [d e ⇒ c] ∈ Aa–d,Ma–d : Jk = iKif [d e ⇒ c] ∈ Ae–h,Me–h : Jk = iZ+1K

·

if [d e ⇒ c] ∈ Aa–d,Me–h : JrB = 〈k,V (e), j〉Kif [d e ⇒ c] ∈ Ae–h,Ma–d : JrB = 〈j ,V (d), k〉Kif [d e ⇒ c] ∈ Fa–c : JrB = 〈‘–’, ‘–’, ‘–’〉K

(2)

Pκℓ (rK | 〈i , c〉〈i ′, c′〉〈j , d〉〈k, e〉)

def=

if c∈C×{-g}×C ∧ ∃d′ d′ e ⇒ c′ ∧ [d ⇒ d ′] ∈ Ga–b : JrK = 〈i ′,V (d), i〉K

if c∈C×{-g}×C ∧ ∃e′ d e′ ⇒ c′ ∧ [e ⇒ e′] ∈ Ga–b : JrK = 〈i ′,V (e), i〉K

if c∈C×{-g}×C ∧ ∃d′ d′ e ⇒ c′ ∧ [d ⇒ d ′] ∈ Gc : JrK = 〈i , 1, i ′〉K

if c∈C×{-g}×C ∧ ∃e′ d e′ ⇒ c′ ∧ [e ⇒ e′] ∈ Gc : JrK = 〈i , 1, i ′〉K

otherwise : JrK = 〈‘–’, ‘–’, ‘–’〉K

(3)



Pσ(q1..Nt | q

1..Nt−1 xt−1)

def= Pφℓ

(‘–’ | bℓt−1 xt−1) · Pσ′

ℓ(q

1..Nt | q

1..Nt−1 a

ℓt−1)

+ Pφℓ(‘+’ | b

ℓt−1 xt−1) · Pσ′

ℓ+1(q

1..Nt | q

1..Nt−1 xt−1); ℓ

def= max{ℓ

′|q

ℓ′

t−1 6=‘–’} (4)

Pσ′ℓ(q

1..Nt | q

1..Nt−1 a

′)def= Pλℓ

(‘+’ | bℓ−1t−1 a

′) · Ja=a

ℓ−1t−1 K · PβB,ℓ−1

(b | bℓ−1t−1 a

′) · P

σ′′ℓ−1

(q1..Nt | q

1..Nt−1 a b a

′)

+ Pλℓ(‘–’ | b

ℓ−1t−1 a

′) · Pαℓ (a | b

ℓ−1t−1 a

′) · PβA,ℓ

(b | aℓt a

′) · P

σ′′ℓ(q

1..Nt | q

1..Nt−1 a b a

′) (5)

Pσ′′ℓ(q

1..Nt | q

1..Nt−1 a b a

′)def= Pνℓ−1 (‘+’ | a

ℓ−1t−1 b

ℓ−1t−1 a

ℓt−1 b) · Pκℓ−1 (r

K| b

nt−1 b

ℓt−1 a

′b) · P

σ′′′ℓ−1

(q1..Nt | q

1..Nt−1 a b)

+ Pνℓ−1 (‘–’ | aℓ−1t−1 b

ℓ−1t−1 a

ℓt−1 b) · Pκℓ−1 (r

K| b

nt−1 b

ℓt−1 a

′b) · P

σ′′′ℓ

(q1..Nt | q

1..Nt−1 a b) (6)

Pσ′′′ℓ

(q1..Nt | q

1..Nt−1 a b)

def= Jq

1..ℓ−1t =q

1..ℓ−1t−1 K · Ja

ℓt =aK · Jb

ℓt =bK · Jq

ℓ+1..Nt =‘–’K (7)


Bibliography I

Chen, E., Gibson, E., and Wolf, F. (2005).Online syntactic storage costs in sentence comprehension.Journal of Memory and Language, 52(1):144–169.

Chomsky, N. and Miller, G. A. (1963).Introduction to the formal analysis of natural languages.In Handbook of Mathematical Psychology, pages 269–321. Wiley, NewYork, NY.

Demberg, V. and Keller, F. (2008).Data from eye-tracking corpora as evidence for theories of syntacticprocessing complexity.Cognition, 109(2):193–210.


Bibliography II

Fossum, V. and Levy, R. (2012).Sequential vs. hierarchical syntactic models of human incrementalsentence processing.In Proceedings of CMCL 2012. Association for ComputationalLinguistics.

Gibson, E. (2000).The dependency locality theory: A distance-based theory of linguisticcomplexity.In Image, language, brain: Papers from the first mind articulationproject symposium, pages 95–126, Cambridge, MA. MIT Press.

Hale, J. (2001).A probabilistic earley parser as a psycholinguistic model.In Proceedings of the second meeting of the North American chapter ofthe Association for Computational Linguistics, pages 159–166,Pittsburgh, PA.


Bibliography III

Hale, J. (2003).Grammar, Uncertainty and Sentence Processing.PhD thesis, Cognitive Science, The Johns Hopkins University.

Nguyen, L., van Schijndel, M., and Schuler, W. (2012).Accurate unbounded dependency recovery using generalized categorialgrammars.In Proceedings of the 24th International Conference on ComputationalLinguistics (COLING ’12), Mumbai, India.

Petrov, S., Barrett, L., Thibaux, R., and Klein, D. (2006).Learning accurate, compact, and interpretable tree annotation.In Proceedings of the 44th Annual Meeting of the Association forComputational Linguistics (COLING/ACL’06).


Bibliography IV

Pynte, J., New, B., and Kennedy, A. (2008).On-line contextual influences during reading normal text: Amultiple-regression analysis.Vision research, 48(21):2172–2183.

Rimell, L., Clark, S., and Steedman, M. (2009).Unbounded dependency recovery for parser evaluation.In Proceedings of EMNLP 2009, volume 2, pages 813–821.

van Schijndel, M., Exley, A., and Schuler, W. (2013).A model of language processing as hierarchic sequential prediction.Topics in Cognitive Science.

van Schijndel, M. and Schuler, W. (2013).An analysis of frequency- and recency-based processing costs.In Proceedings of NAACL-HLT 2013. Association for ComputationalLinguistics.


Bibliography V

Wu, S., Bachrach, A., Cardenas, C., and Schuler, W. (2010).Complexity metrics in an incremental right-corner parser.In Proceedings of the 48th Annual Meeting of the Association forComputational Linguistics (ACL’10), pages 1189–1198.


incremental semantic dependencyparsing · 2020. 12. 9. · introduction: motivation the person who...

Documents