dependency parsing prashanth mannem prashanth@research.iiit.ac.in

Dependency Parsing

Prashanth Mannemprashanth@research.iiit.ac.in

Outline

April 20, 2023Dependency Parsing (P. Mannem)2

Introduction Phrase Structure Grammar Dependency Grammar Comparison

Dependency Parsing Formal definition

Parsing Algorithms Introduction Dynamic programming Constraint satisfaction Deterministic search

Introduction

The syntactic parsing of a sentence consists of finding the correct syntactic structure of that sentence in a given formalism/grammar.

Dependency Grammar (DG) and Phrase Structure Grammar (PSG) are two such formalisms.

Phrase Structure Grammar (PSG)

Breaks sentence into constituents (phrases) Which are then broken into smaller constituents

Describes phrase structure, clause structure

E.g.. NP, PP, VP etc..

Structures often recursive The clever tall blue-eyed old … man

Draw the phrase structure tree

Red figures on the screens indicated falling stocks

Phrase Structure Tree

Dependency Grammar

Syntactic structure consists of lexical items, linked by binary asymmetric relations called dependencies

Interested in grammatical relations between individual words (governing & dependent words)

Does not propose a recursive structure Rather a network of relations

These relations can also have labels

Draw the dependency tree

Red figures on the screens indicated falling stocks

Dependency Tree

Dependency Tree Example

Phrasal nodes are missing in the dependency structure when compared to constituency structure.

Dependency Tree with Labels

Comparison

Dependency structures explicitly represent Head-dependent relations (directed arcs) Functional categories (arc labels) Possibly some structural categories (parts-of-

speech)

Phrase structure explicitly represent Phrases (non-terminal nodes) Structural categories (non-terminal labels) Possibly some functional categories (grammatical

functions)

Learning DG over PSG

Dependency Parsing is more straightforward Parsing can be reduced to labeling each token w i with

Direct encoding of predicate-argument structure Fragments are directly interpretable

Dependency structure independent of word order Suitable for free word order languages (like Indian

languages)

Outline

Introduction Phrase Structure Grammar Dependency Grammar Comparison

Dependency Tree

Formal definition An input word sequence w1…wn

Dependency graph D = (W,E) where W is the set of nodes i.e. word tokens in the input seq. E is the set of unlabeled tree edges (wi, wj) (wi, wj є W).

(wi, wj) indicates an edge from wi (parent) to wj (child).

Task of mapping an input string to a dependency graph satisfying certain conditions is called dependency parsing

Well-formedness

A dependency graph is well-formed iff

Single head: Each word has only one head.

Acyclic: The graph should be acyclic.

Connected: The graph should be a single tree with all the words in the sentence.

Projective: If word A depends on word B, then all words between A and B are also subordinate to B (i.e. dominated by B).

Non-projective dependency tree

Ram saw a dog yesterday which was a Yorkshire Terrier

* Crossing lines

English has very few non-projective cases.

Outline

Introduction Phrase Structure Grammar Dependency Grammar Comparison and Conversion

Dependency Parsing

Dependency based parsers can be broadly categorized into Grammar driven approaches

Parsing done using grammars. Data driven approaches

Parsing by training on annotated/un-annotated data.

Dependency Parsing

Dependency based parsers can be broadly categorized into Grammar driven approaches

Parsing done using grammars. Data driven approaches

Parsing by training on annotated/un-annotated data.

These approaches are not mutually exclusive

Covington’s Incremental Algorithm

Incremental parsing in O(n2) time by trying to link each new word to each preceding one [Covington 2001]:

PARSE(x = (w1, . . . ,wn))1. for i = 1 up to n2. for j = i − 1 down to 13. LINK(wi , wj )

Covington’s Incremental Algorithm

Incremental parsing in O(n2) time by trying to link each new word to each preceding one [Covington 2001]:

PARSE(x = (w1, . . . ,wn))1. for i = 1 up to n2. for j = i − 1 down to 13. LINK(wi , wj )

Different conditions, such as Single-Head and Projectivity, can be incorporated into the LINK operation.

Parsing Methods

Three main traditions Dynamic programming

CYK, Eisner, McDonald

Constraint satisfaction Maruyama, Foth et al., Duchier

Deterministic search Covington, Yamada and Matsumuto, Nivre

Dynamic Programming

Basic Idea: Treat dependencies as constituents. Use, e.g. , CYK parser (with minor modifications)

April 20, 2023Dependency Parsing (P. Mannem) 24

Dependency Chart Parsing

Grammar is regarded as context-free, in which each node is lexicalized

Chart entries are subtrees, i.e., words with all their left and right dependents

Problem: Different entries for different subtrees spanning a sequence of words with different heads

Time requirement: O(n5)

Generic Chart Parsing

for each of the O(n2) substrings,for each of O(n) ways of splitting it,

for each of S analyses of first half

for each of S analyses of second half,for each of c ways of combining them: combine, & add result to chart if best

[cap spending] + [at $300 million] = [[cap spending] [at $300 million]]

S analyses S analyses cS2 analysesof which we keep S

O(n3 S2 c)

Slide from [Eisner, 1997]

Headed constituents ...

... have too many signatures.

How bad is (n3S2c)?

For unheaded constituents, S is constant: NP, VP ...

(similarly for dotted trees). So (n3).

But when different heads different signatures,the average substring has (n) possible heads

and S=(n) possible signatures. So (n5).

Slide from [Eisner, 1997]

Dynamic Programming Approaches

Original version [Hays 1964] (grammar driven) Link grammar [Sleator and Temperley 1991]

(grammar driven) Bilexical grammar [Eisner 1996] (data driven) Maximum spanning tree [McDonald 2006] (data

driven)

Eisner 1996

Two novel aspects: Modified parsing algorithm Probabilistic dependency parsing

Time requirement: O(n3) Modification: Instead of storing subtrees, store

spans Span: Substring such that no interior word links

to any word outside the span. Idea: In a span, only the boundary words are

active, i.e. still need a head or a child One or both of the boundary words can be

active

Example

Red figures on the screen indicated falling stocks_ROOT_

Example

Spans:

indicated falling stocksRed figures

Assembly of correct parse

Red figures

Start by combining adjacent words to minimal spans

figures on on the

Combine spans which overlap in one word; this word must be governed by a word in the left or right span.

on the the screen+ → on the screen

figures on + →on the screen figures on the screen

Invalid span

Red figures on the screen

+ → indicated falling stocksfalling stocksindicated falling

Eisner’s Model

Recursive Generation Each word generates its actual dependents Two Markov chains:

Left dependents Right dependents

Eisner’s Model

where tw(i) is ith tagged word

lc(i) & rc(i) are the left and right children of ith word

wherelcj(i) is the jth left child of the ith wordt(lcj-1(i)) is the tag of the preceding left child

McDonald’s Maximum Spanning Trees

Score of a dependency tree = sum of scores of dependencies

Scores are independent of other dependencies If scores are available, parsing can be

formulated as maximum spanning tree problem Two cases:

Projective: Use Eisner’s parsing algorithm. Non-projective: Use Chu-Liu-Edmonds algorithm [Chu

and Liu 1965, Edmonds 1967] Uses online learning for determining weight

vector w

Parsing Methods

Deterministic parsing Covington, Yamada and Matsumuto, Nivre

Constraint Satisfaction

Uses Constraint Dependency Grammar Grammar consists of a set of boolean

constraints, i.e. logical formulas that describe well-formed trees

A constraint is a logical formula with variables that range over a set of predefined values

Parsing is defined as a constraint satisfaction problem

Constraint satisfaction removes values that contradict constraints

Constraint Satisfaction

Parsing is an eliminative process rather than a constructive one such as in CFG parsing

Constraint satisfaction in general is NP complete

Parser design must ensure practical efficiency Different approaches

Constraint propagation techniques which ensure local consistency [Maruyama 1990]

Weighted CDG [Foth et al. 2000, Menzel and Schroder 1998]

Maruyama’s Constraint Propagation

Three steps:1) Form initial constraint network using a “core”

grammar2) Remove local inconsistencies3) If ambiguity remains, add new constraints and

repeat step 2

Weighted Constraint Parsing

Robust parser, which uses soft constraints Each constraint is assigned a weight between

0.0 and 1.0 Weight 0.0: hard constraint, can only be

violated when no other parse is possible Constraints assigned manually (or estimated

from treebank) Efficiency: uses a heuristic transformation-

based constraint resolution method

Transformation-Based Constraint Resolution

Heuristic search Very efficient Idea: first construct arbitrary dependency

structure, then try to correct errors Error correction by transformations Selection of transformations based on

constraints that cause conflicts Anytime property: parser maintains a

complete analysis at anytime → can be stopped at any time and return a complete analysis

Parsing Methods

Deterministic parsing Covington, Yamada and Matsumuto, Nivre

Deterministic Parsing

Basic idea: Derive a single syntactic representation

(dependency graph) through a deterministic sequence of elementary parsing actions

Sometimes combined with backtracking or repair

Motivation: Psycholinguistic modeling Efficiency Simplicity

Shift-Reduce Type Algorithms

Data structures: Stack [. . . ,wi ]S of partially processed tokens Queue [wj , . . .]Q of remaining input tokens

Parsing actions built from atomic actions: Adding arcs (wi → wj , wi ← wj ) Stack and queue operations

Left-to-right parsing Restricted to projective dependency graphs

Yamada’s Algorithm

Three parsing actions: Shift [. . .]S [wi , . . .]Q

[. . . , wi ]S [. . .]Q Left [. . . , wi , wj ]S [. . .]Q

[. . . , wi ]S [. . .]Q wi → wj

Right [. . . , wi , wj ]S [. . .]Q

[. . . , wj ]S [. . .]Q wi ← wj

Multiple passes over the input give time complexity O(n2)

Yamada and Matsumoto

Parsing in several rounds: deterministic bottom-up O(n2)

Looks at pairs of words 3 actions: shift, left, right

Shift: shifts focus to next word pair

Yamada and Matsumoto

Right: decides that the right word depends on the left word

Left: decides that the left word depends on the right one

Parsing Algorithm

Go through each pair of words Decide which action to take

If a relation was detected in a pass, do another pass

E.g. the little girl First pass: relation between little and girl Second pass: relation between the and girl

Decision on action depends on word pair and context

Nivre’s Algorithm

Four parsing actions:Shift [. . .]S [wi , . . .]Q

[. . . , wi ]S [. . .]Q

Reduce [. . . , wi ]S [. . .]Q Эwk : wk → wi

[. . .]S [. . .]QLeft-Arc [. . . , wi ]S [wj , . . .]Q ¬Эwk : wk → wi

[. . .]S [wj , . . .]Q wi ← wj

Right-Arc [. . . ,wi ]S [wj , . . .]Q ¬Эwk : wk → wj

[. . . , wi , wj ]S [. . .]Q wi → wj

Nivre’s Algorithm

Characteristics: Arc-eager processing of right-dependents Single pass over the input gives time worst case

complexity O(2n)

Example

Red figures on the screen indicated falling stocks_ROOT_S Q

Example

Red figures on the screen indicated falling stocks_ROOT_S Q

Example

figures on the screen indicated falling stocks_ROOT_S Q

Left-arc

Example

Right-arc

Example

figures on the

screen indicated falling stocks_ROOT_S Q

Left-arc

Example

figures on the

Right-arc

Example

figures on the

Reduce

Example

figures on

Reduce

Example

figures

Left-arc

Example

figures

Right-arc

Example

figures

Example

figures

screen indicated falling

stocks_ROOT_S Q

Left-arc

Example

figures

stocks_ROOT_S Q

Right-arc

Example

figures

stocks_ROOT_S Q

Reduce

Example

figures

screen indicated

falling

stocks_ROOT_S Q

Reduce

Classifier-Based Parsing

Data-driven deterministic parsing: Deterministic parsing requires an oracle. An oracle can be approximated by a classifier. A classifier can be trained using treebank data.

Learning algorithms: Support vector machines (SVM) [Kudo and Matsumoto

2002, Yamada and Matsumoto 2003,Isozaki et al. 2004, Cheng et al. 2004, Nivre et al. 2006]

Memory-based learning (MBL) [Nivre et al. 2004, Nivre and Scholz 2004]

Maximum entropy modeling (MaxEnt) [Cheng et al. 2005]

Feature Models

Learning problem: Approximate a function from parser states,

represented by feature vectors to parser actions, given a training set of gold standard derivations.

Typical features: Tokens and POS tags of :

Target words Linear context (neighbors in S and Q) Structural context (parents, children, siblings in G)

Can not be used in dynamic programming algorithms.

Dependency Parsers for download

MST parser by Ryan McDonald Malt parser by Joakim Nivre Stanford parser

Summary

Provided an intro to dependency parsing and various dependency parsing algorithms

Read up Nivre’s and McDonald’s tutorial on dependency parsing at ESSLLI’ 07

dependency parsing prashanth mannem prashanth@research.iiit.ac.in

dependency structures

labels dependency

syntactic parsing

dependency parsingprashanth

dependency treered figures

speechphrase structure

constituency structure

mannemphrase structure

Documents

dr. m. s. prashanth -...

prashanth raman - arxiv · 2019-06-10 · the positive...

yashaswi verma ankush gupta prashanth mannem c. v....

methods of irrigation angrau prashanth

presented by manoj prashanth yuvaraj sanjana...

multi dimensional scaling -angrau - prashanth

mr prashanth nayak - commonwealthnurses.org · mr prashanth...

prashanth ramasamy - resume (21 nov 2016)

prashanth portfolio

c - language by prashanth

proceedings of icon10 nlp tools contest:...

power transmission prashanth

madokaram prashanth iyengar - brahmavidya · 1 capsulized...

organic cotton ppt prashanth

prashanth 0385 seasonality stock market

prashanth mannar

mds prashanth

vè;k; 6 - prashanth...

ieg201602 share prashanth kuttuva_derivatives_analytics

gsoc...