- lexical syntax for statistical machine translation · hany hassan introduction syntax for...

Hany Hassan

Introduction

Syntax for Phrase-basedSMT

SupertaggedPhrase-based SMT

From Supertagged toDependency-basedLanguage Models

IncrementalDependency-basedLanguage Model (IDLM)

DTM2

Dependency-based SMT

Future Work

Conclusion andDiscussion

Lexical Syntax forStatistical Machine Translation

Hany Hassan

DCU & IBM

In collaboration with:Andy Way and Khalil Sima’an

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction

Syntax for Phrase-based SMT

Supertagged Phrase-based SMT

From Supertagged to Dependency-based Language Models

Incremental Dependency-based Language Model (IDLM)

DTM2


Future Work

Conclusion and Discussion

Hany Hassan

Introduction





DTM2


Future Work


Can linguistic syntax improve PBSMT?

(Koehn et al 2003) tried to impose syntactic constituentson phrase extraction

Hierarchical Phrase structure (Chiang 2005)

I Allows for hierarchical phrasesI Handles a range of reordering problemsI The syntax induced is not linguistically motivated.

Syntactified target phrases (Marcu et. al. 2006)

I Induces millions of xRs rules from parallel corpusI Mismatch between constituent (xRs) and phraseI Subtrees for phrases: leads to spurious ambiguity in phrase table

Do subtrees/constituents fit well with phrases?

Hany Hassan

Introduction





DTM2


Future Work



S

NP

VP

V NP

NP NP

The president meets Saudi economic officials

Hany Hassan

Introduction





DTM2


Future Work


Spurious Ambiguity:

meetsThe president economical officials in Riad next weekSaudi

NP V NP NP PP NP

VP

NP

S

obj

Hany Hassan

Introduction





DTM2


Future Work



Why subtress do not match SMT phrases?

I Syntactic constituents mismatch phrase conceptI Which level of tree structure should be incorporated ?I This leads to spurious ambiguity

Can linguistic syntax improve PBSMT?

Trees/constituents do NOT fit well with phrases

What syntax does fit then ?

Hany Hassan

Introduction





DTM2


Future Work


Lexical Syntax (Supertags) Matches Phrases

Hany Hassan

Introduction





DTM2


Future Work


Lexical Syntax (Supertags)

Linguistics offers lexical-syntax (Supertags):

I Lexicalized Tree Adjoining Grammar (LTAG) : (Joshi &Schabes, 1992) & (Srinivas & Joshi, 1999)

I Combinatory Categorical Grammar (CCG) (Steedman,2000)

Rich lexical categories

I Localizing syntactic dependenciesI Representing predicate argument constraints on the word levelI Markovian language model on the sequence produce almost

parsingI Handful of Combination Operators are used to construct

dependency tree

Hany Hassan

Introduction





DTM2


Future Work


The purchase price includes taxes

Hany Hassan

Introduction





DTM2


Future Work



NP/NP (NP) NP (S\NP)/NP NP

Hany Hassan

Introduction





DTM2


Future Work



NP/NP (NP) NP (S\NP)/NP NP> FA > FA

NP S\NP

Hany Hassan

Introduction





DTM2


Future Work




NP S\NP> FA

NP

Hany Hassan

Introduction





DTM2


Future Work




NP S\NP> FA

NP< BA

S

Hany Hassan

Introduction





DTM2


Future Work


Lexical Syntax for SMT

Two levels of support:

I Supertagged TM & LMI Fully incremental parsing

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


Syntactically Lexicalized Phrase Based SMT

Can linguistic syntax improve the output of Phrase-basedSMT systems?

I Which syntax could fit with PBSMT ?I Lexical Syntax : LTAG/CCG SupertagsI Supertags improve the performance of state-of-the-art PBSMT

system on large data sets:I Arabic-to-English NIST’05I German-to-English shared task 07

Hany Hassan

Introduction





DTM2


Future Work


Baseline PBSMT vs Supertags PBSMT

Baseline PBSMTI Many candidate phrasesI Not constrained enoughI N-gram LM can not choose best candidates

Supertags PBSMT

I Many candidate phrasesI Syntactically Constrained PhrasesI Further sophisticated techniques could choose best candidates

Hany Hassan

Introduction





DTM2


Future Work


Supertagged PBSMT Model

Supertags PBSMT: Noisy Channel Model

argmaxt

∑

ST

P (s | t, ST )PST (t, ST ) ≈

arg maxt,ST

P (s | t, ST )PST (t, ST ) ≈

arg maxσ,t,ST

TM w.sup.tags︷︸︸︷

P (φs | φt,ST )

distortion︷︸︸︷

P (Os | Ot)λo

LM w.sup.tags︷︸︸︷

PST (t, ST )

Hany Hassan

Introduction





DTM2


Future Work



Supertags PBSMT: Log-Linear Model

t∗ = arg maxt,σ,ST

∏

f∈F

Hf (s, t, σ, ST )λf

I Log-linear model representationI Added features for supertags

Hany Hassan

Introduction





DTM2


Future Work



Supertags Language Model

P (t, ST ) =

n∏

i=1

P (sti|sti−1

i−4)P (ti|sti)

I Log-linear Language Model for SupertagsI 5-gram Markov Language Model over supertags sequence

Hany Hassan

Introduction





DTM2


Future Work



Supertagged Phrase Translation Probability

P (φs | φt,ST ) ≈∏

〈si,tiSTi〉∈(φs×φt,ST )

P (si | ti, STi)

P (φt,ST | φs) ≈∏

〈si,tiSTi〉∈(φs×φt,ST )

P (ti, STi | si)

Here is a supertagged phrase pair consisting

I Phrase translation probability and its reverseI Generate target words and supertags simultaneously

Hany Hassan

Introduction





DTM2


Future Work


LMs with Global Grammaticality Measures

I Log-linear featureI Smoothing factor for supertags LMI Number of operator violations

John bought quickly shares

NNP_NN VBD_(S[dcl]\NP)/NP RB|(S\NP)\(S\NP) NNS_N

2 Violations

Hany Hassan

Introduction





DTM2


Future Work


LMs with Global Grammaticality Measures

He believes in what he saidNP (S[dcl]\NP)/S[dcl] PP/NP NP/(S/NP) NP (S\NP)/NP

The supertag of “believes” (in boldface) demands directly to its right(for “in”) an “S[dcl]” (Forward Application); however, it finds a “(inPP/NP)” instead. This counts as a single violation V = 1 .Note that the supertag that fits best in the given sequence for“believes” is “(S\NP)/PP”.

Hany Hassan

Introduction





DTM2


Future Work


Experimental Setup

Two Language Pairs:

I Arabic to English NIST 05I German to English Shared Task 07

Supertaggers:

I LTAG supertaggers (XTAG and Bangalore’s Maxent tagger)I CCG supertagger (C&C tools)

Supertags:

I N-gram Language model on supertags sequence for LTAG &CCG

I Grammatical validation for CCG operators

Hany Hassan

Introduction





DTM2


Future Work


Scalability: Larger Training Corpora

Performance on large training data

System BLEU ScoreBase-LARGE 0.4418LTAG-LARGE 0.4600CCG-LARGE 0.4609

Hany Hassan

Introduction





DTM2


Future Work


LM with CCG Grammaticality

Adding a grammaticality factor

System BLEU ScoreBase-LARGE 0.4418CCG-LARGE 0.4609

CCG-LARGE-GRAM 0.4688

Hany Hassan

Introduction





DTM2


Future Work


German to English Results

System BLEU ScoreBase-Line 0.2704Supertags 0.2755

Supertags no Brevity 0.2947

Hany Hassan

Introduction





DTM2


Future Work


Results Analysis

Table: How CCG improves over baseline

N=50 test sentencesReason # %Inserting verb omitted by baseline 11 %22Better reordering 11 %22Better word/phrase selection 5 %10Other reasons 23 %46

Hany Hassan

Introduction





DTM2


Future Work


Arabic to English Examples

Reference: Annan opened an internal investigation in February butcancelled it in March in preparation for a broader, independentinvestigation.Baseline: Annan was to internally in February but abolished in Marchas a prelude to broader and independent .Supertags: Annan conducted an internal inquiry in February butabolished in March in preparation for broader and independent .

Hany Hassan

Introduction





DTM2


Future Work


Arabic to English Examples

Refrence: Rabat 1-14 (AFP) - A sharp debate is raging in Moroccoon the freedom of the press with regard to matters connec tedpersonally to King Mohamed VI following the publication of articlescriticizing the Moroccan monarch’s income and activit ies.Baseline: Rabat 14-1 ( afp ) - was a sharp controversy in morocco onpress freedom in terms of topics affecting king Mohamed VI himselfafter publishing articles critical of the revenues of the moroccanSupertags: Rabat 14-1 ( afp ) - a sharp controversy in Morocco onpress freedom in respect of topics affecting king Mohamed VIpersonally after the publication of articles criticizing the moroccanmonarch revenues.

Hany Hassan

Introduction





DTM2


Future Work


German to English Examples

Source: Ich habe nicht fr den Bericht Mann gestimmt, denn bei allemtatschlich notwendigen Streben nach Gleichbehandlung in Beschftigung undBeruf braucht deswegen noch nicht im bereifer soweit gegangen zu werden,dassder Schutz der Freiheiten und die Achtung des Rechtsstaates dabeivlligin Vergessenheit geraten.Reference: I have not voted for the Mann report because, while it is indeednecessary to seek equal treatment for people in employment and occupation,it is also necessary to refrain from pushing zeal to the point of abandoning allprotection of freedoms and all respect for the rule of law.Baseline: I have voted in favour of the report because , in particular , man isactually needed quest for equal treatment in employment and occupation istherefore not yet in excess of zeal went so far as to say , the protection offreedoms and respect for the rule of law is completely forgotten .Supertags: I have not voted for the Mann report because , in fact , with all thenecessary search for equal treatment in employment and occupation istherefore not yet gone so far in excess of zeal , that the protection offreedoms and respect for the rule of law is being completely forgotten .

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


From Supertagged to Dependency-basedLanguage Models

Almost prsing for MT:

I A n-gram language model over the sequence of supertags (‘almost parsing’).

I ‘almost parsing’ for monolingual parsingI ‘almost parsing’ for bilingual parsing

Hany Hassan

Introduction





DTM2


Future Work


What is the parsing mechanism we need for SMT?

I Suppoet long-range dependenciesI Distinguish between different translation candidates based on

their role in constructing the parse structureI Satisfy the syntactic dependenciesI Work in an incremental manner similar to SMT decodersI Be computationally efficient to be integrated into SMT

Hany Hassan

Introduction





DTM2


Future Work


What is the parsing mechanism we need for SMT?

Our proposed IDLM differs from the related work in fourmajor respects:

I It is based on incremental parsing that seamlessly matches theincremental nature of SMT decoders.

I It is deterministic, in the sense that it maintains a limited numberof parse-states that represent possible parsing decisions at eachword position. This characteristic is very important forincorporating IDLM into large-scale MT systems due to itscomputational efficiency.

I The grammatical representation is based on CCG structureswhich enable the handling of non-constituent constructions.

I The parser seeks out intermediate connected structures, unlikeprevious approaches which deployed dependency relations orhead words to enable syntax-based probabilities into thelanguage model.

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Representation

John likes MaryNP (S\NP)/NP NPNOP BC FANP S/NP S

likes

S0

S1 John

likes

S2

John Mary

John

S0 S1S2

Hany Hassan

Introduction





DTM2


Future Work


Incremental CCG

Mr. Warren will remain on the company 's board

S1 S2 S3 S4 S5 S6 S7 S8 S9

S(S/NP)(S/(NP\NP))

(S/(NP\NP))/NP

(S/NP)(S/PP)S/(S\NP)NPNP/NPState Cat.

FAFCFATRFCFCFCTRFCFANOPOperator

NP(NP/NP)\NP

NPNP/NPPP/NP

(S\NP)/PP

(S\NP)/ (S\NP)

NPNP/NPSupertag

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Training

CCGBank

Transformation to Incremental representation

Supertags Operators

MaxEnt Framework

SupertaggerOperator Tagger

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Runtime

Sentence to parse

Supertagger

Operator Tagger

State Realizer

Dependency Structures

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Features: AppositionHandling

The man , who plays tennis , likes football

S0 : NP/NP NP1 APSV (NP\NP)/(S\NP) (S\NP1)/NP NP APSV (S\NP)/NP NP> FA

S1: NP> INTR

S2: NULL> NOP

S3: NP/(S\NP)> FC

S4: NP/NP> FA

S5: NP> INTR

S6: NP> TRFC

S7: S/NP> FA

S8: S

Figure: Apposition Handling.

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Features: CoordinationHandling

He plays football and tennis

S1 : NP (S\NP)/NP NP2 (NP1\NP2)/NP3 NP3> TRFC

S2: S/NP> FA

S3: S> COORD

S4: S/NP> FA

S5: SFigure: Coordination Handling.

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing Features: WH-movementHandling

He bought what she sold

S0 : NP (S\NP)/NP NP/(S/NP) NP1 (S\NP1)/NP2> TRFC

S1: S/NP> FC

S2: S/(S/NP)> TRFC

S3: S/((S/NP) \NP)< WHMV

S4: SFigure: WH-movement Handling.

Hany Hassan

Introduction





DTM2


Future Work


Incremental Parsing EvaluationAccuracy %

Input Our tags CCGbank tagsGold std. POS 92.39 92.46System POS 91.7 91.81

Table: Supertagger Results.

Input/Features Accuracy %Gold standard POS and Supertags 96.73System POS and Supertags 90.90Preceding correct state as feature 99.22

Table: Operator Tagger Results.

Input F-MeasureGold standard POS and Supertags 87.5System POS and Supertags 86.7

Table: Unlabeled dependency results for section 23.

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


MERT Estimation for log-linear Models:

I Approximiation for Maximum Entroy log-linear modelsI Can handle a few number of paremeters ( in order of ten)I A bottleneck to further serious development of features-rich

SMT systemsI Parameters of different components are not related

Hany Hassan

Introduction





DTM2


Future Work


DTM2 Phrase Structues

Algnp of the X committeeAlmrkzyp central

llhzb of the X Party

Figure: Phrase structures in DTM2. X represents a variable in the targetphrase

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


Dependency Direct Translation Model(DDTM)

P(T|S) = P0(T, J|S)/Z exp∑

i

λiφi(T, J, S) (1)

I P0 is the prior distribution for the phrase probabilityI J is the skip reordering factor for this phrase pair

Hany Hassan

Introduction





DTM2


Future Work


DDTM Features

In our DDTM, we have implemented many features along with thebaseline DTM2 features:

I Supertag-Word features: these features examine the targetphrase words with their associated supertags.

I Supertag sequence features: these features encode n-gramsupertags (equivalent to the n-gram supertags Language Model).

I Supertag-Operator features: these features encode supertags andtheir associated operators.

I Supertag-State features: these features encode states andsupertags co-occurrence.

I State sequence features: these features encode n-gram statesfeatures and are equivalent to an n-gram states Language Model.

I Word-State sequence features: these features encode words andstates co-occurrence.

Hany Hassan

Introduction





DTM2


Future Work


DDTM Decoder

I A beam search decoder similar to decoders used in standardphrase-based log-linear systems

I Performs incremental dependency parsing during decodingI Supports new pruning strategies to handle the large search space

Hany Hassan

Introduction





DTM2


Future Work


DDTM Decoder

e:a : --------P:1S1:NULL

e: attacksa: *----P:=.162ST=NPS2=NP

e: attacksa: *-------P:=.092ST=(S\NP)/NPS4= UNDEF

O:TRFC

e: Riyadha: -*------P:=.142ST=NP/NPS3=NP/NP

e: rockeda: --*--P:=.083ST=(S\NP)/NPS5=S/NP

O:NOP

O:NOP

O:TRFC

e: rockeda: --*------P:=.01ST=(S\NP)/NPS8=S/NP

attacks

attacks rocked

e: Riyadha: --*--P:=.04ST=NPS7=S

O:FC

attacks rocked Riyadh

e: attacksa: *-------P:=.07ST=NPS6=NP

O:TRFC

Riyadh attacks rocked

O:FA

Hany Hassan

Introduction





DTM2


Future Work


DDTM Decoder

Attacks

NP

NOP

NP

rocked Riyadh by two car bombs

tonight

trapped cars

booby

Riayadh

rocked tonight attacks by two booby trapped cars

(S \NP)/NPTRFC

NPFA ((S\NP)\(S\NP))/((S\NP)\(S\NP)

BC((S\NP)\(S\NP))/NP

BC

NP/NPFC

NP/NPFC NP

FA

NP/NPFC

NP/NPFC

NPFA

S/NP S S S/NP S/NP S/NP S

S

NP

NOP

(S \NP)/NPTRFC

NP/NPFC

NPFA ((S\NP)\(S\NP))/NP

BC

NP/NPFC

NP/NPFC

NPFA (NP\NP)/PP

BC

NP S/NP S/NP S S/NP S/NP S/NP S/PPS

Hany Hassan

Introduction





DTM2


Future Work


Experiments

I Arabic–English with 3.7M parallel sentences.I 5-gram LM trained on English Gigaword Corpus.I Testset: Arabic–English MT05I Baseline is top ranked in two recent MT large scale evaluations

Hany Hassan

Introduction





DTM2


Future Work


Evaluated Systems

I IBM-PB: IBM Phrase–based SMT baseline system.I DTM2: the baseline Direct Translation model system.I D-SW: examines Supertag-Word features.I D-SLM: examines Supertag-Word features and supertags

n-gram features.I D-SO: examines Supertag-Operator features.I D-OLM: examines operator n-gram features.I D-SS : examines supertags and states features with parse-state

construction.I D-WS : examines words and states features with parse-state

construction.I D-SLM: examines n-gram states features with parse-state

construction.I DDTM: fully fledged system with all features that proved useful

above.

Hany Hassan

Introduction





DTM2


Future Work


Results

System BLEU Score on MT05IBM-PB 50.16

DTM2-Baseline 52.24D-SW 52.28

D-SLM 52.29D-SO 52.01

D-OLM 51.87D-SS 52.39D-WS 52.03

D-SLM 52.53DDTM 52.61

Table: DDTM Results with various features.

Hany Hassan

Introduction





DTM2


Future Work


Examples

Source: �é£Qå��Ë @ ZAJ.£@ Yg@ A �ë@Qk. @ �HA�ñj ®Ë ½Ë X YªK. © � kðReference: He then underwent medical examinations by a police doctor .Baseline: He was subjected after that tests conducted by doctors of the police .DDTM: Then he underwent tests conducted by doctors of the police .

Hany Hassan

Introduction





DTM2


Future Work


Example

Source: á��Jj� j ®Ó á��KPAJ��. àAÓñj. ë ÐñJË @ ZA�Ó �AKQË @ Që Y�̄ð

Reference: Riyadh was rocked tonight by two car bomb attacks..Baseline: Riyadh rocked today night attacks by two booby - trapped cars.DDTM: Attacks rocked Riyadh today evening in two car bombs.

Figure: DDTM provides better syntatctic structure with more concisetranslation.

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


Future Work

I Source Dependency informationI Enhance the Dependency parser accuracyI Possible implementation of the framework into Moses.I Extend the approach for logical semantics as well

Hany Hassan

Introduction





DTM2


Future Work


Outline

Introduction





DTM2


Future Work


Hany Hassan

Introduction





DTM2


Future Work


I We introduced a novel model of supertagged Phrase-based SMTwhich integrates supertags into the target language model andthe target side of the translation

I We introduced a novel dependency-based LM which isdeterministic in that it maintains a limited number of parsingdecisions at each state which. Furthermore, it is incremental inMarkovian fashion similar to Phrase-based SMT decoders and itcan naturally handle non-constituent constructions, being basedon CCG.

I We introduced an extension to direct translation models thatintegrates incremental dependency parsing while retaining thelinear decoding assumed in conventional Phrase–based SMTsystems.

Hany Hassan

Introduction





DTM2


Future Work


Thanks

Thanks for Listening

- lexical syntax for statistical machine translation · hany hassan introduction syntax for...

Documents