machine translation zoo - univerzita karlovaufal.mff.cuni.cz/~popel/papers/2013_05_06_zoo.pdf ·...

Machine Translation ZooTree-to-tree transfer and Discriminative learning

Martin Popel

UFAL (Institute of Formal and Applied Linguistics)Charles University in Prague

May 5th 2013, Seminar of Formal Linguistics, Prague

Today’s Menu

1 MT IntroTaxonomyHybrids

2 Online LearningPerceptronStructured Prediction

3 Guided Learning

4 Back to MTEasy-First Decoding in MTGuided Learning in MT

MT Intro Online Learning Guided Learning Back to MT

Phrase-based MT (Moses)

Training

word-alignment(Giza++ & symmetrization)

phrase extraction

tune parameters (MERT)

Decoding

get all matching rules

find one derivationwith a maximum score (beam search)


TectoMT

Training

analyze CzEng to t-layer

t-node alignment

learn one MaxEnt model for each source lemma and formeme

Decoding

get all translation variants for each lemma and formeme

find a labeling with a maximum score (HMTM)


TectoMT – MaxEnt Model

ANALYSIS

TRANSFER

SYNTHESIS

He agreed with the unions to cut all overtime. Dohodl se s odbory na zrušení všech přesčasů.

agree / v:fintense=past, voice=activenegation=0, sempos=v

he / n:subj union /n:with+X

overtime / n:obj

cut / v:inf,has_left_child=0, sempos=v,has_right_child=1, tag=VB,position=right, named_entity=0

all / adj:attr

chop, saw, trim, shorten, lumber, hew,

lower, delete, cropabolish, cancel,...


Machine Translation Taxonomy

Level of transfer:

surface, shallow syntax, tectogrammatical

Base translation unit (BTU):

word, phrase, phrase with gaps, treelet, node

Extract more segmentations in training?

Try (search) more segmentations in decoding?

Use more segmentations in the output translation?

What is the context X in P(BTUtarget |BTUsource ,X )?

Considering just Translation Model:

(Brown et al., 1993)word-based



Level of transfer: surface

, shallow syntax, tectogrammatical

Base translation unit (BTU):word

, phrase, phrase with gaps, treelet, node

Extract more segmentations in training? no

Try (search) more segmentations in decoding? no

Use more segmentations in the output translation? no

What is the context X in P(BTUtarget |BTUsource ,X )?Considering just Translation Model: nothing






Base translation unit (BTU):word, phrase

, phrase with gaps, treelet, node

Extract more segmentations in training? yes

Try (search) more segmentations in decoding? yes




(Koehn et al., 2003)phrase-based





Base translation unit (BTU):word, phrase, phrase with gaps

, treelet, node







(Chiang, 2005)hierarchical



Level of transfer: surface, shallow syntax

, tectogrammatical

Base translation unit (BTU):word, phrase, phrase with gaps, treelet

, node




What is the context X in P(BTUtarget |BTUsource ,X )?Considering just Translation Model: neighboring treelets




(Quirk and Menezes, 2006)dep. treelet to string



Level of transfer: surface, shallow syntax, tectogrammatical

Base translation unit (BTU):word, phrase, phrase with gaps, treelet, node




What is the context X in P(BTUtarget |BTUsource ,X )?Considering just Translation Model: neighboring nodes





(Marecek et al., 2010)TectoMT



Level of transfer: surface, shallow syntax, tectogrammatical

Base translation unit (BTU):word, phrase, phrase with gaps, treelet, node



Use more segmentations in the output translation? yes






(Marecek et al., 2010)TectoMT

(Arun, 2011)Monte Carlo


Hybrids: TectoMoses

Linearize source t-trees (two factors: lemma and formeme), translatewith Moses, project dependencies and use TectoMT synthesis.

& blocks

source language (English) target language (Czech)

morphological layer

analytical layer

tectogramatical layer

a-layer

m-layer

w-layer

ANALYSIS TRANSFER SYNTHESIS

t-layer

tokenizationlemmatizationtagger (Morce)

parser (McDonald's MST)analytical functions

mark edges to contract

build t-tree

fill formems grammatemes useHMTM

querydictionary

fill morphological categories

impose agreement

add functional words

generatewordforms

concatenate

rule based statistical

segmentation


Hybrids: PhraseFix

Done for WMT 2013 by Petra Galuscakova:

Post-edit TectoMT output using Moses

trained on cs-tectomt → cs-reference (whole CzEng).

How to post-edit only when confident?

filter phrase tableadd “confidence” feature for MERTimprove alignment (monolingual)boost phrase table (e.g. with identities)

Future work:

use also source (English) sentences ⇒ multi-source translation

project only content words (using TectoMT)

factored translation with non-synchronous(overlapping) factors


Even More Hybrids: DepFix, AddToTrain, Chimera

DepFix (Rosa et al., 2012)

post-edit SMT using syntactic analysis and rules

exploit also the source sentences, robust parsing

AddToTrain (Bojar, Galuscakova)

translate monolingual news (or WMT devsets) with TectoMT

add this to Moses parallel training data

Chimera

post-edit AddToTrain output with DepFix

sent to WMT 2013 in attempt to beat Google

Today’s Menu



3 Guided Learning



General Algorithm for Online Learning

w := 0while (x, ygold) := get new data()

ypred := prediction(w, x)w += update(x, ygold , ypred)

Output: w





Output: w

initialize all weights to zero

for each instance (observation)1. get its features x2. do the prediction ypred3. get the correct label ygold4. update the weights





Output: w

initialize all weights to zerofor each instance (observation)

1. get its features x2. do the prediction ypred3. get the correct label ygold4. update the weights





Output: w

initialize all weights to zerofor each instance (observation)1. get its features x

2. do the prediction ypred3. get the correct label ygold4. update the weights





Output: w

initialize all weights to zerofor each instance (observation)1. get its features x2. do the prediction ypred

3. get the correct label ygold4. update the weights





Output: w

initialize all weights to zerofor each instance (observation)1. get its features x2. do the prediction ypred3. get the correct label ygold

4. update the weights





Output: w

initialize all weights to zerofor each instance (observation)1. get its features x2. do the prediction ypred3. get the correct label ygold4. update the weights





Output: w


Definition: conservative online learning

no error ⇒ no updatei.e., if ypred = ygold then update(x, ygold , ypred) = 0




ypred := prediction(w, x)w += update(x, ygold , ypred)//prediction(w, x) = ygold

Output: w


Definition: conservative online learning

no error ⇒ no updatei.e., if ypred = ygold then update(x, ygold , ypred) = 0

Definition: aggressive online learning

after the update, the instance would be classified correctly


Perceptron



Output: w

prediction(w, x)def=

update(x, ygold , ypred)def=

Binary Perceptron

[w · x > 0]

α(ygold − ypred) · x

Multi-class Perceptron

arg maxy w · f(x, y)

α(f(x, ygold)− f(x, ypred)

)


Perceptron



Output: w

dot product (similarity score)of weights and featuresw · x =

∑i wixi



Binary Perceptron

[w · x > 0]





)


Perceptron



Output: w


∑i wixi

Iverson bracket

[P] =

{1 if P is true;

0 otherwise.



Binary Perceptron

[w · x > 0]





)


Perceptron



Output: w


∑i wixi

Iverson bracket

[P] =

{1 if P is true;

0 otherwise.



Binary Perceptron

[w · x > 0]





)

learning rate (step size) α > 0


Perceptron



Output: w



Binary Perceptron

[w · x > 0]





)learning rate (step size) α > 0


Perceptron



Output: w

Special case:multi-prototype features

f(x, y)def= [y = class1] · x ,

[y = class2] · x ,· · ·[y = classC ] · x



Binary Perceptron

[w · x > 0]





)w := w + αf(x, ygold)− αf(x, ypred)


Perceptron



Output: w

General case:any label-dependent features, e.g.

f101(x, y)def= [(y=NNP or y=NNPS)

and x capitalized ]



Binary Perceptron

[w · x > 0]





)


Structured Prediction

the number of possible labels is huge

labels y have a structure (graph, tree, sequence,. . . )

usually can be decomposed (factorized) into subproblems

local features

fi (x, y, j) can use whole x, but only such yk where k is “near” j

f101(x, y, j)def= [ (yj=NNP or yj=NNPS) and word xj capitalized ]

f102(x, y, j)def= [ yj=NNP and yj−1=NNP and |x| ≤ 6 ]

global features

Fi (x, y)def=∑

j fi (x, y, j)F101 . . . number of capitalized words with tag NNP or NNPSF102 . . . number of NNP followed by NNP

or 0 if the sentence is longer than six wordsWe can define also features that cannot be decomposed


Structured Prediction using Online Learning

1 local approach

update after each local decisionoutput of previous decisions used in local featurese.g. Structured Perceptron (Collins, 2002)ypred = arg maxy

∑i wi fi (x, yj , yj−1, ...)

2 global approach

generate n-best list (lattice) of outputs y for the whole xcompute global features, do update for each x (sentence)we are re-ranking the n-best liste.g. MIRA (Crammer and Singer, 2003)ypred = arg maxy

∑i wiFi (x, y)


Margin-based Online Learning

Definitions

score(y) = w · f(x, y)

margin(y) =score(ygold)− score(y)

margin > 0⇒ no error|margin| ∼ confidence

hinge loss(y) =max

(0, 1−margin(y)

)Online Prediction and Update

ypreddef= arg maxw · f(x, y)

w += α(f(x, ygold)− f(x, ypred)

)

Perceptron

αPerc

def= 1 (or any fixed value > 0)

Passive Aggressive (PA)

αPA

def=

hinge loss(ypred )||f(x,ygold )−f(x,ypred )||2

Passive Aggressive I

αPA-I

def= min {C , αPA}

Passive Aggressive II

αPA-II

def=

hinge loss(ypred )

||f(x,ygold )−f(x,ypred )||2+ 12C


Cost-sensitive Online Learning

Definitions

cost(y) = external error metric (non-negative)e.g. 1 - similarity of y and ygold

hinge loss(y) = max(0, cost(y)−margin(y)

)Hope and Fear

w += α(f(x, ygold)− f(x, ypred)

)min-cost yhope

def= arg maxy −cost(y)

max-score yfeardef= arg maxy score(y)

cost-diminished yhopedef= arg maxy score(y)− cost(y)

cost-augmented yfeardef= arg maxy score(y) + cost(y)

max-cost yfeardef= arg maxy cost(y)



Definitions

cost(y) = external error metric (non-negative)e.g. 1 - similarity of y and ygold

hinge loss(y) = max(0, cost(y)−margin(y)

)Hope and Fear

w += α(f(x, yhope)− f(x, yfear )

)min-cost yhope

def= arg maxy −cost(y)

max-score yfeardef= arg maxy score(y)

cost-diminished yhopedef= arg maxy score(y)− cost(y)

cost-augmented yfeardef= arg maxy score(y) + cost(y)

max-cost yfeardef= arg maxy cost(y)



Hypothesis Selection -cost

score

min-cost(max-BLEU) hope

max-score fear (ypred)

cost-diminished hope

cost-augmented fear

n-best listmax-cost fear




score




cost-augmented fear

n-best list

max-cost fear




score




cost-augmented fear

n-best listmax-cost fear


Application to MT

x = source sentenceygold = its reference translation

more references sometimes available

reference may be unreachable

we score derivations (which include latent variables)one translation may have more derivations

Today’s Menu



3 Guided Learning



Easy-First Decoding (PoS Tagging)

score

Agatha found that book interesting

NN

VB

JJ

DT

DT

f123def= [yj = DT , yj+1 = NN]

IN

RB

f124def= [yj = RB, yj+1 = NN]

VBN

VBD

NNP

yfear

yhope

(Shen, Satta and Joshi, 2007)


Easy-First Decoding (PoS Tagging)

score


NN

VB

JJDT

DT


IN

RB


VBN

VBD

NNP

yfear

yhope



Guided Learning (PoS Tagging)

score


NN

VB

JJDT

DT


IN

RB


VBN

VBD

NNP

yfear

yhope



Easy-First Decoding (Dependency Parsing)

(Goldberg and Elhadad, 2010)

Today’s Menu



3 Guided Learning



Easy-First Decoding (Phrase-Based MT)

score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava

Agatha

prisla

nasel

zjistil, ze

ze

ta

ze rezervovat

ta kniha

ze kniha

kniha

rezervovat

kniha zajımava

kniha zajımavy

zajımavy

zajımava



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava

ze rezervovat

ta kniha

ze kniha

kniha

rezervovat

kniha zajımava

kniha zajımavy



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava

kniha zajımava

kniha zajımavy

zajımavy

zajımava

language model



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava

kniha zajımava

kniha zajımavy

zajımavy

zajımava



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava



score

Agatha

Agate

found

prisla

that

ta

book

kniha

interesting

zajımava

prisla

nasel

zjistil, ze

ze

ta


Features for Guided Learning in MT

Source Segment Features

segment size (number of words)

entropy P(target|source) = −∑

i P(src , trgi ) · logP(trgi |src)

log count(source)

source language model: log P(source)

word identity, e.g. f42def= [src=found that]

PoS identity, e.g. f43def= [src pos=VBD IN]

Target-dependent Features

log P(trg|src)

target language model: log P(target | previous segment)

log count(target)?

identity, e.g. f142def= [src=found that & trg=zjistil]

Combinations and Quantizations

[size(src) = 3] · logP(trg |src)[size(src) = 3 &− 3 < logP(trg |src) < −2]etc.


Application to Tecto Trees

findv:fin

Agathan:subj

bookn:obj

thisadj:attr

interestingadj:compl


Application to Tecto Trees

v:fin

find

n:subj

Agatha

n:obj

book

adj:attr

this

adj:compl

interesting


What have you seen in the Zoo


Predictions?

Hope and Fear

http://syd.korpus.cz/

http://books.google.com/ngrams/

machine translation zoo - univerzita karlovaufal.mff.cuni.cz/~popel/papers/2013_05_06_zoo.pdf ·...

Documents