lexical translation models 1idemo.clab.cs.cmu.edu/sp2013-11731/slides/05.lexical2.pdf · •model...

Lexical Translation Models 1I

January 29, 2013

Tuesday, February 19, 13

Last Time ...

AlignmentX

Alignment

p( )Translationp( )= Translation,

Last Time ...

AlignmentX

Alignment

Alignment Translation | Alignment⇥X

Alignment

p( p() )=

Last Time ...

AlignmentX

Alignment

Alignment Translation | Alignment⇥X

Alignment

p( p() )=

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

| {z } | {z }z }| { z }| {

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

p(ei | fai , fai�1)

p(ei | fai , ei�1)

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

p(ei | fai , fai�1)

p(ei | fai , ei�1)

p(ei, ei+1 | fai)

What is the problem here?Tuesday, February 19, 13

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

a2[0,n]m

1 + n| {z }p(a|f,m)

p(ei | fai)

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

a2[0,n]m

1 + n| {z }p(a|f,m)

p(ei | fai)

a2[0,n]m

1 + np(ei | fai)

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

a2[0,n]m

1 + n| {z }p(a|f,m)

p(ei | fai)

a2[0,n]m

1 + np(ei | fai)

a2[0,n]m

p(ai)⇥ p(ei | fai)

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

a2[0,n]m

1 + n| {z }p(a|f,m)

p(ei | fai)

a2[0,n]m

1 + np(ei | fai)

a2[0,n]m

p(ai)⇥ p(ei | fai)

Can we do something better here?

a2[0,n]m

p(ai)⇥ p(ei | fai)p(e | f,m)

a2[0,n]m

p(ai | i,m, n)⇥ p(ei | fai)Model 2

• Model alignment with an absolute position distribution

• Probability of translating a foreign word at position to generate the word at position (with target length and source length )

• EM training of this model is almost the same as with Model 1 (same conditional independencies hold)

a2[0,n]m

p(ai | i,m, n)

a2[0,n]m

natürlich ist das haus klein

a2[0,n]m

natürlich ist das haus klein

natürlich natürlich istdas haus klein

of course the house is small

• Pros

• Non-uniform alignment model

• Fast EM training / marginal inference

• Cons

• Absolute position is very naive

• How many parameters to model

a2[0,n]m

p(ai | i,m, n)

a2[0,n]m

How much do we knowwhen we only know thesource & target lengthsand the current position?

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

How many parametersdo we need to model this?

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

pos in target

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

pos in target pos in source

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

target len

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

target len source len

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

b(j | i,m, n) =exp�h(j, i,m, n)Pj0 exp�h(j

0, i,m, n)

a2[0,n]m

j0 = 1

j0 = 2

j0 = 3

j0 = 4

j0 = 5

}m = 6

h(j, i,m, n) = ��i

m� j

��

b(j | i,m, n) =exp�h(j, i,m, n)Pj0 exp�h(j

0, i,m, n)

p(ai | i,m, n) =

(p0 if ai = 0

(1� p0)b(ai | i,m, n) otherwise

Words reorder in groups. Model this!

a2[0,n]m

p(ai | ai�1)⇥ p(ei | fai)HMM

• Insight: words translate in groups

• Condition on previous alignment position

• Probability of translating a foreign word at position given that the previous position translated was

• EM training of this model using forward-backward algorithm (dynamic programming)

aiai�1

p(ai | ai�1)

a2[0,n]m

• Improvement: model “jumps” through the source sentence

• Relative position model rather than absolute position model

a2[0,n]m

p(ai | ai�1) = j(ai � ai�1)

-4 0.0008

-3 0.0015

-2 0.08

-1 0.18

0 0.0881

2 0.16

3 0.064

4 0.0256

• Be careful! NULLs must be handled carefully. Here is one option (due to Och):

a2[0,n]m

p(ai | ai�ni) =

(p0 if ai = 0

(1� p0)j(ai � ai�ni) otherwise

ni is the index of the first non-null alignedword in the alignment to the left of .i

• Other extensions: certain word-types are more likely to be reordered

a2[0,n]m

Condition the jump probability on the previousword translated

j(� | f)

a2[0,n]m

j(� | f)

j(� | f, e)

Condition the jump probability on the previousword translated, and how it was translated

a2[0,n]m

j(� | f)

j(� | f, e)

Condition the jump probability on the previousword translated, and how it was translated

j(� | C(f))

j(� | A(f),B(e))

Fertility Models

• The models we have considered so far have been efficient

• This efficiency has come at a modeling cost:

• What is to stop the model from “translating” a word 0, 1, 2, or 100 times?

• We introduce fertility models to deal with this

IBM Model 3

Fertility• Fertility: the number of English words generated by a foreign

• Modeled by categorical distribution

• Examples:

n(� | f)

0 0.00008

2 0.0002

4 0.009

Unabhaengigkeitserklaerung

0 0.01

3 0.0009

4 0.0001

zum = (zu + dem)

0 0.01

1 0.92

2 0.07

Fertility

• Fertility models mean that we can no longer exploit conditional independencies to write as a series of localalignment decisions.

• How do we compute the statistics required for EM training?

p(a | f,m)

Fertility

• Fertility models mean that we can no longer exploit conditional independencies to write as a series of localalignment decisions.

• How do we compute the statistics required for EM training?

p(e | f,m) =X

a2[0,n]m

p(a | f,m)⇥mY

p(ei | fai)

p(a | f,m)

EM Recipe reminder

• If alignment points were visible, training fertility models would be easy

• We would _______ and ________

• But, alignments are not visible

n(� = 3 | f = Unabhaenigkeitserklaerung) =count(3,Unabhaenigkeitserklaerung)

count(Unabhaenigkeitserklaerung)

EM Recipe reminder

• If alignment points were visible, training fertility models would be easy

• We would _______ and ________

• But, alignments are not visible

n(� = 3 | f = Unabhaenigkeitserklaerung) =count(3,Unabhaenigkeitserklaerung)

count(Unabhaenigkeitserklaerung)

n(� = 3 | f = Unabhaenigkeitserklaerung) =E[count(3,Unabhaenigkeitserklaerung)]E[count(Unabhaenigkeitserklaerung)]

Expectation & Fertility

• We need to compute expected counts under p(a | f,e,m)

• Unfortunately p(a | f,e,m) doesn’t factorize nicely. :(

• Can we sum exhaustively? How many different a’s are there?

• What to do?

Sample Alignments• Monte-Carlo methods

• Gibbs sampling

• Importance sampling

• Particle filtering

• For historical reasons

• Use model 2 alignment to start (easy!)

• Weighted sum over all alignment configurations that are “close” to this alignment configuration

• Is this correct? No! Does it work? Sort of.

Pitfalls of Conditional Models

IBM Model 4 alignment Our model's alignment

Figure 2: Example English-Urdu alignment under IBM Model 4 (left) and our discriminative model (right). Model4 displays two characteristic errors: garbage collection and an overly-strong monotonicity bias. Whereas our modeldoes not exhibit these problems, and in fact, makes no mistakes in the alignment.

pervised setting. The contrastive estimation tech-nique proposed by Smith and Eisner (2005) is glob-ally normalized (and thus capable of dealing with ar-bitrary features), and closely related to the model wedeveloped; however, they do not discuss the problemof word alignment. Berg-Kirkpatrick et al. (2010)learn locally normalized log-linear models in a gen-erative setting. Globally normalized discriminativemodels with latent variables (Quattoni et al., 2004)have been used for a number of language processingproblems, including MT (Dyer and Resnik, 2010;Blunsom et al., 2008a). However, this previouswork relied on translation grammars constructed us-ing standard generative word alignment processes.

7 Future Work

While we have demonstrated that this model can besubstantially useful, it is limited in some importantways which are being addressed in ongoing work.First, training is expensive, and we are exploring al-ternatives to the conditional likelihood objective thatis currently used, such as contrastive neighborhoodsadvocated by (Smith and Eisner, 2005). Addition-ally, there is much evidence that non-local featureslike the source word fertility are (cf. IBM Model 3)useful for translation and alignment modeling. To betruly general, it must be possible to utilize such fea-tures. Unfortunately, features like this that dependon global properties of the alignment vector, a, make

the inference problem NP-hard, and approximationsare necessary. Fortunately, there is much recentwork on approximate inference techniques for incor-porating nonlocal features (Blunsom et al., 2008b;Gimpel and Smith, 2009; Cromieres and Kurohashi,2009; Weiss and Taskar, 2010), suggesting that thisproblem too can be solved using established tech-niques.

8 Conclusion

We have introduced a globally normalized, log-linear lexical translation model that can be traineddiscriminatively using only parallel sentences,which we apply to the problem of word alignment.Our approach addresses two important shortcomingsof previous work: (1) that local normalization ofgenerative models constrains the features that can beused, and (2) that previous discriminatively trainedword alignment models required supervised align-ments. According to a variety of measures in a vari-ety of translation tasks, this model produces superioralignments to generative approaches. Furthermore,the features learned by our model reveal interestingcharacteristics of the language pairs being modeled.

AcknowledgmentsThis work was supported in part by the DARPA GALEprogram; the U. S. Army Research Laboratory and theU. S. Army Research Office under contract/grant num-

Pitfalls of Conditional Models

IBM Model 4 alignment Our model's alignment

Figure 2: Example English-Urdu alignment under IBM Model 4 (left) and our discriminative model (right). Model4 displays two characteristic errors: garbage collection and an overly-strong monotonicity bias. Whereas our modeldoes not exhibit these problems, and in fact, makes no mistakes in the alignment.

pervised setting. The contrastive estimation tech-nique proposed by Smith and Eisner (2005) is glob-ally normalized (and thus capable of dealing with ar-bitrary features), and closely related to the model wedeveloped; however, they do not discuss the problemof word alignment. Berg-Kirkpatrick et al. (2010)learn locally normalized log-linear models in a gen-erative setting. Globally normalized discriminativemodels with latent variables (Quattoni et al., 2004)have been used for a number of language processingproblems, including MT (Dyer and Resnik, 2010;Blunsom et al., 2008a). However, this previouswork relied on translation grammars constructed us-ing standard generative word alignment processes.

7 Future Work

While we have demonstrated that this model can besubstantially useful, it is limited in some importantways which are being addressed in ongoing work.First, training is expensive, and we are exploring al-ternatives to the conditional likelihood objective thatis currently used, such as contrastive neighborhoodsadvocated by (Smith and Eisner, 2005). Addition-ally, there is much evidence that non-local featureslike the source word fertility are (cf. IBM Model 3)useful for translation and alignment modeling. To betruly general, it must be possible to utilize such fea-tures. Unfortunately, features like this that dependon global properties of the alignment vector, a, make

the inference problem NP-hard, and approximationsare necessary. Fortunately, there is much recentwork on approximate inference techniques for incor-porating nonlocal features (Blunsom et al., 2008b;Gimpel and Smith, 2009; Cromieres and Kurohashi,2009; Weiss and Taskar, 2010), suggesting that thisproblem too can be solved using established tech-niques.

8 Conclusion

We have introduced a globally normalized, log-linear lexical translation model that can be traineddiscriminatively using only parallel sentences,which we apply to the problem of word alignment.Our approach addresses two important shortcomingsof previous work: (1) that local normalization ofgenerative models constrains the features that can beused, and (2) that previous discriminatively trainedword alignment models required supervised align-ments. According to a variety of measures in a vari-ety of translation tasks, this model produces superioralignments to generative approaches. Furthermore,the features learned by our model reveal interestingcharacteristics of the language pairs being modeled.

AcknowledgmentsThis work was supported in part by the DARPA GALEprogram; the U. S. Army Research Laboratory and theU. S. Army Research Office under contract/grant num-

Lexical Translation• IBM Models 1-5 [Brown et al., 1993]

• Model 1: lexical translation, uniform alignment

• Model 2: absolute position model

• Model 3: fertility

• Model 4: relative position model (jumps in target string)

• Model 5: non-deficient model

• HMM translation model [Vogel et al., 1996]

• Relative position model (jumps in source string)

• Latent variables are more useful these days than the translations

• Widely used Giza++ toolkit

p(e|f)

A few tricks...

p(f|e)

A few tricks...

p(f|e) p(e|f)

A few tricks...

p(f|e) p(e|f)

Announcements• Upcoming language-in-10

• Thursday: Weston - 官话

• Tuesday: Jon/Austin - Русский

• Leaderboard is functional

lexical translation models 1idemo.clab.cs.cmu.edu/sp2013-11731/slides/05.lexical2.pdf · •model...

Documents

sp2013 wcm bootcamp - keynote

samuel zürcher service applications in sp2013

deploying and managing sp2013 apps

language model algorithms - carnegie mellon...

social features sp2013

me525 sp2013 lecture 26

syntax-based translation models – principles, approaches...

évènements sp2013 en france

migration sp2013

wastewater treatment bs 105 sp2013.ppt

sp2013 what's new for developers : spsstl

markise sp2013 - otto · 2020. 3. 5. · sp2013 - 36 - d it...

11-731 (spring2013) lecture 22: example-based machine...

phrase-based mt: decoding - carnegie mellon...

case sp2013

repaso sp2013

day1 sp2013 intro

sp2013 communities

banca def sp2013

11731 p1-78.pdf