SwitchOut: An Efficient Data Augmentation for NeuralMachine Translation

Xinyi Wang∗, Hieu Pham∗, Zihang Dai, Graham Neubig

November 2, 2018

∗:equal contribution1 / 41

Data Augmentation

Neural models are data hungry, while collecting data is expensive

Prevalent in computer vision1

More difficult for natural languageI Discrete vocabularyI NMT sensitive to arbitrary noise

1image source:Medium2 / 41

Data Augmentation

Neural models are data hungry, while collecting data is expensive

Prevalent in computer vision1

More difficult for natural languageI Discrete vocabularyI NMT sensitive to arbitrary noise

1image source:Medium3 / 41

Data Augmentation

Neural models are data hungry, while collecting data is expensive

Prevalent in computer vision1

More difficult for natural languageI Discrete vocabularyI NMT sensitive to arbitrary noise

1image source:Medium4 / 41

Existing Strategies

Word replacement

Dictionary [Fadaee et al., 2017]

Word dropout [Sennrich et al., 2016a]

Reward Augmented Maximum Likelihood (RAML)[Norouzi et al., 2016]

→ Can we characterize all of the related approaches together?

5 / 41

Existing Strategies

Word replacement

Dictionary [Fadaee et al., 2017]

Word dropout [Sennrich et al., 2016a]

Reward Augmented Maximum Likelihood (RAML)[Norouzi et al., 2016]

→ Can we characterize all of the related approaches together?

6 / 41

Existing Strategies

Word replacement

Dictionary [Fadaee et al., 2017]

Word dropout [Sennrich et al., 2016a]

Reward Augmented Maximum Likelihood (RAML)[Norouzi et al., 2016]

→ Can we characterize all of the related approaches together?

7 / 41

Existing Strategies

Word replacement

Dictionary [Fadaee et al., 2017]

Word dropout [Sennrich et al., 2016a]

Reward Augmented Maximum Likelihood (RAML)[Norouzi et al., 2016]

→ Can we characterize all of the related approaches together?

8 / 41

Existing Strategies

Word replacement

Dictionary [Fadaee et al., 2017]

Word dropout [Sennrich et al., 2016a]

Reward Augmented Maximum Likelihood (RAML)[Norouzi et al., 2016]

→ Can we characterize all of the related approaches together?

9 / 41

Existing Strategies: RAML

RAML [Norouzi et al., 2016]

Motivation: NMT relies on imperfect partial translation at test time,but trained only on gold standard target

Solution: Sample corrupted target during training

Gold target y , corrupted y , similarity measure ry

q∗(y |y , τ) =exp {ry (y , y)/τ}∑y ′ exp {ry (y ′, y)/τ}

10 / 41

Existing Strategies: RAML

RAML [Norouzi et al., 2016]

Motivation: NMT relies on imperfect partial translation at test time,but trained only on gold standard target

Solution: Sample corrupted target during training

Gold target y , corrupted y , similarity measure ry

q∗(y |y , τ) =exp {ry (y , y)/τ}∑y ′ exp {ry (y ′, y)/τ}

11 / 41

Existing Strategies: RAML

RAML [Norouzi et al., 2016]

Motivation: NMT relies on imperfect partial translation at test time,but trained only on gold standard target

Solution: Sample corrupted target during training

Gold target y , corrupted y , similarity measure ry

q∗(y |y , τ) =exp {ry (y , y)/τ}∑y ′ exp {ry (y ′, y)/τ}

12 / 41

Formalize Data Augmentation

Real data distribution: x , y ∼ p(X ,Y )

Observed data distribution: x , y ∼ p(X ,Y )→ Problem: p(X ,Y ) and p(X ,Y ) might have large discrepancy

Data augmentation: x , y ∼ q(X , Y )

13 / 41

Formalize Data Augmentation

Real data distribution: x , y ∼ p(X ,Y )

Observed data distribution: x , y ∼ p(X ,Y )

→ Problem: p(X ,Y ) and p(X ,Y ) might have large discrepancy

Data augmentation: x , y ∼ q(X , Y )

14 / 41

Formalize Data Augmentation

Real data distribution: x , y ∼ p(X ,Y )

Observed data distribution: x , y ∼ p(X ,Y )→ Problem: p(X ,Y ) and p(X ,Y ) might have large discrepancy

Data augmentation: x , y ∼ q(X , Y )

15 / 41

Formalize Data Augmentation

Real data distribution: x , y ∼ p(X ,Y )

Observed data distribution: x , y ∼ p(X ,Y )→ Problem: p(X ,Y ) and p(X ,Y ) might have large discrepancy

Data augmentation: x , y ∼ q(X , Y )

16 / 41

Design a good q(X , Y )

q: function of observed (x , y)

How should q approximate p?I Diversity: larger support with all valid data pairs (x , y)

F Entropy H[q(x , y |x , y)

]is large

I Smoothness: probability of similar data pairs are similarF q maximizes similarity measure rx(x , x), ry (y , y)

τ : control effect of diversity; q should maximize

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]

17 / 41

Design a good q(X , Y )

q: function of observed (x , y)How should q approximate p?

I Diversity: larger support with all valid data pairs (x , y)F Entropy H

[q(x , y |x , y)

]is large

I Smoothness: probability of similar data pairs are similarF q maximizes similarity measure rx(x , x), ry (y , y)

τ : control effect of diversity; q should maximize

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]

18 / 41

Design a good q(X , Y )

q: function of observed (x , y)How should q approximate p?

I Diversity: larger support with all valid data pairs (x , y)F Entropy H

[q(x , y |x , y)

]is large

I Smoothness: probability of similar data pairs are similarF q maximizes similarity measure rx(x , x), ry (y , y)

τ : control effect of diversity; q should maximize

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]

19 / 41

Design a good q(X , Y )

q: function of observed (x , y)How should q approximate p?

I Diversity: larger support with all valid data pairs (x , y)F Entropy H

[q(x , y |x , y)

]is large

I Smoothness: probability of similar data pairs are similarF q maximizes similarity measure rx(x , x), ry (y , y)

τ : control effect of diversity; q should maximize

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]

20 / 41

Design a good q(X , Y )

q: function of observed (x , y)How should q approximate p?

I Diversity: larger support with all valid data pairs (x , y)F Entropy H

[q(x , y |x , y)

]is large

I Smoothness: probability of similar data pairs are similarF q maximizes similarity measure rx(x , x), ry (y , y)

τ : control effect of diversity; q should maximize

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]

21 / 41

Mathematically Optimal q

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]Solve for the best q

q∗(x , y |x , y) =exp {s(x , y ; x , y)/τ}∑

x ′,y ′ exp {s(x ′, y ′; x , y)/τ}

Decompose x and y

q∗(x , y |x , y) =exp {rx(x , x)/τx}∑x ′ exp {rx(x ′, x)/τx}

× exp {ry (y , y)/τy}∑y ′ exp {ry (y ′, y)/τy}

Formulate existing methodsI Dictionary: jointly on x and y , but deterministic and not diverseI Word dropout: only x side with null tokenI RAML: only y side

22 / 41

Mathematically Optimal q

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]Solve for the best q

q∗(x , y |x , y) =exp {s(x , y ; x , y)/τ}∑

x ′,y ′ exp {s(x ′, y ′; x , y)/τ}

Decompose x and y

q∗(x , y |x , y) =exp {rx(x , x)/τx}∑x ′ exp {rx(x ′, x)/τx}

× exp {ry (y , y)/τy}∑y ′ exp {ry (y ′, y)/τy}

Formulate existing methodsI Dictionary: jointly on x and y , but deterministic and not diverseI Word dropout: only x side with null tokenI RAML: only y side

23 / 41

Mathematically Optimal q

J(q) = τ ·H[q(x , y |x , y)

]+ Ex ,y∼q

[rx(x , x) + ry (y , y)

]Solve for the best q

q∗(x , y |x , y) =exp {s(x , y ; x , y)/τ}∑

x ′,y ′ exp {s(x ′, y ′; x , y)/τ}

Decompose x and y

q∗(x , y |x , y) =exp {rx(x , x)/τx}∑x ′ exp {rx(x ′, x)/τx}

× exp {ry (y , y)/τy}∑y ′ exp {ry (y ′, y)/τy}

Formulate existing methodsI Dictionary: jointly on x and y , but deterministic and not diverseI Word dropout: only x side with null tokenI RAML: only y side

24 / 41

Formulate SwitchOut

Augment both x and y !

Sample for x , y independently

Define rx(x , x) and ry (y , y)I Negative Hamming Distance, following RAML

25 / 41

Formulate SwitchOut

Augment both x and y !

Sample for x , y independently

Define rx(x , x) and ry (y , y)I Negative Hamming Distance, following RAML

26 / 41

Formulate SwitchOut

Augment both x and y !

Sample for x , y independently

Define rx(x , x) and ry (y , y)I Negative Hamming Distance, following RAML

27 / 41

SwitchOut: Sample efficiently

Given a sentence s = {s1, s2, ...s|s|}1 How many words to corrupt?

Assumption: only one token for swapping.

P(n) ∝ exp(−n)/τ

2 What is the corrupted sentence?

P(randomly swap si by another word) =n

|s|

See Appendix: Efficient batch implementation in PyTorch and Tensorflow

28 / 41

SwitchOut: Sample efficiently

Given a sentence s = {s1, s2, ...s|s|}1 How many words to corrupt?

Assumption: only one token for swapping.

P(n) ∝ exp(−n)/τ

2 What is the corrupted sentence?

P(randomly swap si by another word) =n

|s|

See Appendix: Efficient batch implementation in PyTorch and Tensorflow

29 / 41

Experiments

DatasetsI en-vi: IWSLT 2015I de-en: IWSLT 2016I en-de: WMT 2015

ModelsI Transformer modelI Word-based, standard preprocessing

30 / 41

Results: RAML and word dropout

SwitchOut on source > word dropout

SwitchOut on source and target > RAML

Methoden-de de-en en-vi

src trg

N/A N/A 21.73 29.81 27.97WordDropout N/A 20.63 29.97 28.56SwitchOut N/A 22.78† 29.94 28.67†

N/A RAML 22.83 30.66 28.88WordDropout RAML 20.69 30.79 28.86SwitchOut RAML 23.13† 30.98† 29.09

31 / 41

Results: RAML and word dropout

SwitchOut on source > word dropout

SwitchOut on source and target > RAML

Methoden-de de-en en-vi

src trg

N/A N/A 21.73 29.81 27.97WordDropout N/A 20.63 29.97 28.56SwitchOut N/A 22.78† 29.94 28.67†

N/A RAML 22.83 30.66 28.88WordDropout RAML 20.69 30.79 28.86SwitchOut RAML 23.13† 30.98† 29.09

32 / 41

Results: RAML and word dropout

SwitchOut on source > word dropout

SwitchOut on source and target > RAML

Methoden-de de-en en-vi

src trg

N/A N/A 21.73 29.81 27.97WordDropout N/A 20.63 29.97 28.56SwitchOut N/A 22.78† 29.94 28.67†

N/A RAML 22.83 30.66 28.88WordDropout RAML 20.69 30.79 28.86SwitchOut RAML 23.13† 30.98† 29.09

33 / 41

Where does SwitchOut help?

More gain for sentences more different from training data

1350 2700 4050 5400Top K sentences

-0.25

0

0.25

0.5

0.75

Gainin

BLEU

253 506 759 1012Top K sentences

-1

-0.5

0

0.5

1

Gainin

BLEU

Figure: Left: IWSLT 16 de-en. Right: IWSLT 15 en-vi.

34 / 41

Final Thoughts

SwitchOut sampling is efficient and easy-to-use

Work with any NMT architecture

Formulation of data augmentation encompasses existing works andinspires future direction

Thanks a lot for listening! Questions?

35 / 41

Final Thoughts

SwitchOut sampling is efficient and easy-to-use

Work with any NMT architecture

Formulation of data augmentation encompasses existing works andinspires future direction

Thanks a lot for listening! Questions?

36 / 41

Final Thoughts

SwitchOut sampling is efficient and easy-to-use

Work with any NMT architecture

Formulation of data augmentation encompasses existing works andinspires future direction

Thanks a lot for listening! Questions?

37 / 41

Final Thoughts

SwitchOut sampling is efficient and easy-to-use

Work with any NMT architecture

Formulation of data augmentation encompasses existing works andinspires future direction

Thanks a lot for listening! Questions?

38 / 41

References

Norouzi et al. (2016) Reward Augmented Maximum Likelihood for NeuralStructured Prediction. In NIPS.

Sennrich et al. (2016a) Edinburgh neural machine translation systems for wmt 16.In WMT.

Sennrich et al. (2016b) Improving neural machine translation models withmonolingual data. In ACL.

Currey et al. (2017) Copied Monolingual Data Improves Low-Resource NeuralMachine Translation. In WMT.

Fadaee et al. (2017) Data Augmentation for Low-Resource Neural MachineTranslation. In ACL.

39 / 41

Results: Back Translation (WMT en-de 2015)

SwitchOut > Back Translation

Switchout + RAML + back translate wins

Method en-de

Transformer 21.73+SwitchOut 22.78

+BT 21.82

+BT +RAML 21.53+BT +SwitchOut 22.93+BT +RAML +SwitchOut 23.76

40 / 41

Results: Back Translation (WMT en-de 2015)

SwitchOut > Back Translation

Switchout + RAML + back translate wins

Method en-de

Transformer 21.73+SwitchOut 22.78

+BT 21.82+BT +RAML 21.53+BT +SwitchOut 22.93+BT +RAML +SwitchOut 23.76

41 / 41