esr7 carolina scarton - expert summer school - malaga 2015

Finding Ways to Assess Machine Translated Documents for Document-level Quality Prediction

Carolina Scarton [email protected]

Supervisor: Dr Lucia Specia

EXPERT – Scientific and Technological Workshop – Malaga, Spain – 27/06/2015


2

Agenda

Introduction

Quality Estimation Framework

Related Work

Document-level Quality Estimation

Quality Label problem

Two-stage post-edition experiment

Large-scale experiments

Conclusion


3

Agenda

Introduction


Related Work





Conclusion


4

Introduction

Quality estimation (QE) of machine translations

– quality predictions for new, unseen machine translated texts


5

Introduction



– use of machine learning techniques – only few labelled data points


6

Introduction



– use of machine learning techniques – only few labelled data points

– different from BLEU-style metrics – QE does not rely on reference translations


7

Introduction

Open problems:

– Granularity level?• Word-level• Sentence-level• Document-level


8

Introduction

Open problems:


– Which are the best features?• Linguistic features have been explored: but not

much on discourse features!


9

Introduction

Open problems:


– Which are the best features?• Linguistic features have been explored: but not

much on discourse features!

– Which are the best quality labels?• Likert• HTER• BLEU-style


10

Target documents

Source documents

Introduction


11

Target documents

Feature extractor

Source documents

Introduction


12

Target documents

Features for QE

Feature extractor

Source documents

Introduction


13

Target documents

Features for QE

Feature extractor

Source documents

QE model training

Introduction


14

Target documents

Features for QE

Feature extractor

Source documents

Quality labels Likert HTER BLEU ...

QE model training

Introduction


15

Target documents

Features for QE

Feature extractor

QE modelSource documents


QE model training

Introduction


16

Target documents

Features for QE

Feature extractor

QE model

Predictions

Source documents


QE model training

Introduction


17

Target documents

Features for QE

Feature extractor

QE model

Predictions

Source documents


QE model training

Defining the ideal quality label for document-level prediction is a

challenge

Introduction


18

Agenda

Introduction


Related Work





Conclusion


19


QuEst (www.quest.dcs.shef.ac.uk)

– Framework for sentence-level QE


20




– QuEst++ → recent extension for word and document levels • https://github.com/ghpaetzold/questplusplus


21





– Feature Extraction module (Java)


22





– Feature Extraction module (Java)

– Machine Learning module (Python)


23



24

Agenda

Introduction


Related Work





Conclusion


25

Related Work

Soricut and Echihabi (2010) → TrustRank– Ranking documents according to BLEU scores


26

Related Work


Scarton and Specia (2014)– Document-level QE prediction using discourse features – also

predicted BLEU scores


27

Related Work




Carpuat and Simard (2012)– Lexical consistency study of MT outputs → MT is overall consistent!


28

Related Work





Meyer and Weber (2013)– Implicit discourse connectives in MT


29

Related Work






Li et al. (2014)– Discourse connectives → improve MT → correlations between

discourse connectives and HTER


30

Related Work






Li et al. (2014)– Discourse connectives → improve MT → correlations between

discourse connectives and HTER

Guzmán et al. (2014)– Document-level evaluation metric using RST


31

Agenda

Introduction


Related Work





Conclusion


32


Quality labels are a challenge:

– Which is the ideal quality label for document-level QE?

Document-level QE


33




– How can we assess documents?

Document-level QE


34





• Sentence-level scores aggregation?

Document-level QE


35





• Sentence-level scores aggregation?

• New assessment score of the document as a whole?

Document-level QE


36



– BLEU-style metrics as quality labels• LIG corpus (FR-EN) → 119 documents

Document-level QE


37



– BLEU-style metrics as quality labels• WMT corpus (EN-DE) → 52 documents (1215

paragraphs)

– Low STDEV → documents have similar quality• Is it really true?

Document-level QE


38




paragraphs)


Document-level QE


39




paragraphs)


Document-level QE


40

Two-stage post-edition method

PE1:

– Post-edition of sentences without context

Document-level QE


41


PE1:

– Post-edition of sentences without context• Wir brauchen das kulturelle Fundament, aber wir haben jetzt

mehr Schriftsteller als Leser.

Document-level QE


42


PE1:



PE2:

– Post-edition of sentence with context

Document-level QE


43


PE1:



PE2:

– Post-edition of sentence with context• - St. Petersburg bietet nicht viel kulturelles Angebot, Moskau hat

viel mehr Kultur, es hat eine Grundlage. Es ist schwer fr die Kunst, sich in unserem Umfeld durchzusetzen. Wir brauchen das kulturelle Fundament, aber wir haben jetzt mehr Schriftsteller als Leser. Das ist falsch. In Europa gibt es viele neugierige Menschen, die auf Kunstausstellungen, Konzerte gehen. Hier ist diese Schicht ist dünn.

Document-level QE


44


Hypothesis:

– There are problems in MT outputs that can only be solved in context

Document-level QE


45


Hypothesis:


– Measuring the difference from PE1 to PE2

Document-level QE


46


Hypothesis:



• Isolating document-level problems

Document-level QE


47


Hypothesis:




• Using the difference to create a better quality label

Document-level QE


48


Hypothesis:




• Using the difference to create a better quality label

Document-level QE


49


Experiments:

– Data: 1215 paragraphs → WMT EN-DE corpus

Document-level QE


50


Experiments:

– Data: 1215 paragraphs → WMT EN-DE corpus • Filter 1: only paragraphs with more than 3

sentences (less than 8)

Document-level QE


51


Experiments:


sentences (less than 8)• Filter 2: Paragraphs ordered by number of

discourse phenomena (discourse connectives and pronouns)

Document-level QE


52


Experiments:




• Final data: 200 paragraphs

Document-level QE


53


Experiments:





– Annotators → students of “translation studies” in Saarland University, Saarbrücken, Germany

Document-level QE


54


Experiments:





– Annotators → students of “translation studies” in Saarland University, Saarbrücken, Germany

– 16 sets → evaluate agreement

Document-level QE


55


Annotator's agreement:

Document-level QE


56



Document-level QE


57



Document-level QE


58



Document-level QE


59



Document-level QE


60



Document-level QE


61



Document-level QE


62



Document-level QE


63


Changes from PE1 to PE2 – paragraphs perspective:

Document-level QE


64


Changes from PE1 to PE2 – paragraphs perspective:

Document-level QE

All paragraphswere changed


65


Paragraph changes example:

Document-level QE


66



Document-level QE


67



Document-level QE

Better wordchoices


68


Paragraph changes → manual analysis

Document-level QE


69


Paragraph changes → manual analysis– Discourse/context changes

Document-level QE


70


Paragraph changes → manual analysis– Discourse/context changes– Stylistic changes

Document-level QE


71


Paragraph changes → manual analysis– Discourse/context changes– Stylistic changes– Other changes

Document-level QE


72



Document-level QE


73



– Low agreement• Annotators should not made lots of stylistic

changes!

Document-level QE


74


Final results:

Document-level QE


75


Final results:

– 116 paragraphs analysed

Document-level QE


76


Final results:


– Some changes → only with paragraph context

Document-level QE


77


Final results:


– Some changes → only with paragraph context

– However

• How to combine the results into a quality label?

Document-level QE


78

Agenda

Introduction


Related Work





Conclusion


79


Extending the research:


80



– Data: ~ 1000 data points • Different language pairs• Entire documents (?)


81




– Annotators: expert annotators (familiar with post-editing)• Improving guidelines and training


82





– Evaluation: combining PE2 – PE1 with other metrics (HTER, BLEU, …)


83





– Evaluation: combining PE2 – PE1 with other metrics (HTER, BLEU, …)

– Alternative approach: • Post-editions in contexts → available• Apply PE1 (post-editing the sentences again →

without context)• PE2 – PE1 as usual


84

Agenda

Introduction


Related Work





Conclusion


85

Conclusion

Two-stage post-edition method → promising!

– Problems that can only be solved in context


86

Conclusion



How to compute a quality label?


87

Conclusion



How to compute a quality label?

– Combine PE2-PE1 with other metrics?

– Use PE2-PE1?


88

Acknowledgement

Saarland University: Marcos Zampieri, Mihaela Vela, Heike Przybyl and Josef Van Genabith

Reviewers from EXPERT Workshop

Thank you!

Carolina Scarton [email protected]

Supervisor: Dr Lucia Specia


esr7 carolina scarton - expert summer school - malaga 2015

Data & Analytics