how to evaluate mt quality based on effectiveness (adam lamontagne, language technology dev. &...

Evaluating (MT) quality based on effectiveness

Adam LaMontagne (Language Technology Dev. & Deployment Manager, Moravia)

TAUS Roundtable, 15 March 2016, Vienna

Agenda

Quality Usability Effectiveness An Interconnected System Beyond Individual Quality Measures Discussion



What is Quality?


Measuring QualityAutomated

BLEU F-Measure METEOR Levenshtein NIST ROUGE TER(p) WER

Human


What is Usability?

• Effectiveness - can users complete tasks, achieve goals with the product, i.e. do what they want to do?

• Efficiency - how much effort do users require to do this? (Often measured in time)

• Satisfaction – what do users think about the products ease of use?

Did the user achieve their goal?

Credit: http://www.usabilitynet.org/

Measuring Usability

TAUS Usability Comprehension tests Questionnaires Participant observation Screen recording Think-Aloud Protocols Eye Tracking


What is Effectiveness?

Help the user to take action:• User documentation• Help content• Training material• User Interface

Help the user to make a decision:• Product/Service

descriptions

• Marketing content Help the user to

communicate:• Chat• Email• Social media

Help to protect the user:• Signage• Legal


What does the content really want?Different content types have different purposes:

Measuring Effectiveness

Active User Feedback Passive User Feedback Metadata (Indirect/abstracted measures of

effectiveness)




Active User Feedback User ratings User feedback & continuous

improvement



Example: Microsoft Collaborative Translation Framework (CTF)

Live examples from: https://support.microsoft.com/fr-fr/kb/274703CTF Overview: https://www.microsoft.com/en-us/translator/ctf.aspx



Passive User Feedback Web analytics : “User Engagement”

Clicks Click depth Duration Conversion Bounce rate Drop-off rate

Screenshot from Google Analytics



Metadata (Indirect/Abstracted effectiveness Metrics) SEO results Native-language support communication Call/help center costs Sales/revenue data by market

An Interconnected System

EffectivenessUsabilityQuality


Beyond Individual Quality Measures


Combining & correlating evaluation metrics

Quality Metrics Automated QA & scoring LQA DQF metrics Errors/error typology Automated MT metrics

Effectiveness Metrics User feedback Web analytics Metadata

Usability Metrics Comprehension tests Questionnaires Participant observation…

Discussion
