a case study on multi-modal machine translation · 2017-11-30 · a case study on multi-modal...

Translating using Images A case study on multi-modal machine translation Industry Challenge: eBay users create product listings in many different languages • Machine Translation of product titles is essential • Can images help MT models translate product titles better, while being flexible enough to exploit large, text only MT corpora and deliver state of the art (SOTA) translation quality? Our Solution: We built a neural MT model to translate a source sentence by scanning around the source words AND specific parts of an image • Two independent attention mechanisms in one decoder • Spatial (convolutional) image features • Build on SOTA image processing pipelines The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme (Grant 13/RC/2106) and is co-funded under the European Regional Development Fund. Dr Iacer Calixto, Dr Atul Nautiyal, Ahmed Abdelkader and Prof Andy Way To learn more about innovative ADAPT technologies, contact: [email protected] • Neural machine translation (NMT) models that incorporate images • Flexibility to exploit large text-only machine translation (MT) corpora • Fine-tune models on in-domain multi-modal (MM) data Industry Benefits Exploit readily usable MM data Deliver better translations using image information Obtain expertise on MM language processing Stay ahead of competition, improve user experience Increased user engagement Our models can be directly used to allow users to engage in different languages seamlessly Use Cases Multi-modal data exploitation Use and connect additional multi-modal data together to build better MT models Co-Developed with:

Upload: others

Post on 17-Jun-2020

3 views

Category:

Documents

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: A case study on multi-modal machine translation · 2017-11-30 · A case study on multi-modal machine translation Industry Challenge: eBay users create product listings in many different

Translating using ImagesA case study on multi-modal machine translation

Industry Challenge: eBay users create product listings in many different languages

• Machine Translation of product titles is essential• Can images help MT models translate product titles better, while

being flexible enough to exploit large, text only MT corpora and deliver state of the art (SOTA) translation quality?

Our Solution:We built a neural MT model to translate a source sentence by scanning around the source words AND specific parts of an image

• Two independent attention mechanisms in one decoder• Spatial (convolutional) image features• Build on SOTA image processing pipelines

The ADAPT Centre for Digital Content Technology is funded under the SFI Research Centres Programme

(Grant 13/RC/2106) and is co-funded under the European Regional Development Fund.

Dr Iacer Calixto, Dr Atul Nautiyal, Ahmed Abdelkader and Prof Andy WayTo learn more about innovative ADAPT technologies, contact: [email protected]

• Neural machine translation (NMT) models that incorporate images• Flexibility to exploit large text-only machine translation (MT) corpora • Fine-tune models on in-domain multi-modal (MM) data

Industry Benefits

Exploit readily usable MM dataDeliver better translations using

image informationObtain expertise on MM language

processingStay ahead of competition, improve

user experience

Increased user engagementOur models can be directly used to allow users to engage in different

languages seamlessly

Use Cases

Multi-modal data exploitationUse and connect additional multi-modal data together to build better MT

models

Co-Developed with:

Machine translation vs human translation