applying machine learning - abdessamad echihabi at sdl connect 16

46
Applying Machine Learning 11/17/2016 Samad Echihabi

Upload: sdl

Post on 27-Jan-2017

54 views

Category:

Software


1 download

TRANSCRIPT

Page 1: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Applying Machine Learning

11/17/2016

Samad Echihabi

Page 2: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

2

Machine Learning

Machine

data

model

input output

Page 3: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

3

Machine Learning

Machine

data

model

input output

Page 4: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

4

Machine Learning: French to English Translation

data

model

Méditerranée: 3200 personnes secourues en cinq jours

Mediterranean: 3200 people rescued in five days

Page 5: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

5

Machine Translation Models

Language Model

Translation Model

Page 6: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Source Target Good Translations?bonjour hellobonjour bluebonjour morningbonjour good morningbonjour hi

Source Target Good Translations?bonjour hello ✔bonjour blue ✗bonjour morning ~ bonjour good morning ✔bonjour hi ✔

MT: Translation Model

Page 7: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Target Good Language?Be the change that you wish to see in the world.Be the world that you wish to see in the change.The be change which you wish to see on the world. Be that the you world to in wish see change . the

Target Good Language?Be the change that you wish to see in the world. ✔

Be the world that you wish to see in the change. ✗The be change which you wish to see on the world. ✗✗Be that the you world to in wish see change . the ✗✗✗✗✗

MT: Language Model

Page 8: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

MT: Training

Statistical Analysis

Translation Model

la the 80%

la a 12%

la 8%

capitale capital 70%

capitale death 30%

de of 53%

de from 47%

france france 100%

Is est 75%

Is was 25%

paris paris 100%

Language Model

the death of 54%

the capital of 34%

a capital of 11%

capital of france 41%

capital from france 9%

of france is 45%

of the france 2%

france is paris 23%

france was paris 22%

………………english………

P(s/t) P(t)

parallelmonolingual

………………english………

………………french………

Statistical Analysis

Page 9: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

MT: Decoding

Statistical Search

Translation Score

the capital of france is paris 94%

capital of france is paris 71%

a capital of france is paris 65%

... …

a death from france was paris 3%

Translation Model

la the 80%

la a 12%

la 8%

capitale capital 70%

capitale death 30%

de of 53%

de from 47%

france france 100%

Is est 75%

Is was 25%

paris paris 100%

Language Model

the death of 54%

the capital of 34%

a capital of 11%

capital of france 41%

capital from france 9%

of france is 45%

of the france 2%

france is paris 23%

france was paris 22%

Input

la capitale de la france est paris

Page 10: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

SMT Models

Adaptive ModelsNeural Models

• Translation Model P(s/t)• Language Model P(t)• Distortion• Alignment• Phrase• POS• Syntactic Translation• Syntactic Language• Reordering• Lexicalized Reordering• Preordering• Word Deletion• Lexicalized Smoothing• Capitalization• Morphology• Transliteration• Semantic• Informal Models• Social Media Components

Page 11: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Applying Machine Learning – Use cases

Page 12: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Social Media Translation

Page 13: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Character Repetition

Page 14: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Spelling Errors

Page 15: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Dialect

Page 16: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Morphology

Page 17: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Romanization

Page 18: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Metadata

Page 19: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Social Media Translation Challengesأااااااحسن احسنالخ�لييييييج� لخليجا Normalization

نزيفه

نظيفةوظيفةنظيفنزيفنزيفه

Spelling Correction

# القدم_ كرة #soccerSocial Metadata

المرفهين Morphologicalين + مرفه + ال

Segmentation

bessa7a

wel3afya

habibi والعافيةحبيبي بالصحةhabibi Deromanizationبساحة

+62% Improvement

Page 20: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Source Generic MT

Social Media Translation

• la2a hia katir fi lakhbar. • ma 3ajbanish kida. Lazim t3'iyyer

l3ounouane • Enty habla ? • Kalemni lama t3raf ezay tebatal

teshtemni• 3andy soda3 fi rassi... 5oshy namy badal

chat. a7san lik Ah sa7

• La2a hia katir Fi lakhbar. • Ma 3ajbanish kida. lazim T3 (iyyer

L3ounouane • enty habla? • kalemni Lama T3RAF ezay tebatal

teshtemni • 3Andy soda3 Fi rassi ... 5oshy namy

badal Chat. A7San lik Ah SA7

Page 21: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Source Social Media MT

Social Media Translation

• la2a hia katir fi lakhbar. • ma 3ajbanish kida. Lazim t3'iyyer

l3ounouane • Enty habla ? • Kalemni lama t3raf ezay tebatal

teshtemni• 3andy soda3 fi rassi... 5oshy namy badal

chat. a7san lik Ah sa7

• No, it is very much in the news. • I don't like this. We must change the

title• Are you an idiot? • Talk to me when you know how to stop

insulting me• I have a headache in my head. Go to

sleep, instead of chat. It is better for you, Yes, sa7

Page 22: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Broadcast NewsTranslations

Page 23: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Broadcast News Translation

Speech Recognition

Machine Translation Distillation

Audio Channels

Video Channels

Actionable Information

Page 24: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Received Tuesday in Warsaw by Bronislaw Komorowski, Barack Obama has participated in ceremonies marking the twenty-fifth anniversary of the first democratic elections in Poland

Broadcast News Translation

Reçu mardi à Varsovie par Bronislaw Komorowski, Barack Obama a participé aux cérémonies marquant le vingt-cinquième anniversaire des premières élections démocratiques en Pologne.

Received Tuesday in Warsaw by Bronislaw Komorowski, Barack Obama has participated in ceremonies marking the twenty-fifth anniversary of the first democratic elections in Poland

Page 25: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Travel Reviews

Page 26: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Travel User Reviews Translation

published

Translated User Reviews

post-edited

good translationbad translation

Automatic Quality Prediction

Page 27: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 28: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Post-Editing Machine Translation

Page 29: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Post-Editing Machine Translation

Page 30: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Post-Editing Adaptive Machine Translation

Page 31: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Post-Editing Adaptive Machine Translation

Page 32: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 33: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 34: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 35: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 36: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 37: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 38: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 39: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 40: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 41: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 42: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 43: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16
Page 44: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Applying Machine Learning

Volume

Quality

Data Domain

Models

Delivery

Security Speed

Privacy

Evaluation

Integration

Adaptation

Page 45: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

ANSWERS&

QUESTIONS

Page 46: Applying Machine Learning - Abdessamad Echihabi at SDL Connect 16

Copyright © 2008-2017 SDL plc. All rights reserved. All company names, brand names, trademarks,

service marks, images and logos are the property of their respective owners.

This presentation and its content are SDL confidential unless otherwise specified, and may not be

copied, used or distributed except as authorised by SDL.

Software and Services for Human Understanding