100 years of progress and innovation © 2011 ibm corporation the value of post editing - ibm case...

100 years of progress and innovation © 2011 IBM Corporation

The value of Post Editing - IBM Case Study

Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta, Álex Martínez Corrià, Salim Roukos, Helena Chapman, Saroj K. Vohra

June 2011

© 2011 IBM Corporation2

IBM Case Study – MT Post Editing

Introduction

MT Innovation

Process Overview

Findings

Conclusion / Recommendations


IBM World Wide Translation Operations

24 Centers World Wide~115 Translation Suppliers

Process ~2.8 B WordsTranslate ~0.4 B Words

~60 language pairs

One Stop Shop for all Translation Services

Marketing Material

Web

Product IntegratedInformation

Publications

Legal/Safety/Contracts

Machine Translation

Multimedia

FrancizationCultural Consultancy Centralized DTP

Overall End to End

ProcessManagement


IBM Professional Translation Services

0

5 0

1 0 0

1 5 0

2 0 0

2 5 0

2 0 0 1 2 0 0 2 2 0 0 3 2 0 0 4 2 0 0 5 2 0 0 6 2 0 0 7 2 0 0 8 2 0 0 9

Professional Memory

72% 85% Re-Use

Unit Cost

>50% Reduction

2001 2002 2003 2004 2005 2006 2007 2008 2009

TraditionalTechnology

ProcessMgmt

Human Skill

Consistent Quality Standards

Global Brand Identity

Professional Quality Standards

1

2

3

Future:– Ability to reduce cost using conventional methods reaching limits– Business pressure for additional cost elimination– Looking to MT Technology as next wave to reach business goals


- MT portal- Generic crowdsourcing - Text translation services

June 2008

Historical Perspective

2006

2007

2008

2009

2011

2012

2010

RTTS introduced in 2006as platform for speech and text translation, developed

by IBM Research

2010 MT piloting Pilot: SPA, ITA, FRE, GER-------------------------------------

New E2E processPartnership: WWTO/n.Fluent

8.6 M words

2011 MT Training Pilot: GER, BPR, JPN, CHS-------------------------------------MT payment profiles ready

16.0 M words target

eSupport (www)“Translate This Page”

JPN pilot /rule engine

Statistic

al MT Engines

Rule Based MT Engines

n.Fluent customized withWWTO translation memories

eSupport“Translate This Page”

switch to n.Fluent

Hybrid M

T Engines

RTTS licensed to IBM partners

Initial n.Fluent/WWTO Spanish MT pilot

-------------------------------------Improve efficiency of

professional translators


MT Critical Success Metrics

Necessary and sufficient condition to measure success – 5.0 M words sampled– Minimum of 3 languages– Net Contribution to ROI by MT Engine:

10% of payable words should be MT– No more than 5% adverse impact to Overall Quality Index– No more than 5% impact to Customer Satisfaction

Lack of industry metrics and guidance. – Active research on MT technology... no guidance on operational impacts– A business vacuum existed on how to integrate MT services– No operational process had been defined for MT services


IBM’s Watson Q&A computer

Google’s autonomous car

Technologies to understand and produce natural human speech

Instantaneous, high-quality machine translation

Smartphones / App phones in the developing world

*Andrew McAfee is a principal research scientist in the MIT Sloan School of Business

Recent Digital Innovations with Biggest Impact in the Business World*


Real-Time Translation Server (RTTS) & n.Fluent

Real Time Translation Server (RTTS) IBMs MT Engine RTTS provides machine translation for n.Fluent & other applications APIs allow other applications to access these translation services. Customization tools – Domains, chat-specific models, … Commercially licensed to IBM partners

Language Pairs to/from English:

n.Fluent IBMs MT translation application Providing machine translation services for:

Text, web pages, and documents (Word, Excel, …) Instant Messaging chats (via IM plug-in) Mobile translation application (BlackBerry and others)

Enabled with LEARNING via crowdsourcing (internal 450K IBMers) Deployed for eSupport self serving tech support (external)

العربية

中文

Deutsch

English

Français

Italiano

日本語

PortuguêsРусский

Español한국어

•0

•0.05

•0.1

•0.15

•0.2

•0.25

•0.3

•0.35

•0.4

•0.45

•0.5•BLEU

Qu

alit

y

Base 29k 180k 350k Words

IT HELP DESK


- MT portal- Generic crowdsourcing - Text translation services

June 2008

Historical Perspective

2006

2007

2008

2009

2011

2012

2010

RTTS introduced in 2006as platform for speech and text translation, developed

by IBM Research

2010 MT piloting Pilot: SPA, ITA, FRE, GER-------------------------------------

New E2E processPartnership: WWTO/n.Fluent

8.6 M words

2011 MT Training Pilot: GER, BPR, JPN, CHS-------------------------------------MT payment profiles ready

16.0 M words target

eSupport (www)“Translate This Page”

JPN pilot /rule engine

Statistic

al MT Engines

Rule Based MT Engines

n.Fluent customized withWWTO translation memories

eSupport“Translate This Page”

switch to n.Fluent

Hybrid M

T Engines

RTTS licensed to IBM partners

Initial n.Fluent/WWTO Spanish MT pilot

-------------------------------------Improve efficiency of

professional translators


TM

MT

New /

Changed

100%Exact Match

MT Pre-Process

Editing Session

MT Post Editing End to End Workflow

Upfront & on-going MT tuning via IBM TM professional translations– Professional translation = Best context

Matching methods– Traditional TM – breaks down content @ segment level– Machine TM – breaks down segments @ block level using MT models

– reconstructs segments preserving formats/mark-up tags

MT service level integration

TM Pre-Process

Shipment

EnglishTM

MatchAnalysis

CAT Translation

1.Show best choice

vs vs

2.Select best choice(Post Edit rules)

3. Commit language

TESTING

QUALITYMT

Model &

Trans.

= Localization Kit (NLV Folder)


IBM ConfidentialApril 18, 2023

MT Pre-processing

TM

New /Changed

100%Exact Match

Build dynamic,domain specific

MT model

MT

MTinitial corpus

General parallel training

corpus

Domain specificparallel training

corpus

ALL segment“no match segments”

Translation ofno match segments

Initial MT corpus– done before start of project

Lo

calization

kit

TM

MT

New /

Changed

100%Exact Match


IBM ConfidentialApril 18, 2023

Xxx xxx xx xxx xxx xxx. La aplicación desprotege los archivos antes de exportarlos. Yy yyy yyy

TM Editing Environment

TM EnvironmentXxx xxx xx xxx xxx xxx. The application unprotects files before exporting them. Yy yyy yyy

Translation Memory0 - The application unprotects files before exporting them.1[m] – La aplicación desprotege archivos antes de exportarlos.2[f 85%] - La aplicación protege los archivos antes de exportarlos

TM Environment

[Ctrl + 1]

Typed

Translator optionsIgnore fuzzy and MTPost edit MTPost edit fuzzy

Two Seconds Rule:Translators are trained on several strategies to make a quick choiceTMMT


Productivity Measurements

Start segment– Choose action

End segment

MT productivity evaluation log (MTeval Log)– N events– Words | Time | Existing Proposal | Used Proposal | ...

Examine productivity per payment category– SUM(Words) / SUM(Time) – Use of IBM Business Analytic Tool (SPSS)– Trim events that fall into 5% (slowest) and 95% (fastest) percentile

1. accept match [~0 time]

2. edit match [X time]

3. reject match [manual translation]

Eac

h e

ven

t

EM : ExactRM : ReplaceFM : FuzzyMT : MachineNP : No Proposal

A) = “best” Existing ProposalB) = “alternative” Existing ProposalC) = reject all Existing Proposal, 100% human labor


Total # events : 2,309 (377+1,932)

Total words: 24,150 Total time: 27,362 – 3,911 w/ MT match 11,377 w/ MT match– 20,239 w/o MT match 15,985 w/o MT match

MT impact to productivity – MT : 0.44 words/sec [1777 words / 4071 sec]– NP

• 0.21 w/ MT match• 0.32 w/o MT match Baseline (placebo)

MT Leverage : 71.8% [1777 / (1777+697)]

Single Shipment EXAMPLE

SEGMENTID WORDS TIME Prod_W_T SEGMENTID WORDS TIME Prod_W_T

Count Sum Sum Median Count Sum Sum Median

1-EM 0 . . . 1350 10593 3022 2.00

2-RM 4 18 43 .42 239 3905 3085 1.50

3-FM 129 1419 3870 .46 334 5610 9466 .71

5-MT 111 1777 4071 .50 0 . . .

6-NP 133 697 3393 .20 9 131 412 .33

Total 377 3911 11377 .37 1932 20239 15985 1.67

MT NO MT

Used MT

rate(MT) / rate(NP): 1.37

i.e. Translator can complete 37% more words in the same time. K

ey m

etri

cs


MT Impact on Fuzzy Match : 4Q10 Findings

When FM & MT matches exist simultaneously

Productivity: rate(MT) / rate(NP): a. Case : Translator edits FMb. FM-MT Combined casec. Case: Translator edits MT

** Findings subject to change with additional sampling.

Overall – Machine matches not as

good as professional (fuzzy) matches

– No statistical impact to fuzzy productivity to include MT matches. • SPA highest sample

case

28.6% 4.4% 57.6% 46.9%FM-MT Pick Rate:

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

FRE GER ITA SPA

Pro

du

ctiv

ity

rat

ioFMFM-MTMT


MT Key Metrics: 4Q10 Findings

8.6 M words sampled in real time translation service.

SPA : Qualified MT engine 4Q10

ITA : Qualified MT engine 4Q10

FRA : Qualified MT engine 1Q11• While rate(MT) / rate(NP) is high, the findings were not statistically significant in 4Q.

GER : Insufficient productivity from MT engine

# EventsWords

New/ChangedMT

(% of NP)MT

Leverage

FRE 20417 209347 2.87 68.9%

GER 36634 250238 1.32 5.4%

ITA 78483 715557 2.70 46.2%

SPA 783238 7424298 1.74 55.2%

Total 918772 8599440



Overall Savings Assessment

Overall savings %– Word savings due to MT efficiency

• Convert time savings MT payment factor % – MT payment factor X [MT % words + NP % words]

• Results in less payable words.

MT productivity savings drives a overall savings– These are not the same due to MT % distribution.

Supply chain has to consider cost of MT services

% EM % FM %MT %NPOverallSavings

FRE 47.8% 35.5% 10.6% 6.0% 11.1%

GER 55.9% 29.9% 0.5% 13.6% 2.4%

ITA 19.1% 29.4% 20.4% 31.1% 12.4%

SPA 39.6% 40.1% 9.5% 10.8% 9.0%



Pay for MT Words Translated not MT Matches

We pay for final results (MT payable words) not MT matches– MT matches considered “opinion” until chosen by a human– Too many opinions & opinions by immature MT models are less efficient.

Actual MT payable words have value beyond the specific project– Post Edited words are reused in future and unknown MT context

Engine has to deliver consistent MT payable words – Minimum needed to quality an MT engine for compensation

• High MT productivity [rate(MT) / rate(NP)]• High MT leverage [% of MT matches used]

– Compensation to be based on MT payment factor


Variance across Languages

There is no single maturity path when modeling MT engines across many languages.

IBM Pilot: each trained MT engine is a unique asset.– Some languages require more modeling/tuning than others.– Language pairs that service “Loose -> Structured” languages are struggling

• German requires more effort than Spanish

Are there limitations to statistical MT engines?– New thinking may need to be explored?

Each MT engine will have separate MT payment factors.


Perspective of MT Post Edit Pilots

Translation Service Hierarchy Professional Translation Services(Professional LSP)

Community Translation Services(Controlled Social Crowd)

Volunteer Translation Services(General Crowds)

Free Services(Individual)

Qu

ality / Reliab

ility

LOWER

HIGHER

General

DomainSpecific

internal IBM

All IBMexternal/internal

Pubs / UI

external(2011 Pilots)

internal IBMn.Fluent

“machine”

WWTO“human”

New

Mem

ory A

ssets

MT Post Editing has impacts across entire Translation Service Hierarchy


1. Professional (Human) memories are the best assets and deliver the highest quality.

2. Professional memories are a key asset for MT success.

3. All Memory assets need to be protected and managed.

4. Flow of memories between Professional and Machine must be properly balanced.

5. Dynamic modeling offers significant advantage over static modeling.

6. Continuous business analytics is needed to optimize machine assets.

7. A single cost model per language is needed, independent of MT services/engines.

8. An aggressive yet cautious approach is warranted to go forward.

MT Post Editing Project – Key Lessons

MT Post Editing does improve productivity and efficiency of a localization supply chain.

100 years of progress and innovation © 2011 ibm corporation the value of post editing - ibm case...

Documents

mt services

mt technology

ibm research

mt piloting pilot

mt training pilot

translation suppliers

ibm partners initial

fluentwwto spanish mt