avenue architecture

Download Avenue Architecture

Post on 02-Jan-2016




0 download

Embed Size (px)


Elicitation. Morphology. Rule Learning. Run-Time System. Rule Refinement. Translation Correction Tool. Word-Aligned Parallel Corpus. Learning Module. INPUT TEXT. Run Time Transfer System. Learning Module. Learned Transfer Rules. Rule Refinement Module. Elicitation Corpus. - PowerPoint PPT Presentation


  • Avenue Architecture

  • Interactive and Automatic Refinement of translation RulesProblem: Improve Machine Translation Quality.

    Proposed Solution: Put bilingual speakers back into the loop; use their corrections to detect the source of the error and automatically improve the lexicon and the grammar.

    Approach: Automate post-editing efforts by feeding them back into the MT system.Automatic refinement of translation rules that caused an error beyond post-editing.

    Goal: Improve MT coverage and overall quality.

  • Technical ChallengesElicit minimal MT information from non-expert usersAutomatic Evaluation of Refinement process

  • Error Typology for Automatic Rule Refinement (simplified)Missing wordExtra wordWrong word order

    Incorrect word

    Wrong agreementInteractive elicitation of error information

  • TCTool (Demo) Add a word Delete a word Modify a word Change word orderActions:Interactive elicitation of error information

    precisionrecallerror detection90%89%error classification72%71%

  • Types of Refinement Operations1. Refine a translation rule:R0 R1 (change R0 to make it more specific or more general)

    Automatic Rule AdaptationR0:R1:a nice houseuna casa bonitoa nice houseuna casa bonitaN gender = ADJ gender

  • Types of Refinement Operations2. Bifurcate a translation rule:R0 R0 (same, general rule) R1 (add a new more specific rule)

    Automatic Rule AdaptationR0:R1:a nice houseuna casa bonitaa great artistun gran artistaADJ type: pre-nominal

  • Error Information Elicitation Refinement Operation Typology Automatic Rule AdaptationChange word orderSL: Gaud was a great artist

    MT system output:TL: Gaud era un artista grande

    Ucorrection: *Gaud era un artista grande Gaud era un gran artistaA concrete exampleclue worderrorcorrection

  • Finding Triggering Feature(s): (error word, corrected word) =

    need to postulate a new binary feature: feat1

    Blame assignment (from MT system output)

    tree: Automatic Rule AdaptationS,1NP,1NP,8GrammarADJ::ADJ |: [great] -> [grande]((X1::Y1)((x0 form) = great)((y0 agr num) = sg)((y0 agr gen) = masc))ADJ::ADJ |: [great] -> [gran]((X1::Y1)((x0 form) = great)((y0 agr num) = sg)((y0 agr gen) = masc))

  • Refining RulesBifurcate NP,8 NP,8 (R0) + NP,8 (R1) (flip order of ADJ-N){NP,8} NP::NP : [DET ADJ N] -> [DET ADJ N]( (X1::Y1) (X2::Y2) (X3::Y3) ((x0 def) = (x1 def)) (x0 = x3) ((y1 agr) = (y3 agr)) ; det-noun agreement ((y2 agr) = (y3 agr)) ; adj-noun agreement (y2 = x3) ((y2 feat1) =c + ))Automatic Rule Adaptation

  • Refining Lexical EntriesADJ::ADJ |: [great] -> [grande]((X1::Y1)((x0 form) = great)((y0 agr num) = sg)((y0 agr gen) = masc)((y0 feat1) = -))

    ADJ::ADJ |: [great] -> [gran]((X1::Y1)((x0 form) = great)((y0 agr num) = sg)((y0 agr gen) = masc)((y0 feat1) = +))Automatic Rule Adaptation

  • Evaluating ImprovementAutomatic Rule AdaptationGiven the initial and final Translation Lattices, the Rule Refinement module needs to take into account, whether the following are present:Corrected Translation SentenceOriginal Translation Sentence (labelled as incorrect by the user)

    un artista granun gran artista un grande artista*un artista grande

  • Evaluating ImprovementAutomatic Rule AdaptationGiven the initial and final Translation Lattices, the Rule Refinement module needs to take into account, whether the following are present:Corrected Translation SentenceOriginal Translation Sentence (labelled as incorrect by the user)

    *un artista granun gran artista *un grande artista*un artista grande

  • Challenges and future workCredit and Blame assignment from TCTool Log Files and Xfer engines trace

    Order of corrections matters ~ explore rule interactions

    Explore the space between batch mode and fully interactive system

    Online TCTool always running to collect corrections from bilingual speakers make it into a game with rewards for the best users

  • PublicationsFont Llitjs, A., J.G. Carbonell and A. Lavie. "A Framework for Interactive and Automatic Refinement of Transfer-based Machine Translation" EAMT 10th Annual Conference 30-31 May 2005, Budapest, Hungary.

    Font Llitjs, A., R. Aranovich and L. Levin. "Building Machine translation systems for indigenous languages". Second Conference on the Indigenous Languages of Latin America (CILLA II), 27-29 October 2005, Texas, USA.

    Font Llitjs, A., K. Probst and J.G. Carbonell . "Error Analysis of Two Types of Grammar for the Purpose of Automatic Rule Refinement". AMTA, 2004, Washington, USA.

    Font Llitjs, A. and J.G. Carbonell . "The Translation Correction Tool: English-Spanish user studies. LREC, 2004. Lisbon, Portugal.

  • QuechuaSpanish MTV-Unit: funded Summer project in Cusco (Peru) June-August 2005 [preparations and data collection started earlier]

    Intensive Quechua course in Centro Bartolome de las Casas (CBC)

    Worked together with two Quechua native and one non-native speakers on developing infrastructure (correcting elicited translations, segmenting and translating list of most frequent words)

  • Quechua Spanish prototype MT system Stem Lexicon (semi-automatically generated): 753 lexical entries Suffix lexicon: 21 suffixes (150 Cusihuaman)Quechua morphology analyzer25 translation rulesSpanish morphology generation module

    User-Studies: 10 sentences, 3 users (2 native, 1 non-native)

    Base: crucial component, without which it makes no sense to try to automatically learn Refinement operations. Minimal in this context refers to minimal post-editing, which usually means to make the least amount of changes possible for producing an understandable working document, rather than producing a high quality document.However, unlike Allens (2003) definition of MPE, I do not mean MPE in the sense of producing MT output of less quality, but rather of having bilingual users perform the least amount of changes to the translation so that the same rule that originally generated the SL sentence can be refined. Because my goal is to elicit rather technical information from non-expert users, I need to find a way to classify errors that is not too hard for naive users to do accurately, and at the same time is useful for automatic RR.2. (minimal description length) MDL = parsimony so that grammar size doesn't increase unnecessarily (blow up) with lots of local refinements Part of the challenge of this second aspect of my thesis is to refine both manual and automatically learned translation rulesIn order to be able to improve translation quality and coverage automatically,We need to devise a method evaluate the refined system output also automatically so that evaluation metrics results can drive the refinement process.

    After looking at several hundred sentences (eng2spa) I organized the different types of MT errors into a typology,keeping in mind the ultimate goal of automatic rule refinement.

    Need to Find appropriate level of granularity for MT error classification

    To address the types of MT errors identified (described in previous page), we built an online GUI which allows non-expert bilingual users to reliably detect and minimally correct MT errors by doing one of the following correcting actions:(missing a word ) Adding a word(Extra word ) deleting a word, etc.(incorrect word + wrong agreement ) Modify a word(wrong word order ) change word order for edit-word window, just say: asks user to choose among several non-technical statements (as opposed to asking them to classify between linguistically motivated classes)

    Given:SL sentence (e.g. I see them)TL sentence (e.g. Yo veo los)word-to-word alignments (I-yo, see-veo, them-los)(context)

    There are 2 main refinement operations that can be applied both to grammar rules and lexical entries, with some minor differences.Refine: change existing rule, replace old (incorrect) rule with the new corrected rule

    Sometimes a rule is mostly correct, but is missing an agreement constraint: Example on this slide

    Automatically learned grammars some times over-generalize, and such cases the rules have an extra agreement constraint, which makes the rule incorrect.

    Results in higher precision and a tighter grammar reducing size of candidate listThe second type of operation Ill like to mention is bifurcate, which adds a copy of the general rule and makes it more specific rule without removing the original (more general) ruleRop1: leave original R0 rule as is, modify the duplicate, R1, This example shows the refinement operation required to also cover the pre-nominal order of ADJ in Spanish (exception to the general rule).

    You can find a more comprehensive discussion of possible types of refinement operations in the proposal document. Unfortunately I cannot cover them all in my talk.

    In the document I give a more comprehensive discussion of

    When users add or delete a word, the safest Rop type is bifurcate, until we have evidence that the original rule can never apply ( interactive mode)

    High precision, but increases size of candidate listCoverage in terms of TL patterns: every time R0 is modified ->R0, the coverage on the TL side increases

    Error detection: 90% precision 89% recallError classification (9 classes): 72% p 71% rEmpty delta set = feature language is not expressive enough to distinguish between gran and grande, need to postulate a new feature

    Blame assignment = what rules need to be refined (from parse output by xfer engine): This is the MT output tracing what


View more >