PARADIGM: Paraphrase Diagnostics through Grammar Matching

Jonathan Weese and Juri GanitkevitchJohns Hopkins University

Chris Callison-BurchUniversity of Pennsylvania

Abstract

Paraphrase evaluation is typically done ei-ther manually or through indirect, task-based evaluation. We introduce an in-trinsic evaluation PARADIGM which mea-sures the goodness of paraphrase col-lections that are represented using syn-chronous grammars. We formulate twomeasures that evaluate these paraphrasegrammars using gold standard sententialparaphrases drawn from a monolingualparallel corpus. The first measure calcu-lates how often a paraphrase grammar isable to synchronously parse the sentencepairs in the corpus. The second mea-sure enumerates paraphrase rules from themonolingual parallel corpus and calculatesthe overlap between this reference para-phrase collection and the paraphrase re-source being evaluated. We demonstratethe use of these evaluation metrics on para-phrase collections derived from three dif-ferent data types: multiple translationsof classic French novels, comparable sen-tence pairs drawn from different newspa-pers, and bilingual parallel corpora. Weshow that PARADIGM correlates with hu-man judgments more strongly than BLEU

on a task-based evaluation of paraphrasequality.

1 Introduction

Paraphrases are useful in a wide range of natu-ral language processing applications. A varietyof data-driven approaches have been proposed togenerate paraphrase resources (see Madnani andDorr (2010) for a survey of these methods). Fewobjective metrics have been established to evalu-ate these resources. Instead, paraphrases are typi-cally evaluated using subjective manual evaluationor through task-based evaluations.

Different researchers have used different crite-ria for manual evaluations. For example, Barzilayand McKeown (2001) evaluated their paraphrasesby asking judges whether paraphrases were “ap-proximately conceptually equivalent.” Ibrahimet al. (2003) asked judges whether their para-phrases were “roughly interchangeable given thegenre.” Bannard and Callison-Burch (2005) re-placed phrases with paraphrases in a number ofsentences and asked judges whether the substitu-tions “preserved meaning and remained grammat-ical.” The results of these subjective evaluationsare not easily reusable.

Other researchers have evaluated their para-phrases through task-based evaluations. Lin andPantel (2001) measured their potential impact onquestion-answering. Cohn and Lapata (2007)evaluate their applicability in the text-to-text gen-eration task of sentence compression. Zhao et al.(2009) use them to perform sentence compressionand simplification and to compute sentence simi-larity. Several researchers have demonstrated thatparaphrases can improve machine translation eval-uation (c.f. Kauchak and Barzilay (2006), Zhouet al. (2006), Madnani (2010) and Snover et al.(2010)).

We introduce an automatic evaluation met-ric called PARADIGM, PARAphrase DIagnosticsthrough Grammar Matching. This metric eval-uates paraphrase collections that are representedusing synchronous grammars. Synchronous tree-adjoining grammars (STAGs), synchronous treesubstitution grammars (STSGs), and synchronouscontext free grammars (SCFGs) are popular for-malisms for representing paraphrase rules (Dras,1997; Cohn and Lapata, 2007; Madnani, 2010;Ganitkevitch et al., 2011). We present two mea-sures that evaluate these paraphrase grammars us-ing gold standard sentential paraphrases drawnfrom a monolingual parallel corpus, which havebeen previously proposed as a good resource

for paraphrase evaluation (Callison-Burch et al.,2008; Cohn et al., 2008).

The first of our two proposed metrics calculateshow often a paraphrase grammar is able to syn-chronously parse the sentence pairs in a test set.The second measure enumerates paraphrase rulesfrom a monolingual parallel corpus and calculatesthe overlap between this reference paraphrase col-lection, and the paraphrase resource being evalu-ated.

2 Related work and background

The most closely related work is ParaMetric(Callison-Burch et al., 2008), which is a set ofobjective measures for evaluating the quality ofphrase-based paraphrases. ParaMetric extracts aset of gold-standard phrasal paraphrases from sen-tential paraphrases that have been manually word-aligned. The sentential paraphrases used in Para-Metric were drawn from a data set originally cre-ated to evaluate machine translation output usingthe BLEU metric. Cohn et al. (2008) argue thatthese sorts of monolingual parallel corpora are ap-propriate for evaluating paraphrase systems, be-cause they are naturally occurring sources of para-phrases.

Callison-Burch et al. (2008) calculated threetypes of metrics in ParaMetric. The manual wordalignments were used to calculate how well anautomatic paraphrasing technique is able to alignthe paraphrases in a sentence pair. This measureis limited to a class of paraphrasing techniquesthat perform alignment (like MacCartney et al.(2008)). Most methods produce a list of para-phrases for a given input phrase. So Callison-Burch et al. (2008) calculate two more gener-ally applicable measures by comparing the para-phrases in an automatically extracted resource togold standard paraphrases extracted via the align-ments. These allow a lower-bound on precisionand relative recall to be calculated.

Liu et al. (2010) introduce the PEM metric as analternative to BLEU, since BLEU prefers iden-tical paraphrases. PEM uses a second languageas a pivot to judge semantic equivalence. This re-quires use of some bilingual data. Chen and Dolan(2011) suggest using BLEU together with theirmetric PINC, which uses n-grams to measure lex-ical difference between paraphrases.

PARADIGM extends the ideas in ParaMetricfrom lexical and phrasal paraphrasing techniques

to paraphrasing techniques that also generate syn-tactic templates, such as Zhao et al. (2008), Cohnand Lapata (2009), Madnani (2010) and Ganitke-vitch et al. (2011). Instead of extracting gold stan-dard paraphrases using techniques from phrase-based machine translation, we use grammar ex-traction techniques (Weese et al., 2011) to ex-tract gold standard paraphrase grammar rules fromParaMetric’s word-aligned sentential paraphrases.Using these rules, we calculate the overlap be-tween a gold standard paraphrase grammar and anautomatically generated paraphrase grammar.

Moreover, like ParaMetric, PARADIGM is ableto do further analysis on a restricted class of para-phrasing models. In this case, PARADIGM evalu-ates how well certain models are able to producesynchronous parses of sentence pairs drawn frommonolingual parallel corpora. PARADIGM’s dif-ferent metrics are explained in Section 4, but firstwe give background on synchronous parsing andsynchronous grammars.

2.1 Synchronous parsing with SCFGsSynchronous context-free grammarsAn SCFG (Lewis and Stearns, 1968; Aho andUllman, 1972) is similar to a context-free gram-mar, except that it generates pairs of stringsin correspondence. Each production rule in anSCFG rewrites a non-terminal symbol as a pair ofphrases, which may have contain a mix of wordsand non-terminals symbols. The grammar is syn-chronous because both phrases in the pair musthave an identical set of non-terminals (though theycan come in different orders), and correspondingnon-terminals must be rewritten using the samerule.

Much recent work in MT (and, by extension,paraphrasing approaches that use MT machinery)has been focused on choosing an appropriate set ofnon-terminal symbols. The Hiero model (Chiang,2007) used a single non-terminal symbolX . Otherapproaches have read symbols from constituentparses of the training data (Galley et al., 2004;Galley et al., 2006; Zollmann and Venugopal,2006). Labels based combinatory categorial gram-mar (Steedman and Baldridge, 2011) have alsobeen used (Almaghout et al., 2010; Weese et al.,2012).

Synchronous parsingWu (1997) introduced a parsing algorithm usinga variant of CKY. Dyer recently showed (2010)

and

him

impeach

towant

some

.down

step

tohim

expect

others

.

resign

to

him

want

others

while

,

him

impeach

to

propose

people

some

DT

NPVBP VB

PRP

CC NNS

VBP

PRP

VB PRT

VPVP

SVP

S

NPVP

VP

NP

SVP

S

S

.

Figure 1: PARADIGM extracts lexical, phrasal andsyntactic paraphrases from parsed, word-alignedsentence pairs.

that the average parse time can be significantly im-proved by using a two-pass algorithm.

The question of whether a source-reference pairis reachable under a model must be addressed inend-to-end discriminative training in MT (Lianget al., 2006a; Gimpel and Smith, 2012). Auli etal. (2009) showed that only approximately 30% oftraining pairs are reachable under a phrase-basedmodel. This result is confirmed by our results inparaphrasing.

3 Paraphrase grammar extraction

Like ParaMetric, PARADIGM extracts gold stan-dard paraphrases from word-aligned sententialparaphrases. PARADIGM goes further by parsingone of the two input sentences, and uses the parsetree to extract syntactic paraphrase rules, follow-ing recent advances in syntactic approaches to ma-chine translation (like Galley et al. (2004), Zoll-mann and Venugopal (2006), and others). Figure 1shows an example of a parsed sentence pair. Fromthat pair it is possible to extract a wide varietyof non-identical paraphrases, which include lexi-cal paraphrases (single word synonyms), phrasalparaphrases, and syntactic paraphrases that in-clude a mix of words and syntactic non-terminal

CC→ and whileVBP→ want proposeVBP→ expect want

DT→ some some peopleS→ him to step down him to resign

VP→ step down resignVP→ to step down to resignVP→ want to impeach him propose to impeach himVP→ want VP propose VPVP→ want to impeach PRP propose to impeach PRPVP→ VBP him to step down VBP him to resign

S→ PRP to step down PRP to resign

Figure 2: Four examples each of lexical, phrasal,and syntactic paraphrases that can be extractedfrom the sentence pair in Figure 1.

symbols. Figure 2 shows a set of four examplesfor each type that can be extracted from Figure 1.

These rules are formulated as SCFG rules,with a syntactic left-hand nonterminal symboland two English right-hand sides representing theparaphrase. The examples above include non-terminal symbols that represent whole syntac-tic constituents. It is also possible to createmore complex non-terminal symbols that describeCCG-like non-constituent phrases. For example,we could extract a rule like

S/VP→ <NNS want him to, NNS expect him to>

Using constituents only, we are able to ex-tract 45 paraphrase rules from Figure 1. AddingCCG-style slashed constituents yields 66 addi-tional rules.

4 PARADIGM: Evaluating paraphrasegrammars

By considering a paraphrase model as a syn-chronous context-free grammar, we propose tomeasure the model’s goodness using the followingcriteria:

1. What percentage of sentential paraphrasesare reachable under the model? That is, givena collection of sentence pairs (ai, bi) and anSCFG G, where each pair of a and b are sen-tential paraphrases, how many of the pairs arein the language of G? We evaluate this byproducing a synchronous parse for the pairs,as shown in Figure 3.

2. Given a collection of gold-standard para-phrase rules, how many of those paraphrasesexist as rules in G? To calculate this, welook at the overlap of grammars (described in

of the

twelve cartoons insulting mohammadCD NNS JJ

NP

NP

VP

NP

12 the islamic prophetCD NNS JJ

NP

NPVP

NP

cartoons offensivethat were to sparked riots

violent unrest was caused by

NP

NPVBD

VBD

VP

VP

S

S

Figure 3: We measure the goodness of paraphrasegrammars by determine how often they can beused to synchronously parse gold-standard sen-tential paraphrases. Note we do not require thesynchronous derivation to match a gold-standardparse tree.

Section 4.2 below), examining different cate-gories of rules and thresholding based on howfrequently the rule was used in the gold stan-dard data.

These criteria correspond to properties that wethink are desirable in paraphrase models. Theyalso have the advantage that they do not dependon human judgments and so can be calculated au-tomatically.

4.1 Synchronous parse coverage

Paraphrase grammars should be able to explainsentential paraphrases. For example, Figure3 shows a sentence pair that is synchronouslyparseable by one paraphrase grammar. In general,we say that the more such sentence pairs that aparaphrase grammar can synchronously parse, thebetter it is.

The synchronous derivation allows us to drawinferences about parts of the sentence pair that arein correspondence; for instance, in Figure 3, vi-olent unrest corresponds to riots and mohammadcorresponds to the islamic prophet.

4.2 Grammar overlap defined

We measure grammar overlap by comparing thesets of production rules for two different gram-mars. If the grammars contain rules that are equiv-alent, the equivalent rules are in the grammars’overlap.

We consider two types of overlapping, whichwe will call strict and non-strict overlap. For strictoverlap, we say that two rules are equivalent ifthey are identical, that is, if they have the same

left-hand side non-terminal symbol, their sourcesides are identical strings, and their target sides areidentical strings. (This includes identical indexingon non-terminal symbols on the right hand sidesof the rule.)

To calculate non-strict overlap, we ignore theidentities of non-terminal symbols in the left-handand right-hand sides of the rules. That is, two rulesare considered equivalent if they are identical afterall the non-terminal symbols have been replacedby one equivalent symbol.

For example, in non-strict overlap, the syntacticrule

NP → 〈N1 ’s N2; the N2 of N1〉

would match the Hiero rule

X → 〈X1 ’s X2; the X2 of X1〉

If we are considering two Hiero grammars,strict and non-strict intersection are the same op-eration since they only have on non-terminal X .

4.3 Precision lower bound and relative recallCallison-Burch et al. (2008) use the notion of over-lap between two paraphrase sets to define two met-rics, precision lower bound and relative recall.These are calculated the same way as standardprecision and recall. Relative recall is qualifiedas “relative” because it is calculated on a poten-tially incomplete set of gold standard paraphrases.There may exist valid paraphrases that do not oc-cur in that set. Similarly, only a lower bound onprecision can be calculated because the candidateset may contain valid paraphrases that do not oc-cur in the gold standard set.

5 Experiments

5.1 DataWe extracted paraphrase grammars from a vari-ety of different data sources, including four collec-tions of sentential paraphrases. These included:

• Multiple translation corpora that werecompiled by the Linguistics Data Consortium(LDC) for the purposes of evaluating ma-chine translation quality with the BLEU met-ric. We collected eight LDC corpora that allhave multiple English translations.1

1LDC Catalog numbers LDC2002T01, LDC2005T05,LDC2010T10, LDC2010T11, LDC2010T12, LDC2010T14,LDC2010T17, and LDC2010T23.

sentence totalCorpus pairs words

LDC Multiple Translations 83,284 2,254,707Classic French Literature 75,106 682,978MSR Paraphrase Corpus 5,801 219,492

ParaMetric 970 21,944

Table 1: Amount of English–English parallel data.LDC data has 4 parallel translations per sentence.Literature data is from Barzilay and McKeown(2001). MSR data is from Quirk et al. (2004)and Dolan et al. (2004). ParaMertic data is fromCallison-Burch et al. (2008).

• Classic French Literature that were trans-lated by different translators, and which werecompiled by Barzilay and McKeown (2001).

• The MSR Paraphrase corpus which con-sists of sentence pairs drawn from compara-ble news articles drawn from different websites in the same date rate. The sentence pairswere aligned heuristically aligned and thenmanually judged to be paraphrases.

• The ParaMetric data which consists of 900manually word-aligned sentence pairs col-lected by Cohn et al. (2008). 300 sentencepairs were drawn from each of the 3 abovesources. We use this to extract the gold stan-dard paraphrase grammar.

The size of the data from each source is summa-rized in Table 1.

For each dataset, after tokenizing and normaliz-ing, we parsed one sentence in each English pairusing the Berkeley constituency parser (Liang etal., 2006b). We then obtained word-level align-ments, either using GIZA++ (Och and Ney, 2000)or, in the case of ParaMetric, using human annota-tions.

We used the Thrax grammar extractor (Weeseet al., 2011) to extract Hiero-style and syntacticSCFGs from the paraphrase data. In the syntac-tic setting we allowed labeling of rules with ei-ther constituent labels or CCG-style slashed cat-egories. The size of the extracted grammars isshown in Table 2.

We also used version 0.2 of the SCFG-basedparaphrase collection known as the ParaPhraseDataBase or PPDB (Ganitkevitch et al., 2013).The PPDB paraphrases were extracted using thepivoting technique (Bannard and Callison-Burch,

Grammar RulesLDC Hiero 52,784,462Lit. Hiero 3,288,546

MSR Hiero 2,456,513ParaMetric Hiero 584,944

LDC Syntax 23,978,477Lit. Syntax 715,154

MSR Syntax 406,115ParaMetric Syntax 317,772PPDB-v0.2-small 1,292,224PPDB-v0.2-large 9,456,356

PPDB-v0.2-xl 46,592,161

Table 2: Size of various paraphrase grammars.

Grammar freq. ≥ 1 freq. ≥ 2ParaMetric Syntax 317,772 21,709

LDC Hiero 5,840 (1.8%) 416 (1.9%)Lit. Hiero 6,152 (1.9%) 359 (1.7%)

MSR Hiero 10,012 (3.2%) 315 (1.5%)LDC Syntax 48,833 (15.3%) 7,748 (35.6%)Lit. Syntax 14,431 (4.5%) 1,960 (9.0%)

MSR Syntax 21,197 (6.7%) 2,053 (9.5%)PPDB-v0.2-small 15,831 (5.0%) 5,673 (26.1%)PPDB-v0.2-large 31,277 (9.8%) 8,245 (37.9%)

PPDB-v0.2-xl 47,720 (15.0%) 10,049 (46.2%)

Table 3: Size of strict overlap (number of rules and% of the gold standard) of each grammar with asyntactic grammar derived from ParaMetric. freq.≥ 2 means we first removed all rules that ap-peared only once from the ParaMetric grammar.The number in parentheses shows the percentageof ParaMetric rules that are present in the overlap.

2005) on bilingual parallel corpora containingover 42 million sentence pairs.

The PPDB release includes a tool for pruningthe grammar to a smaller size by retaining onlyhigh-precision paraphrases. We include PPDBgrammars for several different pruning settings inour analysis.

5.2 Experimental setupWe calculated our two metrics for each of thegrammars listed in Table 2.

To perform synchronous parsing, we used theJoshua decoder (Post et al., 2013), which includesan implementation of Dyer’s two-pass parsing al-gorithm (2010). After splitting the LDC data into10 equal pieces, we trained paraphrase models onnine-tenths of the data and parsed the other tenth.

Grammars trained from other sources (the MSRcorpus, French literature domain, and PPDB) werealso evaluated on the held-out tenth of LDC data.

Grammar freq. ≥ 1 freq. ≥ 2ParaMetric Syntax 200,385 20,699

LDC Hiero 41,346 (20.6%) 5,323 (25.8%)Lit. Hiero 36,873 (18.4%) 4,606 (22.3%)

MSR Hiero 58,970 (29.4%) 6,741 (32.6%)LDC Syntax 37,231 (11.7%) 5,055 (24.5%)Lit. Syntax 19,530 (9.7%) 3,121 (15.1%)

MSR Syntax 28,016 (14.0%) 3,564 (17.2%)PPDB-v0.2-small 13,003 (6.5%) 3,661 (17.7%)PPDB-v0.2-large 22,431 (11.2%) 4,837 (23.4%)

PPDB-v0.2-xl 31,294 (15.6%) 5,590 (27.0%)

Table 4: Size of non-strict overlap of each gram-mar with the syntactic grammar derived fromParaMetric. The number in parentheses shows thepercentage of ParaMetric rules that are present inthe overlap.

Grammar syntactic phrasal lexicalParaMetric 238,646 73,320 5,806LDCSyn 36,375 (15%) 8,806 (12%) 3,652 (62%)MSRSyn 7,734 (3%) 11,254 (15%) 2,209 (38%)PPDB-xl 40,822 (17%) 3,765 (5%) 3,142 (54%)

Table 5: Number of paraphrases of each typein each grammar’s strict overlap with the syntac-tic ParaMetric grammar. Numbers in parenthesesshow the percentage of ParaMetric rules of eachtype.

Note that the LDC data contains 4 independenttranslations of each foreign sentence, giving 6 pos-sible (unordered) paraphrase pairs. We evaluatedcoverage in two ways (corresponding to the twocolumns in Table 6): first, considering all possiblesentence pairs from the test data, how many wereable to be parsed?

Secondly, if we consider all the English sen-tences that correspond to one foreign sentence,how many foreign sentences had at least one pairof English translations that could be parsed syn-chronously?

For grammar overlap, we perform both strictand non-strict calculations (see Section 4.2)against a syntactic grammar derived from hand-aligned ParaMetric data.

5.3 Grammar overlap results

In Table 5 we see a breakdown of the types of para-phrases in the overlap for three of the models. Al-though the PPDB-xl overlap is much larger thanthe other two, about 80% of its rules are syntac-tic transformations. The LDC and MSR modelshave a much larger proportion of phrasal and lexi-cal rules.

Next we will look at the grammar overlap num-

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

10 100 1k 10k 100k 0

0.04

0.08

0.12

0.16

Prec

isio

n Lo

wer

Bou

nd

Rel

ativ

e R

ecal

l

Number of rules

RecallPrec.

Figure 4: Precision lower bound and relative recallwhen overlapping different sizes of PPDB with thesyntactic ParaMetric grammar.

bers presented in Table 3 and Table 4.Note the non-intuitive result that for some

grammars (notably PPDB), the non-strict overlapis smaller than the strict overlap. This is becauserules with different non-terminals only count oncein the non-strict overlap; for example, in PPDB-small,

NN→〈 answer ; reply 〉VB→〈 answer ; reply 〉

count as separate entries when calculating strictly,but when ignoring non-terminals, they count asonly one type of rule.

The fact that the non-strict overlaps are smallermeans that there must be many rules in PPDB thatare identical except for non-terminal labels.

5.4 Precision and recall results

Figure 4 shows relative recall and precision lowerbound calculated for various sizes of PPDB rela-tive to the ParaMetric grammar. The x-axis rep-resents the size of the grammar as we vary fromkeeping only the most probable rules to includingless probable ones. Restricting to high probabilityrules makes the grammar much smaller, resultingin higher precision.

5.5 Synchronous parsing results

Table 6 shows the percentage of sentence pairs thatwere reachable in a held-out portion of the LDCmultiple-translation data.

We find that a grammar trained on LDC datavastly outperforms data from any other domain.This is not surprising — we shouldn’t expect amodel trained on French literature to be able to

Grammar % (all) % (any)LDC Hiero 9.5 33.0Lit. Hiero 1.8 9.6

MSR Hiero 1.7 9.2LDC Syntax 9.1 30.2Lit. Syntax 2.0 10.7

MSR Syntax 1.9 10.4PM Syntax 1.7 9.8

PPDB-v0.2-small 1.8 3.3PPDB-v0.2-large 2.5 4.5

PPDB-v0.2-xl 3.5 6.2

Table 6: Parse coverage on held-out LDC data.The all column considers every possible sententialparaphrase in the test set. The any column consid-ers a sentence parsed if any of its paraphrases wasable to parsed.

handle some of the vocabulary found in news sto-ries that were originally in Arabic or Chinese.

The PPDB data outperforms both French litera-ture and MSR models if we look all possible sen-tence pairs from test data (the column labeled “all”in the table). However, when we consider whetherany pair from a set of 4 translations can be trans-lated, the PPDB models do not do as well. Thisimplies that PPDB tends to be able to reach manypairs from the same set of translations, but thereare many translations that it cannot handle at all.By contrast, the literature- and MSR-trained mod-els can reach at least one pair from 10% of thetest examples, even though the absolute numberof pairs they can reach is lower.

5.6 Effects of grammar size and choice ofsyntactic labels

Table 2 shows that the PPDB-derived grammarsare much larger than the syntactic models derivedfrom other domains. It may seem surprising thatthey should perform worse, but adding more rulesto the grammar just by varying non-terminal labelsisn’t likely to help overall parse coverage. Thissuggests a new pruning method: keep only the topk label variations for each rule type.

If we compare the syntactic models to the Hi-ero models trained from the same data, we seethat their overall reachability performance is notvery different. This implies that paraphrases canbe annotated with linguistic information withoutnecessarily hurting their ability to explain partic-ular sentence pairs. Contrast this result, with, for

example, those of Koehn et al. (2003), showingthat restricting translation models to only syntac-tic phrases hurts overall translation performance.The comparable performance between Hiero andsyntactic models seems to hold regardless of do-main.

6 Correlation with human judgments

To validate PARADIGM, we calculated its correla-tion with human judgments of paraphrase qualityon the sentence compression text-to-text genera-tion task, which has been used to evaluate para-phrase grammars in previous research (Cohn andLapata, 2007; Zhao et al., 2009; Ganitkevitch etal., 2011; Napoles et al., 2011). We created sen-tence compression systems for five of the para-phrase grammars described in Section 5.1. We fol-lowed the methodology outlined by Ganitkevitchet al. (2011) and did the following:

• Each paraphrase grammar was augmentedwith an appropriate set of rule-level featuresthat capture information pertinent to the task.In this case, the paraphrase rules were giventwo additional features that shows how thenumber of words and characters changed af-ter applying the rule.

• Similarly to how the weights of the mod-els are set using minimum error rate trainingin statistical machine translation, the weightsfor each of the paraphrase grammars usingthe PRO tuning method (Hopkins and May,2011).

• Instead of optimizing to the BLEU metric, asis done in machine translation, we optimizedto PRECIS, a metric developed for sentencecompression that adapts BLEU so that it in-cludes a “verbosity penalty” (Ganitkevitch etal., 2011) to encourage the compression sys-tems to produce shorter output.

• We created a development set with sentencecompressions by selecting 1000 pairs of sen-tences from the multiple translation corpuswhere two English translations of the sameforeign sentences differed in each other by alength ratio of 0.67–0.75.

• We decoded a test set of 1000 sentences us-ing each of the grammars and its optimized

weights with the Joshua decoder (Ganitke-vitch et al., 2012). The selected in the samefashion as the dev sentences, so each one hada human-created reference compression.

We conducted a human evaluation to judge themeaning and grammaticality of the sentence com-pressions derived from each paraphrase grammar.We presented workers on Mechanical Turk withthe input sentence to the compression sentence(the long sentence), along with 5 shortened out-puts from our compression systems. To ensurethat workers were producing reliable judgmentswe also presented them with a positive control (areference compression written by a person) and anegative controls (a compressed output that wasgenerated by randomly deleted words). We ex-cluded judgments from workers who did not per-form well on the positive and negative controls.

Meaning and grammaticality were scored on5-point scales where 5 is best. These humanscores were averaged over 2000 judgments (1000sentences x 2 annotators) for each system. Thesystems’ outputs were then scored with BLEU,PRECIS, and their paraphrase grammars werescored PARADIGM’s relative recall and precisionlower-bound estimates. For each grammar, wealso calculated the average length of parseablesentences.

We calculated the correlation between the hu-man judgements and the automatic scores, usingSpearman’s rank correlation coefficient ρ. Thisis methodology is the same that is used to quan-tify the goodness of automatic evaluation metricsin the machine translation literature (Przybocki etal., 2008; Callison-Burch et al., 2010). The pos-sible values of ρ range between 1 (where all sys-tems are ranked in the same order) and −1 (wherethe systems are ranked in the reverse order). Thusan automatic evaluation metric with a higher abso-lute value for ρ is making predictions that are moresimilar to the human judgments than an automaticevaluation metric with a lower absolute ρ.

Table 7 shows that our PARADIGM scores cor-relate more highly with human judgments than ei-ther BLEU or PRECIS for the 5 systems in our eval-uation. This suggests that it may be a better predic-tor of the goodness of paraphrase grammars thanMT metrics, when the paraphrase grammars areused for text-to-text generation tasks.

MEANING GRAMMAR

BLEU -0.7 -0.1PRECIS -0.6 +0.2PINC +0.1 +0.4PARADIGMprecision +0.6 +0.1PARADIGMrecall +0.1 +0.4PARADIGMavg−len -0.3 +0.4

Table 7: The correlation (Spearman’s ρ) of dif-ferent automatic evaluation metrics with humanjudgments of paraphrase quality for the text-to-text generation task of sentence compression.

7 Summary

We have introduced two new metrics for evaluat-ing paraphrase grammars, and looked at severalmodels from a variety of domains. Using thesemetrics we can perform a variety of analyses aboutSCFG-based paraphrase models:

• Automatically-extracted grammars can parsea small fraction of held-out data (≤30%).This is comparable to results in MT (Auli etal., 2009).

• In-domain training data is necessary in or-der to parse held-out data. A model trainedon newswire data parsed 30% of held-outnewswire sentence pairs, versus to <10% forliterature or parliamentary data.

• SCFGs with syntactic labels perform just aswell as simpler models with a single non-terminal label.

• Automatically-extracted syntactic grammarstend to have a reasonable overlap with gram-mars derived from human-aligned data, in-cluding more 45% of the gold-standard gram-mar’s paraphrase rules that occurred at leasttwice.

• We showed that PARADIGM more stronglycorrelates with human judgments of themeaning and grammaticality of paraphrasesproduced by sentence compression systemsthan standard automatic evaluation measureslike BLEU.

PARADIGM will help researchers developingparaphrase resources to perform similar diagnos-tics on their models, and quickly evaluate theirsystems.

Acknowledgements

This material is based on research sponsored bythe NSF under grant IIS-1249516 and DARPAunder agreement number FA8750-13-2-0017 (theDEFT program). The U.S. Government is autho-rized to reproduce and distribute reprints for Gov-ernmental purposes. The views and conclusionscontained in this publication are those of the au-thors and should not be interpreted as representingofficial policies or endorsements of DARPA or theU.S. Government.

ReferencesAlfred V. Aho and Jeffrey D. Ullman. 1972. The The-

ory of Parsing, Translation, and Compiling. Pren-tice Hall.

Hala Almaghout, Jie Jiang, and Andy Way. 2010.CCG augmented hierarchical phrase-based machinetranslation. In Proc. of IWSLT.

Michael Auli, Adam Lopez, Hieu Hoang, and PhilippKoehn. 2009. A systematic analysis of translationmodel search spaces. In Proc. WMT.

Colin Bannard and Chris Callison-Burch. 2005. Para-phrasing with bilingual parallel corpora. In Pro-ceedings of ACL.

Regina Barzilay and Kathleen R. McKeown. 2001.Extracting paraphrases from a parallel corpus. InProc. of ACL.

Chris Callison-Burch, Trevor Cohn, and Mirella Lap-ata. 2008. ParaMetric: An automatic evaluationmetric for paraphrasing. In Proc. of COLING.

Chris Callison-Burch, Philipp Koehn, Christof Monz,Kay Peterson, Mark Przybocki, and Omar F. Zaidan.2010. Findings of the 2010 joint workshop on sta-tistical machine translation and metrics for machinetranslation. In Proceedings of the Fourth Workshopon Statistical Machine Translation (WMT10).

David L. Chen and William Dolan. 2011. Collect-ing highly parallel data for paraphrase evaluation. InProc. of ACL.

David Chiang. 2007. Hierarchical phrase-based trans-lation. Computational Linguistics, 33(2):201–228.

Trevor Cohn and Mirella Lapata. 2007. Large mar-gin synchronous generation and its application tosentence compression. In Proceedings of EMNLP-CoLing.

Trevor Cohn and Mirella Lapata. 2009. Sentence com-pression as tree transduction. Journal of ArtificialIntelligence Research (JAIR), 34:637–674.

Trevor Cohn, Chris Callison-Burch, and Mirella Lap-ata. 2008. Constructing corpora for the develop-ment and evaluation of paraphrase systems. Com-putational Linguistics, 34(4).

William Dolan, Chris Quirk, and Chris Brockett. 2004.Unsupervised construction of large paraphrases cor-pora: Exploiting massively parallel news sources. InProc. of COLING.

Mark Dras. 1997. Representing paraphrases usingsynchronous tree adjoining grammars. In Proceed-ings of the 35th Annual Meeting of the Associa-tion for Computational Linguistics, pages 516–518,Madrid, Spain, July. Association for ComputationalLinguistics.

Chris Dyer. 2010. Two monolingual parses are bet-ter than one (synchronous parse). In Proceedings ofHLT/NAACL, pages 263–266. Association for Com-putational Linguistics.

Michel Galley, Mark Hopkins, Kevin Knight, andDaniel Marcu. 2004. What’s in a translation rule?In HLT-NAACL 2004: Main Proceedings, pages273–280.

Michel Galley, Jonathan Graehl, Kevin Knight, DanielMarcu, Steve Deneefe, Wei Wang, and IgnacioThayer. 2006. Scalable inference and training ofcontext-rich syntactic translation models. In Proc.of ACL, pages 961–968.

Juri Ganitkevitch, Chris Callison-Burch, CourtneyNapoles, and Benjamin Van Durme. 2011. Learn-ing sentential paraphrases from bilingual parallelcorpora for text-to-text generation. In Proceedingsof EMNLP.

Juri Ganitkevitch, Yuan Cao, Jonathan Weese, MattPost, and Chris Callison-Burch. 2012. Joshua 4.0:Packing, pro, and paraphrases. In Proceedings ofthe Seventh Workshop on Statistical Machine Trans-lation, pages 283–291, Montreal, Canada, June. As-sociation for Computational Linguistics.

Juri Ganitkevitch, Benjamin Van Durme, and ChrisCallison-Burch. 2013. PPDB: The paraphrasedatabase. In Proc. NAACL.

Kevin Gimpel and Noah A. Smith. 2012. Structuredramp loss minimization for machine translation. InProc. of NAACL.

Mark Hopkins and Jonathan May. 2011. Tuning asranking. In Proceedings of the 2011 Conference onEmpirical Methods in Natural Language Process-ing, pages 1352–1362, Edinburgh, Scotland, UK.,July. Association for Computational Linguistics.

Ali Ibrahim, Boris Katz, and Jimmy Lin. 2003. Ex-tracting structural paraphrases from aligned mono-lingual corpora. In Proc. of the Second InternationalWorkshop on Paraphrasing.

David Kauchak and Regina Barzilay. 2006. Para-phrasing for automatic evaluation. In Proceedingsof EMNLP.

Philipp Koehn, Franz Josef Och, and Daniel Marcu.2003. Statistical phrase-based translation. InNAACL ’03: Proceedings of the 2003 Conferenceof the North American Chapter of the Associationfor Computational Linguistics on Human LanguageTechnology, pages 48–54, Morristown, NJ, USA.Association for Computational Linguistics.

Philip M. Lewis and Richard E. Stearns. 1968.Syntax-directed transduction. Journal of the ACM,15(3):465–488.

Percy Liang, Alexandre Bouchard-Cote, Dan Klein,and Ben Taskar. 2006a. An end-to-end discrimi-native approach to machine translation. In Proc. ofACL.

Percy Liang, Ben Taskar, and Dan Klein. 2006b.Alignment by agreement. In Proceedings of theHuman Language Technology Conference of theNAACL, Main Conference, pages 104–111, NewYork City, USA, June. Association for Computa-tional Linguistics.

Dekang Lin and Patrick Pantel. 2001. Discovery ofinference rules from text. Natural Language Engi-neering, 7(3):343–360.

Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng.2010. PEM: a paraphrase evaluation metric exploit-ing parallel texts. In Proc. of EMNLP.

Bill MacCartney, Michel Galley, and Christopher D.Manning. 2008. A phrase-based alignment modelfor natural language inference. In Proceedings ofthe 2008 Conference on Empirical Methods in Nat-ural Language Processing, pages 802–811, Hon-olulu, Hawaii, October. Association for Computa-tional Linguistics.

Nitin Madnani and Bonnie Dorr. 2010. Generat-ing phrasal and sentential paraphrases: A surveyof data-driven methods. Computational Linguistics,36(3):341–388.

Nitin Madnani. 2010. The Circle of Meaning: FromTranslation to Paraphrasing and Back. Ph.D. the-sis, Department of Computer Science, University ofMaryland College Park.

Courtney Napoles, Benjamin Van Durme, and ChrisCallison-Burch. 2011. Evaluating sentence com-pression: Pitfalls and suggested remedies. In Pro-ceedings of the Workshop on Monolingual Text-To-Text Generation, pages 91–97, Portland, Oregon,June. Association for Computational Linguistics.

Franz Och and Hermann Ney. 2000. Improved sta-tistical alignment models. In Proceedings of the38th Annual Meeting of the Association for Com-putational Linguistics, pages 440–447, Hong Kong,China, October.

Matt Post, Juri Ganitkevitch, Luke Orland, JonathanWeese, Yuan Cao, and Chris Callison-Burch. 2013.Joshua 5.0: Sparser, better, faster, server. In Proc. ofWMT.

Mark Przybocki, Kay Peterson, and Sebastian Bron-sart. 2008. Official results of the NIST 2008 “Met-rics for MAchine TRanslation” challenge (Metrics-MATR08). In AMTA-2008 workshop on Metrics forMachine Translation.

Chris Quirk, Chris Brockett, and William Dolan. 2004.Monlingual machine translation for paraphrase gen-eration. In Proc. of EMNLP.

Matthew Snover, Nitin Madnani, Bonnie Dorr, andRichard Schwartz. 2010. Ter-plus: paraphrase, se-mantic, and alignment enhancements to translationedit rate. Machine Translation, 23(2-3):117–127.

Mark Steedman and Jason Baldridge. 2011. Combi-natory categorial grammar. In Robert Borsley andKersti Borjars, editors, Non-Transformational Syn-tax. Wiley-Blackwell.

Idan Szpektor, Eyal Shnarch, and Ido Dagan. 2007.Instance-based evaluation of entailment rule acqui-sition. In Proc. of ACL.

Jonathan Weese, Juri Ganitkevitch, Chris Callison-Burch, Matt Post, and Adam Lopez. 2011. Joshua3.0: Syntax-based machine translation with the thraxgrammar extractor. In Proceedings of the SixthWorkshop on Statistical Machine Translation, pages478–484, Edinburgh, Scotland, July. Association forComputational Linguistics.

Jonathan Weese, Chris Callison-Burch, and AdamLopez. 2012. Using categorial grammar to labeltranslation rules. In Proc. of WMT.

Dekai Wu. 1997. Stochastic inversion transductiongrammars and bilingual parsing of parallel corpora.Computational Linguistics, 23(3):377–404.

Shiqi Zhao, Haifeng Wang, Ting Liu, and Sheng Li.2008. Pivot approach for extracting paraphrase pat-terns from bilingual corpora. In Proceedings ofACL/HLT.

Shiqi Zhao, Xiang Lan, Ting Liu, and Sheng Li. 2009.Application-driven statistical paraphrase generation.In Proceedings of ACL.

Liang Zhou, Chin-Yew Lin, Dragos Stefan Munteanu,and Eduard Hovy. 2006. Paraeval: Using para-phrases to evaluate summaries automatically. InProceedings of HLT/NAACL.

Andreas Zollmann and Ashish Venugopal. 2006. Syn-tax augmented machine translation via chart parsing.In Proceedings on the Workshop on Statistical Ma-chine Translation, pages 138–141, New York City,June. Association for Computational Linguistics.