Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 185–194November 7–11, 2021. ©2021 Association for Computational Linguistics

185

SUMMARY EXPLORERVisualizing the State of the Art in Text Summarization

Shahbaz Syed †* Tariq Yousef †* Khalid Al-Khatib †

Stefan Jänicke ‡ Martin Potthast †

†Leipzig University ‡University of Southern Denmark<shahbaz.syed@uni-leipzig.de> <tariq.yousef@uni-leipzig.de>

Abstract

This paper introduces SUMMARY EXPLORER,a new tool to support the manual inspec-tion of text summarization systems by com-piling the outputs of 55 state-of-the-art sin-gle document summarization approaches onthree benchmark datasets, and visually explor-ing them during a qualitative assessment. Theunderlying design of the tool considers threewell-known summary quality criteria (cover-age, faithfulness, and position bias), encapsu-lated in a guided assessment based on tailoredvisualizations. The tool complements existingapproaches for locally debugging summariza-tion models and improves upon them. The toolis available at https://tldr.webis.de/.

1 Introduction

Automatic text summarization is the task of gen-erating a summary of a long text by condensingit to its most important parts. This longstandingtask originated in automatically creating abstractsfor scientific documents (Luhn, 1958), and laterextended to documents such as web pages (Saltonet al., 1994) and news articles (Wasson, 1998).

There are two paradigms of automatic sum-marization: extractive and abstractive. The for-mer extracts important information from the to-be-summarized text, while the latter additionallyinvolves paraphrasing, sentence-fusion, and natu-ral language generation to create fluent summaries.Neural summarization approaches trained on large-scale datasets have significantly advanced bothparadigms by improving the overall document un-derstanding and text generation capabilities of themodels to generate fluent summaries.

Currently, the progress in text summarizationis tracked primarily using automatic evaluationwith ROUGE (Lin, 2004) as the de facto standardfor quantitative evaluation. ROUGE has proven

* Equal contribution.

effective for evaluating extractive systems, measur-ing the overlap of word n-grams between a gener-ated summary and a reference summary (groundtruth). Still, it only provides an approximation ofa model’s capability to generate summaries thatare lexically similar to the ground truth. More-over, ROUGE is unsuitable for evaluating abstrac-tive summarization systems, mainly due to its in-adequacy in capturing all semantically equivalentvariants of the reference (Ng and Abrecht, 2015;Kryscinski et al., 2019; Fabbri et al., 2021). Be-sides, a reliable automatic evaluation of a summaryis challenging (Lloret et al., 2018) and stronglydependent on its purpose(Jones et al., 1999).

A robust method to analyze the effectivenessof summarization models is to manually inspecttheir outputs from individual perspectives such ascoverage of key concepts and linguistic quality.However, manual inspection requires obtaining theoutputs of certain models, delineating a guidelinethat comprises particular assessment criteria, andideally utilizing proper visualization techniques toexamine the outputs efficiently.

To this end, we present SUMMARY EXPLORER (Fig-ure 1), an online interactive visualization tool thatassists humans (researchers, experts, and crowds)to inspect the outputs of text summarization mod-els in a guided fashion. Specifically, we com-pile and host the outputs of several state-of-the-artmodels (currently 55) dedicated to English single-document summarization. These outputs coverthree benchmark summarization datasets compris-ing semi-extractive to highly abstractive groundtruth summaries. The tool facilitates a guided vi-sual analysis of three important summary qual-ity criteria: coverage, faithfulness, and positionbias, where tailored visualizations for each crite-rion streamline both absolute and relative manualevaluation of summaries. Overall, our use cases(see Section 5) demonstrate the ability of SUMMARY

EXPLORER to provide a comparative exploration of

186

CNN DailyMail XSum TL;DR A B C D E

Document Summary A

Summary B

Summary C

Summary D

Content coverageWhat parts of a doc. are captured by its summary?

Hallucination faithfulnessWhat are the hallucinations in a summary?

Entity coverage / faithfulnessWhich entities from a doc. are captured by its summary?

Relation coverage / faithfulnessWhich relations from a doc. are captured by its summary?

Position bias within a documentWhich parts of a single doc. do all summaries come from?

Position bias across documentsWhich parts of all docs. do all summaries come from?

1

2

3

4

Figure 1: Overview of SUMMARY EXPLORER. Its guided assessment process works in four steps: (1) corpusselection, (2) quality aspect selection, (3) model selection, and (4) quality aspect assessment. Exemplified is theassessment of the content coverage of the summaries of four models for a source document from the CNN/DMcorpus. For each summary sentence, its two most related source document sentences are highlighted on demand.

the state-of-the-art text summarization models, andto discover interesting cases that cannot likely becaptured by automatic evaluation.

2 Related Work

Leaderboards such as Paperswithcode,1 Explaina-Board2 and NLPProgress3 provide an overview ofstate of the art in text summarization mainly ac-cording to ROUGE. These leaderboards simplyaggregate the scores as reported by the models’developers, where the reported scores can be ob-tained using different implementations. Hence, afair comparison become less feasible. For instance,the Bottom-Up model (Gehrmann et al., 2018) usesa different implementation of ROUGE,4 comparedto the BanditSum model (Dong et al., 2018).5 Be-sides, for a qualitative comparison of the models,one needs to manually inspect the generated sum-maries, which are missing from such leaderboards.

To address these shortcomings, VisSeq (Wanget al., 2019) aids developers to locally comparetheir model’s outputs with the ground truth, provid-ing lexical and semantic comparisons along withstatistics such as most frequent n-grams and sen-tence score distributions. LIT (Tenney et al., 2020)provides similar functionality for a broader range1https://paperswithcode.com/task/text-summarization2http://explainaboard.nlpedia.ai/leaderboard/task-summ/3https://nlpprogress.com/english/summarization.html4https://github.com/sebastianGehrmann/rouge-baselines5https://github.com/pltrdy/rouge

of NLP tasks, implementing a work-bench-style de-bugging of model behavior, including visualizationof model attention, confusion matrices, and prob-ability distributions. Closely related to our workis SummVis (Vig et al., 2021), the recently pub-lished tool that provides a visual text comparisonof summaries with a reference summary as well asa source document, facilitating local debugging ofhallucinations in the summaries.

SUMMARY EXPLORER draws from these develop-ments and adds three missing features: (1) Quality-criteria-driven design. Based on a careful literaturereview of qualitative evaluation of summaries, wederive three key quality criteria and encode themexplicitly in the interface of our tool. Other ex-isting tools render these criteria implicit in theirunderlying design. (2) A step-by-step process forguided analysis. From the chosen quality crite-ria, we formulate concise and specific questionsneeded for a qualitative evaluation, and provide atailored visualization for each question. While pre-vious tools utilize visualization and enable users to(de)activate certain features, they oblige the usersto figure out the process themselves, which can beoverwhelming to non-experts. (3) Compilation ofthe state of the art. We collect the outputs of morethan 50 models on three benchmark datasets pro-viding a comprehensive overview of the progressin text summarization.

SUMMARY EXPLORER complements these tools and

187

also provides direct access to the state of the art intext summarization, encouraging rigorous analysisto support the development of novel models.

3 Designing Visual Summary Exploration

The design of SUMMARY EXPLORER derives fromfirst principles, namely the three quality criteriacoverage, faithfulness, and position bias of a sum-mary in relation to its source document. Thesehigh-level criteria are frequently manually assessedthroughout the literature. Since their definitionsvary, however, we derive from each criterion a totalof six specific aspects that are more straightfor-wardly operationalized in a visual exploration (seeFigure 1, Step 2). To render the aspects more di-rectly accessible to users, each is “clarified” by aguiding question that can be answered by a tailoredvisualization. Below, the three quality criteria arediscussed, followed by the visual design.

3.1 Summary Quality CriteriaCoverage A primary goal of a summary is tocapture the important information from its sourcedocument. Accordingly, a standard practice in sum-mary evaluation is to assess its coverage of thekey content (Paice, 1990; Mani, 2001; Jones et al.,1999). In many cases, a comparison to the groundtruth (reference) summary can be seen as a proxyfor coverage, which is essentially the core ideaof ROUGE. However, since it is hard to establishan ideal reference summary (Mani et al., 1999), acomparison against the source document is moremeaningful. Although an automatic comparisonagainst it is feasible (Louis and Nenkova, 2013;ShafieiBavani et al., 2018), deciding what is impor-tant content is highly subjective (Peyrard, 2019).Therefore, authors resort to a manual comparisoninstead (Hardy et al., 2019). We operationalizecoverage assessment by visualizing a document’soverlap in terms of content, entities, and entity rela-tions with its summary. Content coverage refers towhether a summary condenses information from allimportant parts of a document, measured by com-mon similarity measures; entity coverage contraststhe sets of named entities identified in both sum-mary and document; and relation coverage doesthe same, but for extracted entity relations.

Faithfulness A more recent criterion that gainedprominence especially in relation to neural sum-marization is the faithfulness of a summary to itssource document (Cao et al., 2018; Maynez et al.,

2020). Whereas coverage asks if the document issufficiently reflected in the summary, faithfulnessasks the reverse, namely if the summary adds some-thing new, questioning its appropriateness. Due totheir autoregressive nature, neural summarizationmodels have the unique property to “hallucinate”new content (Kryscinski et al., 2020; Zhao et al.,2020). This is what enables abstractive summariza-tion, but also bears the risk of generating content ina summary that is unrelated to the source document.The only acceptable hallucinated content in a sum-mary must be textually entailed by its source docu-ment, which renders an automatic assessment chal-lenging (Falke et al., 2019; Durmus et al., 2020).We operationalize faithfulness assessment by visu-alizing previously unseen words in a summary incontext, aligned with the best-matching sentencesof its source document.

Position bias Data-driven approaches, such asneural summarization models, can be biased bythe domain of their training data and learn to ex-ploit common patterns. For example, news articlesare typically structured according to an “invertedpyramid,” where the most important informationis given in the first few sentences (PurdueOWL,2019), and which models learn to exploit (Wasson,1998; Kedzie et al., 2018). Non-news texts, suchas social media posts, however, do not adopt thisstructure and thus require an unbiased considera-tion to obtain proper summaries (Syed et al., 2019).We operationalize position bias assessment by visu-alizing the parts of a document that are the sourceof its summary’s sentences, as well as the ones thatare common among a set of summaries.

3.2 Visual DesignGuided Assessment SUMMARY EXPLORER imple-ments a streamlined process to guide summaryquality assessment, consisting of four steps (seeFigure 1). (1) A benchmark dataset is selected.(2) A list of available summary quality aspects isoffered each with a preview of its tailored visual-ization and its interactive use. (3) Applying Shnei-derman’s (1996) well-known Visual Information-seeking Mantra (“overview first, zoom and filter,then details-on-demand”), an overview of all mod-els as a heatmap over averages of several quantita-tive metrics is shown (Figure 2a), which enablesa targeted filtering of the models based on theirquantitative performance. The heatmap of averagevalues paints only a rough picture; upon model

188

(a)

(b) (g)

(c)

(d) (e)

(f)

Figure 2: (a) Heatmap overview of 45 models for the CNN/DM corpus; ones selected for analysis are highlightedred. Views for (b) the content coverage, (c) the entity coverage, (d) the relation coverage, (e) the position biasacross models for a single document, (f) the position bias of a model across all documents as per lexical andsemantic alignment, (g) the distribution of quantitative metric scores for a model.

189

selection, histograms of each model’s score dis-tribution for each metric are available. (4) Aftermodels have been selected, the user is forwardedto the corresponding quality aspect’s view.

The visualizations for the individual aspects ofthe three quality criteria share the property that twotexts need to be visually aligned with one another.6

Despite this commonality, we abstain from creatinga single-view visualization “stuffed” with alterna-tive options. We rather adopt a minimalistic designfor the assessment of individual quality aspects.

Coverage View (Figure 2b,c,d) Content cover-age is visualized as alignment of summary sen-tences and document sentences at the semantic andlexical level in a full-text side-by-side view. Col-orization indicates different types of alignments.For entity coverage (relation coverage), a corre-sponding side-by-side view lists named entities (re-lations) in a summary and aligns them with namedentities (relations) in its source document. Forunaligned relations, corresponding document sen-tences can be retrieved.

Faithfulness View (Figure 3, Case A) Hallucina-tions are visualized by highlighting novel words ina summary. For each summary sentence with a hal-lucination, semantically and lexically similar docu-ment sentences are highlighted on demand. Sincenamed entities and thus also entity relations form asubset of hallucinated words, the above coverageviews do the same. Also, in an aggregated view,hallucinations found in multiple summaries are or-dered by frequency, allowing to inspect a particularmodel with respect to types of hallucinations.

Position Bias View (Figure 2e,f) Position biasis visualized for all models given a source doc-ument, and for a specific model with respect toall its summaries in a corpus. The former is visu-alized as a text heatmap, where a gradient colorindicates for every sentence in a source documenthow many different summaries contain a seman-tically or lexically corresponding sentence. Thelatter is visualized by a different kind of heatmapfor 50 randomly selected model summaries, whereeach summary is projected on a single horizontalbar representing the source document. Bar lengthreflects document length in sentences and alignedsentences are colored to reflect lexical or semanticalignment.6A visualization paradigm recently surveyed by Yousef andJänicke (2021).

Aggregation Options Most of the above visual-izations show individual pairs of source documentsand a summary. This enables the close inspectionof a given summary, and thus the manual assess-ment of a model by sequentially inspecting a num-ber of summaries for different source documentsgenerated by the same model. For these views,the visualizations also support displaying a numberof summaries from different models for a relativeassessment of their summaries.

4 Collection of Model Outputs

We collected the outputs of 55 summarizationapproaches on the test sets of three benchmarkdatasets for the task of single document summa-rization: CNN/DM, XSum and Webis-TLDR-17.Each dataset has a different style of ground truthsummaries, ranging from semi-extractive to highlyabstractive, providing a diverse selection of models.Outputs were obtained from NLPProgress, meta-evaluations such as SummEval (Fabbri et al., 2021),REALSumm (Bhandari et al., 2020), and in corre-spondence with the model’s developers.7

4.1 Summarization CorporaThe most popular dataset, CNN/DM (Hermannet al., 2015; Nallapati et al., 2016), contains newsarticles with multi-sentence summaries that aremostly extractive in nature (Kryscinski et al., 2019;Bommasani and Cardie, 2020). We obtained theoutputs from 45 models. While the original testsplit of the dataset contained 11,493 articles, wediscarded ones that were not summarized by allmodels, resulting in 11,448 articles total. This mi-nor discrepancy is due to inconsistent usage byauthors, such as reshuffling the order of examples,de-duplication of articles in the test set, choice oftokenization, text capitalization, and truncation.

For the XSum dataset (Narayan et al., 2018), theoutputs of six models for its test split (10,360 ar-ticles) were obtained. XSum contains news arti-cles with more abstractive single-sentence sum-maries compared to CNN/DM. The Webis-TLDR-17 dataset (Völske et al., 2017) contains highlyabstractive, self-authored (single to multi-sentence)summaries of Reddit posts, although slightly nois-ier than the other datasets (Bommasani and Cardie,2020). We obtained the outputs from the four sub-missions of the TL;DR challenge (Syed et al., 2019)for 250 posts.7We sincerely thank all the developers for their efforts toreproduce and share their models’ outputs with us.

190

Aligned Document Spans Summary

A) Hallucinations via Sentence Alignment

B) Hidden Errors via Relation Alignment

Aligned Document Span

Corresponding Summary Sentence

Summary Relations

Figure 3: Two showcases for identifying inconsistencies in abstractive summaries using SUMMARY EXPLORER.Case A depicts the verification of the correctness of hallucinations by aligning document sentences. Case B depictsuncovering more subtle hallucination errors by comparing unaligned relations.

4.2 Text PreprocessingIn a preprocessing pipeline, the input of a collectionof documents, their ground truth summaries, andthe generated summaries from a given model werenormalized. First, basic normalization, such asde-tokenization, unifying model-specific sentencedelimiters, and sentence segmentation were carriedout. Second, additional information, such as namedentities and relations were extracted using Spacy8

and Stanford OpenIE (Angeli et al., 2015), respec-tively. The latter extracts redundant relations wherepartial components such as either the subject or theobject are already captured by longer counterparts.Such “contained” relations are merged into uniquerepresentative relations for each subject.

Alignment Every output summary is alignedwith its source document, identifying the top twolexically and semantically related document sen-tences for each summary sentence. Lexical align-ment relies on averaged ROUGE-{1,2,L} scoresamong the document and summary sentences. Thehighest scoring document sentence is taken as thefirst match. The second match is identified by re-moving all content words from the summary sen-tence already captured by the first match, and re-peating the process as per Lebanoff et al. (2019).For semantic alignment, the rescaled BERTScore(Zhang et al., 2020) is computed between a sum-mary sentence and all source document sentences,with the top-scoring two sentences as candidates.8https://spacy.io

Summary Evaluation Measures Several stan-dard evaluation measures enable quantitative com-parisons and filtering of models for detailed anal-ysis: (1) compression as the word ratio between adocument and its summary (Grusky et al., 2018),(2) n-gram abstractiveness as per Gehrmann et al.(2019) calculates a normalized score for novelty bytracking parts of a summary that are already amongthe n-grams it has in common with its document,(3) summary length as word count (not tokens),(4) entity-level factuality as per (Nan et al., 2021)as percentage of named entities in a summary foundin its source document, and (5) relation-level factu-ality as percentage of relations in a summary foundin its source document. Finally, for consistency, werecompute ROUGE-{1,2,L}9 for all the models.

5 Assessment Case Studies

We showcase the use and effectiveness of SUMMARY

EXPLORER by investigating two models (IMPROVE-ABS-NOVELTY, and IMPROVE-ABS-NOVELTY-LM)from Kryscinski et al. (2018) that improve the ab-straction in summaries by including more novelphrases. We investigate the correctness of theirhallucinations (novel words in the summary), andidentify hidden errors introduced by the sentencefusion of the abstractive models.9https://github.com/google-research/google-research/tree/master/rouge

191

Hallucinations via Sentence Alignment Hallu-cinations are novel words or phrases in a summarythat warrant further inspection. Accordingly, ourtool highlights them (Figure 3, Case A), direct-ing the user to the respective candidate summarysentences whose related document sentences canbe seen on demand. For IMPROVE-ABS-NOVELTY,we see that the first candidate improves abstrac-tion via paraphrasing, is concisely written, andcorrectly substitutes the term “offenses” with thenovel word “charges”. The second candidate alsoimproves abstraction via sentence fusion, wheretwo pieces of information are combined: “bennettallegedly drove her daughter”, and “victim advisedshe thought she was going to die”. The novel word

“told” also fits. However, the sentence fusion cre-ates a wrong relation between the different actors(“bennett allegedly told her daughter that she wasgoing to die”), which can be easily identified viathe visual sentence alignment provided.

Hidden Errors via Relation Alignment Theabove showcase does not capture all hallucinations.SUMMARY EXPLORER also aligns relations extractedfrom a summary and its source document to iden-tify novel relations. For IMPROVE-ABS-NOVELTY-LM, we see that the relation “she was arrested” isunaligned to any relation in the source document(Figure 3, Case B). Aligning the summary sentenceto the document, we note that it is unfaithful to thesource despite avoiding hallucinations (“Bennettwas released on $10,500 bail”, and not “arrestedon $10,500 bail”). The word “arrested” was sim-ply extracted from the document sentence (Figure 3,Case A). Without the visual support, identifyingthis small but important mistake would have beenmore cognitively demanding for an assessor.

6 Conclusion

In this paper, we present SUMMARY EXPLORER, an on-line interactive visualization tool to assess the stateof the art in text summarization in a guided fash-ion. In enables analysis akin to close and distantreading in particular facilitating the challenginginspection of hallucinations by abstractive summa-rization models. The tool is available open source10

enabling local use. We also welcome submissionsof summaries from newer models trained on the ex-isting datasets as part of our collaboration with thesummarization community. We aim to expand the10https://github.com/webis-de/summary-explorer

tool’s features in future work, exploring novel vi-sual comparisons of documents to their summariesfor more reliable qualitative assessments of sum-mary quality. Finally, it is important to note that theaccuracy of some of the views is influenced by theintrinsic drawbacks of the toolkits used for namedentity recognition and information extraction.

7 Ethical Statement

Visualization plays a major role in the usage and ac-cessibility of our tool. In this regard, to accommo-date for color blindness, we primarily use gradient-based visuals for key modules such as model se-lection, aggregating important content, and textalignment. This renders the tool usable also in amonochromatic setting. Regarding the hosted sum-marization models, the key goal is to allow a wideraudience comprising of model developers, the endusers, and practitioners to openly compare and as-sess the strengths, limitations and possible ethicalbiases of these systems. Here, our tool supportsmaking informed decisions about the suitability ofcertain models to the downstream applications.

Acknowledgments

We thank the reviewers for their valuable feed-back. This work was supported by the German Fed-eral Ministry of Education and Research (BMBF,01/S18026A-F) by funding the competence centerfor Big Data and AI (ScaDS.AI Dresden/Leipzig).

ReferencesGabor Angeli, Melvin Jose Johnson Premkumar, and

Christopher D. Manning. 2015. Leveraging linguis-tic structure for open domain information extraction.In Proceedings of the 53rd Annual Meeting of theAssociation for Computational Linguistics and the7th International Joint Conference on Natural Lan-guage Processing of the Asian Federation of NaturalLanguage Processing, ACL 2015, July 26-31, 2015,Beijing, China, Volume 1: Long Papers, pages 344–354. The Association for Computer Linguistics.

Manik Bhandari, Pranav Narayan Gour, Atabak Ash-faq, Pengfei Liu, and Graham Neubig. 2020. Re-evaluating evaluation in text summarization. In Pro-ceedings of the 2020 Conference on Empirical Meth-ods in Natural Language Processing (EMNLP), On-line. Association for Computational Linguistics.

Rishi Bommasani and Claire Cardie. 2020. Intrinsicevaluation of summarization datasets. In Proceed-ings of the 2020 Conference on Empirical Methodsin Natural Language Processing (EMNLP), pages

192

8075–8096, Online. Association for ComputationalLinguistics.

Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2018.Faithful to the original: Fact aware neural abstrac-tive summarization. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence,(AAAI-18), the 30th innovative Applications of Arti-ficial Intelligence (IAAI-18), and the 8th AAAI Sym-posium on Educational Advances in Artificial Intel-ligence (EAAI-18), New Orleans, Louisiana, USA,February 2-7, 2018, pages 4784–4791. AAAI Press.

Yue Dong, Yikang Shen, Eric Crawford, Herke vanHoof, and Jackie Chi Kit Cheung. 2018. Banditsum:Extractive summarization as a contextual bandit. InProceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, Brussels,Belgium, October 31 - November 4, 2018, pages3739–3748. Association for Computational Linguis-tics.

Esin Durmus, He He, and Mona Diab. 2020. FEQA: Aquestion answering evaluation framework for faith-fulness assessment in abstractive summarization. InProceedings of the 58th Annual Meeting of the As-sociation for Computational Linguistics, ACL 2020,Online, July 5-10, 2020, pages 5055–5070. Associa-tion for Computational Linguistics.

Alexander R. Fabbri, Wojciech Kryscinski, BryanMcCann, Caiming Xiong, Richard Socher, andDragomir R. Radev. 2021. Summeval: Re-evaluating summarization evaluation. Trans. Assoc.Comput. Linguistics, 9:391–409.

Tobias Falke, Leonardo F. R. Ribeiro, Prasetya AjieUtama, Ido Dagan, and Iryna Gurevych. 2019.Ranking generated summaries by correctness: An in-teresting but challenging application for natural lan-guage inference. In Proceedings of the 57th Confer-ence of the Association for Computational Linguis-tics, ACL 2019, Florence, Italy, July 28- August 2,2019, Volume 1: Long Papers, pages 2214–2220. As-sociation for Computational Linguistics.

Sebastian Gehrmann, Yuntian Deng, and Alexander M.Rush. 2018. Bottom-up abstractive summarization.In Proceedings of the 2018 Conference on EmpiricalMethods in Natural Language Processing, Brussels,Belgium, October 31 - November 4, 2018, pages4098–4109. Association for Computational Linguis-tics.

Sebastian Gehrmann, Zachary M. Ziegler, and Alexan-der M. Rush. 2019. Generating abstractive sum-maries with finetuned language models. In Proceed-ings of the 12th International Conference on Natu-ral Language Generation, INLG 2019, Tokyo, Japan,October 29 - November 1, 2019, pages 516–522. As-sociation for Computational Linguistics.

Max Grusky, Mor Naaman, and Yoav Artzi. 2018.Newsroom: A dataset of 1.3 million summaries withdiverse extractive strategies. In Proceedings of the

2018 Conference of the North American Chapterof the Association for Computational Linguistics:Human Language Technologies, NAACL-HLT 2018,New Orleans, Louisiana, USA, June 1-6, 2018, Vol-ume 1 (Long Papers), pages 708–719. Associationfor Computational Linguistics.

Hardy, Shashi Narayan, and Andreas Vlachos. 2019.Highres: Highlight-based reference-less evaluationof summarization. In Proceedings of the 57th Con-ference of the Association for Computational Lin-guistics, ACL 2019, Florence, Italy, July 28- August2, 2019, Volume 1: Long Papers, pages 3381–3392.Association for Computational Linguistics.

Karl Moritz Hermann, Tomás Kociský, Edward Grefen-stette, Lasse Espeholt, Will Kay, Mustafa Suleyman,and Phil Blunsom. 2015. Teaching machines toread and comprehend. In Advances in Neural Infor-mation Processing Systems 28: Annual Conferenceon Neural Information Processing Systems 2015,December 7-12, 2015, Montreal, Quebec, Canada,pages 1693–1701.

K Sparck Jones et al. 1999. Automatic summarizing:factors and directions. Advances in automatic textsummarization, pages 1–12.

Chris Kedzie, Kathleen R. McKeown, and Hal DauméIII. 2018. Content selection in deep learning mod-els of summarization. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 1818–1828. Associationfor Computational Linguistics.

Wojciech Kryscinski, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019.Neural text summarization: A critical evaluation.In Proceedings of the 2019 Conference on Empiri-cal Methods in Natural Language Processing andthe 9th International Joint Conference on NaturalLanguage Processing, EMNLP-IJCNLP 2019, HongKong, China, November 3-7, 2019, pages 540–551.Association for Computational Linguistics.

Wojciech Kryscinski, Bryan McCann, Caiming Xiong,and Richard Socher. 2020. Evaluating the factualconsistency of abstractive text summarization. InProceedings of the 2020 Conference on EmpiricalMethods in Natural Language Processing, EMNLP2020, Online, November 16-20, 2020, pages 9332–9346. Association for Computational Linguistics.

Wojciech Kryscinski, Romain Paulus, Caiming Xiong,and Richard Socher. 2018. Improving abstractionin text summarization. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 1808–1817. Associationfor Computational Linguistics.

Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt,Doo Soon Kim, Seokhwan Kim, Walter Chang, andFei Liu. 2019. Scoring sentence singletons and pairs

193

for abstractive summarization. In Proceedings ofthe 57th Conference of the Association for Compu-tational Linguistics, ACL 2019, Florence, Italy, July28- August 2, 2019, Volume 1: Long Papers, pages2175–2189. Association for Computational Linguis-tics.

Chin-Yew Lin. 2004. ROUGE: A package for auto-matic evaluation of summaries. In Text Summariza-tion Branches Out, pages 74–81, Barcelona, Spain.Association for Computational Linguistics.

Elena Lloret, Laura Plaza, and Ahmet Aker. 2018.The challenging task of summary evaluation: anoverview. Lang. Resour. Evaluation, 52(1):101–148.

Annie Louis and Ani Nenkova. 2013. Automaticallyassessing machine summary content without a goldstandard. Comput. Linguistics, 39(2):267–300.

Hans Peter Luhn. 1958. The automatic creation of lit-erature abstracts. IBM Journal of Reseach and De-vopment, 2(2):159–165.

Inderjeet Mani. 2001. Summarization evaluation: Anoverview.

Inderjeet Mani, David House, Gary Klein, LynetteHirschman, Therese Firmin, and Beth Sundheim.1999. The tipster summac text summarization eval-uation. In EACL 1999, 9th Conference of the Euro-pean Chapter of the Association for ComputationalLinguistics, June 8-12, 1999, University of Bergen,Bergen, Norway, pages 77–85. The Association forComputer Linguistics.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, andRyan T. McDonald. 2020. On faithfulness and fac-tuality in abstractive summarization. In Proceedingsof the 58th Annual Meeting of the Association forComputational Linguistics, ACL 2020, Online, July5-10, 2020, pages 1906–1919. Association for Com-putational Linguistics.

Ramesh Nallapati, Bowen Zhou, Cícero Nogueira dosSantos, Çaglar Gülçehre, and Bing Xiang. 2016.Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the20th SIGNLL Conference on Computational NaturalLanguage Learning, CoNLL 2016, Berlin, Germany,August 11-12, 2016, pages 280–290. ACL.

Feng Nan, Ramesh Nallapati, Zhiguo Wang,Cícero Nogueira dos Santos, Henghui Zhu, DejiaoZhang, Kathy McKeown, and Bing Xiang. 2021.Entity-level factual consistency of abstractive textsummarization. In Proceedings of the 16th Confer-ence of the European Chapter of the Association forComputational Linguistics: Main Volume, EACL2021, Online, April 19 - 23, 2021, pages 2727–2733.Association for Computational Linguistics.

Shashi Narayan, Shay B. Cohen, and Mirella Lapata.2018. Don’t give me the details, just the summary!

topic-aware convolutional neural networks for ex-treme summarization. In Proceedings of the 2018Conference on Empirical Methods in Natural Lan-guage Processing, Brussels, Belgium, October 31 -November 4, 2018, pages 1797–1807. Associationfor Computational Linguistics.

Jun-Ping Ng and Viktoria Abrecht. 2015. Better sum-marization evaluation with word embeddings forROUGE. In Proceedings of the 2015 Conference onEmpirical Methods in Natural Language Processing,EMNLP 2015, Lisbon, Portugal, September 17-21,2015, pages 1925–1930. The Association for Com-putational Linguistics.

Chris D Paice. 1990. Constructing literature abstractsby computer: techniques and prospects. InformationProcessing & Management, 26(1):171–186.

Maxime Peyrard. 2019. A simple theoretical model ofimportance for summarization. In Proceedings ofthe 57th Conference of the Association for Compu-tational Linguistics, ACL 2019, Florence, Italy, July28- August 2, 2019, Volume 1: Long Papers, pages1059–1073. Association for Computational Linguis-tics.

PurdueOWL. 2019. Journalism and journalistic writ-ing: The inverted pyramid structure.

Gerard Salton, James Allan, Chris Buckley, and AmitSinghal. 1994. Automatic analysis, theme genera-tion, and summarization of machine-readable texts.Science, 264(5164):1421–1426.

Elaheh ShafieiBavani, Mohammad Ebrahimi, Ray-mond K. Wong, and Fang Chen. 2018. Summa-rization evaluation in the absence of human modelsummaries using the compositionality of word em-beddings. In Proceedings of the 27th InternationalConference on Computational Linguistics, COLING2018, Santa Fe, New Mexico, USA, August 20-26,2018, pages 905–914. Association for Computa-tional Linguistics.

B. Shneiderman. 1996. The eyes have it: a task by datatype taxonomy for information visualizations. InProceedings 1996 IEEE Symposium on Visual Lan-guages, pages 336–343.

Shahbaz Syed, Michael Völske, Nedim Lipka, BennoStein, Hinrich Schütze, and Martin Potthast. 2019.Towards summarization for social media - results ofthe tl;dr challenge. In Proceedings of the 12th Inter-national Conference on Natural Language Genera-tion, INLG 2019, Tokyo, Japan, October 29 - Novem-ber 1, 2019, pages 523–528. Association for Compu-tational Linguistics.

Ian Tenney, James Wexler, Jasmijn Bastings, TolgaBolukbasi, Andy Coenen, Sebastian Gehrmann,Ellen Jiang, Mahima Pushkarna, Carey Radebaugh,Emily Reif, and Ann Yuan. 2020. The language in-terpretability tool: Extensible, interactive visualiza-tions and analysis for NLP models. In Proceedings

194

of the 2020 Conference on Empirical Methods inNatural Language Processing: System Demonstra-tions, EMNLP 2020 - Demos, Online, November 16-20, 2020, pages 107–118. Association for Computa-tional Linguistics.

Jesse Vig, Wojciech Kryscinski, Karan Goel, andNazneen Fatema Rajani. 2021. Summvis: Interac-tive visual analysis of models, data, and evaluationfor text summarization. CoRR, abs/2104.07605.

Michael Völske, Martin Potthast, Shahbaz Syed, andBenno Stein. 2017. Tl;dr: Mining reddit tolearn automatic summarization. In Proceedingsof the Workshop on New Frontiers in Summariza-tion, NFiS@EMNLP 2017, Copenhagen, Denmark,September 7, 2017, pages 59–63. Association forComputational Linguistics.

Changhan Wang, Anirudh Jain, Danlu Chen, and Ji-atao Gu. 2019. Vizseq: a visual analysis toolkitfor text generation tasks. In Proceedings of the2019 Conference on Empirical Methods in Natu-ral Language Processing and the 9th InternationalJoint Conference on Natural Language Processing,EMNLP-IJCNLP 2019, Hong Kong, China, Novem-ber 3-7, 2019 - System Demonstrations, pages 253–258. Association for Computational Linguistics.

Mark Wasson. 1998. Using leading text for news sum-maries: Evaluation results and implications for com-mercial summarization applications. In Proc. of the36th Annual Meeting of the Association for Com-putational Linguistics and 17th Int. Conference onComputational Linguistics, pages 1364–1368.

Tariq Yousef and Stefan Jänicke. 2021. A survey oftext alignment visualization. IEEE Transactions onVisualization and Computer Graphics, 27(2):1149–1159.

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.Weinberger, and Yoav Artzi. 2020. Bertscore: Eval-uating text generation with BERT. In 8th Inter-national Conference on Learning Representations,ICLR 2020, Addis Ababa, Ethiopia, April 26-30,2020. OpenReview.net.

Zheng Zhao, Shay B. Cohen, and Bonnie Webber. 2020.Reducing quantity hallucinations in abstractive sum-marization. In Findings of the Association for Com-putational Linguistics: EMNLP 2020, Online. Asso-ciation for Computational Linguistics.