chart-to-text: generating natural language descriptions
TRANSCRIPT
Proceedings of The 13th International Conference on Natural Language Generation, pages 138–147,Dublin, Ireland, 15-18 December, 2020. c©2020 Association for Computational Linguistics
138
Chart-to-Text: Generating Natural Language Descriptions for Charts byAdapting the Transformer Model
Jason ObeidSchool of Information Technology
York University, [email protected]
Enamul HoqueSchool of Information Technology
York University, [email protected]
Abstract
Information visualizations such as bar chartsand line charts are very popular for exploringdata and communicating insights. Interpretingand making sense of such visualizations canbe challenging for some people, such as thosewho are visually impaired or have low visual-ization literacy. In this work, we introduce anew dataset and present a neural model for au-tomatically generating natural language sum-maries for charts. The generated summariesprovide an interpretation of the chart and con-vey the key insights found within that chart.Our neural model is developed by extendingthe state-of-the-art model for the data-to-textgeneration task, which utilizes a transformer-based encoder-decoder architecture. We foundthat our approach outperforms the base modelon a content selection metric by a wide margin(55.42% vs. 8.49%) and generates more infor-mative, concise, and coherent summaries.
1 Introduction
Information visualizations such as bar charts andline charts are commonly utilized by people to getinsights from data and make informed decisions.However, understanding and getting insights fromcharts can be difficult and time-consuming. Thisis because people often need to visually comparebetween several graphical marks (e.g. bars) of achart to infer key insights from data, which maybe challenging when the chart involves many dataitems (Kim et al., 2020).
Generating a natural language summary to ex-plain a chart has numerous benefits and potentialapplications. It can help users in understandingand interpreting charts by conveying key pointsabout the chart by focusing on temporal, causal,and evaluative aspects (Carenini et al., 2013). It canhelp people identify insights from charts that theyotherwise might have missed. A previous study
conducted on a chart corpus found that the textassociated with the chart failed to convey any in-sight from that chart in 35% of the instances, whilein another 26% of cases the text conveyed only aportion of the chart’s intended message (Carberryet al., 2006). Therefore, effective chart summariza-tion could help data analysts, business analysts, orjournalists in better preparing reports from data.Such summaries could also enable people who arevisually impaired or have low cognitive abilitiesto comprehend charts and perform analytical tasksthrough audio (Ferres et al., 2013).
However, natural language generation (NLG)systems for charts are still in their infancy. Earlywork mostly focused on statistical methods for find-ing salient information from charts and planning-based approaches for structuring content (Reiter,2007) to generate textual captions from basiccharts (Fasciano and Lapalme, 1996; Mittal et al.,1998; Green et al., 2004; Demir et al., 2012).Commercial systems such as Quill (Quill, 2020)and Wordsmith (Wordsmith, 2020) have similarlyadopted statistical algorithms and template-basedNLG methods which are used to produce datafacts in textual form. Unfortunately, the predefinedtemplate-based NLG methods and planning-basedarchitecture for generating summaries often lackgenerality and may not offer variations in style.Moving beyond template and planning-based ap-proaches (Reiter, 2007), more recently researchersconsidered data-driven neural models for generat-ing text from data tables (Mei et al., 2016; Gonget al., 2019). However, they do are not designedfor chart summarization as they do not considerthe chart specific features (e.g. chart type). Toour knowledge, there have not been any efforts todevelop deep neural models that are specificallytailored for generating chart summaries.
In this paper, we present a neural model for chartsummarization by extending a transformer-based
139
Figure 1: An example of a generated natural languageexplanation from a chart from our model.
model that was originally designed for the data-to-text generation task. A key challenge in devel-oping neural models for chart summarization isthe lack of a suitable dataset containing a large setof charts and corresponding human-written sum-maries. To address the challenge we first devel-oped a corpus of 8,305 charts, where for each chartwe crawled the corresponding data table and thehuman-written summary. Our model learns contentselection and text generation from such a collectionof chart-summary pairs to generate summaries ofcharts. We also introduce a data variable substitu-tion method that replaces the tokens referring tochart data with variables to generate more factuallycorrect statements than a competitive baseline. Asshown in Figure 1, the resulting summary is quiteinformative, concise, and coherent.
The contributions of our paper are three-fold:• First, we introduce a new large-scale corpus
on chart summarization consisting of human-written summaries of charts along with chartimages and their underlying data.
• Second, we adopt a transformer-based modelto generate chart summaries, which learnsfrom chart-summary pairs from our dataset.To our knowledge, our work is the first to in-vestigate the problem of chart summarizationusing a data-driven deep neural model.
• Finally, we perform a series of evaluations tocompare our model’s performance with a base-line model derived from Gong et al. (Gonget al., 2019). As a secondary contribution,we will make our source codes and the new
dataset used in this research publicly avail-able.
2 Related Work
2.1 Chart Summarization
Early work on generating natural language summa-rization follows some planning-based approacheswhich can be captured by Reiter’s NLG architec-ture for data-to-text generation (Reiter, 2007). Forexample, (Mittal et al., 1998) developed a captiongeneration system which determines the structureof the caption based on the mappings between thedata and marks of the charts (such as text, lines,and bars) and chooses a complexity metric to selectdetails of the caption. It then uses a text plan-ner to generate the description of the chart. TheiGRAPH-Lite system (Ferres et al., 2013) aims tomake charts accessible to blind users via generat-ing captions and supporting navigation through thechart using a keyboard. Like (Mittal et al., 1998)it uses templates to provide a short description ofwhat the chart looks like. Nonetheless, these sys-tems describe the chart only in terms of how tointerpret the chart rather than explaining high-levelinsights conveyed by the chart.
Other research has focused on generating multi-media presentations combining both text summaryand charts (Fasciano and Lapalme, 2000; Greenet al., 2004). PostGraphe takes user-specified inten-tion (e.g. increasing trend) and a spreadsheet file asinput and then generates a simple chart and captionseparately using some heuristics (Fasciano and La-palme, 2000). Autobrief generates presentations intext and information graphics (Green et al., 2004)using an integrated planning-based approach in thedomain of transportation scheduling.
There has also been growing interest to automat-ically extract insights from a dataset and presentthem using template-based NLG. Examples ofsuch automatic insight generation include researchprototypes such as DataSite (Cui et al., 2019),DataShot (Wang et al., 2019) and Voder (Srini-vasan et al., 2018) as well as commercial systemslike Quill1, Arria2, and Wordsmith3. These systemstypically perform some statistical analysis to inferpotentially important or interesting facts about thedata and then present them in natural language sen-tences and charts. (Demir et al., 2012) present a
1Narrative Science. Quill2Arria, Arria3Wordsmith Wordsmith
140
method that computes some statistics (e.g. min,max, trends) and identifies the intended messagethat a bar chart is conveying using a Bayesian in-ference system. Then, it generates the summary ina bottom–up approach to simultaneously constructthe discourse and sentence structures of textualsummaries. More recently, (Chen et al., 2019) usesthe encoder-decoder architecture where a ResidualNetwork (ResNet) (He et al., 2016) in the encoderis used to recognize the input chart from an im-age, and Long Short-Term Memory (LSTM) andattention in the decoder to create template-basedcaptions.
A common limitation of the above body of workis that the sentences are generated using predefinedtemplate-based approaches which may lack gen-erality and offer fewer variations in grammaticalstyle and lexical choices compared to data-drivenmodels. In contrast, we focus on learning languagevariations and automatically finding important in-sights from charts using a deep learning model ona large collection of chart-summary pairs.
2.2 Data-to-Text Generation
The objective of data-to-text generation is to gen-erate a descriptive summary given structured data.Data-to-text generation focuses on creating a de-scriptive summary from structured data which canbe encoded as a table of records. Data to text gener-ation has been explored for various domain-specifictasks such as summarizing sport game data (Barzi-lay and Lapata, 2005; Liang et al., 2009; Wisemanet al., 2017), weather-forecast data (Reiter et al.,2005), recipe generation (Yang et al., 2017) andbiography generation (Lebret et al., 2016).
Several recent methods have primarily focusedon using sequence-to-sequence learning methods(Mei et al., 2016; Lebret et al., 2016; Wisemanet al., 2017). For example, (Mei et al., 2016) pro-pose an encoder-decoder model that uses recurrentneural networks with LSTM units for jointly learn-ing content selection and surface realization forweather forecast and soccer commentaries. (Wise-man et al., 2017) present a new dataset on NBAgame summarization and evaluate several modelsincluding attention-based encoder-decoder models.They found that neural text generation techniquesfrom data perform poorly at content selection andlack inter-sentential coherence. (Puduppully et al.,2019) attempt to address this problem by incorpo-rating content selection and planning mechanisms
within the neural model. (Gong et al., 2019) foundthat the transformer model yielded outputs morefluent and coherent when compared to their seq2seqcounterparts, which is why we use it as the basemodel.
3 Chart Summarization Dataset
There have been several benchmark datasets thatare made available recently for the data-to-text gen-eration tasks (Lebret et al., 2016; Wiseman et al.,2017; Chen et al., 2020; Parikh et al., 2020). How-ever, to the best of our knowledge, there are no pub-licly available large datasets of chart data pairedwith human-generated summaries. While someprior work on generating captions and summariesfrom charts (e.g. (Mittal et al., 1998; Green et al.,2004; Demir et al., 2012)) exist, to our knowledgethey do not provide any large datasets with chart-summary pairs.
With the above challenges in mind, we createa new dataset for chart summarization which isavailable at https://github.com/JasonObeid/
Chart2Text. In order to find the best source ofdata for our corpus, we have analyzed publiclyavailable charts from various sources such as text-books, research papers, news articles, and websitesthat contain data charts and facts. This led us tochoose Statista4 as our data source. Statista regu-larly publishes charts from data collected by marketand opinion research institutes, and data derivedfrom the economic sector. Since Statista has allthe necessary metadata including the data tables,titles, and concise summaries of charts it was asuitable source for our purpose. The dataset wascrawled from 23,382 freely accessible pages fromstatista.com in early March of 2020, yielding a to-tal of 8,305 charts, and associated summaries. Foreach chart, we downloaded the chart image, theunderlying data table, the title, the axis labels, anda human-written summary describing the statistic.
After examining the crawled dataset, we foundthat out of 8,305 charts, 7,726 charts were lack-ing the x-axis labels. To address this problem wesearched by regular expressions and applied namedentity recognition using CoreNLP (Manning et al.,2014) on the data table to automatically identifyif the x-axis represented common temporal dimen-sions such as years or months. For the remaining1,353 missing labels we used human annotatorswho examined the title, summary and the chart to
4https://www.statista.com/
141
Line Bar Total:
Simple 3564 3199 6763Complex 902 640 1542
Total: 4466 3839
Table 1: Chart type distribution
Statistic Value
Mean Token Count 113.4Mean Sentence Count 5.2Vocab Size 19,150Total Tokens 941.8kMean Data Cells 32.3
Table 2: Dataset statistics
come up with short descriptive labels for the x-axis.
The resulting dataset can be described as fol-lows: let the set of chart-summaries CS =[I,D, T, L, S], where for each chart-summary inCS there is a chart image i ∈ I , data table d ∈ D,title t ∈ T , axis labels l ∈ L, and summary s ∈ S.The dataset consists of both bar and line charts (Ta-ble 2). A simple bar chart only has a set of bars, anda simple line chart contains a single line. Complexbar charts include stacked and grouped bar chartswhile complex line charts have multiple lines.
The summaries of the charts are concise, with anaverage sentence count of 5.2, and an average to-ken count of 113.4. The data tables were moderatein size with an average of 32.3 cells. The discoursestructures of these summaries were usually simi-lar. They usually start by describing the chart ata high-level in terms of what this chart is aboutand then describing and comparing salient pointsin the chart. Some common salient information andstatistics are extremes (e.g. highest/lowest values)or trends (upward/downward tendencies) or simplevalue retrieval (e.g. mentioning the first/last datapoint’s value).
4 The Chart-to-Text Model
In this section, we first describe the Transformer-based Model for Data-to-Text (Gong et al., 2019)which we use as our base model, followed by ouradaptations to this model for generating chart sum-maries. Then, we describe our experiments withmodel parameters and the training procedure.
4.1 Base Model
Our base model (Gong et al., 2019) extends thestandard transformer (Vaswani et al., 2017) byadding a binary prediction layer and a content se-lection training step. The input layer of the modelaccepts a tuple of four features (entity, type, value,information) as input. The model then generatesthe latent representation of each record in the inputsequence and passes it through the binary predic-tion layer. The decoder of this model is the sameas the original Transformer model and predicts thesummary based on encoder’s output. The modelalso removes the positional embedding of the trans-former encoder, as there is no ordered relationshipwithin the records of their dataset.
4.2 Our Proposed Approach
Figure 2 shows an overview of our proposedapproach. The model takes data records andother chart information as input. Like the basemodel (Gong et al., 2019), it uses the self-attentionmechanism of the transformer encoder to gener-ate the latent representations of the input, and thebinary prediction at the top of the Transformer en-coder output to decide whether or not a record willbe mentioned in the target summary. Finally, thedecoder predicts the next token in the context ofthe encoder output and the previous tokens fromthe summary sequence.
In order to adapt the enhanced transformermodel (Gong et al., 2019) for generating chart sum-marization, we introduce three main changes. First,we modify the four features of the record tuplesused as input to the model. This is done to includeadditional information which is important for chartsummarization. Second, we reintroduce positionalembeddings to the encoder since charts tend to con-tain ordered relationships. Finally, we introduce aprocess of substituting tokens with data variablesto minimize the amount of generated hallucina-tions, a problem commonly found in NLG where apredicted word is not grounded in truth (Wisemanet al., 2017; Parikh et al., 2020). We now discusseach of these changes in more details:
Input embedding: For each data table d ∈ D,we pre-process it into a set of records r ∈ R. Eachof these records are placed in a tuple with fourfeatures as follows:
• ri(0) contains the column header,• ri(1) contains the table cell value,• ri(2) contains the column index, and
142
Figure 2: An overview of the proposed approach for chart summarization using a transformer-based model. Themodel takes the data table and some chart metadata as input (on the left) and generates a summary containing datavariables that refer values within the data table.
• ri(3) contains the chart type
Each feature is embedded into a vector, and to-gether they are concatenated to represent the recordas shown below:∀ r ∈ R:ri = [ri(0); ri(1); ri(2); ri(3)]Our model takes each of these records ri as input,
and outputs a set of predicted tokens which we willdenote as y ∈ Y. To get the final summary, for eachtoken yi in y, if the token is a data variable then itis substituted with the corresponding data value.
Positional embedding: Unlike the sportsdataset used by (Gong et al., 2019), chart data of-ten involves an ordered dimension such as temporaldata (e.g. year, month), or ordinal categories in barcharts. In order to generate summaries that bet-ter capture salient features from such sequentialdata, we re-introduce positional embeddings to ourmodel.
Data variable substitution: A critical issuewith various existing models for data-to-text gener-ation problem such as (Gong et al., 2019) is thatthey treat data records mentioned in the summariesas regular tokens which often results in hallucina-tion problem. As a consequence, the model some-times predicts tokens that are irrelevant to the chartand thus results in factually incorrect sentences.This problem becomes even more serious for ourchart dataset which is not focused on a specific do-main (e.g. sports). In order to improve the factualaccuracy of the generated summaries we introducea data variable substitution method.
The idea is that before training we first modifythe gold summaries so that whenever a token ref-erences something from the data table, chart title,or axis labels we replace them with one of the pre-defined data variables. We then use these modifiedsummaries to train the model so that it learns howto generate the summary more generally with datavariables as opposed to actual values in the datatable or tokens from the title. Then, during thetesting phase, if a generated token matches a prede-fined data variable, we make a look-up operation toconvert the data variable into the referenced chartdata (see Figure 3).
We define seven categories of data variableslisted in order of priority: subjects, dates, axis la-bels, titles, table cells, trends, and scales. Heresubjects refer to entities relevant to the data table(e.g. ‘Liverpool FC’) and trends refer to tokensthat indicate positive or negative trends (e.g. grow-ing, decreasing, etc). Scales refer to tokens such as‘millions’ and ‘percentage’ that are associated withnumeric tokens. For each token ti in a gold sum-mary s, if that token matches any of these variablecategories, then it is substituted by that variable.We apply named entity recognition (Manning et al.,2014) to detect the subjects and dates. The restof the variables are detected using simple stringpattern matching techniques. For each variable,we also assign the relevant index position in thedata table or other chart data. For example, givena sentence in the gold summary “Broadcasting isthe largest source of revenue for Liverpool FC”,
143
Figure 3: Demonstration of data variable substitution.
the system replaces the token ‘Broadcasting’ with< templateLabel[2][0] > which refers to the thirdcolumn’s header label.
During the testing phase, our approach performsthe reverse operation where it substitutes data vari-ables with their referenced tokens. Figure 3 demon-strates how the data variables are substituted withactual tokens from the chart data. Here, the corre-sponding tokens and variables are highlighted bythe same color.
4.3 Training
The model requires two types of labels for its train-ing step. The first type of label is for the chart’sdata, which for each record ri in the set of records rthere is either a label of 1 if ri is mentioned in thesummary s, or a 0 if not. The second type of labelis for the chart’s summary, where for each token ti∈ a gold summary s there is a label of 1 if t is alsopresent in some records in r, or a 0 otherwise.
The dataset was divided into training, valida-tion, and testing sets with a 70%:15%:15% ratiofor training. We trained the model for 80 epochswith an epoch size of 1000, using the followinghyper-parameters: 1 encoder layer, 6 decoder lay-ers, embedding size of 512, batch size of 6, beamsize of 4, sinusoidal positional embeddings, andGELU (Hendrycks and Gimpel, 2016) activations.
5 Evaluation
We first perform an automatic evaluation to approx-imately measure the performance of our approachcompared to the base model (Gong et al., 2019) andthen we conduct a human evaluation. Finally, weperform some qualitative analysis to have a better
Summarization Method
Evaluation Method Our Model Gong et al.
BLEU Score 18.54 17.06Content Selection 55.42 (24.06) 8.49 (9.99)
Table 3: Results of automatic evaluation for two sum-marization methods (The content section measure isshown in percentage and the standard deviation is men-tioned in the bracket).
understanding of the effectiveness and trade-offsof our approach.
5.1 Automatic EvaluationFor automated evaluation of our summary quality,we employ two different metrics: (1) the BLEUscore (?), and (2) a content selection metric in-spired by (Wiseman et al., 2017). We calculateour content selection metric as the percentage ofrecords mentioned in the generated summary thatare mentioned in the gold summary.
Table 3 shows the results of automatic evalua-tion of our model compared to the base model. Weobserve that our model slightly improves over thebaseline on the BLEU metric and outperforms iton content selection by a wide margin. In particu-lar, the huge improvement on the content selectionmetric suggests that by learning how to generatesummaries in terms of data variables rather thanactual data values, our approach largely addressesthe hallucination problem.
5.2 Human EvaluationTo further investigate the quality of the generatedtext, we perform a human evaluation. For thispurpose, we randomly sample 40 different chartswhere we have 10 charts from each of four charttypes (simple bar, complex bar, simple line, com-plex line). We use the model from (Gong et al.,2019) trained on our dataset with positional embed-dings enabled as our baseline. We created a Me-chanical Turk study and surveyed three unique re-spondents per statistic. We use the survey to assessthe quality of each summary from four independentperspectives: (1) Informativeness: How informa-tive is the summary of the chart? (2) Concise-ness: How concise is the summary of the chart?,(3) Coherence: How coherent is the summary ofthe chart? and (4) Fluency: How fluent or gram-matically correct are the sentences in the summaryof the chart? The questions were asked using a
144
Summary Method
Evaluation Category Our Model Gong et al.
Informativeness 3.42 (1.16) 2.13 (1.44)Conciseness 3.58 (1.11) 2.40 (1.47)Coherence 3.32 (1.25) 2.27 (1.43)Fluency 3.73 (1.07) 3.78 (0.85)
Table 4: Results of human evaluation (standard devia-tion in brackets)
Summary Method
Response Our Model Gong et al.
Yes 51.02% 22.70%No 26.30% 66.02%Partial 15.56% 7.24%Can’t decide 7.13% 4.04%
Table 5: Responses for factual correctness
5-point Likert scale from 1 (the worst) to 5 (thebest).
As shown in Table 4, on average our modeloutperforms the base model in terms of informa-tiveness, conciseness, and coherence by at leastover 1 point. Overall, it indicates that our methodcan generate summaries that are more insightful,that have better connections between sentences andwith fewer repetitions. In terms of fluency, thebase model performs slightly better (3.78) than ourmodel (3.73).
We also wanted to evaluate the factuality aspectof the generated summaries. For this purpose, weasked respondents to evaluate whether the factsstated in each sentence of the summary were sup-ported by the chart. There were four possible re-sponses to this question: yes, no, partially, andcan’t decide (explain why). Table 5 shows the per-centage of statements within the summary that wereperceived as factually correct or not. We find thatover 50% sentences generated by our model wereverified as factually correct, whereas only 22% sen-tences from the baseline model’s summaries wereperceived to be correct. Another 15.56% state-ments generated from our model were verified aspartially correct compared to 7.24% from the basemodel. This result suggests that our model gener-ates more factually correct statements compared tothe base model.
Figure 4: Comparison of summary generation meth-ods. Here, factually correct statements are highlightedin green and incorrect statements are highlighted in red.
5.3 Case Study
In order to better understand how our approachworks we reviewed one random sample from ourdataset (see Figure 4). We notice that the gold sum-mary is relatively longer than the two generatedsummaries with some additional external facts thatare not mentioned in the chart data (e.g. ‘the NewEngland Patriots’ and ‘their Twitter following’).This illustrates the challenge for generating chartsummaries because without direct connections be-tween the chart and its summary, it becomes moredifficult for the model to learn what is relevant. Ourmodel’s summary is similar in structure to the gold,but without mention of these external statistics.The summary is mostly accurate, as it correctlymentions the most and least followed teams alongwith their corresponding fan counts (highlighted ingreen), but it incorrectly predicts tokens related togame wins, and also incorrectly mentions the thirdhighest team (highlighted in red) right after it cor-
145
Figure 5: Comparison of generated summaries withpositional embeddings enabled and disabled (Here thetext that refers to a bar in the chart is highlighted withthe same fill-color of that bar).
rectly mentioned the lowest team. When analyzingthe summary generated from (Gong et al., 2019),we can readily see that it makes more false predic-tions than our model. It correctly mentions the firstteam name, but then after that it suffers from thehallucination problem as it repeats a memorizedsummary which it was trained on. As a result, thissummary is mostly irrelevant to the given chart.This is also reflected from the automatic evalua-tion (Table 3) where the model from Gong et al.performed poorly on the content selection metric.
We also investigated the impact of positional em-beddings on our model’s performance. As we cansee from Figure 5, for a sample chart the model thatenables positional embeddings yielded summarieswith a more meaningful discourse structure by bet-ter connecting between sentences and with moremeaningful salient information. This support thatpositional embeddings allow the model to track se-quential data as suggested by (Vaswani et al., 2017)for the original Transformer model.
5.4 Error Analysis
While our evaluation reveals that our model is moreeffective at chart summary generation than the basemodel, it still has much room for improvement.In particular, we identify two categories of com-mon errors which are present in our generated sum-
Figure 6: Comparison of generated summaries by ourmodel based on data coverage.
maries. The most common error is the fact halluci-nation, which is demonstrated in Figure 4. Whileour model has largely addressed this issue, irrel-evant tokens still occasionally occur. Our erroranalysis on 50 random samples indicates that hallu-cination can occur in two main ways in our model.First, sometimes the model predicts a data variableat an incorrect index, for example it may try to referto one index of a table row, but the correct index isactually that of another index. The second halluci-nation method is when the model simply predicts atoken which is irrelevant to the data, which we seecommonly in other data-to-text works (Wisemanet al., 2017; Parikh et al., 2020). This error usu-ally occurs when the model generates summariesof charts about a domain that has low coveragein the training set. For example, our training sethas several charts on finance and sports. For thesecharts, the generated summaries are quite fluentand tokens that are irrelevant to the correspondingchart are rare. Figure 6 shows how the generatedsummaries differ in fluency and amount of irrel-evant tokens for two different charts: one from adomain with low training data coverage and theother with high training data coverage.
146
6 Conclusion and Future Work
In this paper, we tackle the challenge of automaticchart summarization by introducing a new datasetand proposing a neural approach based on the trans-former architecture. Our approach learns how togenerate natural language descriptions from chartsby modifying the gold summaries to replace refer-ences to chart data values with data variables. Asa result, the model learns how to summarize in amore generalized way, and generates more factu-ally correct statements compared to the base model.Our model also generates summaries that are moreinformative, concise, and coherent according tohuman evaluations. We hope that our work willmotivate researchers to further improve the qualityof summaries automatically generated from charts,which is a highly under-explored research area.
In the future, we would like to develop largerdatasets that cover more diverse domains and ad-ditional chart types (e.g. pie charts, scatterplots,heatmaps etc.) to further improve the quality andgeneralizability of our model. Also, while wecompared our method with a strong baseline from(Gong et al., 2019), we will perform further com-parisons with other models which were developedfor the data-to-text generation problem. Finally, wewould like to build applications such as an interac-tive chart summarization system with a focus on en-hancing the accessibility of charts so that blind andvisually impaired people can comprehend chartsvia audio.
Acknowledgments
This work was supported by the Natural Sci-ences and Engineering Research Council (NSERC),Canada. We thank Dhruv Nayyar for helping withpreparing the dataset.
ReferencesRegina Barzilay and Mirella Lapata. 2005. Collective
content selection for concept-to-text generation. InProceedings of Human Language Technology Con-ference and Conference on Empirical Methods inNatural Language Processing, pages 331–338, Van-couver, British Columbia, Canada. Association forComputational Linguistics.
Sandra Carberry, Stephanie Elzer, and Seniz Demir.2006. Information graphics: an untapped resourcefor digital libraries. In Proceedings of the 29th an-nual international ACM SIGIR conference on Re-search and development in information retrieval,pages 581–588.
Giuseppe Carenini, Cristina Conati, Enamul Hoque,and Ben Steichen. 2013. User task adaptation inmultimedia presentations. In Proceedings of the 1stInternational Workshop on User-Adaptive Informa-tion Visualization in conjunction with the 21st con-ference on User Modeling, Adaptation and Person-alization (UMAP).
Charles Chen, Ruiyi Zhang, Eunyee Koh, SungchulKim, Scott Cohen, Tong Yu, Ryan A. Rossi, andRazvan C. Bunescu. 2019. Figure captioningwith reasoning and sequence-level training. CoRR,abs/1906.02850.
Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, andWilliam Yang Wang. 2020. Logical natural lan-guage generation from open-domain tables. arXivpreprint arXiv:2004.10404.
Zhe Cui, Sriram Karthik Badam, M Adil Yalcin, andNiklas Elmqvist. 2019. Datasite: Proactive vi-sual data exploration with computation of insight-based recommendations. Information Visualization,18(2):251–267.
Seniz Demir, Sandra Carberry, and Kathleen F. McCoy.2012. Summarizing information graphics textually.Computational Linguistics, 38(3):527–574.
Massimo Fasciano and Guy Lapalme. 1996. Post-graphe: a system for the generation of statisticalgraphics and text. In Eighth International NaturalLanguage Generation Workshop.
Massimo Fasciano and Guy Lapalme. 2000. Intentionsin the coordinated generation of graphics and textfrom tabular data. Knowledge and Information Sys-tems, 2:310–339.
Leo Ferres, Gitte Lindgaard, Livia Sumegi, and BruceTsuji. 2013. Evaluating a tool for improving acces-sibility to charts and graphs. ACM Trans. Comput.-Hum. Interact., 20(5).
Li Gong, Josep M Crego, and Jean Senellart. 2019.Enhanced transformer model for data-to-text genera-tion. In Proceedings of the 3rd Workshop on NeuralGeneration and Translation, pages 148–156.
Nancy L Green, Giuseppe Carenini, Stephan Kerped-jiev, Joe Mattis, Johanna D Moore, and Steven FRoth. 2004. Autobrief: an experimental system forthe automatic generation of briefings in integratedtext and information graphics. International Jour-nal of Human-Computer Studies, 61(1):32–70.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and JianSun. 2016. Deep residual learning for image recog-nition. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 770–778.
Dan Hendrycks and Kevin Gimpel. 2016. Bridgingnonlinearities and stochastic regularizers with gaus-sian error linear units. CoRR, abs/1606.08415.
147
Dae Hyun Kim, Enamul Hoque, and ManeeshAgrawala. 2020. Answering questions about chartsand generating visual explanations. In Proceedingsof the 2020 CHI Conference on Human Factors inComputing Systems, CHI 20, page 1–13.
Remi Lebret, David Grangier, and Michael Auli. 2016.Neural text generation from structured data withapplication to the biography domain. In Proceed-ings of the 2016 Conference on Empirical Methodsin Natural Language Processing, pages 1203–1213,Austin, Texas. Association for Computational Lin-guistics.
Percy Liang, Michael I Jordan, and Dan Klein. 2009.Learning semantic correspondences with less super-vision. In Proceedings of the Joint Conference ofthe 47th Annual Meeting of the ACL and the 4th In-ternational Joint Conference on Natural LanguageProcessing of the AFNLP, pages 91–99.
Christopher D. Manning, Mihai Surdeanu, John Bauer,Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural lan-guage processing toolkit. In Association for Compu-tational Linguistics (ACL) System Demonstrations,pages 55–60.
Hongyuan Mei, TTI UChicago, Mohit Bansal, andMatthew R Walter. 2016. What to talk about andhow? selective generation using lstms with coarse-to-fine alignment. In Proceedings of NAACL-HLT,pages 720–730.
Vibhu O. Mittal, Johanna D. Moore, GiuseppeCarenini, and Steven Roth. 1998. Describing com-plex charts in natural language: A caption genera-tion system. Computational Linguistics, 24(3):431–467.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann,Manaal Faruqui, Bhuwan Dhingra, Diyi Yang,and Dipanjan Das. 2020. Totto: A controlledtable-to-text generation dataset. arXiv preprintarXiv:2004.14373.
Ratish Puduppully, Li Dong, and Mirella Lapata. 2019.Data-to-text generation with content selection andplanning. In Proceedings of the AAAI Conference onArtificial Intelligence, volume 33, pages 6908–6915.
Quill. 2020. Narrative science.
Ehud Reiter. 2007. An architecture for data-to-textsystems. In Proceedings of the Eleventh EuropeanWorkshop on Natural Language Generation, pages97–104. Association for Computational Linguistics.
Ehud Reiter, Somayajulu Sripada, Jim Hunter, Jin Yu,and Ian Davy. 2005. Choosing words in computer-generated weather forecasts. Artificial Intelligence,167(1-2):137–169.
Arjun Srinivasan, Steven M Drucker, Alex Endert, andJohn Stasko. 2018. Augmenting visualizations withinteractive data facts to facilitate interpretation and
communication. IEEE transactions on visualizationand computer graphics, 25(1):672–681.
Ashish Vaswani, Noam Shazeer, Niki Parmar, JakobUszkoreit, Llion Jones, Aidan N Gomez, ŁukaszKaiser, and Illia Polosukhin. 2017. Attention is allyou need. In Advances in neural information pro-cessing systems, pages 5998–6008.
Yun Wang, Zhida Sun, Haidong Zhang, Weiwei Cui,Ke Xu, Xiaojuan Ma, and Dongmei Zhang. 2019.Datashot: Automatic generation of fact sheets fromtabular data. IEEE transactions on visualization andcomputer graphics, 26(1):895–905.
Sam Wiseman, Stuart M Shieber, and Alexander MRush. 2017. Challenges in data-to-document gen-eration. arXiv preprint arXiv:1707.08052.
Wordsmith. 2020. Wordsmith by automated insights,inc.
Zichao Yang, Phil Blunsom, Chris Dyer, and WangLing. 2017. Reference-aware language models. InProceedings of the 2017 Conference on EmpiricalMethods in Natural Language Processing, pages1850–1859.