cnrs/loria, nancy university of edinburgh claire gardent ... · deep learning approaches to text...
TRANSCRIPT
![Page 1: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/1.jpg)
Deep Learning Approaches to Text ProductionClaire Gardent Shashi Narayan
CNRS/LORIA, Nancy University of Edinburgh
AIPHES Darmstadt, 14 September 2018
![Page 2: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/2.jpg)
Tutorial Outline ● Text production and its relevance
● Why deep learning for text production?
![Page 3: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/3.jpg)
Text Production: From What and What for?
Communicative Goal
Input
DatabasesKnowledge bases
Dialog actsSentences
Documents...
VerbaliseRespond
SummarizeSimplify
...
![Page 4: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/4.jpg)
Text Production: From What and What for?
Communication Goal
Input
DatabasesKnowledge bases
Dialog actsSentences
Documents...
VerbaliseRespond
SummarizeSimplify
...
● Meaning representations ● Data● Text
![Page 5: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/5.jpg)
Meaning Representation to Text Production
Communication Goal
Input
AMRsDependency trees
Dialog actsLambda-terms
...
Verbalise
![Page 6: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/6.jpg)
Generating from Dependency Trees
Surface Realization Challenge 2011 and 2018
● Shallow and deep approaches● Universal dependency trees
Bills on immigration were submitted by Senator Brownback, a Republican of Kansas.
![Page 7: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/7.jpg)
Generating from Abstract Meaning Representations
SemEval Shared Task 2017: AMR Generation and Parsing
A boy wants to visit New York City.A boy wanted to visit New York City.
![Page 8: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/8.jpg)
Generating from Dialog Moves
Mairesse and Young 2014
Wen et al. 2015, 2016
End-to-End Natural Language Generation Challenge 2017
Input:name[The Eagle],eatType[coffee shop],food[French],priceRange[moderate],customerRating[3/5],area[riverside],kidsFriendly[yes],near[Burger King]
Output: “The three star coffee shop, The Eagle, gives families a mid-priced dining experience featuring a variety of wines and cheeses. Find The Eagle near Burger King.”
![Page 9: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/9.jpg)
Data to Text Production
Communication Goal
InputDatabases
Knowledge bases...
VerbaliseSummariseCompare
![Page 10: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/10.jpg)
Generating from Knowledge Bases (RDFs)The WebNLG Challenge 2017 (John_E_Blaha birthDate
1942_08_26) (John_E_Blaha birthPlace San_Antonio) (John_E_Blaha occupation Fighter_pilot)
“John E Blaha, born in San Antonio on 1942-08-26, worked as a fighter pilot.”
![Page 11: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/11.jpg)
Generating from Databases
Angeli, Liang and Klein 2010. Konstas and Lapata 2012
![Page 12: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/12.jpg)
Generating from Data to Documents
Wiseman, Shieber and Rush 2017
![Page 13: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/13.jpg)
Generating from Loosely Aligned Data
Perez and Lapata, NAACL 2018
Robert Joseph Flaherty, (February 16, 1884 July 23, 1951) was an American film-maker. Flaherty was married to Frances H. Flaherty until his death in 1951.
![Page 14: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/14.jpg)
Other Data Modalities to Text ProductionGenerating Image Captions
Vinyals et al. 2015
![Page 15: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/15.jpg)
Text to Text Production
Communication Goal
InputDialog turns
SentencesDocuments
...
SummarizeSimplify
ParaphraseCompressRespond
![Page 16: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/16.jpg)
Text to Text Production
![Page 17: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/17.jpg)
Sentence Simplification
Complex: In 1964 Peter Higgs published his second paper in Physical Review Letters describing Higgs mechanism which predicted a new massive spin-zero boson for the first time.
Simple: Peter Higgs wrote his paper explaining Higgs mechanism in 1964. Higgs mechanism predicted a new elementary particle.
![Page 18: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/18.jpg)
Paraphrasing and Question Answering
![Page 19: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/19.jpg)
Abstract Generation
![Page 20: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/20.jpg)
Story Highlights Generation
Headline or Title Generation
![Page 21: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/21.jpg)
Multi-document Summarization
![Page 22: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/22.jpg)
Conversational Agents
Li, Monroe, Ritter, Galley, Gao and Jurafsky 2016
![Page 23: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/23.jpg)
Summary
● Many different inputs
Data, Meaning Representations, Text
● Many different communicative goals
Verbalise, summarise, compress, simplify, respond, compare ….
![Page 24: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/24.jpg)
Pre-neural Approaches
![Page 25: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/25.jpg)
Previous Approaches to Text ProductionData-to-Text Generation
Simplification, Compression and Paraphrasing
Summarisation
Content Selection
Text Planning Sentence Planning
Split Rewrite Reorder Delete
Content Selection Aggregation Generalisation
![Page 26: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/26.jpg)
The Data-to-Text Generation Pipeline
(Figure from Johanna Moore)
ffffff
![Page 27: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/27.jpg)
The Data-to-Text Generation PipelinePros
Models the various choices which need to be
made during Generation
Cons
● Many modules to implement
● Error propagation
● Difficult to capture the interactions
between the various choices (joint
learning)
![Page 28: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/28.jpg)
Simplification, Compression and Paraphrasing
Four main operations
● Delete● Reorder● Rewrite● Split
![Page 29: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/29.jpg)
Simplification, Compression and ParaphrasingIn 1964 Peter Higgs published his second paper in Physical Review Letters describing Higgs mechanism [ which ] predicted a new massive spin-zero boson for the first time.
Peter Higgs wrote his paper explaining Higgs mechanism in 1964.
Higgs mechanism predicted a new elementary particle.
REORDER
![Page 30: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/30.jpg)
Simplification, Compression and ParaphrasingIn 1964 Peter Higgs published his second paper in Physical Review Letters describing Higgs mechanism [ which ] predicted a new massive spin-zero boson for the first time.
Peter Higgs wrote his paper explaining Higgs mechanism in 1964.
Higgs mechanism predicted a new elementary particle.
SPLIT
![Page 31: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/31.jpg)
Simplification, Compression and ParaphrasingIn 1964 Peter Higgs published his second paper in Physical Review Letters describing Higgs mechanism [ which ] predicted a new massive spin-zero boson for the first time.
Peter Higgs wrote his paper explaining Higgs mechanism in 1964.
Higgs mechanism predicted a new elementary particle.
DELETE
![Page 32: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/32.jpg)
Simplification, Compression and ParaphrasingIn 1964 Peter Higgs published his second paper in Physical Review Letters describing Higgs mechanism [ which ] predicted a new massive spin-zero boson for the first time.
Peter Higgs wrote his paper explaining Higgs mechanism in 1964.
Higgs mechanism predicted a new elementary particle.
REWRITE
![Page 33: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/33.jpg)
Simplification, Compression and ParaphrasingPros
● Fine grained control over the four
operations
Cons
● Hard to capture the interactions
between operations
![Page 34: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/34.jpg)
Summarization
![Page 35: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/35.jpg)
Summarization
![Page 36: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/36.jpg)
SummarizationPros
● Well-formed (grammatical)
summaries
● Fast
● Reasonable performance
Cons
● Extractive summaries are
still very different from
human written summaries
![Page 37: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/37.jpg)
Questions ?
![Page 38: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/38.jpg)
Neural Text Production
● A single framework for all text production tasks
● End-to-end
![Page 39: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/39.jpg)
Deep Learning: A Uniform Framework for Text Production
RepresentationLearning Generation
input
![Page 40: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/40.jpg)
Deep Learning: A Uniform Framework for Text Production
Generation
How are you doing SentencesDocumentsDialog turns
![Page 41: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/41.jpg)
Deep Learning: A Uniform Framework for Text Production
How are you doing SentencesDocumentsDialog turns
<START> Fine , and
Fine , and you
![Page 42: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/42.jpg)
Deep Learning: A Uniform Framework for Text Production
boy arg0-want arg0-visit arg1-NY AMRsKBs
Databases
<START> A boy wants
A boy wants to
![Page 43: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/43.jpg)
Encoder-Decoder Model for Text Production
Cho et al. 2014, Sutskever et al. 2014
Encoder Decoder

![Page 44: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/44.jpg)
Encoding Input Representations using Recurrent Neural Networks (RNN)
How are you doing input
unrolled
Input representation
Encoding variable length inputs
![Page 45: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/45.jpg)
Encoding Representations using an RNN
st-1 st
xt
The encoder takes as input the previous state st-1 and the previously generated token yt-1
V
U
xt-1
![Page 46: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/46.jpg)
Decoding Representations using an RNN
st-1 st
yt-1 yt
At each time setp, the decoder outputs a probability distribution over the possible outputs (target vocabulary)
W
V
U
yt-2
![Page 47: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/47.jpg)
Generating Text using RNNs
InputHow are you doing?
EncoderInput
representation
![Page 48: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/48.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
vocabulary
Fine
Conditional Generation
![Page 49: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/49.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
vocabularyFine
Fine
softmax
,
![Page 50: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/50.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
vocabularyFine
Fine
softmax
,
,
softmax
and
![Page 51: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/51.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
vocabularyFine
Fine
softmax
,
,
softmax
and
and
softmax
you
![Page 52: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/52.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
Fine
Fine
softmax
,
,
softmax
and
and
softmax
you
you
softmax
?
![Page 53: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/53.jpg)
Generating Text using RNNs
InputHow are you doing?
Encoder
<s>
softmax
Fine
Fine
softmax
,
,
softmax
and
and
softmax
you
you
softmax
?
?
softmax
<eos>
![Page 54: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/54.jpg)
RNN and Long Distance Dependencies● In practice, RNN cannot handle long input because of the Vanishing and
Exploding Gradients issue [Bengio et al. 1994]
The yogi, who had done many sun salutations, was happy.
● LSTM, GRU are alternative recurrent networks which helps learning long distance dependencies
![Page 55: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/55.jpg)
Long Short Term Memory networks (LSTMs)
update
forget
output
ct-1
ht-1
![Page 56: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/56.jpg)
Long Short Term Memory networks (LSTMs)
update
forget
output
Update candidate context
![Page 57: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/57.jpg)
Long Short Term Memory networks (LSTMs)
update
forget
output
Forget Ct-1
![Page 58: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/58.jpg)
Long Short Term Memory networks (LSTMs)
update
forget
output
Create ct
![Page 59: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/59.jpg)
Long Short Term Memory networks (LSTMs)
update
forget
output
Output ht
![Page 60: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/60.jpg)
Gated Recurrent Units (GRUs)
Candidate state
Zt = 1, output candidate state (forget)Zt = 0, output
previous state (memorize)
![Page 61: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/61.jpg)
Bidirectional Recurrent Neural Networks
Ich habe keine Zeit I don’t have time
Er schwieg eine Zeit lang He was silent for a while
![Page 62: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/62.jpg)
Bidirectional Recurrent Neural Networks
eine Zeit lang
Input representation
● Forward RNN encodes left context● Backward RNN encodes right
context● Forward and backward states are
concatenated
![Page 63: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/63.jpg)
Summary: Generating Text using RNNs
A conditional language model with no markov assumption
![Page 64: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/64.jpg)
Summary: Generating Text using RNNs
A conditional language model with no markov assumption
![Page 65: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/65.jpg)
Summary: Generating Text using RNNs
A conditional language model with no markov assumption
![Page 66: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/66.jpg)
Summary: Generating Text using RNNsLearn to predict from a fixed vocabulary distribution
Cross Entropy Loss
![Page 67: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/67.jpg)
Advantages over Pre-Neural Text Production Approaches
● End-to-end training
● Robustness and better generalization
● Leaving behind symbolic representations
![Page 68: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/68.jpg)
Problems and Methods in Neural Text Production
● How to produce adequate texts ? ● How to model various inputs ? ● How to guide learning towards communication goals ? ● Datasets, Challenges and Open Issues
Decoding
Encoding
![Page 69: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/69.jpg)
Producing Adequate Text
![Page 70: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/70.jpg)
Some Problems with Neural Generation
ACCURACY: The output text sometimes contains information that does not match the input.
REPETITIONS: The output text sometimes contains repetitions
COVERAGE: The output text sometimes does not cover all the input
RARE OR UNKNOWN WORDS
DISFLUENT OUTPUT
![Page 71: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/71.jpg)
Example from T2T Generation (Kiddon, 2016) Title: Almond-raspberry thumbprint cookies DeletedIngredients: 1 cup butter or margarine softened, 1 cup sugar, 1 can solo almond filling, 2 egg yolks, 1 tsp almond extract, 2 1/2 cups all-purpose flour, 1/2 tsp baking powder, 1/2 tsp salt, 1 can solo raspberry filling
In a large bowl, beat butter and sugar until light and fluffy. Add eggs, one at a time, beating well after each addition. Blend in flour, baking powder, baking soda, salt and baking soda. Add to creamed mixture and mix well. Stir in nuts and vanilla. Drop dough by rounded table spoonfuls 2 ” apart onto ungreased cookie sheets. Bake 8 to 10 minutes or until edges are lightly browned. Cool on wire racks.
Added Repeated
![Page 72: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/72.jpg)
Example from Summarization Lagos nigeria (CNN). a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at
democracy in africa’s most populous nation.RARE WORD ADDED
Seq2Seq + Attention: UNK UNK says his administration is confident it will be able to destabilize nigeria’s economy. UNK says his administration is confident it will be able to thwart criminals and other nigerians. he says the country has long nigeria and nigeria’s economy.
Pointer-Gen: muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals. See et al. 2017
![Page 73: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/73.jpg)
Example from GenerationREF: A technical committee of indian missile experts stated that the equipement was unimpeachable and irrefutable evidence of a plan to transfer not just missiles but missile-making capabilities.
SYS: A technical committee expert on the technical committee stated that the equipment is not impeached but it is not refutes.
DELETED
DiSFLUENT
ADDED
![Page 74: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/74.jpg)
Attention, Copy and Coverage ATTENTION
○ To improve accuracy
COPY
○ To handle rare or unknown words
○ To copy from the input
COVERAGE
○ To help cover all and only the input
○ To avoid repetitions
![Page 75: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/75.jpg)
Encoder-Decoder without Attention
boy arg0-want arg0-visit arg1-NY AMRsKBs
Databases
<START> A boy wants
A boy wants to
● The input is compressed into a fixed-length vector● Performance decreases with the length of the input [Sutskever
et al. 2014].
![Page 76: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/76.jpg)
Encoder-Decoder with Attention
Takes as input the previous state st-1 , the previously generated token yt-1 and a context vector ct
This context vector
● depends on the previous state and therefore changes at each step● Indicates which part of the input is most relevant to the decoding step
![Page 77: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/77.jpg)
Encoder-Decoder with Attention
h1
st-1st
yt-1
yt-1
yt
h2
h4
h3
+
The context vector ct provides a representation of the input weighted by similarity with the current state.
It shows which part of the input is similar to the current decoding state
Badhanau et al. 2014 Encoder
Decoder
![Page 78: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/78.jpg)
CopyMotivation
● To copy from the input ● To handle rare or unknown words
Method
● Output words are taken either from the target vocabulary or from the input
● At each time step, the model decides whether to copy from the input or to generate from the target vocabulary
![Page 79: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/79.jpg)
Copying vs. GeneratingGeneration Probability
Probability of outputing word w
Vocabulary distribution0 if w is not in VOCAB
Attention distribution0 if w is not in Input
![Page 80: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/80.jpg)
Copy and Generate Lagos nigeria (CNN). a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s
most populous nation.
Pointer-Gen: muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals.
See et al. 2017
![Page 81: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/81.jpg)
Copy and generate in Text ProductionParaphrasing and Simplification: [Ciao et al. AAAI 2017].
Text Summarisation: [Gu, Lu, Li, Li, ACL 2016], [Gulcehre, Ahn, Nallapati, Zhou, Bengio, ACL 2016]
Extractive Summarisation: [Cheng and Lapata. ACL 2016].
Answer Generation: [He, Liu, Liu, Zhao ACL 2017]
![Page 82: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/82.jpg)
Copy, Delexicalisation and Character-Based RNNThe COPY mechanism helps handling rare or unknown words (proper names, dates)
Alternative approaches for handling these are
● Delexicalisation● Character-Based Encoders
![Page 83: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/83.jpg)
Delexicalisation Slot values occurring in training utterances are replaced with a placeholder token representing the slot
At generation time, these placeholders are then copied over from the input specification to form the final output
Wen et al. 2015, 2016
![Page 84: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/84.jpg)
Delexicalisationinform(restaurant name = Au Midi, neighborhood= midtown, cuisine = french)
Au Midi is in Midtown and serves French food.
inform(restaurant name = restaurant name, neighborhood= neighborhood, cuisine = cuisine)
restaurant name is in neighborhood and serves cuisine food.
![Page 85: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/85.jpg)
Character-Based Encoding Uses the open source tf-seq2seq framework to train a char2char model on the E2E NLG Challenge data.
No delexicalization, lowercasing or even tokenization
Input semantics = sequence of characters
Human evaluation shows that
● The output is grammatically perfect● The model does not generate non words
Agarwal et al. 2017
![Page 86: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/86.jpg)
CoverageProblem: Neural models tend to omit or repeat information from the input
Solution
● Use coverage as extra input to attention mechanism● Coverage: cumulative attention, what has been attended to so far● Penalise attending to input that has already been covered
Tu et al. 2017
![Page 87: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/87.jpg)
Coverage in SummarisationA Coverage Vector captures how much attention each input words has received
The attention mechanism is modified to take coverage into account
The loss is modified to penalise any overlap between the coverage vector and the attention distribution
![Page 88: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/88.jpg)
Summarising with coverageLagos nigeria (CNN). a day after winning nigeria’s presidency, muhammadu buhari told cnn’s christiane amanpour that he plans to aggressively fight corruption that has long plagued nigeria and go after the root of the nation’s unrest. buhari said he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria, where the terrorist group boko haram operates. by cooperating with neighboring nations chad, cameroon and niger, he said his administration is confident it will be able to thwart criminals and others contributing to nigeria’s instability. for the first time in nigeria’s history, the opposition defeated the ruling party in democratic elections. buhari defeated incumbent goodluck jonathan by about 2 million votes, according to nigeria’s independent national electoral commission. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s
most populous nation.
Pointer-Gen: muhammadu buhari says he plans to aggressively fight corruption in the northeast part of nigeria. he says he’ll “rapidly give attention” to curbing violence in the northeast part of nigeria. he says his administration is confident it will be able to thwart criminals.
Pointer-Gen-Cov: muhammadu buhari says he plans to aggressively fight corruption that has long plagued nigeria. he says his administration is confident it will be able to thwart criminals. the win comes after a long history of military rule, coups and botched attempts at democracy in africa’s most populous nation.
![Page 89: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/89.jpg)
Coverage successfully eliminates repetitions.
The proportion of duplicate n-grams is similar in the reference summaries and in the summaries produced by the model with coverage.
![Page 90: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/90.jpg)
Coverage in Dialog: SC-LSTM
Wen et al. 2015
![Page 91: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/91.jpg)
Coverage in Dialog: SC-LSTM
Updating the Dialog Move
![Page 92: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/92.jpg)
Coverage in Dialog: SC-LSTM
Conditioning Generation on the dynamically updated Dialog Move
![Page 93: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/93.jpg)
Reading Gate in Action
![Page 94: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/94.jpg)
Text Production with Better Input Representations
![Page 95: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/95.jpg)
Deep Learning: A Uniform Framework for Text Production
RepresentationLearning Generation
input
![Page 96: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/96.jpg)
Deep Learning: A Uniform Framework for Text Production
RepresentationLearning Generation
input
AttentionCopy Coverage
![Page 97: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/97.jpg)
Deep Learning: A Uniform Framework for Text Production
RepresentationLearning Generation
input
Learning representations better suited for Input and Communication Goal
![Page 98: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/98.jpg)
Taking Structure into accountText structure: Abstractive and extractive summarisation
● Hierarchical encoders● Ensemble encoders● Convolutional sentence encoders
Data structure: MR- and data-to-text Generation
● Graph to sequence (AMR to text)● Graph-Based Triple Encoder (RDF to text)● Graph Convolutional Networks
![Page 99: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/99.jpg)
Modeling Sentences as a sequence of Tokens
Generation
How are you doing Sentences
Dialog turns
● Sentence Simplification (Zhang and Lapata, 2017)
● Paraphrasing (Mallinson at al. 2017)
● Sentence Compression (Filippova et al. 2015)
● Conversation Model (Li et al., 2016)
![Page 100: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/100.jpg)
Modeling Document as a sequence of Tokens
Generation
a sequence of wordsDocument
Abstractive Document Summarization (Nallapati et al. 2016, See et al. 2017, Paulus et al. 2017, Pasunuru and Bansal 2018)
![Page 101: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/101.jpg)
Modeling Document as a sequence of Tokens
![Page 102: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/102.jpg)
Modeling Document as a sequence of Tokens
Bidirectional LSTMs to encode document as sequence of words
![Page 103: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/103.jpg)
Modeling Document as a sequence of TokensSimple sequential encoder
Sequential Generators with copy, coverage and attention
Ignores the hierarchical structure of a document
Issues with long range dependencies
![Page 104: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/104.jpg)
Hierarchical Document Encoders
Modeling document with sentence encoders (Cheng and Lapata, 2016, Tan et al. 2017, Narayan et al. 2018)
Modeling document with paragraph encoders (Celikyilmaz et al. 2018)
![Page 105: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/105.jpg)
Modeling Documents with Hierarchical LSTMs
Abstractive Document Summarization (Tan et al. ACL 2017)
![Page 106: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/106.jpg)
Modeling Documents with Hierarchical RNNs
Sentence as a sequence of words
Document as a sequence of sentences
![Page 107: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/107.jpg)
Modeling Documents with Hierarchical LSTMs
Standard Attention Mechanism
![Page 108: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/108.jpg)
Modeling Documents with Hierarchical RNNs
Hierarchical Generation
![Page 109: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/109.jpg)
Modeling Documents with Hierarchical RNNs
Abstractive Document Summarization (Tan et al. ACL 2017)
Without a pointer generator mechanism, model suffers from generating out-of-vocabulary words.
It performs inferior to (See at al, ACL 2017).
![Page 110: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/110.jpg)
Modeling Document with Ensemble Encoders
Abstractive Document Summarization (Celikyilmaz et al. NAACL 2018)
![Page 111: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/111.jpg)
Modeling Document with Ensemble Encoders
Multiple RNN encoders, each modeling a separate paragraph
![Page 112: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/112.jpg)
Modeling Document with Ensemble Encoders
Multi-encoder message passing
![Page 113: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/113.jpg)
Modeling Document with Ensemble Encoders
Multi-encoder message passing
Decoding with Agent Attention
![Page 114: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/114.jpg)
Modeling Document with Ensemble Encoders
Multi-encoder message passing
Decoding with Agent Attention
Pointer generator
![Page 115: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/115.jpg)
Multi-Encoder Message Passing
![Page 116: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/116.jpg)
Modeling Document with Ensemble Encoders
Decoding with Agent Attention
![Page 117: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/117.jpg)
Decoding with Hierarchical AttentionWord attention distribution for each paragraph
Decoder context
Document global agent attention distribution
Agent context vector
Final vocabulary distribution
![Page 118: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/118.jpg)
Modeling Document with Ensemble Encoders
Pointer generator
![Page 119: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/119.jpg)
Modeling Document with Ensemble Encoders
Abstractive Document Summarization (Celikyilmaz et al. ACL 2018)
Model achieves state-of-the-art performance outperforming (See at al, ACL 2017, Tan et al, ACL 2017).
![Page 120: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/120.jpg)
Modeling Document With Convolutional Sentence Encoders
![Page 121: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/121.jpg)
Modeling Document With Convolutional Sentence EncodersConvolutional
SentenceEncoder
![Page 122: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/122.jpg)
Modeling Document With Convolutional Sentence EncodersConvolutional
SentenceEncoder
Document as a sequence of sentences
![Page 123: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/123.jpg)
Extractive Summarization
Sentence Extraction
(Narayan et al., NAACL 2018)
![Page 124: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/124.jpg)
Convolutional Sentence Encoders
Suited for document summarization for capturing salient information
Issues with long range dependencies reduced
Not clear how to utilize convolutional sentence encoders for abstractive summarization
![Page 125: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/125.jpg)
Extractive Summarization with Convolutional Sentence Encoders
Model achieves state-of-the-art performance for extractive summarization
Extractive Document Summarization (Narayan et al., NAACL 2018)
![Page 126: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/126.jpg)
Modeling Graph as a sequence of Tokens
Generation
give :arg0-i :arg1-ball :arg2-dogAMRs
RDFs, KBsDatabases AMR
![Page 127: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/127.jpg)
Modeling Graph as a sequence of Tokens
Generation
● AMR Generation (Konstas et al, 2017, Cao and Clark, 2018)● RDF Generation (The WebNLG Challenge, Gardent et al. 2017)
give :arg0-i :arg1-ball :arg2-dogAMRs
RDFs, KBsDatabases
![Page 128: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/128.jpg)
Modeling Graphs as Sequence of TokensLINEARISATION
Alan_Bean mission Apollo_12 Apollo_12 crewMember Peter_Conrad Apollo_12 operator Nasa Alan_Bean birthDate 1932-03-15 Alan_Bean birthPlace Wheeler_Texas Wheeler_Texas country USA
D2T Generation (Data = RDF)
Creating Training Corpora for NLG Micro-Planners (Gardent et al. 2017)
![Page 129: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/129.jpg)
Modeling Graphs as Sequence of Tokens
Image from : Neural AMR: Sequence-to-Sequence Models for Parsing and Generation (Konstas et al, 2017)
MR-to-Text Generation
(MR = Abstract Meaning Representations)
![Page 130: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/130.jpg)
Problems with Graph Linearization
Local dependencies available in the input turned into long-range dependencies
RNNs often have trouble modeling long-range dependencies
![Page 131: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/131.jpg)
Modeling with Graph Encoders
AMR Generation: A Graph-to-Sequence Model for AMR-to-Text Generation (Song et al., ACL 2018)
RDF Generation: GTR-LSTM: A Triple Encoder for Sentence Generation from RDF Data (Trisedya et al. ACL 2018)
Graph Convolutional Networks for SRL and NMT (Kipf and Welling 2017, Marcheggiani and Titov, 2017, Bastings et al. 2017)
![Page 132: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/132.jpg)
Graph-to-Sequence Model for AMR Generation
Ryan’s description of himself: a genius.
![Page 133: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/133.jpg)
Graph-to-Sequence Model for AMR GenerationAt each time step:
● Encoder operates directly on the graph structure of the input
● Node representations are updated using their dependents in the graph
![Page 134: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/134.jpg)
Graph-to-Sequence Model for AMR Generation
For node , we define incoming and outgoing input representations:
is an input representation for edge .
![Page 135: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/135.jpg)
Graph-to-Sequence Model for AMR Generation
For node , we define incoming and outgoing hidden state representations:
![Page 136: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/136.jpg)
Graph-to-Sequence Model for AMR GenerationGraph state transition
![Page 137: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/137.jpg)
Graph-to-Sequence Model for AMR GenerationGraph state transitions
LSTM
Input gate
Output gate
Forget gate
![Page 138: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/138.jpg)
Graph-to-Sequence Model for AMR Generationt=0: Represents each node itself
t=1: Represents nodes with their immediate parents and children
t=2: Represents nodes with their grandparents and grandchildren
t=longest path length: Represents nodes with whole graph knowledge
![Page 139: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/139.jpg)
Decoding with Graph-to-Sequence Model
● Standard attention-based LSTM decoder
● Decoder initial state: Average of the last states of all
nodes
● Standard attention and copy mechanism can be used
on the last states of all nodes
![Page 140: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/140.jpg)
Graph-to-Sequence Model for AMR Generation
A Graph-to-Sequence Model for AMR-to-Text Generation (Song et al., 2018)
Model achieves state-of-the-art performance outperforming (Konstas et al. 2017) for AMR Generation.
![Page 141: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/141.jpg)
Graph-based Triple Encoder for RDF Generation
![Page 142: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/142.jpg)
Graph-based Triple Encoder for RDF Generation
![Page 143: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/143.jpg)
Graph-based Triple Encoder for RDF Generation
● Traverses the input graph
● When a vertex is visited, the hidden states of adjacent vertices are created
● Each GTR-LSTM unit receives an entity and the incoming property
![Page 144: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/144.jpg)
GTR-LSTM: Triple Encoder for RDF Generation
GTR-LSTM: A Triple Encoder for Sentence Generation from RDF Data. (Trisedya et al. ACL 2018)
Model holds current state-of-the-art performance for RDF Generation on the WebNLG Dataset
![Page 145: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/145.jpg)
Graph Convolutional Networks
Encoding dependency structures for SRL and NMT
(Kipf & Welling 2017)
![Page 146: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/146.jpg)
Graph Convolutional Networks
● Semantic Role Labeling: (Marcheggiani and Titov 2017).
● Syntax-aware Neural Machine Translation: (Bastings et al. 2017)
● AMR or RDF Generation?
(Kipf & Welling, 2017)
![Page 147: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/147.jpg)
Summary: Input Representation and Text Production Hierarchical document encoders and graph encoders are able to better model input for text production
State of art results on Summarization, AMR generation and RDF generation
Many more to come...
![Page 148: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/148.jpg)
Communication Goal-Oriented Deep Generators
![Page 149: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/149.jpg)
Communication Goal-Oriented Deep Generators
Infusing task-specific knowledge to deep architectures
Reinforcement Learning: Optimizing final evaluation metric
![Page 150: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/150.jpg)
Infusing Task-Knowledge to Deep Architectures
Similarity with Machine Translation: Text production tasks such as paraphrase generation and AMR generation often have semantic equivalence between source and target sides.
However, it is not true for text production in general:
● Sentence Compression or Simplification● Generation from noisy data● Document Summarization● Conversational Agents
![Page 151: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/151.jpg)
Summarisation
Identify Key Source Information
Nallapati et al. CONLL 2016: Linguistic features
Zhou et al. ACL 2017: Selective Encoding
Tan et al. 2017: Modified Attention
![Page 152: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/152.jpg)
Sentence Summarization, Sentence Compression or Title Generation
Abstractive Sentence Summarization (Zhou et al. ACL 2017)
![Page 153: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/153.jpg)
Selective Encoding to Capture Salient Information
● Create a sentence representation tailored to highlight key information
● Decode using this tailored sentence representation
Zhou, Yang, Wei and Zhou. ACL 2017
Goal: Select Key Input Words
![Page 154: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/154.jpg)
Encoder : Two LSTMs and a Selective Gate
Input Sentence and Word Representation
![Page 155: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/155.jpg)
Selective Encoding
![Page 156: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/156.jpg)
Selective Encoding to Capture Salient Information
Word Repr.
Sentence Repr.
Updated Word Repr.
![Page 157: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/157.jpg)
Selective Encoding to Capture Salient Information
Input Sentence and Word Representation
Tailored sentence representation
![Page 158: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/158.jpg)
Selective Encoding to Capture Salient Information
Input: The Council of Europe’s human rights commissioner slammed thursday as “unacceptable” conditions in France’s overcrowded and dilapidated jails, where some ## inmates have committed suicide this year.
Output Summary: Council of Europe slams French prisons conditions
Reference summary: Council of Europe again slams French prisons conditions
![Page 159: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/159.jpg)
Document Summarization
Summarization requires model to distinguish the content that is relevant for the summary from the content that is not
Abstractive Document Summarization (Tan et al. ACL 2017)
![Page 160: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/160.jpg)
Document Summarization with Modified Attention
Identifying Salient sentences with topic-sensitive PageRank
(Tan, Wan and Xiao. ACL 2017)
A sentence is important in a document if it is heavily linked with many important sentences.
![Page 161: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/161.jpg)
Summarisation
Identify Correct Key Source Information
Cao et al. AAAI 2018:
Fact Extraction
Dual Encoder and Attention Mechanism
![Page 162: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/162.jpg)
Faifthful to the original
FTSum: 30% of the output summaries contain incorrect information
Input: The repatriation of at least #,### bosnian moslems was postponed friday after the unhcr pulled out of the first joint scheme to return refugees to their homes in northwest bosnia .
Output Summary: bosnian moslems postponed
Reference summary: repatriation of bosnian moslems postponed
(Cao, Wei, Li and Li, AAAI 2018)
![Page 163: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/163.jpg)
Faifthful to the original
FTSum: 30% of the output summaries contain incorrect information
Input: The repatriation of at least #,### bosnian moslems was postponed friday after the unhcr pulled out of the first joint scheme to return refugees to their homes in northwest bosnia .
Output Summary: bosnian moslems postponed
Reference summary: repatriation of bosnian moslems postponed
(Cao, Wei, Li and Li, AAAI 2018)
Goal: Select Correct Information
![Page 164: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/164.jpg)
Faifthful to the originalProposal: Use facts to improve semantic adequacy
Input: The repatriation of at least #,### bosnian moslems was postponed
friday after the unhcr pulled out of the first joint scheme to return refugees to
their homes in northwest bosnia .
Facts: unhcr pulled out of first joint scheme
repatriation was postponed friday
unhcr return refugees to their homes
![Page 165: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/165.jpg)
Faithful to the original
(Cao, Wei, Li and Li, AAAI 2018)
![Page 166: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/166.jpg)
Faithful to the original
(Cao, Wei, Li and Li, AAAI 2018)
![Page 167: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/167.jpg)
D2T Generation
Align Text and Data
Perez-Beltrachini et al. NAACL 2018
Encode data and text into common space to learn similarity function (alignment) between data and text
Multi Task Learning
Reinforcement Learning
![Page 168: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/168.jpg)
Generation from Loosely Aligned Noisy Data
Robert Joseph Flaherty, (February 16, 1884 July 23, 1951) was an American film-maker. Flaherty was married to Frances H. Flaherty until his death in 1951.
(Perez-Beltrachini and Lapata, NAACL 2018)
![Page 169: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/169.jpg)
Generation from Loosely Aligned Noisy DataRobert Joseph Flaherty, (February 16, 1884 July 23, 1951) was an American film-maker. Flaherty was married to Frances H. Flaherty until his death in 1951.
(Perez-Beltrachini and Lapata, NAACL 2018)
Goal: Select Key Input Facts
![Page 170: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/170.jpg)
Generation with Multi-task Objective
Word prediction (generation) objective:
Content selection objective:
Multi-Task Learning:
![Page 171: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/171.jpg)
Generation from Loosely Aligned Noisy Data
Experimental results show that models trained with content-specific objectives improve upon vanilla encoder-decoder architectures which rely
solely on soft attention
(Perez-Beltrachini and Lapata, NAACL 2018)
![Page 172: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/172.jpg)
Reinforcement Learning
Generation: BLEU, NIST, METEOR
Summarisation: ROUGE, Pyramid
Simplification: BLEU, SARI, Compression Rate, Readibility etc.
Task Based Learning Objective
![Page 173: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/173.jpg)
Enforcing Model to Optimize Task-Specific Metric
Cross-Entropy training Objective
![Page 174: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/174.jpg)
Enforcing Model to Optimize Task-Specific Metric
Two problems with Cross-Entropy training objective
● It maximizes the likelihood of the next correct word and not the task-specific evaluation metrics.
● It suffers from exposure bias problem.
![Page 175: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/175.jpg)
Exposure Bias with Cross Entropy Training Training
Predict the next word in a sequence, given the previous reference words and context
Testing
Model generates the entire sequence from scratch
![Page 176: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/176.jpg)
Exposure Bias with Cross Entropy Training
(Ranzato et al., ICLR 2016)
Training
Testing
![Page 177: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/177.jpg)
Text Production as a Reinforcement Learning Problem
Agent
action
Environment
rewardupdate
![Page 178: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/178.jpg)
Policy Gradient to Optimize Task-Specific Metric
Goal of training is to find the parameters of the agent that maximize the expected reward
Loss is the negative expected reward
REINFORCE algorithm (Williams, 1992)
![Page 179: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/179.jpg)
Policy Gradient to Optimize Task-Specific Metric
In practice, we approximate the expected gradient using a single sample for each training example
![Page 180: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/180.jpg)
Sentence Simplification
(Zhang and Lapata, 2017)Optimizes BLEU and SARI jointly
![Page 181: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/181.jpg)
Extractive Document Summarization
Optimizes ROUGE scores (Narayan et al, NAACL 2018)
![Page 182: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/182.jpg)
Abstractive Document Summarization
Optimizes ROUGE scores
● A Deep Reinforced Model for Abstractive Summarization (Paulus 2017)● Multi-Reward Reinforced Summarization with Saliency and Entailment
(Pasunuru and Bansal NAACL 2018)● Deep Communicating Agents for Abstractive Summarization (Celikyilmaz et
al NAACL 2018)
![Page 183: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/183.jpg)
Datasets, Challenges and Example Systems for Text Production
![Page 184: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/184.jpg)
Datasets and Challenges for Neural NLG
DATA to text
WebNLG: Generating from RDF Data
E2E: Generating from Dialog Acts
Meaning Representations to text
SemEval Task 9: Generating from Abstract Meaning Representations
2018 Shared Task (SR'18)
![Page 185: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/185.jpg)
Data OverviewINPUT Output Domain Lge NLG
WebNLG RDF Text 15 domains En MicroPlanning
E2E Dialog Act Text Restaurabt En MicroPlanning
SemEval AMR Stce News En SR
SR Dep. Trees
Stce News En, Fr, Sp SR
![Page 186: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/186.jpg)
Datasets and Challenges for T2T GenerationSimplification:
● Wikismall, Wikilarge, Newsela
Sentence Compression/Summarisation
● English Gigaword [Rush et al, 2015], DUC 2004 Test Set [Over et al., 2007], MSR-ATC Test Set [Toutanova et al. 2016], News Sentence-compression pairs [Filippova et al. 2015]
Paraphrasing
● PPDB,Multiple-Translation Chinese (MTC) corpus
![Page 187: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/187.jpg)
Datasets and Challenges for Simplification
● WikiSmall (Zhu et al., 2010) ○ automatically-aligned complex-simple pairs from the ordinary-simple English Wikipedias. ○ Training: 89, 042 pairs. Test: 100
● WikiLarge (Zhang and Lapata, 2017)○ Larger Wikipedia corpus aggregating pairs from Kauchak (2013), Woodsend and Lapata
(2011), and WikiSmall. ○ All: 296,402 sentence pairs
● Newsela (Xu et al., 2015) ○ Training: 94208 sentence pairs, Test: 1076
![Page 188: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/188.jpg)
Datasets for Paraphrasing● ParaNMT (Wieting and Gimpel 2017)
○ back-translated paraphrase dataset ○ 50M+ back-translated paraphrases from the Czeng1.6 corpus
● PPDB (Ganitkevitch and Callison-Burch, 2014)○ paraphrastic textual fragments extracted automatically from bilingual text
● Multiple-Translation Chinese (MTC) corpus○ 11 translations per input. Used for testing.
![Page 189: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/189.jpg)
Datasets for Sentence CompressionEnglish Gigaword [Rush et al, 2015]
● News: (First sentence, Headline). Train: 3.8M, Test: 189K
DUC 2004 [Over et al., 2007]
● (Sentence, Summary). Test: 500 input with 4 summaries each.
MSR-ATC [Toutanova et al. 2016]
● Test: 26K (Sentence, Summary)
News Sentence-compression pairs [Filippova et al. 2015]
● Test: 10K pairs
![Page 190: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/190.jpg)
Datasets and Challenges for Summarisation● CNN/DailyMail Story Highlights Generation dataset (Hermann et al. 2015) ● NY Times Summarization dataset (Sandhaus, 2008)
Multidocument summarisation DUC and TAC too small
![Page 191: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/191.jpg)
Datasets and Challenges for Summarisation● CNN/DailyMail Story Highlights Generation dataset (Hermann et al. 2015) ● NY Times Summarization dataset (Sandhaus, 2008)
Multidocument summarisation DUC and TAC too smallNewsroom Dataset for Diverse Summarization
(Grusky et al. NAACL 2018)
![Page 192: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/192.jpg)
Open Challenges● Producing text in languages other than English
○ Multilingual SR Task, AMR-to-Chinese○ Byte Pair Encoding,
● Taking the Discourse Structure of input text into account ○ simplification, abstractive summarisation
● Structuring long output (Hierarchical Generation)○ Story generation, Poetry, Data to document
○ Transformer and convSeq2Seq architecture
● Generating under constraints ○ length, emotion,style, user profile, syntax etc
○ VAE, Generative models
![Page 193: CNRS/LORIA, Nancy University of Edinburgh Claire Gardent ... · Deep Learning Approaches to Text Production Claire Gardent Shashi Narayan CNRS/LORIA, Nancy University of Edinburgh](https://reader034.vdocuments.mx/reader034/viewer/2022051916/600858630ac3ae66957ab863/html5/thumbnails/193.jpg)
Thank you!