msr visit talk 2016 - i.konstas · joint generator temperature time min mean max 06:00-21:00 9 15...

NEUGEN !

Text Generation from Meaning Representations

Yannis Konstas!!

Joint work with Mark Yatskar, Luke Zettlemoyer and Yejin Choi (UW)

~ Yonatan Bisk and Daniel Marcu (ISI)

Motivation

Motivation

Machine-generated Representation

Motivation


source block: hk

target block: ms

pos RP: W scale: small

Motivation


source block: hk

target block: ms

pos RP: W scale: small

Place the heineken block west of the mercedes block.

Motivation


source block: hk

target block: ups

pos RP: W scale: big

Motivation


source block: hk

target block: ups

pos RP: W scale: big

Place the heineken block in the first open space to the left of ups block.

Motivation

Motivation


Sentence Compression

Motivation



Sentence Fusion

Motivation



Sentence FusionParaphrasing

NLG Pipeline

Content Selection

Document Planning

Surface Realisation

Sentence Planning

Text

Communicative Goal

Content Planning

A A A A

mInput

Reordering/Linearization

Splitting/Aggregation

Lexicalization

Framework

NLG Pipeline

Content Selection

Document Planning

Surface Realisation

Sentence Planning

Text

Communicative Goal

Content Planning

A A A A

mInput

Reordering/Linearization

Splitting/Aggregation

Lexicalization- Neural Encoder - RNN Decoder - Graph Alignment

Framework

Joint Generator Temperature

Time Min Mean Max

06:00-21:00 9 15 21

Wind Speed

Time Min Mean Max

06:00-21:00 15 20 30

Cloud Sky Cover

Time Percent (%)

06:00-09:00 25-5009:00-12:00 50-75

Wind Direction

Time Mode

06:00-21:00 S

S

...

S

...

S

...

S

...S

...

?? ?

Cloudy, with temperaturesbetween 10 and 20 degrees.South wind around 20 mph.

Training

S ! R(start)R(ri.t)!FS(rj , start)R(rj .t)R(ri.t)!FS(rj , start)FS(r, r.fi)!F(r, r.fj)FS(r, r.fj)FS(r, r.fi)!F(r, r.fj)F(r, r.f)!W(r, r.f)F(r, r.f)F(r, r.f)!W(r, r.f)W(r, r.f)!↵W(r, r.f)!g(f.v)

FS0,1(temp1,start)

FS0,2(temp1,start)

F0,1(temp1,min)

F0,1(temp1,max)

F0,2(temp1,min)

F0,2(temp1,max)

FS1,2(temp1,start)

(2) HypergraphRepresentation

FS0,5(skyCover1.t,start)

0

BB@

mostly cloudy ? the morning

mostly cloudy ? after 11am

mostly cloudy ? then becoming

· · ·

1

CCA

F0,2(skyCover1.t,%)

0

BB@

mostly cloudy

mostly clouds

cloudy ,

· · ·

1

CCAW4,5(skyCover1.t,time)

0

BB@

morning

11am

after

· · ·

1

CCAW0,1(skyCover1.t,%)

0

BB@

mostly

cloudy

sunny

· · ·

1

CCA

W1,2(skyCover1.t,%)

0

BB@

mostly

cloudy

sunny

· · ·

1

CCA

(3) k-best decodingvia integration

Testing

(1) PCFG Grammar

Konstas and Lapata, 2012, 2013

More Complex Structure


know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod

Input (Graph or Tree)


know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod


AMR DAG


know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod


AMR DAG

λ-calculus Expression Tree


know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod


AMR DAG


ECIs Tree


know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod


I know a planet that is inhabited by a lazy man.

Output (Text)

AMR DAG


ECIs Tree


I knew a planet that was inhabited by a lazy man. (lpp_1943.249)

I have known a planet that was inhabited by a lazy man.

know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod



Output (Text)

AMR DAG


ECIs Tree


I knew a planet that was inhabited by a lazy man. (lpp_1943.249)

I have known a planet that was inhabited by a lazy man.

I know about a planet. It is inhabited by a lazy man.

know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod



Output (Text)

AMR DAG


ECIs Tree

NEUGEN Framework

ARG1-of

NEUGEN Framework

ARG1-of

Encoder DecoderG w

NEUGEN Framework

Encoder- Bag of Words: Concepts and roles- Attention mechanism

ARG1-of

Encoder DecoderG w

NEUGEN Framework


Decoder- Left-to-right word LSTM - Greedy beam search

ARG1-of

Encoder DecoderG w

NEUGEN Framework



ARG1-of

Encoder DecoderG wReranker

NEUGEN Framework



Reranker

ARG1-of


NEUGEN Framework



Reranker- CCG parser -> semantics

ARG1-of


NEUGEN Framework



Reranker- CCG parser -> semantics- Partial propositions graph Eincr

ARG1-of


NEUGEN Framework




ARG1-of

Aligner

Encoder DecoderG w

Eincr

G

Aligner- Feed-forward NN with attention

Reranker

NEUGEN Framework




ARG1-of

Aligner

Encoder DecoderG w

Eincr

G salign

Aligner- Feed-forward NN with attention- salign = fit(G, Eincr)

Reranker

Alignment

Alignment

w: A man inhabited a planet

Alignment


man

inhabit

agent

planet

patient

Eincr

Alignment

know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod

G


man

inhabit

agent

planet

patient

Eincr

Alignment

know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod

G


man

inhabit

agent

planet

patient

Eincr

fit(G, Eincr)

Alignment

know-01

I planet

lazy

ARG0 ARG1

inhabit-01

man

ARG1-of

ARG0

mod

G


man

inhabit

agent

planet

patient

Eincr

fit(G, Eincr)

<inhabit, man, agent>!<inhabit, planet, patient>

<know-01, I, A0>!<know-01, planet, A1>!<planet, inhabit-01, A1-of>!<inhabit-01, man, A0>!<man, lazy, mod>

Alignment <know, I, A0>!

!<know, planet, A1>!

!<planet, inhabit, A1-of>!

!<inhabit, man, A0>!

!<man, lazy, mod>

G

[0..1]

fitness score

<inhabit, man, agent>!!!!

<inhabit, planet, patient>

Eincr

E1:

. attention weights

+

max

Alignment

<inhabit, planet, patient>!!!!

<inhabit, man, agent>

<know, I, A0>!!

<know, planet, A1>!!

<planet, inhabit, A1-of>!!

<inhabit, man, A0>!!

<man, lazy, mod>

Eincr

G

E2:

fit(G,Eincr) =1

n

nX

i=0

fit(G,Ei)

[0..1]

fitness score

attention weights

+

max

.

Decoding

encoder(G)

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

G

Decoding

encoder(G)

;know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

G

Decoding

encoder(G)

;

I The A …

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

G

Decoding

encoder(G)

;

I The A …

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

w11: I

Thew12:

Manw13:

Treew14:G

Decoding

encoder(G)

;

I The A …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

w11: I

Thew12:

Manw13:

Treew14:G

sLSTM

Decoding

encoder(G)

;

I The A …

know knew planet …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

w11: I

Thew12:

Manw13:

Treew14:G

sLSTM

Decoding

encoder(G)

;

I The A …


re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

w11: I

Thew12:

Manw13:

Treew14:

…

I know

agent

w21:

I knew

agent

w22:

The planetw23:

Man planetw24:G

sLSTM + salign

Decoding

encoder(G)

;

I The A …


a planet man …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

…

w11: I

Thew12:

Manw13:

Treew14:

…

I know

agent

w21:

I knew

agent

w22:

The planetw23:

Man planetw24:G

sLSTM + salign

Decoding

encoder(G)

;

I The A …


a planet man …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

…

w11: I

Thew12:

Manw13:

Treew14:

…

I know

agent

w21:

I knew

agent

w22:

The planetw23:

Man planetw24:

w41: I know

agent

a planet

patient

w42: I knew

agent

planets that

patient

w43: The planet I knewagentpatient

w44: Man know

agent

a planet

patient

that

…G

sLSTM + salign

?

Decoding

encoder(G)

;

I The A …


a planet man …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

…

w11: I

Thew12:

Manw13:

Treew14:

…

I know

agent

w21:

I knew

agent

w22:

The planetw23:

Man planetw24:

w41: I know

agent

a planet

patient

w42: I knew

agent

planets that

patient


w44: Man know

agent

a planet

patient

that

…G

sLSTM + salign

?

Decoding

encoder(G)

;

I The A …


a planet man …

re-score:

know

I planet

lazy

ARG0 ARG1

inhabit

man

ARG1-of

ARG0

mod

…

…

w11: I

Thew12:

Manw13:

Treew14:

…

I know

agent

w21:

I knew

agent

w22:

The planetw23:

Man planetw24:

w41: I know

agent

a planet

patient

w42: I knew

agent

planets that

patient


w44: Man know

agent

a planet

patient

that

…

inhabit inhabited was …

G

sLSTM + salign

Preliminary Results

Preliminary Results Experimental Setup

Preliminary Results Experimental Setup- Baseline RNN w/o alignment- NeuGen


Dataset


Dataset- AMR LDC2015E86- Train: 15k, Dev: 1.2k, Test: 1.2k


Dataset- AMR LDC2015E86- Train: 15k, Dev: 1.2k, Test: 1.2k

0

5.5

11

16.5

22

BLEU-4 METEOR

BaselineNeuGen

Example Output

Example Output person

Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

Gflee-01

ARG0-of

Example Output Gold: Gallardo is the fugitive founder of Tijuana Drug Cartel.!!Baseline: Gallardo is the founder of Tijuana Drug Cartel.!!NeuGen: Tijunana Drug Cartel is the founder of Gallardo.

person

Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

Gflee-01

ARG0-of


person

Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

Gflee-01

ARG0-of

Gallardo !Tijuana Drug Cartel !State !Country !…


person

Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

Gflee-01

ARG0-of


Gallardo is the founder of !!Gallardo ’s Tijuana Drug Cartel is the !…

patient

agent

-19.82

-17.79

Tijuana Drug Cartel is the founder of !!!Tijuana Drug Cartel was the founder of !…

patient

patient

-18.67

-20.24


person

Gallardo found-01

domain ARG0-of

Tijuana Drug Cartel

ARG1

Gflee-01

ARG0-of


Gallardo is the founder of !!Gallardo ’s Tijuana Drug Cartel is the !…

patient

agent

-19.82

-17.79

Tijuana Drug Cartel is the founder of !!!Tijuana Drug Cartel was the founder of !…

patient

patient

-18.67

-20.24

Gallardo is the founder of Tijuana Drug Cartel.

agentpatient -19.59

Tijuana Drug Cartel is the founder of Gallardo.

agentpatient -17.84

Next Steps

Next Steps

- “Machine-generated” representations - λ-calculus (ATIS) - Blocks World complex descriptions (from ISI)

Next Steps

- “Machine-generated” representations - λ-calculus (ATIS) - Blocks World complex descriptions (from ISI)

- Downstream Text-to-Text Applications

THANK You

msr visit talk 2016 - i.konstas · joint generator temperature time min mean max 06:00-21:00 9 15...

Documents