T-CVAE: Transformer-Based Conditioned Variational Autoencoder for Story Completion

Source : IJCAI 2019

Speaker : Ya-Fang, Hsiao

Advisor : Jia-Ling, Koh

Date : 2020/04/24

CONTENTS

Introduction

Method

Experiment

Conclusion

1

Introduction

for Story Completion

My Dad loves chocolate chip cookies. _______________. I decided I would learn how to make them. I made my first batch the other day. My Dad was very surprised and quite happy!

My Mom doesn't like to make cookies because they take too long.

He has been making them all week.

He always ate them.

2

Method

Transformer-Based Conditioned Variational Autoencoder

T-CVAE:

T-CVAE:

T-CVAE:

ℒ 𝑥 = −KL 𝑞 𝑧 𝑥 ||𝑝(𝑧) + 𝔼𝑞 𝑧 𝑥 [𝑙𝑜𝑔𝑝(𝑥′|𝑧)]

ℒ 𝑥, 𝑥′ = −KL 𝑞 𝑧 𝑥, 𝑐 ||𝑝(𝑧|𝑥) + 𝔼𝑞 𝑧 𝑥, 𝑐 [𝑙𝑜𝑔𝑝(𝑥′|𝑐, 𝑧)]

VAE:

CVAE:

T-CVAE:

ℒ 𝑥, 𝑥′ = −KL 𝑞 𝑧 𝑥, 𝑐 ||𝑝(𝑧|𝑥) + 𝔼𝑞 𝑧 𝑥, 𝑐 [𝑙𝑜𝑔𝑝(𝑥′|𝑐, 𝑧)]

𝐶𝑡 = tanh( 𝑧, 𝐷𝑜𝑢𝑡,𝑡𝐿 𝑊𝑐)

𝑂𝑡 = 𝐶𝑡𝑊𝑜 + 𝑏𝑜

Combination layer:

𝑃𝑡 = 𝑠𝑜𝑓𝑡𝑚𝑎𝑥(𝑂𝑡)

3

Experiment

ROCStory

Ablation Study

Generating k-th sentence

Case Study

4

Conclusion

Generating better coherence and diversity plots

Generating the missing story plot at any position