Upload File

generative adversarial networks (gans) past,...

  • Home
  • Documents
  • Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,
27
Generative Adversarial Networks (GANs) Past, Present, and Future Zhifei Zhang

Upload: buituyen

Post on 06-Sep-2018

230 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

Generative Adversarial Networks (GANs)

Past, Present, and Future

Zhifei Zhang

Page 2: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 2

Outline

1. Basic idea of GANs

2. Time line of development

3. Drawbacks & solutions

4. Future directions

Page 3: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 3

Basic idea of GANs

In probability and statistics, a generative model is a model for randomly generating observable data values, typically given some hidden parameters.

Maximum Likelihood

Page 4: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 4

Basic idea of GANs

• Two neural networks against each other: • A generator network G

• Mimic training samples to fool the discriminator• A discriminator network D

• Discriminate training samples and generated samples

D GTraining samples

Generated samples

Noise

Real/fake?

Page 5: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 5

Basic idea of GANs

D GTraining samples

Generated samples

Noise

Real/fake?

𝒙~𝒒(𝒙) 𝒛~𝒑(𝒛)𝑮(𝒛)

𝒙~𝒑(𝒙|𝒛)

𝒙~𝒒 𝒙 ?

max𝐷

𝔼𝑥~𝑞(𝑥) log 𝐷(𝑥) + 𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

𝑫 𝒙 :

min𝐺

𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

For D:

For G:

Page 6: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 6

Basic idea of GANs

min𝐺max𝐷

𝔼𝑥~𝑞(𝑥) log 𝐷(𝑥) + 𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

The objective function of GANs:

Gz

Real?

Fake?

FeedforwardBackpropagation

x

x' Real?

D

Page 7: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 7

Time line

2014

GAN [NIPS]

2015 2016 2017

LaplacianPyramid [NIPS]

DCGAN [ICLR]

Born Fermenting Booming of Improvements and Applications:• Higher resolution • Flexible manipulation

Theory: Drawbacks &

Solutions

InfoGAN [NIPS]

RNN+GAN [ICML]

VAE+GAN [ICML] AAE [ICLR]

CoGAN [NIPS]

Super-Resolution [ECCV] [CVPR]

Latent-Manipulation [ECCV] [CVPR]

CatGAN [ICLR]

Domain transformation [ICML] [CVPR]

Mode Missing[ICLR]

Instability[ICLR]

Page 8: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 8

Time lineDCGAN [A. Radford and L. Metz, ICLR 2016]

Page 9: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 9

Time lineVAE+GAN [A. B. L. Larsen et al., ICML 2016]

• GAN encourages higher resolution compared to VAE• Controllable generation and manipulation on z

Page 10: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 10

Time lineVAE+GAN [A. B. L. Larsen et al., ICML 2016]

Page 11: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 11

Time lineVAE+GAN [A. B. L. Larsen et al., ICML 2016]

Page 12: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 12

Time lineSuper-Resolution [X. Yu and F. Porikli, ECCV 2016]

• 8× scaling factor

Page 13: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 13

Time lineSuper-Resolution [X. Yu and F. Porikli, ECCV 2016]

Page 14: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 14

Time lineDomain Transformation1 [S. Reed et al., ICML 2016]

Page 15: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 15

Time lineDomain Transformation2 [P. Isola et al., CVPR 2017]

Page 16: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 16

Time lineDomain Transformation2 [P. Isola et al., CVPR 2017]

Page 17: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

D

3/22/2017 17

Summary

Gz

Real?

Fake?

x

x'

• Seldom use original GAN • Concatenate an encoder to G• Concatenate extra feature to z

Ex

y

Page 18: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 18

Drawbacks

• Hard to train, e.g., mode missing.

• Generate something but not real actually.

• Hard to learn to generate discrete data, e.g., text.

Page 19: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 19

Drawbacks (mode missing)

min𝐺max𝐷

𝔼𝑥~𝑞(𝑥) log 𝐷(𝑥) + 𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

D GTraining samples

Generated samples

Noise

Real/fake?

𝒙~𝒒(𝒙) 𝒛~𝒑(𝒛)𝑮(𝒛)

𝒙~𝒑(𝒙|𝒛)

𝒙~𝒒 𝒙 ?𝑫 𝒙 :

min𝐺max𝐷

𝑥

𝑞(𝑥) log 𝐷(𝑥) 𝑑𝑥 + 𝑧

𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧)) 𝑑𝑧

Page 20: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 20

Drawbacks (mode missing)

Fix G, min𝐺max𝐷

𝑥

𝑞(𝑥) log 𝐷(𝑥) 𝑑𝑥 + 𝑧

𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧)) 𝑑𝑧

=max𝐷

𝑥

𝑞 𝑥 log 𝐷 𝑥 + 𝑝𝑔 𝑥 log 1 − 𝐷 𝑥 𝑑𝑥 𝑫∗ 𝒙 =𝒒(𝒙)

𝒒 𝒙 + 𝒑𝒈(𝒙)

Fix D*,=min

𝐺 𝑥

𝑞 𝑥 log𝑞(𝑥)

𝑞 𝑥 + 𝑝𝑔(𝑥)+ 𝑝𝑔 𝑥 log 1 −

𝑞(𝑥)

𝑞 𝑥 + 𝑝𝑔(𝑥)𝑑𝑥

=min𝐺

𝑥

𝑞 𝑥 log𝑞(𝑥)

𝑞 𝑥 + 𝑝𝑔(𝑥)+ 𝑝𝑔 𝑥 log

𝑝𝑔(𝑥)

𝑞 𝑥 + 𝑝𝑔(𝑥)𝑑𝑥

=min𝐺

𝐷𝐾𝐿(𝑞||𝑞 + 𝑝𝑔) + 𝐷𝐾𝐿(𝑝𝑔||𝑞 + 𝑝𝑔)

=min𝐺

𝐷𝐾𝐿(𝑞||𝑞 + 𝑝𝑔

2) + 𝐷𝐾𝐿(𝑝𝑔||

𝑞 + 𝑝𝑔

2)−2 log 2

=min𝐺

2𝐷𝐽𝑆(𝑞||𝑝𝑔)−2 log 2

Jensen-Shannon divergence is a symmetric metric.

q(x)<pg(x): Unrealistic sampleq(x)>pg(x): Mode missing

Page 21: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 21

Drawbacks (mode missing)

However, we cannot ensure D* in practice.

min𝐺

𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

=min𝐺

𝐷𝐾𝐿(𝑝𝑔||𝑞 + 𝑝𝑔

2)− log 2

𝑞(𝑥)

𝑞

• Punish more on generating unrealistic samples• Punish less on mode missing

If relax the assumption of D*,

min𝐺

𝔼𝑧~𝑝(𝑧) log 1 − 𝐷(𝐺(𝑧))

only punish on generating unrealistic samples.

Page 22: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 22

Drawbacks (unrealistic generation)

In GANs, the generated distribution is matched to the distribution specified by D, rather than to the real distribution.

Ideally, pg(x) = q(x), direct matching.

In practice, D(x|pg) = D(x|q), indirect matching.

Page 23: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 23

Drawbacks (Solutions)

• Achieve D*:• Update D multiple times for each update of G.• Unrolled GAN [Metz et al., ICLR 2017]

• Introduce direct matching:• Sample-wide distance.

DGz

Real?

Fake?

x

x'

Ex

Distance min |x - x'|

Page 24: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 24

Drawbacks (Solutions)

x

x'

|x - x'|=|x – F(x)|

pair-wise distance

Mont Carlo q(x) = pg(x)

Page 25: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 25

Drawbacks (Solutions)

GAN

GAN+E

GAN

Unrolled GAN

Page 26: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 26

Future Directions

• After explosion of applications, theoretical understanding becomes more attractive.

• Better objective functions to ensure convergence.

• There is not a good way to quantify how good the generated samples are.

• Generating discrete samples.

• One-shot imitation learning.

• Connection to reinforcement learning.

Page 27: Generative Adversarial Networks (GANs) Past, Present…web.eecs.utk.edu/~zzhang61/docs/reports/2017.03 - GANs_past_pres… · Generative Adversarial Networks (GANs) Past, Present,

3/22/2017 27


Progressive Augmentation of GANs · [email protected] Abstract Training of Generative Adversarial Networks (GANs) is notoriously fragile, re-quiring to maintain a careful balance

Structured Generative Adversarial Networks · Among various DGMs, variational autoencoders (VAEs) and generative adversarial networks (GANs) can be trained unsupervisedly to map a

Data Synthesis based on Generative Adversarial Networks · each other. We propose a data synthesis method based on gener-ative adversarial networks (GANs). GANs are a generative model

A Beginner's Guide to Generative Adversarial Networks (GANs) · GANs, Autoencoders and VAEs It may be useful to compare generative adversarial networks to other neural networks, such

GENERATIVE ADVERSARIAL NETWORKS - Deep Learningdeeplearning.cs.cmu.edu/F20/document/recitation/GANS-ppt.pdf · 2021. 3. 22. · Generative Adversarial Networks Generative Models We

GAN D V U ENERATIVE ADVERSARIAL NETWORKSamygdala.psychdept.arizona.edu/labspace/JclubLab... · better understand their GAN models. 1INTRODUCTION Generative Adversarial Networks (GANs)

GENERATIVE ADVERSARIAL NETWORKS FOR SINGLE …...Generative Adversarial Networks (GANs - Goodfellow et al., 2014) were recently proposed. This paper is focused on providing a reliable

Generative Adversarial Nets(GANs) - UMDdrp.math.umd.edu/Project-Slides/CarySpring2019.pdf · 2019. 8. 22. · •A generative adversarial net is a type of neural net, used in deep

Generative Adversarial Networks (GANs) Synthetic report · 2020. 12. 8. · Generative adversarial networks (GANs) are a generative model with implicit density estimation, part of

Generative Adversarial Networks (GANs)Figure 1: Generative adversarial nets are trained by simultaneously updating the discriminative distribution (D, blue, dashed line) so that it

GANs in Action: Deep learning with generative adversarial ......welcome Welcome to MEAP for GANs in Action: Deep Learning with Generative Adversarial Networks. GANs are an exciting

Generative Adversarial Networks - TJ Machine LearningGenerative Adversarial Networks Justin Zhang July 2017 1 Introduction Generative Adversarial Networks (GANs) are a seminal type

Object Detection: A Reinforcement Learning approach using Adversarial … · 2019-05-13 · Generative Adversarial Networks (GANs). GANs are a class of machine learning systems that

Machine Learning (CSE 446): Generative Adversarial Networks … · Machine Learning (CSE 446): Generative Adversarial Networks (GANs) Sham M Kakade c 2019 University of Washington

Generative Adversarial Networks (GANs) · where D is our discriminator, G is our generator, and and are their corresponding parameters. [ 2 ] E. A. Goodfellow, Ian, Generative Adversarial

Collaborative Sampling in Generative Adversarial … › pdf › 1902.00813.pdfGenerative Adversarial Networks Generative Adversarial Networks (GANs) (Goodfellow et al. 2014) consist

Generative Adversarial Networks (GANs) - Sciencesconf.org · Generative Adversarial Networks (GANs) Anirban Mukhopadhyay TU Darmstadt, Germany. Introduction Generative vs. Discriminative

Generative Adversarial Networks (GANs)courses.missouristate.edu/anthonyclark/790DL/lectures/16-gans.pdf · Generative Adversarial Networks, Goodfellowet al. (2014) GANs Real Data

Semantic Segmentation using Adversarial NetworksTheir generative adversarial networks (GANs) take samples z from a fixed ... An adversarial network is used to define a loss function

Generative Adversarial Networks (GAN) · Unrolled generative adversarial networks.arXivpreprint arXiv:1611.02163. Evaluation of GANs is quite challenging In explicit density models,

Lipschitz constrained GANs via boundedness and continuityKeywords Generative adversarial networks Lipschitz constraint Boundedness Continuity 1 Introduction Generative adversarial

Adversarial Self-Defense for Cycle-Consistent GANs · Generative adversarial networks (GANs) [7] have enabled many recent breakthroughs in image generation, such as being able to

Generative Adversarial Networks (GANs) Adversarial Networks (GANs) Hossein Azizpour Most of the slides are courtesy of Dr. Ian Goodfellow (Research Scientist at OpenAI) and from his

Generative Adversarial Networks (GANs) · 2020. 9. 6. · Generative Adversarial Networks (GANs) Akshan Sameullah April 2020 1 Introduction Imagine two people are locked in a world-class

Data-Driven Crowd Simulation with Generative Adversarial ... · Our work uses Generative Adversarial Networks (GANs) [4], a recent AI development for generating new data. GANs have

Generative Adversarial Networks (GANs) Advanced …...Generative Adversarial Networks (GANs) CS109B, PROTOPAPAS, GLICKMAN Outline Concept and Math Applications Common Problems Wasserstein

Generative Adversarial NetworksGenerative Adversarial Networks Presented by Yi Zhang. Deep Generative Models N(O, I) Variational Auto-Encoders GANs. Unreasonable Effectiveness of GANs

Machine Learning (CSE 446): Generative Adversarial ... · Machine Learning (CSE 446): Generative Adversarial Networks (GANs) Sham M Kakade c 2019 University of Washington [email protected]

Generative Adversarial Networksbhiksha/courses/deeplearning/Spring.2018/... · 2018. 3. 29. · Generative Adversarial Networks Generative adversarial networks (GANs) are relatively

MetricGAN: Generative Adversarial Networks based Black-box ...proceedings.mlr.press/v97/fu19b/fu19b.pdf · Generative adversarial networks (GANs) (Goodfellow et al., 2014) has shown

An Introduction to Generative Adversarial Networks and ... · Generative Adversarial Networks Issues with GANs Deblurring with GANs Conclusion Main Message Brief introduction into

Generative Adversarial Networks (GANs) - Ian Goodfellotions. Meanwhile, deep convolutional generative adversarial networks (GANs) have begun to gen-erate highly compelling images of

Photo Editing With Generative Adversarial Networks (GANs)on-demand.gputechconf.com/gtc/2017/presentation/s... · GTC, May 2017. Greg Heinrich. Photo Editing With Generative Adversarial

“Deep Fakes” using Generative Adversarial Networks (GAN)noiselab.ucsd.edu/ECE228_2018/Reports/Report16.pdfstyles. Generative adversarial networks (GANs) provide us an available

Generative Adversarial Networks (GANs) in Networking: A