GENERATIVE ADVERSARIAL NETWORKS

R. Q. FEITOSA

The GAN Boom

2

source: GAN Zoo - https://github.com/hindupuravinash/the-gan-zoo

Application Examples Image to Image translation

3

images from Isola, et al., 2017

Application Examples Image captioning

4

from /

Application Examples Edges to Image translation

5

Demo: https://affinelayer.com/pixsrv/

Application Examples Edges to Image translation

6

Demo: https://affinelayer.com/pixsrv/

Application Examples Edges to Image translation

7

Demo: https://affinelayer.com/pixsrv/

Application Examples Edges to Image translation

8

Demo: https://affinelayer.com/pixsrv/

𝑥~𝑝𝑑𝑎𝑡𝑎

𝑝𝑚𝑜𝑑𝑒𝑙

𝑦~𝑝𝑚𝑜𝑑𝑒𝑙

Generative Models

9

training samples generated samples

images credited by Goodfellow. 2016

learn the distribution

underlying the

training samples

generate new

samples from the

learned distribution

Generative Models

10

training samples generated samples

learn the distribution

underlying the

training samples

generate new

samples from the

learned distribution

𝑥~𝑝𝑑𝑎𝑡𝑎

𝑝𝑚𝑜𝑑𝑒𝑙

𝑦~𝑝𝑚𝑜𝑑𝑒𝑙

if 𝑝𝑚𝑜𝑑𝑒𝑙 is a high

dimensional complex

distribution, there is no

direct way to do this

Generative Adversarial Network

11

training samples generated samples

learn the distribution

underlying the

training samples

generate new

samples from the

learned distribution

𝑥~𝑝𝑑𝑎𝑡𝑎

𝑝𝑚𝑜𝑑𝑒𝑙

𝑦~𝑝𝑚𝑜𝑑𝑒𝑙 Transformation

Generative Adversarial Network GANs adopt the following alternative approach:

Sample from a simple distribution, e.g., random noise, and learn a transformation to the target distribution

12

Question: How can we design such a complex

function that maps samples from a simple

distribution to a complex target distribution?

Answer: a neural network

z Generator

Network

input:

random noise

output:

sample from training distribution

Ian Goodfellow et al., “Generative Adversarial Nets”, NIPS 2014

Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images

Discriminator network: try to distinguish between real and fake images

13

z

Discriminator

Network

Generator

Network

real image samples

generated image samples

real/fake random noise

Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images

Discriminator network: try to distinguish between real and fake images

Train jointly in minimax game

Minmax objective function:

14

min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

Discriminator output

for real data 𝒙 Discriminator output for

fake data 𝑮 𝒛

Discriminator outputs likelihood in (0,1) real interval

Discriminator parameters Generator parameters

Training GANs: two-player game Generator network: try to fool the discriminator by generation realistic images

Discriminator network: try to distinguish between real and fake images

Train jointly in minimax game

Minmax objective function:

• Discriminator (𝛉𝑑) wants to maximize objective such 𝐷𝛉𝑑 𝒙 is close to 1 for

real and 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 is close to 0 for fake.

• Generator (𝛉𝑔) wants to minimize objective such that 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 is

close to 1 (discriminator is fooled into thinking generated 𝐺𝛉𝑔 𝒛 is real)

15

min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

Discriminator output

for real data 𝒙 Discriminator output for

fake data 𝑮 𝒙

Discriminator outputs likelihood in (0,1) real interval

Training GANs: two-player game Minmax objective function:

Alternate between:

1. Gradient ascent on discriminator (keep 𝛉𝑔 constant)

2. Gradient descent on generator (keep 𝛉𝑑 constant)

16

min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

min𝛉𝑔𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

At the beginning fake samples will be easily recognized by the

discriminator, but the gradient in this region is flat slow learning! 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

Minmax objective function:

Alternate between:

1. Gradient ascent on discriminator (keep 𝛉𝑔 constant)

2. Instead: Gradient ascent on generator – change the objective

Training GANs: two-player game

17

min𝛉𝑔max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

max𝛉𝑑𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙 + 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

max𝛉𝑔𝔼𝒛~𝑝 𝒛 log 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛 Instead of minimizing likelihood of discriminant being correct,

maximize likelihood of discriminator being wrong fast learning!

. 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

log 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

Training GANs: summary for number of training iterations do

for k steps do

• Sample minibatch of m noise samples {𝒛(1)…𝒛(𝑚)} from noise prior 𝑝𝑔 𝒛

• Sample minibatch of m examples {𝒙(1)…𝒙(𝑚)} from data generating

distribution 𝑝𝑑𝑎𝑡𝑎 𝒙

• Update the discriminator

end for

• Sample minibatch of m noise samples {𝒛(1)…𝒛(𝑚)} from noise prior 𝑝𝑔 𝒛

• Update the discriminator by ascending its stochastic gradient (improved

objective)

end for

18

𝛻𝜃𝑑1

𝑚 log𝐷𝜃𝑑 𝒙

𝑖 + log 1 − 𝐷𝜃𝑑 𝐺𝜃𝑔 𝒛𝑖

𝑚

𝑖=1

𝛻𝜃𝑔1

𝑚 log 𝐷𝜃𝑑 𝐺𝜃𝑔 𝒛

𝑖

𝑚

𝑖=1

Generator Implementation DCGANS:

most “deconvs” are batch normalized.

19

Radfort et al. 2015, UNSUPERVISED REPRESENTATION LEARNING WITH DEEP CONVOLUTIONAL

GENERATIVE ADVERSARIAL NETWORKS

Once trained… Generator network: try to fool the discriminator by generation realistic images

Discriminator network: try to distinguish between real and fake images

20

z

Discriminator

Network

Generator

Network

real image samples

generated image samples

real/fake random noise

After training, use

generator network to

generate new images.

Application examples Source to target domain transfer

21

Zhu et al., 2018 , Unpaired Image-to-

Image Translation using Cycle-Consistent

Adversarial Networks

Application examples Text to image synthesis

22

Reed et al., 2015, Generative Adversarial

Text to Image Synthesis

Conditional GANs

23

Discriminator

Network

Generator

Network

real image sample

generated image sample

real/ fake

random noise

condition

condition

condition

z

Conditional GANs

24

z

Conditional GANs GAM

Conditional GAN

25

min𝛉𝑔max𝛉𝑑 𝔼𝒙~𝑝𝑑𝑎𝑡𝑎 𝒙 log 𝐷𝛉𝑑 𝒙

+ 𝔼𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝐺𝛉𝑔 𝒛

min𝛉𝑔max𝛉𝑑 𝔼𝒙,𝒚~𝑝𝑑𝑎𝑡𝑎 𝒙,𝒚 log 𝐷𝛉𝑑 𝒙, 𝒚

+ 𝔼𝒙~𝑝 𝒙 ,𝒛~𝑝 𝒛 log 1 − 𝐷𝛉𝑑 𝒙, 𝐺𝛉𝑔 𝒙, 𝒛

condition real image

generated image

Application examples Image-to-image translation

26

Zhu et al., 2018 , Towards Multimodal Image-to-Image Translation

The GAN Zoo Lots of references on GANs

Click here to access The Gan Zoo

27

GENERATIVE ADVERSARIAL NETWORKS

Thank you!

R. Q. FEITOSA

28

References • Isola, P., Zhu, J. Y., Zhou, T., Efros, A. A., Image-to-Image Translation with

Conditional Adversarial Networks, CVPR, 2017, arXiv:1611.07004v2

29