Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

An Introduction to Generative AdversarialNetworks and Image Deblurring Applications

Dimitrios Gotsis

April 27, 2020

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications1 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Table of Contents

1 Main Message

2 Neural Networks

3 Generative Adversarial Networks

4 Issues with GANs

5 Deblurring with GANs

6 Conclusion

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications2 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Main Message

Brief introduction into neural networks and certain wellknown layers

Give an introduction to Generative Adversarial Networks(GANs) and their potential applications

Introduction to image deblurring problem and potentialsolutions with GANs and its modifications

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications3 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Basic Layers

Will cover some basic layers, many more exist includingvariations on the ones mentioned

ReLU (Rectified Linear Unit): activation function,ReLU(z) = max{0, z}, other activation functions exist likesigmoid, tanh, etc.

Convolutional Layers

Layers that upsample or downsample, e.g. MaxPooling

Linear Layer, y = Wx+ b

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications4 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Structure of a Neural Network

Neural Networks usually pass output from one layer tonext layerMore complex structures can be created that do not just‘stack’, i.e. are not only compositions of functions

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications5 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Training a Neural Network

Typically use gradient descent, i.e. wt+1 = wt − α∇f(wt),or some variant to update weights based on objectivefunction

Need to update weights of all layers, use chain rule toupdate weights due to composition of functions, i.e.[f(g(x))]′ = f ′(g(x))g′(x) = ∂f

∂g∂g∂x

Start with weights from last layer and move backwards totake advantage of chain rule

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications6 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Basic Defintions

A Generative Adversarial Network (GAN) uses two neuralnetworks, a generator (G) and a discriminator (D)

G and D trained in competition to generate images fromnoise to look like they derive from training set

G takes in noise-only images, outputs created images

D takes in created images and real images anddifferentiates between them

When training is complete, discard D and keep G

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications7 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Mathematical Description

Have 0 signify created image and 1 signify real image

Mathematically we have: Dφ : RM×N → [0, 1] andGθ : RL×K → RM×N

M,N dimensions of dimensions of created and real images,K,L dimensions of noise-only image, φ parameters of Dand θ parameters of G

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications8 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Loss Function

The discriminator’s loss function can be shown to be:

LD = −1

2Ex[log(Dφ(x))]− 1

2Ez[log(1−Dφ(Gθ(z)))] (1)

To derive the generator loss function we assume zero sum game,i.e. LD + LG = 0 allowing us to say that:

θ∗ = arg minθ

maxφ

Ex[log(Dφ(x))] + E

z[log(1−Dφ(Gθ(z)))] (2)

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications9 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Training a GAN

Figure: Training a GAN

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications10 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Some Applications

Maps high resolution images to low resolution images well,applications in superresolution and deblurring

Image-to-image translations, such as mapping sketches toimages and aerial photos to maps

Also used in creating artwork

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications11 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Application Examples

Figure: Image-to-Image Translation and Superresolution Examples

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications12 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Mode Collapse

Problem where G maps noise input z to only one potentialoutput point

In practice G maps to some output points but not all whenmode collapse occurs

Caused by the fact that min max is not the same asmax min and during training we simultaneously updateboth networks hoping that solution will be similar

Simultaneous updates do not privilege min max overmax min or vice versa

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications13 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Mode Collapse (cont.)

Figure: No Mode Collapse vs Mode Collapse

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications14 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Vanishing Gradient

During training G or D may overpower one anotherpotentially causing a vanishing gradient value

A small gradient means small updates to weights, if thishappens too soon in training we get bad weights for both Gand D

Many tricks used to avoid problem such as training onenetwork less often during training or changing the modelsize of either network until no vanishing gradient is seen

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications15 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Non-Convergence

Gradient Descent does not converge for all GANs since it isa minimax problem, a competition between two players Gand D

Sometimes you may get oscillatory results meaning thatequilibrium point never reached

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications16 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Deblurring Background

Blurry images very common when taking pictures/video,inconvenient since image information lost

Mathematically can be modeled as IB = h ∗ IS + n, IBblurry image, IS sharp image, h blur kernel, n noise

For simplicity if we assume no noise, and a known kernel

then IS = F−1{IB(ω)H(ω)

}, where F−1 is inverse Fourier

Transform

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications17 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Deblurring Background (cont.)

However, even with very small noise may lead to hugedistortions from fraction if H(ω) has small values

Additionally, h not known in reality and without IS beingknown we can have many potential solutions

Create realistic blur kernels and transform sharp imagesinto blurry images to make dataset

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications18 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Using GANs for Deblurring

Main modification: change input generator input fromnoise-only image to blurry image or blurry image plusnoise-only image pair

Design algorithm for creating realistic blur kernels

In practice other modifications/additions to loss functionalso present in other NN solved problems

See Kupyn et al. (2018)

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications19 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

Conclusion and Summary

NN made up of function compositions, use gradient descent(or variant) plus chain rule to train weights, more complexNN structures also exist

GANs can be used for a variety of different applicationsbut difficult to train due to mode collapse, vanishinggradient and potential non-convergence

Deblurring problem hard to solve due to noise andunknown blur kernel

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications20 / 21

Main MessageNeural Networks

Generative Adversarial NetworksIssues with GANs

Deblurring with GANsConclusion

References

Goodfellow, Ian. “NIPS 2016 Tutorial: GenerativeAdversarial Networks.” ArXiv:1701.00160 [Cs], December31, 2016. http://arxiv.org/abs/1701.00160.

Kupyn, Orest, Volodymyr Budzan, Mykola Mykhailych,Dmytro Mishkin, and Jiri Matas. “DeblurGAN: BlindMotion Deblurring Using Conditional AdversarialNetworks.” In 2018 IEEE/CVF Conference on ComputerVision and Pattern Recognition, 8183–92. Salt Lake City,UT: IEEE, 2018.https://doi.org/10.1109/CVPR.2018.00854.

Dimitrios GotsisAn Introduction to Generative Adversarial Networks and Image Deblurring Applications21 / 21