an introduction to generative adversarial networks · 2021. 8. 1. · unsupervised: circular gans...
TRANSCRIPT
![Page 1: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/1.jpg)
An Introduction to Generative Adversarial Networks
Sagie Benaim
Tel Aviv University
1
![Page 2: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/2.jpg)
![Page 3: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/3.jpg)
![Page 4: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/4.jpg)
![Page 5: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/5.jpg)
![Page 6: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/6.jpg)
Progress on Face Generation
![Page 7: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/7.jpg)
BigGAN – Late 2018
![Page 8: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/8.jpg)
• Depth and Convolution
• Class-conditional Generation
• Wasserstein GAN
• Self Attention
• BigGAN
From GAN to BigGAN
![Page 9: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/9.jpg)
![Page 10: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/10.jpg)
![Page 11: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/11.jpg)
• Depth and Convolution
• Class-conditional Generation
• Wasserstein GAN
• Self Attention
• BigGAN
From GAN to BigGAN
![Page 12: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/12.jpg)
![Page 13: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/13.jpg)
![Page 14: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/14.jpg)
![Page 15: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/15.jpg)
• Depth and Convolution
• Class-conditional Generation
• Wasserstein GAN
• Self Attention
• BigGAN
From GAN to BigGAN
![Page 16: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/16.jpg)
Wasserstein GAN
• Wasserstein Distance: Minimum cost of transporting mass in converting the data distribution q to the data distribution p.
![Page 17: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/17.jpg)
![Page 18: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/18.jpg)
![Page 19: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/19.jpg)
![Page 20: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/20.jpg)
• Depth and Convolution
• Class-conditional Generation
• Wasserstein GAN
• Self Attention
• BigGAN
From GAN to BigGAN
![Page 21: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/21.jpg)
![Page 22: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/22.jpg)
• Depth and Convolution
• Class-conditional Generation
• Wasserstein GAN
• Self Attention
• BigGAN
From GAN to BigGAN
![Page 23: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/23.jpg)
• Scalability: GANs benefit dramatically from scaling. Two architectural changes that improve scalability.
• Robustness: Fine control of the trade-offs between fidelity and variety is possible via the “truncation trick”
• Stability: Devises solutions that minimize the instabilities in Large Scale GANs
BigGAN
![Page 24: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/24.jpg)
![Page 25: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/25.jpg)
Image to Image Translation
25
![Page 26: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/26.jpg)
26
![Page 27: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/27.jpg)
![Page 28: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/28.jpg)
Supervised Unsupervised
Unimodal Pix2pix, CRN, SRGAN DistanceGAN, CycleGAN, DiscoGAN, DualGAN, UNIT, DTN, StarGAN, OST
Multimodal pix2pixHD, BicycleGAN MUNIT, Augmented CycleGAN
![Page 29: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/29.jpg)
……
Paired Unpaired
![Page 30: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/30.jpg)
Fully Supervised: pix2pix
[Isola et al., CVPR 2017]
![Page 31: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/31.jpg)
[Isola et al., CVPR 2017]
![Page 32: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/32.jpg)
Unsupervised: Circular GANsDiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial Networks”. Kim et al. ICML’17.
CycleGAN: “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”. Zhu et al. arXiv:1703.10593, 2017.
DualGAN: “ Unsupervised Dual Learning for Image-to-Image Translation”. Zili et al. arXiv:1704.02510, 2017.
. 32
![Page 33: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/33.jpg)
[Mark Twain, 1903]
[Zhu et al., ICCV 2017]
Cycle-Consistent Adversarial Networks
![Page 34: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/34.jpg)
G(x) F(G x )x F(y) G(F x )𝑦
Cycle Consistency Loss
F G x − x1
G F y − 𝑦1
[Zhu et al., ICCV 2017]
DY(G x )
Reconstructionerror
Reconstructionerror
DG(F x )
See similar formulations [Yi et al. 2017], [Kim et al. 2017]
![Page 35: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/35.jpg)
Collection Style Transfer
Photograph@ Alexei Efros
Monet Van Gogh
Cezanne Ukiyo-e
![Page 36: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/36.jpg)
1 1
22
36
𝑥1 − 𝑥2 1~ 𝐺(𝑥1) − 𝐺(𝑥2) 1
• A pair of images of a given distance are mapped to a pair of outputs with a similar distance
• 𝑥𝑖 − 𝑥𝑗 1and 𝐺 𝑥𝑖 − 𝐺 𝑥𝑗 1
are highly correlated.
Benaim et al., NIPS 2017
DistanceGAN
![Page 37: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/37.jpg)
Motivating distance correlations
37
Analysis of CycleGAN’s horse to zebra resultsBenaim et al., NIPS 2017
![Page 38: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/38.jpg)
Mode Collapse
• GAN: Cycle:
38Benaim et al., NIPS 2017
![Page 39: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/39.jpg)
More than 2 domains
Choi et al., CVPR 2018
![Page 40: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/40.jpg)
More than 2 domains
Choi et al., CVPR 2018
![Page 41: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/41.jpg)
![Page 42: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/42.jpg)
![Page 43: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/43.jpg)
Huang et al., ECCV 2018
![Page 44: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/44.jpg)
Huang et al., ECCV 2018
![Page 45: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/45.jpg)
Huang et al., ECCV 2018
![Page 46: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/46.jpg)
Content Transfer?
![Page 47: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/47.jpg)
One Shot?
• Not only are we unsupervised, but we have only a single sample in the input domain!
“One Shot Unsupervised Cross Domain Mapping”, Benaim, et al. In Submission
![Page 48: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/48.jpg)
![Page 49: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/49.jpg)
Applications Beyond Computer Vision
• Many other Vision Applications: Photo Enhancement, Image Dehazing
• Medical Imaging and Biology [Wolterink et al., 2017]
• Voice conversion [Fang et al., 2018, Kaneko et al., 2017]
• Cryptography [CipherGAN: Gomez et al., ICLR 2018]
• Robotics
• NLP: Unsupervised machine translation.
• NLP: Text style transfer.
• …
![Page 50: An Introduction to Generative Adversarial Networks · 2021. 8. 1. · Unsupervised: Circular GANs DiscoGAN: “Learning to Discover Cross-Domain Relations with Generative Adversarial](https://reader036.vdocuments.mx/reader036/viewer/2022071606/614365e76b2ee0265c020594/html5/thumbnails/50.jpg)
Thank You! Questions?
50