http://bit.ly/idl2018

Xavier Giro-i-Nietoxavier.giro@upc.edu

Associate ProfessorUniversitat Politecnica de CatalunyaTechnical University of Catalonia

The Neural Network Zoo(& some research at UPC)

Day 4 Lecture 4

#DLUPC

2

Acknowledgements

Fjodor Van Veen, “The Neural Network Zoo”

The Asimov Institute (2016)

3

Acknowledgements

4

A Perceptron

J. Alammar, “A visual and interactive guide to the Basics of Neural Networks” (2016)F. Van Veen, “The Neural Network Zoo” (2016)

5

Neural Network = Multi Layer Perceptron

2 layers of perceptronsF. Van Veen, “The Neural Network Zoo” (2016)

6

Deep Neural Network (DNN)

F. Van Veen, “The Neural Network Zoo” (2016)

7Alex Graves, “Supervised Sequence Labelling with Recurrent Neural Networks”

The hidden layers and the output depend from previous

states of the hidden layers

Recurrent Neural Network (RNN)

8

Recurrent Neural Network (RNN)

F. Van Veen, “The Neural Network Zoo” (2016)

9

Recurrent Neural Network (RNN)

F. Van Veen, “The Neural Network Zoo” (2016)

10

Recurrent Neural Network (RNN)

Víctor Campos, Brendan Jou, Jordi Torres, Xavier Giró-i-Nieto, Shih-Fu Chang, “Skip RNN: Learning to Skip State Updates in RNNs”. NIPS Workshop on Time Series 2018.

11

Recurrent Neural Network (RNN)

Deep Learning TV, “Recurrent Neural Networks - Ep. 9”

12

Convolutional Neural Network (CNN)

F. Van Veen, “The Neural Network Zoo” (2016)

13

Convolutional Neural Network (CNN)

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 2278-2324.

LeNet-5

14

Convolutional Neural Network (CNN)

Orange

A Krizhevsky, I Sutskever, GE Hinton “Imagenet classification with deep convolutional neural networks” NIPS 2012.

AlexNet

15

Convolutional Neural Network (CNN)

Junting Pan, Kevin McGuinness, Elisa Sayrol, Noel O'Connor, and Xavier Giro-i-Nieto. "Shallow and Deep Convolutional Networks for Saliency Prediction." CVPR 2016.

JuntingNet

16

Convolutional Neural Network (CNN)

Marc Assens, Kevin McGuinness, Noel E. O’Connor, Xavier Giró-i-Nieto, “SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes”. ICCVW 2017. (winner Salient360 2017)

[ 600 x 300 ]

SaltiNet

17

Convolutional Neural Network (CNN)

Deep Learning TV, “Convolutional Neural Networks - Ep. 8”

18

Deconvolutional Neural Network (Deconv)

Junting Pan, SalGAN (2017)F. Van Veen, “The Neural Network Zoo” (2016)

19

Deep Residual Learning / Skip connections

F. Van Veen, “The Neural Network Zoo” (2016)

He, Kaiming, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. "Deep Residual Learning for Image Recognition." CVPR 2016 [slides]

20Figure: Kilian Weinberger

Deep Residual Learning / Skip connections

21Long, Jonathan, Evan Shelhamer, and Trevor Darrell. "Fully convolutional networks for semantic segmentation." CVPR 2015 & PAMI 2016.

Deep Residual Learning / Skip connections

22Ronneberger, Olaf, Philipp Fischer, and Thomas Brox. "U-net: Convolutional networks for biomedical image segmentation." MICCAI 2015

U-Net

Deep Residual Learning / Skip connections

23

CNN + RNN + Skip connections

Amaia Salvador, Míriam Bellver, Manel Baradad, Ferran Marqués, Jordi Torres, Xavier Giró-i-Nieto, “Recurrent Neural Networks for Semantic Instance Segmentation”. NIPS WiML 2017.

24

Dense connections

Dense Block of 5-layers with a growth rate of k=4

Connect every layer to every other layer of the same filter size.

Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]

DenseNet

25

Dense connections

Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017. [code]

DenseNet

26

Dense connections

Huang, Gao, Zhuang Liu, Kilian Q. Weinberger, and Laurens van der Maaten. "Densely connected convolutional networks." CVPR 2017 [code]

27

Dense connections

Jégou, Simon, Michal Drozdzal, David Vazquez, Adriana Romero, and Yoshua Bengio. "The One Hundred Layers Tiramisu: Fully Convolutional DenseNets for Semantic Segmentation." CVPRW 2017 [code] [slides]

28

Autoencoder (AE)

F. Van Veen, “The Neural Network Zoo” (2016)

29

Variational Autoencoder (VAE)

F. Van Veen, “The Neural Network Zoo” (2016)

30F. Van Veen, “The Neural Network Zoo” (2016)

Restricted Boltzmann Machine (RBM)

31

Deep Belief Networks (DBN)

F. Van Veen, “The Neural Network Zoo” (2016)

32

Adversarial Networks

Goodfellow, Ian, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. "Generative adversarial nets." NIPS 2014Goodfellow, Ian. "NIPS 2016 Tutorial: Generative Adversarial Networks." arXiv preprint arXiv:1701.00160 (2016).

F. Van Veen, “The Neural Network Zoo” (2016)

33

Adversarial Networks

Junting Pan, Cristian Canton, Kevin McGuinness, Noel E. O’Connor, Jordi Torres, Elisa Sayrol and Xavier Giro-i-Nieto. “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks.” CVPRW 2017.

Introduce an additional term to the less to improve performance.

34F. Van Veen, “The Neural Network Zoo” (2016)

Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]

Differentiable Neural Computers (DNC)

35

von Neumann architecture (1952) Neural Turing Machine (2014)

Differentiable Neural Computers (DNC)

Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014). [slides] [code]

End-to-end trainable

36

Differentiable Neural Computers (DNC)Add a trainable external memory to a neural network.

Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]

37

Differentiable Neural Computers (DNC)DNC can solve tasks reading information from a trained memory.

Graves, Alex, Greg Wayne, Malcolm Reynolds, Tim Harley, Ivo Danihelka, Agnieszka Grabska-Barwińska, Sergio Gómez Colmenarejo et al. "Hybrid computing using a neural network with dynamic external memory." Nature 538, no. 7626 (2016): 471-476. [Post by DeepMind]

38

Differentiable Neural Computers (DNC)

[Siraj Raval on YouTube]

39

More architectures to come...

Yann LeCun, “A Path to AI”, Beneficial AI 2017.

von Neumann architecture (1952)

40

The Full Story

F. Van Veen, “The Neural Network Zoo” (2016)

41

The Prequel

Fjodor Van Veen, “Neural Network Zoo Prequel: Cells and Layers”(2017)

42

Questions ?