Upload File

deep learning: a theoretical introduction · deep learning: a theoretical introduction –episode 2...

  • Home
  • Documents
  • Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders
54
[1] Deep Learning: a theoretical introduction – Episode 2 Deep Learning: a theoretical introduction Episode 2 The Turning Point Università degli Studi di Pavia

Upload: others

Post on 15-Oct-2020

18 views

Category:

Documents


0 download

Report

TRANSCRIPT

Page 1: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[1]Deep Learning: a theoretical introduction – Episode 2

Deep Learning:a theoretical introductionEpisode 2The Turning Point

Università degli Studi di Pavia

http://vision.unipv.it/AI/AIRG.html
Page 2: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[2]Deep Learning: a theoretical introduction – Episode 2

A hard AI winter then ensued …

… then, the turning point

Page 3: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[3]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 4: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[4]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 5: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[5]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 6: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[6]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 7: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[7]Deep Learning: a theoretical introduction – Episode 2

An aside question:probabilities or just numbers?

Page 8: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[8]Deep Learning: a theoretical introduction – Episode 2

•

•

Restricted Boltzmann Machine (RBM)

Page 9: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[9]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 10: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[10]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)h hi

hi

hi = 1 x

W i

Page 11: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[11]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

xj = 1 h

Page 12: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[12]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

hi hj x

xi xj h

x

hi

Page 13: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[13]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 14: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[14]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 15: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[15]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 16: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[16]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

S

Page 17: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[17]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

S

Page 18: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[18]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

(i)

Page 19: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[19]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 20: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[20]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 21: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[21]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

x

Page 22: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[22]Deep Learning: a theoretical introduction – Episode 2

•

•

•

Restricted Boltzmann Machine (RBM)

Page 23: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[23]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

T q

Page 24: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[24]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

•

•

•

P

Page 25: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[25]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

x(0)

h(t+1)

x(t+1)

Page 26: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[26]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

x(0)

h(t+1)

x(t+1)

•

•

Page 27: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[27]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 28: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[28]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 29: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[29]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 30: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[30]Deep Learning: a theoretical introduction – Episode 2

Deep Boltzmann Machines (DBM)

Page 31: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[31]Deep Learning: a theoretical introduction – Episode 2

Deep Boltzmann Machines (DBM)

x

hi

Page 32: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[32]Deep Learning: a theoretical introduction – Episode 2

Deep Boltzmann Machines (DBM)

h(2)

Page 33: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[33]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 34: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[34]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 35: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[35]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 36: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[36]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

hi = 1

Page 37: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[37]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 38: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[38]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 39: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[39]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 40: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[40]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 41: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[41]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Network (DBN)

Page 42: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[42]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 43: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[43]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 44: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[44]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 45: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[45]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 46: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[46]Deep Learning: a theoretical introduction – Episode 2

Restricted Boltzmann Machine (RBM)

Page 47: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[47]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 48: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[48]Deep Learning: a theoretical introduction – Episode 2

Auto-encoders

Page 49: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[49]Deep Learning: a theoretical introduction – Episode 2

Auto-encoders

Page 50: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[50]Deep Learning: a theoretical introduction – Episode 2

Deep Belief Networks and Autoencoders

Page 51: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[51]Deep Learning: a theoretical introduction – Episode 2

The Turning Point

Page 52: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[52]Deep Learning: a theoretical introduction – Episode 2

The Turning Point

Page 53: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[53]Deep Learning: a theoretical introduction – Episode 2

What happened after that

Page 54: Deep Learning: a theoretical introduction · Deep Learning: a theoretical introduction –Episode 2 [3] Deep Belief Networks and Autoencoders

[54]Deep Learning: a theoretical introduction – Episode 2

What happened after that


Variational Autoencoders

Deep Learning for Computer Vision - Graduate Center, CUNY€¦ · Deep learning models are a solution to the above problems. ... Different settings for employing deep autoencoders

Deep Learning for Generic Object Detection: A Survey · deep learning, probabilistic models, autoencoders, manifold learning, and deep networks 17 Deep learning LeCun et al. (2015)

Cortical Receptive Fields Using Deep Autoencoders

Theoretical Foundations of Deep Learning via Sparse … · 2018-06-28 · Theoretical Foundations of Deep Learning ... between sparse modeling and deep-learning architectures. This

A Tutorial on Deep Learning Part 2: Autoencoders ...robotics.stanford.edu/~quocle/tutorial2.pdfA Tutorial on Deep Learning Part 2: Autoencoders, Convolutional Neural Networks and Recurrent

A Tutorial on Deep Learning Part 2: Autoencoders, Convolutional

How to Train Deep Variational Autoencoders and Probabilistic ......How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks Figure 3. MNIST test-set log-likelihood

Deep Learning for Natural Language Processing (NLP) using … · 2020. 5. 29. · Deep Learning for Natural Language Processing (NLP) using Variational Autoencoders (VAE) Master`s

GraphRNN: A Deep Generative Model for Graphs · Recent advances in deep generative models, such as varia-tional autoencoders (VAE) (Kingma & Welling,2014) and generative adversarial

Autoencoders and Generative Adversarial Nets€¦ · Autoencoders and Generative Adversarial Nets Chapter 1 [ 5 ] Fixing corrupted data with denoising autoencoders The autoencoders

Deep Inversion, Autoencoders for Learned Regularization of ... · C3: non-linearity; C4: super-resolution Deep learning for model-driven imaging “classical” physics-based reconstruction

Deep convolutional autoencoders for reconstructing

(DL hacks輪読) How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks

Deep Clustering by Gaussian Mixture Variational ...openaccess.thecvf.com/content_ICCV_2019/papers/...Deep Clustering by Gaussian Mixture Variational Autoencoders with Graph Embedding

Introduccion´ Autoencoders RBMs Redes de Deep …´ Autoencoders RBMs Redes de Convolucion´ Introduccion´ Deep Learning Una idea que ha rondado a los investigadores desde hace tiempo

An Evolutionary Method for Training Autoencoders for Deep Learning Networks

AutoEncoders& Kernels

Interpreting atypical conditions in systems with deep conditional … · 2020. 5. 11. · Interpreting atypical conditions in systems with deep conditional Autoencoders: the case

An Introduction to Deep Learning · Deep Learning An Introduction to ... •Deep Learning •Feed Forward Network •Problems of Deep Learning •AutoEncoders •Restricted Boltzmann

Autoencoders, Unsupervised Learning, and Deep Architectures P. Baldi University of California, Irvine

Learning Deep Network Representations with Adversarially ... · Learning Deep Network Representations with Adversarially Regularized Autoencoders∗ Wenchao Yu1, Cheng Zheng1, Wei

Deep Recurrent Neural Network-Based Autoencoders for ...downloads.hindawi.com/journals/cin/2017/4694860.pdf · ResearchArticle Deep Recurrent Neural Network-Based Autoencoders for

How to Train Deep Variational Autoencoders and ... · How to Train Deep Variational Autoencoders and Probabilistic Ladder Networks layer or above. To alleviate these problems we propose

Deep Trajectory Clustering with Autoencoders

Deep Gaussian Process autoencoders for novelty detection

Toward Theoretical Understanding of Deep Learning€¦ · Toward Theoretical Understanding of Deep Learning Sanjeev Arora Princeton University Institute for Advanced Study Support:

Anomaly Detection with Robust Deep Autoencoders€¦ · Anomaly Detection with Robust Deep Autoencoders Chong Zhou Worcester Polytechnic Institute 100 Institute Road Worcester, MA

Theoretical Modelling of Grass Drying in Deep Beds

deep learning - BGU · 2018-12-25 · Deep Learning • Logistic Regression • Neural Network • Autoencoders Example • Recursive Neural Network Example

Intro to Deep learning - Autoencoders

ADVERSARIAL AUTOENCODERS

Deep Autoencoders for Dimensionality Reduction of High-Content Screening … · 2015-01-08 · Deep Autoencoders for Dimensionality Reduction of High-Content Screening Data Lee Zamparo

Workshop Deep Learning - Personal Websitestevenputtemans.github.io/files/DeepLearningWorkshop.pdfnetwerken (ANN, CNN, DNN, RNN, AutoEncoders, GAN, …). •We mogen echter niet vergeten

Algorithm Theoretical Basis Document (ATBD) for Deep