Download - Introduction to Deep learning
Introduc)on to Deep Learning
Massimiliano Ruocco
Outline
• Introduction and Motivation for DL • From NN to Deep Learning • Deep Learning Models • Deep Learning in the Real World • Conclusion
Introduction and Motivation for DL
Introduction Deep Learning - WHAT
Class of ML training algorithm
Introduction Deep Learning - Motivations
• ML Algorithms: – Supervised – Unsupervised – Semi-supervised – Reinforcement Learning
• ML Algorithms: unsupervised learning
Data Representa)on
Input Clustering Output
Example (Marketing/Customer segmentation): • Input : Customers of a specific product • Output: Customer subgroups
Introduction Deep Learning - Motivations
• ML Algorithms: supervised learning
Data Representa)on
Input Classifica)on/ Regression
Output
Training Labeled DataSet
Data Representa)on
Example (spam detection): • Input : Email • Output: Spam/NotSpam • Training Set: Data set of mail labeled as Spam/Not Spam
Introduction Deep Learning in ML and AI
• ML Algorithms: supervised learning
Data Representa)on
Input Classifica)on/ Regression
Output
Training Labeled DataSet
Data Representa)on
Example (spam detection): • Input : Email • Output: Spam/NotSpam • Training Set: Data set of mail labeled as Spam/Not Spam
Introduction Deep Learning – Representation Problem
• Data Representation: – feature set selection – #features
• Main Issues: – Course of dimensionality – Overfitting – Handcrafted features
• How to tackle: Representation Learning
Introduction Deep Learning – Representation Problem
• Deep learning methods: – Representations are expressed in terms of other, simpler representations
Introduction Deep Learning - WHAT
• Deep Learning algorithm as application of Machine Learning to Artificial intelligence
Ar#ficial Intelligence (i.e. knowledge bases)
Machine Learning (i.e. Support Vector Machine)
Representa#on Learning (i.e. Autoencoders)
Deep Learning (i.e. Mul=layer Perceptron)
Introduction Deep Learning in ML and AI
From Neural Network to Deep Learning
• Neural Network: Basic – Different layers of neurons/perceptrons – Human brain analysis – Input, Hidden Layer, Output
• Neural Network: Applications – Classification (Spam Detection) – Pattern Recognition (Character recognition)
Introduction From Neural Network to Deep Learning
• The core: Neuron
Introduction From Neural Network to Deep Learning
W1
W2
W3
x1
x2
xn Sigmoid func)on
1/(1+e-‐z)
Output hw(x)
x = [x0…xn]T w = [w0…wn]T z = wTx
• Neural Network – Single layer
Introduction From Neural Network to Deep Learning
• Forward Propagation: – process of computing the output
Introduction From Neural Network to Deep Learning
x1
x2
x3
a12
a22
W(1)
W(2)
a(2) z(2)
z(3)
X
z(2) = XW(1) a(2) = f(z(2)) z(3) = a(2)W(2) y = f(z(3))
• Training a Neural Network: – Learning the parameters (weights)
• Supervised • Unsupervised • Reinforcement Learning
• Employing a Neural Network: – Selecting the Architecture – # Layers – # Units per layer – Kind of learning algorithm
Introduction From Neural Network to Deep Learning
• Training a Neural Network: – Backward Propagation
• Gradient descent • Objective: Minimize the cost function J
Introduction From Neural Network to Deep Learning
x1
x2
x3
a12
a22
W(1)
W(2)
a(2) z(2)
z(3)
X
• DNN à Typically artificial neural netwok with 3 or more levels of non-linear operations
Introduction From Neural Network to Deep Learning
• Using Back propagation for Deep NN – Does not scale – Bad performance for random initialization – Local Optima – Vanishing gradient problem
Introduction Issues in Training DNN
Introduction The Breakthrough
2006*+ Backward Propaga#on Greedy-‐layer wise training +
Supervised fine tuning
* Hinton et al. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006 + Ranzato et al. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems 19 (NIPS’06),
• Deep learning methods: – Class of ML algorithm – Use cascade of many levels of non linear processing units for feature extraction
– Hierarchy of concepts – Multiple-layered model – NN with high number of hidden layers – NEW LEARNING ALGORITHM Overcoming previous training problems
Introduction Deep Learning - Summary
Deep Learning Models
Deep Learning Models Introduction
• Two main classes: – Generative
• Deep Network for supervised Learning
– Discriminative • Deep Network for unsupervised learning
– Hybrid
Deep Learning Models Generative – Deep Belief Network
• Generative graphic model • Mix directed and undirected between vars • Learn to reconstruct the input
Deep Learning Models Generative – Deep Belief Network
• Training algorithm – Iteratively apply RBM training to each pair of layers
Deep Learning Models Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition – Feed-forward multilayer network – Kind of back propagation for learning – Receptive fields – Learn suitable representation of the image
Deep Learning Models Discriminative – Convolutional NN
• CNN in Computer Vision: Image Recognition – Key concepts:
• Max pooling • Sparse Connectivity • Convolution
Deep Learning in the Real World
• NLP • Image Classification/Computer Vision • Speech Recognition
Introduction Deep Learning – Application Field
• [Google] 2013 acquired DNNresearch of professor Geoff Hinton to improve the state of the art in image recognition in photos
• [Facebook] 2013 hired deep learning expert Yann to head up the company’s new artificial intelligence lab specialized in deep learning for computer vision and image recognition
• [Pinterest] 2014 announced it has acquired Visual Graph
• [Google + Baidu]: 20G13 - Deep Learning Visual Search Engine
Deep Learning in the Real World Facts
• [Baidu] 2013: Deep Learning Visual Search Engine
• [Google] 2013 Photo Search Engine
• [Microsoft] 2013 Search by voice on Xbox console
• [Google] 2014 word2vec for word tagging or text messaging suggestion
Deep Learning in the Real World Products
Thanks for the aUen)on