introduction to deep learning

Introduc)on  to  Deep  Learning  

Massimiliano  Ruocco  

Outline

•  Introduction and Motivation for DL •  From NN to Deep Learning •  Deep Learning Models •  Deep Learning in the Real World •  Conclusion

Introduction and Motivation for DL  

Introduction Deep Learning - WHAT

Class of ML training algorithm

Introduction Deep Learning - Motivations

•  ML Algorithms: –  Supervised –  Unsupervised –  Semi-supervised –  Reinforcement Learning

•  ML Algorithms: unsupervised learning

Data  Representa)on  

Input   Clustering   Output  

Example (Marketing/Customer segmentation): •  Input : Customers of a specific product •  Output: Customer subgroups

Introduction Deep Learning - Motivations

•  ML Algorithms: supervised learning

Data  Representa)on  

Input   Classifica)on/  Regression  

Output  

Training  Labeled    DataSet  

Data  Representa)on  

Example (spam detection): •  Input : Email •  Output: Spam/NotSpam •  Training Set: Data set of mail labeled as Spam/Not Spam

Introduction Deep Learning in ML and AI

•  ML Algorithms: supervised learning

Data  Representa)on  

Input   Classifica)on/  Regression  

Output  

Training  Labeled    DataSet  

Data  Representa)on  

Example (spam detection): •  Input : Email •  Output: Spam/NotSpam •  Training Set: Data set of mail labeled as Spam/Not Spam

Introduction Deep Learning – Representation Problem

•  Data Representation: –  feature set selection –  #features

•  Main Issues: –  Course of dimensionality –  Overfitting –  Handcrafted features

•  How to tackle: Representation Learning

Introduction Deep Learning – Representation Problem

•  Deep learning methods: –  Representations are expressed in terms of other, simpler representations

Introduction Deep Learning - WHAT

•  Deep Learning algorithm as application of Machine Learning to Artificial intelligence

Ar#ficial  Intelligence    (i.e.  knowledge  bases)  

Machine  Learning    (i.e.  Support  Vector  Machine)  

Representa#on  Learning    (i.e.  Autoencoders)  

Deep  Learning    (i.e.  Mul=layer  Perceptron)  

Introduction Deep Learning in ML and AI

From Neural Network to Deep Learning  

•  Neural Network: Basic –  Different layers of neurons/perceptrons –  Human brain analysis –  Input, Hidden Layer, Output

•  Neural Network: Applications –  Classification (Spam Detection) –  Pattern Recognition (Character recognition)

Introduction From Neural Network to Deep Learning

•  The core: Neuron

Introduction From Neural Network to Deep Learning

W1  

W2  

W3  

x1  

x2  

xn  Sigmoid  func)on  

1/(1+e-­‐z)  

Output  hw(x)    

x  =  [x0…xn]T    w  =  [w0…wn]T    z  =  wTx  

•  Neural Network – Single layer

Introduction From Neural Network to Deep Learning

•  Forward Propagation: –  process of computing the output

Introduction From Neural Network to Deep Learning

x1  

x2  

x3  

a12  

a22  

W(1)  

W(2)  

a(2)  z(2)  

z(3)  

X  

z(2)  =  XW(1)  a(2)  =  f(z(2))    z(3)  =  a(2)W(2)  y  =  f(z(3))    

•  Training a Neural Network: –  Learning the parameters (weights)

•  Supervised •  Unsupervised •  Reinforcement Learning

•  Employing a Neural Network: –  Selecting the Architecture –  # Layers –  # Units per layer –  Kind of learning algorithm

Introduction From Neural Network to Deep Learning

•  Training a Neural Network: –  Backward Propagation

•  Gradient descent •  Objective: Minimize the cost function J

Introduction From Neural Network to Deep Learning

x1  

x2  

x3  

a12  

a22  

W(1)  

W(2)  

a(2)  z(2)  

z(3)  

X  

•  DNN à Typically artificial neural netwok with 3 or more levels of non-linear operations

Introduction From Neural Network to Deep Learning

•  Using Back propagation for Deep NN –  Does not scale –  Bad performance for random initialization –  Local Optima –  Vanishing gradient problem

Introduction Issues in Training DNN

Introduction The Breakthrough

2006*+  Backward  Propaga#on   Greedy-­‐layer  wise  training  +    

Supervised  fine  tuning  

* Hinton et al. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006 + Ranzato et al. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems 19 (NIPS’06),

•  Deep learning methods: –  Class of ML algorithm –  Use cascade of many levels of non linear processing units for feature extraction

–  Hierarchy of concepts –  Multiple-layered model –  NN with high number of hidden layers –  NEW LEARNING ALGORITHM Overcoming previous training problems

Introduction Deep Learning - Summary

Deep  Learning  Models  

Deep Learning Models Introduction

•  Two main classes: –  Generative

•  Deep Network for supervised Learning

–  Discriminative •  Deep Network for unsupervised learning

–  Hybrid

Deep Learning Models Generative – Deep Belief Network

•  Generative graphic model •  Mix directed and undirected between vars •  Learn to reconstruct the input

Deep Learning Models Generative – Deep Belief Network

•  Training algorithm –  Iteratively apply RBM training to each pair of layers

Deep Learning Models Discriminative – Convolutional NN

•  CNN in Computer Vision: Image Recognition –  Feed-forward multilayer network –  Kind of back propagation for learning –  Receptive fields –  Learn suitable representation of the image

Deep Learning Models Discriminative – Convolutional NN

•  CNN in Computer Vision: Image Recognition –  Key concepts:

•  Max pooling •  Sparse Connectivity •  Convolution

Deep  Learning  in  the  Real  World  

•  NLP •  Image Classification/Computer Vision •  Speech Recognition

Introduction Deep Learning – Application Field

•  [Google] 2013 acquired DNNresearch of professor Geoff Hinton to improve the state of the art in image recognition in photos

•  [Facebook] 2013 hired deep learning expert Yann to head up the company’s new artificial intelligence lab specialized in deep learning for computer vision and image recognition

•  [Pinterest] 2014 announced it has acquired Visual Graph

•  [Google + Baidu]: 20G13 - Deep Learning Visual Search Engine

Deep Learning in the Real World Facts

•  [Baidu] 2013: Deep Learning Visual Search Engine

•  [Google] 2013 Photo Search Engine

•  [Microsoft] 2013 Search by voice on Xbox console

•  [Google] 2014 word2vec for word tagging or text messaging suggestion

Deep Learning in the Real World Products

Thanks  for  the  aUen)on