introduction to deep learning

Introduc)on to Deep Learning

Massimiliano Ruocco

Outline

•  Introduction and Motivation for DL •  From NN to Deep Learning •  Deep Learning Models •  Deep Learning in the Real World •  Conclusion

Introduction and Motivation for DL

Introduction Deep Learning - WHAT

Class of ML training algorithm

Introduction Deep Learning - Motivations

•  ML Algorithms: –  Supervised –  Unsupervised –  Semi-supervised –  Reinforcement Learning

•  ML Algorithms: unsupervised learning

Data Representa)on

Input Clustering Output

Example (Marketing/Customer segmentation): •  Input : Customers of a specific product •  Output: Customer subgroups

Introduction Deep Learning - Motivations

•  ML Algorithms: supervised learning

Data Representa)on

Input Classifica)on/ Regression

Output

Training Labeled DataSet

Data Representa)on

Example (spam detection): •  Input : Email •  Output: Spam/NotSpam •  Training Set: Data set of mail labeled as Spam/Not Spam

Introduction Deep Learning in ML and AI

•  ML Algorithms: supervised learning

Data Representa)on

Input Classifica)on/ Regression

Output

Training Labeled DataSet

Data Representa)on

Example (spam detection): •  Input : Email •  Output: Spam/NotSpam •  Training Set: Data set of mail labeled as Spam/Not Spam

Introduction Deep Learning – Representation Problem

•  Data Representation: –  feature set selection –  #features

•  Main Issues: –  Course of dimensionality –  Overfitting –  Handcrafted features

•  How to tackle: Representation Learning

Introduction Deep Learning – Representation Problem

•  Deep learning methods: –  Representations are expressed in terms of other, simpler representations

Introduction Deep Learning - WHAT

•  Deep Learning algorithm as application of Machine Learning to Artificial intelligence

Ar#ficial Intelligence (i.e. knowledge bases)

Machine Learning (i.e. Support Vector Machine)

Representa#on Learning (i.e. Autoencoders)

Deep Learning (i.e. Mul=layer Perceptron)

Introduction Deep Learning in ML and AI

From Neural Network to Deep Learning

•  Neural Network: Basic –  Different layers of neurons/perceptrons –  Human brain analysis –  Input, Hidden Layer, Output

•  Neural Network: Applications –  Classification (Spam Detection) –  Pattern Recognition (Character recognition)

Introduction From Neural Network to Deep Learning

•  The core: Neuron


W1

W2

W3

x1

x2

xn Sigmoid func)on

1/(1+e-‐z)

Output hw(x)

x = [x0…xn]T w = [w0…wn]T z = wTx

•  Neural Network – Single layer


•  Forward Propagation: –  process of computing the output


x1

x2

x3

a12

a22

W(1)

W(2)

a(2) z(2)

z(3)

X

z(2) = XW(1) a(2) = f(z(2)) z(3) = a(2)W(2) y = f(z(3))

•  Training a Neural Network: –  Learning the parameters (weights)

•  Supervised •  Unsupervised •  Reinforcement Learning

•  Employing a Neural Network: –  Selecting the Architecture –  # Layers –  # Units per layer –  Kind of learning algorithm


•  Training a Neural Network: –  Backward Propagation

•  Gradient descent •  Objective: Minimize the cost function J


x1

x2

x3

a12

a22

W(1)

W(2)

a(2) z(2)

z(3)

X

•  DNN à Typically artificial neural netwok with 3 or more levels of non-linear operations


•  Using Back propagation for Deep NN –  Does not scale –  Bad performance for random initialization –  Local Optima –  Vanishing gradient problem

Introduction Issues in Training DNN

Introduction The Breakthrough

2006*+ Backward Propaga#on Greedy-‐layer wise training +

Supervised fine tuning

* Hinton et al. A fast learning algorithm for deep belief nets. Neural Computation, 18:1527–1554, 2006 + Ranzato et al. Efficient learning of sparse representations with an energy-based model. Advances in Neural Information Processing Systems 19 (NIPS’06),

•  Deep learning methods: –  Class of ML algorithm –  Use cascade of many levels of non linear processing units for feature extraction

–  Hierarchy of concepts –  Multiple-layered model –  NN with high number of hidden layers –  NEW LEARNING ALGORITHM Overcoming previous training problems

Introduction Deep Learning - Summary

Deep Learning Models

Deep Learning Models Introduction

•  Two main classes: –  Generative

•  Deep Network for supervised Learning

–  Discriminative •  Deep Network for unsupervised learning

–  Hybrid

Deep Learning Models Generative – Deep Belief Network

•  Generative graphic model •  Mix directed and undirected between vars •  Learn to reconstruct the input

Deep Learning Models Generative – Deep Belief Network

•  Training algorithm –  Iteratively apply RBM training to each pair of layers

Deep Learning Models Discriminative – Convolutional NN

•  CNN in Computer Vision: Image Recognition –  Feed-forward multilayer network –  Kind of back propagation for learning –  Receptive fields –  Learn suitable representation of the image

Deep Learning Models Discriminative – Convolutional NN

•  CNN in Computer Vision: Image Recognition –  Key concepts:

•  Max pooling •  Sparse Connectivity •  Convolution

Deep Learning in the Real World

•  NLP •  Image Classification/Computer Vision •  Speech Recognition

Introduction Deep Learning – Application Field

•  [Google] 2013 acquired DNNresearch of professor Geoff Hinton to improve the state of the art in image recognition in photos

•  [Facebook] 2013 hired deep learning expert Yann to head up the company’s new artificial intelligence lab specialized in deep learning for computer vision and image recognition

•  [Pinterest] 2014 announced it has acquired Visual Graph

•  [Google + Baidu]: 20G13 - Deep Learning Visual Search Engine

Deep Learning in the Real World Facts

•  [Baidu] 2013: Deep Learning Visual Search Engine

•  [Google] 2013 Photo Search Engine

•  [Microsoft] 2013 Search by voice on Xbox console

•  [Google] 2014 word2vec for word tagging or text messaging suggestion

Deep Learning in the Real World Products

Thanks for the aUen)on

introduction to deep learning

Science