introduction to deep learning in python and matlab
TRANSCRIPT
![Page 1: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/1.jpg)
Introduction to Hands-on Deep Learning
Imry Kissos Algorithm Researcher
![Page 2: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/2.jpg)
Outline
● Problem Definition● Motivation● Training a Regression DNN● Training a Classification DNN● Open Source Packages● Summary + Questions
2
![Page 3: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/3.jpg)
Problem Definition
3
Deep Convolutional Network
![Page 4: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/4.jpg)
Tutorial
● Goal: Detect faciallandmarks on (normal)face images● Data set provided by Dr. Yoshua Bengio● Tutorial code available:https://github.com/dnouri/kfkd-tutorial/blob/master/kfkd.py
4
![Page 5: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/5.jpg)
Flow
5
Predict Points on Test Set
Train ModelGeneral
Train Model “Nose Tip”
Train Model “Mouth Corners”
![Page 6: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/6.jpg)
Flow
6
Train ImagesTrain Points
Fit TrainedNet
![Page 7: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/7.jpg)
Flow
7
Test Images
Predict Predicted Points
![Page 8: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/8.jpg)
Python DL Framework
Wrapper to Lasagne
Theano extension for Deep Learning
Define, optimize, and evaluate mathematical expressions
Efficient Cuda GPU for DNN
8
Low Level
High Level
HW Supports: GPU & CPUOS: Linux, OS X, Windows
![Page 9: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/9.jpg)
Training a Deep Neural Network
1. Data Analysis2. Architecture Engineering3. Optimization4. Training the DNN
9
![Page 10: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/10.jpg)
Training a Deep Neural Network
1. Data Analysisa. Exploration + Validationb. Pre-Processingc. Batch and Split
2. Architecture Engineering3. Optimization4. Training the DNN
10
![Page 11: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/11.jpg)
Data Exploration + ValidationData:● 7K gray-scale images of detected faces● 96x96 pixels per image● 15 landmarks per image (?)
Data validation:● Some Landmarks are missing
11
1
![Page 12: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/12.jpg)
Pre-Processing
12
Data Normalization
Shuffle train data
![Page 13: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/13.jpg)
Batch-- t - train batch
- validation batch
- - test batch
⇐One Epoch’s data
13train/valid/test splits are constant
![Page 14: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/14.jpg)
Train / Validation Split
14
Classification - Train/Validation preserve classes proportion
![Page 15: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/15.jpg)
Training a Deep Neural Network
1. Data Analysis2. Architecture Engineering
a. Layers Definitionb. Layers Implementation
3. Optimization4. Training
15
![Page 16: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/16.jpg)
Architecture
16
X Y
Conv Pool Dense Output
![Page 17: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/17.jpg)
Layers Definition
17
![Page 18: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/18.jpg)
Activation Function
18
1
ReLU
![Page 19: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/19.jpg)
Dense Layer
19
![Page 20: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/20.jpg)
Dropout
20
![Page 21: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/21.jpg)
Dropout
21
![Page 22: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/22.jpg)
Training a Deep Neural Network
1. Data Analysis2. Architecture Engineering3. Optimization
a. Back Propagationb. Objectivec. SGDd. Updatese. Convergence Tuning
4. Training the DNN 22
![Page 23: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/23.jpg)
Back PropagationForward Path
23
Conv Dense
X Y
OutputPoints
![Page 24: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/24.jpg)
Back PropagationForward Path
24
X Y
ConvOutputPointsDense
X Y
Training Points
![Page 25: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/25.jpg)
Back PropagationBackward Path
25
X Y
Conv Dense
![Page 26: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/26.jpg)
Back PropagationUpdate
26
Conv Dense
For All Layers:
![Page 27: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/27.jpg)
Objective
27
![Page 28: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/28.jpg)
S.G.D
28Updates the network after each batch
![Page 29: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/29.jpg)
Optimization - Updates
29 Alec Radford
![Page 30: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/30.jpg)
Adjusting Learning Rate & Momentum
30
Linear in epoch
![Page 31: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/31.jpg)
Convergence Tuning
31
stops according to validation loss
returns best weights
![Page 32: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/32.jpg)
Training a Deep Neural Network
1. Data Analysis2. Architecture Engineering3. Optimization4. Training the DNN
a. Fitb. Fine Tune Pre-Trainedc. Learning Curves
32
![Page 33: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/33.jpg)
Fit
33
Loop over test batchs
Forward
Loop over train batchs
Forward+BackProp
![Page 34: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/34.jpg)
Fine Tune Pre-Trained
fgd
34
change output layer
load pre-trained weight
fine tune specialist
![Page 35: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/35.jpg)
Learning CurvesLoop over 6 Nets:
35
Epochs
![Page 36: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/36.jpg)
Learning Curves Analysis
36
Net 1
Net 2
OverfittingConvergence Jittering
EpochsEpochs
RM
SE
RM
SE
![Page 37: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/37.jpg)
Part 1 Summary
Training a DNN:
37
![Page 38: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/38.jpg)
Python
● Rich eco-system● State-of-the-art● Easy to port from prototype to production
38https://github.com/yoavram/Py4Eng
![Page 39: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/39.jpg)
Python DL Framework
39
Theano based Packages
![Page 40: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/40.jpg)
Part 1 EndBreak
![Page 41: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/41.jpg)
Part 2
![Page 42: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/42.jpg)
Outline
● Problem Definition● Motivation● Training a regression DNN● Training a classification DNN● Improving the DNN● Open Source Packages● Summary
42
![Page 43: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/43.jpg)
Matlab DL Framework
Open Source CNN Toolbox by
Numerical computing using Parallel Computing Toolbox
Efficient Cuda GPU for DNN
43
Low Level
High Level
HW Supports: GPU & CPUOS: Linux, OS X, Windows
![Page 44: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/44.jpg)
Problem StatementClassify a, b, …, z images into 26 classes:
44http://www.robots.ox.ac.uk/~vgg/practicals/cnn/
Bonus - OCR:
![Page 45: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/45.jpg)
Training a Deep Neural Network
1. Data Analysis2. Training the DNN3. Architecture Engineering4. Optimization
45
![Page 46: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/46.jpg)
Data Analysis
46
Defines training vs validation
Class uint per class [1,26]
![Page 47: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/47.jpg)
Data Pre-Processing
Image
47
scalar
![Page 48: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/48.jpg)
Training Flow
48
![Page 49: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/49.jpg)
Customized Batch Loading
49
How would you add Data Augmentation ?
![Page 50: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/50.jpg)
trainOpts
50
Start from last iter if interrupted
![Page 51: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/51.jpg)
initializeCharCnn()Net Architecture
Layers:● Conv● Pool● Conv● Pool● Conv● Relu● Conv● SoftMaxLoss
51
%f is the W initial std
![Page 52: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/52.jpg)
Optimization
SoftMaxScore (-∞,∞) → probabilities [0,1]
52https://classroom.udacity.com/courses/ud730
![Page 53: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/53.jpg)
One Hot Encoding
Encode class labels
53
![Page 54: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/54.jpg)
Cross Entropy
Distance measure between S(Y) and Labels
54
![Page 55: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/55.jpg)
Cross Entropy
Distance measure between S(Y) and Labels
55
D(S,L) is a positive scalart - index of ground-truth class
![Page 56: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/56.jpg)
Cross Entropy
Distance measure between S(Y) and Labels
56
In vl_nnloss.m:
![Page 57: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/57.jpg)
Training Goal
57
CNN
![Page 58: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/58.jpg)
Minimize Loss
Loss=average cross entropy
58
![Page 59: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/59.jpg)
Minimize loss
- learning rate
59
![Page 60: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/60.jpg)
Error Rate
TopK - Target label is one of the top K predictions
The Error Rate is:
60
![Page 61: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/61.jpg)
Loss & Error Convergence
61
Loss Error Rate
![Page 62: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/62.jpg)
Learned Filters
62
![Page 63: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/63.jpg)
OCR Evaluation
63
![Page 64: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/64.jpg)
OCR Evaluation
64
![Page 65: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/65.jpg)
Beyond Training
1. Training a classification DNN2. Improving the DNN
a. Analysis Capabilitiesb. Augmentation
3. Open Source Packages4. Summary
65
![Page 66: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/66.jpg)
Basic VS Advanced Mode
66
Basic
Advance
![Page 67: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/67.jpg)
Improving the DNN
Very tempting:● >1M images● >1M parameters● Large gap: Theory ↔ Practice
⇒Brute force experiments?!
67
![Page 68: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/68.jpg)
Analysis Capabilities
1. Theoretical explanationa. Eg. dropout/augmentation decrease overfit
2. Empirical claims about a phenomenaa. Eg. normalization helps convergence
3. Numerical understandinga. Eg. exploding / vanishing updates
68
![Page 69: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/69.jpg)
Reduce Overfitting
Solution:Data Augmentation
69
Net 1
Net 2
OverfittingEpochs
![Page 70: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/70.jpg)
Data Augmentation
Horizontal Flip Perturbation
70
1
![Page 71: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/71.jpg)
Convergence Challenges
71Need to monitor forward + backward path
EpochsEpochs
RM
SE
Data ErrorNormalization
![Page 72: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/72.jpg)
Deal with NaN
1. If in first 100 iterations
a. Learning rate is too high
2. Beyond 100 iterations
a. Gradient explosion
i. Consider gradient clipping
b. Illegal math operation
i. SoftMax: inf/inf
ii. Division by zero by one of your customized layers
72http://russellsstewart.com//notes/0.html
![Page 73: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/73.jpg)
The Net Doesn’t Learn Anything
1. Training loss does not reduce after first 100 iterations
a. Reduce the training size to 10 instances (images) to overfit it
i. Achieve 100% training accuracy on a small portion of data
b. Change batch size to 1 to and monitor the error per batch
c. Solve the simplest version of your problem
73http://russellsstewart.com//notes/0.html
![Page 74: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/74.jpg)
Beyond Training
1. Training a classification DNN2. Improving the DNN3. Open Source Packages
a. DL Open Source Packagesb. Effort Estimation
4. Summary
74
![Page 75: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/75.jpg)
Tips from Other PackagesTorch code organization Caffe’s separation
configuration ↔code
NeuralNet → YAML text format defining experiment’s configuration
75
![Page 76: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/76.jpg)
DL Open Source Packages
76
Caffe & MatConvNet for applications Torch, TensorFlow and Theano for research on DL http://fastml.com/torch-vs-theano/
Simple dnnComplex dnn
![Page 77: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/77.jpg)
Disruptive Effort Estimation
Feature Eng Deep Learning
77Modest SW Infra Huge SW Infra
![Page 78: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/78.jpg)
Summary
● Dove into Training a DNN● Presented Analysis Capabilities● Reviewed Open Source Packages
78
![Page 79: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/79.jpg)
ReferencesHinton Coursera Neuronal Networkhttps://www.coursera.org/course/neuralnetsUdacity Tensor Flow coursehttps://classroom.udacity.com/courses/ud730Technion Deep Learning coursehttp://moodle.technion.ac.il/course/view.php?id=4128Oxford Deep Learning coursehttps://www.youtube.com/playlist?list=PLE6Wd9FR--EfW8dtjAuPoTuPcqmOV53FuCS231n CNN for Visual Recognitionhttp://cs231n.github.io/Deep Learning Bookhttp://www.deeplearningbook.org/
79
![Page 80: Introduction to deep learning in python and Matlab](https://reader031.vdocuments.mx/reader031/viewer/2022013109/5870d7741a28ab64768b6e69/html5/thumbnails/80.jpg)
Questions?
80
DNN