urs köster - convolutional and recurrent neural networks
TRANSCRIPT
Deep Learning Deep Dive & Workshop on Convolutional and Recurrent
Neural Networks
Urs Köster
San Diego Deep Learning Meetup December 2, 2015
Outline
2
• Deep Learning
• Nervana
• Neon
• Convolutional Network Demo (meetup.nervanasys.com)
• Recurrent Network Demo (meetup.nervanasys.com)
INTRO TO DEEP LEARNING
3
4
Scene Parsing
*Yann LeCun https://www.youtube.com/watch?v=ZJMtDRbqH40
5
Speech Translation
*Skype https://www.youtube.com/watch?v=eu9kMIeS0wQ
6
Understanding Images
*Karpathy http://cs.stanford.edu/people/karpathy/deepimagesent/
7
What is deep learning?
Historical perspective:
• Input → designed features → output
• Input → designed features → SVM → output
• Input → learned features → SVM → output
• Input → levels of learned features → output
8
What is deep learning?
A method for extracting features at
multiple levels of abstraction
• Features are discovered from data
• Performance improves with more data
• Network can express complex transformations
• High degree of representational power
9
What is deep learning?
No free lunch:
• lots of data
• flexible models
• powerful priors
10
Imagenet ILSVRC ChallengeError rate1
Source: ImageNet1: ImageNet top 5 error rate
0%
10%
20%
30%
2010 2011 2012 2013 2014 2015
Deep learning techniques human performance
NERVANA PLATFORM
11
12
nervana platform for deep learning
neon deep learning
framework
train deploy
nervana cloud
explore
GPUs
CPUs
nervana engine
AWS
VM
S3 S3
Web
VM VM
VM VM VM
S3
13
Deep learning as a core technology
DL
Photos Maps
Voice Search
Self-driving car
Ad Targeting
Machine Translation
‘Google Brain’ model
DL
Image classification
Image localization
Speech recognition
Video indexing Sentiment
analysis
Machine Translation
Nervana Platform
• Architecture optimized for
algorithm
14
Core technology
• Unprecedented compute density
• Scalable distributed architecture
• Learning and inference
15
Markets
Pharma Oil&Gas AgricultureMedical
$Finance Internet Govt
NEON
16
neon: nervana python deep learning library
17
• User-friendly, extensible, abstracts parallelism & data caching
• Support for many deep learning models
• Interface to nervana cloud
• Multiple backends
• nervana engine
• GPU (optimized on assembler level)
• CPU clusterOpen source (Apache 2.0) on github.com/nervanaSystems/neon
18
Image classification (VGG-D) speed comparison
Speed for one full forward/backward pass on VGG model D
Imag
es p
er s
econ
d
0
20
40
60
80
100
120
Neon Caffe
• Neon trains networks about 2x
faster! • Caffe uses CuDNN v3 (NVidia’s
own optimized library) • Same holds for other models
(GoogLeNet, AlexNet) • And other Frameworks (Torch7,
TensorFlow)
1 Soumith Chintala, github.com/soumith/convnet-benchmarks
Benchmarks for convnets1
19
Benchmarks compiled by Facebook. Smaller is better.
20
End-to-end optimized
• GPU Kernels:
Written in SASS
Assembler, near full
utilization for most
layers • Data Loader: neon
never blocks waiting
for data
LibraryWrapper
DataLoader DataLoader DecodeThreads
start
IOThreads
destroythreadpoolstop
next
...
next
createthreadpool
createthreadpool
destroythreadpool
readmacrobatchfile
decode
decode
decode
macrobatchbuffers
minibatchbuffers(pinned)
rawfilebuffers
Control Codes
Dual issue instr.
Fused fp32 multiply add
Load from shared
Barrier sync
Set barrier
Proprietary and confidential. Do not distribute.
21
Running locally:
% python rnn.py # or neon rnn.yaml
Running in nervana cloud:
% ncloud submit rnn.py # or rnn.yaml
% ncloud show <model_id>
% ncloud list
% ncloud deploy <model_id>
% ncloud predict <model_id> <data> # or use REST api
22
HANDS ON!
• Option 1: Interactive ipython notebook in your
browser without installing anything (you should have
paper slip with your URL and password)
• Option 2: Use your linux or mac laptop to download
neon and try our examples. Get the notebooks and
data from meetup.nervanasys.com
Contact
23
github.com/NervanaSystems/neon
\