urs köster - convolutional and recurrent neural networks

Deep Learning Deep Dive & Workshop on Convolutional and Recurrent

Neural Networks

Urs Köster

San Diego Deep Learning Meetup December 2, 2015

Outline

2

• Deep Learning

• Nervana

• Neon

• Convolutional Network Demo (meetup.nervanasys.com)

• Recurrent Network Demo (meetup.nervanasys.com)

INTRO TO DEEP LEARNING

3

4

Scene Parsing

*Yann LeCun https://www.youtube.com/watch?v=ZJMtDRbqH40

5

Speech Translation

*Skype https://www.youtube.com/watch?v=eu9kMIeS0wQ

6

Understanding Images

*Karpathy http://cs.stanford.edu/people/karpathy/deepimagesent/

7

What is deep learning?

Historical perspective:

• Input → designed features → output

• Input → designed features → SVM → output

• Input → learned features → SVM → output

• Input → levels of learned features → output

8

What is deep learning?

A method for extracting features at

multiple levels of abstraction

• Features are discovered from data

• Performance improves with more data

• Network can express complex transformations

• High degree of representational power

9

What is deep learning?

No free lunch:

• lots of data

• flexible models

• powerful priors

10

Imagenet ILSVRC ChallengeError rate1

Source: ImageNet1: ImageNet top 5 error rate

0%

10%

20%

30%

2010 2011 2012 2013 2014 2015

Deep learning techniques human performance

NERVANA PLATFORM

11

12

nervana platform for deep learning

neon deep learning

framework

train deploy

nervana cloud

explore

GPUs

CPUs

nervana engine

AWS

VM

S3 S3

Web

VM VM

VM VM VM

S3

13

Deep learning as a core technology

DL

Photos Maps

Voice Search

Self-driving car

Ad Targeting

Machine Translation

‘Google Brain’ model

DL

Image classification

Image localization

Speech recognition

Video indexing Sentiment

analysis

Machine Translation

Nervana Platform

• Architecture optimized for

algorithm

14

Core technology

• Unprecedented compute density

• Scalable distributed architecture

• Learning and inference

15

Markets

Pharma Oil&Gas AgricultureMedical

$Finance Internet Govt

NEON

16

neon: nervana python deep learning library

17

• User-friendly, extensible, abstracts parallelism & data caching

• Support for many deep learning models

• Interface to nervana cloud

• Multiple backends

• nervana engine

• GPU (optimized on assembler level)

• CPU clusterOpen source (Apache 2.0) on github.com/nervanaSystems/neon

18

Image classification (VGG-D) speed comparison

Speed for one full forward/backward pass on VGG model D

Imag

es p

er s

econ

d

0

20

40

60

80

100

120

Neon Caffe

• Neon trains networks about 2x

faster! • Caffe uses CuDNN v3 (NVidia’s

own optimized library) • Same holds for other models

(GoogLeNet, AlexNet) • And other Frameworks (Torch7,

TensorFlow)

1 Soumith Chintala, github.com/soumith/convnet-benchmarks

Benchmarks for convnets1

19

Benchmarks compiled by Facebook. Smaller is better.

20

End-to-end optimized

• GPU Kernels:

Written in SASS

Assembler, near full

utilization for most

layers • Data Loader: neon

never blocks waiting

for data

LibraryWrapper

DataLoader DataLoader DecodeThreads

start

IOThreads

destroythreadpoolstop

next

...

next

createthreadpool

createthreadpool

destroythreadpool

readmacrobatchfile

decode

decode

decode

macrobatchbuffers

minibatchbuffers(pinned)

rawfilebuffers

Control Codes

Dual issue instr.

Fused fp32 multiply add

Load from shared

Barrier sync

Set barrier

Proprietary and confidential. Do not distribute.

21

Running locally:

% python rnn.py # or neon rnn.yaml

Running in nervana cloud:

% ncloud submit rnn.py # or rnn.yaml

% ncloud show <model_id>

% ncloud list

% ncloud deploy <model_id>

% ncloud predict <model_id> <data> # or use REST api

22

HANDS ON!

• Option 1: Interactive ipython notebook in your

browser without installing anything (you should have

paper slip with your URL and password)

• Option 2: Use your linux or mac laptop to download

neon and try our examples. Get the notebooks and

data from meetup.nervanasys.com

Contact

23

urs@nervanasys.com

github.com/NervanaSystems/neon

\