deep learning at scale

15

Proprietary and confidential. Do not distribute. Nervana’s Deep Learning Platform MAKING MACHINES SMARTER.™ Hanlin Tang, PhD Algorithms Engineer

Upload: nervana-systems

Post on 16-Apr-2017

347 views

Category:

Technology

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Page 1: Deep Learning at Scale

Proprietary and confidential. Do not distribute.

Nervana’s Deep Learning Platform

MAKING MACHINES SMARTER.™

Hanlin Tang, PhDAlgorithms Engineer

Page 2: Deep Learning at Scale

Facebook DeepMask

Silver et al, 2016

The Atlantic, March 2016

“The error rate has been cut by a factor of two in all the languages, more than a factor of two in many cases. That’s mostly due to deep learning and the way we have optimized it …”

Alex Acero, Siri Senior Director, AppleArticle in Backhannel/WIRED, Aug 2016

Deep Learning

Page 3: Deep Learning at Scale

neon deep learning

framework

train deployexplore

nervanaengine

Fastest deep learning framework

cloudn

Page 4: Deep Learning at Scale

• Unprecedented computing power• 10x speedup over current Maxwell GPUs (~55 TeraOps)

• 32 GB High-Bandwidth Memory

• Six bi-directional high-bandwidth links for 3D torus interconnect• 8 chips in a box, seamlessly scale to multiple chassis

Page 5: Deep Learning at Scale

https://github.com/NervanaSystems/neon

Page 6: Deep Learning at Scale

• https://github.com/NervanaSystems/ModelZoo• Pre-trained weights and models

SegNet

Deep Speech 2

Skip-thought

Autoencoders

Deep Dream

Page 7: Deep Learning at Scale

Badrinarayanan et al., 2015

Page 8: Deep Learning at Scale

Neon (ms) Caffe (ms) Speed-upForward 101 719 7.1x

Backward 164 746 4.5xTotal 265 1455 5.5x

Page 9: Deep Learning at Scale

neon v1.6 + mgpu v1.6

neon v2.0Modular dataloader (aeon)Neural machine translation model

neon v3.0•Nervana Graph•Tensorflow inter-operability•Graph-enabled models•Distributed computing

Page 10: Deep Learning at Scale

Page 11: Deep Learning at Scale

“Training neural networks is a dark art.”Hyperparameters:•Number and type of units/layers•Convolution filter size•Weight Initialization•Optimization method•Learning Rate schedule

Page 12: Deep Learning at Scale

Page 13: Deep Learning at Scale

Command Line client Web Interface

Page 14: Deep Learning at Scale

Nervana in actionHealthcare: Tumor detection

Automotive: Speech interfacesFinance: Time-series search engine

Positive:

Negative:

Agricultural Robotics Oil & Gas

Positive:

Negative:

Proteomics: Sequence analysis

Query:

Results:

Page 15: Deep Learning at Scale

+ n

Large-Scale Deep Learning for Intelligent Computer Systems

Deep learning for large scale biodiversity monitoring

Intelligent Computer Systems Large-Scale Deep Learning forstatic.googleusercontent.com/media/research.google.com/en//people/... · Large-Scale Deep Learning for Intelligent Computer

Multi-scale deep learning for gesture detection and localizationgwtaylor/publications/neverova2014… · · 2014-12-10Multi-scale deep learning for gesture detection and localization

Applications: Large-Scale Deep Learningsrihari/CSE676/12.1 LargeScaleSystems.pdf · Deep Learning Srihari Large Scale Deep Learning •Philosophy of connectionism –While an individual

Multi-Scale Deep Learning Architectures for Person Re ......Multi-scale Deep Learning Architectures for Person Re-identiﬁcation Xuelin Qian1 Yanwei Fu2,5,* Yu-Gang Jiang1,3 Tao Xiang4

Benchmarking Deep Learning Workloads on Large-scale HPC ... · Benchmarking Deep Learning Workloads on Large-scale HPC Systems AmmarAhmad Awan and Dhabaleswar K. Panda [email protected],

Deep Learning at Scale

Deep Learning Based Large-Scale Automatic Satellite Crosswalk … · 2017-07-06 · Deep Learning Based Large-Scale Automatic Satellite Crosswalk Classiﬁcation Rodrigo F. Berriel,

Large Scale Deep Learning - … Scale Deep Learning Jeff Dean. ... representation conveyed by the IT cortex. Thus, compared with early visual representations, object manifolds are

Deep Learning at Scale on NVIDIA V100 Accelerators · Deep Learning at Scale on NVIDIA V100 Accelerators HPC and AI Innovation Lab Rengan Xu, Frank Han, ... •Deep Learning Frameworks

Large Scale Deep Learning with TensorFlow

Modular Deep Learning Analysis of Galaxy-Scale Strong

TRAINING DEEP LEARNING MODELS AT SCALE …...TRAINING DEEP LEARNING MODELS AT SCALE USING KUBERNETES Introductions Outline Conversational AI and Deep Learning Need for a Jobs framework

DEEP LEARNING FOR LARGE SCALE MUSIC RECOMMENDATION

Accelerating Large Scale Deep Learning Inference through ... · Accelerating Large Scale Deep Learning Inference through DeepCPU at Microsoft Minjia Zhang, Samyam Rajbandari, Wenhan

Deep Learning at Scale: A Paradigm Shift for Multi ......Deep Learning at Scale: A Paradigm Shift for Multi-Messenger Astrophysics Eliu Huerta Gravity Group gravity.ncsa.illinois.edu

Quoc le tera-scale deep learning

Initial Characterization of I/O in Large-Scale Deep Learning … · 2018-11-11 · Deep Learning (DL) applications demand large-scale computing facilities. DL applications require

Large Scale Deep Learning - Research at Googleresearch.google.com/people/jeff/CIKM-keynote-Nov2014.pdf · Large Scale Deep Learning ... for object recognition. Speciﬁcally,

Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning consists of two phases: Training and inference. As illustrated in Figure 2, training involves learning a

Democratizing Production-Scale Distributed Deep Learning...arXiv:1811.00143v2 [cs.CV] 3 Nov 2018 Democratizing Production-Scale Distributed Deep Learning additional overhead. Examples

Deep Learning on Hadoop Scale out Deep Learning on YARN

Deep Learning Performance Comparing Scale-out vs Scale-up · Deep Learning Performance: Scale-up vs Scale-out Architectures & Technologies Dell EMC | Infrastructure Solutions Group

Scalable and Distributed DNN Training on Modern HPC ...hidl.cse.ohio-state.edu/...distributed_training_dk.pdf · (2) Deep Learning @Scale (3) Non-deep learning analytics @Scale (4)

Deep Reinforcement Learning at Scale - GitHub Pages · Deep Reinforcement Learning at Scale Timothy Lillicrap Research Scientist, DeepMind & UCL ... Scaling Reinforcement Learning

Scalable Machine Learning - Centrum Wiskunde & … Machine...• hardware for deep learning –CPUs (SIMD), GPUs, TPUs • parallel training: does deep learning scale? –Trivially

Distributed Deep Learning At Scale On Apache Spark With BigDL

Large-scale Deep Unsupervised Learning using Graphics ...€¦ · Rajat Raina Anand Madhavan Andrew Y. Ng Stanford University Large-scale Deep Unsupervised Learning using Graphics

Multi-scale deep learning for gesture detection and ... · Multi-scale deep learning for gesture detection and localization 1;2Natalia Neverova 1;2Christian Wolf 3Graham W. Taylor

Large Scale Deep Learning Jeff Dean

Learning Deep Representation with Large-scale Attributesxgwang/papers/ouyangLZWiccv15.pdf · Learning Deep Representation with Large-scale Attributes Wanli Ouyang, Hongyang Li, Xingyu

Amber: Large-Scale Deep Learning for Intelligent Computer System

Deep Learning Performance Scale-Out · 4 Deep Learning Performance Scale-Out Motivation With the recent advances in the field of Machine Learning and especially Deep Learning, it’s

Large-scale Deep Unsupervised Learning using Graphics Processors