tensorflow flexible, scalable, portable
TRANSCRIPT
![Page 1: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/1.jpg)
TensorFlowFlexible, Scalable, Portable
Rajat MongaEngineering Director, TensorFlow
![Page 2: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/2.jpg)
![Page 3: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/3.jpg)
repositoryfor “machine learning”
category on GitHub
#1
Released in Nov. 2015
![Page 4: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/4.jpg)
10,000+ commits since Nov, 2015
450+ contributors
1M+ binary downloads
#20 most popular repository on GitHub by stars
Used in ML classes at many universities:Toronto, Berkeley, Stanford, ...
Some Stats
![Page 5: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/5.jpg)
![Page 6: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/6.jpg)
![Page 7: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/7.jpg)
TensorFlow powered Cucumber Sorter
From: http://workpiles.com/2016/02/tensorflow-cnn-cucumber/
![Page 8: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/8.jpg)
Flexible
![Page 9: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/9.jpg)
![Page 10: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/10.jpg)
MatMul
Add Relu
biases
weights
examples
labels
Xent
Directed graph
![Page 11: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/11.jpg)
Async SGD with Parameter Server
Model Workers
Parameter Server
p∆p
p’ = p - λ *∆p
![Page 12: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/12.jpg)
Async SGD with TensorFlow
Model Workers
PS Workers
p∆p
p’ = p - λ *∆p
![Page 13: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/13.jpg)
Sync SGD with TensorFlow
Model Workers
PS Workers
p∆p
delta = ∑∆pp’ = p - λ * delta
![Page 14: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/14.jpg)
Alternate version of Parameter sharing
Workers
![Page 15: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/15.jpg)
More ML Algorithms● Clustering e.g. k-means● Random Forests● Logistic Regression● Bayesian methods
![Page 16: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/16.jpg)
Scalable
![Page 17: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/17.jpg)
Deferred Execution
Operations build the dataflow graph; eval() fetches the result.
import tensorflow as tf
with tf.Session():
x = tf.constant([[5, 6], [7, 8]])
z = tf.matmul(x, x) + tf.matmul(x, [[1, 0], [0, 1]])
# Run graph to fetch z.
result = z.eval()
MatMul
const x const
MatMul
+
![Page 18: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/18.jpg)
Parallelism in Op implementations
MatMul
...
...
![Page 19: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/19.jpg)
...
...
...
...
...MatMul
MatMul
Task Parallelism in DataFlow graph
![Page 20: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/20.jpg)
Data Parallelism
MatMul
Input
MatMulParam
...
...MatMul
MatMul
...
...
![Page 21: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/21.jpg)
Model Parallelism
MatMul
Matrix
MatrixMatMul
Split Concat
![Page 22: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/22.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution across Devices
![Page 23: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/23.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
![Page 24: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/24.jpg)
GPU 0 CPU
Add Mul
biases
learning rate
−=......
Distribution● TensorFlow inserts Send/Recv Ops to transport tensors across devices● Recv ops pull data from Send ops
Send Recv
Send Recv
Send
Recv SendRecv
![Page 25: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/25.jpg)
Scale across machines
![Page 26: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/26.jpg)
Portable
![Page 27: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/27.jpg)
Platforms
![Page 28: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/28.jpg)
TensorFlow Core Execution Engine
CPU GPU Android iOS ...
Device Abstraction
![Page 29: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/29.jpg)
Languages
![Page 30: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/30.jpg)
TensorFlow Core Execution Engine
CPU GPU Android iOS ...
Language Abstraction
C++Python ...
![Page 31: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/31.jpg)
Now what?
![Page 32: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/32.jpg)
![Page 33: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/33.jpg)
AutoTrash
![Page 34: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/34.jpg)
AutoTrash
![Page 35: TensorFlow Flexible, Scalable, Portable](https://reader034.vdocuments.mx/reader034/viewer/2022051319/58a1a4051a28ab90398b9601/html5/thumbnails/35.jpg)
Rajat Monga@rajatmonga
Thank You!