deep learning on ios #360idev
TRANSCRIPT
![Page 1: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/1.jpg)
360|iDev 2017
Shuichi Tsutsumi @shu223
Deep Learning on iOS
![Page 2: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/2.jpg)
![Page 3: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/3.jpg)
Overview• How to implement “Deep Learning” on iOS
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
![Page 4: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/4.jpg)
Why so exciting? !
![Page 5: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/5.jpg)
AIphaGo
Cancer Detection
Self-driving Car
![Page 6: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/6.jpg)
AutoDraw
![Page 7: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/7.jpg)
Pose Estimation
http://qiita.com/nnn112358/items/a4490d85dac5827db53b
![Page 8: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/8.jpg)
Frontal View Synthesis
![Page 9: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/9.jpg)
AutoHair
![Page 10: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/10.jpg)
AutoDraw Pose Estimation
Frontal View AutoHair
![Page 11: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/11.jpg)
AutoDraw Pose Estimation
Frontal View AutoHair
This evolutional tech works on iOS"
![Page 12: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/12.jpg)
Deep Learning ON iOS
= works on iOS devices
![Page 13: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/13.jpg)
(Demo)
![Page 14: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/14.jpg)
Image
Result60 times / sec
xUsers
#
#
![Page 15: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/15.jpg)
![Page 16: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/16.jpg)
TOOTHPASTE
![Page 17: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/17.jpg)
!
![Page 18: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/18.jpg)
Train InferenceTrained Params
iOS ML frameworks
![Page 19: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/19.jpg)
Implementation
![Page 20: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/20.jpg)
"""Core ML"""
![Page 21: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/21.jpg)
![Page 22: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/22.jpg)
The largest hall was 100% full for the Core ML session.
![Page 23: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/23.jpg)
✋
![Page 24: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/24.jpg)
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
iOS 11
iOS 10
![Page 25: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/25.jpg)
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Core ML
Vision
Your App
iOS 11
iOS 10
![Page 26: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/26.jpg)
Your App
Metal Performance Shaders (MPSCNN) Accelerate (BNNS) iOS 10
• Optimized for GPU (by ) and CPU (by Accelerate) • Available for iOS 10, too • Basically any ML tools can be used to train the models
Still works!
![Page 27: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/27.jpg)
Metal Performance Shaders (MPSCNN) Accelerate (BNNS)
Your App
GPU CPU
![Page 28: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/28.jpg)
MPSCNN 900 results
Core ML 160,000 results
![Page 29: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/29.jpg)
3 steps to implement w/ MPSCNN
![Page 30: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/30.jpg)
How to implement w/ MPSCNN
Step 1: Create the model
![Page 31: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/31.jpg)
Train InferenceTrained Params
-Which tools can be used for the training? -What kind of formats can be used to pass the pre-trained params?
![Page 32: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/32.jpg)
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
![Page 33: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/33.jpg)
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
hdf5
•The “.dat” files are common binary files. - Not specific for iOS or MPS.
•Contains the trained params - Weights / Biases
•Any other format can be used as long as it can be read by the iOS app. - c.f. hdf5
![Page 34: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/34.jpg)
Which ML tools can be used? : ANY What model format can be used? : ANY
Model (Trained Params)
dat
Train
ML Tools
hdf5
•Any tools which can train CNN, and export the params
![Page 35: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/35.jpg)
How to implement w/ MPSCNN
Step 2: Implement the network
![Page 36: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/36.jpg)
MPSCNNConvolution
MPSCNNFullyConnected
MPSCNNPooling
MPSCNNConvolution
MPSCNNConvolution
MPSCNNPooling
![Page 37: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/37.jpg)
MPSCNNConvolution
MPSCNNFullyConnected
MPSCNNPooling
MPSCNNConvolution
MPSCNNConvolution
MPSCNNPooling
• Almost same name -> Easy to find• Complicated Maths or GPU optimization are encapsulated
Classes corresponding to each CNN layers are provided
![Page 38: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/38.jpg)
How to implement w/ MPSCNN
Step 3: Implement the inference
![Page 39: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/39.jpg)
Input image
MPSImage
Result (UInt, etc.)
CNN • Implemented in Step 2 • Trained params (created in Step 1)
are loaded.
Do something
![Page 40: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/40.jpg)
• Step 1: Create the model - Any ML tools can be used
- Any file format can be used for the trained params
• Step 2: Implement the network - Classes corresponding to each CNN layer are provided
• Step 3: Implement the inference - Input to the CNN, and output from the CNN
![Page 41: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/41.jpg)
Demo : Logo detection
![Page 42: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/42.jpg)
![Page 43: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/43.jpg)
Increased by 70 times
![Page 44: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/44.jpg)
Trained Params
dat
f.write(session.run(w_conv1_p).tobytes())
![Page 45: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/45.jpg)
(Demo)
![Page 46: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/46.jpg)
![Page 47: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/47.jpg)
⚠
![Page 48: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/48.jpg)
CNN for the logo detection
![Page 49: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/49.jpg)
GoogLeNet (Inception v3)
![Page 50: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/50.jpg)
GoogLeNet (Inception v3) Apple’s implementation w/ MPSCNN
Inception3Net.swift
![Page 51: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/51.jpg)
![Page 52: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/52.jpg)
2,000 lines&Only for the CNN
![Page 53: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/53.jpg)
![Page 54: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/54.jpg)
Core ML
![Page 55: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/55.jpg)
Development Flow w/ MPSCNNML Tools
Train
some format
Trained Params
dat
Extract
Parse
dat
MPSCNNConvolution
MPSCNNFullyConnected
Implement Network 2,000
lines&App
![Page 56: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/56.jpg)
Development Flow w/ Core MLML Tools
Train
some format
Trained Params
dat
Extract
Parse
dat
MPSCNNConvolution
MPSCNNFullyConnected
Implement NetworkApp 2,000
lines&
1) Convert w/ coremltools
2) Drag & Drop
some format
Generate
![Page 57: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/57.jpg)
Input image
MPSImage
Result (UInt, etc.)
CNN
Do something
let size = MTLSize(width: inputWidth, height: inputHeight, depth: 1) let region = MTLRegion(origin: MTLOrigin(x: 0, y: 0, z: 0), size: size) network.srcImage.texture.replace( region: region, mipmapLevel: 0, slice: 0, withBytes: context.data!, bytesPerRow: inputWidth, bytesPerImage: 0)
Need to know Metal to use MPSCNN
let origin = MTLOrigin(x: 0, y: 0, z: 0) let size = MTLSize(width: 1, height: 1, depth: 1) finalLayer.texture.getBytes(&(result_half_array[4*i]), bytesPerRow: MemoryLayout<UInt16>.size*1*4, bytesPerImage: MemoryLayout<UInt16>.size*1*1*4, from: MTLRegion(origin: origin, size: size), mipmapLevel: 0, slice: i)
![Page 58: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/58.jpg)
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
![Page 59: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/59.jpg)
Input image
MPSImage
Results
CNN
Do something
let ciImage = CIImage(cvPixelBuffer: imageBuffer)let handler = VNImageRequestHandler(ciImage: ciImage)try! handler.perform([self.coremlRequest])
Don’t need to touch Metal to use Vision
guard let results = request.results as? [VNClassificationObservation] else { return }guard let best = results.first?.identifier else { return }
![Page 60: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/60.jpg)
MPSCNN Accelerate (BNNS)
Your App
![Page 61: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/61.jpg)
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
![Page 62: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/62.jpg)
MPSCNN Accelerate (BNNS)
Core ML
Vision
Your App
How about BNNS?
![Page 63: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/63.jpg)
Deep Learning on CPU?!
![Page 64: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/64.jpg)
CPU: 40 days GPU: 6 days
Deep Learning on CPU?!
![Page 65: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/65.jpg)
Asked @ Lab in WWDC17
![Page 66: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/66.jpg)
“How can we utilize both MPSCNN and BNNS?”
![Page 67: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/67.jpg)
' “Basically use MPSCNN.”
![Page 68: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/68.jpg)
“OK, but, when should I choose BNNS?”
![Page 69: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/69.jpg)
' “I don’t know.”
![Page 70: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/70.jpg)
• He added “watchOS might be a case on that you should use BNNS.”
• Because watchOS doesn’t support MPSCNN, but supports BNNS. (I haven’t tried yet.)
![Page 71: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/71.jpg)
My current understanding:• The cost for passing data between CPU and GPU is
not small • When the network is small, the CPU <-> GPU cost
might be bigger than the benefit of parallel processing.
BNNS might be better when the network is small.
![Page 72: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/72.jpg)
Recap• Why is “Deep Learning on iOS” exciting? • How to implement “Deep Learning” on iOS
- w/ MPSCNN (iOS 10)
- w/ Core ML & Vision (iOS 11)
- When to choose BNNS
MPSCNN BNNS
Core ML
Vision
![Page 73: Deep Learning on iOS #360iDev](https://reader033.vdocuments.mx/reader033/viewer/2022061306/5a65aa857f8b9ab3488b4b15/html5/thumbnails/73.jpg)
Thank you!
https://github.com/shu223