efficient deep learning in communications - itu...2018/01/29 · human champ deep net outperforms...
TRANSCRIPT
![Page 1: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/1.jpg)
Image ProcessingFraunhofer
Heinrich Hertz Institute
Fraunhofer Heinrich Hertz Institute, Einsteinufer 37, 10587 Berlin www.hhi.fraunhofer.de
Efficient Deep Learning in Communications
Dr. Wojciech Samek
Fraunhofer HHI, Machine Learning Group
![Page 2: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/2.jpg)
©2
Today’s AI Systems
AlphaGo beats Go
human champ
Deep Net outperforms humans
in image classification
Deep Net beats human at
recognizing traffic signs
DeepStack beats
professional poker players
Computer out-plays
humans in "doom"
Dermatologist-level classification
of skin cancer with Deep Nets
Revolutionizing Radiology
with Deep Learning
![Page 3: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/3.jpg)
©3
Huge volumes of data Computing power Powerful models
Today’s AI Systems
Communications settings are often different.
- Millions of labeled
examples available
- huge models (up to 137 billion
parameters and 1001 layers)
- architectures adapted to images,
speech, text …
- highly parallel processing
- large power consumption
(600 Watts per GPU card)
![Page 4: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/4.jpg)
©4
Many additional requirements: Small size, efficient execution, low energy consumption …
Satellite Communications Autonomous driving
Smartphones Internet of Things 5G Networks
Smart Data
ML in Communications
![Page 5: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/5.jpg)
©5
Distributed setting
Large nonstationarity
Restricted ressources
Communications costs
Interoperability
Security & privacy
Interpretability
Trustworthiness
…
ML in Communications
We need ML techniques which are
adapted to communications
But it’s not only the algorithms, also:
- protocols
- data formats
- frameworks
- mechanisms
- …
![Page 6: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/6.jpg)
©6
Problem 1: Restricted ressources
DNN with Millions of weight parameters
- large size
- energy-hungry training & inference
- floating point operations
Many recent work on compressing neural networks by weight quantization.
![Page 7: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/7.jpg)
©7
Problem 1: Restricted resources
compressed sparse row format
- reduces storage
- fast multiplications
can we do better ?
quantization
(rate-distortion
theory)
![Page 8: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/8.jpg)
©8
Problem 1: Restricted resources
RD-theory based weight quantization does not necessarily lead to sparse matrices.
Weight sharing property: Subsets of connections share the same weight value
rewriting trick
![Page 9: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/9.jpg)
©9
Problem 1: Restricted resources
more efficient format
than CSR
![Page 10: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/10.jpg)
©10
Problem 1: Restricted resources
VGG-16
size: 553 MB, acc: 68.73 %, ops: 30940 M, energy: 71 mJ
Simple quantization (8 bit)
size: 138 MB, acc: 68.52 %, ops: 30940 M, energy: 68 mJ
Sparse format
size: 773 (217) MB, acc: 68.52 %, ops: 29472 M, energy: 65 mJ
iphone8 25 kJ
WS format
size: 247 (99) MB, acc: 68.52 %, ops: 16666 M, energy: 36 mJ
State-of-the-art compression + sparse format
size: 17.8 MB, acc: 68.83 %, ops: 10081 M, energy: 22 mJ
State-of-the-art compression + WS format
size: 12.8 MB, acc: 68.83 %, ops: 7225 M, energy: 16 mJ
![Page 11: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/11.jpg)
©
Black
Box
Problem 2: Interpretability
11
verify
system
understand
weaknesses
legal
aspects learn new
strategies
![Page 12: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/12.jpg)
©
Theoretical interpretation: (Deep) Taylor decomposition of neural network
Black
Box
Problem 2: Interpretability
12
![Page 13: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/13.jpg)
©
Problem 2: Interpretability
Explanation
cat
rooster
dog
?
13
![Page 14: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/14.jpg)
©14
what speaks for / against
classification as “3”
what speaks for / against
classification as “9”
Problem 2: Interpretability
![Page 15: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/15.jpg)
©15
Problem 2: Interpretability
![Page 16: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/16.jpg)
©16
Predictions
25-32 years old
60+ years old
Problem 2: Interpretability
![Page 17: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/17.jpg)
©17
Problem 2: Interpretability
![Page 18: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/18.jpg)
©18
Conclusion
Bringing ML to communications comes with new challenges
AI systems may behave differently than expected
Need for best practices & recommendations (protocols, formats, …)
![Page 19: Efficient Deep Learning in Communications - ITU...2018/01/29 · human champ Deep Net outperforms humans in image classification Deep Net beats human at recognizing traffic signs](https://reader034.vdocuments.mx/reader034/viewer/2022050117/5f4e3a460452c8532f7847ca/html5/thumbnails/19.jpg)
©19
Thank you for your attention
Questions ???
All our papers available on:
http://iphome.hhi.de/samek
Acknowledgement
Simon Wiedemann (HHI)
Klaus-Robert Müller (TUB)
Grégoire Montavon (TUB)
Sebastian Lapuschkin (HHI)
Leila Arras (HHI)
…