![Page 1: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/1.jpg)
Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations
Dayeol Lee†, Gwangmu Lee*, Dongup Kwon*, Sunghwa Lee*, Youngsok Kim*, and Jangwoo Kim*
*Dept. of Electrical and Computer Engineering, Seoul National University†Dept. of Electrical Engineering and Computer Sciences, University of California, Berkeley
![Page 2: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/2.jpg)
2/32
NeuroscienceThe Study of Neurons and Brains
Recognition
Object detection,Classification
Morality,Social Value
Parkinson's Disease,CJD, Dementia
Degeneration
Consciousness Emotion
Sympathy,Happiness, Apathy
![Page 3: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/3.jpg)
3/32
NeuroscienceThe Study of Neurons and Brains
Recognition
Object detection,Classification
Morality,Social Value
Parkinson's Disease,CJD, Dementia
Degeneration
Consciousness Emotion
Sympathy,Happiness, Apathy
DNN did this
Unexplored yet
![Page 4: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/4.jpg)
4/32
SpikeGen.
SpikeGen.
for each time-step:
NeuronSpikeGenerator
Time-step
1
Neuron
Modeling a Brain: Spiking Neural NetworkHow to compute spiking neural network?
Synapse
![Page 5: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/5.jpg)
5/32
SpikeGen.
SpikeGen.
for each time-step:1) stimulus generation
NeuronSpikeGenerator
Time-step
1
Neuron
Modeling a Brain: Spiking Neural NetworkHow to compute spiking neural network?
Synapse
![Page 6: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/6.jpg)
6/32
SpikeGen.
SpikeGen.
for each time-step:1) stimulus generation2) neuron computation
NeuronSpikeGenerator
Time-step
1
Neuron
Modeling a Brain: Spiking Neural NetworkHow to compute spiking neural network?
Synapse
![Page 7: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/7.jpg)
7/32
SpikeGen.
SpikeGen.
NeuronSpikeGenerator
Time-step
1
Neuron
Modeling a Brain: Spiking Neural NetworkHow to compute spiking neural network?
Synapse
for each time-step:1) stimulus generation2) neuron computation3) synapse calculation
…
![Page 8: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/8.jpg)
8/32
10 Representative Benchmarks on CPU/GPUCPU: Intel Xeon E5-2630 v4 CPU (12-core, 2.2 GHz) / GPU: NVIDIA Titan X (Pascal) GPU
Where Does Time Go?
~50% of overheads coming from
Neuron Computation
![Page 9: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/9.jpg)
9/32
MembraneVoltage
(millivolt scale)
Threshold
RestingPotential
Time(millisecond scale)
Neuron
Biological Neuron
How Does a Neuron Behave?
![Page 10: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/10.jpg)
10/32
MembraneVoltage
(millivolt scale)
Threshold
RestingPotential
Time(millisecond scale)
Neuron
Biological Neuron
How Does a Neuron Behave?
![Page 11: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/11.jpg)
11/32
MembraneVoltage
(millivolt scale)
Threshold
RestingPotential
Time(millisecond scale)
Neuron
Biological Neuron
How Does a Neuron Behave?
![Page 12: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/12.jpg)
12/32
MembraneVoltage
(millivolt scale)
Threshold
RestingPotential
Time(millisecond scale)
Neuron
Biological Neuron
How Does a Neuron Behave?
![Page 13: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/13.jpg)
13/32
AxonDendrite Soma
Biological Neuron
Time(millisecond scale)
MembraneVoltage
(millivolt scale)
Spike initiation
Tons of variants exist, depending on their feature set.
Various Neuron Behaviors
Input spikeaccumulation Membrane decay Spike inhibition
We need to support various features for accurate brain simulations.
![Page 14: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/14.jpg)
14/32
CustomHardware
Framework
SoftwareSimulation
Solutions and Limitations
Flexibility
Accuracy
High Performance
Low Energy
![Page 15: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/15.jpg)
15/32
Design Goals & Key Ideas
High PerformanceGoal 1
High FlexibilityGoal 2
Low costGoal 3
Hardware-based
Feature-driven
Spatially-folded
![Page 16: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/16.jpg)
16/32
Neuron Feature #1: Input Spike Accumulation
Current-based Conductance-based(Exponential-shaped)
Conductance-based(Alpha function-shaped)
![Page 17: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/17.jpg)
17/32
Neuron Feature #1: Input Spike Accumulation
Current-based Conductance-based(Exponential-shaped)
Conductance-based(Alpha function-shaped)
![Page 18: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/18.jpg)
18/32
Neuron Feature #2: Spike Initiation
Quadratic(with feature)(without feature)
Exponential
![Page 19: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/19.jpg)
19/32
Neuron Feature #3: Spike-triggered Current
Adaptation Sub-threshold Oscillation(without) (with)
(without) (with)
![Page 20: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/20.jpg)
20/32
Neuron Feature #4: Refractory Period
Absolute Relative
(without) (with) (without) (with)
![Page 21: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/21.jpg)
21/32
(without) (with) (without) (with)
Neuron Feature: Flexible Feature Support
Absolute Relative
“Feature-driven” Flexonsupports 11 major neuron models
(LLIF, SLIF, DSRM0, DLIF, QIF, EIF, Izhikevich, AdEx, …)
![Page 22: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/22.jpg)
22/32
1 10 100 1000
GeomeanNowotny et al.
IzhikevichGeomean
Vogels et al.Vogels-Abbott
Potjans-DiesmannMuller et al.
Destexhe-UpDownDestexhe-LTS
BrunelBrette et al.
GPU
CPU
Speed-up(Normalized to the baseline)
8 CPU + 2 GPU Representative BenchmarksFlexon: TSMC 45nm, Synopsys Design Compiler (neuron), CACTI 6.5 (SRAM)
87.4xOver CPU
8.19xOver GPU
Evaluation (12x Feature-driven Design)
CPU
Intel Xeon E5-2630 v4(12-core, 2.2 GHz)
GPUNVIDIA Titan X (Pascal)
![Page 23: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/23.jpg)
23/32
1 10 100 1000 10000 100000
GeomeanNowotny et al.
IzhikevichGeomean
Vogels et al.Vogels-Abbott
Potjans-DiesmannMuller et al.
Destexhe-UpDownDestexhe-LTS
BrunelBrette et al.
GPU
CPU
Energy Efficiency(Normalized to the baseline)
8 CPU + 2 GPU Representative BenchmarksFlexon: TSMC 45nm, Synopsys Design Compiler (neuron), CACTI 6.5 (SRAM)
6,186xOver CPU
422xOver GPU
Evaluation (12x Feature-driven Design)
CPU
Intel Xeon E5-2630 v4(12-core, 2.2 GHz)
GPUNVIDIA Titan X (Pascal)
![Page 24: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/24.jpg)
24/32
Intrinsic Space-inefficiency
(without) (with) (without) (with)
Absolute Relative
“Feature-driven” Flexonsupports 11 major neuron models
(LLIF, SLIF, DSRM0, DLIF, QIF, EIF, Izhikevich, AdEx, …)
Lots of redundant units(multiplier, adder, …)
![Page 25: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/25.jpg)
25/32
Constructing Spatially-folded Flexon
Conductance-based(Exponential)
Quadratic
Adaptation
Spatially-folded design è reduce area- Remove redundant MAC operators
![Page 26: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/26.jpg)
26/32
Constructing Spatially-folded Flexon
Exponential Relative(ADT: Adaptation)
- Remove redundant MAC operators
Modifications from the baseline- 2-stage pipeline, multi-cycle implementation
Spatially-folded design è reduce area
![Page 27: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/27.jpg)
27/32
Constructing Spatially-folded Flexon
“Spatially-folded” Flexonsupports various major neuron models
6x area saving
- Remove redundant MAC operators
What we should change- 2-stage pipeline, multi-cycle implementation
Spatially-folded design è reduce area
![Page 28: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/28.jpg)
28/32
1 10 100 1000
GeomeanNowotny et al.
IzhikevichGeomean
Vogels et al.Vogels-Abbott
Potjans-DiesmannMuller et al.
Destexhe-UpDownDestexhe-LTS
BrunelBrette et al.
GPU
CPU
Speed-up(Normalized to the baseline)
8 CPU + 2 GPU Representative BenchmarksFlexon: TSMC 45nm, Synopsys Design Compiler (neuron), CACTI 6.5 (SRAM)
122xOver CPU
9.83xOver GPU
Evaluation (72x Spatially-folded Design)
CPU
Intel Xeon E5-2630 v4(12-core, 2.2 GHz)
GPUNVIDIA Titan X (Pascal)
Feature-drivenSpatially-folded
![Page 29: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/29.jpg)
29/32
1 10 100 1000 10000 100000
GeomeanNowotny et al.
IzhikevichGeomean
Vogels et al.Vogels-Abbott
Potjans-DiesmannMuller et al.
Destexhe-UpDownDestexhe-LTS
BrunelBrette et al.
GPU
CPU
Energy Efficiency(Normalized to the baseline)
8 CPU + 2 GPU Representative BenchmarksFlexon: TSMC 45nm, Synopsys Design Compiler (neuron), CACTI 6.5 (SRAM)
Evaluation (72x Spatially-folded Design)
CPU
Intel Xeon E5-2630 v4(12-core, 2.2 GHz)
GPUNVIDIA Titan X (Pascal)
5,413xOver CPU
135xOver GPU
Feature-drivenSpatially-folded
![Page 30: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/30.jpg)
30/32
Baseline Flexon vs. Spatially-folded Flexon
• Baseline “Feature-driven” Flexon−Fast: 87.4x over CPUs, 8.19x over GPUs
−Energy-efficient: 6,186x over CPUs, 422x over GPUs
• “Spatially-folded” Flexon−Fast: 122x over CPUs, 9.83x over GPUs
−Energy-efficient: 5,413x over CPUs, 135x over GPUs
![Page 31: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/31.jpg)
31/32
Conclusion
• Flexon is a flexible feature-driven digital neuron design,capable of realizing various major neuron models.−Flexible & power-efficient (6,186x over CPU)
• Spatially-folded Flexon makes features share units,reducing 6x circuit area.−Flexible & fast when integrated (122x over CPU)
![Page 32: Flexon: A Flexible Digital Neuron for Efficient Spiking ...iscaconf.org/isca2018/slides/4A1.pdf · Flexon: A Flexible Digital Neuron for Efficient Spiking Neural Network Simulations](https://reader034.vdocuments.mx/reader034/viewer/2022042621/5f6d3026da361221b1132f34/html5/thumbnails/32.jpg)
32/32
Thank you for listening
FlexonA Flexible Digital Neuron for EfficientSpiking Neural Network Simulations