welcome to iscaconf.orgtitle microsoft powerpoint - snapea-isca18.pptx author vakhl created date...

Vahideh Akhlaghi Amir YazdanbakhshKambiz Samadi Rajesh K. Gupta Hadi Esmaeilzadeh

SnaPEA:Predictive Early Activation for Reducing Computation

In Deep Convolutional Neural Networks

University of California, San Diego †Georgia Institute of Technology

‡Qualcomm Technologies, Inc

Equal Contribution

ISCA ’18

* *†

‡

*

CNNs perform trillions of operations for one input

. . . Dog

GoogLeNet 283,000,000,000 Ops

SqueezeNet 222,000,000,000 Ops

AlexNet 1,147,000,000,000 OpsVGG-16 16,362,000,000,000 Ops

CNN models Operations for inference

2

. . . Dog

GoogLeNet 283,000,000,000 Ops

SqueezeNet 222,000,000,000 Ops

AlexNet 1,147,000,000,000 OpsVGG-16 16,362,000,000,000 Ops

CNN models Operations for inference

≥ 90% of operations are for convolutional layers

Convolutions dominate CNN computation

3

Research challenge:How to reduce CNN computation with minimal effect on accuracy?

Our solution: SnaPEA1. Leverage algorithmic structure2. Exploit runtime information3. Tune up with static multi-variable optimization

4

(1) Algorithmic structure of CNNs guides SnaPEA

. . . . . .

Pooling

Conv

ReLU

Norm

al.

Conv

ReLU

Conv

ReLU

Kernels

. . .

𝒘𝒌𝒋𝒊𝒙𝒌𝒋𝒊𝒊𝒋𝒌

5

(1) Algorithmic structure of CNNs guides SnaPEA

. . . . . .

Pooling

Conv

ReLU

Norm

al.

Conv

ReLU

Conv

ReLU

Rectified Linear Unit (ReLU)

Kernels

. . .

𝒘𝒌𝒋𝒊𝒙𝒌𝒋𝒊𝒊𝒋𝒌

6

Opportunity to reduce the computation

Large number of negative convolution outputs (61% on average)

0%

20%

40%

60%

80%

100%

AlexNet GoogLeNet SqueezeNet VGGNet Average

Neg

ativ

e In

puts

to th

e Ac

tivat

ion

Laye

rs

. . .

Conv

. . .

ReLU

GoogLeNet

Black pixels are zero values

7

ReLU makes negative outputs zero: cut convolution short

Early termination of convolution

Blue boxes are the performed operations in two highlighted convolutions

InputOutput

Convolution

8

GoogLeNet

. . . . . .

Pooling

Conv

ReLU

Norm

al.

Conv

ReLU

Conv

ReLU

(2) Runtime information enables reducing computation

Varying distribution of zero and non-zero outputs

Rectified Linear Unit (ReLU)

9

SnaPEA: Principles

SnaPEA: Leveraging algorithmic structure of CNNs and runtime information

Reduce computation without accuracy loss

Trade accuracy for further computation reduction

Add minimal hardware overhead

10

+ + + + + + + + + + + +X

+ - + + + - + - - - + -w

+ - - + + + + + - - + -PartialSum

ReLU 0

Original convolution

SnaPEA: An illustrative example

11

+ + + + + + + + + + + +X

+ - + + + - + - - - + -w

+ - - + + + + + - - + -PartialSum

ReLU 0

+ + + + + + - - - - - -w

+ + + + + + + + + + + +X

+ + + + + + + - - - -PartialSum -

ReLU 0

Convolution in SnaPEA (Exact mode)

Original convolution

SnaPEA: An illustrative example (Exact mode)

12

0

20

40

60

80

100

AlexNet GoogleNet SqueezeNet VGGNet

% N

egat

ive

Wei

ghts

Potential benefits in the exact mode

On average, 54% of the weights are negative13

w

X

+ + +PartialSum

≤ th+ + +

+ - +

SnaPEA: An illustrative example (Predictive mode)

Convolution in SnaPEA (Predictive mode)

Partial Sum

+ + + + + + + + + + + +X

+ - + + + + + - - - - -w

+ + + + + + + + - - - -

Yes

Speculation operations

14

+ + + + + + + + + + + +X

+ - + + + + + - - - - -w

+ + + + + + + + - - - -Partial Sum

No

Speculation operations

X* w x * wGroup 1 Group 2 Group n

Largeabsolute value

Large Small

n largest weights

largest weights from each group

Smallabsolute value

X* w x * w15

Speculation parameters:

Th: Threshold N: Number of speculation operations

Find (Th, N) for all kernels in a CNNto minimize operations and satisfy the accuracy

Optimize the level of speculation

16

… … … …All convolution kernels in a CNN

Layer 2 Layer LLayer 1

(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


17

ThresholdValue

# ofOperations

… … … …Layer 2 Layer LLayer 1

Dog

Per kernel sensitivity analysis



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


18

ThresholdValue

# ofOperations


Cat




(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


19

ThresholdValue

# ofOperations



Dog



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


20

Set of configurations per layer


Dog

…Kernel mKernel 2Kernel 1

…Kernel mKernel 2Kernel 1 …

Dog



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


21


…Kernel kKernel 2Kernel 1

…Kernel kKernel 2Kernel 1 …

Set of configurations per layer

Dog

Dog



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


22


Cat…Layer L

Kernel kLayer 1

Kernel 2Layer 1

Kernel 1

Adjust parameters regarding the cross-layer effect



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


23


Bird…Layer L

Kernel kLayer 1

Kernel 2Layer 1

Kernel 1




(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


24



Dog…Layer L

Kernel kLayer 1

Kernel 2Layer 1

Kernel 1



(Th,N)

Kernel Profiling

Local Optimization

Global Optimization


25

SnaPEA: Hardware implementation

Add low-overhead sign checks and threshold checks to the hardware

… …

PE1,1 PE1,m

PEn,1 PEn,m

≤

Partial result

Terminate

Terminate

Sign-bit

Exact mode

Predictive mode

Threshold

…PAU

MAC

PAU PAU

…MAC MAC

PAU

MAC

PAU PAU

…MAC MAC

…PAU

MAC

PAU PAU

…MAC MAC

PAU

MAC

PAU PAU

…MAC MAC

Prediction Activation Unit (PAU)

26

Compute Lane

Weight and In/Out Buffer

Index Buffer

K Compute Lanes

MAC

Prediction Activation Unit (PAU)

SnaPEA: Hardware implementation

Processing Engine (PE)

27

CNN Model AlexNet2012

GoogLeNet2015

SqueezeNet2016

VGG-162014

Top-1 Accuracy 57.2% 68.7% 57.5% 70.5%

Top-5 Accuracy 80.1% 89.0% 80.3% 89.9%

Benchmarks

Optimization

Hardware implementation

Optimization algorithm built on top of Caffe

Simulation: Cycle accuratePower estimation: Design Compiler using TSMC 45 nmBaseline design: Eyeriss with the same number MAC units (256)SnaPEA area overhead compared to Eyeriss: 4.5%

28

Experimental setup

1.27 1.34 1.29 1.24 1.281.37 1.45 1.44

1.261.38

1.54

2.02

1.52 1.511.63

1.892.08

1.83 1.81 1.81

0

0.5

1

1.5

2

2.5

AlexNet GoogleNet SqueezeNet VGGNet Geomean

Exact Predictive (accuracy loss <= 1%)Predictive (accuracy loss <= 2%) Predictive (accuracy loss <= 3%)

Spee

dup

over

Eye

riss

29

Experimental results

Layers in the predictive mode for accuracy loss ≤ 3%

Network % of Conv Layers

Speedup Energy Improvement

AlexNet 60.0 2.11 1.97GoogleNet 84.2 2.14 2.04

SqueezeNet 65.4 1.94 1.84VGGNet 61.5 1.87 1.73


On average, 68% of layers operate inthe predictive mode (3% accuracy drop). 30

Spee

dup

Highest speedup (3.6x) in a layer in GoogLeNet31


SnaPEAExploit algorithmic structure and runtime information Reduce computations in convolutional layersControl the accuracy with multi-variable optimizationAdd minimal hardware overhead

Future directionsLeverage runtime information (e.g., patterns in inputs and activations)Expand to other activation functions (e.g., sigmoid)Tune up the hardware for more parallelism

Conclusion

32

welcome to iscaconf.orgtitle microsoft powerpoint - snapea-isca18.pptx author vakhl created date...

Documents