DOUBLESQUEEZE: Parallel Stochastic Gradient Descent with Double-passError-Compensated Compression

Hanlin Tang 1 Xiangru Lian 1 Chen Yu 1 Tong Zhang 2 Ji Liu 3 1

AbstractA standard approach in large scale machine learn-ing is distributed stochastic gradient training,which requires the computation of aggregatedstochastic gradients over multiple nodes on a net-work. Communication is a major bottleneck insuch applications, and in recent years, compressedstochastic gradient methods such as QSGD (quan-tized SGD) and sparse SGD have been proposedto reduce communication. It was also shownthat error compensation can be combined withcompression to achieve better convergence ina scheme that each node compresses its localstochastic gradient and broadcast the result to allother nodes over the network in a single pass.However, such a single pass broadcast approachis not realistic in many practical implementations.For example, under the popular parameter-servermodel for distributed learning, the worker nodesneed to send the compressed local gradients tothe parameter server, which performs the aggre-gation. The parameter server has to compressthe aggregated stochastic gradient again beforesending it back to the worker nodes. In this work,we provide a detailed analysis on this two-passcommunication model, with error-compensatedcompression both on the worker nodes and onthe parameter server. We show that the error-compensated stochastic gradient algorithm admitsthree very nice properties: 1) it is compatible withan arbitrary compression technique; 2) it admitsan improved convergence rate than the non error-compensated stochastic gradient methods suchas QSGD and sparse SGD; 3) it admits linearspeedup with respect to the number of workers.An empirical study is also conducted to validateour theoretical results.

1University of Rochester 2Hong Kong University of Scienceand Technology 3Seattle AI Lab, FeDA Lab, Kwai Inc. Correspon-dence to: Hanlin Tang <htang14@ur.rochester.edu>.

Proceedings of the 36 th International Conference on MachineLearning, Long Beach, California, PMLR 97, 2019. Copyright2019 by the author(s).

1. IntroductionLarge scale distributed machine learning on big data setsare important for many modern applications (Abadi et al.,2016a; Seide & Agarwal, 2016), and there are many meth-ods being studied to improve the performance of distributedlearning, such as communication efficient learning (Alis-tarh et al., 2017; Bernstein et al., 2018a; Seide et al., 2014),decentralized learning (He et al., 2018b; Lian et al., 2017),and asynchronous learning (Agarwal & Duchi, 2011; Lianet al., 2015; Recht et al., 2011). All these methods havebeen proved to be quite efficient in accelerating distributedlearning under different seneario.

A widely used framework in distributed learning is data par-allelism, where we assume that data are distributed over mul-tiple nodes on a network, with a shared model that needs tobe jointly optimized. Mathematically, the underlying prob-lem can be posed as the following distributed optimizationproblem:

minx

f(x) =1

n

n∑i=1

Eζ∼DiF (x; ζ), (1)

where n is the number of workers, Di is the local datadistribution for worker i (in other words, we do not assumethat all nodes can access the same data set), and F (x; ζ) isthe local loss function of model x given data ζ for worker i.

A standard synchronized approach for solving (1) is parallelSGD (stochastic gradient descent) (Bottou & Bottou, 2010),where each worker i draws ζ(i) from Di, and compute thelocal stochastic gradient with respect to the shared parameterx:

g(i) = ∇F (x; ζ(i)),

The local gradients are sent over the network, where theaggregated SGD is computed as:

g =1

n

n∑i=1

g(i),

and the result are sent back to each local node.

The high communication cost is a main bottleneck for largescale distributed training. In order to alleviate this cost,

DOUBLESQUEEZE

recently it has been suggested that each worker can senda compressed version of the local gradient (Alistarh et al.,2018; Stich et al., 2018; Zhang et al., 2017a), with methodssuch as quantization or sparsification. Specifically, let Qω[·]be a compression operator, one will transmit

Qω[g(i)]

to form aggregated gradient1. However, it is observed thatsuch compression methods slow down the convergence dueto the loss of information under compression. To remedythe problem, error compensation has been proposed (Seideet al., 2014), and successfully used in practical applications.The idea is to keep aggregated compression error in a vectorδ(i), and send Qω

[g(i) + δ(i)

], where we update δ(i) by

using the following recursion at each time step

δ(i) = g(i) + δ(i) −Qω[g(i) + δ(i)

].

It was recently shown that such methods can be effectivelyused to accelerate convergence when the compression ratiois high.

However, previous work assume that the error compensationis done only for each worker, g(i), but not for the aggregatedgradient g (Stich et al., 2018). This is impractical for realworld applications, since it can save up to 50% bandwidth.For example, in the popular parameter server model, theaggregation of local gradient is done at the parameter server,and then sent back to each worker node. Although eachworker can send sparsified local stochastic gradient to theparameter server, and thus reduce the communication cost.However, the aggregated gradient g can become dense, andto save the communication cost, it needs to be compressedagain before sending back to the worker nodes (Wangniet al., 2018). In such case, it is necessary to incorporateerror compensation on the parameter server as well becauseonly the parameter server can keep track of the historiccompression error. In this paper, we study an error compen-sated compression of stochastic gradient algorithm namelyDOUBLESQUEEZE under this more realistic setting.

The contribution of this paper can be summarized as follows:

• Better tolerance to compression: Our theoreticalanalysis suggests that the proposed DOUBLESQUEEZEenjoys a better tolerance than the non-error-compensated algorithms.

• Optimal communication cost: There are only nrounds of communication at each iteration (comparedto Alistarh et al. (2018); Wu et al. (2018) where thereare n2 rounds) and we could ensure that all the infor-mation to be sent is compressed (compared to (Stich

1Qω[·] could also include randomness.

et al., 2018) where only half of the information sendfrom workers to the server is compressed).

• Prove for parallel case: To the best of our knowl-edge, this is the first work that gives the conver-gence rate analysis for a parallel implementation oferror-compensated SGD, and our result shows a linearspeedup corresponding to the number of workers n.To the best of our knowledge, this is the first resultto show the speedup property for error compensatedalgorithms.

• Proof of acceleration for Non-Convex case: To thebest of our knowledge, this is the first work where theloss function in our work is only assumed to be non-convex, which is the case for most of the real worlddeep neural network, and still prove that the error-compensated SGD admits a factor of improvementover the non-compensated SGD. In Wu et al. (2018)they only consider the quadratic loss function and inStich et al. (2018) they consider a strongly-convex lossfunction. Alistarh et al. (2018) considers a non-convexloss function but they did not prove an acceleration oferror-compensated SGD.

Notations and definitions Throughout this paper, we usethe following notations and definitions

• ∇f(x) denotes the gradient of a function f(·).

• f∗ denotes the optimal solution to (1).

• ‖ · ‖ denotes the l2 norm for vectors.

• ‖ · ‖2 denotes the spectral norm for matrix.

• fi(x) := Eζ∼DiF (x; ζ).

• . means “less than equal to up to a constant factor”.

2. Related Work2.1. Distributed Learning

Nowadays, distributed learning has been proved to the keystrategy for accelerating the deep learning training. Thereare two kinds of designs for parallelism: centralized de-sign (Agarwal & Duchi, 2011; Recht et al., 2011), wherethe network is designed to ensure that all workers can getinformation from all others, and decentralized design (Heet al., 2018a; Li & Yan, 2017; Lian et al., 2017; 2018; Shiet al., 2015; Tang et al., 2018b), where each worker is onlyallowed to communicate with its neighbors.

DOUBLESQUEEZE

Centralized Parallel Training In centralized paralleltraining, the network is designed to ensure that all work-ers can get information of all others. One key primitivein centralized training is to aggregate all local models orgradients. This primitive is called the collective communica-tion operator in HPC literature (Thakur et al., 2005). Thereare different implementations for information aggregationin centralized systems. For example, the parameter server(Abadi et al., 2016b; Li et al., 2014) and AllReduce thataverages models over a ring topology (Renggli et al., 2018;Seide & Agarwal, 2016).

Decentralized Parallel Training In decentralized paral-lel training, the network does not ensure that all workerscould get information of all others in a single step. Theycan only communicate with their individual neighbors. De-centralized training can be divided into fixed topology algo-rithms and random topology algorithms. For fixed topologyalgorithms, the communication network is fixed, for exam-ple, (Jin et al., 2016; Lian et al., 2017; Shen et al., 2018;Tang et al., 2018b;c). For the random topology decentralizedalgorithms, the communication network changes over time,for example, (Lian et al., 2018; Nedic & Olshevsky, 2015;Nedic et al., 2017). All of these works provide rigorousanalysis to show the convergence rate. For example, Lianet al. (2017) proves that the decentralized SGD achieves acomparable convergence rate to the centralized SGD algo-rithm, but significantly reduces the communication cost andis more suitable for training a large model.

There has been lots of works studying the implementationof distributed learning from different angles. such as differ-entially private distributed optimization (Jayaraman et al.,2018; Zhang et al., 2018), adaptive distributed ADMM (Xuet al., 2017), adaptive distributed SGD (Cutkosky & Busa-Fekete, 2018), non-smooth distributed optimization (Sca-man et al., 2018), distributed proximal primal-dual algo-rithm (Hong et al., 2017), projection-free distributed onlinelearning (Zhang et al., 2017b). Some works also investigatemethods for parallel backpropgation (Huo et al., 2018; Liet al., 2018).

2.2. Compressed Communication Learning

In order to save the communication cost, a widely usedapproach is to compress the gradients (Shen et al., 2018).In Wang et al. (2017), the communication cost is reducedby sending a sparsified model from the parameter server toworkers. An adaptive approach for doing the compression isproposed in Chen (2018). Alistarh et al. (2017) gives a theo-retical analysis for QSGD and studies the tradeoff betweenlocal update and communication cost. Most of the previouswork used unbiased quantizing operation (Jiang & Agrawal,2018; Tang et al., 2018a; Wangni et al., 2018; Zhang et al.,2017a) to ensure the convergence of the algorithm. Some

extension of communication efficient distributed learning,such as differentially private optimization (Agarwal et al.,2018), optimization on manifolds (Saparbayeva et al., 2018),compressed PCA (Garber et al., 2017), are also studied re-cently.

In Seide et al. (2014), a 1Bit-SGD was proposed to utilizeonly the sign of each element in the gradient vector forstochastic gradient descent. The convergence rate guaranteeof 1Bit-SGD is studied recently in Bernstein et al. (2018a;b).In Wen et al. (2017), authors manipulate the 1Bit-SGD toensure that the compressed is an unbiased estimation of theoriginal gradient, and they prove that this unbiased 1Bit-SGD could ensure the algorithm to converge to the singleminimum.

Some specific methods for implementing the compressionfor other distributed systems is also studied. In Suresh et al.(2017), authors proposed some communication efficientstrategies distributed mean estimation. A Lazily AggregatedGradient (LAG) strategy is studied to reduce the communi-cation cost for Gradient Descent based distributed learning(Chen et al., 2018). Wang et al. (2018) proposed an atomicsparsification strategy for gradient sparsification.

2.3. Error-Compensated SGD

In Seide et al. (2014), authors used an error-compensatestrategy to compensate the error for AllReduce 1Bit-SGD,and found in experiments that the accuracy drop could be mi-nor as long as the error is compensated. Recently, Wu et al.(2018) studied an Error-Compensated SGD for quadraticoptimization via adding two hyperparameters to compensatethe error, but does not successfully prove the advantage ofusing error compensation theoretically. In Stich et al. (2018),authors adapted the error-compensate strategy for compress-ing the gradient, and proved that the error-compensatingprocedure could greatly reduce the influence of the quanti-zation for non-parallel and strongly-convex loss functions.But their theoretical results are restricted to the compressingoperators whose expectation compressing error cannot belarger than the magnitude of the original vector, which isnot the case for some biased compressing methods, such asSignSGD (Bernstein et al., 2018a). Alistarh et al. (2018)studied the error-compensated SGD under a non-convexloss function, but did not prove that the error-compensatemethod could admit a factor of acceleration compared tothe non-compensate ones. All of those works did not provea linear speedup corresponding to the number of workers nfor a parallel learning case.

3. Parallel Error-Compensated AlgorithmsIn this section, we will introduce the parallel error-compensated SGD algorithm, namely DOUBLESQUEEZE.

DOUBLESQUEEZE

Algorithm 1 DOUBLESQUEEZE

1: Input: Initialize x0, learning rate γ, initial error δ = 0,and number of total iterations T .

2: for t = 1, 2, · · · , T do3: On worker4: Compute the error-compensated stochastic gra-

dient v(i) ← ∇F (x; ζ(i)) + δ(i)

5: Compress v(i) into Qω[v(i)]

6: Update the error δ(i) ← v −Qω[v(i)]

7: Send Qω[v(i)]

to the parameter server8: On parameter server9: Average all gradients received from workers

v ← 1n

∑ni=1Qω

[v(i)]

+ δ10: Compress v into Qω [v]11: Update the error δ ← v −Qω [v]12: Send Qω [v] to workers13: On worker14: Update the local model x← x− γQω [v]15: end for16: Output: x

We first introduce the algorithm details. Next we willgive its mathematical updating formulation from a globalview of point in order to get a better understanding of theDOUBLESQUEEZE algorithm.

3.1. Algorithm Description

In this paper, we consider a parameter-server (PS) archi-tecture for parallel training for simplicity – a parameterserver and n workers, but the proposed DOUBLESQUEEZEalgorithm is not limit to the parameter server architecture.DOUBLESQUEEZE essentially applies the error-compensatestrategy on both workers and the parameter to ensure thatall information communicated is compressed.

During the tth iteration, the key updating rule forDOUBLESQUEEZE is described below:

• (Worker: Compute) Each worker i computes thelocal stochastic gradient ∇F (xt; ζ

(i)t ), based on the

global model xt and local sample ζ(i)t . Here i is theindex for worker i and t is the index for iteration num-ber.

• (Worker: Compress) Each worker i computes theerror-compensated stochastic gradient

v(i)t = ∇F (xt; ζ

(i)t ) + δ

(i)t−1, (2)

and update the local error of tth step δ(i)t according to

δ(i)t =v

(i)t −Qω(i)

t[v

(i)t ], (3)

whereQω

(i)t

[v(i)t ] is the compressed error-compensated

stochastic gradient.

• (Parameter server: Compress) All workers sendQω

(i)t

[v(i)t

]to the parameter server, then the param-

eter server average all Qω

(i)t

[v(i)t

]s and update the

global error-compensated stochastic gradient vt, to-gether with the global error δt according to

vt =δt−1 +1

n

n∑i=1

Qω

(i)t

[v(i)t

]δt =vt −Qωt

[vt] . (4)

• (Worker: Update) The parameter server sendsQωt

[vt] to all workers. Then each worker updatesits local model using

xt+1 = xt − γQωt [vt],

where γ is the learning rate.

It is worth noting that all information exchanged betweenworkers and parameter server under the DOUBLESQUEEZEframework is compressed. As a result, the required band-width could be extremely low (much lower than 10%). Com-paring to some recent error compensated algorithms (Stichet al., 2018), they only compress the gradient sent from theworker to the PS and still send dense vector from PS toworkers, which can only save bandwidth up to 50%.

3.2. Compression options

Note that here unlike many existing work (Alistarh et al.,2017; Jiang & Agrawal, 2018), we do not require the com-pression to be unbiased, which means we do not assumeEωQω[x] = x. So the choice of compression in our frame-work is pretty flexible. We list a few commonly options forQω[·]2:

• Randomized Quantization: (Alistarh et al., 2017;Zhang et al., 2017a) For any real number z ∈ [a, b] (a,b are pre-designed low-bit number), with probabilityb−zb−a compress p into a, and with probability z−a

b−a com-press z into b. This compression operator is unbiased.

• 1-Bit Quantization: Compress a vector x into‖x‖sign(x), where sign(x) is a vector whose ele-ment take the sign of the corresponding element inx (see Bernstein et al. (2018a)). This compressionoperator is biased.

2Deterministic operator can be considered as a special case ofthe randomized operator.

DOUBLESQUEEZE

• Clipping: For any real number z, directly set its lowerk bits into zero. For example, deterministically com-press 1.23456 into 1.2 with its lower 4 bits set to zero.This compression operator is biased.

• Top−k sparsification: (Stich et al., 2018) For anyvector x, compress x by retaining the top k largestelements of this vector and set the others to zero. Thiscompression operator is biased.

• Randomized Sparsification: (Wangni et al., 2018)For any real number z, with probability p set z to 0and z

p with probability p. This is also an unbiasedcompression operator.

3.3. Mathematical form of the updating rule byDOUBLESQUEEZE

Below we are going to prove that the updating rule ofDOUBLESQUEEZE admits the form

xt+1 = xt − γ∇f(xt) + γξt − γΩt−1 + γΩt, (5)

where

Ωt :=δt +1

n

n∑i=1

δ(i)t

ξt :=1

n

n∑i=1

(∇f(xt)−∇F

(xt; ζ

(i)t

)). (6)

Here δ(i)t and δt are computed according to (3) and (4).

According to the algorithm description in Section 3.1, weknow that the updating rule for the global model xt can bewritten as

xt+1 − xt

=− γQωt

[δt−1 +

1

n

n∑i=1

Qω

(i)t

[v(i)t

]]

=− γ

((δt−1 +

1

n

n∑i=1

Qω

(i)t

[v(i)t

])− δt

)(from (4))

=− γ

n

n∑i=1

Qω

(i)t

[v(i)]− γδt−1 + γδt

=− γ

n

n∑i=1

(v(i)t − δ

(i)t

)− γδt−1 + γδt (from (3))

=− γ

n

n∑i=1

(∇F (xt; ζ

(i)t ) + δ

(i)t−1 − δ

(i)t

)− γδt−1 + γδt (from (2))

=− γ

n

n∑i=1

∇F (xt; ζ(i)t )− γΩt−1 + γΩt

=− γ∇f(xt) + γξt − γΩt−1 + γΩt.

4. Convergence AnalysisIn this section, we are going to give the convergence rate ofDOUBLESQUEEZE, and from the theoretical result we shallsee that DOUBLESQUEEZE is quite efficient in the way thatit could reduce the side effect of the compression. For theconvenience of further discussion, we first introduce someassumptions that are necessary for theoretical analysis.

Assumption 1. We make the following assumptions:

1. Lipschitzian gradient: f(·) is assumed to be with L-Lipschitzian gradients, which means

‖∇f(x)−∇f(y)‖ ≤ L‖x− y‖, ∀x,∀y,

2. Bounded variance: The variance of the stochastic gra-dient is bounded

Eζ∼Di‖∇F (x; ζ)−∇f(x)‖2 ≤ σ2, ∀x,∀i.

3. Bounded magnitude of error for Qω[·]: The magni-tude of worker’s local errors δ(i)t (defined in (3)), andthe server’s global error δt (defined in (4)), are as-sumed to be bounded by a constant ε

Eω∥∥∥δ(i)t ∥∥∥ ≤ ε2 , ∀t,∀i,

Eω ‖δt‖ ≤ε

2, ∀t.

Here the first and second assumptions are commonly usedfor non-convex convergence analysis. The third assumptionis used to restrict the compression. It can be obtained fromthe following commonly used assumptions (Stich et al.,2018):

E‖Cω[x]− x‖2 ≤α2‖x‖2, α ∈ [0, 1),∀x‖∇f(x)‖2 ≤G2, ∀x,

where α is a constant specifies the compression level and isnot required to be bounded in [0, 1). Because

E‖δt‖2 = E‖Cω[gt − δt−1]− gt + δt−1‖2, (7)

where gt := 1n

∑ni=1∇F (xt; ξt) is the sum of all stochas-

tic gradient at each iteration, then from (7) we have

E‖δt‖2

≤Eα2‖gt − δt−1‖2

≤(1 + ρ)α2E‖gt‖2 +

(1 +

1

ρ

)α2E‖δt−1‖2

≤(1 + ρ)α2E‖gt‖2 +((1 + ρ)α2

)2 E‖gt−1‖2+

((1 +

1

ρ

)α2

)2

E‖δt−2‖2

DOUBLESQUEEZE

≤t∑

s=1

((1 + ρ)α2

)t−s E‖gs‖2 +

((1 +

1

ρ

)α2

)tE‖δ0‖2

≤t∑

s=1

((1 + ρ)α2

)t−s E‖gs‖2 (δ0 = 0)

≤G2 + σ2

n

1− (1 + ρ)α2.

(E‖gt‖2 ≤ G2 +

σ2

n

)Here ρ > 0 can be any positive constant. So ‖δt‖2 wouldbe bounded as long as α < 1√

1+ρ, which is equivalent to

α ∈ [0, 1) since ρ can be any positive number.

Next we are ready to present the main theorem forDOUBLESQUEEZE.

Theorem 1. Under Assumption 1, for DOUBLESQUEEZE,we have the following convergence rate(

γ

2− Lγ2

2

) T−1∑t=0

E ‖∇f(xt)‖2

≤Ef(x0)− Ef(x∗) +Lγ2σ2T

2n+ 2L2ε2γ3T.

Given the generic result in Theorem 1, we obtain the conver-gence rate for DOUBLESQUEEZE with appropriately chosenthe learning rate γ.

Corollary 2. Under Assumption 1, for DOUBLESQUEEZE,choosing

γ =1

4L+ σ√

Tn + ε

23T

13

,

we have the following convergence rate

1

T

T−1∑t=0

E‖∇f(xt)‖2 .σ√nT

+ε

23

T23

+1

T,

where we treat f(x1)− f∗ and L as constants.

This result suggests that

• (Comparison to SGD) DoubleSqueeze essentially ad-mits the same convergence rate as SGD in the sensethat both of them admit the asymptotical convergencerate O(1/

√T );

• (Linear Speedup) The asymptotical convergence rateof DOUBLESQUEEZE is O(1/

√nT ), the same conver-

gence rate as Parallel SGD. It implies that the averagedsample complexity is O(1/(nε2)). To the best of ourknowledge, this is the first analysis to show the linearspeedup for the error compensated type of algorithms.

• (Advantage over non error-compensated SGD (Al-istarh et al., 2017; Wangni et al., 2018)) For non

error-compensated SGD, there is no guarantee for con-vergence in general unless the compression operatoris unbiased. Using the existing analysis for SGD’sconvergence rate

1

T

T−1∑t=0

E‖∇f(xt)‖2 .σ′√nT

+1

T

where σ′ is the stochastic variance, it is not hard toobtain the following convergence rate for unbiasedcompressed SGD:(one-pass compressed SGD on workers such as QSGD(Alistarh et al., 2017) and sparse SGD (Wangni et al.,2018))

1

T

T−1∑t=0

E‖∇f(xt)‖2 .σ√nT

+ε√nT

+1

T

(double-pass compressed SGD on workers and the pa-rameter server)

1

T

T−1∑t=0

E‖∇f(xt)‖2 .σ√T

+ε√T

+1

T.

Note that ε measures the upper bound of the (stochas-tic) compression variance. Therefore, when ε is domi-nant, the convergence rate for DOUBLESQUEEZE has amuch better dependence on ε in terms of iteration num-ber T . It means that DOUBLESQUEEZE has a muchbetter tolerance on the compression variance or bias.It makes sense since DOUBLESQUEEZE does not dropany information in stochastic gradients just delay toupdate some portion in them.

5. ExperimentsWe validate our theory with experiments that comparedDOUBLESQUEEZE with other compression implementa-tions. We run experiments with 1 parameter server and8 workers, and show that, the DOUBLESQUEEZE convergessimilar to SGD without compression, but runs much fasterthan vanilla SGD and other compressed SGD algorithmswhen bandwidth is limited.

5.1. Experiment setting

Datasets and models We evaluate DOUBLESQUEEZE bytraining ResNet-18 (He et al., 2016) on CIFAR-10. Themodel size is about 44MB.

Implementations and setups We evaluate five SGD im-plementations:

1. DOUBLESQUEEZE. Both workers and the parame-ter server compress gradients. The error caused by

DOUBLESQUEEZE

compression are saved and used to compensate newgradients as shown in Algorithm 1. We evaluateDOUBLESQUEEZE with two compression methods:

• 1-bit compression: The gradients are quantizedinto 1-bit representation (containing the sign ofeach element). Accompanying the vector, a scal-ing factor is computed as

magnitude of compensated gradientmagnitude of quantized gradient

.

The scaling factor is multiplied onto the quan-tized gradient whenever the quantized gradient isused, so that the recovered gradient has the samemagnitude of the compensated gradient.

• Top-k compression: The compensated gradientsare compressed so that only the largest k elements(in the sense of absolute value) are kept, and allother elements are set to 0.

2. QSGD (Alistarh et al., 2017). The workers quantizethe gradients into a tenary representation, where eachelement is in the set −1, 0, 1. Assuming the elementwith maximum absolute value in a gradient vector ism, for any other element e, it has a probability of|e|/|m| to be quantized to sign(e), and a probabilityof 1 − |e|/|m| to be quantized to 0. A scaling factorlike the one in DOUBLESQUEEZE is computed as

magnitude of original gradientmagnitude of compressed gradient

.

The parameter server aggregates the gradients andsends the aggregated gradient back to all workers with-out compression.

3. Vanilla SGD. This is the common centralized parallelSGD implementation without compression, where theparameter server aggregates all gradients and sends itback to each worker.

4. MEM-SGD. As in DOUBLESQUEEZE, workers doboth compression and compensation. However, theparameter server aggregates all gradients and sendsit back to all workers without compression as shownin Stich et al. (2018). For MEM-SGD, we also eval-uate both 1-bit compression and top-k compressionmethods.

5. Top-k SGD. This is vanilla SGD with top-k compres-sion in each worker, without compensation.

For more direct comparison, no momentum and weightdecay are used in the optimization process. The learningrate starts with 0.1 and is reduced by a factor of 10 every160 epochs. The batch size is set to 256 on each worker.Each worker computes gradients on a Nvidia 1080Ti.

0 100 200 300epoch

0.0

0.5

1.0

1.5

Loss

Vanilla SGDDoubleSqueezeMEM-SGDQSGD

Figure 1: Training loss w.r.t. epochs for DOUBLESQUEEZE(1-bit compression), QSGD, Vanilla SGD, and MEM-SGD(1-bit compression) on CIFAR-10.

0 100 200 300epoch

020406080

100

Accu

racy

Vanilla SGDDoubleSqueezeMEM-SGDQSGD

Figure 2: Testing accuracy w.r.t. epochs forDOUBLESQUEEZE (1-bit compression), QSGD, VanillaSGD, and MEM-SGD (1-bit compression) on CIFAR-10.

5.2. Experiment results

The empirical study is conducted on two compression ap-proaches: 1-bit compression and top-k compression.

1-bit compression We apply the 1-bit compression toDOUBLESQUEEZE, MEM-SGD, QSGD, and report resultsfor the training loss w.r.t. epochs in Figure 1. The resultshows that with 1-bit compression DOUBLESQUEEZE andMEM-SGD converge similarly w.r.t. epochs as Vanilla SGD,while QSGD converges much slower due to the lack of com-pensation. For testing accuracy, we have similar results, asshown in Figure 2.

While DOUBLESQUEEZE, MEM-SGD, and Vanilla SGDconverges similarly w.r.t. epochs, when network bandwidthis limited, DOUBLESQUEEZE can be much faster than otheralgorithms as shown in Figure 3.

Top-k compression For the top-k compression method,we choose k = 300000, which is about 1/32 of the numberof parameters in the model. We report results for the trainingloss and testing accuracy w.r.t. epochs in Figure 4 and

DOUBLESQUEEZE

0.02 0.04 0.06 0.08 0.10Bandwidth (1/Mb)

0

100

200

300

400

500se

cond

sVanilla SGDDoubleSqueezeMEM-SGDQSGD

Figure 3: Per iteration time cost for DOUBLESQUEEZE (1-bit compression), QSGD, MEM-SGD (1-bit compression),and Vanilla SGD, under different network environments.The x-axis represents the inverse of the bandwidth of theparameter server. The y-axis is the number of secondsneeded to finish one iteration.

0 50 100 150 200epoch

0.0

0.5

1.0

Loss

Vanilla SGDDoubleSqueezeMEM-SGDTop-k SGD

Figure 4: Training loss w.r.t. epochs for DOUBLESQUEEZE(top-k compression), Top-k SGD, Vanilla SGD, and MEM-SGD (top-k compression) on CIFAR-10. k = 300000.

Figure 5, respectively, for Vanilla SGD, DOUBLESQUEEZE,MEM-SGD, and Top-k SGD. With the top-k compression,all methods converge similarly w.r.t. epochs. The Top-kSGD method converges a little bit slower.

Similar to what we observed in the 1-bit compres-sion experiment, when network bandwidth is limited,DOUBLESQUEEZE can be much faster than other algo-rithms as shown in Figure 6.

6. ConclusionIn this paper, we study an error-compensated SGD algo-rithm, namely DOUBLESQUEEZE that performs the com-pression on both the worker’s side and the parameter server’sside, to ensure that all information exchanged over the net-work is compressed. As a result, this approach can sig-nificantly save the bandwidth, unlike many existing errorcompensated algorithms that can only save bandwidth upto 50%. Theoretical convergence for DOUBLESQUEEZE

0 50 100 150 200epoch

020406080

100

Accu

racy

Vanilla SGDDoubleSqueezeMEM-SGDTop-k SGD

Figure 5: Testing accuracy w.r.t. epochs forDOUBLESQUEEZE (top-k compression), Top-k SGD,Vanilla SGD, and MEM-SGD (top-k compression) onCIFAR-10. k = 300000.

0.02 0.04 0.06 0.08 0.10Bandwidth (1/Mb)

0

100

200

300

400

500

seco

nds

Vanilla SGDDoubleSqueezeMEM-SGDTop-k SGD

Figure 6: Per iteration time cost for DOUBLESQUEEZE (top-k compression), Top-k SGD, MEM-SGD (top-k compres-sion), and Vanilla SGD, under different network environ-ments. The x-axis represents the inverse of the bandwidthof the parameter server. The y-axis is the number of secondsneeded to finish one iteration.

is also provided. It implies that DOUBLESQUEEZE admitsthe linear speedup corresponding to the number of workers,and has a better tolerance to the compression bias and noisethan those non-error-compensated approaches. Empiricalstudy is also conducted to validate the DOUBLESQUEEZEalgorithm.

AcknowledgementsThis project is in part supported by NSF CCF1718513, IBMfaculty award, and NEC fellowship.

DOUBLESQUEEZE

ReferencesAbadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,

J., Devin, M., Ghemawat, S., Irving, G., Isard, M., Kudlur,M., Levenberg, J., Monga, R., Moore, S., Murray, D. G.,Steiner, B., Tucker, P., Vasudevan, V., Warden, P., Wicke,M., Yu, Y., and Zheng, X. Tensorflow: A system forlarge-scale machine learning. In Proceedings of the 12thUSENIX Conference on Operating Systems Design andImplementation, OSDI’16, pp. 265–283, Berkeley, CA,USA, 2016a. USENIX Association. ISBN 978-1-931971-33-1.

Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean,J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.Tensorflow: a system for large-scale machine learning.In OSDI, volume 16, pp. 265–283, 2016b.

Agarwal, A. and Duchi, J. C. Distributed delayed stochasticoptimization. In Shawe-Taylor, J., Zemel, R. S., Bartlett,P. L., Pereira, F., and Weinberger, K. Q. (eds.), Advancesin Neural Information Processing Systems 24, pp. 873–881. Curran Associates, Inc., 2011.

Agarwal, N., Suresh, A. T., Yu, F. X. X., Kumar, S., andMcMahan, B. cpsgd: Communication-efficient anddifferentially-private distributed sgd. In Bengio, S., Wal-lach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N.,and Garnett, R. (eds.), Advances in Neural InformationProcessing Systems 31, pp. 7575–7586. Curran Asso-ciates, Inc., 2018.

Alistarh, D., Grubic, D., Li, J., Tomioka, R., and Vojnovic,M. QSGD: communication-efficient SGD via gradientquantization and encoding. In Guyon, I., von Luxburg,U., Bengio, S., Wallach, H. M., Fergus, R., Vishwanathan,S. V. N., and Garnett, R. (eds.), Advances in Neural In-formation Processing Systems 30: Annual Conferenceon Neural Information Processing Systems 2017, 4-9 De-cember 2017, Long Beach, CA, USA, pp. 1707–1718,2017.

Alistarh, D., Hoefler, T., Johansson, M., Konstantinov, N.,Khirirat, S., and Renggli, C. The convergence of spar-sified gradient methods. In Bengio, S., Wallach, H.,Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar-nett, R. (eds.), Advances in Neural Information Process-ing Systems 31, pp. 5977–5987. Curran Associates, Inc.,2018.

Bernstein, J., Wang, Y.-X., Azizzadenesheli, K., and Anand-kumar, A. signsgd: compressed optimisation for non-convex problems. In ICML, 2018a.

Bernstein, J., Zhao, J., Azizzadenesheli, K., and Anandku-mar, A. signsgd with majority vote is communicationefficient and byzantine fault tolerant. 10 2018b.

Bottou, L. and Bottou, L. Large-scale machine learningwith stochastic gradient descent. IN COMPSTAT, 2010.doi: 10.1.1.419.462.

Chen, C.-Y. AdaComp : Adaptive Residual Gradient Com-pression for Data-Parallel Distributed Training.; AAAI.2018.

Chen, T., Giannakis, G., Sun, T., and Yin, W. Lag: Lazily ag-gregated gradient for communication-efficient distributedlearning. In Bengio, S., Wallach, H., Larochelle, H.,Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),Advances in Neural Information Processing Systems 31,pp. 5055–5065. Curran Associates, Inc., 2018.

Cutkosky, A. and Busa-Fekete, R. Distributed stochasticoptimization via adaptive sgd. In Bengio, S., Wallach,H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., andGarnett, R. (eds.), Advances in Neural Information Pro-cessing Systems 31, pp. 1914–1923. Curran Associates,Inc., 2018.

Garber, D., Shamir, O., and Srebro, N. Communication-efficient algorithms for distributed stochastic principalcomponent analysis. In Precup, D. and Teh, Y. W. (eds.),Proceedings of the 34th International Conference on Ma-chine Learning, volume 70 of Proceedings of MachineLearning Research, pp. 1203–1212, International Conven-tion Centre, Sydney, Australia, 06–11 Aug 2017. PMLR.

He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn-ing for image recognition. In Proceedings of the IEEEconference on computer vision and pattern recognition,pp. 770–778, 2016.

He, L., Bian, A., and Jaggi, M. Cola: Decentralized linearlearning. In Bengio, S., Wallach, H., Larochelle, H.,Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),Advances in Neural Information Processing Systems 31,pp. 4541–4551. Curran Associates, Inc., 2018a.

He, L., Bian, A., and Jaggi, M. Cola: Decentralized linearlearning. In Advances in Neural Information ProcessingSystems, pp. 4541–4551, 2018b.

Hong, M., Hajinezhad, D., and Zhao, M.-M. Prox-PDA:The proximal primal-dual algorithm for fast distributednonconvex optimization and learning over networks. InPrecup, D. and Teh, Y. W. (eds.), Proceedings of the34th International Conference on Machine Learning, vol-ume 70 of Proceedings of Machine Learning Research,pp. 1529–1538, International Convention Centre, Sydney,Australia, 06–11 Aug 2017. PMLR.

Huo, Z., Gu, B., qian Yang, and Huang, H. Decoupledparallel backpropagation with convergence guarantee. InDy, J. and Krause, A. (eds.), Proceedings of the 35th In-ternational Conference on Machine Learning, volume 80

DOUBLESQUEEZE

of Proceedings of Machine Learning Research, pp. 2098–2106, Stockholmsmassan, Stockholm Sweden, 10–15 Jul2018. PMLR.

Jayaraman, B., Wang, L., Evans, D., and Gu, Q. Distributedlearning without distress: Privacy-preserving empiricalrisk minimization. In Bengio, S., Wallach, H., Larochelle,H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),Advances in Neural Information Processing Systems 31,pp. 6346–6357. Curran Associates, Inc., 2018.

Jiang, P. and Agrawal, G. A linear speedup analysis ofdistributed deep learning with sparse and quantized com-munication. In Bengio, S., Wallach, H., Larochelle, H.,Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),Advances in Neural Information Processing Systems 31,pp. 2530–2541. Curran Associates, Inc., 2018.

Jin, P. H., Yuan, Q., Iandola, F., and Keutzer, K. Howto scale distributed deep learning? arXiv preprintarXiv:1611.04581, 2016.

Li, M., Andersen, D. G., Park, J. W., Smola, A. J., Ahmed,A., Josifovski, V., Long, J., Shekita, E. J., and Su, B.-Y.Scaling distributed machine learning with the parameterserver. In OSDI, volume 14, pp. 583–598, 2014.

Li, Y., Yu, M., Li, S., Avestimehr, S., Kim, N. S., andSchwing, A. Pipe-sgd: A decentralized pipelined sgdframework for distributed deep net training. In Bengio, S.,Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi,N., and Garnett, R. (eds.), Advances in Neural Infor-mation Processing Systems 31, pp. 8056–8067. CurranAssociates, Inc., 2018.

Li, Z. and Yan, M. A primal-dual algorithm with optimalstepsizes and its application in decentralized consensusoptimization. arXiv preprint arXiv:1711.06785, 2017.

Lian, X., Huang, Y., Li, Y., and Liu, J. Asynchronousparallel stochastic gradient for nonconvex optimization.In Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama,M., and Garnett, R. (eds.), Advances in Neural Infor-mation Processing Systems 28, pp. 2737–2745. CurranAssociates, Inc., 2015.

Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W., andLiu, J. Can decentralized algorithms outperform central-ized algorithms? a case study for decentralized parallelstochastic gradient descent. In Advances in Neural Infor-mation Processing Systems, pp. 5330–5340, 2017.

Lian, X., Zhang, W., Zhang, C., and Liu, J. Asynchronousdecentralized parallel stochastic gradient descent. In In-ternational Conference on Machine Learning, 2018.

Nedic, A. and Olshevsky, A. Distributed optimization overtime-varying directed graphs. IEEE Transactions on Au-tomatic Control, 60(3):601–615, 2015.

Nedic, A., Olshevsky, A., and Shi, W. Achieving geomet-ric convergence for distributed optimization over time-varying graphs. SIAM Journal on Optimization, 27(4):2597–2633, 2017.

Recht, B., Re, C., Wright, S., and Niu, F. Hogwild: Alock-free approach to parallelizing stochastic gradientdescent. In Shawe-Taylor, J., Zemel, R. S., Bartlett, P. L.,Pereira, F., and Weinberger, K. Q. (eds.), Advances inNeural Information Processing Systems 24, pp. 693–701.Curran Associates, Inc., 2011.

Renggli, C., Alistarh, D., and Hoefler, T. Sparcml: High-performance sparse communication for machine learning.arXiv preprint arXiv:1802.08021, 2018.

Saparbayeva, B., Zhang, M., and Lin, L. Communication ef-ficient parallel algorithms for optimization on manifolds.In Bengio, S., Wallach, H., Larochelle, H., Grauman,K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advancesin Neural Information Processing Systems 31, pp. 3578–3588. Curran Associates, Inc., 2018.

Scaman, K., Bach, F., Bubeck, S., Massoulie, L., and Lee,Y. T. Optimal algorithms for non-smooth distributedoptimization in networks. In Bengio, S., Wallach, H.,Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Gar-nett, R. (eds.), Advances in Neural Information Process-ing Systems 31, pp. 2745–2754. Curran Associates, Inc.,2018.

Seide, F. and Agarwal, A. Cntk: Microsoft’s open-sourcedeep-learning toolkit. In Proceedings of the 22nd ACMSIGKDD International Conference on Knowledge Dis-covery and Data Mining, pp. 2135–2135. ACM, 2016.

Seide, F., Fu, H., Droppo, J., Li, G., and Yu, D. 1-bit stochas-tic gradient descent and application to data-parallel dis-tributed training of speech dnns. In Interspeech 2014,September 2014.

Shen, Z., Mokhtari, A., Zhou, T., Zhao, P., and Qian, H.Towards more efficient stochastic decentralized learning:Faster convergence and sparse communication. In Dy, J.and Krause, A. (eds.), Proceedings of the 35th Interna-tional Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pp. 4624–4633, Stockholmsmassan, Stockholm Sweden, 10–15 Jul2018. PMLR.

Shi, W., Ling, Q., Wu, G., and Yin, W. Extra: An exact first-order algorithm for decentralized consensus optimization.SIAM Journal on Optimization, 25(2):944–966, 2015.

Stich, S. U., Cordonnier, J.-B., and Jaggi, M. Sparsified sgdwith memory. In Bengio, S., Wallach, H., Larochelle, H.,Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),

DOUBLESQUEEZE

Advances in Neural Information Processing Systems 31,pp. 4452–4463. Curran Associates, Inc., 2018.

Suresh, A. T., Yu, F. X., Kumar, S., and McMahan, H. B.Distributed mean estimation with limited communication.In Precup, D. and Teh, Y. W. (eds.), Proceedings of the34th International Conference on Machine Learning, vol-ume 70 of Proceedings of Machine Learning Research,pp. 3329–3337, International Convention Centre, Sydney,Australia, 06–11 Aug 2017. PMLR.

Tang, H., Gan, S., Zhang, C., Zhang, T., and Liu, J. Com-munication compression for decentralized training. InBengio, S., Wallach, H., Larochelle, H., Grauman, K.,Cesa-Bianchi, N., and Garnett, R. (eds.), Advances inNeural Information Processing Systems 31, pp. 7663–7673. Curran Associates, Inc., 2018a.

Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. d2:Decentralized training over decentralized data. In Dy, J.and Krause, A. (eds.), Proceedings of the 35th Interna-tional Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pp. 4848–4856, Stockholmsmassan, Stockholm Sweden, 10–15 Jul2018b. PMLR.

Tang, H., Lian, X., Yan, M., Zhang, C., and Liu, J. D2:Decentralized training over decentralized data. arXivpreprint arXiv:1803.07068, 2018c.

Thakur, R., Rabenseifner, R., and Gropp, W. Optimizationof collective communication operations in mpich. TheInternational Journal of High Performance ComputingApplications, 19(1):49–66, 2005.

Wang, H., Sievert, S., Liu, S., Charles, Z., Papailiopoulos,D., and Wright, S. Atomo: Communication-efficientlearning via atomic sparsification. In Bengio, S., Wallach,H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., andGarnett, R. (eds.), Advances in Neural Information Pro-cessing Systems 31, pp. 9872–9883. Curran Associates,Inc., 2018.

Wang, J., Kolar, M., Srebro, N., and Zhang, T. Efficientdistributed learning with sparsity. In Precup, D. and Teh,Y. W. (eds.), Proceedings of the 34th International Con-ference on Machine Learning, volume 70 of Proceedingsof Machine Learning Research, pp. 3636–3645, Interna-tional Convention Centre, Sydney, Australia, 06–11 Aug2017. PMLR.

Wangni, J., Wang, J., Liu, J., and Zhang, T. Gradient spar-sification for communication-efficient distributed opti-mization. In Bengio, S., Wallach, H., Larochelle, H.,Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.),Advances in Neural Information Processing Systems 31,pp. 1306–1316. Curran Associates, Inc., 2018.

Wen, W., Xu, C., Yan, F., Wu, C., Wang, Y., Chen, Y.,and Li, H. Terngrad: Ternary gradients to reduce com-munication in distributed deep learning. In Guyon, I.,Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vish-wanathan, S., and Garnett, R. (eds.), Advances in NeuralInformation Processing Systems 30, pp. 1509–1519. Cur-ran Associates, Inc., 2017.

Wu, J., Huang, W., Huang, J., and Zhang, T. Error compen-sated quantized SGD and its applications to large-scaledistributed optimization. In Dy, J. and Krause, A. (eds.),Proceedings of the 35th International Conference on Ma-chine Learning, volume 80 of Proceedings of MachineLearning Research, pp. 5325–5333, Stockholmsmassan,Stockholm Sweden, 10–15 Jul 2018. PMLR.

Xu, Z., Taylor, G., Li, H., Figueiredo, M. A. T., Yuan,X., and Goldstein, T. Adaptive consensus ADMM fordistributed optimization. In Precup, D. and Teh, Y. W.(eds.), Proceedings of the 34th International Conferenceon Machine Learning, volume 70 of Proceedings of Ma-chine Learning Research, pp. 3841–3850, InternationalConvention Centre, Sydney, Australia, 06–11 Aug 2017.PMLR.

Zhang, H., Li, J., Kara, K., Alistarh, D., Liu, J., and Zhang,C. ZipML: Training linear models with end-to-end lowprecision, and a little bit of deep learning. In Precup, D.and Teh, Y. W. (eds.), Proceedings of the 34th Interna-tional Conference on Machine Learning, volume 70 ofProceedings of Machine Learning Research, pp. 4035–4043, International Convention Centre, Sydney, Australia,06–11 Aug 2017a. PMLR.

Zhang, W., Zhao, P., Zhu, W., Hoi, S. C. H., and Zhang, T.Projection-free distributed online learning in networks.In Precup, D. and Teh, Y. W. (eds.), Proceedings of the34th International Conference on Machine Learning, vol-ume 70 of Proceedings of Machine Learning Research,pp. 4054–4062, International Convention Centre, Sydney,Australia, 06–11 Aug 2017b. PMLR.

Zhang, X., Khalili, M. M., and Liu, M. Improving theprivacy and accuracy of ADMM-based distributed algo-rithms. In Dy, J. and Krause, A. (eds.), Proceedings ofthe 35th International Conference on Machine Learning,volume 80 of Proceedings of Machine Learning Research,pp. 5796–5805, Stockholmsmassan, Stockholm Sweden,10–15 Jul 2018. PMLR.

DOUBLESQUEEZE

SupplementaryA. Proof to Theorem 1Proof. As already proved, the updating rule of DOUBLESQUEEZE admits the formulation

xt+1 = xt − γ∇f(xt) + γξt − γΩt−1 + γΩt.

Moreover, since we have (from Assumption 1)

E[∇F

(xt; ξ

(i)t

)]=∇fi(xt),

E∥∥∥δ(l,j)t

∥∥∥2 ≤ε2,it can be easily verified that for all t, we have

Eξt =1

n

n∑i=1

(∇f(xt)− E

[∇F

(xt; ζ

(i)t

)])= 0,

E‖ξt‖2 =1

n2E

∥∥∥∥∥n∑i=1

(∇f(xt)−∇F

(xt; ζ

(i)t

))∥∥∥∥∥2

=1

n2

n∑i=1

E∥∥∥∇f(xt)−∇F

(xt; ζ

(i)t

)∥∥∥2 +1

n2

n∑i 6=i′

⟨∇f(xt)−∇F

(xt; ζ

(i)t

),∇f(xt)−∇F

(xt; ζ

(i′)t

)⟩,

=1

n2

n∑i=1

E∥∥∥∇f(xt)−∇F

(xt; ζ

(i)t

)∥∥∥2≤σ

2

n,

E‖Ωt‖2 =E

∥∥∥∥∥δt +1

n

n∑i=1

δ(i)t

∥∥∥∥∥2

≤2E ‖δt‖2 + 2E

∥∥∥∥∥ 1

n

n∑i=1

δ(i)t

∥∥∥∥∥2

≤ε2 +2

n

n∑i=1

E∥∥∥δ(i)t ∥∥∥2

≤2ε2.

Introducing the auxiliary sequence yt defined as

yt =xt − γΩt−1.

The updating rule of yt could be deducted by

yt+1 =xt+1 − γΩt

=xt − γ∇f(xt) + γξt − γΩt−1 + γΩt − γΩt

=xt − γΩt−1 − γ∇f(xt) + γξt

=yt − γ∇f(xt) + γξt.

Meanwhile, since f(x) is with L-Lipschitz gradients, then we have

E‖∇f(yt)−∇f(xt)‖ ≤ L2E‖yt − xt‖2 = L2γ2E‖Ωt−1‖2 ≤ 2L2γ2ε2, (8)

DOUBLESQUEEZE

and

Ef(yt+1)− Ef(yt)

≤E 〈yt+1 − yt,∇f(yt)〉+L

2E‖yt+1 − yt‖2

=− γE 〈∇f(xt),∇f(yt)〉+ γE 〈ξt,∇f(yt)〉+Lγ2

2E‖∇f(xt)− ξt‖2

=− γE 〈∇f(xt),∇f(yt)〉+Lγ2

2E‖∇f(xt)‖2 +

Lγ2

2E‖ξt‖2 (due to Eξt = 0)

≤− γE 〈∇f(xt),∇f(yt)〉+Lγ2

2E‖∇f(xt)‖2 +

Lγ2σ2

2n

=− γE ‖∇f(xt)‖2 − γE 〈∇f(xt),∇f(yt)−∇f(xt)〉+Lγ2

2E‖∇f(xt)‖2 +

Lγ2σ2

2n

≤− γE ‖∇f(xt)‖2 +(γ

2E ‖∇f(xt)‖2 + 2γE ‖∇f(yt)−∇f(xt)‖2

)+Lγ2

2E‖∇f(xt)‖2 +

Lγ2σ2

2n

≤(−γ

2+Lγ2

2

)E ‖∇f(xt)‖2 +

Lγ2σ2

2n+ 4L2ε2γ3. (from (8))

Summing up the inequality above from t = 0 to t = T − 1, we get

Ef(yT )− Ef(y0) ≤ −(γ

2− Lγ2

2

) T−1∑t=0

E ‖∇f(xt)‖2 +Lγ2σ2T

2n+ 4L2ε2γ3T,

which can be also written as(γ

2− Lγ2

2

) T−1∑t=0

E ‖∇f(xt)‖2 ≤Ef(y0)− Ef(yT ) +Lγ2σ2T

2n+ 4L2ε2γ3T

≤Ef(x0)− Ef(x∗) +Lγ2σ2T

2n+ 4L2ε2γ3T.

It completes the proof.

A.1. Proof to Corollary 2

Proof. Given the choice of γ = 1

4L+σ√

Tn +ε

23 T

13

, we have

1− γL ≤ 1

2. (9)

Also, from Theorem 1, we obtain(γ

2− Lγ2

2

) T−1∑t=0

E ‖∇f(xt)‖2 ≤Ef(x0)− Ef(x∗) +Lγ2σ2T

2n+ 4L2ε2γ3T,

followed by

(1− Lγ)1

T

T−1∑t=0

E ‖∇f(xt)‖2 ≤2 (Ef(x0)− Ef(x∗))

γT+Lγσ2

n+ 8L2ε2γ2. (10)

Combing (9) and (10) together we get

1

2T

T−1∑t=0

E ‖∇f(xt)‖2 ≤2 (Ef(x0)− Ef(x∗))

γT+Lγσ2

n+ 8L2ε2γ2.

DOUBLESQUEEZE

Replacing γ = 1

4L+σ√

Tn +ε

23 T

13

in the equation above we get

1

T

T−1∑t=0

E ‖∇f(xt)‖2 ≤4 (Ef(x0)− Ef(x∗))

T

(4L+ σ

√T

n+ ε

23T

13

)+Lγσ2

n+ 8L2ε2γ2

≤4 (Ef(x0)− Ef(x∗))

T

(4L+ σ

√T

n+ ε

23T

13

)+

Lσ√nT

+8L2ε

23

T23

≤ (4Ef(x0)− 4Ef(x∗) + 4L)σ√nT

+

(4Ef(x0)− 4Ef(x∗) + 8L2

)ε

23

T23

+4Ef(x0)− 4Ef(x∗)

T.

Taking f(x0)− f∗ and L as constants, the inequality above gives

1

T

T−1∑t=0

E‖∇f(xt)‖2 .σ√nT

+ε

23

T23

+1

T,

which completes the proof.