data poisoning against differentially-private learners

Data Poisoning against Differentially-Private Learners:Attacks and Defenses

Yuzhe Ma , Xiaojin Zhu and Justin HsuUniversity of Wisconsin-Madison

yzm234, jerryzhu, [email protected]

AbstractData poisoning attacks aim to manipulate the modelproduced by a learning algorithm by adversariallymodifying the training set. We consider differentialprivacy as a defensive measure against this type ofattack. We show that private learners are resistant todata poisoning attacks when the adversary is onlyable to poison a small number of items. However,this protection degrades as the adversary is allowedto poison more data. We emprically evaluate thisprotection by designing attack algorithms targetingobjective and output perturbation learners, two stan-dard approaches to differentially-private machinelearning. Experiments show that our methods areeffective when the attacker is allowed to poison suf-ficiently many training items.

1 IntroductionAs machine learning is increasingly used for consequential de-cisions in the real world, their security has received more andmore scrutiny. Most machine learning systems were originallydesigned without much thought to security, but researchershave since identified several kinds of attacks under the um-brella of adversarial machine learning. The most well-knownexample is adversarial examples [Szegedy et al., 2013], whereinputs are crafted to fool classifiers. More subtle methods aimto recover training examples [Fredrikson et al., 2015], or evenextract the model itself [Tramer et al., 2016].

In data poisoning attacks [Biggio et al., 2012; Koh et al.,2018; Mei and Zhu, 2015; Alfeld et al., 2016; Li et al., 2016;Xiao et al., 2015; Munoz-Gonzalez et al., 2017; Jun et al.,2018; Zhao et al., 2017; Zhang and Zhu, 2019], the adver-sary corrupts examples at training time to manipulate thelearned model. As is generally the case in adversarial ma-chine learning, the rise of data poisoning attacks has outpacedthe development of robust defenses. While attacks are oftenevaluated against specific target learning procedures, effec-tive defenses should provide protection—ideally guaranteed—against a broad class of attacks. In this paper, we study dif-ferential privacy [Dwork et al., 2006; Dwork, 2011] as asimple and general defensive technique against data poison-ing. While the original motivation of differential privacy wasoriginally capture notions of privacy—the output should not

depend too much on any single individual’s (private) data—italso provides a defense against data poisoning. Concretely,we establish quantitative bounds on how much an adversarycan change the distribution over learned models by manipu-lating a fixed number of training items. We are not the firstto design defenses to data poisoning [Steinhardt et al., 2017;Raghunathan et al., 2018], but our proposal has several no-table strengths compared to previous work. First, our defenseprovides provable protection against a worst-case adversary.Second, our defense is general. Differentially-private learningprocedures are known for a wide variety of tasks, and newalgorithms are continually being developed. These learnersare all resilient against data poisoning.

We complement our theoretical bounds with an empiri-cal evaluation of data poisoning attacks against differentiallyprivate learners. Concretely, we design an attack based onstochastic gradient descent to search for effective training ex-amples to corrupt against specific learning algorithms. Weevaluate our attack on two private learners: objective perturba-tion [Kifer et al., 2012] and output perturbation [Chaudhuri etal., 2011]. Our evaluation confirms the theoretical predictionthat private learners are vulnerable to data poisoning attackswhen the adversary can poison sufficiently many examples. Agap remains between the performance of attack and theoreti-cal limit of data poisoning attacks—it seems like differentialprivacy provides a substantially stronger defense in practicethan in theory. We discuss possible causes and leave furtherinvestigation to future work.

2 PreliminariesLet the data space be Z = X × Y , where X is the featurespace and Y is the label space. We write D = ∪∞n=0Zn forthe space of all training sets, and write D ∈ D for a particulartraining set. A randomized learnerM : D × Rd 7→ Θ maps atraining set D and noise b ∈ Rd drawn from a distribution νto a model in the model space Θ.

Definition 1. (Differential Privacy)M is (ε, δ)-differentially-private if ∀D, D ∈ D that differ by one item, and ∀S ⊂ Θ,

P (M(D, b) ∈ S) ≤ eεP(M(D, b) ∈ S

)+ δ, (1)

where the probability is taken over b ∼ ν. When δ = 0, wesay thatM is ε-differentially private.

Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence (IJCAI-19)

4732

Many standard machine learning tasks can be phrased asoptimization problems modeling empirical risk minimiza-tion (ERM). Broadly speaking, there are two families oftechniques in the privacy literature for solving these prob-lems. Objective perturbation injects noise into the objec-tive function of a vanilla machine learner to train a ran-domized model [Chaudhuri et al., 2011; Kifer et al., 2012;Nozari et al., 2016]. In contrast, output perturbation runs thevanilla learner but after training, it randomly perturbs the out-put model [Chaudhuri et al., 2011]. We will consider datapoisoning attacks against both kinds of learners. We first fixthe adversarial attack setting.

Knowledge of the attacker: The attacker has full knowledgeof the full training setD and the differentially-private machinelearnerM, including the noise distribution ν. However, theattacker does not know the realized noise b.

Ability of the attacker: The attacker is able to modify theclean training set D by changing the features or labels, addingadditional items, or deleting existing items, but the attackercan modify at most k items. Formally, the poisoned trainingset D must lie in the ball B(D, k) ⊂ D, where the center isthe clean data D and the radius k is the maximum number ofitem changes.

Goal of the attacker: The attacker aims to drive the (stochas-tic) model learned from the poisoned data θ = M(D, b) toachieve a certain goal. Formally, we define a cost functionC : Θ 7→ R that measures how much the poisoned model θdeviates from the attack target. Then the attack problem canbe formulated as minimizing an objective function J :

minD∈B(D,k)

J(D) := Eb

[C(M(D, b))

]. (2)

This formulation differs from previous works (e.g. [Biggio etal., 2012; Mei and Zhu, 2015; Xiao et al., 2015]) because thedifferentially-private learner produces a stochastic model, sothe objective takes the expected value of C.Example 1. (Parameter-Targeting Attack) If the attackerwants the output model to be close to a target model θ′, theattacker can define C(θ) = 1

2‖θ − θ′‖2. Minimizing J(D)

pushes the poisoned model close to θ′ in expectation.Example 2. (Label-Targeting Attack) If the attacker wantssmall prediction error on an evaluation set z∗i i∈[m], theattacker can define C(θ) = 1

m

∑mi=1 `(θ, z

∗i ) where `(θ, z∗i )

is the loss on item z∗i = (x∗i , y∗i ). Minimizing J(D) pushes

the prediction on x∗i towards the target label y∗i .Example 3. (Label-Aversion Attack) If the attacker wants toinduce large prediction error on an evaluation set z∗i i∈[m],the attacker can define C(θ) = − 1

m

∑mi=1 `(θ, z

∗i ), a non-

positive cost. Minimizing J(D) pushes the prediction on x∗iaway from the true label y∗i .

Now, we turn to our two main contributions: i) we showthat differentially-private learners have a degree of naturalimmunity against data poisoning attacks. Specifically, in sec-tion 3 we present a lower bound on how much the attackercan reduce J(D) by manipulating up to k items; and ii) as kincreases, the attacker’s power grows and it becomes easier

to attack differentially-private learners. In section 4 we pro-pose a stochastic gradient descent algorithm to solve the attackproblem (2). To our knowledge, this is the first data poisoningattack targeting specifically differentially-private learners.

3 Differential Privacy Resists Data PoisoningFixing the number of poisoned items k and a differentiallyprivate learner, can a powerful attacker achieve arbitrarily lowattack cost J(D)? We show that the answer is no: differen-tial privacy implies a lower bound on how far J(D) can bereduced, so private learners are inherently resistant to datapoisoning attacks. Previous work [Lecuyer et al., 2018] pro-posed defenses based on differential privacy, but for test-timeattacks. While effective, these defenses require the classifier tomake randomized predictions. In contrast, our defense againstdata poisoning only requires the learning algorithm, not theclassification process, to be randomized.

Our first result is for ε-differentially private learners: if anattacker can manipulate at most k items in the clean data, thenthe attack cost is lower bounded.Theorem 1. LetM be an ε-differentially-private learner. LetJ(D) be the attack cost, where D ∈ B(D, k), then

J(D) ≥ e−kεJ(D) (C ≥ 0),

J(D) ≥ ekεJ(D) (C ≤ 0).(3)

Note that for C ≥ 0, Theorem 1 shows that no attackercan achieve the trivial lower bound J(D) = 0. For C ≤ 0,although J(D) could be unbounded from below, Theorem 1shows that no attacker can reduce J(D) arbitrarily. Theseresults follow from the differential privacy property of M.Corollary 2 re-states the theorem in terms of the minimumnumber of items an attacker has to modify in order to suffi-ciently reduce the attack cost J(D).Corollary 2. Let M be an ε-differentially-private learner.Assume J(D) 6= 0. Let τ ≥ 1. Then the attacker has tomodify at least k ≥ d 1

ε log τe items in D in order to achieve

i). J(D) ≤ 1τ J(D) (C ≥ 0).

ii). J(D) ≤ τJ(D) (C ≤ 0).

To generalize Theorem 1 to (ε, δ)-private learners, we needan additional assumption that C is bounded.

Theorem 3. LetM be (ε, δ)-differentially-private and J(D)

be the attack cost, where |C| ≤ C and D ∈ B(D, k). Then,

J(D) ≥ maxe−kε(J(D) +Cδ

eε − 1)− Cδ

eε − 1, 0(C ≥ 0),

J(D) ≥ maxekε(J(D)− Cδ

eε − 1) +

Cδ

eε − 1,−C(C ≤ 0).

(4)As a corollary, we can lower-bound the minimum number

of modifications needed to reduce the attack cost to a certainlevel.Corollary 4. LetM be an (ε, δ)-differentially-private learner.Assume J(D) 6= 0. Then


4733

i). (0 ≤ C ≤ C): in order to achieve J(D) ≤ 1τ J(D) for

τ ≥ 1, the attacker has to modify at least the followingnumber of items in D.

k ≥ d1ε

log(eε − 1)J(D)τ + Cδτ

(eε − 1)J(D) + Cδτe. (5)

ii). (−C ≤ C ≤ 0): in order to achieve J(D) ≤ τJ(D) forτ ∈ [1,− C

J(D) ], the attacker has to modify at least thefollowing number of items in D.

k ≥ d1ε

log(eε − 1)J(D)τ − Cδ(eε − 1)J(D)− Cδ

e. (6)

Note that in (5), the lower bound on k is always finite evenif τ = ∞, which means an attacker might be able to reducethe attack cost J(D) to 0. This is in contrast to ε-differentially-private learner, where J(D) = 0 can never be achieved. Thus,the weaker (ε, δ)-differential privacy guarantee gives weakerprotection against attacks.

4 Data Poisoning Attacks on Private LearnersThe results in the previous section provide a theoretical boundon how effective a data poisoning attack can be against aprivate learner. To evaluate how strong the protection isin practice, we propose a range of attacks targeting generaldifferentially-private learners. Our adversary will modify (notadd or delete) any continuous features (for classification or re-gression) and the continuous labels (for regression) on at mostk items in the training set. Restating the attack problem (2):

minD⊂B(D,k)

J(D) = Eb

[C(M(D, b))

](7)

This is a combinatorial problem—the attacker must pick kitems to modify. We propose a two-step attack procedure.

step I) use heuristic methods to select k items to poison.

step II) apply stochastic gradient descent to reduce J(D).Step II) could lead to poisoned items taking arbitrary featuresor labels. However, differentially-private learners typicallyassume training examples are bounded. To avoid trivially-detectable poisoned examples, we project poisoned featuresand labels back to a bounded space after each iteration of SGD.We now detail step II), assuming that the attacker has fixed kitems to poison. We will return to step I) later.

4.1 SGD on Differentially-Private Victims (DPV)Our attacker uses stochastic gradient descent to minimize thecost J(D). We first show how to compute a stochastic gradientwith respect to a training item.Proposition 5. AssumeM(D, b) is a differentiable functionof D, then under certain conditions (details deferred to thefull paper), a stochastic gradient of J(D) with respect to aparticular item zi is ∂C(M(D,b))

∂zi, where b ∼ ν(b).

The concrete form of the stochastic gradient depends onthe target private learnerM, as well as the attack cost (whichencodes the type of attack). In the following, we derive thestochastic gradient for objective perturbation and output per-turbation in the context of parameter-targeting attack with costfunction C(θ) = 1

2‖θ − θ′‖2, where θ′ is the target model.

Instantiating Attack on Objective PerturbationWe first consider objective-perturbed private learners:

M(D, b) = argminθ∈Θ

n∑i=1

`(θ, zi) + λΩ(θ) + b>θ, (8)

where ` is some convex learning loss, Ω is a regularization,and Θ is the model space. Note that the noise b enters theobjective via linear product with the model θ. By Proposition 5,the stochastic gradient is ∂C(M(D,b))

∂zi= ∂C(θ)

∂zi, where θ =

M(D, b) is the poisoned model. By the chain rule, we have

∂C(θ)

∂zi=

(∂θ

∂zi

)>dC(θ)

dθ. (9)

Note that dC(θ)

dθ= θ − θ′. Next we focus on deriving ∂θ

∂zi.

Since (8) is convex, the learned model θ must satisfy thefollowing Karush-Kuhn-Tucker (KKT) condition:

f(θ, zi) :=n∑j=1

∂`(θ, zj)

∂θ+ λ

dΩ(θ)

dθ+ b = 0. (10)

By using the derivative of implicit function, we have ∂θ∂zi

=

−(∂f∂θ

)−1 ∂f∂zi

. We now give two examples where the baselearner is logistic regression and ridge regression, respectively.

In the case `(θ, z) = log(1 + exp(−yθ>x)) and Ω(θ) =12‖θ‖

2, learner (8) is objective-perturbed logistic regression:


n∑i=1

log(1+exp(−yiθ>xi))+λ

2‖θ‖2+b>θ,

(11)where Θ = Rd. Our attacker will only modify the features,thus yi = yi. We now derive stochastic gradient for xi. Definesj = exp(yj θ

>xj). By (9), one can verify that ∂C(θ)∂xi

=(yi

1 + siI − siθx

>i

(1 + si)2

)λI +n∑j=1

sj xj x>j

(1 + sj)2

−1

(θ−θ′).

Note that the noise b enters into stochastic gradient throughθ =M(D, b).

In the case `(θ, z) = 12 (y − x>θ)2 and Ω(θ) = 1

2‖θ‖2,

learner (8) is the objective-perturbed ridge regression:


1

2‖Xθ − y‖2 +

λ

2‖θ‖2 + b>θ, (12)

where Θ = θ ∈ Rd : ‖θ‖2 ≤ ρ, X ∈ Rn×d is the featurematrix, and y ∈ Rn is the label vector. Unlike logistic re-gression, the objective-perturbed ridge regression requires themodel space to be bounded (see e.g. [Kifer et al., 2012]). Theattacker can modify both the features and the labels. By (9),the stochastic gradient of xi is ∂C(θ)

∂xi=(

θx>i + (x>i θ − yi)I)(

X>X + (λ+ µ)I)−1

(θ′ − θ),


4734

where µ is the dual variable of constraint 12‖θ‖

2 ≤ ρ2

2 whensolving (12). The stochastic gradient of yi is

∂C(θ)

∂yi= x>i

(X>X + (λ+ µ)I

)−1

(θ − θ′).

Note that the noise b enters into the stochastic gradient throughθ =M(D, b).

Instantiating Attack on Output PerturbationWe now consider the output perturbation mechanism:

M(D, b) = b+ argminθ∈Θ

n∑i=1

`(θ, zi) + λΩ(θ)

. (13)

The stochastic gradients for this target are similar to those forobjective perturbation. Again, we instantiate on two exam-ples where the base learner is logistic regression and ridgeregression, respectively.

The output-perturbed logistic regression takes the followingform:

M(D, b) = b+argminθ∈Rd

n∑i=1

log(1+exp(−yiθ>xi))+λ

2‖θ‖2,

(14)Let sj = exp(yj(θ − b)>xj), then we have ∂C(θ)

∂xi=(

yi1 + si

I − si(θ − b)x>i(1 + si)2

)λI +n∑j=1

sj xj x>j

(1 + sj)2

−1

(θ−θ′).

The output-perturbed ridge regression takes the followingform

M(D, b) = b+ argminθ∈Θ

1

2‖Xθ − y‖2 +

λ

2‖θ‖2, (15)

where Θ = θ : ‖θ‖2 ≤ ρ. Let µ be the dual variable forthe constraint 1

2‖θ‖2 ≤ ρ2

2 when solving the argmin in (15).Then the stochastic gradients of xi and yi are

∂C(θ)

∂xi=(

(θ − b)x>i + x>i (θ − b)I − yiI)

·(X>X + (λ+ µ)I)

)−1

(θ′ − θ).

∂C(θ)

∂yi= x>i

(X>X + (λ+ µ)I

)−1

(θ − θ′).

4.2 SGD on Surrogate Victims (SV)We also consider an alternative, simpler way to perform stepII). When b = 0, the differentially-private learning algorithmsrevert to the base learner which returns a deterministic modelM(D,0). The adversary can simply attack the base learner(without b) as a surrogate for the differentially-private learnerM (with b) by solving the following problem:

minD

C(M(D,0)). (16)

Since the objective is deterministic, we can work with itsgradient rather than its stochastic gradient, plugging in b = 0into all derivations in section 4.1. Note that the attack foundby (16) will still be evaluated with respect to differentially-private learners in our experiments.

4.3 Selecting Items to PoisonNow, we return to step I) of our attack: how can we selectthe k items for poisoning? We give two heuristic methods:shallow and deep selection.

Shallow selection selects the k items with the largest ini-tial gradient norm ‖∂J(D)

∂zi|D=D ‖. Modifying these items

will reduce the attack cost the most, at least initially. Theprecise gradients to use during selection depend on the attackin step II). When targeting differentially-private learners di-rectly, computing the gradient of J(D) is difficult. We useMonte-Carlo sampling to approximate the gradient gi of itemzi: gi ≈ 1

m

∑ms=1

∂∂ziC(M(D, bs)) |D=D, where bs are ran-

dom samples of privacy parameters. Then, the attacker picksthe k items with the largest ‖gi‖ to poison. When targetingsurrogate victim, the objective is C(M(D,0)), thus the at-tacker computes the gradient of C(M(D,0)) of each cleanitem zi: gi = d

dziC(M(D,0)) |D=D, and then picks the k

items with the largest ‖gi‖.Deep selection selects items by estimating the influence of

an item on the final attack cost. When targeting a differentially-private victim directly, the attacker first solves the followingrelaxed attack problem: minD J(D) + αR(D). Importantly,here D allows changes to all training items, not just k ofthem. R(D) is a regularizer penalizing data modification withweight α > 0. We take R(D) =

∑ni=1 r(zi), where r(zi) is

the distance between the poisoned item zi and the clean itemzi. We define r(zi) = 1

2‖xi − xi‖2 for logistic regression and

r(zi) = 12‖xi−xi‖

2 + 12 (yi−yi)2 for ridge regression. After

solving the relaxed attack problem with SGD, the attackerevaluates the amount of change r(zi) for all training itemsand pick the top k to poison. When targeting a surrogatevictim, the attacker can solve the following relaxed attack first:minD C(M(D,0))+αR(D), and then select the k top items.

In summary, we have four attack algorithms based on com-binations of methods used in step I) and step II): shallow-SV,shallow-DPV, deep-SV and deep-DPV. In the following, weevaluate the performance of these four algorithms.

5 ExperimentsWe now evaluate our attack with experiments, taking objective-perturbed learners [Kifer et al., 2012] and output-perturbedlearners [Chaudhuri et al., 2011] as our victims. Throughout,we use α = 10−4 for deep selection and fix a constant step sizeη = 1 for (stochastic) gradient descent. After each iteration ofSGD, we project poisoned items to ensure feature norms areat most 1 and labels are in [−1, 1].

5.1 Attacks Intelligently Modify DataAs a first example, we use label-aversion attack to illustratethat our attack modifies data intelligently. The victim is anobjective-perturbed learner for ε-differentially private logisticregression, with ε = 0.1 and regularizer λ = 10. The trainingset contains n = 21 items uniformly sampled in the interval[−1, 1] with labels yi = 1 [xi ≥ 0], see Figure 1(a). To gen-erate the evaluation set, we construct m = 21 evenly-spaceditems in the interval [−1, 1], i.e., x∗i = 0.1i,−10 ≤ i ≤ 10


4735

and labeled as y∗i = 1 [x∗i ≥ 0]. The cost function is definedas C(θ) = − 1

m

∑mi=1 log(1 + exp(−y∗i x∗i

>θ)). To achievesmall attack cost J , a negative model θ < 0 is desired.1 Werun deep-DPV with k = n (attacker can modify all items) forT = 300 iterations. In Figure 1(a), we show the position ofthe poisoned items on the vertical axis as the SGD iteration, t,grows. As expected, the attacker flips the positions of positive(blue) items and negative (red) items. As a result, our attackcauses the learner to find a model with θ < 0.

5.2 Attacks Can Achieve Different GoalsWe now study a 2D example to show that our attack is ef-fective for different attack goals. The victim is the same asin section 5.1. The clean training set in Figure 1(b) containsn = 317 items uniformly sampled in a unit sphere with labelsyi = 1

[x>i θ

∗ ≥ 0]

where θ∗ = (1, 1). We now describe theattack settings for different attack goals.

For label-aversion attack, we generate an evaluation set con-tainingm = 317 grid points (x∗i , y

∗i ) lying in a unit sphere and

labeled by a vertical decision boundary, see Figure 1(c). Thecost function is C(θ) = − 1

m


>θ)).For label-targeting attack, the evaluation set remains the

same as in the label-aversion attack. The cost function isC(θ) = 1

m


>θ)).For parameter-targeting attack, we first train vanilla logistic

regression on the evaluation set to obtain the target modelθ′ = (2.6, 0), then define cost function as C(θ) = 1

2‖θ− θ′‖2.

(a) poisoning trajectory (b) training set (c) evaluation set

Figure 1: (a) 1D example (b, c) 2D example

We run deep-DPV with k = n for T = 5× 103 iterations.In Figure 2(a)-(c), we show the poisoning trajectories for thethree attack goals respectively. Each translucent point is anoriginal training point, and the connected solid point is its finalposition after attack. The curve connecting them shows thetrajectory as the attack algorithm gradually poisons the data.

In label-aversion attack the attacker aims to maximize thelogistic loss on the evaluation set. It ends up moving positive(negative) items to the left (right), so that the poisoned datadeviates from the evaluation set.

In contrast, the label-targeting attack tries to minimize thelogistic loss, thus items are moved to produce a poisoned dataaligned with the evaluation set. However, the attack does notproduce exactly the evaluation set. To understand that, wecompute the model learnt by vanilla logistic regression onthe evaluation set and the poisoned data, which are (2.60,0)and (2.94,0.01). Note that both models are aligned with the

1Differentially-private machine learning typically considers ho-mogeneous learners, thus the 1D model is just a scalar.

(a) label-aversion (b) label-targeting (c) parameter-targeting

(d) label-aversion (e) label-targeting (f) parameter-targeting

Figure 2: Poisoning trajectories in 2D.

evaluation set, but the latter has larger norm, which leads tosmaller attack cost, thus our result is a better attack.

In parameter-targeting attack, we compute the model learntby vanilla logistic regression on the clean data and poisoneddata, which are (1.86,1.85) and (2.21, 0.04). Note that theattack pushes the model closer to the target model (2.6,0).

To quantify the attack performance, we evaluate the attackcost J(D) via Monte-Carlo sampling. We run private learnersTe = 103 times, each time with a random bs to produce a costCs = C(M(D, bs)). Then we average to obtain an estimateof the cost J(D) ≈ 1

Te

∑Te

s=1 Cs. In Figure 2(d)-(f), we showthe attack cost J(D) and its 95% confidence interval (green,±2∗stderr) up to t = 103 iterations. J(D) decreases in allplots, showing that our attack is effective for all attack goals.

5.3 Attacks Become More Effective as k IncreasesThe theoretical protection provided by differential privacyweakens as the adversary is allowed to poison more train-ing items. We can show empirically that our attacks indeedbecome more effective as the number of poisoned items kincreases. The data set is the same as in section 5.2. Weconsider k from 20 to 100 in steps of 20. For each k, werun our four attack algorithms for T = 5 × 103 iterationsand estimate the attack cost J(D) on Te = 2× 103 samples.Figure 3(a)-(c) show that the attack cost J(D) decreases ask grows, indicating better attack performance. We also showthe poisoning trajectory produced by deep-DPV with k = 10in Figure 3(d)-(f). The corresponding attack costs J(D) are−0.60, 0.49 and 3.43 respectively. Compared to that of k = n,−1.79, 0.32 and 1.20, we see that poisoning only 10 itemsis not enough to reduce the attack cost J(D) significantly,although the attacker is having some effect.

5.4 Attacks Effective on Both Privacy MechanismsWe now show experiments on real data to illustrate that ourattacks are effective on both objective and output perturbationmechanisms. We focus on label-targeting attack.

The first real data set is vertebral column from the UCI Ma-chine Learning Repository [Dua and Karra Taniskidou, 2017].


4736

(a) label-aversion (b) label-targeting (c) parameter-targeting

(d) label-aversion (e) label-targeting (f) parameter-targeting

Figure 3: (a-c) the attack cost J(D) decreases as k grows. (d-f) thesparse attack trajectories for k = 10.

The data set contains 6 features of 310 orthopaedic patients,and the task is to predict if the vertebra of any patient is abnor-mal (positive class) or not (negative class). We normalize thefeatures so that the norm of any item is at most 1. Since thisis a classification task, we use private logistic regression withε = 0.1 and λ = 10 as the victim. To generate the evaluationset (x∗i , y∗i ), we randomly pick one positive item and find its10 nearest neighbors within the positive class, and set y∗i = −1for all 10. Intuitively, this attack targets a small cluster of ab-normal patients and the goal is to mislead the learner to classifythem as normal. The other experimental parameters remain thesame as in section 5.2. Note that in order to push the predictionon x∗i toward y∗i , the attacker requires y∗i x

∗i>θ > 0, which

indicates C(θ) = log(1 + exp(−y∗i x∗i>θ)) < log 2 ≈ 0.69,

so should J(D). As shown in Figure 4, the attacker indeedreduces the attack cost J(D) below 0.69 for both privacymechanisms, thus our attack is successful.

(a) objective perturbation (b) output perturbation

Figure 4: Attack on objective and output-perturbed logistic regres-sion.

The second data set is red wine quality from UCI. The dataset contains 11 features of 1598 wine samples, and the taskis predict the wine quality, a number between 0 and 10. Wenormalize the features so that the norm of any item is at most1. Labels are also normalized to ensure the value is in [−1, 1].The victim is private ridge regression with ε = 1, λ = 10,and bound on model space ρ = 2. To generate the evaluationset, we pick an item x∗i with the smallest quality value, and

set the target label to y∗i = 1. The cost function is defined asC(θ) = 1

2 (x∗i>θ − y∗i )2. The other experimental parameters

are the same as in section 5.2. This attack aims to force thelearner to predict a low-quality wine as having a high quality.Figure 5 shows the result. Note that the attack cost can besignificantly reduced for both privacy mechanisms even if theattacker poisons only 100/1598 ≈ 6.3% items of the data set.

(a) objective perturbation (b) output perturbation

Figure 5: Attack on objective and output-perturbed ridge regression.

We make two overall observations. First, deep-DPV is ingeneral the most effective method among our four attacks.Second, there remains a gap between the performance of deep-DPV and the theoretical bound, leaving space to design moreefficient attacks or to tighten the theoretical bound.

5.5 Attack Is Easier with Weaker PrivacyFinally, we show that the attack is more effective as the privacyguarantee becomes weaker, i.e., as we increase ε. To illustrate,we run experiments on the data set in section 5.2, where we fixk = 100 and vary ε from 0.1 to 0.5. In Figure 6, the attack costapproaches the lower bound as ε grows. This means weakerprivacy guarantees (smaller ε) lead to easier attacks.

Figure 6: Attacker can reduce cost J(D) more as ε grows.

6 Conclusion and Future WorkWe showed that differentially private learners are resistant todata poisoning attacks, with the protection degrading as theattacker poisons more items. We proposed attack algorithmsand demonstrated that our attacks are effective on different pri-vacy mechanisms and machine learners. There remains a gapbetween the theoretical limit and the empirical performanceof our attacks; closing the gap remains future work.

AcknowledgmentsWe thank Steve Wright for helpful discussions. This workis supported in part by NSF 1545481, 1561512, 1623605,1704117, 1836978, the MADLab AF Center of ExcellenceFA9550-18-1-0166, a Facebook TAV grant, and the Universityof Wisconsin.


4737

References[Alfeld et al., 2016] Scott Alfeld, Xiaojin Zhu, and Paul Bar-

ford. Data poisoning attacks against autoregressive models.In Thirtieth AAAI Conference on Artificial Intelligence,2016.

[Biggio et al., 2012] Battista Biggio, Blaine Nelson, andPavel Laskov. Poisoning attacks against support vectormachines. arXiv preprint arXiv:1206.6389, 2012.

[Chaudhuri et al., 2011] Kamalika Chaudhuri, Claire Mon-teleoni, and Anand D Sarwate. Differentially private em-pirical risk minimization. Journal of Machine LearningResearch, 12(Mar):1069–1109, 2011.

[Dua and Karra Taniskidou, 2017] Dheeru Dua and EfiKarra Taniskidou. UCI machine learning repository, 2017.

[Dwork et al., 2006] Cynthia Dwork, Frank McSherry, KobbiNissim, and Adam Smith. Calibrating noise to sensitivity inprivate data analysis. In Theory of cryptography conference,pages 265–284. Springer, 2006.

[Dwork, 2011] Cynthia Dwork. Differential privacy. Ency-clopedia of Cryptography and Security, pages 338–340,2011.

[Fredrikson et al., 2015] Matt Fredrikson, Somesh Jha, andThomas Ristenpart. Model inversion attacks that exploitconfidence information and basic countermeasures. In Pro-ceedings of the 22nd ACM SIGSAC Conference on Com-puter and Communications Security, pages 1322–1333.ACM, 2015.

[Jun et al., 2018] Kwang-Sung Jun, Lihong Li, Yuzhe Ma,and Jerry Zhu. Adversarial attacks on stochastic bandits. InAdvances in Neural Information Processing Systems, pages3640–3649, 2018.

[Kifer et al., 2012] Daniel Kifer, Adam Smith, andAbhradeep Thakurta. Private convex empirical risk mini-mization and high-dimensional regression. In Conferenceon Learning Theory, pages 25–1, 2012.

[Koh et al., 2018] Pang Wei Koh, Jacob Steinhardt, and PercyLiang. Stronger data poisoning attacks break data sanitiza-tion defenses. arXiv preprint arXiv:1811.00741, 2018.

[Lecuyer et al., 2018] Mathias Lecuyer, Vaggelis Atlidakis,Roxana Geambasu, Daniel Hsu, and Suman Jana. Certi-fied robustness to adversarial examples with differentialprivacy. In Certified Robustness to Adversarial Exampleswith Differential Privacy. IEEE, 2018.

[Li et al., 2016] Bo Li, Yining Wang, Aarti Singh, and Yev-geniy Vorobeychik. Data poisoning attacks on factorization-based collaborative filtering. In Advances in Neural In-formation Processing Systems (NIPS), pages 1885–1893,2016.

[Mei and Zhu, 2015] Shike Mei and Xiaojin Zhu. Using ma-chine teaching to identify optimal training-set attacks onmachine learners. In The 29th AAAI Conference on Artifi-cial Intelligence, 2015.

[Munoz-Gonzalez et al., 2017] Luis Munoz-Gonzalez, Bat-tista Biggio, Ambra Demontis, Andrea Paudice, Vasin

Wongrassamee, Emil C Lupu, and Fabio Roli. Towardspoisoning of deep learning algorithms with back-gradientoptimization. In Proceedings of the 10th ACM Workshopon Artificial Intelligence and Security, pages 27–38. ACM,2017.

[Nozari et al., 2016] Erfan Nozari, Pavankumar Tallapragada,and Jorge Cortes. Differentially private distributed convexoptimization via objective perturbation. In 2016 AmericanControl Conference (ACC), pages 2061–2066. IEEE, 2016.

[Raghunathan et al., 2018] Aditi Raghunathan, Jacob Stein-hardt, and Percy Liang. Certified defenses against adversar-ial examples. arXiv preprint arXiv:1801.09344, 2018.

[Steinhardt et al., 2017] Jacob Steinhardt, Pang Wei W Koh,and Percy S Liang. Certified defenses for data poisoningattacks. In Advances in Neural Information ProcessingSystems (NIPS), pages 3517–3529, 2017.

[Szegedy et al., 2013] Christian Szegedy, Wojciech Zaremba,Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow,and Rob Fergus. Intriguing properties of neural networks.arXiv preprint arXiv:1312.6199, 2013.

[Tramer et al., 2016] Florian Tramer, Fan Zhang, Ari Juels,Michael K Reiter, and Thomas Ristenpart. Stealing ma-chine learning models via prediction apis. In 25th USENIXSecurity Symposium, pages 601–618, 2016.

[Xiao et al., 2015] Huang Xiao, Battista Biggio, GavinBrown, Giorgio Fumera, Claudia Eckert, and Fabio Roli.Is feature selection secure against training data poisoning?In International Conference on Machine Learning, pages1689–1698, 2015.

[Zhang and Zhu, 2019] Xuezhou Zhang and XiaojinZhu. Online data poisoning attack. arXiv preprintarXiv:1903.01666, 2019.

[Zhao et al., 2017] Mengchen Zhao, Bo An, Wei Gao, andTeng Zhang. Efficient label contamination attacks againstblack-box learning models. In IJCAI, pages 3945–3951,2017.


4738

data poisoning against differentially-private learners

Documents