support vector machines under adversarial label...

Support Vector Machines under Adversarial Label Contamination

Huang Xiaoa, Battista Biggiob,∗, Blaine Nelsonb, Han Xiaoa, Claudia Eckerta, Fabio Rolib

aDepartment of Computer Science, Technical University of Munich, Boltzmannstr. 3, 85748, Garching, GermanybDepartment of Electrical and Electronic Engineering, University of Cagliari, Piazza d’Armi, 09123, Cagliari, Italy

Abstract

Machine learning algorithms are increasingly being applied in security-related tasks such as spam and malware detection, althoughtheir security properties against deliberate attacks have not yet been widely understood. Intelligent and adaptive attackers mayindeed exploit specific vulnerabilities exposed by machine learning techniques to violate system security. Being robust to adversarialdata manipulation is thus an important, additional requirement for machine learning algorithms to successfully operate in adversarialsettings. In this work, we evaluate the security of Support Vector Machines (SVMs) to well-crafted, adversarial label noise attacks.In particular, we consider an attacker that aims to maximize the SVM’s classification error by flipping a number of labels in thetraining data. We formalize a corresponding optimal attack strategy, and solve it by means of heuristic approaches to keep thecomputational complexity tractable. We report an extensive experimental analysis on the effectiveness of the considered attacksagainst linear and non-linear SVMs, both on synthetic and real-world datasets. We finally argue that our approach can also provideuseful insights for developing more secure SVM learning algorithms, and also novel techniques in a number of related researchareas, such as semi-supervised and active learning.

Keywords: Support Vector Machines, Adversarial Learning, Label Noise, Label Flip Attacks

1. Introduction

Machine learning and pattern recognition techniques are in-creasingly being adopted in security applications like spam, in-trusion and malware detection, despite their security to adver-sarial attacks has not yet been deeply understood. In adversar-ial settings, indeed, intelligent and adaptive attackers may care-fully target the machine learning components of a system tocompromise its security. Several distinct attack scenarios havebeen considered in a recent field of study, known as adversar-ial machine learning [1–4]. For instance, it has been shownthat it is possible to gradually poison a spam filter, an intrusiondetection system, and even a biometric verification system (ingeneral, a classification algorithm) by exploiting update mecha-nisms that enable the adversary to manipulate some of the train-ing data [5–13]; and that the detection of malicious samples bylinear and even some classes of non-linear classifiers can beevaded with few targeted manipulations that reflect a properchange in their feature values [13–17]. Recently, poisoning andevasion attacks against clustering algorithms have also been for-malized to show that malware clustering approaches can be sig-nificantly vulnerable to well-crafted attacks [18, 19].

Research in adversarial learning not only investigates thesecurity properties of learning algorithms against well-crafted

∗Corresponding authorEmail addresses: [email protected] (Huang Xiao),

[email protected] (Battista Biggio),[email protected] (Blaine Nelson), [email protected] (HanXiao), [email protected] (Claudia Eckert),[email protected] (Fabio Roli)

attacks, but it also focuses on the development of more se-cure learning algorithms. For evasion attacks, this has beenmainly achieved by explicitly embedding knowledge into thelearning algorithm of the possible data manipulation that canbe performed by the attacker, e.g., using game-theoretical mod-els for classification [15, 20–22], probabilistic models of thedata distribution drift under attack [23, 24], and even multipleclassifier systems [25–27]. Poisoning attacks and manipulationof the training data have been differently countered with datasanitization (i.e., a form of outlier detection) [5, 6, 28], mul-tiple classifier systems [29], and robust statistics [7]. Robuststatistics have also been exploited to formally show that the in-fluence function of SVM-like algorithms can be bounded undercertain conditions [30]; e.g., if the kernel is bounded. This en-sures some degree of robustness against small perturbations oftraining data, and it may be thus desirable also to improve thesecurity of learning algorithms against poisoning.

In this work, we investigate the vulnerability of SVMs to aspecific kind of training data manipulation, i.e., worst-case la-bel noise. This can be regarded as a carefully-crafted attack inwhich the labels of a subset of the training data are flipped tomaximize the SVM’s classification error. While stochastic la-bel noise has been widely studied in the machine learning liter-ature, to account for different kinds of potential labeling errorsin the training data [31, 32], only a few works have consideredadversarial, worst-case label noise, either from a more theoret-ical [33] or practical perspective [34, 35]. In [31, 33] the im-pact of stochastic and adversarial label noise on the classifica-tion error have been theoretically analyzed under the probably-approximately-correct learning model, deriving lower bounds

Preprint submitted to Neurocomputing July 26, 2014

on the classification error as a function of the fraction of flippedlabels η; in particular, the test error can be shown to be lowerbounded by η/(1−η) and 2η for stochastic and adversarial labelnoise, respectively. In recent work [34, 35], instead, we have fo-cused on deriving more practical attack strategies to maximizethe test error of an SVM given a maximum number of allowedlabel flips in the training data. Since finding the worst label flipsis generally computationally demanding, we have devised suit-able heuristics to find approximate solutions efficiently. To ourknowledge, these are the only works devoted to understandinghow SVMs can be affected by adversarial label noise.

From a more practical viewpoint, the problem is of interestas attackers may concretely have access and change some of thetraining labels in a number of cases. For instance, if feedbackfrom end-users is exploited to label data and update the system,as in collaborative spam filtering, an attacker may have accessto an authorized account (e.g., an email account protected bythe same anti-spam filter), and manipulate the labels assignedto her samples. In other cases, a system may even ask directlyto users to validate its decisions on some submitted samples,and use them to update the classifier (see, e.g., PDFRate,1 anonline tool for detecting PDF malware [36]). The practical rel-evance of poisoning attacks has also been recently discussed inthe context of the detection of malicious crowdsourcing web-sites that connect paying users with workers willing to carryout malicious campaigns (e.g., spam campaigns in social net-works) — a recent phenomenon referred to as crowdturfing. Infact, administrators of crowdturfing sites can intentionally pol-lute the training data used to learn classifiers, as it comes fromtheir websites, thus being able to launch poisoning attacks [37].

In this paper, we extend our work on adversarial label noiseagainst SVMs [34, 35] by improving our previously-defined at-tacks (Sects. 3.1 and 3.3), and by proposing two novel heuris-tic approaches. One has been inspired from previous work onSVM poisoning [12] and incremental learning [38, 39], andmakes use of a continuous relaxation of the label values togreedily maximize the SVM’s test error through gradient ascent(Sect. 3.2). The other exploits a breadth first search to greedilyconstruct sets of candidate label flips that are correlated in theireffect on the test error (Sect. 3.4). As in [34, 35], we aim atanalyzing the maximum performance degradation incurred byan SVM under adversarial label noise, to assess whether theseattacks can be considered a relevant threat. We thus assumethat the attacker has perfect knowledge of the attacked systemand of the training data, and left the investigation on how todevelop such attacks having limited knowledge of the trainingdata to future work. We further assume that the adversary in-curs the same cost for flipping each label, independently fromthe corresponding data point. We demonstrate the effectivenessof the proposed approaches by reporting experiments on syn-thetic and real-world datasets (Sect. 4). We conclude in Sect. 5with a discussion on the contributions of our work, its limita-tions, and future research, also related to the application of theproposed techniques to other fields, including semi-supervisedand active learning.

1Available at: http://pdfrate.com

2. Support Vector Machines and Notation

We revisit here structural risk minimization and SVM learn-ing, and introduce the framework that will be used to motivateour attack strategies for adversarial label noise.

In risk minimization, the goal is to find a hypothesis f :X → Y that represents an unknown relationship between an in-putX and an output spaceY, captured by a probability measureP. Given a non-negative loss function ` : Y×Y → R assessingthe error between the prediction y provided by f and the trueoutput y, we can define the optimal hypothesis f ? as the onethat minimizes the expected risk R ( f , P) = E(x,y)∼P

[` ( f (x), y)

]over the hypothesis space F , i.e., f ? = arg min f∈F R ( f , P).Although P is not usually known, and thus f ? can not be com-puted directly, a set Dtr = (xi, yi)ni=1 of i.i.d. samples drawnfrom P are often available. In these cases a learning algorithmL can be used to find a suitable hypothesis. According to struc-tural risk minimization [40], the learner L minimizes a sum ofa regularizer and the empirical risk over the data:

L(Dtr) = arg minf∈F

[Ω ( f ) + C · R ( f ,Dtr)

], (1)

where the regularizer Ω ( f ) is used to penalize excessive hy-pothesis complexity and avoid overfitting, the empirical riskR ( f ,Dtr) is given by 1

n∑n

i=1 ` ( f (xi), yi), and C > 0 is a parame-ter that controls the trade-off between minimizing the empiricalloss and the complexity of the hypothesis.

The SVM is an example of a binary linear classifier devel-oped according to the aforementioned principle. It makes pre-dictions inY = −1,+1 based on the sign of its real-valued dis-criminant function f (x) = w>x+b; i.e., x is classified as positiveif f (x) ≥ 0, and negative otherwise. The SVM uses the hingeloss ` ( f (x), y) = max (0, 1 − y f (x)) as a convex surrogate lossfunction, and a quadratic regularizer on w, i.e., Ω ( f ) = 1

2 w>w.Thus, SVM learning can be formulated according to Eq. (1) asthe following convex quadratic programming problem:

minw,b

12 w>w + C

∑ni=1 max (0, 1 − yi f (xi)) . (2)

An interesting property of SVMs arises from their dual for-mulation, which only requires computing inner products be-tween samples during training and classification, thus avoid-ing the need of an explicit feature representation. Accordingly,non-linear decision functions in the input space can be learnedusing kernels, i.e., inner products in implicitly-mapped featurespaces. In this case, the SVM’s decision function is given asf (x) =

∑ni=1 αiyik(x, xi) + b, where k(x, z) = φ(x)>φ(z) is the

kernel function, and φ the implicit mapping. The SVM’s dualparameters (α, b) are found by solving the dual problem:

min0≤α≤C

12α>Qα − 1>α , s.t. y>α = 0 , (3)

where Q = yy> K is the label-annotated version of the (train-ing) kernel matrix K. The bias b is obtained from the corre-sponding Karush-Kuhn-Tucker (KKT) conditions, to satisfy theequality constraint y>α = 0 (see, e.g., [41]).

2

In this paper, however, we are not only interested in how thehypothesis is chosen but also how it performs on a second vali-dation or test dataset Dvd, which may be generally drawn froma different distribution Q. We thus define the error measure

VL (Dtr,Dvd) = ‖ fDtr‖2 + C · R

(fDtr ,Dvd

), (4)

which implicitly uses fDtr = L(Dtr). This function evaluatesthe structural risk of a hypothesis fDtr that is trained onDtr butevaluated on Dvd, and will form the foundation for our labelflipping approaches to dataset poisoning. Moreover, since weare only concerned with label flips and their effect on the learnerwe use the notation VL (z, y) to denote the above error measurewhen the datasets differ only in the labels z used for training andy used for evaluation; i.e., VL (z, y) = VL ( (xi, zi) , (xi, yi) ).

3. Adversarial Label Flips on SVMs

In this paper we aim at gaining some insights on whetherand to what extent an SVM may be affected by the presenceof well-crafted mislabeled instances in the training data. Weassume the presence of an attacker whose goal is to cause a de-nial of service, i.e., to maximize the SVM’s classification error,by changing the labels of at most L samples in the training set.Similarly to [12, 35], the problem can be formalized as follows.

We assume there is some learning algorithm L known to theadversary that maps a training dataset into hypothesis space:L : (X × Y)n → F . Although this could be any learningalgorithm, we consider SVM here, as discussed above. Theadversary wants to maximize the classification error (i.e., therisk), that the learner is trying to minimize, by contaminatingthe training data so that the hypothesis is selected based ontainted data drawn from an adversarially selected distributionP over X × Y. However, the adversary’s capability of manipu-lating the training data is bounded by requiring P to be withina neighborhood of (i.e., “close to”) the original distribution P.

For a worst-case label flip attack, the attacker is restricted toonly change the labels of training samples inDtr and is allowedto change at most L such labels in order to maximally increasethe classification risk of L — L bounds the attacker’s capability,and it is fixed a priori. Thus, the problem can be formulated as

y′ = arg maxz∈Yn

R ( fz(x), P) , (5)

s.t. fz = L((xi, zi)ni=1

),∑n

i=1 I[zi , yi] ≤ L ,

where I[] is the indicator function, which returns one if theargument is true, and zero otherwise. However, as with thelearner, the true risk R cannot be assessed because P is alsounknown to the adversary. As with the learning paradigm de-scribed above, the risk used to select y′ can be approximated us-ing by the regularized empirical risk with a convex loss. Thusthe objective in Eq. (5) becomes simply VL (z, y) where, no-tably, the empirical risk is measured with respect to the truedataset Dtr with the original labels. For the SVM and hinge

loss, this yields the following program:

maxz∈−1,+1n

12α>Qα + C

∑ni=1 max(0, 1 − yi fz(xi)) , (6)

s.t. (α, b) = LS V M

((xi, zi)ni=1

),∑n

i=1 I[zi , yi] ≤ L ,

where fz(x) =∑n

j=1 z jα jK(x j, x) + b is the SVM’s dual discrim-inant function learned on the tainted data with labels z.

The above optimization is aNP-hard subset selection prob-lem, that includes SVM learning as a subproblem. In the nextsections we present a set of heuristic methods to find approxi-mate solutions to the posed problem efficiently; in particular, inSect. 3.1 we revise the approach proposed in [35] according tothe aforementioned framework, in Sect. 3.2 we present a novelapproach for adversarial label flips based on a continuous relax-ation of Problem (6), in Sect. 3.3 we present an improved, modi-fied version of the approach we originally proposed in [34], andin Sect. 3.4 we finally present another, novel approach for ad-versarial label flips that aims to flips clusters of labels that are‘correlated’ in their effect on the objective function.

3.1. Adversarial Label Flip Attack (alfa)We revise here the near-optimal label flip attack proposed

in [35], named Adversarial Label Flip Attack (alfa). It is for-mulated under the assumption that the attacker can maliciouslymanipulate the set of labels to maximize the empirical loss ofthe original classifier on the tainted dataset, while the classifica-tion algorithm preserves its generalization on the tainted datasetwithout noticing it. The consequence of this attack misleads theclassifier to an erroneous shift of the decision boundary, whichmost deviates from the untainted original data distribution.

As discussed above, given the untainted dataset Dtr withn labels y, the adversary aims to flip at most L labels to formthe tainted labels z that maximize VL (z, y). Alternatively, wecan pose this problem as a search for labels z that achieve themaximum difference between the empirical risk for classifierstrained on z and y, respectively. The attacker’s objective canthus be expressed as

minz∈−1,+1n

VL (z, z) − VL (y, z) , (7)

s.t.∑n

i=1 I[zi , yi] ≤ L .

To solve this problem, we note that the R ( f ,Dvd) compo-nent of VL is a sum of losses over the data points, and the eval-uation set Dvd only differs in its labels. Thus, for each datapoint, either we will have a component ` ( f (xi), yi) or a com-ponent ` ( f (xi),−yi) contributing to the risk. By denoting withan indicator variable qi which component to use, the attacker’sobjective can be rewritten as the problem of minimizing the fol-lowing expression with respect to q and f :

‖ f ‖2 + Cn∑

i=1

(1 − qi) · [` ( f (xi), yi) − `(

fy(xi), yi

)]

+qi · [` ( f (xi),−yi) − `(

fy(xi),−yi

)].

In this expression, the dataset is effectively duplicated and ei-ther (xi, yi) or (xi,−yi) are selected for the set Dvd. The qi

3

variables are used to select an optimal subset of labels yi to beflipped for optimizing f .

When alfa is applied to the SVM, we use the hinge lossand the primal SVM formulation from Eq. (2). We denote withξ0

i = max(0, 1−yi fy(xi)) and ξ1i = max(0, 1+yi fy(xi)) the ith loss

of classifier fy when the ith label is respectively kept unchangedor flipped. Similarly, ε0

i and ε1i are the corresponding slack vari-

ables for the new classifier f . The above attack framework canthen be expressed as:

minq,w,ε0,ε1,b

12‖w‖2 + C

n∑i=1

[(1 − qi) · (ε0

i − ξ0i )

+qi · (ε1i − ξ

1i )

], (8)

s.t. 1 − ε0i ≤ yi(w>xi + b) ≤ ε1

i − 1, ε0i , ε

1i ≥ 0 ,∑n

i=1 qi ≤ L, qi ∈ 0, 1 .

To avoid integer programming which is generallyNP-hard, theindicator variables, qi, are relaxed to be continuous on [0, 1].The minimization problem in Eq. (8) is then decomposed intotwo iterative sub-problems. First, by fixing q, the summandsξ0

i + qi(ξ0i − ξ

1i ) are constant, and thus the minimization reduces

to the following QP problem:

minw,ε0,ε1,b

12‖w‖2 + C

n∑i=1

[(1 − qi)ε0

i + qiε1i

], (9)

s.t. 1 − ε0i ≤ yi(w>xi + b) ≤ ε1

i − 1, ε0i , ε

1i ≥ 0 .

Second, fixing w and b yields a set of fixed hinge losses, ε0

and ε1. The minimization over (continuous) q is then a linearprogramming problem (LP):

minq∈[0,1]n

Cn∑

i=1

(1 − qi)(ε0i − ξ

0i ) + qi(ε1

i − ξ1i ) , (10)

s.t.∑n

i=1 qi ≤ L .

After convergence of this iterative approach, the largest subsetof q corresponds to the near-optimal label flips within the bud-get L. The complete alfa procedure is given as Algorithm 1.

3.2. ALFA with Continuous Label Relaxation (alfa-cr)

The underlying idea of the method presented in this sec-tion is to solve Problem (6) using a continuous relaxation of theproblem. In particular, we relax the constraint that the taintedlabels z ∈ −1,+1n have to be discrete, and let them take oncontinuous real values on a bounded domain. We thus maxi-mize the objective function in Problem (6) with respect to z ∈[zmin, zmax]n ⊆ Rn. Within this assumption, we optimize theobjective through a simple gradient-ascent algorithm, and iter-atively map the continuous labels to discrete values during thegradient ascent. The gradient derivation and the complete at-tack algorithm are respectively reported in Sects. 3.2.1 and 3.2.2.

3.2.1. Gradient ComputationLet us first compute the gradient of the objective in Eq. (6),

starting from the loss-dependent term∑

i max (0, 1 − yi fz(xi)).Although this term is not differentiable when y fz(x) = 1, it is

Algorithm 1: alfaInput : Untainted training setDtr = xi, yi

ni=1,

maximum number of label flips L.Output: Tainted training setD′tr.

1 Find fy by solving Eq. (2) onDtr ; /* QP */

2 foreach (xi, yi) ∈ Dtr do3 ξ0

i ← max(0, 1 − yi fy(xi));4 ξ1

i ← max(0, 1 + yi fy(xi));5 ε0

i ← 0, and ε1i ← 0;

6 repeat7 Find q by solving Eq. (10); /* LP */

8 Find ε0, ε1 by solving Eq. (9); /* QP */

9 until convergence;10 v←Sort([q1, . . . , qn], “descend”);

/* v are sorted indices from n + 1 */

11 for i← 1 to n do zi ← yi for i← 1 to L dozv[i] ← −yv[i]returnD′tr ← (xi, zi)ni=1;

possible to consider a subgradient that is equal to the gradientof 1−y fz(x), when y fz(x) < 1, and to 0 otherwise. The gradientof the loss-dependent term is thus given as:

∂

∂z

∑i

max (0, 1 − yi fz(xi))

= −

n∑i=1

δi∂vi

∂z, (11)

where δi = 1 (0) if yi fz(xi) < 1 (otherwise), and

vi = yi

n∑j=1

Ki jz j(z)α j(z) + b(z)

− 1 , (12)

where we explicitly account for the dependency on z. To com-pute the gradient of vi, we derive this expression with respect toeach label z` in the training data using the product rule:

∂vi

∂z`= yi

n∑j=1

Ki jz j∂α j

∂z`+ Ki`α` +

∂b∂z`

. (13)

This can be compactly rewritten in matrix form as:

∂v∂z

=(yz> K

) ∂α∂z

+ K (yα>) + y∂b∂z

, (14)

where, using the numerator layout convention,

∂α

∂z=

∂α1∂z1

· · ·∂α1∂zn

.... . .

...∂αn∂z1

· · ·∂αn∂zn

, ∂b∂z

>

=

∂b∂z1

· · ·∂b∂zn

, and simil.∂v∂z

.

The expressions for ∂α∂z and ∂b

∂z required to compute the gra-dient in Eq. (14) can be obtained by assuming that the SVM so-lution remains in equilibrium while z changes smoothly. Thiscan be expressed as an adiabatic update condition using thetechnique introduced in [38, 39], and exploited in [12] for a sim-ilar gradient computation. Observe that for the training sam-ples, the KKT conditions for the optimal solution of the SVM

4

training problem can be expressed as:

g = Qα + zb − 1

if gi > 0, i ∈ Rif gi = 0, i ∈ Sif gi < 0, i ∈ E

(15)

h = z>α = 0 , (16)

where we remind the reader that, in this case, Q = zz> K.The equality in condition (15)-(16) implies that an infinitesimalchange in z causes a smooth change in the optimal solution ofthe SVM, under the constraint that the sets R, S, and E do notchange. This allows us to predict the response of the SVMsolution to the variation of z as follows.

By differentiation of Eqs. (15)-(16), we obtain:

∂g∂z

= Q∂α

∂z+ K (zα>) + z

∂b∂z

+ S , (17)

∂h∂z

>

= z>∂α

∂z+ α> , (18)

where S = diag(K(z α) + b) is an n-by-n diagonal matrix,whose elements S i j = 0 if i , j, and S ii = fz(xi) elsewhere.

The assumption that the SVM solution does not change struc-ture while updating z implies that

∂gs

∂z= 0,

∂h∂z

= 0, (19)

where s indexes the margin support vectors in S (from theequality in condition 15). In the sequel, we will also use r, e,and n, respectively to index the reserve vectors in R, the errorvectors in E, and all the n training samples. The above assump-tion leads to the following linear problem, which allows us topredict how the SVM solution changes while z varies:[

Qss zs

z>s 0

] [∂αs∂z∂b∂z

]= −

[Ksn (zsα

>) + Ssn

α>

]. (20)

The first matrix can be inverted using matrix block inversion [42]:[Qss zs

z>s 0

]−1

=1ζ

[ζQ−1

ss − υυ> υ

υ> −1

], (21)

where υ = Q−1ss zs and ζ = z>s Q−1

ss zs. Substituting this result tosolve Problem (20), one obtains:

∂αs

∂z=

(1ζυυ> −Q−1

ss

) (Ksn (zsα

>) + Ssn

)−

1ζυα> , (22)

∂b∂z

= −1ζυ>

(Ksn (zsα

>) + Ssn

)+

1ζα> . (23)

The assumption that the structure of the three sets S,R,E ispreserved also implies that ∂αr

∂z = 0 and ∂αe∂z = 0. Therefore, the

first term in Eq. (14) can be simplified as:

∂v∂z

=(yz>s Kns

) ∂αs

∂z+ K (yα>) + y

∂b∂z

. (24)

Eqs. (22) and (23) can be now substituted into Eq. (24),and further into Eq. (11) to compute the gradient of the loss-dependent term of our objective function.

As for the regularization term, the gradient can be simplycomputed as:

∂

∂z

(12α>Qα

)= α [K(α z)] +

∂α

∂z

>

Qα . (25)

Thus, the complete gradient of the objective in Problem (6) is:

∇zVL = α [K(α z)] +∂α

∂z

>

Qα −Cn∑

i=1

δi∂vi

∂z. (26)

The structure of the SVM (i.e., the sets S,R,E) will clearlychange while updating z, hence after each gradient step weshould re-compute the optimal SVM solution along with its cor-responding structure. This can be done by re-training the SVMfrom scratch at each iteration. Alternatively, since our changesare smooth, the SVM solution can be more efficiently updatedat each iteration using an active-set optimization algorithm ini-tialized with the α values obtained from the previous iterationas a warm start [43]. Efficiency may be further improved bydeveloping an ad hoc incremental SVM under label perturba-tions based on the above equations. This however includes thedevelopment of suitable bookkeeping conditions, similarly to[38, 39], and it is thus left to future investigation.

3.2.2. AlgorithmOur attack algorithm for alfa-cr is given as Algorithm 2.

It exploits the gradient derivation reported in the previous sec-tion to maximize the objective function VL(z, y) with respectto continuous values of z ∈ [zmin, zmax]n. The current best setof continuous labels is iteratively mapped to the discrete set−1,+1n, adding a label flip at a time, until L flips are obtained.

3.3. ALFA based on Hyperplane Tilting (alfa-tilt)

We now propose a modified version of the adversarial labelflip attack we presented in [34]. The underlying idea of the orig-inal strategy is to generate different candidate sets of label flipsaccording to a given heuristic method (explained below), andretain the one that maximizes the test error, similarly to the ob-jective of Problem (6). However, instead of maximizing the testerror directly, here we consider a surrogate measure, inspiredby our work in [44]. In that work, we have shown that, underthe agnostic assumption that the data is uniformly distributedin feature space, the SVM’s robustness against label flips canbe related to the change in the angle between the hyperplane wobtained in the absence of attack, and that learnt on the tainteddata with label flips w′. Accordingly, the alfa-tilt strategyconsidered here, aims to maximize the following quantity:

maxz∈−1,+1n

〈w′,w〉||w′||||w||

=α′>Qzyα

√α′>Qzzα′

√α>Qyyα

, (27)

where Quv = K (uv>), being u and v any two sets of traininglabels, and α and α′ are the SVM’s dual coefficients learnt fromthe untainted and the tainted data, respectively.

Candidate label flips are generated as explained in [34]. La-bels are flipped with non-uniform probabilities, depending on

5

Algorithm 2: alfa-crInput : Untainted training setDtr = xi, yi

ni=1,

maximum num. of label flips L, maximum num.of iterations N (N ≥ L), gradient step size t.

Output: Tainted training setD′tr.1 z← y ; /* Initialize labels */

2 zbest ← y ;3 α, b ← learn SVM onDtr (Eq. 2) ;4 p← 0 ; /* Number of current flips */

5 k ← 0 ; /* Number of iterations */

6 while p < L do7 k ← k + 1 ;

/* Compute gradient from Eq. (11) using

current SVM solution */

8 z← z + t∇zVL ;9 Project z onto the feasible domain [zmin, zmax]n ;

10 α, b ← update SVM solution on xi, zini=1 ;

11 if VL(z, y) ≥ VL(zbest, y) then12 zbest ← z ; /* Best (soft) labels */

13 if mod (k, bN/Lc) = 0 then/* Project the best (soft) labels to

p (hard) label flips */

14 p← p + 1 ; /* Update flip count */

15 z← Flip the first p labels from y according to thedescending order of |zbest − y|;

16 returnD′tr ← (xi, zi)ni=1;

how well the corresponding training samples are classified bythe SVM learned on the untainted training set. We thus increasethe probability of flipping labels of reserve vectors (as they arereliably classified), and decrease the probability of label flipsfor margin and error vectors (inversely proportional to α). Theformer are indeed more likely to become margin or error vectorsin the SVM learnt on the tainted training set, and, therefore, theresulting hyperplane will be closer to them. This will in turn in-duce a significant change in the SVM solution, and, potentially,in its test error. We further flip labels of samples in differentclasses in a correlated way to force the hyperplane to rotate asmuch as possible. To this aim, we draw a random hyperplanewrnd, brnd in feature space, and further increase the probabil-ity of flipping the label of a positive sample x+ (respectively, anegative one x−), if w>rndx+ + brnd > 0 (w>rndx− + brnd < 0).

The full implementation of alfa-tilt is given as Algo-rithm 3. It depends on the parameters β1 and β2, which tune theprobability of flipping a point’s label based on how well it isclassified, and how well it is correlated with the other consid-ered flips. As suggested in [34], they can be set to 0.1, since thisconfiguration has given reasonable results on several datasets.

3.4. Correlated Clusters

Here, we explore a different approach to heuristically opti-mizing VL (z, y) that uses a breadth first search to greedily con-struct subsets (or clusters) of label flips that are ‘correlated’ intheir effect on VL. Here, we use the term correlation loosely.

Algorithm 3: alfa-tilt [34]Input : Untainted training setDtr = xi, yi

ni=1,

maximum num. of label flips L, maximum num.of trials N, and weighting parameters β1 and β2.

Output: Tainted training setD′tr.1 α, b ← learn SVM onDtr (Eq. 2) ;2 for i = 1, . . . , n do3 si ← yi[

∑nj=1 y jα jK(xi, x j) + b]

4 normalize sini=1 dividing by maxi=1,...,n si ;

5 (αrnd, brnd)← generate a random SVM (draw n + 1numbers from a uniform distribution) ;

6 for i = 1, . . . , n do7 qi ← yi[

∑nj=1 y jα

rndj K(xi, x j) + brnd]

8 normalize qini=1 dividing by maxi=1,...,n qi ;

9 for i = 1, . . . , n do10 vi ← αi/C − β1si − β2qi

11 (k1, . . . , kn)← sort (v1, . . . , vn) in ascending order, andreturn the corresponding indexes ;

12 z← y ;13 for i = 1, . . . , L do14 zki = −zki

15 train an SVM on xi, zini=1 ;

16 estimate the hyperplane tilt angle using Eq. (27) ;17 repeat N times from 5, selecting z to maximize Eq. (27) ;18 returnD′tr ← xi, zi

ni=1 ;

The algorithm starts by assessing how each singleton flipimpacts VL and proceeds by randomly sampling a set of P ini-tial singleton flips to serve as initial clusters. For each of theseclusters, k, we select a random set of mutations to it (i.e., a mu-tation is a change to a single flip in the cluster), which we thenevaluate (using the empirical 0-1 loss) to form a matrix ∆. Thismatrix is then used to select the best mutation to make amongthe set of evaluated mutations. Clusters are thus grown to max-imally increase the empirical risk.

To make the algorithm tractable, the population of candi-date clusters is kept small. Periodically, the set of clusters arepruned to keep the population to size M by discarding the worstevaluated clusters. Whenever a new cluster achieves the highestempirical error, that cluster is recorded as being the best candi-date cluster. Further, if clusters grow beyond the limit of L, thebest deleterious mutation is applied until the cluster only hasL flips. This overall process of greedily creating clusters withrespect to the best observed random mutations continues for aset number of iterations N at which point the best flips untilthat point are returned. Pseudocode for the correlated clustersalgorithm is given in Algorithm 4.

4. Experiments

We evaluate the adversarial effects of various attack strate-gies against SVMs on both synthetic and real-world datasets.Experiments on synthetic datasets provide a conceptual repre-

6

Algorithm 4: correlated-clustersInput : Untainted training setDtr = xi, yi

ni=1,

maximum number of label flips L, maximumnumber of iterations N (N ≥ L).

Output: Tainted training setD′tr.1 Let err(z) = R (L((xi, zi),Dtr);2 fy ← L(Dtr), Ey ← err(y);3 E? ← −∞, z? ← y;/* Choose random singleton clusters */

4 for i=1. . . M do5 j← rand(1, n);6 zi ← f lip(y, j);7 Ei ← err(zi) − Ey;8 if Ei > E? then E? ← Ei, z? ← zi for j=1. . . n do9 if rand[0,1] < L/n then ∆i, j ← err( f lip(zi, j))

else ∆i, j ← −∞

/* Grow new clusters by mutation */

10 for t=1 . . . N do11 (i, j)← arg max(i, j) ∆i, j ∆i, j ← −∞

zM+1 ← f lip(zi, j);12 if ‖zM+1 − y‖1 > 2L then13 Find best flip to reverse and flip it

14 EM+1 ← err(zM+1) − Ey;15 if EM+1 > E? then E? ← EM+1, z? ← zM+1 for

k=1. . . n do16 if rand[0,1] < L/n then

∆M+1,k ← err( f lip(zM+1, k)) else ∆M+1,k ← −∞

17 Delete worst cluster and its entries in E and ∆;

18 returnD′tr ← (xi, z?i )ni=1;

sentation of the rationale according to which the proposed at-tack strategies select the label flips. Their effectiveness, and thesecurity of SVMs against adversarial label flips, is then moresystematically assessed on different real-world datasets.

4.1. On Synthetic Datasets

To intuitively understand the fundamental strategies and dif-ferences of each of the proposed adversarial label flip attacks,we report here an experimental evaluation on two bi-dimensionaldatasets, where the positive and the negative samples can beperfectly separated by a linear and a parabolic decision bound-ary, respectively.2 For these experiments, we learn SVMs withthe linear and the RBF kernel on both datasets, using LibSVM [41].We set the regularization parameter C = 1, and the kernelparameter γ = 0.5, based on some preliminary experiments.For each dataset, we randomly select 200 training samples, andevaluate the test error on a disjoint set of 800 samples. The pro-posed attacks are used to flip L = 20 labels in the training data(i.e., a fraction of 10%), and the SVM model is subsequentlylearned on the tainted training set. Besides the four proposed

2Data is available at http://home.comcast.net/~tom.fawcett/

public_html/ML-gallery/pages/index.html.

attack strategies for adversarial label noise, further three attackstrategies are evaluated for comparison, respectively referred toas farfirst, nearest, and random. As for farfirst andnearest, only the labels of the L farthest and of the L nearestsamples to the decision boundary are respectively flipped. Asfor the random attack, L training labels are randomly flipped.To mitigate the effect of randomization, each random attack se-lects the best label flips over 10 repetitions.

Results are reported in Fig. 1. First, note how the proposedattack strategies alfa, alfa-cr, alfa-tilt, and correlatedcluster generally exhibit clearer patterns of flipped labels thanthose shown by farfirst, nearest, and random, yieldingindeed higher error rates. In particular, when the RBF ker-nel is used, the SVM’s performance is significantly affectedby a careful selection of training label flips (cf. the error ratesbetween the plots in the first and those in the second row ofFig. 1). This somehow contradicts the result in [30], wherethe use of bounded kernels has been advocated to improve ro-bustness of SVMs against training data perturbations. The rea-son is that, in this case, the attacker does not have the abilityto make unconstrained modifications to the feature values ofsome training samples, but can only flip a maximum of L la-bels. As a result, bounding the feature space through the useof bounded kernels to counter label flip attacks is not helpfulhere. Furthermore, the security of SVMs may be even wors-ened by using a non-linear kernel, as it may be easier to sig-nificantly change (e.g., “bend”) a non-linear decision boundaryusing carefully-crafted label flips, thus leading to higher errorrates. Amongst the attacks, correlated cluster shows thehighest error rates when the linear (RBF) kernel is applied to thelinearly-separable (parabolically-separable) data. In particular,when the RBF kernel is used on the parabolically-separabledata, even only 10% of label flips cause the test error to in-crease from 2.5% to 21.75%. Note also that alfa-cr andalfa-tilt outperform alfa on the linearly-separable dataset,but not on the parabolically-separable data. Finally, it is worthpointing out that applying the linear kernel on the parabolically-separable data, in this case, already leads to high error rates,making thus difficult for the label flip attacks to further increasethe test error (cf. the plots in the third row of Fig. 1).

4.2. On Real-World DatasetsWe report now a more systematic and quantitative assess-

ment of our label flip attacks against SVMs, considering fivereal-world datasets publicly available at the LibSVM website.3

In these experiments, we aim at evaluating how the performanceof SVMs decreases against an increasing fraction of adversar-ially flipped training labels, for each of the proposed attacks.This will indeed allow us to assess their effectiveness as wellas the security of SVMs against adversarial label noise. For afair comparison, we randomly selected 500 samples from eachdataset, and select the SVM parameters from C ∈ 2−7, 2−6, . . . , 210

and γ ∈ 2−7, 2−6, . . . , 25 by a 5-fold cross validation proce-dure. The characteristics of the datasets used, along with the

3http://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/

binary.html

7

Lin

ea

r ke

rne

l

2.25% 4.875% 8% 5.125% 10% 4.875% 4.125% 5.5%

Lin

ea

r ke

rne

l

25.5% 27.5% 26.875% 25% 27.5% 26.75% 25% 25.75%

RB

F k

ern

el

2.25% 10.375% 14.5% 16.375% 11.875% 2.375% 6.25% 5.5%

Original Data

RB

F k

ern

el

libsvm

2.5% 14.625%

alfa

13.125%

alfa−cr

8.625%

alfa−tilt

21.75%

correlated cluster farfirst

2.875% 6%

nearest

5.875%

random

C=1, γ=0.5

Figure 1: Results on synthetic datasets, for SVMs with the linear (first and third row) and the RBF (second and fourth row) kernel, trained with C = 1 and γ = 0.5.The original data distribution is shown in the first column (first and second row for the linearly-separable data, third and fourth row for the parabolically-separabledata). The decision boundaries of SVMs trained in the absence of label flips, and the corresponding test errors are shown in the second column. The remainingcolumns report the results for each of the seven considered attack strategies, highlighting the corresponding L = 20 label flips (out of 200 training samples).

Characteristics of the real-world datasetsName Feature set size SVM parameters

dna 124Linear kernel C = 0.0078

RBF kernel C = 1, γ = 0.0078

acoustic 51Linear kernel C = 0.016

RBF kernel C = 8, γ = 0.062

ijcnn1 23Linear kernel C = 4


seismic 51Linear kernel C = 4


splice 60Linear kernel C = 0.062


Table 1: Feature set sizes and SVM parameters for the real-world datasets.

optimal values of C and γ as discussed above, are reported inTable 1. We then evaluate our attacks on a separate test set of500 samples, using 5-fold cross validation. The correspondingaverage error rates are reported in Fig. 2, against an increasingfraction of label flips, for each considered attack strategy, andfor SVMs trained with the linear and the RBF kernel.

The reported results show how the classification performanceis degraded by the considered attacks, against an increasing per-centage of adversarial label flips. Among the considered at-tacks, correlated cluster shows an outstanding capabilityof subverting SVMs, although requiring significantly increasedcomputational time. In particular, this attack is able to induce atest error of almost 50% on the dna and seismic data, when theRBF kernel is used. Nevertheless, alfa and alfa-tilt can

achieve similar results on the acoustic and ijcnn1 data, whilebeing much more computationally efficient. In general, all theproposed attacks show a similar behavior for both linear andRBF kernels, and lead to higher error rates on most of the con-sidered real-world datasets than the trivial attack strategies farfirst,nearest, and random. For instance, when 20% of the la-bels are flipped, correlated cluster, alfa, alfa-cr, andalfa-tilt almost achieve an error rate of 50%, while farfirst,nearest and random hardly achieve an error rate of 30%. It isnevertheless worth remarking that farfirst performs rather wellagainst linear SVMs, while being not very effective when theRBF kernel is used. This reasonably means that non-linearSVMs may not be generally affected by label flips that are farfrom the decision boundary.

To summarize, our results demonstrate that SVMs can besignificantly affected by the presence of well-crafted, adversar-ial label flips in the training data, which can thus be considereda relevant and practical security threat in application domainswhere attackers can tamper with the training data.

5. Conclusions and Future Work

Although (stochastic) label noise has been well studied es-pecially in the machine learning literature (e.g., see [32] fora survey), to our knowledge few have investigated the robust-ness of learning algorithms against well-crafted, malicious labelnoise attacks. In this work, we have focused on the problem of

8

0 0.05 0.1 0.15 0.2−10

0

10

20

30

40

50

60dna

Flip rate on training set

Err

or

rate

on test

set (%

)

Linear kernel, c=0.0078

0 0.05 0.1 0.15 0.2

20

30

40

50

60acoustic


Err

or

rate

on test

set (%

)


0 0.05 0.1 0.15 0.2−10

0

10

20

30

40

50

60ijcnn1


Err

or

rate

on test

set (%

)

Linear kernel, c=4

0 0.05 0.1 0.15 0.20

10

20

30

40

50

60seismic


Err

or

rate

on test

set (%

)

Linear kernel, c=4

0 0.05 0.1 0.15 0.20

10

20

30

40

50

60splice


Err

or

rate

on test

set (%

)


0 0.05 0.1 0.15 0.2−10

0

10

20

30

40

50

60dna


Err

or

rate

on test

set (%

)

RBF kernel, c=1 g=0.0078

0 0.05 0.1 0.15 0.2

10

20

30

40

50

60acoustic


Err

or

rate

on test

set (%

)


0 0.05 0.1 0.15 0.2−10

0

10

20

30

40

50

60ijcnn1


Err

or

rate

on test

set (%

)


0 0.05 0.1 0.15 0.20

10

20

30

40

50

60seismic


Err

or

rate

on test

set (%

)


0 0.05 0.1 0.15 0.20

10

20

30

40

50

60splice


Err

or

rate

on test

set (%

)


libsvm alfa alfa−cr alfa−tilt farfirst nearest random correlated cluster

Figure 2: Results on real-world datasets (in different columns), for SVMs with the linear (first row) and the RBF (second row) kernel. Each plot shows the averageerror rate (± half standard deviation, for readability) for each attack strategy, estimated from 500 samples using 5-fold cross validation, against an increasing fractionof adversarially flipped labels. The values of C and γ used to learn the SVMs are also reported for completeness (cf. Table 1).

learning with label noise from an adversarial perspective, ex-tending our previous work on the same topic [34, 35]. In par-ticular, we have discussed a framework that encompasses dif-ferent label noise attack strategies, revised our two previously-proposed label flip attacks accordingly, and presented two novelattack strategies that can significantly worsen the SVM’s classi-fication performance on untainted test data, even if only a smallfraction of the training labels are manipulated by the attacker.

An interesting future extension of this work may be to con-sider adversarial label noise attacks in which the attacker haslimited knowledge of the system, e.g., when the feature set orthe training data are not completely known to the attacker, to seewhether the resulting error remains considerable also in morepractical attack scenarios. Another limitation that may be easilyovercome in the future is the assumption of equal cost for eachlabel flip. In general, indeed, a different cost can be incurreddepending on the feature values of the considered sample.

We nevertheless believe that this work provides an inter-esting starting point for future investigations on this topic, andmay serve as a foundation for designing and testing SVM-basedlearning algorithms to be more robust against a deliberate labelnoise injection. To this end, inspiration can be taken from previ-ous work on robust SVMs to stochastic label noise [34, 45]. Al-ternatively, one may exploit our framework to simulate a zero-sum game between the attacker and the classifier, that respec-tively aim to maximize and minimize the classification error onthe untainted test set. This essentially amounts to re-trainingthe classifier on the tainted data, having knowledge of which la-bels might have been flipped, to learn a more secure classifier.Different game formulations can also be exploited if the playersuse non-antagonistic objective functions, as in [22].

We finally argue that our work may also provide useful in-sights for developing novel techniques in machine learning ar-

eas which are not strictly related to adversarial learning, suchas semi-supervised and active learning. In the former case, byturning the maximization in Problems (5)-(6) into a minimiza-tion problem, one may find suitable label assignments z forthe unlabeled data, thus effectively designing a semi-supervisedlearning algorithm. Further, by exploiting the continuous la-bel relaxation of Sect. 3.2, one can naturally implement a fuzzyapproach, mitigating the influence of potentially outlying in-stances. As for active learning, minimizing the objective ofProblems (5)-(6) may help identifying the training labels whichmay have a higher impact on classifier training, i.e., some ofthe most informative ones to be queried. Finally, we conjecturethat our approach may also be applied in the area of structuredoutput prediction, in which semi-supervised and active learningcan help solving the inference problem of finding the best struc-tured output prediction approximately, when the computationalcomplexity of that problem is not otherwise tractable.

Acknowledgments

We would like to thank the anonymous reviewers for pro-viding useful insights on how to improve our work. This workhas been partly supported by the project “Security of patternrecognition systems in future internet” (CRP-18293) funded byRegione Autonoma della Sardegna, L.R. 7/2007, Bando 2009,and by the project “Automotive, Railway and Avionics Multi-core Systems - ARAMiS” funded by the Federal Ministry ofEducation and Research of Germany, 2012.

References

[1] M. Barreno, B. Nelson, R. Sears, A. D. Joseph, J. D. Tygar, Can machinelearning be secure?, in: ASIACCS ’06: Proc. of the ACM Symp. onInform., Computer and Comm. Sec., ACM, NY, USA, 2006, pp. 16–25.

9

[2] M. Barreno, B. Nelson, A. Joseph, J. Tygar, The security of machinelearning, Machine Learning 81 (2010) 121–148.

[3] L. Huang, A. D. Joseph, B. Nelson, B. Rubinstein, J. D. Tygar, Adver-sarial machine learning, in: 4th ACM Workshop on Artificial Intell. andSecurity, Chicago, IL, USA, 2011, pp. 43–57.

[4] B. Biggio, G. Fumera, F. Roli, Security evaluation of pattern classifiersunder attack, IEEE Trans. on Knowl. and Data Eng. 26 (2014) 984–996.

[5] B. Nelson, M. Barreno, F. J. Chi, A. D. Joseph, B. I. P. Rubinstein,U. Saini, C. Sutton, J. D. Tygar, K. Xia, Exploiting machine learningto subvert your spam filter, in: 1st Usenix Workshop on Large-Scale Ex-ploits and Emergent Threats, USENIX Assoc., CA, USA, 2008, pp. 1–9.

[6] B. Nelson, M. Barreno, F. Jack Chi, A. D. Joseph, B. I. P. Rubinstein,U. Saini, C. Sutton, J. D. Tygar, K. Xia, Misleading learners: Co-optingyour spam filter, in: Machine Learning in Cyber Trust, Springer US,2009, pp. 17–51.

[7] B. I. Rubinstein, B. Nelson, L. Huang, A. D. Joseph, S.-h. Lau, S. Rao,N. Taft, J. D. Tygar, Antidote: Understanding and defending against poi-soning of anomaly detectors, in: 9th ACM SIGCOMM Internet Measure-ment Conf., IMC ’09, ACM, New York, NY, USA, 2009, pp. 1–14.

[8] B. Biggio, G. Fumera, F. Roli, L. Didaci, Poisoning adaptive biometricsystems, in: G. Gimel’farb, E. Hancock, A. Imiya, A. Kuijper, M. Kudo,S. Omachi, T. Windeatt, K. Yamada (Eds.), Structural, Syntactic, and Sta-tistical Pattern Recognition, volume 7626 of LNCS, Springer Berlin Hei-delberg, 2012, pp. 417–425.

[9] B. Biggio, L. Didaci, G. Fumera, F. Roli, Poisoning attacks to compro-mise face templates, in: 6th IAPR Int’l Conf. on Biometrics, Madrid,Spain, 2013, pp. 1–7.

[10] M. Kloft, P. Laskov, Online anomaly detection under adversarial impact,in: 13th Int’l Conf. on Artificial Intell. and Statistics, 2010, pp. 405–412.

[11] M. Kloft, P. Laskov, Security analysis of online centroid anomaly detec-tion, J. Mach. Learn. Res. 13 (2012) 3647–3690.

[12] B. Biggio, B. Nelson, P. Laskov, Poisoning attacks against support vectormachines, in: J. Langford, J. Pineau (Eds.), 29th Int’l Conf. on MachineLearning, Omnipress, 2012.

[13] B. Biggio, I. Corona, B. Nelson, B. Rubinstein, D. Maiorca, G. Fumera,G. Giacinto, F. Roli, Security evaluation of support vector machines inadversarial environments, in: Y. Ma, G. Guo (Eds.), Support Vector Ma-chines Applications, Springer Int’l Publishing, 2014, pp. 105–153.

[14] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Srndic, P. Laskov, G. Gi-acinto, F. Roli, Evasion attacks against machine learning at test time,in: H. Blockeel, K. Kersting, S. Nijssen, F. Zelezny (Eds.), EuropeanConf. on Machine Learning and Principles and Practice of KnowledgeDiscovery in Databases (ECML PKDD), Part III, volume 8190 of LNCS,Springer Berlin Heidelberg, 2013, pp. 387–402.

[15] N. Dalvi, P. Domingos, Mausam, S. Sanghai, D. Verma, Adversarial clas-sification, in: 10th ACM SIGKDD Int’l Conf. on Knowledge Discoveryand Data Mining, Seattle, 2004, pp. 99–108.

[16] D. Lowd, C. Meek, Adversarial learning, in: A. Press (Ed.), Proc. of theEleventh ACM SIGKDD Int’l Conf. on Knowledge Discovery and DataMining (KDD), Chicago, IL., 2005, pp. 641–647.

[17] B. Nelson, B. I. Rubinstein, L. Huang, A. D. Joseph, S. J. Lee, S. Rao,J. D. Tygar, Query strategies for evading convex-inducing classifiers, J.Mach. Learn. Res. 13 (2012) 1293–1332.

[18] B. Biggio, I. Pillai, S. R. Bulo, D. Ariu, M. Pelillo, F. Roli, Is data cluster-ing in adversarial settings secure?, in: Proc. of the 2013 ACM Workshopon Artificial Intell. and Security, ACM, NY, USA, 2013, pp. 87–98.

[19] B. Biggio, S. R. Bulo, I. Pillai, M. Mura, E. Z. Mequanint, M. Pelillo,F. Roli, Poisoning complete-linkage hierarchical clustering, in: Struc-tural, Syntactic, and Statistical Pattern Recognition, 2014, In press.

[20] A. Globerson, S. T. Roweis, Nightmare at test time: robust learning byfeature deletion, in: W. W. Cohen, A. Moore (Eds.), Proc. of the 23rdInt’l Conf. on Machine Learning, volume 148, ACM, 2006, pp. 353–360.

[21] C. H. Teo, A. Globerson, S. Roweis, A. Smola, Convex learning withinvariances, in: J. Platt, D. Koller, Y. Singer, S. Roweis (Eds.), NIPS 20,MIT Press, Cambridge, MA, 2008, pp. 1489–1496.

[22] M. Bruckner, C. Kanzow, T. Scheffer, Static prediction games for adver-sarial learning problems, J. Mach. Learn. Res. 13 (2012) 2617–2654.

[23] B. Biggio, G. Fumera, F. Roli, Design of robust classifiers for adversarialenvironments, in: IEEE Int’l Conf. on Systems, Man, and Cybernetics,2011, pp. 977–982.

[24] R. N. Rodrigues, L. L. Ling, V. Govindaraju, Robustness of multimodal

biometric fusion methods against spoof attacks, J. Vis. Lang. Comput. 20(2009) 169–179.

[25] A. Kolcz, C. H. Teo, Feature weighting for improved classifier robustness,in: 6th Conf. on Email and Anti-Spam, Mountain View, CA, USA, 2009.

[26] B. Biggio, G. Fumera, F. Roli, Multiple classifier systems under attack,in: N. E. Gayar, J. Kittler, F. Roli (Eds.), 9th Int’l Workshop on MultipleClassifier Systems, volume 5997 of LNCS, Springer, 2010, pp. 74–83.

[27] B. Biggio, G. Fumera, F. Roli, Multiple classifier systems for robustclassifier design in adversarial environments, Int’l Journal of MachineLearning and Cybernetics 1 (2010) 27–41.

[28] G. F. Cretu, A. Stavrou, M. E. Locasto, S. J. Stolfo, A. D. Keromytis,Casting out demons: Sanitizing training data for anomaly sensors, in:IEEE Symposium on Security and Privacy, IEEE Computer Society, LosAlamitos, CA, USA, 2008, pp. 81–95.

[29] B. Biggio, I. Corona, G. Fumera, G. Giacinto, F. Roli, Bagging classifiersfor fighting poisoning attacks in adversarial environments, in: C. San-sone, J. Kittler, F. Roli (Eds.), 10th Int’l Workshop on Multiple ClassifierSystems, volume 6713 of LNCS, Springer-Verlag, 2011, pp. 350–359.

[30] A. Christmann, I. Steinwart, On robust properties of convex risk mini-mization methods for pattern recognition, J. Mach. Learn. Res. 5 (2004)1007–1034.

[31] M. Kearns, M. Li, Learning in the presence of malicious errors, SIAM J.Comput. 22 (1993) 807–837.

[32] B. Frenay, M. Verleysen, Classification in the presence of label noise: asurvey, IEEE Trans. on Neural Netw. and Learn. Systems PP (2013) 1–1.

[33] N. Bshouty, N. Eiron, E. Kushilevitz, Pac learning with nasty noise, in:O. Watanabe, T. Yokomori (Eds.), Algorithmic Learning Theory, volume1720 of LNCS, Springer Berlin Heidelberg, 1999, pp. 206–218.

[34] B. Biggio, B. Nelson, P. Laskov, Support vector machines under adversar-ial label noise, in: J. Mach. Learn. Res. - Proc. 3rd Asian Conf. MachineLearning, volume 20, 2011, pp. 97–112.

[35] H. Xiao, H. Xiao, C. Eckert, Adversarial label flips attack on supportvector machines, in: 20th European Conf. on Artificial Intelligence, 2012.

[36] C. Smutz, A. Stavrou, Malicious pdf detection using metadata and struc-tural features, in: Proc. of the 28th Annual Computer Security Applica-tions Conf., ACSAC ’12, ACM, New York, NY, USA, 2012, pp. 239–248.

[37] G. Wang, T. Wang, H. Zheng, B. Y. Zhao, Man vs. machine: Practi-cal adversarial detection of malicious crowdsourcing workers, in: 23rdUSENIX Security Symposium, USENIX Association, CA, 2014.

[38] G. Cauwenberghs, T. Poggio, Incremental and decremental support vectormachine learning, in: T. K. Leen, T. G. Dietterich, V. Tresp (Eds.), NIPS,MIT Press, 2000, pp. 409–415.

[39] C. P. Diehl, G. Cauwenberghs, SVM incremental learning, adaptation andoptimization, in: Int’l J. Conf. on Neural Networks, 2003, pp. 2685–2690.

[40] V. N. Vapnik, The nature of statistical learning theory, Springer-VerlagNew York, Inc., New York, NY, USA, 1995.

[41] C.-C. Chang, C.-J. Lin, LibSVM: a library for support vector machines,2001. URL: http://www.csie.ntu.edu.tw/~cjlin/libsvm/.

[42] H. Lutkepohl, Handbook of matrices, John Wiley & Sons, 1996.[43] K. Scheinberg, An efficient implementation of an active set method for

svms, J. Mach. Learn. Res. 7 (2006) 2237–2257.[44] B. Nelson, B. Biggio, P. Laskov, Understanding the risk factors of learn-

ing in adversarial environments, in: 4th ACM Workshop on ArtificialIntell. and Security, AISec ’11, Chicago, IL, USA, 2011, pp. 87–92.

[45] G. Stempfel, L. Ralaivola, Learning svms from sloppily labeled data, in:Proc. of the 19th Int’l Conf. on Artificial Neural Networks: Part I, ICANN’09, Springer-Verlag, Berlin, Heidelberg, 2009, pp. 884–893.

Huang Xiao got his B.Sc. degree in Com-puter Science from the Tongji Universityin Shanghai in China. After that, he got theM.Sc. degree in Computer Science fromthe Technical University of Munich (TUM)in Germany. His research interests includesemi-supervised and nonparametric learn-

ing, machine learning in anomaly detection, adversarial learn-ing, causal inference, Bayesian network, Copula theory. He isnow a 3rd year PhD candidate at the Dept. of Information Se-curity at TUM, supervised by Prof. Dr. Claudia Eckert.

10

Battista Biggio received the M. Sc. degreein Electronic Eng., with honors, and thePh. D. in Electronic Eng. and ComputerScience, respectively in 2006 and 2010, fromthe University of Cagliari, Italy. Since 2007he has been working for the Dept. of Elec-trical and Electronic Eng. of the same Uni-versity, where he is now a postdoctoral re-

searcher. In 2011, he visited the University of Tubingen, Ger-many, for six months, and worked on the security of machinelearning algorithms to training data contamination. His researchinterests currently include: secure / robust machine learningand pattern recognition methods, multiple classifier systems,kernel methods, biometric authentication, spam filtering, andcomputer security. He serves as a reviewer for several interna-tional conferences and journals, including Pattern Recognitionand Pattern Recognition Letters. Dr. Biggio is a member of theIEEE Computer Society and IEEE Systems, Man and Cyber-netics Society, and of the Italian Group of Italian Researchersin Pattern Recognition (GIRPR), affiliated to the InternationalAssociation for Pattern Recognition.

Blaine Nelson is currently a postdoctoralresearcher at the University of Cagliari, Italy.He previously was a postdoctoral researchfellow at the University of Potsdam andat the University of Tubingen, Germany,and completed his doctoral studies at theUniversity of California, Berkeley. Blainewas a co-chair of the 2012 and 2013 AISecworkshops on artificial intelligence and se-

curity and was a co-organizer of the Dagstuhl Workshop “Ma-chine Learning Methods for Computer Security” in 2012. Hisresearch focuses on learning algorithms particularly in the con-text of security-sensitive application domains. Dr. Nelson in-vestigates the vulnerability of learning to security threats andhow to mitigate them with resilient learning techniques.

Han Xiao is a Ph.D. candidate at Tech-nical University of Munich (TUM), Ger-many. He got the M.Sc. degree at TUM in2011. His advisors are Claudia Eckert andRen Brandenberg. His research interestsinclude online learning, semi-supervised learn-ing, active learning, Gaussian process, sup-port vector machines and probabilistic graph-ical models, as well as their applications in

knowledge discovery. He is supervised by Claudia Eckert andRene Brandenberg. From Sept. 2013 to Jan. 2014, he was a vis-iting scholar in Shou-De Lin’s Machine Discovery and SocialNetwork Mining Lab at National Taiwan University.

Claudia Eckert got her diploma in com-puter science from the University of Bonn.And she got the PhD in 1993 and in 1999she completed her habilitation at the TUMunich on the topic “Security in DistributedSystems”. Her research and teaching ac-tivities can be found in the fields of op-erating systems, middleware, communica-tion networks and information security. In

2008, she founded the center in Darmstadt CASED (Center forAdvanced Security Research Darmstadt), she was the deputydirector until 2010. She is a member of several scientific advi-sory boards, including the Board of the German Research Net-work (DFN), OFFIS, Bitkom and the scientific committee ofthe Einstein Foundation Berlin. She also advises governmentdepartments and the public sector at national and internationallevels in the development of research strategies and the imple-mentation of security concepts. Since 2013, she is the memberof the bavarian academy of science.

Fabio Roli received his M. Sc. degree,with honors, and Ph. D. degree in Elec-tronic Eng. from the University of Genoa,Italy. He was a member of the researchgroup on Image Processing and Understand-ing of the University of Genoa, Italy, from1988 to 1994. He was adjunct professor

at the University of Trento, Italy, in 1993 and 1994. In 1995,he joined the Dept. of Electrical and Electronic Eng. of theUniversity of Cagliari, Italy, where he is now professor of com-puter engineering and head of the research group on patternrecognition and applications. His research activity is focusedon the design of pattern recognition systems and their applica-tions to biometric personal identification, multimedia text cat-egorization, and computer security. On these topics, he haspublished more than two hundred papers at conferences and onjournals. He was a very active organizer of international con-ferences and workshops, and established the popular workshopseries on multiple classifier systems. Dr. Roli is a member ofthe governing boards of the International Association for Pat-tern Recognition and of the IEEE Systems, Man and Cyber-netics Society. He is Fellow of the IEEE, and Fellow of theInternational Association for Pattern Recognition.

11

support vector machines under adversarial label...

Documents