Exposing the Probabilistic Causal Structure ofDiscrimination

Francesco Bonchi1,2 Sara Hajian2 Bud Mishra3 Daniele Ramazzotti4

1Algorithmic Data Analytics Lab 2EURECAT 3New York University 4Milano-Bicocca UniversityISI Foundation, Turin, Italy Barcelona, Spain New York, USA Milan, Italy

francesco.bonchi@isi.it sara.hajian@eurecat.org mishra@nyu.edu daniele.ramazzotti@disco.unimib.it

ABSTRACTDiscrimination discovery from data is an important taskaiming at identifying patterns of illegal and unethicaldiscriminatory activities against protected-by-law groups,e.g., ethnic minorities. While any legally-valid proof of dis-crimination requires evidence of causality, the state-of-the-art methods are essentially correlation-based, albeit, as it iswell known, correlation does not imply causation.

In this paper we take a principled causal approach tothe data mining problem of discrimination detection indatabases. Following Suppes’ probabilistic causation theory,we define a method to extract, from a dataset of historicaldecision records, the causal structures existing among theattributes in the data. The result is a type of constrainedBayesian network, which we dub Suppes-Bayes Causal Net-work (SBCN). Next, we develop a toolkit of methods basedon random walks on top of the SBCN, addressing differ-ent anti-discrimination legal concepts, such as direct and in-direct discrimination, group and individual discrimination,genuine requirement, and favoritism. Our experiments onreal-world datasets confirm the inferential power of our ap-proach in all these different tasks.

1. INTRODUCTIONThe importance of discrimination discovery. At thebeginning of 2014, as an answer to the growing concernsabout the role played by data mining algorithms in decision-making, USA President Obama called for a 90-day reviewof big data collecting and analysing practices. The result-ing report1 concluded that “big data technologies can causesocietal harms beyond damages to privacy”. In particular, itexpressed concerns about the possibility that decisions in-formed by big data could have discriminatory effects, evenin the absence of discriminatory intent, further imposing lessfavorable treatment to already disadvantaged groups.

1http://www.whitehouse.gov/sites/default/files/docs/big_data_privacy_report_may_1_2014.pdf

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies arenot made or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwise, torepublish, to post on servers or to redistribute to lists, requires prior specificpermission and/or a fee.Copyright 20XX ACM X-XXXXX-XX-X/XX/XX ...$10.00.

Discrimination refers to an unjustified distinction of indi-viduals based on their membership, or perceived member-ship, in a certain group or category. Human rights lawsprohibit discrimination on several grounds, such as gender,age, marital status, sexual orientation, race, religion or be-lief, membership in a national minority, disability or illness.Anti-discrimination authorities (such as equality enforce-ment bodies, regulation boards, consumer advisory coun-cils) monitor, provide advice, and report on discriminationcompliances based on investigations and inquiries. A fun-damental role in this context is played by discriminationdiscovery in databases, i.e., the data mining problem of un-veiling discriminatory practices by analyzing a dataset ofhistorical decision records.

Discrimination is causal. According to current legisla-tion, discrimination occurs when a group is treated “lessfavorably” [1] than others, or when “a higher proportion ofpeople not in the group is able to comply” with a qualify-ing criterion [2]. Although these definitions do not directlyimply causation, as stated in [3] all discrimination claims re-quire plaintiffs to demonstrate a causal connection betweenthe challenged outcome and a protected status characteris-tic. In other words, in order to prove discrimination, author-ities must answer the counterfactual question: what wouldhave happened to a member of a specific group (e.g., non-white), if he or she had been part of another group (e.g.,white)?

“The Sneetches”, the popular satiric tale2 against discrim-ination published in 1961 by Dr. Seuss, describes a societyof yellow creatures divided in two races: the ones with agreen star on their bellies, and the ones without. The Star-Belly Sneetches have some privileges that are instead deniedto Plain-Belly Sneetches. There are, however, Star-On andStar-Off machines that can make a Plain-Belly into a Star-Belly, and viceversa. Thanks to these machines, the causalrelationship between race and privileges can be clearly mea-sured, because stars can be placed on or removed from anybelly, and multiple outcomes can be observed for an individ-ual. Therefore, we could readily answer the counterfactualquestion, saying with certainty what would have happenedto a Plain-Belly Sneetch had he or she been a Star-BellySneetch.

In the real world however, proving discrimination episodesis much harder, as we cannot manipulate race, gender, orsexual orientation of an individual. This highlights the needto assess discrimination as a causal inference problem [4]

2http://en.wikipedia.org/wiki/The_Sneetches_and_Other_Stories

arX

iv:1

510.

0055

2v3

[cs

.DB

] 8

Mar

201

7

from a database of past decisions, where causality can beinferred probabilistically. Unfortunately, the state of theart of data mining methods for discrimination discovery indatabases does not properly address the causal question, asit is mainly based on correlation-based methods (surveyedin Section 2).

Correlation is not causation. It is well known that cor-relation between two variables does not necessarily implythat one causes the other. Consider a unique cause X oftwo effects, Y and Z: if we do not take into account X, wemight derive wrong conclusions because of the observablecorrelation between Y and Z. In this situation, X is said toact as a confounding factor for the relationship between Yand Z.

Variants of the complex relation just discussed can ariseeven if, in the example, X is not the actual cause of eitherY or Z, but it is only correlated to them, for instance, be-cause of how the data were collected. Consider for instancea credit dataset where there exists high correlation betweena variable representing low income and another variable rep-resenting loan denial and let us assume that this is due toan actual legitimate causal relationship in the sense that, le-gitimately, a loan is denied if the applicant has low income.Let us now assume that we also observe high correlation be-tween low income and being female, which, for instance, canbe due to the fact that the women represented in the spe-cific dataset in analysis, tend to be underpaid. Given thesesettings, in the data we would also observe high correlationbetween the variable gender being female and the variablerepresenting loan denial, due to the fact that we do not ac-count for the presence of the variable low income. Followingcommon terminologies, we will say that such situations aredue to spurious correlations.

However, the picture is even more complicated: it couldbe the case, in fact, that being female is the actual causeof the low income and, hence, be the indirect cause of loandenial through low income. This would represent a causalrelationship between the gender and the loan denial, thatwe would like to detect as discrimination.

Disentangling these two different cases, i.e., female is onlycorrelated to low income in a spurious way, or being female isthe actual cause of low income, is at the same time importantand challenging. This highlights the need for a principledcausal approach to discrimination detection.

Another typical pitfall of correlation-based reasoning isexpressed by what is known as Simpson’s paradox3 accord-ing to which, correlations observed in different groups mightdisappear when these heterogeneous groups are aggregated,leading to false positives cases of discrimination discovery.One of the most famous false-positive examples due to Simp-son’s paradox occurred when in 1973 the University of Cali-fornia, Berkeley was sued for discrimination against womenwho had applied for admission to graduate schools. In fact,by looking at the admissions of 1973, it first appeared thatmen applying were significantly more likely to be admit-ted than women. But later, by examining the individualdepartments carefully, it was discovered that none of themwas significantly discriminating against women. On the con-trary, most departments had exercised a small bias in fa-vor of women. The apparent discrimination was due to thefact that women tended to apply to departments with lower

3http://en.wikipedia.org/wiki/Simpson’s_paradox

rates of admission, while men tended to apply to depart-ments with higher rates [5]. Later in Section 5.5 we willuse the dataset from this episode to highlight the differencesbetween correlation-based and causation-based methods.

Spurious correlations can also lead to false negatives (i.e.,discrimination existing but not being detected) as is com-monly seen in “reverse-discrimination”. The typical case iswhen authorities take affirmative actions, e.g., with compen-satory quota systems, in order to protect a minority groupfrom a potential discrimination. Such actions, while tryingto erase the supposed discrimination (i.e., the spurious corre-lation), fail to address the real underlying causes for discrim-ination, potentially ending up denying individual membersof a privileged group from access to their rightful shares ofsocial goods. In the early 70’s, a case involving the Univer-sity of California at Davis Medical School highlighted onesuch incident as the school’s admissions program reserved16 of the 100 places in its entering class for “disadvantaged”applicants, thus unintentionally reducing the chances of ad-mission for a qualified applicant.4

These are just few typical examples of the pitfalls ofcorrelation-based reasoning in the discovery of discrimina-tion. Later in Section 5.5 we show concrete examples fromreal-world datasets where correlation-based methods to dis-crimination discovery are not satisfactory.

Our proposal and contributions. In this paper we takea principled causal approach to the data mining problemof discrimination detection in databases. Following Suppes’probabilistic causation theory [6, 7] we define a method toextract, from a dataset of historical decision records, thecausal structures existing among the attributes in the data.

In particular, we define the Suppes-Bayes Causal Net-work (SBCN), i.e., a directed acyclic graph (dag) wherewe have a node representing a Bernulli variable of the type〈attribute = value〉 for each pair attribute-value present inthe database. In this dag an arc (A,B) represents the ex-istence of a causal relation between A and B (i.e., A causesB). Moreover, each arc is labeled with a score, representingthe strength of the causal relation.

Our SBCN is a constrained Bayesian network recon-structed by means of maximum likelihood estimation (MLE)from the given database, where we force the conditionalprobability distributions induced by the reconstructed graphto obey Suppes’ constraints: i.e., temporal priority and prob-ability rising. Imposing Suppes’ temporal priority and prob-ability raising we obtain what we call the prima facie causesgraph [7], which might still contain spurious causes (falsepositives). In order to remove these spurious case we add abias term to the likelihood score, favoring sparser causal net-works: in practice we sparsify the prima facie causes graphby extracting a minimal set of edges which best explain thedata. This regularization is done by means of the BayesianInformation Criterion (BIC) [8].

The obtained SBCN provides a clear summary, amenableto visualization, of the probabilistic causal structures foundin the data. Such structures can be used to reason aboutdifferent types of discrimination. In particular, we show howusing several random-walk-based methods, where the nextstep in the walk is chosen proportionally to the edge weights,we can address different anti-discrimination legal concepts.

4http://en.wikipedia.org/wiki/Regents_of_the_University_of_California_v._Bakke

Our experiments show that the measures of discriminationproduced by our methods are very strong, almost binary,signals: our measures are very clearly separating the dis-crimination and the non-discrimination cases.

To the best of our knowledge this is the first proposal ofdiscrimination detection in databases grounded in probabilis-tic causal theory.

Roadmap. The rest of the paper is organized as follows.Section 2 discusses the state of the art in discriminationdetection in databases. In Section 3 we formally introducethe SBCN and we present the method for extracting suchcausal network from the input dataset. Once extracted ourSBCN, in Section 4 we show how to exploit it for differentconcepts of discrimination detection, by means of random-walk methods. Finally Section 5 presents our experimentalassessment and comparison with correlation-based methodson two real-world datasets.

2. RELATED WORKDiscrimination analysis is a multi-disciplinary problem,

involving sociological causes, legal reasoning, economic mod-els, statistical techniques [9, 10]. Some authors [11, 12] studyhow to prevent data mining from becoming itself a source ofdiscrimination. In this paper instead we focus on the datamining problem of detecting discrimination in a dataset ofhistorical decision records, and in the rest of this section wepresent the most related literature.

Pedreschi et al. [13, 14, 15] propose a technique based onextracting classification rules (inductive part) and rankingthe rules according to some legally grounded measures ofdiscrimination (deductive part). The result is a (possiblylarge) set of classification rules, providing local and over-lapping niches of possible discrimination. This model onlydeals with group discrimination.

Luong et al. [16] exploit the idea of situation-testing [17]to detect individual discrimination. For each member of theprotected group with a negative decision outcome, testerswith similar characteristics (k-nearest neighbors) are con-sidered. If there are significantly different decision outcomesbetween the testers of the protected group and the testers ofthe unprotected group, the negative decision can be ascribedto discrimination.

Zliobaite et al. [18] focus on the concept of genuine re-quirement to detect that part of discrimination which maybe explained by other, legally grounded, attributes. In [19]Dwork et al. address the problem of fair classification thatachieves both group fairness, i.e., the proportion of mem-bers in a protected group receiving positive classification isidentical to the proportion in the population as a whole, andindividual fairness, i.e., similar individuals should be treatedsimilarly.

The above approaches assume that the dataset under anal-ysis contains attributes that denote protected groups (i.e.,direct discrimination). This may not be the case when suchattributes are not available, or not even collectable at amicro-data level as in the case of the loan applicant’s race.In these cases we talk about indirect discrimination discov-ery. Ruggieri et al. [20, 21] adopt a form of rule inferenceto cope with the indirect discovery of discrimination. Thecorrelation information is called background knowledge, andis itself coded as an association rule.

Mancuhan and Clifton [22] propose Bayesian networks

as a tool for discrimination discovery. Bayesian networksconsider the dependence between all the attributes and usethese dependencies in estimating the joint probability dis-tribution without any strong assumption, since a Bayesiannetwork graphically represents a factorization of the jointdistribution in terms of conditional probabilities encoded inthe edges. Although Bayesian networks are often used torepresent causal relationships, this needs not be the case,in fact a directed edge from two nodes of the network doesnot imply any causal relation between them. As an exam-ple, let us observe that the two graphs A → B → C andC → B → A impose exactly the same conditional inde-pendence requirements and, hence, any Bayesian networkwould not be able to disentangle the direction of any causalrelationship among these events.

Our work departs from this literature as:

1. it is grounded in probabilistic causal theory instead ofbeing based on correlation;

2. it proposes a holistic approach able to deal with differ-ent types of discrimination in a single unifying frame-work, while the methods in the state of the art usuallydeal with one and only one specific type of discrimina-tion;

3. it is the first work to adopt graph theory and socialnetwork analysis concepts, such as random-walk-basedcentrality measures and community detection, for dis-crimination detection;

Our proposal has also lower computational cost than meth-ods such as [13, 14, 15] which require to compute a poten-tially exponential number of association/classification rules.

3. SUPPES-BAYES CAUSAL NETWORKIn order to study discrimination as a causal inference prob-

lem, we exploit the criteria defined in the theories of prob-abilistic causation [6]. In particular, we follow [7], whereSuppes proposed the notion of prima facie causation that isat the core of probabilistic causation. Suppes’ definition isbased on two pillars: (i) any cause must happen before itseffect (temporal priority) and (ii) it must raise the probabil-ity of observing the effect (probability raising).

Definition 1 (Probabilistic causation [7]). Forany two events h and e, occurring respectively at times thand te, under the mild assumptions that 0 < P(h),P(e) < 1,the event h is called a prima facie cause of the event e if itoccurs before the effect and the cause raises the probabilityof the effect, i.e. th < te and P(e | h) > P(e | ¬h) .

In the rest of this section we introduce our method toconstruct, from a given relational table D, a type of causalBayesian network constrained to satisfy the conditions dic-tated by Suppes’ theory, which we dub Suppes-Bayes CausalNetwork (SBCN).

In the literature many algorithms exist to carry out struc-tural learning of general Bayesian networks and they usu-ally fall into two families [23]. The first family, constraintbased learning, explicitly tests for pairwise independence ofvariables conditioned on the power set of the rest of thevariables in the network. These algorithms exploit struc-tural conditions defined in various approaches to causality [6,24, 25]. The second family, score based learning, constructs

a network which maximizes the likelihood of the observeddata with some regularization constraints to avoid overfit-ting. Several hybrid approaches have also been recently pro-posed [26, 27, 28].

Our framework can be considered a hybrid approach ex-ploiting constrained maximum likelihood estimation (MLE)as follows: (i) we first define all the possible causal rela-tionship among the variables in D by considering only theoriented edges between events that are consistent with Sup-pes’ notion of probabilistic causation and, subsequently, (ii)we perform the reconstruction of the SBCN by a score-basedapproach (using BIC), which considers only the valid edges.

We next present in details the whole learning process.

3.1 Suppes’ constraintsWe start with an input relational table D defined over

a set A of h categorical attributes and s samples. In casecontinuous numerical attributes exists in D, we assume theyhave been discretized to become categorical. From D, wederive D′, an m× s binary matrix representing m Bernoullivariables of the type 〈attribute = value〉, where an entry is1 if we have an observation for the specific variable and 0otherwise.

Temporal priority. The first constraint, temporal priority,cannot be simply checked in the data as we have no timinginformation for the events. In particular, in our context theevents for which we want to reason about temporal priorityare the Bernoulli variables 〈attribute = value〉.

The idea here is that, e.g., income = low cannot be acause of gender = female, because the time when the gen-der of an individual is determined is antecedent to that ofwhen the income is determined. This intuition is imple-mented by simply letting the data analyst provide as inputto our framework a partial temporal order r : A→ N for theh attributes, which is then inherited from the m Bernoullivariables 5.

Based on the input dataset D and the partial order r weproduce the first graph G = (V,E) where we have a nodefor each of the Bernoulli variables, so |V | = m, and we havean arc (u, v) ∈ E whenever r(u) ≤ r(v). This way we willimmediately rule out causal relations that do not satisfy thetemporal priority constraint.

Probability raising. Given the graph G = (V,E) builtas described above the next step requires to prune thearcs which do not satisfy the second constraint, probabil-ity raising, thus building G′ = (V,E′), where E′ ⊆ E.In particular we remove from E each arc (u, v) such thatP(v | u) ≤ P(v | ¬u). The graph G′ so obtained is calledprima facie graph.

We recall that the probability raising condition is equiva-lent to constraining for positive statistical dependence [27]:in the prima facie graph we model all and only the posi-tive correlated relations among the nodes already partiallyordered by temporal priority, consistently with Suppes’ char-acterization of causality in terms of relevance.

5Note that our learning technique requires the input order rto be correct and complete in order to guarantee its conver-gence. Nevertheless, if this is not the case, it is still capableof providing valuable insights about the underlying causalmodel, although with the possibility of false positive or falsenegative causal claims.

3.2 Network simplificationSuppes’ conditions are necessary but not sufficient to eval-

uate causation [28]: especially when the sample size is small,the model may have false positives (spurious causes), evenafter constraining for Suppes’ temporal priority and prob-ability raising criteria (which aim at removing false nega-tives). Consequently, although we expect all the statisticallyrelevant causal relations to be modelled in G′, we also expectsome spurious ones in it.

In our proposal, in place of other structural conditionsused in various approaches to causality, (see e.g., [6, 24,25]), we perform a network simplification (i.e., we sparsifythe network by removing arcs) with a score based approach,specifically by relying on the Bayesian Information Criterion(BIC) as the regularized likelihood score [8].

We consider as inputs for this score the graph G′ and thedataset D′. Given these, we select the set of arcs E∗ ⊆ E′

that maximizes the score:

scoreBIC(D′, G′) = LL(D′|G′)− log s

2dim(G′).

In the equation, G′ denotes the graph, D′ denotes the data,s denotes the number of samples, and dim(G′) denotesthe number of parameters in G′. Thus, the regularizationterm −dim(G′) favors graphs with fewer arcs. The coeffi-cient log s/2 weighs the regularization term, such that thehigher the weight, the more sparsity will be favored over“ex-plaining” the data through maximum likelihood. Note thatthe likelihood is implicitly weighted by the number of datapoints, since each point contributes to the score.

Assume that there is one true (but unknown) probabil-ity distribution that generates the observed data, which is,eventually, uniformly randomly corrupted by false positivesand negatives rates (in [0, 1)). Let us call correct model,the statistical model which best approximate this distribu-tion. The use of BIC on G′ results in removing the falsepositives and, asymptotically (as the sample size increases),converges to the correct model. In particular, BIC is at-tempting to select the candidate model corresponding to thehighest Bayesian Posterior probability, which can be provedto be equivalent to the presented score and its log(s) penal-ization factor.

We denote with G∗ = (V,E∗) the graph that we obtain af-ter this step. We note that, as for general Bayesian network,G∗ is a dag by construction.

3.3 Confidence scoreUsing the reconstructed SBCN, we can represent the prob-

abilistic relationships between any set of events (nodes).As an example, suppose to consider the nodes represent-ing respectively income = low and gender = female be-ing the only two direct causes (i.e., with arcs toward) ofloan = denial. Given SBCN, we can estimate the condi-tional probabilities for each node in the graph, i.e., proba-bility of loan = denial given income = low AND gender =female in the example, by computing the conditional prob-ability of only the pair of nodes directly connected by an arc.For an overview of state-of-the-art methods for doing this,see [23]. However, we expect to be mostly dealing with fulldata, i.e., for every directly connected node in the SBCN,we expect to have several observations of any possible com-bination attribute = value. For this reason, we can simplyestimate the node probabilities by counting the observations

sex_Male

workclass_Self_emp_not_inc

0.06

marital_status_Married_civ_spouse

0.456

occupation_Craft_repair

0.157

occupation_Protective_serv

0.019

relationship_Husband

0.604

age_old

0.11

0.088

0.88

education_Bachelors

0.038

positive_dec

0.382

education_Assoc_voc

0.006

education_Masters

0.026

occupation_Exec_managerial

0.066

education_Prof_school

0.019

workclass_Self_emp_inc

0.043 education_Doctorate

0.012

0.069

0.39

0.371

workclass_Local_gov

0.035

0.085

0.267

0.018

0.011 0.052

0.015

native_country_Cuba

0.002

0.132

workclass_State_gov

0.071

0.028

occupation_Prof_specialty

0.151

0.153

0.208

0.066 0.036

workclass_Never_worked

0.001

0.272 0.0760.159 0.6560.509 0.1220.059 0.141 0.06

0.35

0.178

0.328

0.172

0.6110.492

native_country_India

0.213

0.129

Figure 1: One portion of the SBCN extracted from the Adult dataset. This subgraph corresponds to the C2

community reported later in Table 3 (Section 5) extracted by a community detection algorithm.

in the data. Moreover, we will exploit such conditional prob-abilities to define the confidence score of each arc in termsof their causal relationship.

In particular, for each arc (v, u) ∈ E∗ involving the causalrelationship between two nodes u, v ∈ V , we define a con-fidence score W (v, u) = P(u | v) − P(u | ¬v), which, intu-itively, aims at estimating the observations where the causev is followed by its effect u, that is P(u | v), and the oneswhere this is not observed, i.e., P(u | ¬v), because of imper-fect causal regularities. We also note that, by the constraintsdiscussed above, we require P(u | v) � P(u | ¬v) and, forthis reason, each weight is positive and no larger than 1, i.e.,W : E∗ → (0, 1].

Combining all of the concepts discussed above, we con-clude with the following definition.

Definition 2 (Suppes-Bayes Causal Network).Given an input dataset D′ of m Bernoulli variables ands samples, and given a partial order r of the variables,the Suppes-Bayes Causal Network SBCN = (V,E∗,W )subsumed by D′ is a weighted dag such that the followingrequirements hold:

• [Suppes’ constraints] for each arc (v, u) ∈ E∗ in-volving the causal relationship between nodes u, v ∈ V ,under the mild assumptions that 0 < P(u),P(v) < 1:

r(v) ≤ r(u) and P(u | v) > P(u | ¬v) .

• [Simplification] let E′ be the set of arcs satisfying theSuppes’ constraints as before; among all the subsets ofE′, the set of arcs E∗ is the one whose correspondinggraph maximizes BIC:

E∗ = arg maxE⊆E′,G=(V,E)

(LL(D′|G)− log s

2dim(G)) .

• [Score] W (v, u) = P(u | v)− P(u | ¬v), ∀(v, u) ∈ E∗

An example of a portion of a SBCN extracted from a real-world dataset is reported in Figure 1.

Algorithm 1 summarizes the learning approach adoptedfor the inference of the SBCN . Given D′ an input datasetover m Bernoulli variables and s samples, and r a partialorder of the variables, Suppes’ constraints are verified (Lines4-9) to construct a dag as described in Section 3.1.

The likelihood fit is performed by hill climbing (Lines 12-21), an iterative optimization technique that starts with anarbitrary solution to a problem (in our case an empty graph)and then attempts to find a better solution by incrementallyvisiting the neighbourhood of the current one. If the newcandidate solution is better than the previous one it is con-sidered in place of it. The procedure is repeated until thestopping criterion is matched.

The !StoppingCriterion occurs (Line 12) in two situa-tions: (i) the procedure stops when we have performed alarge enough number of iterations or, (ii) it stops when noneof the solutions in Gneighbors is better than the current Gfit.Note that Gneighbors denotes all the solutions that are deriv-able from Gfit by removing or adding at most one edge.

Time and space complexity. The computation of thevalid dag according to Suppes’ constraints (Lines 4-10) re-quires a pairwise calculation among the m Bernoulli vari-ables. After that, the likelihood fit by hill climbing (Lines11-21) is performed. Being an heuristic, the computationalcost of hill climbing depends on the sopping criterion. How-ever, constraining by Suppes’ criteria tends to regularize theproblem leading on average to a quick convergence to a goodsolution. The time complexity of Algorithm 1 is O(sm) andthe space required is O(m2), where m however is usually nottoo large, being the number of attribute-value pairs, and notthe number of examples.

Algorithm 1 Learning the SBCN

1: Inputs: D′ an input dataset of m Bernoulli variables ands samples, and r a partial order of the variables

2: Output: SBCN(V,E∗,W ) as in Definition 23: [Suppes’ constraints]4: for all pairs (v, u) among the m Bernoulli variables do5: if r(v) ≤ r(u) and P(u | v) > P(u | ¬v) then6: add the arc (v, u) to SBCN .7: end if8: end for9: [Likelihood fit by hill climbing]

10: Consider G(V,E∗,W )fit = ∅.11: while !StoppingCriterion() do12: Let G(V,E∗,W )neighbors be the neighbor solutions of

G(V,E∗,W )fit.13: Remove from G(V,E∗,W )neighbors any solution

whose arcs are not included in SBCN .14: Consider a random solution Gcurrent in

G(V,E∗,W )neighbors.15: if scoreBIC(D′, Gcurrent) > scoreBIC(D′, Gfit)

then16: Gfit = Gcurrent.17: ∀ arc (v, u) of Gfit, W (v, u) = P(u | v)−P(u | ¬v).18: end if19: end while20: SBCN = Gfit.21: return SBCN .

3.4 Expressivity of a SBCNWe conclude this Section with a discussion on the causal

relations that we model by a SBCN .Let us assume that there is one true (but unknown) prob-

ability distribution that generates the observed data whosestructure can be modelled by a dag. Furthermore, let usconsider the causal structure of such a dag and let us alsoassume each node with more then one cause to have conjunc-tive parents: any observation of the child node is precededby the occurrence of all its parents. As before we call cor-rect model, the statistical model which best approximate thedistribution. On these settings, we can prove the followingtheorem.

Theorem 1. Let the sample size s → ∞, the providedpartial temporal order r be correct and complete and thedata be uniformly randomly corrupted by false positives andnegatives rates (in [0, 1)), then the SBCN inferred from thedata is the correct model.

Proof. [Sketch] Let us first consider the case where theobserved data have no noise. On such an input, we observethat the prima facie graph has no false negatives: in fact∀[c → e] modelling a genuine causal relation, P(e ∧ c) =P(e), thus the probability raising constraint is satisfied, soit is the temporal priority given that we assumed r to becorrect and complete.

Furthermore, it is know that the likelihood fit performedby BIC converges to a class of structures equivalent in termsof likelihood among which there is the correct model: allthese topologies are the same unless the directionality ofsome edges. But, since we started with the prima facie graphwhich is already ordered by temporal priority, we can con-clude that in this case the SBCN coincides with the correctmodel.

To extend the proof to the case of data uniformly ran-domly corrupted by false positives and negatives rates (in[0, 1)), we note that the marginal and joint probabilitieschange monotonically as a consequence of the assumptionthat the noise is uniform. Thus, all inequalities used in thepreceding proof still hold, which concludes the proof.

In the more general case of causal topologies where anycause of a common effect is independent from any othercause (i.e., we relax the assumption of conjunctive parents),the SBCN is not guaranteed to converge to the correctmodel but it coincides with a subset of it modeling all theedges representing statistically relevant causal relations (i.e.,where the probability raising condition is verified).

4. DISCRIMINATION DISCOVERY BYRANDOM WALKS

In this section we propose several random-walk-basedmethods over the reconstructed SBCN, to deal with differentdiscrimination-detection tasks.

4.1 Group discrimination and favoritismThe basic problem in the analysis of direct discrimination

is precisely to quantify the degree of discrimination sufferedby a given protected group (e.g., an ethnic group) with re-spect to a decision (e.g., loan denial). In contrast to dis-crimination, favoritism refers to the case of an individualtreated better than others for reasons not related to indi-vidual merit or business necessity: for instance, favoritismin the workplace might result in a person being promotedfaster than others unfairly. In the following we denote fa-voritism as positive discrimination in contrast with negativediscrimination.

Given an SBCN we define a measure of group discrimina-tion (either negative or positive) for each node v ∈ V . Recallthat each node represents a pair 〈attribute = value〉, so itis essentially what we refer to as a group, e.g., 〈gender =female〉. Our task is to assign a score of discriminationds− : V → [0, 1] to each node, so that the closer ds−(v) isto 1 the more discriminated is the group represented by v.

We compute this score by means of a number n of randomwalks that start from v and reaches either the node repre-senting the positive decision or the one representing the neg-ative decision. In these random walks the next step is chosenproportionally to the weights of the out-going arcs. Supposea random walk has reached a node u, and let degout(u) de-note the set of outgoing arcs from u. Then the arc (u, z) ischosen with probability

p(u, z) =W (u, z)∑

e∈degout(u)W (e)

.

When a random walk ends in a node with no outgoing arcbefore reaching either the negative or the positive decision,it is restarted from the source node v.

Definition 3 (Group discrimination score).Given an SBCN = (V,E∗,W ), let δ− ∈ V and δ+ ∈ Vdenote the nodes indicating the negative and positivedecision, respectively. Given a node v ∈ V , and a numbern ∈ N of random walks to be performed, we denote asrwv→δ− the number of random walks started at node vthat reach δ− earlier than δ+. The discrimination score for

the group corresponding to node v is then defined as

ds−(v) =rwv→δ−

n.

This implicitly also defines a score of positive discrimination(or favoritism): ds+(v) = 1− ds−(v).

Taking advantage of the SBCN we also propose two ad-ditional measures capturing how far a node representing agroup is from the positive and negative decision respectively.This is done by computing the average number of steps thatthe random walks take to reach the two decisions: we denotethese scores as as−(v) and as+(v).

4.2 Indirect discriminationThe European Union Legislation [2] provides a broad def-

inition of indirect discrimination as occurring “where an ap-parently neutral provision, criterion or practice would putpersons of a racial or ethnic origin at a particular disadvan-tage compared with other persons”. In other words, the ac-tual result of the apparently neutral provision is the same asan explicitly discriminatory one. A typical legal case studyof indirect discrimination is concerned with redlining : e.g.,denying a loan because of ZIP code, which in some areas isan attribute highly correlated to race. Therefore, even if theattribute race cannot be required at loan-application time(thus would not be present in the data), still race discrimina-tion is perpetrated. Indirect discrimination discovery refersto the data mining task of discovering the attributes valuesthat can act as a proxy to the protected groups and lead todiscriminatory decisions indirectly [13, 14, 11].

In our setting, indirect discrimination can be detected byapplying the same method described in Section 4.1.

4.3 Genuine requirementThe legal concept of genuine requirement refers to detect-

ing that part of the discrimination which may be explainedby other, legally-grounded, attributes; e.g., denying creditto women may be explainable by the fact that most of themhave low salary or delay in returning previous credits. Atypical example in the literature is the one of the “genuineoccupational requirement”, also called “business necessity”in [29, 30]. In the state of the art of data mining methodsfor discrimination discovery, it is also known as explainablediscrimination [31] and conditional discrimination [18].

The task here is to evaluate to which extent the dis-crimination apparent for a group is “explainable” on a legalground. Let v ∈ V be the node representing the group whichis suspected of being discriminated, and ul ∈ V be a nodewhose causal relation with a negative or positive decision islegally grounded. As before, δ− and δ+ denote the nega-tive and positive decision, respectively. Following the samerandom-walk process described in Section 4.1, we define thefraction of explainable discrimination for the group v:

fed−(v) =rwv→ul→δ−

rwv→δ−,

i.e., the fraction of random walks passing trough ul amongthe ones started in v and reaching δ− earlier than δ+. Simi-larly we define fed+(v), i.e., the fraction of explainable pos-itive discrimination.

4.4 Individual and subgroup discriminationIndividual discrimination requires to measure the amount

of discrimination for a specific individual, i.e., an entirerecord in the database. Similarly, subgroup discriminationrefers to discrimination against a subgroup described bya combination of multiple protected and non-protected at-tributes: personal data, demographics, social, economic andcultural indicators, etc. For example, consider the case ofgender discrimination in credit approval: although an ana-lyst may observe that no discrimination occurs in general, itmay turn out that older women obtain car loans only rarely.

Both problems can be handled by generalizing the tech-nique introduced in Section 4.1 to deal with a set of startingnodes, instead of only one. Given an SBCN = (V,E∗,W )let v1, . . . , vn be the nodes of interest. In order to define adiscrimination score for v1, . . . , vn, we perform a personal-ized PageRank [32] computation with respect to v1, . . . , vn.In personalized PageRank, the probability of jumping to anode when abandoning the random walk is not uniform, butit is given by a vector of probabilities for each node. In ourcase the vector will have the value 1

nfor each of the nodes

v1, ..., vn ∈ V and zero for all the others. The output ofpersonalized PageRank is a score ppr(u|v1, ..., vn) of prox-imity/relevance to {v1, ..., vn} for each other node u in thenetwork. In particular, we are interested in the score of thenodes representing the negative and positive decision: i.e.,ppr(δ−|v1, ..., vn) and ppr(δ+|v1, ..., vn) respectively.

Definition 4 (Generalized discrimination score).Given an SBCN = (V,E∗,W ), let δ− ∈ V and δ+ ∈ Vdenote the nodes indicating the negative and positivedecision, respectively. Given a set of nodes v1, ..., vn ∈ V ,we define the generalized (negative) discrimination score forthe subgroup or the individual represented by {v1, ..., vn}as

gds−(v1, ..., vn) =ppr(δ−|v1, ..., vn)

ppr(δ−|v1, ..., vn) + ppr(δ+|v1, ..., vn).

This implicitly also defines a generalized score of positivediscrimination: gds+(v1, ..., vn) = 1− gds−(v1, ..., vn).

5. EXPERIMENTAL EVALUATIONThis section reports the experimental evaluation of our

approach on four datasets, Adult, German credit andcensus-income from the UCI Repository of Machine Learn-ing Databases6, and Berkeley Admissions Data from [34].These are well-known real-life datasets typically used indiscrimination-detection literature.

Adult: consists of 48,842 tuples and 10 attributes, whereeach tuple correspond to an individual and it is describedby personal attributes such as age, race, sex, relationship,education, employment, etc. Following the literature, in or-der to define the decision attribute we use the income levels,≤50K (negative decision) or >50K (positive decision). Weuse four levels in the partial order for temporal priority: age,race, sex, and native country are defined in the first level;education, marital status, and relationship are defined in thesecond level; occupation and work class are defined in thethird class, and the decision attribute (derived from income)is the last level.

6http://archive.ics.uci.edu/ml

Table 1: SBCN main characteristics.Dataset |V | |A| avgDeg maxInDeg maxOutDeg

Adult 92 230 2.5 7 19German credit 73 102 1.39 3 7Census-income 386 1426 3.69 8 54

0.0 0.2 0.4 0.6 0.8 1.0

0.0

0.4

0.8

edge scores

cdf

AdultGermancensus−income

Figure 2: Distribution of edge scores.

German credit: consists of 1000 tuples with 21 attributeson bank account holders applying for credit. The decision at-tribute is based on repayment history, i.e., whether the cus-tomer is labeled with good or bad credit risk. Also for thisdataset the partial order for temporal priority has four or-ders. Personal attributes such as gender, age, foreign workerare defined in the first level. Personal attributes such as em-ployment status and job status are defined in the secondlevel. Personal properties such as savings status and credithistory are defined in the third level, and finally the decisionattribute is the last level.

Census-income: consists of 299,285 tuples and 40 at-tributes, where each tuple correspond to an individual andit is described by demographic and employment attributessuch as age, sex, relationship, education, employment, ext.Similar to Adult dataset, the decision attribute is the in-come levels and we define four levels in the partial order fortemporal priority.

Building the SBCN just take a handful of seconds in Ger-man credit and Adult, and few minutes in Census-income ona commodity laptop. The main characteristics of the ex-tracted SBCN are reported in Table 1, while the distributionof the edges scores W (e) is plotted in Figure 2.

As discussed in the Introduction we also use the datasetfrom the famous 1973 episode at University of Californiaat Berkeley, in order to highlight the differences betweencorrelation-based and causation-based methods.

Berkeley Admissions Data: consists of 4,486 tuples andthree attributes, where each tuple correspond to an indi-vidual and it is described by the gender of applicants andthe department that they apply for it. For this dataset thepartial order for temporal priority has three orders. Genderis defined in the first level, department in the second level,and finally the decision attribute is the last level. Table 2is a three-way table that presents admissions data at theUniversity of California, Berkeley in 1973 according to thevariables department (A, B, C, D, E), gender (male, female),and outcome (admitted, denied). The table is adapted fromdata in the text by Freedman, et al. [34].

Male FemaleAdmitted Denied Admitted Denied Department512 313 89 19 A313 207 17 8 B120 205 202 391 C138 279 131 244 D53 138 94 299 E22 351 24 317 F

Table 2: Berkeley Admission Data

5.1 Community detection on the SBCN

Given that our SBCN is a directed graph with edge weight,as a first characterization we try to partition it using arandom-walks-based community detection algorithm, calledWalktrap and proposed in [33], whose unique parameter isthe maximum number of steps in a random walk (we set itto 8), and which automatically identifies the right number ofcommunities. The idea is that short random walks tend tostay in the same community (densely connected area of thegraph). Using this algorithm over the reconstructed SBCNfrom Adult dataset, we obtain 5 communities: two largerones and three smaller ones (reported in Table 3). Inter-estingly, the two larger communities seem built around thenegative (C1) and the positive (C2) decisions.

Figure 1 in Section 3 shows the subgraph of the SBCNcorresponding to C2 (that we can call, the favoritism clus-ter): we note that such cluster also contains nodes suchas sex Male, age old, relationship Husband. The other largecommunity C1, can be considered the discrimination cluster:beside the negative decision it contains other nodes repre-senting disadvantaged groups such as sex Female, age young,race Black, marital status Never married. This good separa-bility of the SBCN in the two main clusters of discriminationand favoritism, highlights the goodness of the causal struc-ture captured by the SBCN.

5.2 Group discrimination and favoritismWe next focus on assessing the discrimination score ds−

we defined in Section 4.1, as well as the average number ofsteps that the random walks take to reach the negative andpositive decisions, denote as−(v) and as+(v) respectively.

Tables 4, 5 and 6 report the top-5 and bottom-5 nodesw.r.t. the discrimination score ds−, for datasets Adult, Ger-man and Census-income, respectively. The first and mostimportant observation is that our discrimination score pro-vides a very clear signal, with some disadvantaged groupshaving very high discrimination score (equal to 1 or veryclose), and similarly clear signals of favoritism, with groupshaving ds−(v) = 0, or equivalently ds+(v) = 1. This is moreclear in the Adult dataset, where the positive and negativedecisions are artificially derived from the income attribute.In the German credit dataset, which is more realistic as thedecision attribute is truly about credit, both discriminationand favoritism are less palpable. This is also due to the factthat German credit contains less proper causal relations, asreflected in the higher sparsity of the SBCN. A consequenceof this sparsity is also that the random walks generally needmore steps to reach one of the two decisions. In Census-income dataset, we observe favoritism with respect to mar-ried and asian pacific individuals.

Table 3: Communities found in the SBCN extractedfrom the Adult dataset by Walktrap[33]. In the ta-ble the attributes are shortened as in parenthesis:age (ag), education (ed), marital status (ms), na-tive country (nc), occupation (oc), race(ra), rela-tionship (re), sex (sx), workclass (wc).

C1

negative dec, wc:Private, ed:Some college, ed:Assoc acdm,ms:Never married, ms:Divorced, ms:Widowed,ms:Married AF spouse, oc:Sales, oc:Other service,

oc:Priv house serv, re:Own child, re:Not in family, re:Wife,re:Unmarried, re:Other relative, ra:Black, oc:Armed Forces,oc:Handlers cleaners, oc:Tech support, oc:Transport moving,

ed:7th 8th, ed:10th, ed:12th, ms:Separated,ed:HS grad,ed:11th, nc:Outlying US Guam USVI etc,

nc:Haiti, ag:young, sx:Female, ra:Amer Indian Eskimo,nc:Trinadad Tobago, nc:Jamaica, oc:Machine op inspct,

ms:Married spouse absent, oc:Adm clerical,C2

positive dec, oc:Prof specialty, wc:Self emp not inc,ms:Married civ spouse, oc:Craft repair,oc:Protective serv,

re:Husband, ed:Prof school, wc:Self emp inc,ag:old , wc:Local gov, oc:Exec managerial,

ed:Bachelors, ed:Assoc voc, ed:Masters, wc:Never worked,wc:State gov, ed:Doctorate, sx:Male, nc:India, nc:Cuba

C3

oc:Farming fishing, wc:Without pay, nc:Mexico, nc:Canada,nc:Italy, nc:Guatemala, nc:El Salvador, ra:White,

nc:Poland, ed:1st 4th, ed:9th,ed:Preschool, ed:5th 6thC4

nc:Iran, nc:Puerto Rico, nc:Dominican Republic,nc:Columbia, nc:Peru, nc:Nicaragua, ra:Other

C5

nc:Philippines, nc:Cambodia, nc:China, nc:South,nc:Japan, nc:Taiwan, nc:Hong, nc:Laos, nc:Thailand,

nc:Vietnam, ra:Asian Pac Islander

5.3 Genuine requirementWe next focus on genuine requirement (or explainable dis-

crimination). Table 7 reports some examples of fraction ofexplainable discrimination (both positive and negative) onthe Adult dataset. We can see how some fractions of dis-crimination against protected groups, can be “explained” byintermediate nodes such as having a low education profile,or a simple job. In the case these intermediate nodes areconsidered legally grounded, then one cannot easily file adiscrimination claim.

Similarly, we can observe that the favoritism towardsgroups such as married men, is explainable, to a large ex-tent, by higher education and good working position, suchas managerial or executive roles.

5.4 Subgroup and Individual DiscriminationWe next turn our attention to subgroup and individual

discrimination discovery. Here the problem is to assign ascore of discrimination not to a single node (a group), but tomultiple nodes (representing the attributes of an individualor a subgroup of citizens). In Section 4.4 we have introducedbased on the PageRank of the positive and negative decision,ppr(δ+) and ppr(δ−) respectively, personalized on the nodesof interest. Figure 3 presents a scatter plot of ppr(δ+) versusppr(δ−) for each individual in the German credit dataset.We can observe the perfect separation between individualscorresponding to a high personalized PageRank with respectto the positive decision, and those associated with a highpersonalized PageRank relative to the negative decision.

Table 4: Top-5 and bottom-5 groups by discrimina-tion score ds−(v) in Adult dataset.

ds−(v) as−(v) as+(v)relationship Unmarried 1 1.164 -

marital status Never married 0.996 1.21 2.14age Young 0.995 2.407 3.857race Black 0.994 2.46 4.4sex Female 0.98 2.60 3.76

ds−(v) as−(v) as+(v)relationship Husband 0 - 2

marital status Married civ spouse 0 - 2.06sex Male 0 - 3.002

native country India 0.002 4.0 3.25age Old 0.018 2.062 2.14

Table 5: Top-5 and bottom-4 groups by discrimina-tion score ds−(v) in German credit. We report only thebottom-4, because there are only 4 nodes in whichds+(v) > ds−(v).

ds−(v) as−(v) as+(v)residence since le 1d6 1 6.0 -residence since gt 2d8 1 2.23 -

residence since from 1d6 le 2d2 1 6.0 -age gt 52d6 0.86 3.68 4.0

personal status male single 0.791 5.15 5.0

ds−(v) as−(v) as+(v)job unskilled resident 0 - 2.39

personal status male mar or wid 0.12 8.0 4.4age le 30d2 0.186 7.0 3.34

personal status female 0.294 6.48 4.4div or sep or mar

Table 6: Top-5 and bottom-5 groups by discrimina-tion score ds−(v) in Census-income dataset.

ds−(v) as−(v) as+(v)MIGSAME Not in universe under 1 year old 0.71 4.09 8.82

WKSWORK 94 5 inf 0.625 3.0 6.76AWKSTAT Not in labor force 0.59 2.0 6.16

VETYN 0 5 20 5 0.58 1.01 5.17MARSUPWT 3188 455 4277 98 0.55 5.0 9.25

ds−(v) as−(v) as+(v)AHGA Doctorate degreePhD EdD 0 - 3.07

AMARITL Married A F spouse present 0 - 4.49AMJOCC Sales 0 - 2.0

ARACE Asian or Pacific Islander 0 - 6.47VETYN 20 5 32 5 0 - 5.89

Table 7: Fraction of explainable discrimination forsome exemplar pair of nodes in the Adult dataset.

Source node Intermediate fed−(v)race Amer Indian Eskimo education HS grad 0.481

sex Female occupation Other service 0.310age Young occupation Other service 0.193

relationship Unmarried education HS grad 0.107race Black education 11th 0.083

Source node Intermediate fed+(v)relationship Husband occupation Exec managerial 0.806

sex Male occupation Exec managerial 0.587native country Iran education Bachelors 0.480native country India education Prof school 0.415

age Old occupation Exec managerial 0.39

Such good separation is also reflected in the generalizeddiscrimination score (Definition 4) that we obtain by com-bining ppr(δ+) versus ppr(δ−).

0.010 0.015 0.020 0.025

0.01

00.

014

0.01

80.

022

ppr(+)

ppr(

−)

Figure 3: Scatter plot of ppr(δ+) versus ppr(δ−) foreach individual in the German credit dataset.

gds−

Fre

quen

cy

0.3 0.4 0.5 0.6 0.7

020

6010

0

Figure 4: Individual discrimination: histogram rep-resenting the distribution of the values of the gener-alized discrimination score gds− for the populationof the German credit dataset.

In Figure 4 we report the distribution of the generalizeddiscrimination score gds− for the population of the Germancredit dataset: we can make a note of the clear separationbetween the two subgroups of the population.

In the Adult dataset (Figure 5) we do not observe thesame neat separation in two subgroups as in the Germancredit dataset, also due to the much larger number of points.Nevertheless, as expected, ppr(δ+) and ppr(δ−) still exhibitanticorrelation. In Figure 5 we also use colors to show twodifferent groups: red dots are for age Young and blue dotsare for age Old individuals. As expected we can see that thered dots are distributed more in the area of higher ppr(δ−).

The plots in Figure 6 have a threshold t ∈ [0, 1] on theX-axis, and the fraction of tuples having gds−() ≥ t on theY-axis, and they show this for different subgroups. The firstplot, from the Adult dataset, shows the group female, young,and young female. As we can see the individuals that areboth young and female have a higher generalized discrim-ination score. Similarly, the second plot shows the groupsold, single male, and old single male from the German creditdataset. Here we can observe much lower rates of discrimina-tion with only 1/5 of the corresponding populations havinggds−() ≥ 0.5, while in the previous plot it was more than85%.

0.02 0.03 0.04 0.05 0.06 0.07 0.08

0.04

0.08

ppr(+)

ppr(

−)

Figure 5: Individual discrimination: scatter plot ofppr(δ+) versus ppr(δ−) for each individual in the Adultdataset. Red dots are for age Young and blue dotsare for age Old.

0.4 0.5 0.6 0.7 0.8

0.0

0.4

0.8

t

femaleyoungyoung female

0.3 0.4 0.5 0.6 0.7

0.0

0.4

0.8

t

oldsingle maleold single male

Figure 6: Subgroup discrimination: plots reportinga threshold t ∈ [0, 1] on the X-axis, and the fractionof tuples having gds−() ≥ t on the Y-axis. The topplot is from Adult, while the bottom is from Germancredit.

5.5 Comparison with prior artIn this section, we discuss examples in which our

causation-based method draws different conclusions fromthe correlation-based methods presented in [13, 14, 15] usingthe same datasets and the same protected groups 7.

The first example involves the foreign worker group fromGerman Credit dataset, whose contingency table is reportedin Figure 7. Following the approaches of [13, 14, 15] theforeign worker group results strongly discriminated. In factFigure 7 shows an RD value (risk difference) of 0.244 whichis considered a strong signal: in fact RD > 0 is alreadyconsidered discrimination [15].

However, we can observe that the foreign worker group isper se not very significant, as it contains 963 tuples out of

7We could not compare with [22] due to repeatability issues.

decision- +

foreign worker=yes 298 667 968foreign worker=no 2 30 32

300 700 1000

p1 = 298/968 = 0.307p2 = 2/32 = 0.0625

RD = p1 − p2 = 0.244

Figure 7: Contingency table for foreign worker in theGerman credit dataset.

decision- +

race=black 4119 566 4685race 6=black 33036 11121 44157

37155 11687 48842

p1 = 4119/4685 = 0.879p2 = 33036/44157 = 0.748

RD = p1 − p2 = 0.13

Figure 8: Contingency table for race black in theAdult dataset.

1000 total. In fact our causal approach does not detect anydiscrimination with respect to foreign worker which appearsas a disconnected node in the SBCN.

The second example is in the opposite direction. Considerthe race black group from Adult dataset whose contingencytable is shown in Figure 8. Our causality-based approach de-tects a very strong signal of discrimination (ds−() = 0.994),while the approaches of [13, 14, 15] fail to discover discrim-ination against black minority when the value of minimumsupport threshold used for extracting classification rules ismore than 10%. On the other hand, when such minimumsupport threshold is kept lower, the number of extractedrules might be overwhelming.

Finally, we turn our attention to the famous example offalse-positive discrimination case happened at Berkeley in1973, that we discussed in Section 1. Figure 9 presents theSBCN extracted by our approach from Berkeley AdmissionData. Interestingly, we observe that there is no direct edgebetween node sex Female and Admission No. And sex Femaleis connected to node Admission No through nodes of Dep C,Dep D, Dep E, and Dep F, which are exactly the depart-ments that have lower admission rate. By running our ran-dom walk-based methods over SBCN we obtain the value of1 for the score of explainable discrimination confirming thatapparent discrimination in this dataset is due the fact thatwomen tended to apply to departments with lower rates ofadmission.

Similarly, we observe that there is no direct edge be-tween node sex Male and Admission Yes. And sex Male isconnected to node Admission Yes through nodes of Dep A,and Dep B, which are exactly the departments that havehigher admission rate. By running our random walk-basedmethods over SBCN we obtain the value of 1 for the scoreof explainable discrimination confirming that apparent fa-voritism towards men is due to the fact that men tended toapply to departments with higher rates of admission.

However, following the approaches of [13, 14, 15], thecontingency table shown in Figure 10 can be extracted fromBerkeley Admission Data. As shown in Figure 10, the valueof RD suggests a signal of discrimination versus women.

Male

Dep_A

0.252

Dep_B

0.183

Admission_Yes

0.33 0.254

Female

Dep_C

0.201

Dep_D

0.047

Dep_E

0.142

Dep_F

0.045

Admission_No

0.039 0.052 0.15 0.378

Figure 9: The SBCN constructed from Berkeley Ad-mission Data dataset.

decision- +

gender=female 1278 557 1835gender=male 1493 1158 2651

2771 1715 4486

p1 = 1278/1835 = 0.696p2 = 1493/2651 = 0.563RD = p1 − p2 = 0.133

Figure 10: Contingency table for female in the Berke-ley Admission Data dataset.

This highlights once more the pitfalls of correlation-basedapproaches to discrimination detection and the need for aprincipled causal approach, as the one we propose in thispaper.

6. CONCLUSIONSDiscrimination discovery from databases is a fundamental

task in understanding past and current trends of discrimi-nation, in judicial dispute resolution in legal trials, in thevalidation of micro-data before they are publicly released.While discrimination is a causal phenomenon, and any dis-crimination claim requires to prove a causal relationship, thebulk of the literature on data mining methods for discrimi-nation detection is based on correlation reasoning.

In this paper, we propose a new discrimination discoveryapproach that is able to deal with different types of dis-crimination in a single unifying framework. It is the firstdiscrimination detection method grounded in probabilisticcausal theory. We define a method to extract a graph rep-resenting the causal structures found in the database, andthen we propose several random-walk-based methods overthe causal structures, addressing a range of different dis-crimination problems.

Our experimental assessment confirmed the great flexibil-ity of our proposal in tackling different aspects of the dis-crimination detection task, and doing so with very cleansignals, clearly separating discrimination cases.

Repeatability. Our software together with the datasetsused in the experiments are available at http://bit.ly/

1GizSIG.

7. REFERENCES[1] Australian Legislation, “Equal opportunity

act–victoria state,(b) anti-discriminationact–queensland state, 2008.”

[2] EU Legislation, “(a) race equalitydirective,2000/43/ec, 2000; (b) employment equalitydirective, 2000/78/ec, 2000; (c) equal treatment ofpersons, european parliament legislative resolution,2009.”

[3] S. R. Foster, “Causation in antidiscrimination law:Beyond intent versus impact,” Hous. L. Rev., vol. 41,p. 1469, 2004.

[4] M. Dabady, R. M. Blank, C. F. Citro et al., Measuringracial discrimination. National Academies Press,2004.

[5] P. J. Bickel, E. A. Hammel, and J. W. O’Connell, “Sexbias in graduate admissions: Data from berkeley,”Science, vol. 187, no. 4175, 1975.

[6] C. Hitchcock, “Probabilistic causation,” in TheStanford Encyclopedia of Philosophy, 2012.

[7] P. Suppes, A probabilistic theory of causality.North-Holland Publishing Company, 1970.

[8] G. Schwarz, “Estimating the dimension of a model,”Annals of Statistics, vol. 6, no. 2, pp. 461–464, 1978.

[9] B. Custers, T. Calders, B. Schermer, and T. Zarsky,Discrimination and Privacy in the InformationSociety: Studies in Applied Philosophy, Epistemologyand Rational Ethics. Springer, 2012, vol. 3.

[10] A. Romei and S. Ruggieri, “A multidisciplinary surveyon discrimination analysis,” The KnowledgeEngineering Review, pp. 1–57, 2013.

[11] S. Hajian and J. Domingo-Ferrer, “A methodology fordirect and indirect discrimination prevention in datamining,” TKDE, 25(7), 2013.

[12] F. Kamiran and T. Calders, “Data preprocessingtechniques for classification without discrimination,”KAIS, 33(1), 2012.

[13] D. Pedreshi, S. Ruggieri, and F. Turini,“Discrimination-aware data mining,” in KDD, 2008.

[14] D. Pedreschi, S. Ruggieri, and F. Turini, “Measuringdiscrimination in socially-sensitive decision records.” inSDM, 2009.

[15] S. Ruggieri, D. Pedreschi, and F. Turini, “Data miningfor discrimination discovery,” TKDD, 4(2), 2010.

[16] B. T. Luong, S. Ruggieri, and F. Turini, “k-nn as animplementation of situation testing for discriminationdiscovery and prevention,” in KDD, 2011.

[17] I. Rorive, “Proving discrimination cases: The role ofsituation testing,” 2009.

[18] I. Zliobaite, F. Kamiran, and T. Calders, “Handlingconditional discrimination,” in ICDM, 2011.

[19] C. Dwork, M. Hardt, T. Pitassi, O. Reingold, andR. Zemel, “Fairness through awareness,” in ITCS,2012.

[20] D. Pedreschi, S. Ruggieri, and F. Turini, “Integratinginduction and deduction for finding evidence ofdiscrimination,” in ICAIL, 2009.

[21] S. Ruggieri, S. Hajian, F. Kamiran and X. Zhang,“Anti-discrimination Analysis Using Privacy AttackStrategies,” in PKDD, 2014.

[22] K. Mancuhan and C. Clifton, “Combatingdiscrimination using bayesian networks,” ArtificialIntelligence and Law, 22(2), 2014.

[23] D. Koller and N. Friedman, Probabilistic graphicalmodels: principles and techniques. MIT press, 2009.

[24] P. Menzies, “Counterfactual theories of causation,” inThe Stanford Encyclopedia of Philosophy, spring2014 ed., E. N. Zalta, Ed., 2014.

[25] J. Woodward, “Causation and manipulability,” in TheStanford Encyclopedia of Philosophy, winter 2013 ed.,E. N. Zalta, Ed., 2013.

[26] E. Brenner and D. Sontag, “Sparsityboost: A newscoring function for learning bayesian networkstructure,” arXiv:1309.6820, 2013.

[27] L. O. Loohuis, G. Caravagna, A. Graudenzi,D. Ramazzotti, G. Mauri, M. Antoniotti, andB. Mishra, “Inferring tree causal models of cancerprogression with probability raising,” PLoS ONE,9(12), 2014.

[28] D. Ramazzotti et al. “Capri: Efficient inference ofcancer progression models from cross-sectional data,”Bioinformatics, 31(18), 2015.

[29] US Federal Legislation, “(a) equal credit opportunityact, (b) fair housing act, (c) intentional employmentdiscrimination, (d) equal pay act, (e) pregnancydiscrimination act, (f) civil right act., 2010.”

[30] E. Ellis and P. Watson, EU anti-discrimination law.Oxford University Press, 2012.

[31] S. Hajian, J. Domingo-Ferrer, A. Monreale,D. Pedreschi, and F. Giannotti, “Discrimination-andprivacy-aware patterns,” DAMI, 29(6), 2015.

[32] G. Jeh and J. Widom, “Scaling personalized websearch,” in WWW 2003.

[33] P. Pons and M. Latapy, “Computing communities inlarge networks using random walks,” in ISCIS, 2005.

[34] D. Feedman, R. Pisani, and R. Purves, “Statistics,”3rd Edition, WW Norton & Company, New York(1998).