correlation implies causation ? saad saleh team lead, wisnet lab, seecs saad.saleh@seecs.edu.pk

Correlation implies

Causation ? Saad Saleh

Team Lead, Wisnet Lab, SEECS

saad.saleh@seecs.edu.pk

Contents

• Correlation

• Causality

• Examples

• Causal Research

• Causality Techniques:

• Granger Causality

• Zhang Causality

• Peter Causality

• LINGAM Causality

• Practical Applications

• Conclusion

2

Correlation• Correlation means how closely related two sets of data

are

• In statistics, Dependence refers to any statistical relationship between two random variables or two sets of data. Correlation refers to any of a broad class of statistical relationships involving dependence.

[wiki : http://en.wikipedia.org/wiki/Correlation_and_dependence]

• Relates to closeness, implying a relationship between objects, people, events, etc.

For example, people often believe there are more bizarre behaviors exhibited when the moon is full.

3

Causality

• Causality (also referred to as causation) is the relation between an event (the cause) and a second event (the effect), where the second event is understood as a consequence of the first.

[Random House Unabridged Dictionary]

4

Correlation ExamplesDrivers Age vs Sign Legibility

distance

Driver’s age is negatively correlated with Sign Legibility Distance5

Speed vs Fuel Consumption

6

Speed is correlated with the fuel consumption by the vehicle

Speed vs Fuel Consumption

7

Incentive vs Percentage Returned

8

Incentive is positively correlated with the Percentage Returned

9

Gun ownership vs Crime rate

Gun ownership and crime

r = .71

Gun Ownership correlates positively with crime rate

In a Gallup poll, surveyors asked, “Do you believe correlation implies

causation?”

• 64% of American’s answered “Yes” .

• 28% replied “No”.

• The other 8% were undecided.

10

See 10 simple questions to check

the influence of correlation over causality

11

Does Ice cream consumption

leads to drowning ??

Ice cream consumption is positivey correlated with number of drowning people

12

Do Pirates Stop Global Warming ??

13

No. of pirates are positivey correlated with Global Temperature

Does Shoe Size increases Reading Ability??

14

Shoe Size is positivey correlated with Reading Ability

Do Firemen cause Large Fire Damage??

15

Firemen are positivey correlated with amount of damage

Does Nationality effect SAT Score??

16

SAT scores are positivey correlated with nationality

Is Cholestrol level affected by Facebook??

17

Cholesterol level is correlated with Facebook invention

Are bad movies made because of low sale of

newspapers??

18

Shyamalin bad movies production is correlated with Newspapers

Can Internet Explorer effect Murder Rate??

19

Use of Internet explorer is correlated with murder Rate

Can Mexican lemon imports effect highway deaths??

20

Mexican Lemon imports are correlated with Highway deaths

The number of Nobel prizes won by a country (adjusting for population) correlates well with per capita chocolate consumption. 21

(New England Journal of Medicine)

Are noble prizes won by chocolate consumption??

RealityCorrelation vs. Causation

• ‘‘The correlation between workers’ education levels and wages is strongly positive”

• Does this mean education “causes” higher wages?

• We don’t know for sure !

• Correlation tells us two variables are related BUT does not tell us why

22

• Possibility 1

• Education improves skills and skilled workers

get better paying jobs

• Education causes wages to

• Possibility 2

• Individuals are born with quality A which is relevant for success in education and on the job

• Quality (NOT education) causes wages to

23

RealityCorrelation vs. Causation

Correlation vs Causation

24

Without proper interpretation, causation should not be assumed,

or even implied.

25

Causal Research

• If the objective is to determine which variable might be causing a certain behavior (whether there is a cause and effect relationship between variables) causal research must be undertaken.

26

Causal discovery

Which actions will have beneficial effects?

…your health?

…climate changes?… the economy?

What affects…

27

Available data

• A lot of “observational” data.

Correlation Causality!

• Experiments are often needed, but:

• Costly

• Unethical

• Infeasible

28

Establishing CausalityEstablishing Causality

• To establish whether two variables are causally related, that is, whether a change in the independent variable X results in a change in the dependent variable Y, you must establish:

• Time order: The cause must have occurred before the effect

• Co-variation (statistical association): Changes in the value of the independent variable must be accompanied by changes in the value of the dependent variable

• Rationale: There must be a logical and compelling explanation for why these two variables are related

• Non-spuriousness: It must be established that the independent variable X, and only X, was the cause of changes in the dependent variable Y; rival explanations must be ruled out.

29

Establishing CausalityEstablishing Causality

•Note that it is never possible to prove causality, but only to show to what degree it is probable.

30

Causation Possibilities

• A causes B.

• B causes A.

• A and B both partly cause each other.

• A and B are both caused by a third factor, C.

• The observed correlation was due purely to chance.

31

Third or Missing Variable Problem

A relationship other than causal might exist between the two variables.

It is possible that there is some other variable or factor that is causing the outcome.

32

Lung Cancer

Smoking Genetics

Coughing

AttentionDisorder

Allergy

Anxiety Peer Pressure

Yellow Fingers

Car Accident

Born an Even Day

Fatigue

Causal graph example

33

A ? B

A

BA ->

BB =Temperature

A = log(Altitude)

34

Best fit: A -> B

A -> B A <- B

35

Linear case?

A -> B A <- B

• Linear function• Gaussian input• Gaussian noise

36

Google Trends &

Google Correlate

37

38

39

40

Approach 1:Granger Causality

Prof. Clive W.J. Granger, recipient of the 2003 Nobel Prize in Economics

History In the early 1960's, I was

considering a pair of related stochastic processes which were clearly inter-related and I wanted to know if this relationship could be broken down into a pair of one way relationships. It was suggested to me to look at a definition of causality proposed by a very famous mathematician, Norbert Weiner, so I adapted this definition (Wiener 1956) into a practical form and discussed it.

Applied economists found the definition understandable and useable and applications of it started to appear. However, several writers stated that "of course, this is not real causality, it is only Granger causality.“

Clive W. J. Granger 

42

Grangers Idea“If variables are cointegrated, the

relationship among them can be expressed as Error

Correction Mechanism (ECM)”.

43

Granger Causality

• Suppose that we have three terms, Xt , Yt , and Wt , and that we first attempt to forecast Xt+1 using past terms of Xt and Wt  (without Yt). 

• We then try to forecast Xt+1 using past terms of Xt , Wt ,and Yt (with Yt). 

• If the second forecast is found to be more successful, according to standard cost functions, then the past of Y appears to contain information helping in forecasting Xt+1 that is not in past Xt or Wt . 

• In short, Yt would "Granger cause" Xt+1 if

• Yt occurs before Xt+1 ; 

• it contains information useful in forecasting Xt+1 that is not found in a group of other appropriate variables.

44

Vector Autoregression (VAR)Mathematical Definition

[Y]t = [A][Y]t-1 + … + [A’][Y]t-k + [e]t or

where: p = the number of variables be considered in the systemk = the number of lags be considered in the system[Y]t, [Y]t-1, …[Y]t-k = the 1x p vector of variables

[A], … and [A'] = the p x p matrices of coefficients to be estimated[e]t = a 1 x p vector of innovations that may be contemporaneously

correlated but are uncorrelated with their own lagged values and uncorrelated with all of the right-hand side variables.

pt

t

t

t

pkt

kt

kt

kt

ppppp

p

p

p

pt

t

t

t

ppppp

p

p

p

pt

t

t

t

e

e

e

e

Y

Y

Y

Y

AAAA

AAAA

AAAA

AAAA

Y

Y

Y

Y

AAAA

AAAA

AAAA

AAAA

Y

Y

Y

Y

......

...

...............

...

...

...

...

...

...

...............

...

...

...

...3

2

1

3

2

1

'3

'2

'1

'

3'

33'

32'

31'

2'

23'

22'

21'

1'

13'

12'

11'

1

31

21

11

321

3333231

2232221

1131211

3

2

1

45

Vector Autoregression (VAR)Example

Consider a case in which the number of variables n is 2, the number of lags p is 1 and the constant term is suppressed. For concreteness, let the two variables be called money, mt and output, yt .

The structural equation will be:

yttttt

mttttt

ymyy

ymym

1221212

1121111

46

Vector Autoregression (VAR)Example

Then, the reduced form is

ttt

ytmtttt

ym

ymm

1112111

21

1

211

21

221121

21

21111

11

1

11

47

ttt

ytmtttt

ym

ymy

2122121

2121

21

21

122221

21

11221

1

1

111

Vector Autoregression (VAR)Example

Among the statistics computed from VARs are helpful in predicting Granger Causality.

Granger causality tests – which have been interpreted as testing, for example, the validity of the monetarist proposition that autonomous variations in the money supply have been a cause of output fluctuations.

48

Vector Autoregression (VAR)Vector Autoregression (VAR)Granger CausalityGranger Causality

In a regression analysis, we deal with the dependence of one variable on other variables, but it does not necessarily imply causation.

In our GDP and M example, the often asked question is whether GDP M or M GDP. Since we have two variables, we are dealing with bilateral causality.

Given the previous GDP and M VAR equations:

yttttt

mttttt

ymmy

ymym

1221212

1121111

49

Vector Autoregression (VAR)Granger Causality

We can distinguish four cases:

Unidirectional causality from M to GDP Unidirectional causality from GDP to M Feedback or bilateral causality Independence

Assumptions: Stationary variables for GDP and M Number of lag terms Error terms are uncorrelated – if it is, appropriate

transformation is necessary

50

Vector Autoregression (VAR)Granger Causality – Estimation (t-test)

A variable, say mt is said to fail to Granger cause another variable, say yt, relative to an information set consisting of past m’s and y’s if: E[ yt | yt-1, mt-1, yt-2, mt-2, …] = E [yt | yt-1, yt-2, …].

mt does not Granger cause yt relative to an information set consisting of past m’s and y’s iff 21 = 0.yt does not Granger cause mt relative to an information set consisting of past m’s and y’s iff 12 = 0. In a bivariate case, as in our example, a t-test can be used to test

the null hypothesis that one variable does not Granger cause another variable. In higher order systems, an F-test is used.

tttt

tttt

ymy

ymm

2122121

1112111

51

1. Regress current GDP on all lagged GDP terms but do not include the lagged M variable (restricted regression). From this, obtain the restricted residual sum of squares, RSSR.

2. Run the regression including the lagged M terms (unrestricted regression). Also get the residual sum of squares, RSSUR.

3. The null hypothesis is Ho: i = 0, that is, the lagged M terms do not belong in the regression.

5. If the computed F > critical F value at a chosen level of significance, we reject the null, in which case the lagged m belong in the regression. This is another way of saying that m causes y.

Vector Autoregression (VAR)Granger Causality – Estimation (F-test)

)/(

/)(

knRSS

mRSSRSSF

UR

URR

52

Criticisms of Causality Tests Granger causality test, much used

in VAR modelling, however do not explain some aspects of the VAR:

• It does not give the sign of the effect, we do not know if it is positive or negative

• It does not show how long the effect lasts for.

• It does not provide evidence of whether this effect is direct or indirect.

53

54

Prof. Dr. Bernhard Schölkopf 

Kun Zhang

Max Planck at centre, 1931

55

Approach 2

“Distinguishing Causes from Effects using

Nonlinear Acyclic Causal Models”

Kun Zhang, Aapo Hyv¨arinen

Background

• Model-based causal discovery assumes a generative model to explain the data generating process.

• If the assumed model is close to the true one, such methods could not only detect the causal relations, but also discover the form in which each variable is influenced by others.

• For example, • Granger causality assumes that effects must follow causes and that

the causal effects are linear (Granger,1980).

• If the data are generated by a linear acyclic causal model and at most one of the disturbances is Gaussian, independent component analysis (ICA) (Hyv¨arinen et al., 2001)can be exploited to discover the causal relations in a convenient way (Shimizu et al., 2006).

57

Shortcomings

• Previous models were too restrictive for real-life problems.

If the assumed model is wrong, model-based causal discovery may give misleading results.

58

Zhang Approach

In a large class of real-life problems, the following three effects usually exist.

1. The effect of the causes is usually nonlinear.

2. The final effect received by the target variable from all its causes contains some noise which is independent

from the causes.

3. Sensors or measurements may introduce nonlinear distortions into the observed values of the variables.

Assumption: Involved nonlinearities are invertible.

59

Proposed Solution:

Each observed variable is non-linear function of its parents with additive noise, followed by non-linear distortion

If all non-linearities are invertible, conditions are given for causal relationship

Two-step method: Constrained nonlinear ICA followed by statistical independence tests, to distinguish the cause from the effect in the two-variable case

60

Proposed Causal Model:

Xi = fi,2 { fi,1 (pai) + ei}

Non-linear Distortion

(Continuous and Invertible)

Non-linear transformation

(Not necessarily Invertible)

Noise Effect in transmission from pai to xi

First stage: a nonlinear transformation of its parents pai, denoted by fi,1(pai), plus some noise (or disturbance) ei (which is independent from pai). Second stage: a nonlinear distortion fi,2 is applied to the output of the first stage to produce xi.

61

Zhang Approach

• Suppose the causal relation under examination is x1 → x2. If this causal relation holds, there exist nonlinear functions f2,2 and f2,1 such that

• e2 = f−1 2,2 (x2)−f2,1(x1) is independent from x1.

y1 = x1, y2 = g2(x2) − g1(x1).

• Use Multi-Layer perceptrons (MLP’s) to model the nonlinearities g1 and g2.

• Parameters in g1 and g2 are learned by making y1 and y2 as independent as possible.

62

Multilayer Perceptron (MLP)• A multilayer perceptron (MLP) is

a feedforward artificial neural network model that maps sets of input data onto a set of appropriate outputs.

63

Zhang Analysis

• y1 and y2 produced by the first step are the assumed cause and the estimated corresponding disturbance, respectively.

• In the second step, one needs to verify if they are independent.

• If y1 and y2 are independent, it implies x1 causes x2, and that g1 and g2 provide an estimate of f2,1 and f−1

2,2 , respectively.64

Success !!

• Zhang approach solved the problem “CauseEffectPairs” in the Pot-luck challenge, and successfully identified causes from effects

• Earned Reward : 200$

65

Approach 3

“Nonlinear causal discovery

with additive noise models”

Patrik O. Hoyer, Dominik Janzing, Joris Mooij, Jonas Peters, Bernhard Sch¨olkopf

Claim:

“Non-linearities are a blessing rather than a curse” -- Hoyer

Idea:In reality, many causal relationships are non-linear.

How about generalizing Basic linear framework to non-linear models??

67

Hoyer Approach

When causal relationships are nonlinear it typically helps break the symmetry between the observed variables and allows the identification of causal directions.

As Friedman and Nachman have pointed out, non-invertible functional relationships between the observed variables can provide clues to the generating causal model.

We show that the phenomenon is much more general; for nonlinear models with additive noise almost any nonlinearities (invertible or not) will typically yield identifiable models.

68

Model:

xi := fi ( xpa(i) ) + ni

where

fi is an arbitrary function (possibly different for each i),

xpa(i) is a vector containing the elements xj such that there is an edge from j to i in the DAG G,

the noise variables ni may have arbitrary probability densities pni (ni),

69

Hoyer Approach

Hoyer Model Estimation

Test whether x and y are statistically independent.

If not : Test whether a model

y := f(x)+n

is consistent with the data, simply by doing a nonlinear regression of y on x (to get an estimate f’ of f), calculating the corresponding residuals n’ = y - f(x),

and testing whether n’ is independent of x. If so, accept the model

y := f(x) + n;

if not, reject it.

Similarly test whether the reverse model x := g(y) + n fits the data

70

Hoyer Test Results

the “Old Faithful” dataset

• Obtains a p-value of 0.5 for the (forward) model “current duration causes next interval length” and

• a p-value of 4:4*10-9 for the (backward) model “next interval length causes current duration”

71

the “Abalone” dataset from the UCI ML repository

• The correct model “age causes length” leads to a p-value of 0.19, • The reverse model “length causes age” comes with p < 10-15

72

Hoyer Test Results

Temperature Alitude Statistics

• The correct model “altitude causes temperature” leads to p = 0:017, • “Temperature causes altitude” can clearly be rejected (p = 8*10-15)

73

Hoyer Test Results

Approach 4

“A Linear Non-Gaussian Acyclic

Model for Causal Discovery (LINGAM)”

Shohei Shimizu, Patrik O. Hoyer, Aapo Hyv¨arinen, Antti Kerminen

Assumptions

1.Data Generating Process is Linear

2.No unobserved confounders

3.Disturbance variables have non-gaussian distribution of non-zero variances

Approach:

Use of Independent Component Analysis (ICA)----- called Linear Non-Gaussian Acyclic Model (LINGAM ) Analysis

“when working with continuous-valued data, a significant advantagecan be achieved by departing from the Gaussianity assumption”

75

LINGAM Model• Linear Non-Gaussina Acyclic Model

• Data Generating process:

76

LINGAM Idea

• Key to Solution :

Observed variables are linear functions of the disturbance variables, and the disturbance variables are mutually independent and non-Gaussian.

x = Bx+e,

x= Ae,

where A = (I−B)−1.

77

LINGAM Algorithm

LINGAM can be briefly summarized as follows:

• First, use a standard ICA algorithm (e.g., FastICA algorithm) to obtain an estimate of the mixing matrix A (or equivalently of W),

• subsequently permute it and normalize it appropriately before using it to compute B containing the sought connection strengths bij.3

78

LINGAM Algorithm(1)Given : m*n data matrix X (m<<n) where each column

contains one sample vector X.

(a) Subtract mean from each row of X

(b) Apply ICA to get X = A*S, where S contains independent components in its rows

(c) Note : W= A-1

(2)Find W1 where W1 contains NO zeros on main diagonal and is obtained by permutting rows of W.

(3) Divide each row of W1 by corresponding diagonal element to get W1` with all 1’s on main diagonal

79

(4) Find B^ such that B^ = I – W~`

(5) To find causal order, find permutation matrix P of B^ which yields

B~ = P*B^*PT

B~ (close to strictly lower triangular) can be measured using

summation{i<=j} (Bij2)

80

LINGAM Algorithm

Practical Experiments

Project

Detecting Covert Links in Instant Messaging (IM) Networks Using Flow Level

Log Data

81

• Users sending Instant Messages (IM) to relay server

• Relay server forwards messages to corresponding users

• All packets contain source and destination IP addresses of user and server IP addresses only

82

Introduction

Scenario # 1

83

Scenario # 2

Introduction

• Users may be communicating behind a proxy server

• Users behind proxy servers are visible in scenario#2.

Data Set

• Yahoo! Messenger IM network.

• Data Set Details: • Area: New York City area.

• Time: 12am to 12am

• Data Set Files:

• Input Data File:

• User-to-server traffic traces.

• Ground Truth Data File:

• Record of the actual user-to-user connections.

84

Data Set StatisticsTime Duration Users Messages Sessions

8-8:10a 10 mins 3,420 15,370 1,968

8-8:20a 20 mins 5,405 33,192 3,265

8-8:30a 30 mins 7,438 53,649 4,661

8-8:40a 40 mins 9,513 75,810 6,179

8-8:50a 50 mins 11,684 99,721 7,669

8-9a 60 mins 13,953 126,694 9,264

85

Granger Causality

86

F-test statistics for Granger Causalty test

Zhang Approach Results

87

Zhang results for talking and non-talking pairs for IM networks in Yahoo!

Just for Knowledge• Classifier Tool

WEKA (Waikato Environment for Knowledge Analysis) -> popular suite of machine learning software written in Java, developed at the University of Waikato, New Zealand

88

WEKA Bird : Found in New Zealand,Vulnerable Species.

WEKA

89

Conclusion

90

Given: A causes of B;To Prove: Is it must that A and

B are correlated??Result: YES or NO;

why?? Can you show??91

92