slides distance co variance

TECHNIQUES FOR BIG DATA

FEATURE EXTRACTION USING

DISTANCE COVARIANCE

BASED PCA

Big Data

Big Data' is a blanket term for any collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

Big data requires exceptional technologies to efficiently process large quantities of data within tolerable elapsed times. A 2011 McKinsey report suggests suitable technologies include crowdsourcing, data fusion and integration, genetic algorithms, machine learning, natural language processing, signal processing, simulation, time series analysis and visualization.

How Big is Big Data?

Very large, distributed aggregations of loosely structured data – often incomplete and inaccessible.

Petabytes/exabytes of data Millions/billions of people Billions/trillions of records.

Loosely-structured and often distributed data.

Flat schemas with few complex interrelationships

Often involving time-stamped events

Often made up of incomplete data

Often including connections between data elements that must be probabilistically inferred.

Applications that involved Big-data can be: Transactional (e.g., Facebook, PhotoBox), or, Analytic (e.g., ClickFox, Merced Applications).

(Reference Wikibon.org)

Big Data

Big Data Can be of three types:

1. Large number of attributes (>16)

2. Large number of samples

3. Large number both of attributes and samples

I have tried to work on the first case.

What is Dimensionality Reduction?

Dimensionality reduction or dimension reduction

is the process of reducing the number of random

variables under consideration (or attributes or

features or descriptors), and can be divided into

feature selection and feature extraction.

Feature Selection

Filters: Pearson’s Correlation

Wrappers: Run a classifier again and again, each

time with a new set of features selected using

backward selection or forward selection.

Feature Extraction

Feature extraction transforms the data in the high-

dimensional space to a space of fewer dimensions.

The data transformation may be linear, as in

principal component analysis (PCA), but many

nonlinear dimensionality reduction techniques also

exist. For multidimensional data, tensor

representation can be used in dimensionality

reduction through multilinear subspace learning.

Feature Extraction

The main linear technique for dimensionality

reduction, principal component analysis, performs a

linear mapping of the data to a lower-dimensional

space in such a way that the variance of the data in

the low-dimensional representation is maximized

What is Principal Component Analysis?

Principal component analysis (PCA) is a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. The number of principal components is less than or equal to the number of original variables. This transformation is defined in such a way that the first principal component has the largest possible variance (that is, accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding components. Principal components are guaranteed to be independent if the data set is jointly normally distributed. PCA is sensitive to the relative scaling of the original variables.

That is fine, but show me the MATH!

Online tutorial

(http://www.cs.otago.ac.nz/cosc453/student_tutori

als/principal_components.pdf)

PCA and BIG DATA

BIG DATA containing thousands will require a lot of

computation time for an average computer.

PCA becomes an important tool while drawing

inference from such large data sets.

What is Distance Correlation?

Distance correlation is a measure of statistical dependence between two random variables or two random vectors of arbitrary, not necessarily equal dimension. An important property is that this measure of dependence is zero if and only if the random variables are statistically independent. This measure is derived from a number of other quantities that are used in its specification, specifically: distance variance, distance standard deviation and distance covariance. These take the same roles as the ordinary moments with corresponding names in the specification of the Pearson product-moment correlation coefficient.

Distance Covariance Solved Example

Sample Data

Column 1 Column 2

1 1

2 0

-1 2

0 3

Mean 0.5 1.5

Distances

For Column 1 (aij = pow((ai^2 – aj^2), 0.5))

0 1.73 0 1

1.73 0 1.73 2

0 1.73 0 1

1 2 1 0

Using Euclidean formula to calculate distances

Mean 0.62 1.365 0.62 1

0.62

1.365

0.62

1

Grand Mean : 0.932

Similarly

Distances for column 2 (bij)

0 1 1.73 2.8

1 0 2 3

1.73 2 0 2.23

2.8 3 2.23 0

Mean

Mean 1.38 1.5 1.49

2.66

1.38

1.5

1.49

2.66

Grand Mean : 1.595

Centering both the columns

Aij = aij – ~ai – ~aj + ~a;

where

~ai = row mean of ai

~aj = column mean of aj

~a = grand mean of a

Aij

-0.308 0.677 -0.308 0.312

0.677 -1.668 0.677 0.567

-0.308 0.677 -0.308 0.312

0.312 0.567 0.312 -1.608

Similarly

We can calculate Bij

Distance Covariance = (Aij*Bij)/n^2

Distance Covariance Principal

Component Analysis

After we have obtained distance covariance, we

can find the highest eigen vectors of the covariance

matrix and then use those eigen vectors to extract

new features

These eigen vectors can be multiplied by the real

dataset to generate the reduced dataset.

PCA vs D-PCA

The classical measure of dependence, the Pearson correlation coefficient, is mainly sensitive to a linear relationship between two variables. Distance correlation was introduced in 2005 by Gabor J Szekely in several lectures to address this deficiency of Pearson’s correlation, namely that it can easily be zero for dependent variables. Correlation = 0 (uncorrelatedness) does not imply independence while distance correlation = 0 does imply independence. The first results on distance correlation were published in 2007 and 2009.

Confusion Matrix

Modifications of D-PCA

1. pow((ai^2 – aj^2),0.5)/ai+aj

2. pow((ai^2 – aj^2),0.5)/ai

These modification can be used to scale the data

which can then eliminate Normalization Step.

Results

Drawbacks

Cannot handle time series data

Cannot handle noisy data

Assumes data distribution to be normal

Sensitive to scaling of the data

Future work

Rank correlation

Distance based source separation