dimension reduction and classification using pca, factor analysis and ant functions a short overview

8/3/2019 Dimension Reduction and Classification Using PCA, Factor Analysis and ant Functions a Short Overview

http://slidepdf.com/reader/full/dimension-reduction-and-classification-using-pca-factor-analysis-and-ant-functions 1/43

OutlinePrincipal Components

Factor AnalysisDiscriminant Functions

References

Dimension Reduction and ClassificationUsing PCA, Factor Analysis and

Discriminant Functions - A Short Overview

Dipayan Maiti

Laboratory for Interdisciplinary Statistical Analysis

Department of Statistics

Virginia Tech

http://www.stat.vt.edu/consult/

October 28, 2008

Dipayan Maiti Dimension Reduction and Classification Using PCA, Factor An







References

Outline

Principal ComponentsThe ConceptExampleApplications

Factor AnalysisThe ConceptExampleApplications - Disussions

Difference between Principal Components and Factor AnalysisDiscriminant Functions

The ConceptExampleApplications






References


The Problem

What to do when you have too many predictors in a model?

For example you have expression level data for 1000 genes!

Or you have customer attributes in hundreds and you areinterested in making a predictive model based on customerattributes!

Or you have second by second stock market data over a

trading day for stocks! Or in survey data where multiple questions might capture the

same kind of information (highly correlated)






References


The Cars Example

A researcher wants to build a model to find out which variables aremost significant in predicting the demand for cars but believes thata lot of variables have high correlation and the study can beeffectively done on a small number of variables without losingmuch information.


O li





References


The Problem

Given a data set with N observations like X = (x 1, . . . , x p ) for avery large p .

Figure: Data with 11 possible predictors


O tli





References


The Problem

How do we reduce the number of columns in X but still not throwaway too much information?


Outline





References


The Problem

JMP → Analyze → Multivariate Methods → PrincipalComponents → Mutlivariate(Tab) → Scatterplot Matrix


Outline





References


The Problem

Notice the highly correlated variables! We will attempt to explain most of the variability in the data,

but use a small number of principal components (parsimony)if it is possible.


Outline





References


The Geometric Interpretation

We intend to come up with rotations and projections in p

dimensions that captures most of the variability.

Figure: Plot of Principal Components in three dimensionsDipayan Maiti Dimension Reduction and Classification Using PCA, Factor An

Outline





References


The Geometric Interpretation - Eigens

We can write the principal components as:

Y 1 = a1X

. . .

Y p = apX

such that the Y s are uncorrelated and the variances for each Y isas large as possible.We find out the eigenvalues λ of the data matrix and rank them in

terms of their size.The a’s are obtained from the corresponding eigenvectors and theeigenvalues correspond to corresponding variances.Since Total population Variance = λ1 + · · · + λp

Variance explained by the k th principal component = λk

λ1+···+λp


Outline





References


The Geometric Interpretation - Eigens Summary

Principal components are determined by our predictors There is a principal component for every eigenvalue

The value of the eigenvalue gives a measure of much variationthe corresponding principal component explains


Outline



Principal ComponentsFactor Analysis

Discriminant FunctionsReferences


The Geometric Interpretation - Eigens Summary

By choosing the first few principal components (and henceeigenvalues) we might be able to explain a lot of the variationamong the predictors (not all!)

Hence we throw away some information but hopefully notmuch


Outline






The Cars Example

We have data about 387 cars with the following variables

Suggested Retail Price

Invoice price

Engine Size (liters) Number of Cylinders (=-1 if rotary engine)

Horsepower

City Miles Per Gallon

Highway Miles Per Gallon Weight (Pounds)

Wheel Base (inches)

Length (inches)

Width (inches)


Outline






The Cars Example Again

A researcher wants to build a model to find out which variables aremost significant in predicting the demand for cars but believes that

a lot of variables have high correlation and the study can beeffectively done on a small number of variables without losingmuch information.

But how to choose a fewer number of predictors?

Principal Components Analysis!


OutlineP i i l C Th C






The Cars Example

Use JMP → Analyze → Multivariate Methods → PrincipalComponents


OutlineP i i l C t Th C t






The Cars Example

Let us first look at the correlations between the variables.

Figure: CorrelationsDipayan Maiti Dimension Reduction and Classification Using PCA, Factor An

OutlinePrincipal Components The Concept






The Cars Example

What about the principal components? Can we interpret them?

Figure: Principal Components








The Cars Example

How many principal components do we need? How much of thevariation is explained?








Key Points

Principal components are functions of the predictors

The first few principal components can give us almost all theinformation in terms of the variability in the data








Applications - Discussion

To reduce the number of predictors As a first step for a predictive model where we would like to

remove correlated variables

General dimension reduction - expecting a low dimensionalstructure where higher dimensions are basically noise



The ConceptE l



p pFactor Analysis


ExampleApplications - DisussionsDifference between Principal Components and Factor Analysis

The Problem

Sometimes inherent structure of the data motivates theresearcher to group the data based on some unseen underlyingfactors.

This inherent structure can be identified through thecorrelation matrix of X .



The ConceptExample



p pFactor Analysis



The Subject Scores Problem

Consider examination scores in 6 subjects for 220 male students.The 6 subjects are Latin, English, History, Arithmetic, Algebra andGeometry. Consider the correlation matrix for the scores.

1.000.439 1.000.410 .351 1.000.288 .354 .164 1.000.329 .320 .190 .595 1.000.248 .329 .181 .470 .464 1.000



The ConceptExample




References


The Problem

The researcher believes that the subject scores will be correlatedamongst themselves in groups.

A possible hypothesis might be that there are probably twounderlying factors for the students’ scores - a factor thatcaptures the liberal arts scores and another that captures thescience scores.

But how to verify such a hypothesis? Factor Analysis!



The ConceptExample




References


Factor Loadings

For our problem the researcher thinks that there are two underlyingfactors.The underlying factors correspond to two different loadings on the

6 subjects.

Latin = L11F 1 + L12F 2 + 1

English = L21F 1 + L22F 2 + 2

. . .

Geometry = L61F 1 + L62F 2 + 6

The loadings L’s will hopefully help us interpret the factors.



F A l i

The ConceptExample




References


The Approach

Data has underlying factors

→ researcher determines number of factors

→ factor loadings to be obtained through the covariancematrix

→ researcher interprets factors based on loadings



F t A l i

The ConceptExample




References

pApplications - DisussionsDifference between Principal Components and Factor Analysis

Factor Loadings for the Subject Scores Example

Variable F1 F2 Communalities

Latin .553 .429 .490English .568 .288 .406History .392 .450 .356

Arithmetic .740 -.273 .623Algebra .724 -.211 .569

Geometry .595 -.132 .372

The factor loadings do not give us any immediately identifiable

groups or factor interpretation.Or DOES it?

Communalities give a measure of how much of the variance of thevariable is explained by the factor structure.





Factor Analysis

The ConceptExample




References

Applications - DisussionsDifference between Principal Components and Factor Analysis

The Factor Rotation

The factors are not immediately identifiable

What do we do now?

Factor structure in terms of variance explained remainsunchanged if we rotate the factors

Lets rotate and see if the factor loadings become interpretable



Factor Analysis

The ConceptExample




References


Rotated Factor Loadings for the Subject Scores Example

Variable F1 F2 Communalities

Latin .369 .594 .490

English .433 .467 .406History .211 .558 .356Arithmetic .789 -.001 .623

Algebra .752 -.054 .569Geometry .604 -.083 .372

Rotation makes the two factors immediately identifiable



Factor Analysis

The ConceptExampleA li i Di i




References


Rotated Factor Loadings Plot

Figure: Plot of factor loadings with two factors for the scores example



Factor Analysis

The ConceptExampleA li ti Di i



yDiscriminant Functions

References


Factor Analysis Approach - Summary

Decide on number of factors

Obtain factor loadings for the variables

Interpret factors

If interpretation not obvious rotate factors and check loadingsagain



Factor Analysis

The ConceptExampleApplications Disussions



yDiscriminant Functions

References


Applications

Psychometrics, Psychology, human factors - identify ”factors”that explain a variety of results on different tests

Marketing - Identify the salient attributes consumers use toevaluate products in this category.

Physical sciences, geochemistry, ecology, and hydrochemistry



Factor Analysis

The ConceptExampleApplications - Disussions





Differences

Principal components capture most of the variability in data

by using fewer dimensions that where the data exists Hence the principal components lie in the same space as data

Factor analysis conceptually tries to search for underlying butunobserved factors that define the correlation in the data

Hence factors lie in a different space than the data



Factor AnalysisThe ConceptExample




Applications

The Problem

Given data for two groups X1 = {x 11, . . . , x 1p } and

X2 = {x 21, . . . , x 2p }.x s can be thought of as explanatory variables.We would like to come up with a classification rule based on data.When we see new data we can use the classification rule to assignthe new data to any of the two groups.



Factor AnalysisDi i i t F ti

The ConceptExampleA li ti




Applications

The Classification Criteria

The rule should be based on some criteria.Our Criteria → Minimize Expected Miss-classification Cost

Expected Miss-classification Cost depends on Information about prior classification probability

Cost of miss-classifying

Note: Cost of miss-classifying can be assymetric.

In the absence of prior beliefs about classification probability wewill assume a 50:50 chance.



Factor AnalysisDiscriminant F nctions





Applications

The Classification Criteria

We intend to come up with a hyperplane in p dimensions thatseparates the two groups after minimizing the cost.

Figure: Plot of discriminant function in two dimensions with correctlyclassified and miss-classified dataDipayan Maiti Dimension Reduction and Classification Using PCA, Factor An







Applications

Normal populations with equal variances

When we minimize miss-classification our classification criteria foran arbitrary data point X is given by:

if f 1(X)f 2(X) > (cost ratio)*(prior probability ratio) then group 1

else group 2

Suppose that our two groups have normal densities f 1(x) and f 2(x)with means µ1 and µ2 and equal covariance Σ

In this case the classification criteria reduces to checking on whichside of a linear discriminant function the arbitrary data point X lies.


OutlinePrincipal ComponentsFactor Analysis

Discriminant Functions





Applications

The Hemophilia Example

To construct a procedure for detecting potential hemophilia A

carriers based on measurements of two variableslog 10(AHFActivity ) and log 10(AHF − likeAntigen)One group was from a population that did not carry thehemophilia gene and the other group was from known populationof hemophilia carriers.








Applications

The Hemophilia Example - The Discriminant Function

Figure: The discriminant function in two dimensions for Hemophilia dataDipayan Maiti Dimension Reduction and Classification Using PCA, Factor An







Applications

The Hemophilia Example - The Posterior Probabilities

Figure: The posterior probabilities that the data belongs to a specificDipayan Maiti Dimension Reduction and Classification Using PCA, Factor An






Referencespp

The Hemophilia Example - JMP Output

JMP gives the following output:

JMP rotates the data by the canonical axes

All calculations in JMP correspond to the rotated axes It gives the classification matrix and the percent data

miss-classfied

It gives the predicted group and the posterior probability thata data point belongs to a specific group

It gives the estimates of the population means, µ1 and µ2 andthe covariance Σ for the normal populations







References

Predictive Classification Applications

Educational research - To investigate which variablesdiscriminate between high school graduates who decide (1) togo to college, (2) to attend a trade or professional school, or(3) to seek no further training or education.

Medical research - Record different variables relating topatients’ backgrounds in order to learn which variables bestpredict whether a patient is likely to recover completely(group 1), partially (group 2), or not at all (group 3).

Biology - Record different characteristics of similar types(groups) of flowers, and then perform a discriminant functionanalysis to determine the set of characteristics that allows forthe best discrimination between the types.






References

References

Richard Johnson, Dean Wishern - Applied MultivariateStatistical Analysis, 5e


dimension reduction and classification using pca, factor analysis and ant functions a short overview

Documents