1

Principal Component Analysis (PCA) or Empirical Orthogonal Function (EOF) (Lorenz, 1956) Hotelling, H., 1935: The most predictable criterion. J. Ed. Phych., 26, 139-142. (from Jackson, 1991 and Graham,1996 (class notes)) Jackson, J. E., 1991: A user’s guide to principal components. Wiley & Sons, Inc., 569 pp. Objective: Reduce a data set containing a large number of variables to a data set containing many fewer variables, but that still represents a large fraction of the variability contained in the original data set. In a field: “What are the important patterns that appears in this field and how much of the variance is explained by them?”

2

Given a set of data X we want to find a transformation of X (say U) that has the maximum variance of all such linear transformations. Var (XU) = max

x1 x2 10.0 10.7 10.4 9.8 9.7 10.0 9.7 10.1

11.7 11.5 11.0 10.8 8.7 8.8 9.5 9.3

10.1 9.4 9.6 9.6

10.5 10.4 9.2 9.0

11.3 11.6 10.1 9.8 8.5 9.2

=

=

7343.067930.6793.079860.

ss

ssS

2221

2121

( ) 89.0ss

sr

21

21 ==

=

=0.10

0.10

x

xx

2

1

8

9

10

11

12

8 9 10 11 12

3

If X is the data set and U the transformation, the transformed data will be: Z = XU and

but is the variance – covariance matrix S so that var(Z) = U’SU and we want it to be max To solve the maximization problem,

1- constrain U to have unit length, so that: U’U = 1

2- define a Lagrange multiplier to allow the side constraint into the problem (Lagrange multiplier is used to find the extreme of f(x) subject to the constraint g(x) = c), so that the problem becomes:

var(Z) = f(U) = U’SU – λ(U’U – I) = max

or *** (S – λI) U = 0

1nXU'X'U

1n)XU()'XU(

)notationmatrixin(1n

Z'Z1n

)z( var(Z)

2

−=

−=

−=

−= ∑

1nX'X−

0U

)U(f=

∂∂

4

Eigenstructure of a square matrix A For a square matrix A, there is a vector E such that: AE = λE where λ is scalar. E contains the so called eigenvectors and λ the eigenvalues How to calculate λ and E? AE = λE AE – λE = 0 AIE – λIE = 0 (A – λI)E = 0 if E ≠ 0 - Solving for λ: If (A – λI) has an inverse (it is non-singular and det (A – λI) ≠ 0) then (A – λI)-1 (A – λI) E = 0 it means that E = 0 (no good) Then (A – λI) must be singular, i.e., det (A – λI) = 0 and λ can be determined from this equation (λ has as many values as the dimension of A). - Solving for E is just to substitute each value of λ in (A – λI)E = 0 and find E for each of them. We can determine the eigenstructure of a square matrix from the so called the characteristic equation

(A – λI)E = 0

5

Comparing the characteristic equation to the equation from the maximization problem, we can see that they have the same form.

(S – λI) U = 0 and (A – λI)E = 0 It means that the variation maximizing transformation of S is given by its eigenvectors, and the eigenmode with the largest eigenvalues gives the transformation with the largest variance. Due to U’U=1, the variance represented by each eigenmode is equal to the eigenvalues and

total variance Graphically, the eigenvectors give the axis of deformation and the eigenvalues, the magnitude.

∑=

=λn

1ii

8

9

10

11

12

8 9 10 11 12

6

Applying (S – λI) U = 0 to our data set: 1- Solving for λ: *** λ1 = 1.4465

λ2 = 0.0864 2- Solving for U: ***

The eigenvectors are often normalized so that they can be compared, and that is easy. Each column of E is called “principal component” or “mode”.

−=

−=

=

=

=

0.19538.09538.00.1

Eso

0.19538.0

ee

eand9538.00.1

ee

e22

122

21

111

00

07343.067930.6793.079860.

det =

λ

λ−

=

−

00

ee

4465.1004465.1

7343.067930.6793.079860.

21

11

=

−

00

ee

0864.0000864.0

7343.067930.6793.079860.

22

12

i'i

ii

eeeen =

7

Import property!! E is orthonornal e1’ e1 = 1; e2’ e2 = 1 e1’ e2 = 0 The new variables um, that will account successively for the maximum amount of variance are calculated by projecting the old variables (x) in the new axis (eigenvectors).

u = E’ X u1=e1’X is the linear combination of elements of X with the greatest variance. u2=e2’X is the linear combination with the greatest variance that is uncorrelated with u1, and so on.

PCA is possible to be applied to a data set if there are substantial correlations among the variables contained in

the original data set (it means redundant information). The elements of the new vectors are called the principal

components and are uncorrelated (orthogonal).

≠=

= ji,0ji,1

ee j'i

8

In practice: Given a data set Xnt,nx that has the dimension time x variable

1- Standardize or Normalize your data. 2- Extract the mean per column. For better physical

interpretation, PCA is only applied on centered data (anomalies).

3- Calculate the variance – covariance matrix of X (S). 4- Obtain the eigenvalues and eigenvectors of S using

whatever means you like.

5- Plot your eigenvectors (map or graphic). Each explains λ*100% of the total variance.

6- Analyze your results.

9

Given a set of data represented as a matrix X in which each column represents a variable:

X =

1- PCA can only be conducted on centered data (anomalies), i.e., the average value by column must be zero.

2- PCA (EOF) can be applied both to the covariance or to

the correlation matrix but the covariance matrix can be used only when all variables have the same units (variance has unit).

3- One can always use covariance matrix if the data is

normalized before. 4- The correlation matrix can be used always (correlation has

no unit).

x1,1 x1,2 x1,3 … x1,mx2,1 x2,2 x2,3 … x2,mx3,1 x3,2 x3,3 … x3,m xn,1 xn,2 xn,3 … xn,m

10

Analysis of the PC / EOF eigenvectors: New time series: projection of the original data on the new axes

u = E’ X u1=e1’X is the linear combination of elements of X with the greatest variance. u2=e2’X is the linear combination with the greatest variance that is uncorrelated with u1, and so on.

11

Plot the new time series and analyze them. e.g.: Fraedrich (1993) Mean zonal wind over Singapure

12

Maps - eigenvectors - local variance (squared correlation between the time series and the original data)

If you have station data: Uvo (2003)

Interpolate the data in grid points (griddata on Matlab) Plot the line contours

Stn 1 2 3 4 … ntime 1 2 3 4 … t

13

14

15

In case your data is given as grid points on a maps you have to redistribute it so the you can calculate EOF. t1 t2 tn t1 t2 tn Follow steps 1 to 6 on page 8 Return the eigenvectors to map format to plot them. Important: Before applying PCA, Replace missing data. Missing data should not be more than 10% of the total data. Exclude land grid points (in case of oceanic data) or ocean grid points (in case of land data).

16

Gurgel and Ferreira (2003)

99.49 % mean vegetation

0.12% summer/winter

0.04% spring/autumn

0.03% semi – annual cycle

17

Gurgel and Ferreira (2003)

18

Greatbatch 2000

Artic Oscillation

19

Extended EOF – EEOF The ordinary EOF provides a “snap shot” of the spatially evolving behaviour of your data. Suppose you are studying a phenomenon that varies in time and space. How to solve this problem? time map 1 y1 y2 y3 … yn-1 yn 2 y1 y2 y3 … yn-1 yn 3 y1 y2 y3 … yn-1 yn t y1 y2 y3 … yn-1 yn Time 1 map1 map2 map3 map42 map2 map3 map4 map53 map3 map4 map5 map64 map4 map5 map6 map7… … … … … t-3 map t-3 map t-2 map t-1 map t Look at this just as a new data matrix with dimension t x n*4. Apply steps 1 to 7 (page 8). The ordering can be done however seems useful Time 1 map1 map3 map5 map7 2 map2 map4 map6 map8 3 map3 map5 map7 map9 4 map4 map6 map8 map10… … … … …

20

For seasonally varying systems, it is common to use: Time 1 1960 Jan Feb Mar Apr2 1961 Jan Feb Mar Apr3 1962 Jan Feb Mar Apr4 1963 Jan Feb Mar Apr… … … … … … t Jan Feb Mar Apr EEOF are specially suitable for detecting patterns of systematic evolution such as propagation

21

Example: Fraedrich 1993

22

Rotated EOF Used when the physical interpretation of the principal components (eigenvectors) is important. Mathematically, this process is the relaxation of the orthogonality constrain on the principal components (eigenvectors). Physically, it means that the second and further eigenvectors are chosen as physical representation of the signal present in the data set and not only as orthogonal to the previous eigenvector. Used mainly when the variance is distributed among several eigenvectors, i.e., not concentrated in the first eigenvector. Rotate or not rotate? Varimax method – the most commonly used. Found in most of the statistical packages.

23

24

Important references: Bretherton et al 1992 JC Chen and Haar 1993 MWR Fraedrich et al 1993 JAS Fraedrich et al 1997 JAS Gurgel Ferreira 2003 PCA Korres et al 2000 JC – EOF Uvo 2003 Luo and Yamagata 2002 Busuioc A, Chen DL, Hellstrom C. Temporal and spatial variability of precipitation in Sweden and its link with the large-scale atmospheric circulation TELLUS SERIES A-DYNAMIC METEOROLOGY AND OCEANOGRAPHY 53 (3): 348-367 MAY 2001 Most references can be downloaded by ftp from air.tvrl.lth.se username: course passwd: courseTSA cd papers mget * or taken from fire/public/cintia/multivariate analysis

25

Report You are expected to read through the references above before doing your report on EOF. They are several examples about how to use PCA/EOF in geosciences. Describe your data. Apply EOF to your data set. Interpret your results physically.

How is the explained variance distributed among the modes?

How many modes you consider important, why? Can you explain physically the modes you consider important? What do they mean?

Rotate your EOF (varimax)

Does it improve the physical analysis of your higher modes? Can you explain physically more modes than before?

Apply extended EOF to your data set (anyway you like).

Does it provide any further information about the time evolution?

26

A general routine to perform PCA/EOF % General routine to perform PCA clear load %load your data file z = <data file>; [nt,nz] = size(z); %---------------------------------------------------------------- % check for zero mean per column on z %---------------------------------------------------------------- %Calculation of the anomalies of a matrix time vs. variable (z) anom = z - (1/size(z,1))*ones(size(z,1))*z; clear z % standardization d = diag(std(anom)); z = anom*d^(-1); % z is the standardized matrix data clear d anom %---------------------------------------------------------------- % calculation of PCA covz = cov(z); [pc,variances,explained] = pcacov(covz); h = pc; s = variances; expvar = explained; clear pc variances explained

27

%--------------------------------------------------------------- % Saving the explained variance fprintf('explained variance %10.2f \n',expvar(1:5) ) fid = fopen('expvar.dat','w'); fprintf(fid,'%5.2f \n',expvar); fclose(fid); %---------------------------------------------------------------- % Working on eigenvectors % Calculating and plotting the time series bk = z * h; % time series (dimension time x mode) for i=1:7 % this 7 is the number of modes ( you choose) plot(bk(:,i)')

title([' time series for mode 'num2str(i) ' var … explained 'num2str(expvar(i)) ])

pause end %---------------------------------------------------------------- % here eigenvectors are saved as they are, in asc format fid1 = fopen('eigenvec.dat','w');

fprintf(fid1,'%10.5f %10.5f\n',h(:,1:7)') % 7 is the … number of modes

fclose(fid1);

28

%---------------------------------------------------------------- % Calculating local variance (squared correlation btw the time % series and the original data ser) for i= 1:nz for j = 1:7 cor = corrcoef([bk(:,j) z(:,i)]); cc(i,j) = cor(1,2); % dimension variable X ... mode end end clear h h = cc; %----------------------------------------------------------- fclose('all')

29

Program pcarot_general.m % Routine to perform EOF using SVD % rotation is made using varmaxt.m function clear load 'data file' z = <data file>; [nt,nz] = size(z); %---------------------------------------------------------------- % check for zero mean per column on z %---------------------------------------------------------------- %Calculation of the anomalies of a matrix time vs. variable (z) anom = z - (1/size(z,1))*ones(size(z,1))*z; clear z % standardization d = diag(std(anom)); z = anom*d^(-1); % z is the standardized matrix data clear d anom %---------------------------------------------------------------- % preparing for the calculation % cross covariance matrix ccov = z' * z / (nt-1); [g,s,h] = svd(ccov,0); % diag(s(1:5,1:5)) h2 = h*sqrt(s); %------------------------------------------------------------- % rotating [hrot,at,cscor,vrot,a] = varmaxt(h2,3,'Y'); %---------------------------------------------------------------

30

% % calculation of the new explained variance expvar = vrot/trace(s); fprintf('explained variance %10.2f \n',expvar(1:3)*100 ) fid = fopen('expvar.dat','w'); fprintf(fid,'%5.2f \n',expvar); fclose(fid); %---------------------------------------------------------------- % Calculating and ploting the rotated time series bk = z * h; % time series dimention time x mode % normalizing bk % check for zero mean on z [r,n] = size(bk); clear m m = mean(bk(:,1:3)); for ii=1:r bk(ii,1:3) = bk(ii,1:3)-m; end % standardize z sz = std(bk(:,1:3)); for ii=1:r bk(ii,1:3) = bk(ii,1:3)./sz; end clear ii sz m % rotating bkrot = bk(:,1:3) * at; for i=1:3 plot(bkrot(:,i)') title(['pcp rotated time series for mode ' num2str(i) ' var explained ' num2str(expvar(i)*100) ])

31

pause print -dps -append tseries.ps end %---------------------------------------------------------------- % here I save the eigenvectors as they are, in asc format % fid1 = fopen('/usr/cintia/scand/pcpeigenvec.dat','w'); % fprintf(fid1,'%10.5f %10.5f\n',h(:,1:2)') % fclose(fid1); %return %---------------------------------------------------------------- % Calculating local variance (squared correlation btw the time % series and the original data ser) % reg_var %----------------------------------------------------------- fclose('all')