smooth principal components for investigating changes in covariances over time

© 2012 Royal Statistical Society 0035–9254/12/61000

Appl. Statist. (2012)61, Part 5, pp.

Smooth principal components for investigatingchanges in covariances over time

Claire Miller and Adrian Bowman

University of Glasgow, UK

[Received February 2010. Final revision December 2011]

Summary. The complex interrelated nature of multivariate systems can result in relationshipsand covariance structures that change over time. Smooth principal components analysis isproposed as a means of investigating whether and how the covariance structure of multipleresponse variables changes over time, after removing a smooth function for the mean, andthis is motivated and illustrated by using data from an aircraft technology study and a lakeecosystem. Inferential procedures are investigated in the cases of independent and dependenterrors, with a bootstrapping procedure proposed to detect changes in the direction or varianceof components.

Keywords: Bootstrapping; Principal components analysis; Smoothing; Time

1. Introduction

Multivariate systems often consist of multiple related response variables whose covariance struc-ture may change as a result of a covariate such as time. A simple example is provided by data onaircraft technology (Bowman and Foster, 1993; Saviotti, 1996) which consist of responses suchas total engine power, wingspan, length, maximum take-off weight, maximum speed and rangefor different aircraft along with their year of first manufacture between 1914 and 1984. Hereeach observation refers to a different aircraft and so the data can be considered to be indepen-dent throughout time. These responses will be highly related and standard multivariate analysismay aid in reducing the dimensionality. However, with advances in technology the covariancestructure of these characteristics is very likely to have changed over the years and it would beof interest to investigate this.

A more complex and topical example relates to lake ecosystems. Assessing the water qualityof lakes is essential to ensure policy targets such as the European Community Water Frame-work Directive 2000 are achieved. The level of nutrient inputs to lakes, as a result of pollution,must be controlled to minimize risks to the environment and human health from eutrophicationand cyanobacterial blooms (Tyler et al., 2009). Therefore, to reduce risks and to improve waterquality, the directive requires that all lakes should be at ‘good’ ecological status by 2015 (Euro-pean Parliament, 2000). To establish whether such targets for water quality are achievable it isnecessary to have a thorough understanding of all processes and relationships at work withinsuch systems.

The complex interrelated nature of ecosystems presents several statistical challenges. Suchsystems consist of multiple physical, biological and chemical variables many of which can beconsidered as both responses and covariates within the system, with auto- and cross-correlation

Address for correspondence: Claire Miller, School of Mathematics and Statistics, University of Glasgow, 15University Gardens, Glasgow, G12 8QW, UK.E-mail: [email protected]

2 C. Miller and A. Bowman

present. Relationships and covariance structures can change over time with current environ-mental issues, such as climate change, likely to impact on such changes.

Loch Leven (Scotland) is a large shallow eutrophic lake in the UK. It has been regularlymonitored by the Centre for Ecology & Hydrology in Edinburgh since 1967 and hence anexcellent long-term data series exists to explore the ecological system. The variables that aremeasured at the loch cover the chemistry, biology and climate with key variables of chlorophylla, phosphorus and zooplankton (Daphnia: water fleas which graze on phytoplankton). For theLoch Leven data, it is of interest to assess the water quality over time along with investigatinghow relationships between variables change throughout the year. Ferguson et al. (2007, 2008,2009) have investigated models to describe the mean water quality and changes in relationshipsthroughout the year. However, interpretation of such models with multiple responses becomesdifficult as a result of incorporating different covariates for each response and a correlated errorstructure. An alternative approach would be to investigate the multivariate nature of the data byassessing whether the covariance structure between ecological variables in the system changesthroughout time.

Principal components analysis (PCA) is a widely used dimension reduction technique formultivariate data (see, for example, Jolliffe (2002)). It is a natural way to explore the covariancestructure among multiple related response variables and to assess whether dimensionality canbe reduced by forming linear combinations of these variables. For regularly spaced data with atime component, PCA could be performed at each time point. However, if the data are sparse orirregular or if the time point of interest does not correspond to any observed data points, then adifferent approach is required. Any method adopted should also allow for both auto-correlationand cross-correlation, which are likely to be present in a time series context, such as the secondexample for Loch Leven.

If the purpose of PCA is purely descriptive then lack of independence does not seriouslyaffect the results. However, auto-correlation in time series data particularly needs to be recog-nized if appropriate inference is to be performed on the resulting components. Techniques forperforming PCA with time series data have been presented in the literature in both the time (see,for example, Jolliffe (2002)) and the frequency (see, for example, Brillinger (2001)) domains.

Jolliffe (2002) summarized a variety of techniques that can be used for PCA of time series andother types of dependent data. These include singular spectrum analysis and multichannel sin-gular spectrum analysis to investigate dominant periodicities; see, for example, Allen and Smith(1997). Related techniques such as principal oscillation pattern (POP) analysis (Hasselmann,1988), for several time series that can be assumed to have a multivariate first-order auto-regres-sive process, and PCA for dealing with periodicity in a time series (Kim and Wu, 1999) are alsopresented.

To investigate evolutions in multiple series over time, Jassby and Powell (1990) and Salvadoret al. (2003) proposed dynamic principal components (PCs) with Zuur et al. (2003a,b) pre-senting dynamic factor analysis. The main aim is to investigate how the related series evolve toidentify common trends over time. Applications include monthly births in the USA from 1948to 1978 and biomass of macro zoobenthic species over 15 stations in the Wadden Sea.

PCA can also be performed in the frequency domain for time series data and Boudou andViguier-Pla (2006) have provided technical results relating PCA in the frequency domain toPCA in the time domain. Brillinger (2001) devoted a chapter to PCA in the frequency domain,which provides estimation and inferential results along with an example on temperature mea-surements at 14 meteorological stations in America and Europe. The techniques are primarilyfor multichannel signals that are typically stable throughout time with a short dependence span.Results in the frequency domain have been developed, in particular, for brain imaging data from

Smooth Principal Components 3

electroencephalograms to consider time-dependent frequency domain PCA (Ombao and Ho,2006) for multichannel non-stationary signals. The main aim is to extract time varying featuresand to identify channels that are the most plausible source of variation. Lagerlund et al. (2004)stated that PCA is performed in the frequency domain for such multichannel signals since thewaveforms are of differing frequencies but have similar amplitudes and similar spatial distri-butions. This can result in the waveforms being extracted into the same component if PCA isperformed in the time domain, thus making it impossible to separate them.

In many of the applications presented by the researchers above, in both the time and the fre-quency domains, the main interest lies in investigating a single time series or multiple time seriesrecorded for a single variable (e.g. temperature, species or multichannel electroencephalogramsignals) to extract common trends over time or to identify time points with similarities in spec-tral frequencies. Missing data are also likely to be an issue for many of these techniques, withimputation required. In this paper we present smooth PCA to investigate how the covariancestructure of multiple response variables changes over time, after removing a smooth functionfor the mean. For smooth PCA the initial series do not have to be stationary and missing dataare not an issue. We develop the technique that was proposed by Prvan and Bowman (2003) whopresented non-parametric time-dependent PCA. This consists of performing PCA at each time tby using a smoothed covariance matrix, where the amount of neighbouring data that contributeis controlled by the choice of the smoothing parameter. We provide a more general version ofthe function that can be used to consider whether the covariance structure changes throughouttime in a variety of contexts involving multivariate data with a time component. In particular,alternative weight functions are used to adapt the method of smoothing to the nature of the timeseries, e.g. where periodic (seasonal) effects are of interest. Methods of inference are presentedfor data that are independent throughout time and for variables that have auto-correlation, anddifferent graphical outputs are presented to highlight how to display results effectively.

In Section 2, the technique of smooth PCA is described and illustrated by using the aircrafttechnology data, where data are independent throughout time, and the data from Loch Leven(the main motivation for this paper), which have auto- and cross-correlation. Approaches toinference are presented in Section 3 for independent and dependent data, and the techniquesare applied in Section 4 to both case-studies. A simulation study to assess the power and size ofthe inferential methods is presented in Section 5 with a discussion in Section 6.

The aircraft data that are analysed in the paper can be obtained from

http://www.blackwellpublishing.com/rss

2. Smooth principal components analysis

In smooth PCA interest lies in assessing whether the covariance structure between relatedresponse variables changes with an explanatory variable, which in the present context is time,after removing a simple smooth mean structure. The modelling involves a two-step process ofremoving a simple mean for each univariate response and then investigating the covariancestructure between the residuals of all responses.

2.1. Step 1In this paper we fit a non-parametric function for the mean of each response and hence, in thesimplest case, for each univariate response yi .i=1, . . . , n/, we have

yi =m.ti/+ "i .1/


where m.·/ is a univariate smooth function of a covariate, which is time in this context, andthe errors "i are not necessarily independent throughout time. An estimate at time t can beconstructed through a wide variety of smoothing techniques. In this paper, either local linear(2) or local mean regression (3) is implemented (see Cleveland and Devlin (1988) and Bowmanand Azzalini (1997) for further details) and an estimate at time t arises as the solution α fromthe least squares criterion expression (2) or (3) respectively, as this defines the position of thelocal regression line at point t:

minα,β

[n∑

i=1{yi −α−β.ti − t/}2 w.ti − t; h/

], .2/

minα

[n∑

i=1.yi −α/2 w.ti − t; h/

]: .3/

In expression (2) the weight function w is taken to be a normal kernel density with mean 0 andstandard deviation h, where h is the smoothing parameter and t is a particular covariate value,e.g. a time point. The kernel weight function ensures that it is observations close to the pointof interest t which have most weight in determining the estimate and the smoothing parameterh controls the width of the kernel function and hence the degree of smoothing. If the covariateis cyclic in nature, e.g. representing a seasonal component, the local mean criterion (3) is moreappropriate with a circular weight function defined as

w.ti − t, h/= exp{

1h

cos(

2πti − t

r

)}.4/

where r is the period of the function. A convenient expression is the vector matrix notationm=Sy, where m denotes the vector of estimates at a set of n evaluation time points of interest,S denotes an n × n smoothing matrix whose rows consist of the weights that are appropriateto the estimation at each evaluation point from expression (2) or (3) and y denotes the univar-iate observed response in vector form. Regardless of the weight function used, the smoothingparameter h is determined by specifying the degrees of freedom df, where df = tr.S/, enablingsmoothing parameters to be specified on a common scale. More complex functions for the meancould be removed as required and this will be discussed further in Section 4.

2.2. Step 2After estimating a smooth function for the mean of each response, local p×p covariance matri-ces can then be constructed as

Vt = .Y − M.t//TSt.Y − M.t//

for each time point t (over a grid of g time points), where Y is the n×p matrix of all p responsevariables (Y1, . . . , Yp), M.t/ is an n×p matrix containing the smooth mean estimates at time tevaluated at each observation for each response and St is a diagonal n × n matrix containingthe vector of weights for smoothing at time t, i.e. row t from S.

An eigenanalysis is then performed on the local covariance matrices to produce smooth PCsfor each time point of interest t. The p PCs indicate the directions of maximum variability inthe data, at each time point t, with the eigenvalue λtd of PC d specifying the variance of the PCand the eigenvector qtd , the loadings for PC d. It is therefore of interest to assess whether thedirections of maximum variability identified at each time point change throughout time. Theseresults can be presented graphically through plots of the eigenvalues and eigenvectors over time.


For a particular PC d , the direction and variability can be displayed at each time point t byinterpolating

√λtdqT

td over a range of suitable points.In many applications, it is common for the response variables to be on different scales and

hence data may also have to be standardized before implementing PCA. Results are presentedhere for local covariance matrices of standardized data. These smooth PCs are rotationallyinvariant. Translating and rotating the data results in the eigenvalues being unchanged and theeigenvectors being simply rotated versions of the eigenvectors of the original local estimate; seethe on-line supplementary material for further details.

The theoretical properties of smoothing techniques are well understood and well documentedin the literature through asymptotic approximations of bias and variance for estimates of smoothfunctions and guarantees of convergence if the degree of smoothing tends to zero at a suitablerate; see Bowman and Azzalini (1997). In the present context, these results can be applied tothe estimation of each element of a covariance matrix. The complexity of the spectral decom-position of this matrix as a function of its individual components prevents a simple deriva-tion of expressions for the bias and variance of eigenvectors and eigenvalues. However, if werestrict attention to the case where the true covariance matrices at each time point have aunique spectral decomposition then the convergence of the estimated covariance matrix atleast ensures the convergence of the estimated eigenvectors and eigenvalues to their true val-ues.

2.3. Illustrations of smooth principal components analysisSmooth PCA will now be illustrated by using the aircraft technology and Loch Leven appli-cations to investigate how the design characteristics of aircraft have changed over time and toassess how the covariance structure between ecological variables changes throughout the yearat Loch Leven. After implementing smooth PCA, a three-dimensional plot can be produced toillustrate the direction of maximum variability (PC 1) throughout time. In this plot the directionsof the first PC at each time point describe a surface and the width of the surface at each timepoint corresponds to the variance that is associated with the first PC.

2.3.1. Aircraft technologyThe aircraft data consist of six responses of interest. However, in this illustrative example,we apply smooth PCA to two of these variables, total engine power (kilowatts) and wingspan(metres), to investigate changes in the covariance structure of these characteristics over the yearswhile allowing the trend in the mean over time to be modelled by a smooth function. Each ofthese variables has been log-transformed, to stabilize the variance, before standardizing, sincevariables are on a different scale. To estimate the mean and local covariance functions, locallinear regression was applied with year as a covariate over a grid of g = 25 time points. Theweights were from a normal density function with mean 0 and standard deviation h, where his determined by applying 4 degrees of freedom for smoothing (h=10:7) to allow a reasonableamount of flexibility in the smooth functions.

Fig. 1(a) illustrates the direction of maximum variability (PC 1) throughout time, relative tothe smooth mean (the thick curve). The plot highlights that the direction of PC 1 is close to ver-tical in the early years, indicating little relationship between log-power and log-span. However,this appears to change smoothly to produce a relatively strong positive relationship by 1984,suggesting that larger aircraft are more powerful in the later years. The variance also appearsto have changed over time with more variability evident in the size and power of aircraft by1984.


(a) (b)

4

3.5

2.5

2

20 40 60 80 4 6log(Power)

log(TP)

54.5

3.531210

864

2

2

3

4

5

log(chla)

Month4

log(Span)

Year8 10

3

Fig. 1. Surface to highlight the direction and variance of PC 1 over the years from a smooth PCA with (a)log-power, log-span and year as a covariate and (b) log(chlorophyll), log(TP) and month as a covariate: �,data points; , smooth mean

2.3.2. Loch LevenMonthly mean data from Loch Leven will be considered only for the period January 1988–December 2007, as a result of changing regulations and substantive missing periods in the earliertime window; see Ferguson et al. (2008, 2009) for further details. The relationship between theamount of chlorophyll a (micrograms per litre), as an indicator of phytoplankton biomass, andtotal phosphorus TP (micrograms per litre), a measure of the total concentration of phosphorusavailable, will be investigated initially. Both variables are indicators of water quality and it is ofinterest to investigate whether the covariance structure between them changes throughout theyear. A natural log-transform has been applied to each variable to stabilize the variance, beforestandardizing for different scales. Smooth PCA was performed with the covariate month, whichis cyclic, and hence a circular smoother (4) was applied in local mean regression with 4 degreesof freedom (h= 0:35) over a grid of g = 25 time points. There are four missing data points foreach variable and hence there are 236 observations for each response.

Fig. 1(b) illustrates the direction of maximum variability (PC 1) throughout the months of theyear, relative to the smooth mean. The plot highlights a strong positive relationship in January,which flattens out slightly throughout the year before returning to strongly positive in Decem-ber. The variance appears to be smaller in the early and later months of the year, with highervariance evident in the summer months. Since both variables are indicators of water quality, thishighlights the strong relationship in winter months. However, moving through the year, both ofthese variables become affected by, and affect, other variables. For example, Daphnia graze onphytoplankton, reducing chlorophyll levels (Ferguson et al., 2007), changing this relationshipand hence the direction of variability. It therefore appears that the covariance structure betweenthese two variables does change throughout the year.

3. Inference

The general null hypothesis (H0) of interest here is that the covariance matrix of multiple relatedresponse variables does not change throughout time. This will be investigated by assessingwhether the observed changes in PCs over time are consistent with sampling variation. Investi-gation of the sampling properties of the coefficients and variances of the sample PCs is equivalent


to looking at the sampling properties of the eigenvalues and eigenvectors (Jolliffe, 2002). There-fore, to assess the null hypothesis, that the covariance matrix does not change throughout time,inference will be performed to investigate whether the eigenvalues and eigenvectors of particularPCs change over time.

Anderson (1963), Kollo and Neudecker (1993) and Jolliffe (2002) have given asymptoticresults for eigenvalues and normalized eigenvectors of sample variance and correlation matri-ces. However, it is difficult to see how such results can be modified to suit the local nature ofthe eigenvalues and eigenvectors (for local covariance matrices) that are presented here or toaccommodate auto-correlation.

3.1. Reference informationAn alternative approach will be used here to construct reference information illustrating thedirection that would be expected for a particular PC if the null hypothesis, that the covariancematrix does not change throughout time, is true. For independent data, such as in the aircrafttechnology study, this can be constructed by permuting the time points that are associatedwith the responses and performing smooth PCA for each new sample of data. However, inthe presence of auto-correlation for individual response variables or cross-correlation betweenvariables, such as in the Loch Leven data, this method is not appropriate.

In the latter case bootstrapping can be employed and there are many references on bootstrap-ping methods for time series (see, for example, Efron and Tibshirani (1986), Cordaus (1992)and Politis (2003)). These techniques include block bootstrapping of time series (Politis andRomano, 1994) or simulating data from a fitted model with known covariance structure toprovide a reference distribution. The latter option, namely the parametric bootstrap (see, forexample, Efron and Tibshirani (1986) and Davison et al. (2003)), will be considered for this paperto construct reference information in the presence of correlation by simulating data from a fittedmodel with an appropriate error covariance structure that does not change throughout time.

The results for two responses, from permutations in the independence case or the parametricbootstrap in the correlated case, can then be displayed graphically as a reference band or asshading, on the surfaces that were presented in Section 2 which highlight the direction of maxi-mum variability over time. Shading will be used here to indicate changes in standard deviationof direction or variance relative to the mean, from the permuted or bootstrapped samples. Fordirection, this has been done by comparing the angles, from horizontal, of the fitted surface andthe mean surface computed through permutations or bootstrapping, plus and minus 1, 2 and3 standard deviations (SDs). Approximate distributions can also be produced for eigenvaluesand eigenvectors, under hypothesis H0. These can be used to produce reference bands for ei-genvalues and eigenvectors of specific PCs to display where, for example, 95% of the sampleswould be expected to lie if H0 is true.

3.2. Test statisticsIn addition, it is also useful to have a test statistic and associated p-value to assess the nullhypotheses that the eigenvalues and/or eigenvectors do not change throughout time.

3.2.1. EigenvaluesA simple test statistic for the eigenvalue of PC d can be computed by considering the one-dimen-sional distance between the estimated eigenvalue at each time point after performing smoothPCA and the estimated eigenvalue calculated from performing standard PCA (ignoring time).


Standard PCA can be performed under an assumption of no time effect to produce λ0d , theeigenvalue for PC d. Using smooth PCA the eigenvalue for PC d can be computed at each timepoint of interest over a grid of points, to produce λtd at each time t, and the absolute differencebetween λtd and λ0d calculated. These differences can then be aggregated across the grid of gtime points to form an appropriate test statistic.

This process can be repeated for all the permuted or bootstrap samples, constructing manypossible values for the test statistic when hypothesis H0 is true. The proportion of the test sta-tistics based on the permuted or bootstrap samples that are greater than or equal to the teststatistic produced using the original data can be computed and used as an empirical p-value.

Therefore, for the eigenvalues of PC d , compute

Λd =g∑

t=1|λtd −λ0d |

and hence Λd is the test statistic for the eigenvalue. For each of the permuted or bootstrappedsamples of data compute

ΛÅd =

g∑t=1

|λÅtd −λÅ

0d |

where λÅtd is the eigenvalue of PC d at time t from a permuted or bootstrapped sample and λÅ

0dthe eigenvalue by using standard PCA from a permuted or bootstrapped sample. The p-valueis therefore the proportion of times that the value ΛÅ

d is greater than or equal to the value Λd .

3.2.2. EigenvectorsA similar test statistic can be constructed for the eigenvector of PC d. Standard PCA can beperformed on the responses to produce q0d , the eigenvector of PC d (ignoring any time effect).The p-dimensional Euclidean distance between q0d and qtd , the eigenvector for PC d estimatedat time point t by using smooth PCA, can then be computed and aggregated over the grid of gtime points to form a test statistic.

This process can be repeated for all the permuted or bootstrap samples to construct manypossible values for the test statistic when hypothesis H0 is true. Care must be taken here toavoid difficulties arising from the fact that eigenvectors are invariant to the sign of their direc-tion (multiplication by −1). Where necessary, eigenvectors should be multiplied by −1 beforecomputing the test statistics to ensure that the appropriate signed directions are compared.

Therefore, for the eigenvectors of PC d , compute

Ωd =g∑

t=1‖qtd −q0d‖

which is the test statistic for the eigenvector, where qtd and q0d are of the same signed direction.The corresponding statistic ΩÅ

d =Σgt=1‖qÅ

td −qÅ0d‖ can be computed for each permuted or boot-

strap sample and the estimated p-value is therefore the proportion of times that the value ΩÅd is

greater than the value Ωd .The performance of an alternative test statistic from common PCs (Schott, 1991, 1996, 1998,

2003) will also be investigated in this context. Jolliffe (2002) stated that the idea of commonPCs arises if we suspect that the same components underlie the covariance matrices for eachgroup (time point in this context), i.e. the same components may be present over time but theirrelative importance may vary with time. Therefore, a particular case of this would be the cur-rent context where the observed changes in the covariance matrices over time are consistentwith sampling variation, i.e. the covariance matrices do not change over time. If the covariance


matrices for the time points that are considered are different, then there is no uniquely definedset of PCs that is common to all time points. Flury (1987) developed ways of estimating andtesting for common PCs for covariance matrices. Schott (1991) extended these ideas to commonPC subspaces, spanned by the most important PC vectors, for both covariance and correlationmatrices. Schott (1991) provided test statistics for common PC subspaces across several groupsbased on eigenprojections, where each group would correspond to a particular time point inthis context.

Schott (1991) stated that if the same p variables are being measured on objects in g differentgroups and the first d PCs are sufficient for each group then the g groups have a common PCsubspace if the eigenprojection

P td = qt1qTt1 + . . . + qtd qT

td , .5/

where qtd is the eigenvector for group t and PC d , is the same for all groups. Schott (1991)provided a test statistic using eigenprojections with an asymptotic distribution that has a χ2-distribution for covariance matrices and is approximated by a χ2-distribution for correlationmatrices, to test for the common subspace hypothesis that

H0 : P1d = . . . =Pgd:

As a result of the local nature of the covariance matrices for smooth PCA and potential auto-cor-relation in variables of interest for this paper, these results do not hold here. However, the sametest statistic can be considered with inference provided through permutations or bootstrapping.

To apply the results of Schott (1991), instead of investigating a common PC subspace forseveral groups, we take the groups to be time points. Schott (1991) constructed a test based onthe differences vec.P ·dP td/−vec.P ·dPkd/ through their sums of squares, specifically

Td =g∑

t,k=1t<k

vec.P ·dP td − P ·dPkd/T vec.P ·dP td − P ·dPkd/

where g is the number of time points in this context, d is the number of largest roots considered(i.e. the number of PCs considered to be important) and P ·d is the eigenprojection correspond-ing to the d largest roots of the pooled sample correlation (or standardized covariance) matrix.Specifically, P ·d is the eigenprojection of n1V1 + . . . +ngVg, where n1, . . . , ng is the sample sizefor each group.

This test statistic (Td), based on the eigenprojections, can be constructed by using the eigen-vectors produced after applying smooth PCA on the original data. Test statistics (say T Å

d ) canalso be calculated by using the eigenvectors that are produced after applying smooth PCA toeach of the new samples from a permutation or parametric bootstrap procedure. An estimatedp-value can then be computed by calculating the proportion of test statistics T Å

d that have avalue that is greater than or equal to Td .

4. Applications

In Sections 2.3.1 and 2.3.2 it was highlighted that the relationship between power and span doesappear to change over time and the relationship between chlorophyll and TP does appear tochange throughout the year; see Fig. 1. The inferential methods in Section 3 will now be usedto assess formally whether the covariance structure is changing with time in each case-study.


In the first application it is of interest to investigate how the design characteristics of air-craft (i.e. power and span) have changed over time where it can be assumed that the errorsare independent throughout time for each variable. The second application assesses how thecovariance structure between ecological variables (i.e. chlorophyll and TP), which have auto-and cross-correlation, changes throughout the year at Loch Leven.

In both applications, a grid of g =25 time points has been applied with 4 degrees of freedomfor smoothing and 500 iterations of each permutation or bootstrap procedure. Changes havebeen assessed by using graphical techniques and the test statistics from Section 3.2 based oneigenvalues, eigenvectors and the test statistic using eigenprojections. For two response vari-ables, the fitted surfaces corresponding to PC 1, the direction of maximum variability, fromsmooth PCA will be displayed along with shading to highlight whether or not the directionand variance have changed throughout time. To illustrate a change in the direction of maxi-mum variability the shading illustrates how many SDs away from the mean direction (acquiredthrough permutations or bootstrapping) the surface lies. To illustrate a change in the variancethe shading illustrates how many SDs lower or higher than the mean variance the variance ofthe surface lies.

4.1. Aircraft technologyFig. 2 illustrates the variance and direction of PC 1 for power and span over the years. Thesurface can be shaded to illustrate how many SDs away from the mean direction (Fig. 2(a))and mean variance (Fig. 2(b)), obtained from the permuted samples, the surface lies at eachtime point of interest. For direction (Fig. 2(a)), the shading illustrates a significant change overthe first 50 years (approximately) from what would be expected under hypothesis H0. The laterperiod has less shading. However, a slight change is still evident around the 1980s. For variance(Fig. 2(b)), the shading highlights that it is significantly smaller than expected under hypothesisH0 in the early period and significantly larger from around 1955–1975.

For these two variables, interest lies specifically in the first PC, as the direction of maxi-mum variability. Applying the test statistics that were presented in Section 3.2 for PC 1 provides

4

3.5

2.5

2

log(Span)

3

4

3.5

2.5

2

log(Span)

3

20 40 60 80Year

20 40 60 80Year4 6 log(Power)8 10

4 6 log(Power)8 10

−4

−2

02

4

(a) (b)

Fig. 2. Surface to highlight the direction of PC 1 over the years from a smooth PCA with log-power andlog-span: (a) with shading to illustrate how many SDs away from the mean direction the direction of thesurface lies; (b) with shading to illustrate how many SDs away from the mean variance the variance of thesurface lies


p-values of 0.002 and less than 0.001 for the eigenvalue and eigenvector test statistics respectively.The eigenprojection is

P td = qt1qTt1

where qt1 is the eigenvector of time point t and the first PC, and a p-value of less than 0.001is computed for the test statistic based on the eigenprojection. These results highlight that theeigenvalues and eigenvectors are both changing throughout time and hence the covariancestructure is changing throughout time for the power and span of aircraft technology data. Fig.2 illustrates that the variance increases over the 70-year period, with the direction of maximumvariability changing from being mainly vertical to being quite strongly positive. This reflects thefact that, in the earlier years, aircraft tended to be smaller with less variability in size and littlerelationship with power. However, as time and technology have progressed, the range of sizesof aircraft has increased with the larger aircraft becoming more powerful.

4.2. Loch LevenPrevious studies (Ferguson et al., 2007, 2009) have used an auto-regressive AR(1) and vectorauto-regressive VAR(1) structure for the errors in this application. However, permutations ofthe covariate (as in the aircraft example) will remove such structure. To account for this correla-tion a parametric bootstrap was employed to simulate data from a fitted model that incorporatesthe correlation. Since we are only interested in the residuals after removing a smooth mean, wecan simulate data of the form

δt =Aδt−1 +ηt .6/

where ηt ∼N.0, Σ/ are independent and identically distributed with a covariance structure thatdoes not change over time. To do this we need to remove a smooth mean initially from each oflog(chlorophyll) and log(TP) and obtain residuals εt . A VAR(1) model(

"1t

"2t

)=(

a11 a12a21 a22

)("1,t−1"2,t−2

)+(

η1t

η2t

)

can then be fitted to the residuals ε to obtain estimates for the coefficients a11, . . . , a22 of A andwhite noise η. These final residuals η are now assumed to be independent.

Smooth PCA can then be applied using new samples of data from equation (6), which incorpo-rate the correlation structure, to produce reference information and enable computation of teststatistics for the eigenvalues and eigenvectors. The original surface can be shaded to illustratehow many SDs away from the mean direction and mean variance, obtained from bootstrapping,the surface lies at each time point of interest. This is illustrated in Fig. 3. The greatest changeover the year appears to be for the variance, Fig. 3(b), with only one or two time points towardsthe end of the year displaying a change in direction that is greater than 2 SDs from the mean:Fig. 3(a).

Applying the test statistics from Section 3.2 for the first PC, after performing the parametricbootstrap, produces p-values of less than 0.001 and 0.012 for the eigenvalue and eigenvectortest statistics respectively and 0.098 for the test statistic based on the eigenprojection. Althoughthe eigenprojection test statistic p-value is not significant, at a level of significance of 0.05, it isquite small. The other test statistic values are significant, suggesting that the eigenvalues andeigenvectors are not constant throughout the year. Relationships between variables within thesystem appear to be dependent on the availability of other variables at particular points in theyear.


−4

(a) (b)

−2

02

4

2

2 4 6 8 10 12Month

3

4

5

log(chla)

log(TP)

54.5

3.53

4

2

2 4 6 8 10 12Month

3

4

5

log(chla)

log(TP)

54.5

3.53

4

Fig. 3. Surface to highlight the direction of PC 1 over the months of the year after applying smooth PCA tolog(chlorophyll) and log(TP): (a) with shading to illustrate how many SDs away from the mean direction thedirection of the surface lies; (b) with shading to illustrate how many SDs away from the mean variance thevariance lies

Although there is a discrepancy between the two test statistics for direction (eigenvector andeigenprojection) the changes in direction here are small. Simulation results in Section 5 highlightthat, for moderate changes in direction, the power to detect change by using the eigenprojectiontest statistic is very similar to the power to detect change by using the eigenvector test statistic.This will be discussed more fully later.

4.2.1. Removing trend and seasonalityFor the Loch Leven data, a smooth function of month was removed initially before applyingsmooth PCA and essentially this removes a seasonal pattern over the year. However, a slightdecreasing trend is also evident for log(TP) over the period 1988–2007 and hence it may also beappropriate to remove time trend over the period. To do this an additive model can be fitted toeach response variable as

y =α+m1.year/+m2.month/+ ": .7/

This removes a more complex function for the mean before assessing whether the local covari-ance matrices of the residuals change throughout the year. To fit the additive model, local linearsmoothing is applied for the smooth function of year and local mean smoothing for the cycliccomponent of month. The same smoothing parameters have been used as in Ferguson et al.(2009), which were based on 4 degrees of freedom for each covariate. After fitting this modelfor the mean, smooth PCA with the covariate month is then applied to the residuals in a similarway to that in Section 2.3.2. The results are displayed in Fig. 4, excluding the smooth mean,which is more complex here. It can be seen that there is still evidence of changes in directionand variance early and late in the year. However, for the variance (Fig. 4(b)), changes no longerappear significant in the middle of the year. Applying the three test statistics for PC 1, p-valuesof 0.008 and less than 0.001 are calculated for the eigenvalue and eigenvector test statistics andthe test statistic based on eigenprojections gives a p-value of 0.180. Therefore, the test statistics


−4

−2

02

4

(a) (b)

2

3

4

5

log(chla)

2 4 6 8 10 12Monthlog(TP)

54.5

3.534

2

3

4

5

log(chla)

2 4 6 8 10 12Month log(TP)

54.53.53

4

Fig. 4. Surface to highlight the direction of PC 1 over the months of the year, for log(chlorophyll) and log(TP),after removing trend and seasonality: (a) with shading to illustrate how many SDs away from the mean direc-tion the direction of the surface lies; (b) with shading to illustrate how many SDs away from the mean variancethe variance lies

for eigenvalues and eigenvectors still show evidence of change over time but there is little evi-dence from the eigenprojection. The non-significant p-value for the eigenprojection appears tobe a result of the small change in the eigenvector over time and the shape of the function. It isobserved empirically that, when the change is quite small, data generated by bootstrapping canresult in larger changes between time points than the original data, resulting in the p-value forthe eigenprojection being non-significant.

4.2.2. More than two responsesThe three-dimensional plots are effective for displaying the results of the applications above.However, to display p variables more generally, the eigenvalues for PC d can be plotted againsttime with a 95% reference band to highlight where the curve would be expected to lie ifhypothesis H0 is true. Similarly, the loadings of each variable can be plotted as separate curvesagainst time with symbols to represent where the curve lies within or outside a 95% referenceband.

This has been investigated for log(chlorophyll), log(TP) and log(secchi) (a water clarity indi-cator, recorded as metres of visibility) from the Loch Leven data. After removing a smoothfunction for the mean, based on the additive model (7), smooth PCA was applied to thesethree variables with the parametric bootstrap used for inference and the results are displayed inFig. 5 for PC 1. Fig. 5(a) highlights that there are significant changes in the variance, with lowvalues at the beginning and end of the year, with small changes in the loadings for log(chloro-phyll) and log(secchi) evident in Fig. 5(b). Applying the test statistics based on the eigenvalue,eigenvector and eigenprojection for PC 1, to focus on the direction of maximum variability,provides p-values of 0.014, 0.002 and 0.450 respectively. Again, this highlights a change in thedirection and variance over time, with the exception of the p-value for the eigenprojection.However, as illustrated in Fig. 5 (and the previous examples) the change in direction is quitesmall.


2 4 6 8 10 12

1.0

1.5

2.0

2.5

3.0

Month(a)

(b)

eige

nval

ue 1

2 4 6 8 10 12

−1.

0−

0.5

0.0

0.5

1.0

Month

PC

1 lo

adin

gs

Fig. 5. (a) Eigenvalue 1 across months based on smooth PCA with responses of log(chlorophyll), log(TP)and log(secchi), with a 95% reference band displayed, and (b) PC 1 loadings across months based on smoothPCA with responses of log(chlorophyll) ( ), log(TP) (– �– �– �) and log(secchi) (. . . . . . .) (�, outside a 95%reference band; 4, within a 95% reference band)


5. Type 1 error and power

To investigate the performance of the three proposed test statistics for eigenvalues, eigenvectorsand the test statistic based on eigenprojections a simulation study was designed to assess thetype 1 error (size) and power for inference based on permutations and the parametric boot-strap. Three scenarios were investigated to mimic the applications that are presented in thispaper. Firstly, independent data were generated and changes assessed by using permutations.Secondly, the same data were assessed by using the parametric bootstrap and finally cycliccorrelated data were simulated and assessed using the parametric bootstrap. In each case 500repetitions of every scenario were performed with 500 iterations for permuting or bootstrappingand sample sizes n of 50, 100, 200 and 500. Only results for n= 100 and n= 200 are displayedhere and a level of significance 0.05 was used in each case. All three scenarios were investigatedin two dimensions with only the first and third scenarios investigated in three dimensions andfour dimensions.

5.1. Type 1 error (size)To investigate the probability that we reject hypothesis H0 (that the covariance matrix does notchange throughout time), when H0 is true, response data were simulated that were similar tothe aircraft data by using

MVN{(

34

),(

2:3 0:40:4 0:2

)}: .8/

This assumes that the covariance structure does not vary with time and data are independent.The covariate was taken as time from 1, . . . , g, where g is the number of time points (takenhere to be 25), and the number of data points at each time point was n=g. Smooth PCA wasperformed and the probability of rejecting H0 when H0 is true was calculated for each of the teststatistics using permutations and the parametric bootstrap. The null hypothesis was rejected ata level of significance of 0.05 and the results are displayed in Table 1, sections 1 and 2. For three

Table 1. Type 1 error—two dimensional

Results for thefollowing values of n:

n=100 n=200

1, permutations (independent)Eigenprojection 0.05 0.04Eigenvalue 0.02 0.04Eigenvector 0.06 0.05

2, parametric bootstrap (independent)Eigenprojection 0.01 0.02Eigenvalue 0.01 0.06Eigenvector 0.02 0.04

3, parametric bootstrap (correlated)Eigenprojection 0.01 0.01Eigenvalue 0.04 0.06Eigenvector 0.01 0.01


dimensions and four dimensions, data were simulated from distributions similar to distribution(8) but extended (based on the aircraft data) for three and four variables. The simulations fortype 1 error were performed in the same way as the two-dimensional case and the results aredisplayed in section 1 of Table 2.

To incorporate correlation, the data were generated to be similar to the Loch Leven examplewith two, three and four responses. For example, in two dimensions,

(Y1t

Y2t

)=(

μ1t

μ2t

)+Aδt−1 +ηt .9/

with mean μl for each response l. Results are displayed in section 3 of Table 1 and section 2 ofTable 2. For all results in Tables 1 and 2 it should be noted that the error, assuming a true sizeof 0.05, is approximately 2

√.0:95×0:05=500/=0:02.

Tables 1 and 2 highlight that the type 1 error rate is close to 0.05 for the permutations onindependent data at a sample size of n=200 and across all dimensions. Results for a sample sizeof 500 are not displayed here but all values are similarly close to 0.05. However, as the samplesize decreases to 100 the values decrease slightly. In two dimensions (Table 1), the permutationresults are slightly closer to 0.05 than for the parametric bootstrap for independent data atn=100 and n=200. For the parametric bootstrap in three dimensions and four dimensions theresults are close to 0.05, whereas in two dimensions some values are closer to 0.01. These smallerresults are partly a result of the fact that for small sample sizes there can be an abrupt changein direction of the PC from one time point to the next, instead of a smooth gradual change.This is a result of a large change in the component loadings, resulting in the direction changingabruptly from one time point to the next. In such cases the degrees of freedom were graduallydecreased until the component loadings changed smoothly from one time point to the next. Ifabrupt changes still occurred with df=2:5 then the data were rejected and new data simulated.This method was adopted as an alternative to simply rejecting the data immediately to improvecomputational time. As smoothing increases we are less likely to reject the null hypothesis giventhat the data are generated under the null hypothesis of no change and hence values for size arelikely to be less than 0.05.

Table 2. Type 1 error—three dimensional and four dimensional

Results for Results forthree dimensional four dimensional

n=100 n=200 n=100 n=200

1, permutations (independent)Eigenprojection 0.04 0.05 0.04 0.05Eigenvalue 0.04 0.07 0.03 0.04Eigenvector 0.04 0.05 0.03 0.06

2, parametric bootstrap (correlated)Eigenprojection 0.02 0.05 0.06 0.05Eigenvalue 0.05 0.05 0.03 0.04Eigenvector 0.03 0.05 0.05 0.06


5.2. PowerTo investigate the probability that hypothesis H0 is rejected when H0 is false, multivariate nor-mal data for two responses were simulated with a covariance structure that changes throughouttime. To construct such data, an eigenanalysis was performed on the covariance matrix fromdistribution (8) to obtain a vector λ of two eigenvalues and a 2 × 2 matrix of eigenvectors E.The matrix of eigenvectors E was then premultiplied by rotation matrices of the form(

cos.θt/ −sin.θt/

sin.θt/ cos.θt/

).10/

to produce eigenvector matrices Et that vary smoothly over a grid of g time points. The angleθt was determined by the smooth function

θt =π=6− .π=6/

(t −g=2

g=2

)2

where t is a time point of interest in a grid of g time points. In the cyclic case the angles weredetermined by using the smooth function

π=12− .π=12/ cos(

2πt −g=2

g=2

):

The eigenvalues λ were multiplied by 0:25t to provide a vector of eigenvalues at each time pointλt that increase smoothly over time. For cyclic data this was altered to multiply by

2:5−2:5 cos(

2πt −g=2

g=2

)

to obtain a smooth cyclic change. These functions were chosen to create similar magnitudes ofchange in both the standard and the cyclic cases and g was taken to be 25 in each case.

Mardia et al. (1979) showed that a matrix Y of p response variables and n data points can beconstructed from eigenvectors and PC scores by using

Y = Y +CET .11/

where Y is an n×p matrix containing the mean of each of the p response variables, C is an n×p

matrix of scores and E is the p×p eigenvector matrix. The PC scores C can be generated from

C ∼MVN.0, λD/ .12/

where λD is a p×p diagonal matrix of the eigenvalues. Therefore, at time t, blocks of n=g datapoints, for each response, were generated by using the eigenvalue vector λt and eigenvectormatrix Et corresponding to time t in expressions (11) and (12). This provides n data points,where n=g data points have the same covariance structure.

In the correlated case, data were generated from a multivariate AR(1) process using the errormatrices and A-matrix, of lag 1 coefficients, from the Loch Leven data (9). An eigenanalysis wasthen applied to these data and the eigenvectors and eigenvalues stored. The same procedure asabove was then applied to rotate the eigenvectors and to modify the eigenvalues at each timepoint (with appropriate changes to the functions for cyclic data as described above). To generatenew data, the PC scores C were computed from a multivariate AR(1) process with the A-matrixdiagonal with lag 1 correlations and Σ diagonal with eigenvalues for each time point as above.The remainder of the process was the same as in the independent case.

These methods can be extended to three and four dimensions by applying appropriate rota-


tion matrices (Noll, 1967; Arfken and Weber, 2005). In three dimensions the three matricesdescribing the rotation are

Rx.α/=(1 0 0

0 cos.α/ −sin.α/

0 sin.α/ cos.α/

),

Ry.β/=( cos.β/ 0 −sin.β/

0 1 0sin.β/ 0 cos.β/

),

Rz.γ/=( cos.γ/ −sin.γ/ 0

sin.γ/ cos.γ/ 00 0 1

)

for angles α, β and γ. These matrices can be combined to give the complete rotation R, whereR = RxRyRz, and applied with the above procedure to investigate power in three dimensions.In general, if rotation is in the plane xa and xb (the a–b-plane), then the rotation matrix Rab.α/

has the following elements at position ij:

rii =1, except, raa = rbb = cos.α/;

rij =0, except, rab =−rba =−sin.α/:

Therefore, in four dimensions (e.g. x, y, z and w), the complete rotation can be formed bycombining the rotations for each plane.

This procedure was implemented in two dimensions to investigate power for independent datausing permutations, independent data using a parametric bootstrap and cyclic correlated datausing a parametric bootstrap. Power was investigated for the first and third scenarios in threedimensions and four dimensions and the results for all dimensions are displayed in Tables 3 and4. The same angle was used for rotation in each plane for three dimensions and four dimensions.

Table 3. Power, two dimensional

Results for thefollowing values of n:

n=100 n=200

1, permutations (independent)Eigenprojection 0.95 1Eigenvalue 0.89 1Eigenvector 0.98 1

2, parametric bootstrap (independent)Eigenprojection 1 1Eigenvalue 0.94 1Eigenvector 1 1

3, parametric bootstrap (correlated)Eigenprojection 0.27 0.53Eigenvalue 0.98 1Eigenvector 0.35 0.62


Table 4. Power, three dimensional and four dimensional

Results for Results forthree dimensional four dimensional

n=100 n=200 n=100 n=200

1, permutations (independent)Eigenprojection 0.93 1 0.91 1Eigenvalue 0.92 1 0.95 1Eigenvector 0.95 1 0.93 1

2, parametric bootstrap (correlated)Eigenprojection 0.96 1 0.65 0.96Eigenvalue 1 1 0.41 0.79Eigenvector 0.98 1 0.67 0.97

For all results in Tables 3 and 4 it should be noted that the maximum error, assuming a powerof 0.5, is approximately 2

√.0:5×0:5=500/=0:04.

Reassuringly, the results highlight that the power increases as the sample size increases in allcases. For independent data at all dimensions, the power is indistinguishable from 1 for all teststatistics at n = 200. In two dimensions (Table 3), the permutation and parametric bootstrapresults for independent data are similar for both n=100 and n=200 highlighting that for inde-pendent data either approach appears reasonable with a moderate amount of data. The resultsin two dimensions for the cyclic correlated data (section 3 of Table 3) have lower powers forthe eigenprojection and eigenvector test statistics, in contrast with three dimensions and fourdimensions (Table 4) where the powers are much higher at both sample sizes and close to 1 forn = 200. Even though the underlying changes in variance and direction were set to be of thesame size in all dimensions, the randomly generated data and added dimensions result in largerestimated changes in the direction of the PCs in three dimensions and four dimensions than intwo dimensions. Increasing the size of the change in two dimensions increases the powers forthe eigenprojection and eigenvector.

In most cases the powers for the eigenprojection are slightly smaller than for the eigenvectorand this is consistent with the results in the earlier examples, where the p-values for the eigen-projection appear slightly larger than for the eigenvector. However, the simulation results showthat there is little difference overall between the two test statistics.

It is difficult to compare results between the independent and correlated cases and acrossdimensions since the underlying structure of the data and hence the changes estimated through-out time are different. It should also be noted that there is still an occasional issue here withabrupt changes in the direction of the PCs from one time point to the next. This can occur atsmaller sample sizes and high dimensions. This has been dealt with in the same way as describedfor size.

6. Discussion

Smooth PCA has been applied effectively to investigate whether the covariance structure in anaircraft technology study and in the lake ecosystem at Loch Leven changes throughout time.Inferential methods have been presented and applied for both the contexts of independent errors


throughout time and in the presence of auto- and cross-correlation. Changes in covariances overtime were found for both applications. The eigenprojection and eigenvector test statistics for theecological example were small as a result of the small changes in direction and hence evidence ofchange here was mainly for the variance. However, the figures and the eigenvector test statisticalso suggested changes in direction. This indicates changes in the system over the year as othervariables, such as grazers, impact on the water quality.

Smooth PCA is presented as a graphical and inferential procedure to assess whether covari-ance structures between multiple related response variables change with an explanatory vari-able. Several graphical representations have been presented to explore changes throughout time.Permutations and bootstrapping have been used to provide distributions for eigenvalues andeigenvectors of the PCs, enabling inference to be performed for independent and correlateddata. Shading has been added to the graphics, using these results to indicate where changes overtime are likely to be significant, and three test statistics were proposed to provide p-values forthe null hypothesis that the eigenvalues and/or eigenvectors do not change throughout time.

Simulations for size and power have shown that the three test statistics that were presentedprovide accurate results for testing hypotheses regarding changes in the eigenvectors and eigen-values. Many of the results are very good for sample sizes of 100 or more data points and formoderate changes over time. Since the signs of the loadings of the PCs are not unique care mustbe taken here to ensure that the eigenvectors being compared are all of the same signed directionbefore computing the test statistics and constructing reference bands. The direction of the PCsmust also be tracked to ensure that it does not change abruptly from one time point to the next(e.g. as the result of a large change in the component loadings). This is not an issue if thereare sufficient data. However, for small sample sizes smoothing sometimes must be increased toprevent abrupt changes and this was a feature of the simulations. It was also a feature at largerdimensions for a few of the simulations where the same change in angle in each plane resultedin large changes in direction from one time point to the next.

The local covariance matrices are calculated for a grid of points over the time periods ofinterest and in the applications here this was taken to be a grid of 25 with 4 degrees of freedomfor smoothing. Both of these can be changed to suit the particular application. However, herethe grid size and smoothing parameters worked well to display the smooth changes over timewithout displaying unnecessary variation. The smoothing parameters were chosen on the basisof previous analysis of the data and to allow a reasonable amount of flexibility, and a sensitivityanalysis was performed on the size of the grid for both applications. Taking the grid to be ofsizes between five time points and 40 time points, the p-values differed in a few cases by about0.01. However, all conclusions were unchanged.

Smooth PCA has been applied in the context of time here up to a maximum of four dimen-sions. However, the method can be applied more generally for higher dimensional data and toassess whether or not covariances change as a result of a general covariate. Further examplesare provided as on-line supplementary material.

Software

Code to implement smooth PCA is available in the sm package (Bowman and Azzalini, 2010)for the R system (R Development Core Team, 2009).

Acknowledgement

The authors gratefully acknowledge the Centre for Ecology & Hydrology at Edinburgh forproviding access to the Loch Leven data.


References

Allen, M. R. and Smith, L. A. (1997) Optimal filtering in singular spectrum analysis. Phys. Lett. A, 234, 419–428.Anderson, T. (1963) Asymptotic theory for principal component analysis. Ann. Math. Statist., 34, 122–148.Arfken, G. B. and Weber, H. J. (2005) Mathematical Methods for Physicists. Amsterdam: Elsevier.Boudou, A. and Viguier-Pla, S. (2006) On the proximity between PCA in the frequency domain and usual PCA.

Statistics, 40, 447–464.Bowman, A. and Azzalini, A. (1997) Applied Smoothing Techniques for Data Analysis. Oxford: Clarendon.Bowman, A. and Azzalini, A. (2010) R Package ‘sm’: Nonparametric Smoothing Methods, Version 2.2-4. (Available

from http://www.stats.gla.ac.uk/∼adrian/sm.)Bowman, A. and Foster, P. (1993) Density based exploration of bivariate data. Statist. Comput., 3, 171–177.Brillinger, D. R. (2001) Time Series Data Analysis and Theory, 4th edn. Philadelphia: Society for Industrial and

Applied Mathematics.Cleveland, W. and Devlin, S. (1988) Locally weighted regression: an approach to regression analysis by local

fitting. J. Am. Statist. Ass., 83, 596–610.Cordaus, M. (1992) Bootstrapping moving average models. J. Ital. Statist. Soc., 2, 227–234.Davison, A. C., Hinkley, D. V. and Young, G. A. (2003) Recent developments in bootstrap methodology. Inst.

Math. Sci., 18, 141–157.Efron, B. and Tibshirani, R. (1986) Bootstrap methods for standard errors, confidence intervals and other mea-

sures of statistical accuracy. Statist. Sci., 1, 54–75.European Parliament (2000) Directive 2000/60/EC of the European Parliament, establishing a framework for

community action in the field of water policy. Off. J. Eur. Commts, 327, 1–72.Ferguson, C. A., Bowman, A. W., Scott, E. M. and Carvalho, L. (2007) Model comparison for a complex ecologi-

cal system. J. R. Statist. Soc. A, 170, 691–711.Ferguson, C., Bowman, A., Scott, E. and Carvalho, L. (2009) Multivariate varying-coefficient models for an

ecological system. Environmetrics, 20, 460–476.Ferguson, C., Carvalho, L., Scott, E., Bowman, A. and Kirika, A. (2008) Assessing ecological responses to

environmental change using statistical models. J. Appl. Ecol., 45, 193–203.Flury, B. K. (1987) Two generalizations of the common principal component model. Biometrika, 74, 59–69.Hasselmann, K. (1988) PIPs and POPs: the reduction of complex dynamical systems using principal interaction

and oscillation patterns. J. Geophys. Res., 93, 11015–11021.Jassby, A. and Powell, T. (1990) Detecting changes in ecological time series. Ecology, 71, 2044–2052.Jolliffe, I. (2002) Principal Component Analysis. New York: Springer.Kim, K. Y. and Wu, Q. (1999) A comparison study of EOF techniques: analysis of nonstationary data with

periodic statistics. J. Clim., 12, 185–199.Kollo, T. and Neudecker, H. (1993) Asymptotics of eigenvalues and unit-length eigenvectors of sample variance

and correlation matrics. J. Multiv. Anal., 47, 283–300.Lagerlund, T. D., Sharbrough, F. W. and Busacker, N. E. (2004) Use of principal component analysis in the

frequency domain for mapping electroenchephalographic activities: comparison with phase-encoded fourierspectral analysis. Brain Topog., 17, 73–84.

Mardia, K., Kent, J. T. and Bibby, J. (1979) Multivariate Analysis. New York: Academic Press.Noll, A. M. (1967) A computer technique for displaying n-dimensional hyperobjects. Communs ACM, 10, 469–473.Ombao, H. and Ho, M. R. (2006) Time-dependent frequency domain principal components analysis of multi-

channel non-stationary signals. Computnl Statist. Data Anal., 50, 2339–2360.Politis, D. N. (2003) The impact of bootstrap methods on time series analysis. Statist. Sci., 18, 219–230.Politis, D. N. and Romano, J. P. (1994) The stationary bootstrap. J. Am. Statist. Ass., 89, 1303–1313.Prvan, T. and Bowman, A. (2003) Nonparametric time dependent principal components analysis. Anziam J., 44,

627–643.R Development Core Team (2009) R: a Language and Environment for Statistical Computing. Vienna: R Foun-

dation for Statistical Computing.Salvador, M., Gallizo, J. and Gargallo, P. (2003) A dynamic principal components analysis based on multivariate

matrix normal dynamic linear models. J. Forecast., 22, 457–478.Saviotti, P. (1996) Technological Evolution, Variety and Economy. Cheltenham: Elgar.Schott, J. (1991) Some tests for common principal component subspaces in several groups. Biometrika, 78,

771–777.Schott, J. (1996) Eigenprojections and the equality of latent roots of a correlation matrix. Computnl Statist. Data

Anal., 23, 229–238.Schott, J. (1998) Estimating correlation matrices that have common eigenvectors. Computnl Statist. Data Anal.,

27, 445–459.Schott, J. (2003) Weighted chi-squared tests for partial common principal component subspaces. Biometrika, 90,

411–421.Tyler, A., Hunter, P., Carvalho, L., Codd, G., Elliott, J., Ferguson, C., Hanley, N., Hopkins, D., Maberly, S.,

Mearns, K. and Scott, E. (2009) Strategies for monitoring and managing mass populations of toxic cyanobac-teria in recreational waters: a multi-interdisciplinary approach. Environ. Hlth, 8, suppl. 1, article S11.


Zuur, A., Fryer, R., Jolliffe, I., Dekker, R. and Beukema, J. (2003a) Estimating common trends in multivariatetime series using dynamic factor analysis. Environmetrics, 14, 665–685.

Zuur, A., Tuck, I. and Bailey, N. (2003b) Dynamic factor analysis to estimate common trends in fisheries timeseries. Can. J. Fish. Aquat. Sci., 60, 542–552.

Supporting informationAdditional ‘supporting information’ may be found in the on-line version of this article:

‘Supplementary Material: Smooth prinicipal components for investigating changes in covariances over time’.

Please note: Wiley–Blackwell are not responsible for the content or functionality of any supporting materials suppliedby the authors. Any queries (other than missing material) should be directed to the author for correspondence for thearticle.

smooth principal components for investigating changes in covariances over time

Documents