Spatiotemporal Pattern Extractionby Spectral Analysis of Vector-valued Observables

Dimitris Giannakis

Center for Atmosphere Ocean ScienceCourant Institute of Mathematical Sciences

New York University

Geometry and Topology of DataICERM, 12/11/2017

Collaborators: Abbas Ourmazd, Joanna Slawinska, Jane Zhao

10−4 m 10−1 m 102 m

105 m 107 m 1025 m

Setting

~F (xn)

~F

AX

x0

xn = Φnτ x0

~F (A)

HY

Y

~F (x0)

• Dynamical flow Φt : X 7→ X on a manifold with an ergodic invariant measureµ, supported on a compact set A ⊆ X

• Compact spatial domain Y , equipped with a finite measure ρ

• Continuous, vector-valued observation map ~F : X 7→ C(Y )

Objective. Given time-ordered measurements ~F (x0), ~F (x1), . . ., with

xn = Φnτ (x0), decompose ~F into spatiotemporal patterns ~φj : X 7→ C(Y ),

~F =∑j

cj ~φj , cj ∈ R

Separable space-time patterns

A widely used approach is to recover temporal patterns through theeigenfunctions of an operator T on HA = L2(A, µ),

Tϕk = λkϕk , ϕk ∈ HA

Many choices for T , including:

• Covariance operators (POD, PCA, SSA, . . . )• Heat operators (Laplacian eigenmaps, diffusion maps, . . . )• Koopman operators (DMD, EDMD, . . . )

Spatial patterns ψk in HY = L2(Y , ν) can then be obtained by pointwiseprojection of the observation map onto the temporal patterns:

ψk(y) = 〈ϕk ,Fy 〉HA , Fy : x ∈ A 7→ ~F (x)(y)

This is equivalent to treating ~F as a function in the tensor product spaceHA ⊗ HY , and performing the decomposition

~F ≈ Fl =l∑

k=0

ϕk ⊗ ψk

• In the presence of symmetries and/or spatiotemporal intermittency, puretensor product patterns, ϕk ⊗ ψk , may suffer from poor descriptive efficiencyand physical interpretability (Aubry et al. 1993)

Hilbert spaces of observables

We have the Hilbert space isomorphisms

H ' HA ⊗ HY ' HM ,

HA = L2(A, µ), HY = L2(Y , ν), H = L2(A, µ;HY ), HM = L2(M, ρ),

with M = A× Y ⊆ Ω = X × Y , ρ = µ× ν

As a result, the observation map ~F can be equivalently thought of as:

1 A vector-valued observable ~F : A 7→ HY in H

2 An element of the tensor product space HX ⊗ HY , i.e., ~F =∑

jk cjkeAj ⊗ eYk

for bases eAj of HA and eYk of HY

3 A scalar-valued observable F : M 7→ R in HM , s.t. F (x , y) = ~F (x)(y)

Given x ∈ A, the function t 7→ ~F (Φt(x)) corresponds to a spatiotemporalpattern

Vector-valued spectral analysis (VSA) framework

We decompose ~F using the eigenfunctions of a compact operator PQ : H 7→ H,

PQ~φj = λj

~φj , ~F ≈ ~Fl =l∑

j=0

cj ~φj , cj ∈ R

This operator is associated with an operator-valued kernel (Micchelli & Pontil

2005, Caponnetto et al. 2008,Carmeli et al. 2010), constructed from delay-coordinatemapped data with Q delays

PQ~f =

∫A

LQ(·, x)~f (x) dµ(x), LQ : X × X 7→ L(HY )

Desirable properties include:

• Ability to recover patterns without a tensor product structure

• Symmetry group actions are naturally factored out

• Asymptotic commutation property with Koopman operators allows to identifyintrinsic dynamical timescales

Operator-valued kernel construction

For the purposes of this work:

1 A scalar-valued kernel on Ω = X × Y will be a continuous function

k : Ω ×Ω 7→ R+,

bounded above and away from zero on compact sets

2 An operator-valued kernel on X will be a continuous function

κ : X × X 7→ L(HY )

Associated with k and κ are kernel integral operators K : HM 7→ HM andK : H 7→ H, respectively, where

Kf =

∫M

k(·, ω)f (ω) dρ(ω), K~f =

∫A

κ(·, x)~f (x) dµ(x)

We can assign k to the operator-valued kernel κ, where

κ(x , x ′) = Kxx′ , Kxx′g(y) =

∫Y

k((x , y), (x ′, y ′))g(y ′) dν(y ′)

Kernels from delay-coordinate maps (G. & Majda 2012; Berry et al. 2013; G. 2017; Das & G. 2017)

1 Start from a pseudometric dQ : Ω ×Ω 7→ R0, s.t.,

d2Q((x , y), (x ′, y ′)) =

1

Q

Q−1∑q=0

|F (Φ−qτ (x), y)− F (Φ−qτ (x ′), y ′)|2.

2 Choose a continuous shape function h : R0 7→ [0, 1], and define the kernel

kQ : Ω ×Ω 7→ R+, kQ(ω, ω′) = h(dQ(ω, ω′));

here, h(s) = e−s2/ε, with ε > 0

3 Normalize kQ to obtain a continuous Markov kernel pQ : Ω ×Ω 7→ R+ usingthe procedure introduced in the diffusion maps algorithm (Coifman & Lafon 2006):

pQ(ω, ω′) =kQ(ω, ω′)

lQ(ω)rQ(ω′), rQ =

∫M

kQ(·, ω) dρ(ω), lQ =

∫M

kQ(·, ω)

rQ(ω)dρ(ω)

The kernel pQ induces the compact operators PQ : HM 7→ HM andPQ : H 7→ H, s.t.

PQ f =

∫M

pQ(·, ω)f (ω) dρ(ω), PQ~f =

∫A

LQ(·, x)~f (x) dµ(x)

where LQ : X × X 7→ L(HY ) is the operator-valued kernel associated with pQ

Vector-valued eigenfunctions

• Identify spatiotemporal patterns, t 7→ ~φj(Φt(x)), through the eigenfunctions of

PQ :PQ

~φj = λj~φj , ~φj ∈ H, 1 = λ0 > λ1 ≥ λ2 ≥ · · ·

• Expand the observation map ~F in the ~φj eigenbasis of H, i.e.,

~F =∞∑j=0

cj ~φj , cj = 〈~φ′j , ~F 〉H ,

where ~φ′j are eigenfunctions of P∗Q , satisfying 〈~φ′j , ~φk〉H = δjk

• Operationally, we obtain (λj , ~φj) through the eigenvalue problem for PQ ,

PQφj = λjφj , φj ∈ H, ~φj(x)(y) = φj((x , y))

Remark. The ~φj are not restricted to a pure tensor product form, ϕj ⊗ ψj , withϕj ∈ HA and ψj ∈ HY

Bundle structure of spatiotemporal data

• The kernel kQ can be expressed as a pullback of a kernel κQ on RQ , the spaceof delay-coordinate sequences with Q delays,

kQ(ω, ω′) = kQ(FQ(ω),FQ(ω′)),

FQ(ω) = (F (ω),F (ω−1), . . . ,F (ω−Q+1)), ω = (x , y), ωq = (Φqτ (x), y)

• Defining BQ = FQ(Ω) and πQ : Ω 7→ BQ s.t. πQ(ω) = FQ(ω), the triplet(Ω,BQ , πQ) is a topological bundle, with total space Ω, base space BQ , andprojection map πQ

• This partitions Ω into equivalence classes, [·]Q , s.t. ω′ ∈ [ω]Q ifπQ(ω) = πQ(ω′)

• Every function in the closed subspace

HQ = ranPQ = spanφj : λj > 0 ⊆ HM ,

is a pullback of a function in L2(JQ , αQ), with JQ = πQ(M) and αQ = πQ(ρQ),i.e., it is ρ-a.e. constant on the [·]Q equivalence classes

• HQ is not necessarily expressible as a tensor product of HA and HY subspaces.

Limit of no delays

If no delays are performed (Q = 1), and M is connected, then J1 = π1(M) is aclosed interval

• The eigenfunctions φj are pullbacks of orthogonal functions ηj on J with respectto the L2 inner product associated with the pushforward measure α1 = π1∗ρ,

φj(ω) = ηj(π1(ω)) = ηj(F (ω))

• In particular, the φj are constant on the level sets of the obsevation map F

In a number of cases (e.g., α1 has a C 2 density wrt. Lebesgue measure, and thekernel bandwidth ε is small), η1 will be monotonic

• In such cases, even the one-term expansion F ≈ F1 = c1φ1 recovers thequalitative features of the input signal

Spatial symmetries

An important example with nontrivial [·]Q equivalence classes is that of PDEmodels with equivariant dynamics under the action of a group G on thespatial domain Y

• Suppose that X is a subset of HY (e.g., an inertial manifold of a dissipativePDE system), and there is a group action Γ g

Y : Y 7→ Y , g ∈ G , satisfying

Φt Γ gX = Γ g

X Φt , Γ g

X (x) = x Γ g−1Y

• Then, defining Γ gΩ = Γ g

X ⊗ ΓgY , the following diagram commutes:

Ω Ω

BQ

ΓgΩ

πQπQ

Spatial symmetries

Under the previous assumptions:

1 For every ω ∈ Ω, the G -orbit ΓΩ(ω) = Γ gΩ(ω) | g ∈ G lies in [ω]Q

2 Moreover, the pseudometric dQ has the invariance property

dQ(Γ gΩ(ω), Γ g′

Ω (ω′)) = dQ(ω, ω′),

for all ω, ω′ ∈ Ω and g , g ′ ∈ G

If, in addition, Γ gΩ preserves null sets with respect to ρ, then it induces a

representation of G on HM , with representatives

RgM : HM 7→ HM , Rg

M f = f Γ gΩ

Theorem. The operators PQ and RgM satisfy [PQ ,R

gM ] = 0 and PQR

gM = PQ for

all g ∈ G . As a result, every eigenspace Wj of PQ at nonzero eigenvalue is afinite-dimensional (by compactness of PQ), trivial representation space of G ,i.e., Rg

M f = f for every f ∈Wj .

Remark. In PCA-type decompositions, ϕj ⊗ ψj , the spatial (ψj) and temporal(ϕj) patterns also lie in G representation spaces, but the representations arenot necessarily trivial

Correspondence with Koopman operators

• Consider the unitary group of Koopman operators U t : HA 7→ HA, t ∈ R,acting on scalar-valued observables in HA by composition with the flow map,

U t f = f Φt

• A distinguished class of observables in HA is that of Koopman eigenfunctions,

U tzj = e iωj tzj , 〈zj , zk〉HA = δjk , ωj ∈ R

• This leads to the U t-invariant decomposition

HA = D ⊕D⊥, D = spanzj

• Because the system is ergodic, the eigenspaces of U t are all one-dimensional

• Similarly, we can define a group of unitary Koopman operators U t : HM 7→ HM ,with

U t f = (U t ⊗ IHY )f = f (Φt ⊗ IY )

• In this case, we have the U t-invariant decomposition

HM = D ⊕ D⊥, D = D ⊗ HY ,

but the eigenspaces of U t , spanzj ⊗ HY , are infinite-dimensional

Correspondence with Koopman operators

In the limit of infinitely many delays, the following hold (Das & G., 2017):

1 d∞ = limQ→∞ dQ is well-defined as a function in HM ⊗ HM

2 d∞ lies in D ⊗ D3 d∞ is invariant under U t ⊗ U t

Therefore, we can define a compact operator P∞ : HM 7→ HM , and ranP∞ ⊆ D

Theorem. The operators P∞ and U t commute for all t ∈ R. As a result, theyare simultaneously diagonalizable on the finite-dimensional eigenspaces of P∞.

Corollary. The eigenspaces Wj of P∞ corresponding to nonzero eigenvalue λj

have the form Wj = spanzj ⊗ Vj , where zj ∈ HA is an eigenfunction of U t

and Vj a finite-dimensional subspace of HY

Note also the following:

1 In the presence of symmetries, P∞, U t , and RgM are mutually commuting

operators

2 In Dynamic Mode Decomposition (DMD) (Schmidt & Henningson 2008, Rowley et al.

2009) and related techniques, one assigns a single spatial patternψj(y) = 〈zj ,Fy 〉HA (Koopman mode) to a given Koopman eigenfunction; here,the number of such patterns is equal to dimVj

Data-driven approximation

• In many cases of interest, the invariant set A is a non-smooth subset of X ofzero Lebesgue measure (e.g., a fractal attractor)

• Moreover, in realistic experimental environments, the sampled dynamical statesdo not lie exacty on A

Assumptions for data-driven approximation

1 The measure µ is physical; that is, there exists a set Bµ ⊆ X , of positiveLebesgue measure, such that for every f ∈ C(X ) and x ∈ Bµ,

limNX→∞

1

NX

NX∑n=0

f (Φnτ (x)) =

∫A

f dµ (1)

2 There exists a compact, forward invariant set U ⊆ X , of positive Lebesguemeasure, s.t. A ⊆ U ⊆ Bµ

3 We make measurements on Y at a sequence of points y0, y1, . . . such that ananalog of (1) holds for f ∈ C(Y )

4 Measurements F (xn, yr ), xn = Φnτ (x0), are available along an (unknown) orbitstarting at x0 ∈ Bµ

Data-driven approximation

• As a data-driven analog of HM = L2(M, ρ), we employ the N-dimensional,N = NXNY , Hilbert space HΩ,N = L2(M, ρN) associated with the samplingmeasure,

ρN =1

N

N−1∑j=0

δωj , ωj = (xnj , yrj ), 0 ≤ nj ≤ NX − 1, 0 ≤ rj ≤ NY − 1

• On this space, PQ is approximated by PQ,N : HΩ,N 7→ HΩ,N , where

PQ,N f =

∫Ω

pQ,N(·, ω)f (ω) dρN(ω) =1

N

N−1∑j=0

pQ,N(·, ωj)f (ωj),

pQ,N(ω, ω′) =kQ(ω, ω′)

lQ,N(ω′)rQ,N(ω),

rQ,N =

∫Ω

kQ(·, ω) dρN(ω), lQ,N =

∫Ω

kQ(·, ω)

rQ,N(ω)dρN(ω)

• The eigenvalue problem for PQ,N is equivalent to an N × N matrix eigenvalueproblem

Spectral convergence

• One issue with establishing convergence (pointwise, in operator norm, etc) ofPQ,N to PQ is that, as defined, these operators act on different spaces (HΩ,Nand HM , respectively)

• We thus examine the analogous operators PQ,N and PQ on C(V),V = U × Y ⊆ Ω, defined using the same (continuous) kernels as PQ,N and PQ ,respectively.

Theorem. For every nonzero eigenvalue λj of PQ and correspondingeigenfunction φj ∈ HM :

1 The sequence of eigenvalues λj,N of PQ,N , N ≥ j − 1, converges to λj

2 There exist eigenfunctions φj,N ∈ HΩ,N such that φj,N ∈ C(V) convergesuniformly to φj ∈ C(V), where

φj,N =1

λj,N

∫Ω

pQ,N(·, ω)φj,N(ω) dρN(ω), φj =

∫M

pQ(·, ω)φj(ω) dρ(ω)

Proof. Establish that (i) PQ is compact, and (ii) PQ,N converges compactly toPQ . The claim then follows from spectral approximation results for compactoperators (Von Luxburg et al 2008; Chatelin 2011).

Application to the Kuramoto-Sivashinksy model

• The Kuramoto-Sivashinsky (KS) model is a prototype dissipative PDE modelexhibiting complex spatiotemporal dynamics, while having a number of usefulknown properties such as inertial manifolds (Foias et al. 1986) and symmetries(Kevrekidis et al. 1990, Cvitanovic et al. 2009)

• The governing equation for the real-valued scalar field u(t, ·) : Y 7→ R, t ≥ 0,Y = [0, L], is given by

u = −u∇u + ∆u −∆2u, u ∈ HY = L2(Y , Leb),

subject to periodic boundary conditions

• The domain size parameter L controls the dynamical complexity of the system;here, we apply VSA to data generated by the KS model at the chaotic regimesL = 22 and L = 94

• For our purposes, the state space manifold X ⊆ HY will be an inertial manifoldof the KS system

KS patterns, L = 22

At a small number of delays, Q = 15, the recovered eigenfunctions areapproximately constant on the level sets of the input signal

KS patterns, L = 22

• At a larger number of delays Q = 500, the leading vector-valued eigenfunctionscapture O(2) families of unstable equilibria (wavenumber L/2 structures), andsmaller-scale traveling waves embedded in those structures

• In contrast, NLSA (G. & Majda 2012), a scalar-valued kernel technique alsoutilizing delays, requires several modes to capture these families

KS patterns, L = 94

Conclusions

Kernel algorithms operating on spaces of vector-valued observables have anumber of useful properties for spatiotemporal pattern extraction, including theability to:

• Recover patterns with a non-separable structure in the spatial and temporaldegrees of freedom

• Quotient out symmetries

• Recover intrinsic timescales associated with the point spectrum of theKoopman operator of the dynamical systems

Physical measures allow spectral convergence of data-driven approximations ofsuch operators for ergodic dynamical systems with non-smooth invariant sets

Ongoing and future work includes applications in atmosphere ocean scienceand extensions of VSA to skew-product dynamical systems

References

• Giannakis, D., J. Slawinska, A. Ourmazd, Z. Zhao (2017). Vector-ValuedSpectral Analysis of Space-Time Data. Proceedings of the NIPS 2017 TimeSeries Workshop.

• Giannakis, D., A. Ourmazd, J. Slawinska, Z. Zhao (2017). Spatiotemporalpattern extraction by spectral analysis of vector-valued observables. Submitted.arXiv: 1711.02798.

• Das, S., and D. Giannakis (2017). Delay-coordinate maps and the spectra ofKoopman operators. arXiv:1706.08544

• Giannakis, D. (2017). Data-driven spectral decomposition and forecasting ofergodic dynamical systems. Applied and Computational Harmonic Analysis,doi:10.1016/j.acha.2017.09.001

Research supported by DARPA grant HR0011-16-C-0116, NSF grantDMS-1521775, ONR grant N00014-14-1-0150, and ONR YIP grantN00014-16-1-2649