![Page 1: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/1.jpg)
1–1
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Dynamical Systems for Extreme Eigenspace Computations
Maziar Nikpour
UCL Belgium
![Page 2: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/2.jpg)
1–2
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Co-workers
Iven M. Y. Mareels
Jonathan H. Manton
University of Melbourne, Australia.
Vadym Adamyan
Odessa State University, Ukraine.
Uwe Helmke
University of Wurzberg, Germany.
![Page 3: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/3.jpg)
1–3
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Problem• For Hermitian matrices (A, B), with B > 0;
find the non-trivial solutions (, x) of
with the smallest or largest generalised eigenvalues .
n – size of matrices (A,B)k – no. of desired generalised eigenvalue/eigenvector pairs.
![Page 4: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/4.jpg)
1–4
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Outline
• Introduction• Motivation• Brief history of literature• Penalty function approach• Gradient flow• Convergence• Discrete-time Algorithms• Applications• Conclusions
![Page 5: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/5.jpg)
1–5
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Motivation
• Signal Processing
• Telecommunications
• Control
• Many others…
![Page 6: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/6.jpg)
1–6
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Brief History of Problem
• Numerical Linear Algebra Literature– Methods for general A and B:
• QZ algorithm, Moler and Stewart 1973.
(what MATLAB does when you type ‘eig’)
– Methods for large and sparse A, B.• Trace minimisation method, Sameh & Wisiniewski, 1981.
• Engineering Literature• Methods largely for computing largest/smallest generalised evs
adaptively• Mathew and Reddy 1998 (inflation approach, special case of
approach in this work).
• Strobach, 2000 (tracking algorithms).
![Page 7: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/7.jpg)
1–7
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Brief History of Problem
• Dynamical systems literature– Brockett flow
– Oja
• Above approaches cannot be adapted to the Generalised Eigenvalue problem without manipulating A and/or B.
• Recent paper by Manton et al. presents an approach that can…
![Page 8: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/8.jpg)
1–8
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Penalty Function Approach• The minimisation of the following cost can lead to
algorithms for computing extreme generalised evs.
![Page 9: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/9.jpg)
1–9
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Dynamical Systems for Numerical Computations
Gradient descent like flows on a cost function.
Discretisation of flows.
Efficient numerical algorithms.
![Page 10: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/10.jpg)
1–10
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Examples
• Power flow:
• Oja subspace flow:
• Brockett flow:
![Page 11: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/11.jpg)
1–11
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Contributions• Gradient flow on f(A, B)
• Discretisation of Gradient Flow– Steepest Descent
– Conjugate Gradient
• Stochastic minor/principal component tracking algorithms
• The case B = I, and Z real has already been treated.
(see Manton et al. 2003).
Extending the domain to the complex matrices complicates the analysis substantially…
Allowing B to be any p.d. matrix expands the range of applications…
![Page 12: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/12.jpg)
1–12
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Gradient Flow
• Main Result: For almost all initial conditions, solutions of
converge to a single point in the stable invariant set of the flow.
![Page 13: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/13.jpg)
1–13
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Gradient Flow
• The stable invariant set is:
![Page 14: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/14.jpg)
1–14
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Critical Points of f(A, B)
• Hessian of f(A, B) is degenerate at critical points,
N.B. • Proposition:
![Page 15: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/15.jpg)
1–15
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Stability analysis of critical points
• Linear stability analysis will not suffice.• Use center manifold theorem at each c.p.• Proposition:
Why?
Nullspace of hessian of cost func. = Tangent space of critical subman.
![Page 16: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/16.jpg)
1–16
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Stability analysis of critical pointsReduction principle of dynamical systems
stable
unstable
center
![Page 17: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/17.jpg)
1–17
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
• Main result follows….
• Proposition: level sets are compact => flow converges to one of the critical components.
• Center manifold thm. + reduction principle => converge to a single point on a critical component.
• Converges to stable invariant set for an open dense set of initial conditions.
Stability analysis of critical points
![Page 18: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/18.jpg)
1–18
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Remarks• Conditions used in proof => f(A, B) is a Morse-Bott function
=> solutions converge to a single point instead of a set (see Helmke & Moore, 1994).
• Also f(A, B) is a real analytic function (Cn x k considered as a real vector space) => convergence to a single point (Lojasiewicz, 1984).
![Page 19: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/19.jpg)
1–19
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Further Remarks• Generalised eigenvectors not unique but
convergence to particular g.evs can be achieved by the following flow in reduced dimensions:
where trunc{X} denotes X with imaginary components of diagonal set to 0.
Flow converges to an element of critical component with real diagonal elements.
![Page 20: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/20.jpg)
1–20
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Systems of Flows
• Consider the system of cost functions:
![Page 21: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/21.jpg)
1–21
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Systems of Flows
• System of partial gradient descent flows allows the possibility to add or take away components without affecting the computation of others
• Proposition: Z(t) converges to smallest generalised eigenvalues for a generic initial condition.
![Page 22: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/22.jpg)
1–22
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Discrete-time algorithms
• Since flow evolves on a Euclidean space – discretisation is not complicated:
• Steepest descent:
• Conjugate gradient
![Page 23: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/23.jpg)
1–23
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Discrete-time algorithms
• Can solve the Hermitian definite GEVP without any factorisation or manipulation of A or B.
• Only matrix – small matrix multiplications are required.
• Suitable for cases where A and B are large and sparse.
• Conjugate gradient algorithm – superlinear convergence but no increase in order of computational complexity.
• Complexity O(n2k).
• Exact line search can be performed.
![Page 24: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/24.jpg)
1–24
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
![Page 25: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/25.jpg)
1–25
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Discrete-time algorithms
• Tracking algorithm:
• O(nk2) complexity when Rnn = I.
- signal plus noise model
![Page 26: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/26.jpg)
1–26
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
![Page 27: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/27.jpg)
1–27
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Conclusion
• Proposing and deriving convergence theory of a gradient flow for solving GEVP.
• Modular system of flows.
• Discretisation: CG and SD algorithms.
• Application to Minor component tracking.
![Page 28: Workshop on Dynamical Systems and Computation 1–1 Dynamical Systems for Extreme Eigenspace Computations Maziar Nikpour UCL Belgium](https://reader036.vdocuments.mx/reader036/viewer/2022062720/56649f155503460f94c2a515/html5/thumbnails/28.jpg)
1–28
Wor
ksh
op o
n D
ynam
ical
Sys
tem
s an
d C
ompu
tatio
n
Questions