eigenproblems ii:...
TRANSCRIPT
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Eigenproblems II: Computation
CS 205A:Mathematical Methods for Robotics, Vision, and Graphics
Justin Solomon
CS 205A: Mathematical Methods Eigenproblems II: Computation 1 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Setup
A ∈ Rn×n symmetric
~x1, . . . , ~xn ∈ Rn eigenvectors
|λ1| ≥ |λ2| ≥ · · · ≥ |λn| eigenvalues
Review (Spectral Theorem):What do we know about the eigenvectors?
CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Setup
A ∈ Rn×n symmetric
~x1, . . . , ~xn ∈ Rn eigenvectors
|λ1| ≥ |λ2| ≥ · · · ≥ |λn| eigenvalues
Review (Spectral Theorem):What do we know about the eigenvectors?
CS 205A: Mathematical Methods Eigenproblems II: Computation 2 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Usual Trick
~v ∈ Rn
⇓~v = c1~x1 + · · · + cn~xn
CS 205A: Mathematical Methods Eigenproblems II: Computation 3 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Observation
Ak~v =
λk1
(c1~x1 +
(λ2λ1
)kc2~x2 + · · · +
(λnλ1
)kcn~xn
)
CS 205A: Mathematical Methods Eigenproblems II: Computation 4 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
For Large k
Ak~v ≈ λk1c1~x1
(assuming |λ2| < |λ1|and c1 6= 0)
CS 205A: Mathematical Methods Eigenproblems II: Computation 5 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Power Iteration
~vk = A~vk−1
Question:What if |λ1| > 1?
CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Power Iteration
~vk = A~vk−1
Question:What if |λ1| > 1?
CS 205A: Mathematical Methods Eigenproblems II: Computation 6 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Normalized Power Iteration
~wk = A~vk−1
~vk =~wk‖~wk‖
Question: Which norm?
CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Normalized Power Iteration
~wk = A~vk−1
~vk =~wk‖~wk‖
Question: Which norm?
CS 205A: Mathematical Methods Eigenproblems II: Computation 7 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Eigenvalues of Inverse Matrix
A~v = λ~v =⇒ A−1~v =1
λ~v
Question:What is the largest-magnitude eigenvalue?
CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Eigenvalues of Inverse Matrix
A~v = λ~v =⇒ A−1~v =1
λ~v
Question:What is the largest-magnitude eigenvalue?
CS 205A: Mathematical Methods Eigenproblems II: Computation 8 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Inverse Iteration
~wk = A−1~vk−1
~vk =~wk‖~wk‖
Question: How to make faster?
CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Inverse Iteration
~wk = A−1~vk−1
~vk =~wk‖~wk‖
Question: How to make faster?
CS 205A: Mathematical Methods Eigenproblems II: Computation 9 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Inverse Iteration with LU
Solve L~yk = ~vk−1
Solve U ~wk = ~yk
Normalize ~vk =~wk‖~wk‖
CS 205A: Mathematical Methods Eigenproblems II: Computation 10 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Eigenvalues of Shifted Matrix
A~v = λ~v =⇒ (A− σI)~v = (λ− σ)~v
CS 205A: Mathematical Methods Eigenproblems II: Computation 11 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Shifted Inverse Iteration
To find eigenvalue closest to σ:
~vk+1 =(A− σI)−1~vk‖(A− σI)−1~vk‖
CS 205A: Mathematical Methods Eigenproblems II: Computation 12 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Heuristic: Convergence Rate
Ak~v =
λk1
(c1~x1 +
(λ2λ1
)kc2~x2 + · · · +
(λnλ1
)kcn~xn
)
CS 205A: Mathematical Methods Eigenproblems II: Computation 13 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Strategy for Better Convergence
Find σ with∣∣∣∣λ2 − σλ1 − σ
∣∣∣∣ < ∣∣∣∣λ2λ1∣∣∣∣
CS 205A: Mathematical Methods Eigenproblems II: Computation 14 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Least-Squares Approximation
If ~v0 is approximately an eigenvector:
argminλ‖A~v0 − λ~v0‖22 =
~v>0 A~v0‖~v0‖22
CS 205A: Mathematical Methods Eigenproblems II: Computation 15 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Rayleigh Quotient Iteration
~wk = (A− σkI)−1~vk−1
~vk =~wk‖~wk‖
σk+1 =~v>k A~vk‖~vk‖22
Efficiency per iteration vs. number of iterations?
CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Rayleigh Quotient Iteration
~wk = (A− σkI)−1~vk−1
~vk =~wk‖~wk‖
σk+1 =~v>k A~vk‖~vk‖22
Efficiency per iteration vs. number of iterations?
CS 205A: Mathematical Methods Eigenproblems II: Computation 16 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Unlikely Failure Mode for Iteration
What is ~v0?
What happens when~v0 · ~x1 = 0?
CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Unlikely Failure Mode for Iteration
What is ~v0?
What happens when~v0 · ~x1 = 0?
CS 205A: Mathematical Methods Eigenproblems II: Computation 17 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Bug or Feature?
1. Compute ~x0 via power iteration.
2. Project ~x0 out of ~v0.
3. Compute ~x1 via power iteration.
4. Project span {~x0, ~x1} out of ~v0.
5. · · ·
Assumption: A is symmetric.
CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Bug or Feature?
1. Compute ~x0 via power iteration.
2. Project ~x0 out of ~v0.
3. Compute ~x1 via power iteration.
4. Project span {~x0, ~x1} out of ~v0.
5. · · ·
Assumption: A is symmetric.
CS 205A: Mathematical Methods Eigenproblems II: Computation 18 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Avoiding Numerical Drift
Do power iteration on P>AP where P projects
out known eigenvectors.
DeflationModify A so that power iteration reveals an
eigenvector you have not yet computed.
CS 205A: Mathematical Methods Eigenproblems II: Computation 19 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Similarity Transformations
Similar matricesTwo matrices A and B are similar if there exists
T with B = T−1AT.
PropositionSimilar matrices have the same eigenvalues.
CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Similarity Transformations
Similar matricesTwo matrices A and B are similar if there exists
T with B = T−1AT.
PropositionSimilar matrices have the same eigenvalues.
CS 205A: Mathematical Methods Eigenproblems II: Computation 20 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Householder Asymmetric Deflation
H~x1 = ~e1
=⇒ HAH>~e1 = HAH~e1 by symmetry
= HA~x1 since H2 = I
= λ1H~x1
= λ1~e1
CS 205A: Mathematical Methods Eigenproblems II: Computation 21 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Householder Asymmetric Deflation
HAH> =
(λ1 ~b
>
~0 B
)Similarity transform of A =⇒ same eigenvalues.
Do power iteration on B.
CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Householder Asymmetric Deflation
HAH> =
(λ1 ~b
>
~0 B
)Similarity transform of A =⇒ same eigenvalues.
Do power iteration on B.
CS 205A: Mathematical Methods Eigenproblems II: Computation 22 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
<warning>
Justin’s favorite
algorithm. Ever.
</warning>
CS 205A: Mathematical Methods Eigenproblems II: Computation 23 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Conjugation without Inversion
Q−1 = Q>
=⇒ Q−1AQ = Q>AQ
But which Q?Should involve matrix structure but be easy to
compute.
CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Conjugation without Inversion
Q−1 = Q>
=⇒ Q−1AQ = Q>AQ
But which Q?Should involve matrix structure but be easy to
compute.
CS 205A: Mathematical Methods Eigenproblems II: Computation 24 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Experiment
A = QR
Q−1AQ = ?
CS 205A: Mathematical Methods Eigenproblems II: Computation 25 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
QR Iteration
A1 = A
Factor Ak = QkRk
Multiply Ak+1 = RkQk
CS 205A: Mathematical Methods Eigenproblems II: Computation 26 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Commutativity
Lemma
Take A,B ∈ Rn×n. Suppose that the eigenvectors ofA span Rn and have distinct eigenvalues. Then,AB = BA if and only if A and B have the same set ofeigenvectors (with possibly different eigenvalues).
CS 205A: Mathematical Methods Eigenproblems II: Computation 27 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
If QR Iteration Converges
A∞ = Q∞R∞ = R∞Q∞
(Convergence proof in book.)
CS 205A: Mathematical Methods Eigenproblems II: Computation 28 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Starting Point
(A + δA)(~x + δ~x) = (λ + δλ)(~x + δ~x)
What are the independent and dependentvariables?
Approximation:
Aδ~x + δA · ~x ≈ λδ~x + δλ · ~x
CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Starting Point
(A + δA)(~x + δ~x) = (λ + δλ)(~x + δ~x)
What are the independent and dependentvariables?
Approximation:
Aδ~x + δA · ~x ≈ λδ~x + δλ · ~x
CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Starting Point
(A + δA)(~x + δ~x) = (λ + δλ)(~x + δ~x)
What are the independent and dependentvariables?
Approximation:
Aδ~x + δA · ~x ≈ λδ~x + δλ · ~x
CS 205A: Mathematical Methods Eigenproblems II: Computation 29 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Trick: Left Eigenvector
A~x = λ~x, ~x 6= ~0 =⇒∃~y 6= ~0 such that A>~y = λ~y
CS 205A: Mathematical Methods Eigenproblems II: Computation 30 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Change in Eigenvalue
|δλ| / ‖δA‖2|~y · ~x|
What about symmetric A?
Next
CS 205A: Mathematical Methods Eigenproblems II: Computation 31 / 31
Power Iteration Other Eigenvalues Multiple Eigenvalues QR Iteration Conditioning
Change in Eigenvalue
|δλ| / ‖δA‖2|~y · ~x|
What about symmetric A?
Next
CS 205A: Mathematical Methods Eigenproblems II: Computation 31 / 31