chapter 3
DESCRIPTION
school workTRANSCRIPT
Chapter 3 Iterative methods for Linear systemWhy do we need to solve big linear systems?
E.g. 1 Spectral method: Express solution as : u(x) =∑ni=1 aiφi(x). To look for ai’s We need to solve system
of linear equation.
E.g. 2 Finite difference method : Consider u”(x) = f(x) where 0 < x < 1 and u(0) = a0, u(1) = a1.
In calculus, we know
g(x+ h) ≈ g(x) + hg′(x) +h2
2g”(x)
+g(x− h) ≈ g(x)− hg′(x) +h2
2g”(x)
⇒ g(x+ h) + g(x− h) ≈ 2g(x) + h2g”(x)
∴
f”(x) ≈ g(x+ h)− 2g(x) + g(x− h)
h2
Now, partition [0, 1] into xi = ih; h = 1n+1 .
Then, the differential equation can be approximated by
ui+1 − 2ui + ui−1h2
= f(xi), i = 1, 2, ..., n
⇔ A
u1u2...un
=
f(x1)− u0f(x2)
...f(xn)− u1
← system oflinear system
where
A =1
h2
2 −1−1 2 −1
. . . . . . . . .−1 2 −1
−1 2
How to solve big linear system?
From linear algebra, we learnt gaussian elimination.a11 a12 · · · a1na21 a22 · · · a2n...
. . . . . ....
an1 an2 · · · ann
x1x2...xn
=
b1b2...bn
→ by elementaryrow operation
c11 c12 · · · c1n
c22 · · · a2n. . .
...cnn
x1x2...xn
=
b′1b′2...b′n
→Upper triangular matrix
Solved bybackward substitution
Computational cost : O(n3) [Check it if interested]
From linear algebra, we also learnt LU factorization.Decompose a matrix A into A = LU.Then solve the equation by:
A~x = ~b⇔ L(u~x) = ~b
1
Let ~y = u(~x). Solve for L~y = ~b first (backward substitution). Then solve for u~x = ~y(easy).
For A =symmetric positive definite (~xTA~x > 0 for all ~x), then A = LLT . Decomposition can be done by Choleskydecomposition : (Numerical analysis).
Computational cost : O(n3)
Goal :Develop iterative method : Find a sequence ~x0, ~x1, ~x2, ... such that ~xR → ~x = solution as k → ∞ (Stop when theerror is small enough).
Splitting method for general linear systems
Consider the system A~x = ~f ; A ∈Mn×n(R) (n is big )
We can split A as follows:A = M + (A−M) = M − (M −A) = N − P
Solving A~x = ~f is equivalent to solving:
(N − P )~x = f ⇔ N~x = P~x+ ~f
We can develop an iterative scheme as follows
N~xn+1 = P~xn + ~f
to get a sequence {~xn}∞n=1 .
It can be shown that if {~xn} converges, it converges to the solution of A~x = f .
Many different choices of the splitting !!
Goal :N is simple to take inverse ( such as diagonal ).
Splitting choice 1 : Jacobi Method
Split A as A = D + (A−D), where D contains the diagonal entries of A only.Then A~x = ~f :
D~xk+1 + (A−D)~xk = ~f
⇔ D~xk+1 = (D −A)~xk + ~f
⇔ ~xk+1 = D−1(D −A)~xk +D−1 ~f
This is equivalent to solving : a11x
k+11 + a12x
k2 + · · ·+ a1nx
kn = f1
a21xk1 + a22x
k+12 + · · ·+ a2nx
kn = f2
...an1x
k1 + an2x
k2 + · · ·+ annx
k+1n = fn
for xk+11
for xk+12...
for xk+1n
Example : Consider 5 −2 3−3 9 12 −1 −7
x1x2x3
=
−123
2
Then:
~xk+1 =
5 0 00 9 00 0 −7
−1 0 2 −33 0 −1−2 1 0
~xk +
5 0 00 9 00 0 −7
−123
Start with ~x0 =
000
. The sequence converges in 7 iteration to ~x7 =
−0.1860.331−0.423
.
Splitting Choice 2 : Gauss Seidel Method
Split A as :A = L+D + U
Develop an iterative scheme as :L~xk+1 +D~xk+1 + U~xk = ~f
This is equivalent to : a11x
k+11 + a12x
k2 + · · ·+ a1nx
kn = f1
a21xk+11 + a22x
k+12 + · · ·+ a2nx
kn = f2
...an1x
k+11 + an2x
k+12 + · · ·+ annx
k+1n = fn
for xk+11
for xk+12...
for xk+1n
Gauss-Seidel is equivalent to~xk+1 = −(L+D)−1U~xk + (D + L)−1 ~f
Exampl : Continue with last example
~xk+1 = −
5 0 0−3 9 02 −1 −7
0 −2 30 0 10 0 0
~xk +
5 0 0−3 9 02 −1 −7
−1 −123
Start with ~x0 =
000
. The sequence converges in 7 iteration to : ~x7 =
−0.1860.331−0.423
.
Do Jacobi / Gauss-Seidel Method always converges ?
Example : Consider :(
1 −57 −1
)(x1x2
)=
(−46
).
Then : Jacobi method gives :
~xk+1 =
(1 00 −1
)−1(0 5−7 0
)~xk +
(1 00 −1
)−1( −46
)
Start with ~x0 =
(00
). Then ~x1 =
(−46
), ~x2 =
(−34−34
), ~x3 =
(−174−294
), ...~x7 =
(−214374−300127
)which
doesn’t converges.
The real solution should be :(x1x2
)=
(11
)!
How about Gauss Seidel ?? It also doesn’t converges !
Our next goal is to check when Jacobi method and Gauss-Seidel method would converge.
Answer : Matrix A must satisfy certain property : Strictly diagonal dominant (SDD).
Analysis of CovergenceLet A = N − P .
3
Goal : Solve A~x = ~f ⇔ (N − P )~x = ~f .We have : N~x = P~x+ ~f to obtain iterative scheme :
N~xm+1 = P~xm + ~f, m = 0, 1, 2, ...
Let ~x∗be the solution of A~x = ~b. ∴ A~x∗ = ~b.Define error em := ~xm − ~x∗, m = 0, 1, 2, ...Now
N~xm+1 = P~xm + ~f (1)
N~x∗ = P~x∗ + ~f (2)
(1)− (2) : N(~xm+1 − ~x∗) = P (~xm − ~x∗)⇔ N~em+1 = P~em ⇔ ~em+1 = N−1P~em
So let M = N−1P , we have~em = Mm~e0
Assume {~u1, ..., ~un} is the set of linear independent eigenvectors of M . (~ui can be complex-valued vectors )Let ~e0 =
∑ni=1 ai~ui. Then :
~em = Mm~e0 =
n∑i=1
aiMm~ui =
n∑i=1
aλmi ~ui
where λ1, λ2, ...λn =corresponding eigenvalues (can be C).Suppose we can order the eigenvalues :
|λ1| ≥ |λ2| ≥ |λ3| ≥ ... ≥ |λn|
Then
~em = λm1
{a1~u1 +
n∑i=2
ai
(λiλ1
)m~ui
}Assume |λ1| < 1. Then ~em → 0 as m→∞.In order to reduce error by a factor of 10−m, then we need k iterations when |λ1|k ≤ 10−m.That is
k ≥ m
− log10(ρ(M)):=
m
R
We call ρ(M) : asymptotic convergence factor.We call R : asymptotic convergence rate.
In other words, the spectral radius of M :
ρ(M) = maxk{|λk| : λk = eigenvalues of M}
is a good indicator of the rate of convergence.But finding ρ(M) is difficult !Solution : Numerically (Next Topic)
Useful Theorem: Gerschgorin Theorem
Consider : ~e = (e1, e2, ..., en)T
=eigenvector of A = (aij) with eigenvalues λ.Then :
A~e = λ~e
4
Hence for each i (1 ≤ i ≤ n)
N∑j=1
aijej = λei
⇔ aiiei +
N∑j=1,j 6=i
aijej = λei
⇔ ei(aii − λ) = −N∑
j=1,j 6=i
aijej
⇔ |ei| |aii − λ| ≤N∑
j=1,j 6=i
|aij | |ej |
Suppose the component of the largest absolute value is |el| 6= 0 (∴ |el| ≥ |ej | ∀j).Then :
|el| |all − λ| ≤N∑
j=1,j 6=l
|alj | |ej | ≤N∑
j=1,j 6=l
|alj | |el|
⇒ |all − λ| ≤N∑
j=1,j 6=l
|alj |
So we have
λ ∈ Ball
N∑j=1,j 6=l
|alj |
That is the ball with centre all and with radii
∑Nj=1,j 6=i |alj |.
Note : We don’t know l unless we know λ and ~e.But : we can conclude
λ ∈N⋃l=1
Ball
N∑j=1j 6=l
|alj |
Example : Determine the upper bounds on the eigenvalues for the matrix:
A =
2 −1 0 0−1 2 −1 00 −1 2 −10 0 −1 2
Then all eigenvalues lie within
⋃Nl=1Ball
(∑Nj=1,j 6=l |alj |
).
For l = 1 and 4, Ball(∑N
j=1,j 6=l |alj |)
= {λ : |λ− 2| ≤ 1}.
For l = 2 and 3, Ball(∑N
j=1,j 6=l |alj |)
= {λ : |λ− 2| ≤ 2} .Therefore
N⋃l=1
Ball
N∑j=1,j 6=l
|alj |
= circle with radius 2 and centre at (2,0)
Since A is symmetric, all eigenvalues are real.Thus 0 ≤ λ ≤ 4.
In factm eigenvalues of A are : λ1 = 3.618, λ2 = 2.618, λ3 = 1.382, λ4 = 0.382.
5
That is ρ(A) = λ1 ≤ 4.
To prove the convergence of Jacobi method and the Gauss-Seidel method, let us introduce some definition.
Definition: A matrix M = (aij) is called strictly diagonally dominant (SDD) if:
|aii| >n∑
j=1,j 6=i
|aij | , i = 1, 2, ..., n
Theorem 1 : If a matrix A is SDD, then A must be non-singular.Proof : Recall all eigenvalues λ ∈
⋃Nl=1Ball
(∑Nj=1,j 6=l |alj |
).
Now A is SDD iff :
|all| >N∑
j=1,j 6=l
|alj | , l = 1, 2, ..., n
Therefore every ball Ball(∑N
j=1,j 6=l |alj |)must not contain 0.
Hence, eigenvalue cannot be 0.If A is singular, then ∃~v 6= 0 such that A~v = ~0 = 0~v.This implies λ = 0 is an engenvalue. Contradiction.So A is non-singular.
Now, we prove the convergence of Jacobi Method.Recall that Jacobi Method can be written as :
~xm+1 = D−1 (D −A) ~xm +D−1 ~f
Theorem: The Jacobi method converges to the solution of Ax = ~f if A is strictly diagonally dominant.Proof : Note that
~xm+1i = − 1
aii
n∑j=1,j 6=i
aij~xmj +
fiaii, i = 1, 2, ..., n (1)
Let x∗ be the solution. Then we also have
~x∗i = − 1
aii
n∑j=1,j 6=i
aij~x∗j +
fiaii
(2)
(1)-(2):
~em+1i = − 1
aii
n∑j=1,j 6=i
aij~emj
Therefore
∣∣~em+1i
∣∣ ≤ n∑j=1,j 6=i
∣∣∣∣aijaii∣∣∣∣ ∣∣~emj ∣∣
≤n∑
j=1,j 6=i
∣∣∣∣aijaii∣∣∣∣ ‖~em‖∞ (
‖~em‖ = max{∣∣~emj ∣∣})
= r ‖~em‖∞
r = maxi
n∑j=1,j 6=i
∣∣∣∣aijaii∣∣∣∣ < 1
⇒∥∥~em+1
∥∥∞ ≤ r ‖~em‖∞
Inductively,‖~em‖∞ ≤ r
m∥∥~e0∥∥∞
6
Therefore‖~em‖∞ → 0 as m→∞
Theorem : The Gauss-Seidel Method converges to the solution of A~x = ~f if A is strightly diagonally domi-nant.Proof : Gauss-Seidel Method can be written as:
~xm+1i = −
i−1∑j=1
aijaii~xm+1j −
n∑j=i+1
aijaii~xmj +
fiaii
(1)
Let ~x∗ be the solution of A~x = f. Then :
~x∗i = −i−1∑j=1
aijaii~x∗j −
n∑j=i+1
aijaii~x∗j +
fiaii
(2)
(1)-(2):
~em+1i = −
i−1∑j=1
aijaii~em+1j −
n∑j=i+1
aijaii~emj
Let ~em = (em1 , ..., emn ), and r = maxi
{∑nj=1,j 6=i
∣∣∣aijaii ∣∣∣} Again, we will prove:∣∣~em+1∣∣∞ ≤ r ‖~e
m‖∞ , m = 0, 1, 2, ...∀i
Induction on i :When i = 1,
∣∣~em+11
∣∣ ≤ n∑j=2
∣∣∣∣a1ja11∣∣∣∣ · ∣∣~emj ∣∣
≤ ‖~em‖∞n∑j=2
∣∣∣∣a1ja11∣∣∣∣
≤ r ‖~em‖∞
Assume∣∣~em+1
∣∣ ≤ r ‖~em‖∞ for k = 1, 2, ..., i− 1.Then :
∣∣~em+1i
∣∣ ≤ i−1∑j=1
∣∣∣∣aijaii∣∣∣∣ ∣∣~em+1
j
∣∣+
n∑j=i+1
∣∣∣∣aijaii∣∣∣∣ ∣∣~emj ∣∣
≤ r ‖~em‖∞i−1∑j=1
∣∣∣∣aijaii∣∣∣∣+ ‖~em‖∞
n∑j=i+1
∣∣∣∣aijaii∣∣∣∣
< ‖~em‖∞n∑
j=1,j 6=i
∣∣∣∣aijaii∣∣∣∣
≤ r ‖~em‖∞
By MI,∣∣~em+1i
∣∣ < r ‖~em‖∞.Hence ∥∥~em+1
∥∥∞ < r ‖~em‖∞
Therefore‖~em‖∞ < rm
∥∥~e0∥∥∞ → 0 as m→∞ as r < 1
7
Example : Consider
A~x =
(10 11 10
)(x1x2
)=
(1221
)= ~b
A is SDD. Therefore both Jacobi method and Gauss-Seidel method converge.Compare the convergence rates of the two methods.
Solution : Jacobi method: D =
(10 00 10
)
~xk+1 = D−1(D −A)~xk +D−1(
1221
)=
(10 00 10
)−1(0 −1−1 0
)~xk +D−1
(1221
)Let
M =
(10 00 10
)−1(0 −1−1 0
)Need to check the spectral radius of M :
ρ(M) = ρ
((10 00 10
)−1(0 −1−1 0
))= ρ
((0 − 1
10− 1
10 0
))Eigenvalue λ of M : λ2 − 1
100 = 0 ⇒ λ = 110 or λ = − 1
10Therefore M is diagonalizable and ρ(M) = 1
10 .
Recall: ~em = ρ(M)m ~Kj where ~em = error = ~xm − ~x∗ =(
110
)m ~Kj
Now, consider the Guass-Seidel method:Take
L+D =
(10 01 10
), U =
(0 −10 0
), ~xk+1 =
(10 01 10
)−1(0 −10 0
)~xk + (L+D)−1~b
We need to check:ρ(M) = ρ
((0 − 1
100 1
100
))Eigenvalue of M : (
λ− 1
100
)λ = 0⇒ λ =
1
100or λ = 0
Therefore
ρ(M) =1
100⇒ ~em =
(1
100
)m~KG−S
So Guass Seidel converges faster.In fact, recall that in order to reduce error by a factor of 10−m, we need k iteration such that:
|λ1|k ≤ 10−m ⇔ k ≥ m
− log10 (ρ(M))
For Jacobi, k ≥ m
− log10( 110 )
= m
For Guass-Seidel, k ≥ m
− log10( 1100 )
= m2
Therefore Gauss-Seidel converges twice as fast as Jacobi.
What if M = N−1P is not diagonalizable.Theorem : Let A ∈Mn×n(C) be a complex-valued matrix.
ρ(A) = maxi{|λi|} where λ1, λ2, ...λn = eigenvalues of A
8
where ρ(A) is called the spectral radius. Then
limk→∞
Ak = 0 iff ρ(A) < 1
Proof : (⇒) Let λ be an eigenvalues with eigenvector ~v.Then Ak~v = λk~v. Thus,
limk→0
Ak~v = limk→∞
λk~v
⇒ 0 = ~v limk→∞
λk
∴ limk→∞ λk = 0⇒ |λ|<1. Thus ρ(A)<1.
(⇐) Let λ1, λ2, ..., λs =eigenvalues of A. From linear algebra, ∃ invertible Q ∈Mn×n (C) such that :
A = QJQ−1
where J =Jordan Canonical form of A.Actually,
J =
Jm1(λ1)
Jm2(λ2). . .
Jms(λs)
where
Jmi(λi) =
λi 1
λi 1. . . . . .
λi 1λi
∈Mmi×mi (C) 1 ≤ i ≤ s
Now, Ak = QJkQ−1 and
Jk =
Jkm1
(λ1)Jkm2
(λ2). . .
Jkms(λs)
Also, for k ≥ mi − 1
Jkmi(λi) =
λki
(k1
)λk−1i
(k2
)λk−2i · · ·
(k
mi−1
)λk−mi+1i
0 λki
(k1
)λk−1i · · ·
(k
mi−2
)λk−mi+1i
. . ....
...
λki
(k1
)λk−1i
0 λki
Since ρ(A) < 1, then |λi| < 1 ∀i.Therefore limk→∞ Jkmi
= 0 ∀i and so Jk → 0 as k →∞.Thus,
limk→∞
Ak = limk→∞
QJkQ−1 = 0
Remark : Following the same idea, ρ(A) > 1 implies
limk→∞
∥∥Ak∥∥∞ as k →∞ ‖A‖∞ = max {aij}
9
Corollary : The iteration scheme xk+1 = Mxk + b converges iff ρ(M) < 1Proof : Consider
xk+1 = Mxk + b (1)
x = Mx+ b (2) x = solution
Thereforeek+1 = M(xk − x)⇒ ek+1 = Mek ⇒ ek = Mke0
If ρ(M) < 1, then Mk → 0 as k →∞.
Splitting Choice 3 : Successive overrelaxation Method (SOR)
A = L+D + U
Consider the iterative scheme : (Introduce sequence xk and x̄k)
Lxk+1 +Dx̄k+1 + Uxk = b (∗)
xk+1 = xk + ω(x̄k+1 − xk
)(∗∗)(
⇔ x̄k+1 =1
ω
(xk+1 + (ω − 1)xk
))Put (∗∗) to (∗), we have: (
L+1
ωD
)xk+1 +
1
ω(ωU + (ω − 1)D)xk = b
or (L+
1
ωD
)xk+1 =
(1
ωD − (D + U)
)xk + b (SOR)
Clearly, SOR is equivalent to splitting A:
A = N − P =
(L+
1
ωD
)−(
1
ωD − (D + U)
)In fact, SOR is equivalent to solving :
a11x̄k+11 + a12x
k2 + · · ·+ a1nx
kn = b1
a21xk+11 + a22x̄
k+12 + · · ·+ a2nx
kn = f2
...an1x
k+11 + an2x
k+12 + · · ·+ annx̄
k+1n = fn
for xk+11 = xk1 + ω
(x̄k+11 − xk1
)for xk+1
2 = xk2 + ω(x̄k+12 − xk2
)...
for xk+1n = xkn + ω
(x̄k+1n − xkn
)Remark : SOR = Gauss-Seidel if ω = 1.
Condition for the convegence of SORTheorem : The necessary condition (not sufficient) condition for SOR to converge is 0 < ω < 2.
Proff : Consider:
det(N−1P ) = det
(((L+
1
ω
)D
)−1(1
ωD − (D + U)
))
= det
((1
ωD
)−1)det
((1
ωD −D
))=
[ω det
(D−1
)] [ 1
ωdet ((1− ω)D)
]= det ((1− ω) I) = (1− ω)
n
10
Sonce det(N−1P
)=∏i λi
(λi are eigenvalues of M−1N
)Therefore
(1− ω)n
= det(N−1P
)=∏i
λi ≤ max |λi|n = ρ(N−1P
)n⇒ ρ
(N−1P
)≥ |ω − 1|
Now, SOR method converges iff ρ(N−1P ) < 1.So, |ω − 1| ≤ ρ
(M−1N
)< 1⇒ 0 < ω < 2.
Remark : In general, SOR conveges if and only if
ρ(N−1P
)= ρ
((L+
1
ωD
)−1(1
ωD − (D+U)
))< 1
Therefore, to find the sufficient condition for SOR method to converges, we need to check the eigenvalues of thematrix: (
L+1
ωD
)−1(1
ωD − (D + U)
)
Example: Let go back to Ax = ~b where A =
(10 11 10
).
Recall ρ(MJacobi) = 110 ; ρ (MG−S) = 1
100G-S converges faster !!
Now, consider the convergence rate for SOR method.
Recall the SOR method reads:
~xk+1 =
(L+
1
ωD
)−1(1
ωD − (D + U)
)~xk +
(L+
1
ωD
)−1~b
So,
M =
(L+
1
ωD
)−1(1
ωD − (D + U)
)=
[1
ω(D + ωL)
]−1 [1
ω(D − ω (D + U))
]= ω (D + ωL)
−1 1
ω((1− ω)D − ωU)
= (D + ωL)−1
((1− ω)D − ωU)
We examine ρ(M)
(1− ω)D − ωU = (1− ω)
[10 00 10
]− ω
(0 10 0
)=
(10 (1− ω) −ω
0 10 (1− ω)
)and
(D + ωL) =
(10 0ω 10
)⇒ (D + ωL)
−1=
(110 0− ω
100110
)∴MSOR = (D + ωL)
−1((1− ω)D − ωU) =
(1− ω − ω
10
−ω(1−ω)10ω2
100 + 1− ω
)Characteristic polynomial of MSOR is:
[(1− ω)− λ]
[ω2
100+ 1− ω − λ
]− ω2 (1− ω)
100= 0
11
Simplify:
λ2 − λ[2 (1− ω) +
ω2
100
]+ (1− ω)
2= 0
Then:
λ = (1− ω) +ω2
200± ω
20
√4 (1− ω) +
ω2
100
When ω = 1 (Gauss-Seidel method), λ = 0 or λ = 1100 .
Changing ω changes λ.
Choice of ω??Let us choose ω such that 4 (1− ω) + ω2
100 = 0So, the equation has equal root :
λ = (1− ω) +ω2
200= (1− ω)− 2 (1− ω) = ω − 1(
∵ 4 (1− ω) +ω2
100
)The smallest value of ω (2 > ω > 0) such that 4 (1− ω) + ω2
100 = 0 is:
ω = 1.002512579
(Which is very close to gauss-Seidel)But !! ρ(MSOR) = 0.002512579 (Compare to ρ (MGS) = 0.01, ρ (MJ) = 0.1)∴ converges much faster than G-S!!
Remark :
• ρ(MSOR) is very sensitive to ω. If you can hit the right value of ω, we can improve the spped of convergencesignificantly !!!
• One major task in computational math is to find the right parameter !!
How can we choose optimal ω for simple case??
• In general, difficult to choose ω
• ω is usually chosen as 0 < ω < 2
• But for some special matrix, optimum ω can be easily found.
Definition: Consider the system A~x = ~b. And let A = D + L+ U .If the eigenvalue of :
αD−1L+1
αD−1U, α 6= 0
is independent of α. Then the matrix is said to be consistently ordered.
Theorem: If A is consistently ordered, then the optimum ω for SOR method is:
ω =2
1 +√
1− ρ(MJ)2
where MJ = M in the Jacobi method, MJ = −D−1 (L+ U)
Proof: Consistently ordered means the eigenvalues of(αD−1L+
1
αD−1U
)
12
are the same as those for (D−1L+D−1U
)(Jacobi matrix, put α = 1)
Now consider the eigenvalues of MSOR. The characteristic polynomial
det (MSOR − λI) = 0 or
det(
(D + ωL)−1
((1− ω)D − ωU)− λI)
= 0 or
det (D + ωL)−1
det [(1− ω)D − ωU − λ (D + ωL)] = 0
So, λ satisfies:det [(1− ω − λ)D − ωU − λωL] = 0
Sinbce ω 6= 0, the non-zero eigenvalue λmust satisfy:
det
[((1− ω − λ)
ω√λ
D − 1√λU −
√λL
)ω√λ
]= 0
⇒ det
[√λD−1L+
1√λD−1U − (λ+ ω − 1)
ω√λ
I
]= 0
Since A is consistently ordered, the eigenvalues of
√λD−1L+
1√λD−1U
are the same as those of MJ = −D−1 (L+ U).
Let the eigenvalues of MJ be µ, then the non-zero eigenvalue of MSOR safisfies:
µ =λ+ ω − 1
ω√λ
for some λ (∗)
For ω 6= 0, we can solve (∗) and set
λ = (1− ω) +µ2ω2
2± µω
√(1− ω) +
µ2ω2
4(∗∗)
Each µ gives one or two eigenvalues λ.
λ depends on ω. We want to find ω such that ρ (MSOR) is as small as possible.We can show that this happens whenever the roots in (∗∗) are equal when µ takes the maximum norm. That is,
µω
√(1− ω) +
µ2ω2
4= 0
⇒ ω =2
1∓√
1− µ2
We look for smallest value of ω (0 < ω < 2) ad so
ω =2
1 +√
1− µ2=
2
1 +√
1− ρ(BJ)2
Example: Consider A =
(10 11 10
).
Then: αD−1L+ 1αD−1U =
[0 − 1
10α− α
10 0
]⇒ λ2 −
( −110α
) (− α
10
)= 0⇒ λ2 = 1
100
13
∴A is consistently ordered.∴ Optimal ω :
ω =2
1 +√
1− ρ(MJ)2=
2
1 +√
1− 1100
= 1.0025126 (sameasexample1)
∴The fastest convergence rate is:ρ (MSOR) = 0.0025126
SOR method:Take N = 1
ωD + L; P =(1ω − 1
)D − U .
Then: A = N − P .The iterative scheme:
~xk+1 =
(1
ωD + L
)−1((1
ω− 1
)D − U
)~xk +
(1
ωD + L
)−1~b
= (D + ωL)−1
[(1− ω)D − ωU ] ~xk +
(1
ωD + L
)−1~b
Recall:
• SOR converges ⇒|ω − 1| < 1 or 0 < ω < 2
• In general, convergence of SOR ⇐⇒ ρ(
(D + ωL)−1
[(1− ω)D − ωU ])< 1
• ω = 1→G-S method
Remark: In particular, tridiagonal matrix is consistently ordered:
A =
λ1 ∗
λ2 ∗. . . . . . . . .
∗ λn−1 ∗∗ λn
Example: Solve −u” = f, u(0) = a, u(1) = bPartition [0, 1] by x0 = 0 < x1 = h < x2 = 2h < · · · < 1 = xn
Approximate u” by u(x+h)−2u(x)+u(x−h)h2 , then −u” = f can be approximated by
A
u(x1).........
u(xn−1)
= ~b, where A =
2 −1−1 2 −1
. . . . . . . . .−1 2 −1
−1 2
Example of consistent ordering matrix
Example 1: Consider a block tridiagonal matrix of the form
A =
D1 T12T21 D2 T23
. . . . . . . . .
Tp,p−1 Dp
14
where Di are diagonal matrices. Then A is consistently ordered.To see this, it suffices to see that D−1L+D−1U and zD−1L+ 1
zD−1U are similar for all z 6= 0.
Note,
zD−1L+1
zD−1U = X
(D−1L+D−1U
)X−1
where
X =
I
zIz2I
. . .zp−1I
Example 2: Another type of consistent ordered matrices:
A =
T1 D12
D21 T2 T23. . . . . . . . .
Dp−1,pDp,p−1 Tp
where Ti ∈Mn×n are tridiagonal matrices.Proof: Complicated !
Theorem: [D.Young] Assume:
1. ω ∈ (0, 2)
2. MJAC has only real eigenvalue
3. β = ρ (MJAC) < 1
4. A is consistently ordered
Then: ρ (MSOR,ω) < 1In face,
ρ (MSOR,ω) =
{1− ω + 1
2ω2β2 + ωβ
√1− ω + ω2β2
4 , for 0 < ω ≤ ωoptω − 1 for ωopt ≤ ω < 2
whereωopt =
2
1 +√
1− β2
Convergence condition for SOR
Theorem: If A is symmetric positive definite, then the SOR method converges for all 0 < ω < 2.
Theorem: If A is strictly diagonally dominant, then the SOR converges for 0 < ω ≤ 1.
Proof: If A is SDD, then aii 6= 0 and A is invertible. The SOR method reads:
~xk+1 = MSOR~xk + ~c
whereMSOR = (D + ωL)
−1((1− ω)D − ωU) , (A = L+D + U)
We need to show that if 0 < ω ≤ 1, then ρ(MSOR) < 1We will prove by contradiction.
15
Suppose ∃ eigenvalue such that |λ| ≥ 1. Then
det (λI −MSOR) = 0
Also,
det(λ (D + ωL)
−1)[
(D + ωL)− 1
λ((1− ω)D − ωU)
]= 0
⇒ det(
(D + ωL)−1)
det
((1− 1
λ(1− ω)
)D + ωL+
ω
λU
)= 0
⇒ det (C) = 0
whereC =
(1− 1
λ(1− ω)
)D + ωL+
ω
λU
since aii 6= 0, det (D + ωL)−1 6= 0.
Then
|Cii| =
∣∣∣∣1− 1
λ(1− ω)
∣∣∣∣ |aii| ≥ [1− 1
|λ|(1− ω)
]|aii|
≥ ω |aii| > ω
n∑j=1,j 6=1
|aij | ≥ ωi−1∑j=1
|aij |+ω
|λ|
n∑j=i+1
|aij |
So C is also SDD.Thus det(C) 6= 0. (Contradiction)∴ all eigenvalues of A should satisfy |λ| < 1.Thus ρ(MSOR) < 1 and hence SOR converges.
16