iterative techniques in matrix algebra

8/20/2019 Iterative Techniques in Matrix Algebra

1/107

Iterative Techniques in Matrix Algebra

Jacobi & Gauss-Seidel Iterative Techniques II

Numerical Analysis (9th Edition)

R L Burden & J D Faires

Beamer Presentation Slidesprepared byJohn Carroll

Dublin City University

c 2011 Brooks/Cole, Cengage Learning

http://find/


2/107

Gauss-Seidel Method Gauss-Seidel Algorithm Convergence Results Interpretation

Outline

1 The Gauss-Seidel Method

Numerical Analysis (Chapter 7) Jacobi & Gauss-Seidel Methods II R L Burden & J D Faires 2 / 38

http://find/


3/107


Outline


2 The Gauss-Seidel Algorithm


http://find/


4/107


Outline



3 Convergence Results for General Iteration Methods


http://find/


5/107


Outline




4 Application to the Jacobi & Gauss-Seidel Methods


G S id l M h d G S id l Al i h C R l I i

http://find/


6/107


Outline






G S id l M th d G S id l Al ith C R lt I t t ti

http://find/


7/107


The Gauss-Seidel Method

Looking at the Jacobi MethodA possible improvement to the Jacobi Algorithm can be seen by

re-considering

x (k )i = 1a ii

n j =1 j =i

−a ij x (k −1) j

+ b i

, for i = 1, 2, . . . , n


Gauss Seidel Method Gauss Seidel Algorithm Convergence Results Interpretation

http://find/


8/107




re-considering

x (k )i = 1a ii

n j =1 j =i

−a ij x (k −1) j

+ b i

, for i = 1, 2, . . . , n

The components of x(k −1) are used to compute all the

components x (k )i of x(k ).



http://find/


9/107




re-considering

x (k )i = 1a ii

n j =1 j =i

−a ij x (k −1) j

+ b i

, for i = 1, 2, . . . , n

The components of x(k −1) are used to compute all the

components x (k )i of x(k ).

But, for i > 1, the components x (k )1 , . . . , x

(k )i −1 of x

(k ) have already

been computed and are expected to be better approximations to

the actual solutions x 1, . . . , x i −1 than are x (k −1)1 , . . . , x

(k −1)i −1 .



http://find/


10/107



Instead of using

x (k )i =

1

a ii

n j =1 j =i

−a ij x

(k −1) j

+ b i

, for i = 1, 2, . . . , n

it seems reasonable, then, to compute x (k )i using these most recently

calculated values.



http://find/http://goback/


11/107



Instead of using

x (k )i =

1

a ii

n j =1 j =i

−a ij x

(k −1) j

+ b i

, for i = 1, 2, . . . , n

it seems reasonable, then, to compute x (k )i using these most recently

calculated values.

The Gauss-Seidel Iterative Technique

x (k )i =

1

a ii

−

i −1 j =1

(a ij x (k )

j ) −n

j =i +1

(a ij x (k −1)

j ) + b i

for each i = 1, 2, . . . , n .



http://find/


12/107



Example

Use the Gauss-Seidel iterative technique to find approximate solutionsto

10x 1 − x 2 + 2x 3 = 6

−x 1 + 11x 2 − x 3 + 3x 4 = 25

2x 1 − x 2 + 10x 3 − x 4 = −113x 2 − x 3 + 8x 4 = 15

,



http://goforward/http://find/http://goback/


13/107

g g p


Example


10x 1 − x 2 + 2x 3 = 6

−x 1 + 11x 2 − x 3 + 3x 4 = 25

2x 1 − x 2 + 10x 3 − x 4 = −113x 2 − x 3 + 8x 4 = 15

,

starting with x = (0, 0, 0, 0)t



http://find/


14/107

g g p


Example


10x 1 − x 2 + 2x 3 = 6

−x 1 + 11x 2 − x 3 + 3x 4 = 25

2x 1 − x 2 + 10x 3 − x 4 = −113x 2 − x 3 + 8x 4 = 15

,

starting with x = (0, 0, 0, 0)t and iterating until

x(k )

−x(k −1)

∞x(k )∞


15/107


Example


10x 1 − x 2 + 2x 3 = 6

−x 1 + 11x 2 − x 3 + 3x 4 = 25

2x 1 − x 2 + 10x 3 − x 4 = −113x 2 − x 3 + 8x 4 = 15

,

starting with x = (0, 0, 0, 0)t and iterating until

x(k )

−x(k −1)

∞x(k )∞


16/107


Solution (1/3)

For the Gauss-Seidel method we write the system, for each

k = 1, 2, . . . as

x (k )1 = 110x (k −1)2 − 15

x (k −1)3 + 35

x (k )2 =

1

11x (k )1 +

1

11x (k −1)3 −

3

11x (k −1)4 +

25

11

x (k )3 = −

1

5 x (k )1 +

1

10 x (k )2 +

1

10 x (k −1)4 −

11

10

x (k )4 = −

3

8x (k )2 +

1

8x (k )3 +

15

8



http://find/


17/107


Solution (2/3)

When x(0) = (0, 0, 0, 0)t , we havex(1) = (0.6000, 2.3272, −0.9873, 0.8789)t .



http://find/


18/107


Solution (2/3)

When x(0) = (0, 0, 0, 0)t , we havex(1) = (0.6000, 2.3272, −0.9873, 0.8789)t . Subsequent iterations givethe values in the following table:

k 0 1 2 3 4 5

x (k )1 0.0000 0.6000 1.030 1.0065 1.0009 1.0001

x (k )

2

0.0000 2.3272 2.037 2.0036 2.0003 2.0000

x (k )

3 0.0000 −0.9873 −1.014 −1.0025 −1.0003 −1.0000

x (k )4 0.0000 0.8789 0.984 0.9983 0.9999 1.0000



http://find/


19/107


Solution (3/3)

Becausex(5) − x(4)∞

x(5)∞ =

0.0008

2.000 = 4 × 10−4

x(5) is accepted as a reasonable approximation to the solution.



http://find/


20/107


Solution (3/3)

Becausex(5) − x(4)∞

x(5)∞ =

0.0008

2.000 = 4 × 10−4

x(5) is accepted as a reasonable approximation to the solution.

Note that, in an earlier example, Jacobi’s method required twice as

many iterations for the same accuracy.



http://find/


21/107

The Gauss-Seidel Method: Matrix Form

Re-Writing the Equations

To write the Gauss-Seidel method in matrix form,



http://find/


22/107



To write the Gauss-Seidel method in matrix form, multiply both sides of

x (k )i = 1a ii

−i −1 j =1

(a ij x (k ) j ) −

n j =i +1

(a ij x (k −1) j ) + b i

by a ii and collect all k th iterate terms,



http://find/


23/107



To write the Gauss-Seidel method in matrix form, multiply both sides of

x (k )i = 1

a ii

−

i −1 j =1

(a ij x (k ) j ) −

n j =i +1

(a ij x (k −1) j ) + b i

by a ii and collect all k th iterate terms, to give

a i 1x (k )1 + a i 2x (

k )2 + · · · + a ii x (

k )i = −a i ,i +1x (

k −

1)i +1 − · · · − a in x (

k −

1)n + b i

for each i = 1, 2, . . . , n .



http://find/


24/107


Re-Writing the Equations (Cont’d)

Writing all n equations gives

a11x(k)1 = −a12x

(k−1)2 − a13x

(k−1)3 − · · · − a1nx

(k−1)n + b1

a21x(k)

1 + a

22x(k)

2 = −a

23x(k−1)

3 − · · · − a

2nx(k−1)

n + b

2

.

.

.

an1x(k)1 + an2x

(k)2 + · · · + annx

(k)n = bn



http://find/


25/107



Writing all n equations gives

a11x(k)1 = −a12x

(k−1)2 − a13x

(k−1)3 − · · · − a1nx

(k−1)n + b1

a21x(k)

1 + a

22x(k)

2 = −a

23x(k−1)

3 − · · · − a

2nx(k−1)

n + b

2

.

.

.

an1x(k)1 + an2x

(k)2 + · · · + annx

(k)n = bn

With the definitions of D , L, and U given previously, we have the

Gauss-Seidel method represented by

(D − L)x(k ) = U x(k −1) + b



http://find/


26/107


(D − L)x(k ) = U x(k −1) + b


Solving for x(k ) finally gives

x(k ) = (D − L)−1U x(k −1) + (D − L)−1b, for each k = 1, 2, . . .



http://find/


27/107


(D − L)x(k ) = U x(k −1) + b




Letting T g = (D − L)−1U and cg = (D − L)

−1b, gives the Gauss-Seidel

technique the form

x(k ) = T g x(k −1) + cg



http://find/


28/107


(D − L)x(k ) = U x(k −1) + b




Letting T g = (D − L)−1U and cg = (D − L)

−1b, gives the Gauss-Seidel

technique the form

x(k ) = T g x(k −1) + cg

For the lower-triangular matrix D − L to be nonsingular, it is necessaryand sufficient that a ii = 0, for each i = 1, 2, . . . , n .



http://find/


29/107

Outline







http://find/


30/107

Gauss-Seidel Iterative Algorithm (1/2)

To solve Ax = b given an initial approximation x(0):





31/107



INPUT the number of equations and unknowns n ;

the entries a ij

, 1 ≤

i , j ≤

n of the matrix A;

the entries b i , 1 ≤ i ≤ n of b;

the entries XO i , 1 ≤ i ≤ n of XO = x(0);

tolerance TOL;

maximum number of iterations N .



http://find/


32/107



INPUT the number of equations and unknowns n ;

the entries a ij

, 1 ≤

i , j ≤

n of the matrix A;

the entries b i , 1 ≤ i ≤ n of b;

the entries XO i , 1 ≤ i ≤ n of XO = x(0);

tolerance TOL;

maximum number of iterations N .

OUTPUT the approximate solution x 1, . . . , x n or a message

that the number of iterations was exceeded.



http://find/


33/107


Step 1 Set k = 1

Step 2 While (k ≤ N ) do Steps 3–6:



http://find/


34/107


Step 1 Set k = 1


Step 3 For i = 1, . . . , n

set x i = 1

a ii

−i −1

j =1

a ij x j −n

j =i +1

a ij XO j + b i



http://find/


35/107


Step 1 Set k = 1


Step 3 For i = 1, . . . , n

set x i = 1

a ii

−i −1

j =1

a ij x j −n

j =i +1

a ij XO j + b i

Step 4 If ||x − XO||


36/107


Step 1 Set k = 1


Step 3 For i = 1, . . . , n

set x i = 1

a ii

−i −1

j =1

a ij x j −n

j =i +1

a ij XO j + b i



37/107


Step 1 Set k = 1


Step 3 For i = 1, . . . , n

set x i = 1

a ii

−i −1

j =1

a ij x j −n

j =i +1

a ij XO j + b i



38/107


Step 1 Set k = 1


Step 3 For i = 1, . . . , n

set x i = 1

a ii

−i −1

j =1

a ij x j −n

j =i +1

a ij XO j + b i



39/107

Gauss-Seidel Iterative Algorithm

Comments on the Algorithm

Step 3 of the algorithm requires that a ii = 0, for eachi = 1, 2, . . . , n .



G S id l It ti Al ith

http://find/


40/107



Step 3 of the algorithm requires that a ii = 0, for eachi = 1, 2, . . . , n . If one of the a ii entries is 0 and the system isnonsingular, a reordering of the equations can be performed so

that no a ii = 0.



Ga ss Seidel Iterati e Algorithm

http://find/


41/107




that no a ii = 0.

To speed convergence, the equations should be arranged so that

a ii is as large as possible.



Gauss Seidel Iterative Algorithm

http://find/


42/107




that no a ii = 0.



Another possible stopping criterion in Step 4 is to iterate until

x(k ) − x(k −1)

x(k )

is smaller than some prescribed tolerance.



Gauss Seidel Iterative Algorithm

http://find/


43/107




that no a ii = 0.



Another possible stopping criterion in Step 4 is to iterate until

x(k ) − x(k −1)

x(k )

is smaller than some prescribed tolerance.

For this purpose, any convenient norm can be used, the usual

being the l ∞ norm.



Outline

http://find/


44/107

Outline







Convergence Results for General Iteration Methods

http://find/


45/107


Introduction

To study the convergence of general iteration techniques, we need

to analyze the formula

x(k ) = T x(k −1) + c, for each k = 1, 2, . . .

where x(0) is arbitrary.

The following lemma and the earlier Theorem on convergent

matrices provide the key for this study.






46/107


Lemma

If the spectral radius satisfies ρ(T )


47/107


Lemma



48/107


Lemma



49/107


Lemma



50/107


Proof (2/2)Let

S m = I + T + T 2 + · · · + T m




http://find/


51/107


Proof (2/2)Let

S m = I + T + T 2 + · · · + T m

Then

(I − T )S m = (1 + T + T 2 + · · · + T m ) − (T + T 2 + · · · + T m +1) = I − T m +1




http://find/


52/107


Proof (2/2)Let

S m = I + T + T 2 + · · · + T m

Then


and, since T is convergent, the Theorem on convergent matrices

implies that

limm →∞(I − T )S m = limm →∞(I − T m +1) = I




http://find/


53/107

g

Proof (2/2)Let

S m = I + T + T 2 + · · · + T m

Then


and, since T is convergent, the Theorem on convergent matrices

implies that

limm →∞(I − T )S m = limm →∞(I − T m +1) = I

Thus, (I − T )−1 = limm →∞ S m = I + T + T 2 + · · · =

∞ j =0 T

j




http://find/


54/107

g

Theorem

For any x(0) ∈ IRn , the sequence {x(k )}∞k =0 defined by

x(k )

= T x(k −1)

+c

, for each k ≥ 1

converges to the unique solution of

x = T x + c

if and only if ρ(T )


55/107

g

Proof (1/5)First assume that ρ(T )


56/107

g



57/107



58/107

Proof (1/5)

First assume that ρ(T )


59/107

Proof (1/5)



60/107

Proof (1/5)



61/107

Proof (2/5)The previous lemma implies that

limk →∞

x(k ) = lim

k →∞

T k x(0) +

∞

j =0

T j

c




http://find/http://goback/


62/107


limk →∞

x(k ) = lim

k →∞

T k x(0) +

∞

j =0

T j

c

= 0 + (I − T )−1c




http://find/


63/107


limk →∞

x(k ) = lim

k →∞

T k x(0) +

∞

j =0

T j

c

= 0 + (I − T )−1c

= (I − T )−1c




http://find/


64/107


limk →∞

x(k ) = lim

k →∞

T k x(0) +

∞

j =0

T j

c

= 0 + (I − T )−1c

= (I − T )−1c

Hence, the sequence {x(k )} converges to the vector x ≡ (I − T )−1cand x = T x + c.




http://find/


65/107

Proof (3/5)

To prove the converse, we will show that for any z ∈ IRn , we havelimk →∞ T

k z = 0.






66/107

Proof (3/5)


k z = 0.

Again, by the theorem on convergent matrices, this is equivalentto ρ(T )


67/107

Proof (3/5)


k z = 0.



68/107

Proof (3/5)


k z = 0.



69/107

Proof (3/5)


k z = 0.



70/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)






71/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)

sox − x(k ) = T

x − x(k −1)




http://find/


72/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)

sox − x(k ) = T

x − x(k −1)

= T 2x − x(k −2)






73/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)

sox − x(k ) = T

x − x(k −1)

= T 2x − x(k −2)

=

...




http://find/


74/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)

sox − x(k ) = T

x − x(k −1)

= T 2x − x(k −2)

=

...

= T k x − x(0)




http://find/


75/107

Proof (4/5)

Also,

x − x(k ) = (T x + c) −

T x(k −1) + c

= T x − x(k −1)

sox − x(k ) = T

x − x(k −1)

= T 2x − x(k −2)

=

...

= T k x − x(0)

= T k z




http://find/


76/107

Proof (5/5)

Hence

limk →∞

T k z = limk →∞

T k x − x(0)




http://find/


77/107

Proof (5/5)

Hence

limk →∞

T k z = limk →∞

T k x − x(0)

= limk →∞

x − x(k )




http://find/


78/107

Proof (5/5)

Hence

limk →∞

T k z = limk →∞

T k x − x(0)

= limk →∞

x − x(k )

= 0




http://find/


79/107

Proof (5/5)

Hence

limk →∞

T k z = limk →∞

T k x − x(0)

= limk →∞

x − x(k )

= 0

But z ∈ IRn was arbitrary, so by the theorem on convergentmatrices, T is convergent and ρ(T ) < 1.




http://find/


80/107

Corollary

T


81/107

Corollary

T


82/107

Corollary

T


83/107

Corollary

T


84/107





Numerical Analysis (Chapter 7) Jacobi & Gauss-Seidel Methods II R L Burden & J D Faires 28 / 38 Gauss-Seidel Method Gauss-Seidel Algorithm Convergence Results Interpretation

Convergence of the Jacobi & Gauss-Seidel Methods

http://find/


85/107

Using the Matrix Formulations

We have seen that the Jacobi and Gauss-Seidel iterative techniques

can be written

x(k ) = T j x

(k −1) + c j and

x(k ) = T g x(k −1) + cg





86/107



can be written

x(k ) = T j x

(k −1) + c j and

x(k ) = T g x(k −1) + cg

using the matrices

T j = D −1(L + U ) and T g = (D − L)

−1U

respectively.



http://find/


87/107



can be written

x(k ) = T j x

(k −1) + c j and

x(k ) = T g x(k −1) + cg

using the matrices

T j = D −1(L + U ) and T g = (D − L)

−1U

respectively. If ρ(T j ) or ρ(T g ) is less than 1, then the correspondingsequence {x(k )}∞k =0 will converge to the solution x of Ax = b.



http://find/


88/107

Example

For example, the Jacobi method has

x(k ) = D −1(L + U )x(k −1) + D −1b,



http://find/


89/107

Example


x(k ) = D −1(L + U )x(k −1) + D −1b,

and, if {x(k )}∞k =0

converges to x,



http://find/


90/107

Example


x(k ) = D −1(L + U )x(k −1) + D −1b,


converges to x, then

x = D −1(L + U )x + D −1b



http://find/


91/107

Example


x(k ) = D −1(L + U )x(k −1) + D −1b,



x = D −1(L + U )x + D −1b

This implies that

D x = (L + U )x + b and (D − L − U )x = b



http://find/


92/107

Example


x(k ) = D −1(L + U )x(k −1) + D −1b,



x = D −1(L + U )x + D −1b

This implies that

D x = (L + U )x + b and (D − L − U )x = b

Since D − L − U = A, the solution x satisfies Ax = b.

Numerical Analysis (Chapter 7) Jacobi & Gauss Seidel Methods II R L Burden & J D Faires 30 / 38 Gauss-Seidel Method Gauss-Seidel Algorithm Convergence Results Interpretation


http://find/


93/107

The following are easily verified sufficiency conditions for convergence

of the Jacobi and Gauss-Seidel methods.



http://find/


94/107

The following are easily verified sufficiency conditions for convergence

of the Jacobi and Gauss-Seidel methods.

Theorem

If A is strictly diagonally dominant, then for any choice of x(0), both the

Jacobi and Gauss-Seidel methods give sequences {x(k )}∞k =0 thatconverge to the unique solution of Ax = b.



http://find/


95/107

Is Gauss-Seidel better than Jacobi?



http://find/


96/107

Is Gauss-Seidel better than Jacobi?

No general results exist to tell which of the two techniques, Jacobi

or Gauss-Seidel, will be most successful for an arbitrary linear

system.

Numerical Analysis (Chapter 7) Jacobi & Gauss Seidel Methods II R L Burden & J D Faires 32 / 38

http://find/


97/107



S


98/107

(Stein-Rosenberg) Theorem

If a ij ≤ 0, for each i = j and a ii > 0, for each i = 1, 2, . . . , n , then oneand only one of the following statements holds:

(i) 0 ≤ ρ(T g ) < ρ(T j )


99/107

(Stein-Rosenberg) Theorem

If a ij ≤ 0, for each i = j and a ii > 0, for each i = 1, 2, . . . , n , then oneand only one of the following statements holds:

(i) 0 ≤ ρ(T g ) < ρ(T j )


100/107

Two Comments on the Thoerem

For the special case described in the theorem, we see from part

(i), namely

0 ≤ ρ(T g ) < ρ(T j )


101/107



(i), namely

0 ≤ ρ(T g ) < ρ(T j )


102/107



(i), namely

0 ≤ ρ(T g ) < ρ(T j )


103/107



(i), namely

0 ≤ ρ(T g ) < ρ(T j )


104/107

Eigenvalues & Eigenvectors: Convergent Matrices

Theorem

The following statements are equivalent.


105/107

g q

(i) A is a convergent matrix.

(ii) limn →∞ An = 0, for some natural norm.

(iii) limn →∞ An = 0, for all natural norms.

(iv) ρ(A)


106/107

Let g ∈ C [a , b ] be such that g (x ) ∈ [a , b ], for all x in [a , b ]. Suppose, inaddition, that g exists on (a , b ) and that a constant 0


107/107

Corrollary to the Fixed-Point Convergence Result

If g satisfies the hypothesis of the Fixed-Point Theorem then

|p n − p | ≤ k n

1 − k |p 1 − p 0|

Return to the Corollary to the Convergence Theorem for General Iterative Methods

iterative techniques in matrix algebra

Documents