chapter 5: linear systems: direct methodsascher/ag_2012/ag_slides/chap5.pdf · september 9, 2014...

September 9, 2014

Chapter 5: Linear Systems: Direct Methods

Uri M. Ascher and Chen GreifDepartment of Computer ScienceThe University of British Columbia

{ascher,greif}@cs.ubc.ca

Slides for the bookA First Course in Numerical Methods (published by SIAM, 2011)

http://bookstore.siam.org/cs07/

Linear systems: Direct Methods Goals

Goals of this chapter

• To learn practical methods to handle the most common problem in numericalcomputation;

• to get familiar (again) with the ancient method of Gaussian elimination in itsmodern form of LU decomposition, and develop pivoting methods for itsstable computation;

• to consider LU decomposition in the very important special cases ofsymmetric positive definite and sparse matrices;

• to study the expected quality of the computed solution, introducing as we gothe fundamental concept of a condition number.

Uri Ascher & Chen Greif (UBC Computer Science) A First Course in Numerical Methods September 9, 2014 1 / 56

Linear systems: Direct Methods Outline

Outline

• Gaussian elimination and backward substitution

• LU decomposition

• Pivoting strategies

• Efficient implementation

• Cholesky decomposition

• Sparse matrices

• Permutations and ordering strategies

• Estimating error and the condition number


Linear systems: Direct Methods Motivation

In general

• Here and in Chapter 7 we consider the problem of finding x which solves

Ax = b,

where A is a given, real, nonsingular, n× n matrix, and b is a given, realvector.

• Such problems are ubiquitous in applications!

• Two solution approaches:• Direct methods: yield exact solution in absence of roundoff error.

– Variations of Gaussian elimination.– Considered in this chapter

• Iterative methods: iterate in a similar fashion to what we do for nonlinearproblems.– Use only when direct methods are ineffective.– Considered in Chapter 7


Linear systems: Direct Methods Gaussian Elimination

Backward substitution

• Special case: A is an upper triangular matrix

A =

a11 a12 · · · a1n

a22. . .

.... . .

...ann

,

i.e., all elements below the main diagonal are zero: aij = 0, ∀i > j.

• The algorithm:

for k = n : −1 : 1

xk =bk −

∑nj=k+1 akjxj

akkend



Example

x1 − 4x2 + 3x3 = −2

5x2 − 3x3 = 7

−2x3 = −2

In matrix form:1 −4 30 5 −30 0 −2

x1

x2

x3

=

−27−2

.

Backward substitution: x3 = −2−2 = 1, then x2 = 1

5 (7 + 3 · 1) = 2, thenx1 = −2 + 4 · 2− 3 · 1 = 3.



Forward substitution

• Special case: A is a lower triangular matrix

A =

a11a21 a22...

. . .. . .

an1 an2 · · · ann

,

where all elements above the main diagonal are zero: aij = 0, ∀i < j.

• The algorithm:

for k = 1 : n

xk =bk −

∑k−1j=1 akjxj

akkend



Gaussian elimination

• Can multiply a row of Ax = b by a scalar and add to another row:elementary transformation.

• Use this to transform A to upper triangular form:

MAx = Mb, U = MA.

• Apply backward substitution to solve Ux = Mb.

A b A(1)b(1)

A(2) b(2) A(3)b(3)



Gaussian elimination (basic)

for k = 1 : n− 1

for i = k + 1 : n

lik =aikakk

for j = k + 1 : n

aij = aij − likakj

end

bi = bi − likbk

end

end

Then apply backward substitution.

Note: upper part of A is overwritten by U , lower part no longer of interest.



Cost (flop count)

• For the elimination:

≈ 2n−1∑k=1

(n− k)2 = 2((n− 1)2 + (n− 2)2 + · · ·+ 12) =2

3n3 +O(n2).

• For the backward substitution:

≈ 2

n−1∑k=1

(n− k) = 2(n− 1)n

2≈ n2.



Example

• Solve Ax = b for

A =

1 −4 31 1 03 −2 1

, b =

−256

.

• Gaussian elimination: (A | b) ⇒ 1 −4 3 −20 5 −3 70 10 −8 12

⇒

1 −4 3 −20 5 −3 70 0 −2 −2

.

• Backward substitution: x3 = −2−2 = 1, then x2 = 1

5 (7 + 3 · 1) = 2, thenx1 = −2 + 4 · 2− 3 · 1 = 3.


Linear systems: Direct Methods LU Decomposition

Outline






• Sparse matrices





LU decomposition

• What if we have many right hand side vectors, or we don’t know b rightaway?

• Note that determining transformation M such that MA = U does notdepend on b.

• M = M (n−1) . . .M (2)M (1), where M (k) is the transformation of the kthouter loop step. These are elementary lower triangular matrices, e.g.,

M (2) =

1

1

−l32. . .

.... . .

−ln2 1

.



LU decomposition (cont.)

• The matrix M is unit lower triangular.

• The matrix L = M−1 is also unit lower triangular:

A = LU, L =

1l21 1l31 l32 1...

.... . .

. . .

ln1 ln2 · · · ln,n−1 1

.



LU decomposition (cont.)

A = L U

So, Gaussian elimination is equivalent to:...1 decompose A = LU .Now for a given b we have to solve L

(Ux

)= b :

...2 use forward substitution to solve Ly = b;

...3 use backward substitution to solve Ux = y.



Example

A =

1 −4 31 1 03 −2 1

.

Obtain...1 l21 = 1

1 = 1, l31 = 31 = 3, so

M (1) =

1 0 0−1 1 0−3 0 1

, A(1) = M (1)A =

1 −4 30 5 −30 10 −8

.

...2 l32 = 105 = 2, so

M (2) =

1 0 00 1 00 −2 1

, A(2) = M (2)A(1) =

1 −4 30 5 −30 0 −2

.



Example (cont.)

• We thus obtain

U = A(2) = M (2)A(1) =

1 −4 30 5 −30 0 −2

,

and collect the multipliers l21, l31 and l32 into the unit lower triangularmatrix

L =

1 0 0l21 1 0l31 l32 1

=

1 0 01 1 03 2 1

.

• Indeed, A = LU :1 −4 31 1 03 −2 1

=

1 0 01 1 03 2 1

1 −4 30 5 −30 0 −2

.



Examples where the LU decomposition is useful

• When we have multiple right-hand sides, form once the LU decomposition(which costs O(n3) flops); then for each right-hand side only applyforward/backward substitutions (which are computationally cheap at O(n2)flops each).

• Can compute A−1 by decomposing A = LU once, and then solvingLUx = ek for each column ek of the unit matrix. These are n right handsides, so the cost is approximately 2

3n3 + n · 2n2 = 8

3n3 flops.

(However, typically we try to avoid computing the inverse A−1; the need tocompute it explicitly is rare.)

• Compute determinant of A by

det(A) = det(L) det(U) = Πnk=1ukk.


Linear systems: Direct Methods Pivoting

Outline






• Sparse matrices





Example: need for pivoting

• First step of Gaussion elimination: 1 1 1 11 1 2 21 2 2 3

⇒

1 1 1 10 0 1 10 1 1 2

.

• Second step: Now a(1)22 = 0 and we’re stuck.

• Simple remedy: exchange rows 2 and 3: 1 1 1 11 2 2 31 1 2 2

⇒

1 1 1 10 1 1 20 0 1 1

.

Here the decomposition has been completed without difficulty.



Partial pivoting

• It is rare to hit precisely a zero pivot, but common to hit a very small one.

• Example: 1 1 1 11 1 + 10−12 2 21 2 2 3

⇒

1 1 1 10 10−12 1 10 1 1 2

.

• Now we get a multplier l3,2 = 1/10−12 = 1012, so roundoff error inelimination step is magnified by this factor 1012.

• Employ Gaussian elimination with partial pivoting (GEPP) not just to avoidzero pivots but more generally to obtain a stable algorithm.



GEPP

• At each stage k choose q = q(k) as the smallest integer for which

|a(k−1)qk | = max

k≤i≤n|a(k−1)

ik |,

and interchange rows k and q.

• This ensures that pivots are not too small (unless matrix is close to singular)and |li,k| ≤ 1, all i ≥ k.

• PA = LU where P is permutation matrix, e.g.,

P =

1 0 00 0 10 1 0

.



Simple GEPP algorithm

for k = 1 : n− 1

for i = k + 1 : n

q = arg maxk≤i≤n

|a(k−1)ik |

exchange rows k and q

lik =aikakk

for j = k + 1 : n

aij = aij − lik ∗ akjend

bi = bi − lik ∗ bkend

end



Forming PA = LU

• It’s not so obvious, but it’s true, that with

B = M (n−1)P (n−1) · · ·M (2)P (2)M (1)P (1), P = P (n−1) · · ·P (2)P (1),

we get L lower triangular and

B = L−1P.

• The matrix L is lower triangular, although not the same as it would bewithout pivoting. It is obtained by a similar sequence of steps as before, withthe addition of permutation steps.

• The permutation matrix P is orthogonal, so

A = (PTL)U.

PTL is “psychologically lower triangular”.In practice, keep record of permutations in a 1D array.



Example revisited (1/3)

Same matrix we worked on a few slides ago, now with pivoting:

A =

1 −4 31 1 03 −2 1

.

Go through first column and find pivot:

P (1) =

0 0 10 1 01 0 0

; P (1)A =

3 −2 11 1 01 −4 3

.

So, we have

M (1) =

1 0 0−1

3 1 0−1

3 0 1

, A(1) = M (1)P (1)A =

3 −2 10 5

3 − 13

0 −103

83

.




Now, work on A(1):

P (2) =

1 0 00 0 10 1 0

; P (2)A(1) =

3 −2 10 −10

383

0 53 − 1

3

,

and we have

M (2) =

1 0 00 1 00 1

2 1

, A(2) = M (2)P (2)M (1)P (1)A =

3 −2 10 − 10

383

0 0 1

.

So the upper triangular U is U = A(2) = M (2)P (2)M (1)P (1)A.




• Let us find L and P . Write

U = M (2)P (2)M (1)P (1)A =(M (2)

)︸︷︷︸

M(2)

(P (2)M (1)P (2)T

)︸︷︷︸

M(1)

(P (2)P (1)

)︸︷︷︸

P

A.

• Next, take the elements of L below the diagonal to be those of the M (k)

with flipped signs; the permutation matrix P is just the product of the P (k):

L =

1 0 013 1 013 −1

2 1

; U =

3 −2 10 −10

383

0 0 1

; P =

0 0 11 0 00 1 0

.

Exercise: confirm that indeed, PA = LU .

• In Matlab obtain these matrices by the commandsA=[1 -4 3; 1 1 0; 3 -2 1];

[L,U,P]=lu(A);

• For more on the general principle illustrated in this example, see pages107–108 in the book, as well as Exercises 7 and 8 of Chapter 5.



GEPP stability

• Want to be assured that

gn(A) = maxi,j,k

|a(k)i,j |

does not grow exponentially in n. However, this is easily said than done!

• Bad scaling of rows can fool the GEPP we saw, because multiplying a row of(A | b) by an arbitrary nonzero constant can affect which q = k maximizes

|a(k−1)ik |.

• Can occasionally do better by scaled partial pivoting, where pivotdominance is relative to its original row norm.

• However, provably stable is only the more expensive complete pivoting.And yet, in practice partial pivoting is usually sufficient.

• There are special cases where no pivoting is required, including symmetricpositive definite and diagonally dominant matrices.


Linear systems: Direct Methods Efficient Implementation

Outline






• Sparse matrices





GEPP vectorization

• Memory access and inter-processor communications can be as expensive asfloating point operations.

• A simple way to improve efficiency in Matlab is to avoid if- for- and while-loops where possible.

• Work with array operations rather than on individual elements.

for k = 1:n-1% find pivot q [...]% interchange rows k and q and record this in pA([k,q],:)=A([q,k],:); p([k,q])=p([q,k]);% compute the corresponding column of LJ=k+1:n; A(J,k) = A(J,k) / A(k,k);% update submatrix by outer productA(J,J) = A(J,J) - A(J,k) * A(k,J);

end



Inner and outer products

Example:

y =

321

, z =

013

.

• Inner product

yT z = zTy = 3 ∗ 0 + 2 ∗ 1 + 1 ∗ 3 = 5.

• Outer products

yzT =

0 3 90 2 60 1 3

, zyT =

0 0 03 2 19 6 3

.

• Note that y and z do not need to have the same length for an outer product,although they do for an inner product.



Example

Same matrix as before, now with vectorized GEPP:

A =

1 −4 31 1 03 −2 1

.

Obtain for the first column k = 1 without pivoting

...1 J = [2, 3], A([2, 3], 1) =

(13

), A(1, [2, 3]) =

(−4 3

), so the upate is

...2

A([2, 3], [2, 3]) =

(1 0−2 1

)−(13

)∗(−4 3

)=

(5 −310 −8

).



Fast memory access and BLAS

• Computer memories are built as hierarchies: from faster, smaller and moreexpensive to slower, larger and cheaper.

...1 registers

...2 cache

...3 memory

...4 disk, cloud

• Standardize basic matrix operations into BLAS:...1 BLAS1: a ∗ x+ y (SAXPY)...2 BLAS2: matrix-vector operations...3 BLAS3: matrix-matrix operations


Linear systems: Direct Methods Cholesky Decomposition

Outline






• Sparse matrices





The “square root” of a symmetric positive definite matrix

• Recall that symmetric positive definite is the concept extension of a positivescalar to square real matrices A.

• For a scalar a > 0 there is a real square root, i.e., a real scalar g s.t. g2 = a.

• For a symmetric positive matrix A, the Cholesky decomposition is written as

A = GGT

where G is a lower triangular matrix.

• Obtain Cholesky as a special case of LU decomposition, utilizing bothsymmetry of A and the assurance that no partial pivoting is needed.



Cholesky decomposition

• Since A is symmetric positive definite, we can stably write

A = LU.

• Factor out the diagonal of U

U =

u11

u22

. . .. . .

unn

1 u12

u11· · · · · · u1n

u11

1 u23

u22· · · u2n

u22

. . ....

. . ....1

= DU.

• So A = LDU , and by symmetry, A = LDLT .



Cholesky decomposition cont.

• By an elementary linear algebra theorem, ukk > 0, k = 1, . . . , n, so candefine

D1/2 = diag{√u11, . . . ,

√unn}.

• Obtain A = GGT with G = LD1/2.

• Compute directly, using symmetry, in 13n

3 +O(n2) flops.

• In Matlab, R = chol(A) gives R = GT .



Cholesky Algorithm

Given a symmetric positive definite n× n matrix A, this algorithm overwrites itslower part with its Cholesky factor.

for k = 1 : n

akk =√akk

for i = k + 1 : n

aik =aikakk

end

for j = k + 1 : n

for i = j : n

aij = aij − aikajk

end

end

end


Linear systems: Direct Methods Sparse Matrices

Outline






• Sparse matrices





Sparse matrix

• Suppose A is n× n (although it can more generally be rectangular). Thematrix is sparse if most its elements are zero: nz = m ≪ n2.

• Think of a case where only m = O(n) elements are nonzero.

• Then convenient to represent by triplet form (i, j, v), such that

vk = aik,jk k = 1, 2, . . . ,m.

• Matlab has many special sparse matrix operations, e.g., sparse,spdiags: great for improved efficiency, not-so-great for code readability andfriendliness.



Banded matrices

• The simplest, most well-organized sparse matrix case is where all nonzeros ofA are in a band of diagonals neighboring the main diagonal.

A =

a11 · · · a1q...

. . .. . .

. . .

ap1. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . . an−q+1,n

. . .. . .

. . ....

an,n−p+1 · · · ann

.

• So,

aij = 0, if i ≥ j + p, or if j ≥ i+ q.

The bandwidth is p+ q − 1.



Special banded matrices

• For a diagonal matrix, p = q = 1;

• for a full (or dense) matrix, p = q = n;

• for a tridiagonal matrix, p = q = 2.

A



GE (and LU decomposition) for banded matrices

for k = 1 : n− 1

for i = k + 1 : k + p− 1

lik =aikakk

for j = k + 1 : k + q − 1

aij = aij − likakj

end

bi = bi − likbk

end

end

Computational cost (flop count): if p and q are constant as n grows then cost isO(n), which is a major improvement over the dense matrix case.



GEPP for banded matrices

• Essentially the same algorithm as without pivoting. However:

• The upper bandwidth of U may increase from q to q + p− 1.

• The matrix L, although it remains unit lower triangular, is no longer tightlybanded (although it has the same number of nonzeros per column).


Linear systems: Direct Methods Permutations and Ordering Strategies

Outline






• Sparse matrices





Fill-in

• Consider solving

Ax = b

where A is large and sparse (a vast majority of its elements are zero).

• Want to use GE (LU decomposition).

• But may have problem of fill-in during the elimination process, i.e., L and Utogether may have many more nonzero elements than A.

• Example: Arrow I

A =

× × × × ×× × 0 0 0× 0 × 0 0× 0 0 × 0× 0 0 0 ×

.

(Think of a symmetric matrix with n = 500, say.)

• Here both L and U could have completely full triangles: disastrous fill-in.



Fill-in in banded matrices

A =

a11 · · · a1q...

. . .. . .

. . .

ap1. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . .

. . .. . . an−q+1,n

. . .. . .

. . ....

an,n−p+1 · · · ann

.

Essentially no fill-in outside the band, but possible fill-in inside the band.



Example: model Poisson problem

0 10 20 30 40 50 60

0

10

20

30

40

50

60

nz = 288

The 2√n inner zero diagonals may be filled in upon using LU decomposition...



Permutations and ordering strategies

What can we do to obtain little or no fill-in?

• No perfect solution for this problem.

• Occasionally, permuting rows and columns of A significantly helps in reducingfill-in.

• Leads to considerations involving graph theory.

• Common goals are:• Reduce the bandwidth of the matrix; e.g., reverse Cuthill-McKee (RCM)• Reduce the expected fill-in in the decomposition stage; e.g., (approximate)

minimum degree (AMD).



Examples

...1 Arrow II Exchange first and last rows and likewise columns of Arrow I:

B =

× 0 0 0 ×0 × 0 0 ×0 0 × 0 ×0 0 0 × ×× × × × ×

.

Now there is no fill-in upon applying LU decomposition....2 Model Poisson problem

0 20 40 60

0

10

20

30

40

50

60

nz = 2880 20 40 60

0

10

20

30

40

50

60

nz = 2880 20 40 60

0

10

20

30

40

50

60

nz = 288

Left: the original, Center: RCM, Right: AMD.



Example: fill-in for Poisson

• Permuted matrix patterns

0 20 40 60

0

10

20

30

40

50

60

nz = 2880 20 40 60

0

10

20

30

40

50

60

nz = 2880 20 40 60

0

10

20

30

40

50

60

nz = 288

• After Cholesky decomposition

0 20 40 60

0

10

20

30

40

50

60

nz = 5190 20 40 60

0

10

20

30

40

50

60

nz = 4280 20 40 60

0

10

20

30

40

50

60

nz = 354


Linear systems: Direct Methods Estimating Errors and Condition Number

Outline






• Sparse matrices





Relative error in the solution

• Still consider

Ax = b

but now assess quality of approximate solution obtained somehow.

• Denote exact solution x, computed (or given) approximate solution x. Wantto estimate

∥x− x∥∥x∥

in some vector norm (∞-, 2-, or 1-norm).

• Can compute the residual r = b−Ax and so also ∥r∥∥b∥ .

Does a small relative residual imply small relative error in solution?



Example

• For the problem

A =

(1.2969 .8648.2161 .1441

), b =

(.8642.1440

),

consider the approximate solution

x =

(.9911−.4870

).

• Then

r = b−Ax =

(−10−8

10−8

),

so ∥r∥∞ = 10−8.

• However, the exact solution is

x =

(2−2

), so ∥x− x∥∞ = 1.513.



Conditioning of problem

• Since r = Ax−Ax = A(x− x), get

∥x− x∥ = ∥A−1r∥ ≤ ∥A−1∥∥r∥.

• Since Ax = b, get

1

∥x∥≤ ∥A∥

∥b∥.

• Hence∥x− x∥∥x∥

≤ κ(A)∥r∥∥b∥

,

κ(A) = ∥A∥∥A−1∥.

The scalar κ(A) is the condition number of A.For defining κ(A) use the induced matrix norm corresponding to the vector normemployed in the above estimates.



Quality of solution

• Backward error analysis: associate result of numerical algorithm (GEPP) withthe exact solution of a perturbed problem

(A+ δA)x = b+ δb.

• The job of GEPP is to make δA and δb small.

• Obtain good quality solution (only) if in addition, κ(A) is not too large.

• In our 2× 2 example, in fact, κ(A) ≈ 108, and indeed we saw∥x− x∥ ∼ κ(A)∥r∥.



The condition number

• Always κ(A) ≥ 1.

• For orthogonal matrices, κ2(Q) = 1: ideally conditioned!

• κ(A) indicates how close A is to being singular, which det(A) does not.

• If A is symmetric positive definite with eigenvalues λ1 ≥ · · · ≥ λn > 0 then

κ2(A) =λ1

λn.

• If A is noningular with singular values σ1 ≥ · · · ≥ σn > 0 then

κ2(A) =σ1

σn.

• The exact value of κ(A) rarely (if ever) matters in practice: typically, only itsorder of magnitude is important.


chapter 5: linear systems: direct methodsascher/ag_2012/ag_slides/chap5.pdf · september 9, 2014...

Documents