class17_linalg-i-handout.pdf

7/29/2019 class17_linalg-I-handout.pdf

1/22

High Performance Linear Algebra I

M. D. Jones, Ph.D.

Center for Computational ResearchUniversity at Buffalo

State University of New York

High Performance Computing I, 2012

M. D. Jones, Ph.D. (CCR/UB) High Performance Linear Algebra I HPC-I Fall 2012 1 / 28


2/22

Introduction Matrices, Vectors, and Scalars ... Oh My!

Computer Algebra Recapitulated

Consider an old friend,

A x = b,

where A is an M N matrix, and (therefore) x and b are vectors of

length N and M, respectively.

Otherwise known as M equations for N unknowns

Solution fails analytically when:

M < N (underdetermined)linear dependence exists (i.e. some row or column is a linear

combination of the others), i.e. A is singular.M > N (overdetermined), seek "best fit" solution (linear leastsquares)



3/22

Introduction Matrices, Vectors, and Scalars ... Oh My!

Potential Numerical Pitfalls

Eq. 1 has some special pitfalls numerically:

Near linear dependence exists, in which the machine precision is

unable to distinguish the linear independence

Roundoff errors accumulate in an uncontrolled fashion, whichusually leads to nonsensical results

(Prime motivator for using higher precision arithmetic)

Numerical analysts have produced copious research in the above, and

much of that work is incorporated into the various available algebra

packages



4/22

Gauss-Jordan

Gauss-Jordan elimination is the standard technique for solving linear

systems by inverting the (square, non-singular) matrix A:

Solution is unaffected by interchange of any two rows of A

Solution is unaffected by replacing any row by a linearcombination of other rows

Of course, in these operations we also have to keep track of the

corresponding permutations in x and b



5/22

Gauss-Jordan

Augmentation with the identity matrix yields A1,

A11 A12 . . . A1N 1 0 . . . 0A21 A22 . . . A2N 0 1 . . . 0...

.... . .

......

.... . .

...

AN1 AN2 . . . ANN 0 0 . . . 1

or you can augment with (a set of) solution vectors to solve for x

A11 A12 . . . A1N b11 b12A21 A22 . . . A2N b21 b22...

.... . .

......

...

AN1 AN2 . . . ANN bN1 bN2

Rearranging the rows of A to yield the identity matrix finds the solution

(simultaneously A1).


G J d


6/22

Gauss-Jordan

Attributes of Gauss-Jordan

Prominent features of Gauss-Jordan elimination:

Simple to implement

Stable (when used with pivoting, which we discuss next)

More expensive when you do not need A1

Using A1 for additional solution vectors can be unstable


G J d Pi ti


7/22

Gauss-Jordan Pivoting

Instability

Consider the following simple algorithm for the first steps in

Gauss-Jordan elimination:

do i =1,N1do j = i + 1 ,N1

mult =A( j , i ) / A( i , i )do k = i , N1

A ( j , k ) = A ( j , k ) mu l tA ( i , k )end do

end doend do

See the problem? In this context A(i, i) is called the pivot element.


Gauss Jordan Pivoting


8/22

Gauss-Jordan Pivoting

Pivoting

We can use pivoting to eliminate the instability in Gauss-Jordan (and

later plain Gaussian) elimination

Always unstable without pivoting

Partial pivoting, exchange rows to get desirable (large) value for

pivot

Full pivoting, exchange rows and columns to get desirable (large)

value for pivot

Implicit pivoting, use as pivots those elements that would have

been largest if original equations were scaled to have unity aslargest elements


Gaussian Elimination


9/22


Halfway There

Gauss-Jordan is sometimes called a full elimination scheme, i.e. the

original matrix is (fully) reduced to the identity. Gaussian elimination

(with or without pivoting) stops halfway:

A

11 A

12 . . . A

1N b

1

0 A

22 . . . A

2N b

2...

.... . .

......

0 0 . . . A

NN b

N

when we have a triangular matrix,




10/22


and then proceeds through backsubstitution to obtain the solution

xi =1

A

ii

b

i

N

j=i+1

A

ijxj

,

where i goes from N down to 1.


Gaussian Elimination Gauss or Gauss-Jordan


11/22

Gaussian Elimination Gauss, or Gauss Jordan

Gauss, or Gauss-Jordan?

So which technique, Gaussian or Gauss-Jordan elimination is

preferable, computationally?

Even if you do compute the inverse matrix, Gaussian elimination

has a slight advantage in operation count (N3/3 versus N3, plusN2/2 versus N2 for each right-hand side).

Big drawback for both is that all right-hand side vectors must be

solved at the time of elimination

Neither - the next method, LU factorization is just as efficient,

without the drawbacks


LU Decomposition Triangular Matrices


12/22


Triangular Factors

Suppose that we can write A as a product of a lower triangular matrix,

L and an upper triangular matrix, U (we can if A is invertible and its

principal minors are non-zero),

A = L U,

then our original matrix equation becomes

Ax = b,

L Ux=

b,

and then we can arrange as two coupled equations:




13/22

p g

Ly = b,

Ux = y.

Given the triangular nature of these matrices, we can write the

solutions as:

yi = 1Lii

bii1

j=1

Lijyj ,

known as forward substitution and

xi = 1Uii

yiN

j=i+1

Uijxj ,

as the backward substitution step.




14/22

p g

LU Code

LU code fragment:

f o r ( k = 1 ; k


15/22

Singular Value Decomposition

Useful when Gaussian elimination and LU decomposition fails

Diagnose problematic matrices, and often can yield useful

numerical results

Also useful for nonlinear least-squares problems, although that is

a separate topic


Singular Value Decomposition Mathematical Foundation of SVD


16/22

Foundations of SVD

SVD is based on a linear algebraic theorem:

Theorem

Any M

N matrixA

with M

N can be written as the product of aM N column orthogonal matrixU, a N N diagonal matrixwwith

nonnegative elements, and the transpose of an N N orthogonal

matrixV:

A = Uw VT. (1)


Singular Value Decomposition Mathematical Foundation of SVD


17/22

The column-wise orthogonality of U and V simply means

Mi=1

UijUik = jk,

Ni=1

VijVik = jk.


Singular Value Decomposition SVD for Square Matrices


18/22

SVD For Square Matrices

For M = N, it is easy to invert the orthogonal matrices U and V(just take the transpose) and show that

A1 = V diag(1/wj)

UT

If some of the wj are quite small (or even zero), that is as

indication that A is ill-conditioned.

The condition number of a matrix is simply given by

cond(A) =max(wj)

min(wj)

.

Condition numbers approaching machine preicision (106 single,

1012 double) indicate ill-condition A


Singular Value Decomposition SVD for Square Matrices


19/22

Columns of U whose corresponding values wj are nonzero form

an orthonormal basis (i.e. span) the range of A

Columns of V whose corresponding values wj are zero form anorthonormal basis (i.e. span) the nullspace of A


Iterative Improvement

I i I P S l i


20/22

Iterative Improvements to Poor Solutions

Sometimes you have a nearly singular matrix that leads to roundoff

errors even under the best of circumstances. This can happen for large

matrices as well. But all hope is not lost, as you can apply the following

trick to improve the quality of an existing solution.Start with

Ax = b,

but suppose that x is the exact solution, and ours is just a bit off,

x + x.




21/22

So our poor solution satisfies

A (x + x) = b + b,

which also implies

A x = b,

but substituting for b gives:

A x = A (x + x) b, (2)

but we know the right-hand side of Eq. 2 - it is the residual of the

problem we originally solved, and now want to improve. So solving Eq.2 for x is as simple as backsubstituting the right-hand side using ouroriginal A (if LU decomposed). Rinse and repeat!



It ti I t S h


22/22

Iterative Improvement Scheme

The iterative improvement scheme:

Solve for residual, r = (Ax b),

while (norm(r) > ) do

Solve Ax = rLet x = x + xr = Ax b

end while

When the matrix A is already LU decomposed, solving for x isrelatively straightforward.


class17_linalg-i-handout.pdf

Documents