Sparse matricesWikipedia

Contents

1 Anti-diagonal matrix 11.1 Formal denition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21.5 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2 Band matrix 32.1 Band matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.3 Band storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.4 Band form of sparse matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42.5 Examples and special cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.6 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.7 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.9 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3 Bidiagonal matrix 63.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73.4 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Block matrix 84.1 Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2 Block matrix multiplication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.3 Block matrix inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.4 Block diagonal matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.5 Block tridiagonal matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.6 Block Toeplitz matrices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.7 Direct sum . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114.8 Direct product . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

i

ii CONTENTS

4.9 Application . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.10 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124.11 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

5 Reverse Cuthill-McKee algorithm 135.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145.2 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

6 Diagonal matrix 166.1 Scalar matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166.2 Matrix operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.3 Operator matrix in eigenbasis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.4 Other properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.5 Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176.6 Operator theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.7 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

7 Generalized permutation matrix 197.1 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

7.1.1 Group structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197.1.2 Subgroups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

7.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.3 Generalizations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.4 Signed permutation group . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

7.4.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

7.5.1 Monomial representations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

8 Identity matrix 228.1 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238.2 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238.3 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

9 Nested dissection 249.1 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.2 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25

10 Pentadiagonal matrix 2610.1 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

CONTENTS iii

11 Permutation matrix 2711.1 Denition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2811.2 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2811.3 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2911.4 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

11.4.1 Permutation of rows and columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2911.4.2 Permutation of rows . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

11.5 Explanation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3011.6 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3011.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

12 Reverse Cuthill-McKee algorithm 3212.1 Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3312.2 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3412.3 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34

13 Shear matrix 3513.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3513.2 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3613.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3613.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3613.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36

14 Shift matrix 3714.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3714.2 Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3814.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3814.4 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

15 Single-entry matrix 3915.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

16 Skyline matrix 4016.1 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4016.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40

17 Sparse matrix 4117.1 Storing a sparse matrix . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

17.1.1 Dictionary of keys (DOK) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.1.2 List of lists (LIL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.1.3 Coordinate list (COO) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4217.1.4 Yale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4317.1.5 Compressed row Storage (CRS or CSR) . . . . . . . . . . . . . . . . . . . . . . . . . . . 4317.1.6 Compressed sparse column (CSC or CCS) . . . . . . . . . . . . . . . . . . . . . . . . . . 44

iv CONTENTS

17.2 Special structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4417.2.1 Banded . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4417.2.2 Diagonal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4417.2.3 Symmetric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

17.3 Reducing ll-in . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4417.4 Solving sparse matrix equations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4517.5 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4517.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4517.7 Further reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4517.8 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46

18 Sparse matrix-vector multiplication 4718.1 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

19 Tridiagonal matrix 4819.1 Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

19.1.1 Determinant . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4819.1.2 Inversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4919.1.3 Solution of linear system . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5019.1.4 Eigenvalues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

19.2 Computer programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5019.3 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5019.4 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5019.5 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

20 Zero matrix 5220.1 See also . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5220.2 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5220.3 External links . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5320.4 Text and image sources, contributors, and licenses . . . . . . . . . . . . . . . . . . . . . . . . . . 54

20.4.1 Text . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5420.4.2 Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5520.4.3 Content license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

Chapter 1

Anti-diagonal matrix

In mathematics, an anti-diagonal matrix is a matrix where all the entries are zero except those on the diagonal goingfrom the lower left corner to the upper right corner (), known as the anti-diagonal.

1.1 Formal denition

An n-by-n matrix A is an anti-diagonal matrix if the (i, j) element is zero for all i, j {1, , n} with i + j n + 1.

1.2 Example

An example of an anti-diagonal matrix is

2666640 0 0 0 10 0 0 2 00 0 5 0 00 7 0 0 01 0 0 0 0

377775:

1.3 Properties

All anti-diagonal matrices are also persymmetric.The product of two anti-diagonal matrices is a diagonal matrix. Furthermore, the product of an anti-diagonal matrixwith a diagonal matrix is anti-diagonal, as is the product of a diagonal matrix with an anti-diagonal matrix.An anti-diagonal matrix is invertible if and only if the entries on the diagonal from the lower left corner to the upperright corner are nonzero. The inverse of any invertible anti-diagonal matrix is also anti-diagonal, as can be seen fromthe paragraph above. The determinant of an anti-diagonal matrix has absolute value given by the product of theentries on the diagonal from the lower left corner to the upper right corner. However, the sign of this determinantwill vary because the one nonzero signed elementary product from an anti-diagonal matrix will have a dierent signdepending on whether the permutation related to it is odd or even:More precisely, the sign of the elementary product needed to calculate the determinant of an anti-diagonal matrix isrelated to whether the corresponding triangular number is even or odd. This is because the number of inversions inthe permutation for the only nonzero signed elementary product of any n n anti-diagonal matrix is always equal tothe nth such number.

1

2 CHAPTER 1. ANTI-DIAGONAL MATRIX

1.4 See also Main diagonal

1.5 External links Anti-diagonal matrix at PlanetMath.org.

Chapter 2

Band matrix

In mathematics, particularly matrix theory, a band matrix is a sparse matrix whose non-zero entries are conned toa diagonal band, comprising the main diagonal and zero or more diagonals on either side.

2.1 Band matrixFormally, consider an nnmatrix A=(ai,j ). If all matrix elements are zero outside a diagonally bordered band whoserange is determined by constants k1 and k2:

ai;j = 0 if j < i k1 or j > i+ k2; k1; k2 0:

then the quantities k1 and k2 are called the lower and upper bandwidth, respectively.[1] The bandwidth of thematrix is the maximum of k1 and k2; in other words, it is the number k such that ai;j = 0 if ji jj > k .[2]A matrix is called a band matrix or banded matrix if its bandwidth is reasonably small.A band matrix with k1 = k2 = 0 is a diagonal matrix; a band matrix with k1 = k2 = 1 is a tridiagonal matrix; whenk1 = k2 = 2 one has a pentadiagonal matrix and so on. If one puts k1 = 0, k2 = n1, one obtains the denition of anupper triangular matrix; similarly, for k1 = n1, k2 = 0 one obtains a lower triangular matrix.

2.2 ApplicationsIn numerical analysis, matrices from nite element or nite dierence problems are often banded. Such matrices canbe viewed as descriptions of the coupling between the problem variables; the bandedness corresponds to the fact thatvariables are not coupled over arbitrarily large distances. Such matrices can be further divided - for instance, bandedmatrices exist where every element in the band is nonzero. These often arise when discretising one-dimensionalproblems.Problems in higher dimensions also lead to banded matrices, in which case the band itself also tends to be sparse.For instance, a partial dierential equation on a square domain (using central dierences) will yield a matrix witha bandwidth equal to the square root of the matrix dimension, but inside the band only 5 diagonals are nonzero.Unfortunately, applying Gaussian elimination (or equivalently an LU decomposition) to such a matrix results in theband being lled in by many non-zero elements.

2.3 Band storageBand matrices are usually stored by storing the diagonals in the band; the rest is implicitly zero.For example, a tridiagonal matrix has bandwidth 1. The 6-by-6 matrix

3

4 CHAPTER 2. BAND MATRIX

266666666664

B11 B12 0 0B21 B22 B23

. . . . . . ...0 B32 B33 B34

. . . ...... . . . B43 B44 B45 0... . . . . . . B54 B55 B560 0 B65 B66

377777777775is stored as the 6-by-3 matrix

266666640 B11 B12B21 B22 B23B32 B33 B34B43 B44 B45B54 B55 B56B65 B66 0

37777775:

A further saving is possible when the matrix is symmetric. For example, consider a symmetric 6-by-6 matrix with anupper bandwidth of 2:

266666664

A11 A12 A13 0 0A22 A23 A24

. . . ...A33 A34 A35 0

A44 A45 A46sym A55 A56

A66

377777775:

This matrix is stored as the 6-by-3 matrix:

26666664A11 A12 A13A22 A23 A24A33 A34 A35A44 A45 A46A55 A56 0A66 0 0

37777775:

2.4 Band form of sparse matricesFrom a computational point of view, working with band matrices is always preferential to working with similarlydimensioned square matrices. A band matrix can be likened in complexity to a rectangular matrix whose row di-mension is equal to the bandwidth of the band matrix. Thus the work involved in performing operations such asmultiplication falls signicantly, often leading to huge savings in terms of calculation time and complexity.As sparse matrices lend themselves to more ecient computation than dense matrices, as well as in more ecientutilization of computer storage, there has been much research focused on nding ways to minimise the bandwidth(or directly minimise the ll-in) by applying permutations to the matrix, or other such equivalence or similaritytransformations.[3]

TheCuthillMcKee algorithm can be used to reduce the bandwidth of a sparse symmetricmatrix. There are, however,matrices for which the reverse CuthillMcKee algorithm performs better. There are many other methods in use.The problem of nding a representation of a matrix with minimal bandwidth by means of permutations of rows andcolumns is NP-hard.[4]

2.5. EXAMPLES AND SPECIAL CASES 5

2.5 Examples and special casesThe following are special cases of band matrices:

Diagonal matrices. Tridiagonal matrices. Pentadiagonal matrices. Upper and lower triangular matrices. Upper and lower Hessenberg matrices. Block-diagonal matrices. Shift matrices and shear matrices. Matrices in Jordan normal form. A skyline matrix, also called variable band matrix is a generalization of band matrix

The inverses of Lehmer matrices are constant tridiagonal matrices, and are thus band matrices.

2.6 See also Graph bandwidth

2.7 Notes[1] Golub & Van Loan 1996, 1.2.1.[2] Atkinson 1989, p. 527.[3] Davis 2006, 7.7.[4] Feige 2000.

2.8 References Atkinson, Kendall E. (1989),An Introduction to Numerical Analysis, JohnWiley& Sons, ISBN 0-471-62489-6. Davis, Timothy A. (2006), Direct Methods for Sparse Linesr Systems, Society for Industrial and Applied Math-ematics, ISBN 978-0-898716-13-9.

Feige, Uriel (2000), Coping with the NP-Hardness of the Graph Bandwidth Problem, Algorithm Theory -SWAT 2000, Lecture Notes in Computer Science 1851, pp. 129145, doi:10.1007/3-540-44985-X_2.

Golub, Gene H.; Van Loan, Charles F. (1996), Matrix Computations (3rd ed.), Baltimore: Johns Hopkins,ISBN 978-0-8018-5414-9.

Press, WH; Teukolsky, SA; Vetterling, WT; Flannery, BP (2007), Section 2.4, Numerical Recipes: The Artof Scientic Computing (3rd ed.), New York: Cambridge University Press, ISBN 978-0-521-88068-8.

2.9 External links Information pertaining to LAPACK and band matrices A tutorial on banded matrices and other sparse matrix formats

Chapter 3

Bidiagonal matrix

In mathematics, a bidiagonal matrix is a matrix with non-zero entries along the main diagonal and either the diagonalabove or the diagonal below. This means there are exactly two non zero diagonals in the matrix.When the diagonal above the main diagonal has the non-zero entries the matrix is upper bidiagonal. When thediagonal below the main diagonal has the non-zero entries the matrix is lower bidiagonal.For example, the following matrix is upper bidiagonal:

0BB@1 4 0 00 4 1 00 0 3 40 0 0 3

1CCAand the following matrix is lower bidiagonal:

0BB@1 0 0 02 4 0 00 3 3 00 0 4 3

1CCA:

3.1 UsageOne variant of the QR algorithm starts with reducing a general matrix into a bidiagonal one,[1] and the Singular valuedecomposition uses this method as well.

3.2 See also

Diagonal matrix

List of matrices

LAPACK

Bidiagonalization

Hessenberg form The Hessenberg form is similar, but has more non zero diagonal lines than 2.

Tridiagonal matrix with three diagonals

6

3.3. REFERENCES 7

3.3 References Stewart, G. W. (2001) Matrix Algorithms, Volume II: Eigensystems. Society for Industrial and Applied Mathe-matics. ISBN 0-89871-503-2.

[1] Bochkanov Sergey Anatolyevich. ALGLIB User Guide - General Matrix operations - Singular value decomposition . AL-GLIB Project. 2010-12-11. URL:http://www.alglib.net/matrixops/general/svd.php. Accessed: 2010-12-11. (Archivedby WebCite at http://www.webcitation.org/5utO4iSnR)

3.4 External links High performance algorithms for reduction to condensed (Hessenberg, tridiagonal, bidiagonal) form

Chapter 4

Block matrix

In mathematics, a block matrix or a partitioned matrix is a matrix which is interpreted as having been broken intosections called blocks or submatrices.[1] Intuitively, a matrix interpreted as a block matrix can be visualized as theoriginal matrix with a collection of horizontal and vertical lines which break it out, or partition it, into a collection ofsmaller matrices.[2] Any matrix may be interpreted as a block matrix in one or more ways, with each interpretationdened by how its rows and columns are partitioned.This notion can be made more precise for an n bym matrixM by partitioning n into a collection rowgroups , andthen partitioningm into a collection colgroups . The original matrix is then considered as the total of these groups,in the sense that the (i; j) entry of the original matrix corresponds in a 1-to-1 way with some (s; t) oset entry ofsome (x; y) , where x 2 rowgroups and y 2 colgroups .

4.1 ExampleThe matrix

P =

26641 1 2 21 1 2 23 3 4 43 3 4 4

3775can be partitioned into 4 22 blocks

P11 =1 11 1

;P12 =

2 22 2

;P21 =

3 33 3

;P22 =

4 44 4

:

The partitioned matrix can then be written as

P =P11 P12P21 P22

:

4.2 Block matrix multiplicationIt is possible to use a block partitioned matrix product that involves only algebra on submatrices of the factors. Thepartitioning of the factors is not arbitrary, however, and requires conformable partitions[3] between two matricesA and B such that all submatrix products that will be used are dened.[4] Given an (m p) matrix A with q rowpartitions and s column partitions

8

4.2. BLOCK MATRIX MULTIPLICATION 9

A 168168 element block matrix with 1212, 1224, 24x12, and 2424 sub-Matrices. Non-zero elements are in blue, zero elementsare grayed.

A =

26664A11 A12 A1sA21 A22 A2s... ... . . . ...

Aq1 Aq2 Aqs

37775and a (p n) matrix B with s row partitions and r column partitions

B =

26664B11 B12 B1rB21 B22 B2r... ... . . . ...

Bs1 Bs2 Bsr

37775;that are compatible with the partitions of A , the matrix product

C = AB

can be formed blockwise, yieldingC as an (mn)matrix with q row partitions and r column partitions. Thematricesin your matrix C are calculated by multiplying:

C =sX

=1

AB :

10 CHAPTER 4. BLOCK MATRIX

Or, using the Einstein notation that implicitly sums over repeated indices:

C = AB :

4.3 Block matrix inversionMain article: HelmertWolf blocking

If a matrix is partitioned into four blocks, it can be inverted blockwise as follows:where A, B, C and D have arbitrary size. (A and D must be square, so that they can be inverted. Furthermore, Aand DCA1B must be nonsingular.[5])Equivalently,

4.4 Block diagonal matricesA block diagonalmatrix is a blockmatrix which is a squarematrix, and havingmain diagonal blocks squarematrices,such that the o-diagonal blocks are zero matrices. A block diagonal matrix A has the form

A =

26664A1 0 00 A2 0... ... . . . ...0 0 An

37775where Ak is a square matrix; in other words, it is the direct sum of A1, , An. It can also be indicated as A1 A2 : : : A or diag(A1, A2, : : : , A) (the latter being the same formalism used for a diagonal matrix). Any squarematrix can trivially be considered a block diagonal matrix with only one block.For the determinant and trace, the following properties hold

detA = detA1 : : : detAntrA = trA1 + + trAn:The inverse of a block diagonal matrix is another block diagonal matrix, composed of the inverse of each block, asfollows:

0BBB@A1 0 00 A2 0... ... . . . ...0 0 An

1CCCA1

=

0BBB@A11 0 00 A12 0... ... . . . ...0 0 A1n

1CCCA:The eigenvalues and eigenvectors of A are simply those of A1 and A2 and ... and An (combined).

4.5 Block tridiagonal matricesA block tridiagonal matrix is another special block matrix, which is just like the block diagonal matrix a squarematrix, having square matrices (blocks) in the lower diagonal, main diagonal and upper diagonal, with all other blocksbeing zero matrices. It is essentially a tridiagonal matrix but has submatrices in places of scalars. A block tridiagonalmatrix A has the form

4.6. BLOCK TOEPLITZ MATRICES 11

A =

266666666664

B1 C1 0A2 B2 C2

. . . . . . . . . ...Ak Bk Ck

... . . . . . . . . .An1 Bn1 Cn1

0 An Bn

377777777775where Ak, Bk and Ck are square sub-matrices of the lower, main and upper diagonal respectively.Block tridiagonal matrices are often encountered in numerical solutions of engineering problems (e.g., computationaluid dynamics). Optimized numerical methods for LU factorization are available and hence ecient solution algo-rithms for equation systems with a block tridiagonal matrix as coecient matrix. The Thomas algorithm, used forecient solution of equation systems involving a tridiagonal matrix can also be applied using matrix operations toblock tridiagonal matrices (see also Block LU decomposition).

4.6 Block Toeplitz matricesA block Toeplitz matrix is another special block matrix, which contains blocks that are repeated down the diagonalsof the matrix, as a Toeplitz matrix has elements repeated down the diagonal. The individual block matrix elements,Aij, must also be a Toeplitz matrix.A block Toeplitz matrix A has the form

A =

266666666664

A(1;1) A(1;2) A(1;n1) A(1;n)A(2;1) A(1;1) A(1;2) A(1;n1)

. . . . . . . . . ...A(2;1) A(1;1) A(1;2)

... . . . . . . . . .A(n1;1) A(2;1) A(1;1) A(1;2)A(n;1) A(n1;1) A(2;1) A(1;1)

377777777775:

4.7 Direct sumFor any arbitrary matrices A (of size m n) and B (of size p q), we have the direct sum of A and B, denoted byA B and dened as

A B =

2666666664

a11 a1n 0 0... ... ... ...

am1 amn 0 00 0 b11 b1q... ... ... ...0 0 bp1 bpq

3777777775:

For instance,

1 3 22 3 1

1 60 1

=

26641 3 2 0 02 3 1 0 00 0 0 1 60 0 0 0 1

3775:

12 CHAPTER 4. BLOCK MATRIX

This operation generalizes naturally to arbitrary dimensioned arrays (provided that A and B have the same numberof dimensions).Note that any element in the direct sum of two vector spaces of matrices could be represented as a direct sum of twomatrices.

4.8 Direct productMain article: Kronecker product

4.9 ApplicationIn linear algebra terms, the use of a block matrix corresponds to having a linear mapping thought of in terms of corre-sponding 'bunches of basis vectors. That again matches the idea of having distinguished direct sum decompositionsof the domain and range. It is always particularly signicant if a block is the zero matrix; that carries the informationthat a summand maps into a sub-sum.Given the interpretation via linear mappings and direct sums, there is a special type of block matrix that occursfor square matrices (the case m = n). For those we can assume an interpretation as an endomorphism of an n-dimensional space V; the block structure in which the bunching of rows and columns is the same is of importancebecause it corresponds to having a single direct sum decomposition on V (rather than two). In that case, for example,the diagonal blocks in the obvious sense are all square. This type of structure is required to describe the Jordannormal form.This technique is used to cut down calculations of matrices, column-row expansions, and many computer scienceapplications, including VLSI chip design. An example is the Strassen algorithm for fast matrix multiplication, as wellas the Hamming(7,4) encoding for error detection and recovery in data transmissions.

4.10 Notes[1] Eves, Howard (1980). Elementary Matrix Theory (reprint ed.). New York: Dover. p. 37. ISBN 0-486-63946-0. Retrieved

24 April 2013. We shall nd that it is sometimes convenient to subdivide a matrix into rectangular blocks of elements.This leads us to consider so-called partitioned, or block, matrices.

[2] Anton, Howard (1994). Elementary Linear Algebra (7th ed.). New York: John Wiley. p. 30. ISBN 0-471-58742-7. Amatrix can be subdivided or partitioned into smaller matrices by inserting horizontal and vertical rules between selectedrows and columns.

[3] Eves, Howard (1980). Elementary Matrix Theory (reprint ed.). New York: Dover. p. 37. ISBN 0-486-63946-0. Retrieved24 April 2013. A partitioning as in Theorem 1.9.4 is called a conformable partition of A and B.

[4] Anton, Howard (1994). Elementary Linear Algebra (7th ed.). New York: John Wiley. p. 36. ISBN 0-471-58742-7....provided the sizes of the submatrices of A and B are such that the indicated operations can be performed.

[5] Bernstein, Dennis (2005). Matrix Mathematics. Princeton University Press. p. 44. ISBN 0-691-11802-7.

4.11 References Strang, Gilbert (1999). Lecture 3: Multiplication and inverse matrices. MIT Open Course ware. 18:3021:10.

Chapter 5

Reverse Cuthill-McKee algorithm

Cuthill-McKee ordering of a matrix

In the mathematical subeld of matrix theory, the CuthillMcKee algorithm (CM), named for Elizabeth Cuthill

13

14 CHAPTER 5. REVERSE CUTHILL-MCKEE ALGORITHM

RCM ordering of the same matrix

and J. McKee ,[1] is an algorithm to permute a sparse matrix that has a symmetric sparsity pattern into a band matrixform with a small bandwidth. The reverse CuthillMcKee algorithm (RCM) due to Alan George is the samealgorithm but with the resulting index numbers reversed. In practice this generally results in less ll-in than the CMordering when Gaussian elimination is applied.[2]

The Cuthill McKee algorithm is a variant of the standard breadth-rst search algorithm used in graph algorithms. Itstarts with a peripheral node and then generates levels Ri for i = 1; 2; :: until all nodes are exhausted. The set Ri+1is created from set Ri by listing all vertices adjacent to all nodes in Ri . These nodes are listed in increasing degree.This last detail is the only dierence with the breadth-rst search algorithm.

5.1 Algorithm

Given a symmetric n n matrix we visualize the matrix as the adjacency matrix of a graph. The CuthillMcKeealgorithm is then a relabeling of the vertices of the graph to reduce the bandwidth of the adjacency matrix.The algorithm produces an ordered n-tuple R of vertices which is the new order of the vertices.

5.2. SEE ALSO 15

First we choose a peripheral vertex (the vertex with the lowest degree) x and set R := ({x}).Then for i = 1; 2; : : : we iterate the following steps while |R| < n

Construct the adjacency set Ai of Ri (with Ri the i-th component of R) and exclude the vertices we alreadyhave in R

Ai := Adj(Ri) nR

Sort Ai with ascending vertex order (vertex degree). Append Ai to the Result set R.

In other words, number the vertices according to a particular breadth-rst traversal where neighboring vertices arevisited in order from lowest to highest vertex order.

5.2 See also Graph bandwidth Sparse matrix

5.3 References[1] E. Cuthill and J. McKee. Reducing the bandwidth of sparse symmetric matrices In Proc. 24th Nat. Conf. ACM, pages

157172, 1969.

[2] J. A. George and J. W-H. Liu, Computer Solution of Large Sparse Positive Denite Systems, Prentice-Hall, 1981

CuthillMcKee documentation for the Boost C++ Libraries. A detailed description of the CuthillMcKee algorithm. symrcm MATLABs implementation of RCM.

Chapter 6

Diagonal matrix

In linear algebra, a diagonal matrix is a matrix (usually a square matrix) in which the entries outside the maindiagonal () are all zero. The diagonal entries themselves may or may not be zero. Thus, the matrix D = (d,) withn columns and n rows is diagonal if:

di;j = 0 if i 6= j 8i; j 2 f1; 2; : : : ; ngFor example, the following matrix is diagonal:

241 0 00 4 00 0 2

35The term diagonal matrix may sometimes refer to a rectangular diagonal matrix, which is an m-by-n matrix withonly the entries of the form di,i possibly non-zero. For example:2664

1 0 00 4 00 0 30 0 0

3775 or241 0 0 0 00 4 0 0 00 0 3 0 0

35However, in the remainder of this article we will consider only square matrices. Any square diagonal matrix is alsoa symmetric matrix. Also, if the entries come from the eld R or C, then it is a normal matrix as well. Equivalently,we can dene a diagonal matrix as a matrix that is both upper- and lower-triangular. The identity matrix In and anysquare zero matrix are diagonal. A one-dimensional matrix is always diagonal.

6.1 Scalar matrixA diagonal matrix with all its main diagonal entries equal is a scalar matrix, that is, a scalar multiple I of the identitymatrix I. Its eect on a vector is scalar multiplication by . For example, a 33 scalar matrix has the form:

24 0 00 00 0

35 I3The scalar matrices are the center of the algebra of matrices: that is, they are precisely the matrices that commutewith all other square matrices of the same size.For an abstract vector space V (rather than the concrete vector space Kn ), or more generally a module M over aring R, with the endomorphism algebra End(M) (algebra of linear operators onM) replacing the algebra of matrices,

16

6.2. MATRIX OPERATIONS 17

the analog of scalar matrices are scalar transformations. Formally, scalar multiplication is a linear map, inducinga map R ! End(M); (send a scalar to the corresponding scalar transformation, multiplication by ) exhibitingEnd(M) as a R-algebra. For vector spaces, or more generally free modulesM = Rn , for which the endomorphismalgebra is isomorphic to a matrix algebra, the scalar transforms are exactly the center of the endomorphism algebra,and similarly invertible transforms are the center of the general linear group GL(V), where they are denoted by Z(V),follow the usual notation for the center.

6.2 Matrix operationsThe operations of matrix addition and matrix multiplication are especially simple for diagonal matrices. Writediag(a1,...,an) for a diagonal matrix whose diagonal entries starting in the upper left corner are a1,...,an. Then,for addition, we have

diag(a1,...,an) + diag(b1,...,bn) = diag(a1+b1,...,an+bn)

and for matrix multiplication,

diag(a1,...,an) diag(b1,...,bn) = diag(a1b1,...,anbn).

The diagonal matrix diag(a1,...,an) is invertible if and only if the entries a1,...,an are all non-zero. In this case, wehave

diag(a1,...,an)1 = diag(a11,...,an1).

In particular, the diagonal matrices form a subring of the ring of all n-by-n matrices.Multiplying an n-by-n matrix A from the left with diag(a1,...,an) amounts to multiplying the i-th row of A by ai forall i; multiplying the matrix A from the right with diag(a1,...,an) amounts to multiplying the i-th column of A by aifor all i.

6.3 Operator matrix in eigenbasisMain articles: Finding the matrix of a transformation and Eigenvalues and eigenvectors

As explained in determining coecients of operator matrix, there is a special basis, e1, ..., en, for which the matrixtakes the diagonal form. Being diagonal means that all coecients ai;j but ai;i are zeros in the dening equationA~ej =

Pai;j~ei , leaving only one term per sum. The surviving diagonal elements, ai;i , are known as eigenvalues

and designated with i in the equation, which reduces toA~ei = i~ei . The resulting equation is known as eigenvalueequation[1] and used to derive the characteristic polynomial and, further, eigenvalues and eigenvectors.In other words, the eigenvalues of diag(1, ..., n) are 1, ..., n with associated eigenvectors of e1, ..., en.

6.4 Other propertiesThe determinant of diag(a1, ..., an) is the product a1...an.The adjugate of a diagonal matrix is again diagonal.A square matrix is diagonal if and only if it is triangular and normal.

6.5 UsesDiagonal matrices occur in many areas of linear algebra. Because of the simple description of the matrix operationand eigenvalues/eigenvectors given above, it is always desirable to represent a given matrix or linear map by a diagonalmatrix.

18 CHAPTER 6. DIAGONAL MATRIX

In fact, a given n-by-n matrix A is similar to a diagonal matrix (meaning that there is a matrix X such that X1AX isdiagonal) if and only if it has n linearly independent eigenvectors. Such matrices are said to be diagonalizable.Over the eld of real or complex numbers, more is true. The spectral theorem says that every normal matrix isunitarily similar to a diagonal matrix (if AA* = A*A then there exists a unitary matrix U such that UAU* is diagonal).Furthermore, the singular value decomposition implies that for any matrix A, there exist unitary matrices U and Vsuch that UAV* is diagonal with positive entries.

6.6 Operator theoryIn operator theory, particularly the study of PDEs, operators are particularly easy to understand and PDEs easy tosolve if the operator is diagonal with respect to the basis with which one is working; this corresponds to a separablepartial dierential equation. Therefore, a key technique to understanding operators is a change of coordinates inthe language of operators, an integral transform which changes the basis to an eigenbasis of eigenfunctions: whichmakes the equation separable. An important example of this is the Fourier transform, which diagonalizes constantcoecient dierentiation operators (or more generally translation invariant operators), such as the Laplacian operator,say, in the heat equation.Especially easy are multiplication operators, which are dened as multiplication by (the values of) a xed function the values of the function at each point correspond to the diagonal entries of a matrix.

6.7 See also Anti-diagonal matrix Banded matrix Bidiagonal matrix Diagonally dominant matrix Diagonalizable matrix Multiplication operator Tridiagonal matrix Toeplitz matrix Toral Lie algebra Circulant matrix

6.8 References[1] Nearing, James (2010). Chapter 7.9: Eigenvalues and Eigenvectors (PDF). Mathematical Tools for Physics. ISBN

048648212X. Retrieved January 1, 2012.

Roger A. Horn and Charles R. Johnson, Matrix Analysis, Cambridge University Press, 1985. ISBN 0-521-30586-1 (hardback), ISBN 0-521-38632-2 (paperback).

Chapter 7

Generalized permutation matrix

Inmathematics, a generalized permutationmatrix (ormonomialmatrix) is amatrix with the same nonzero patternas a permutation matrix, i.e. there is exactly one nonzero entry in each row and each column. Unlike a permutationmatrix, where the nonzero entry must be 1, in a generalized permutation matrix the nonzero entry can be any nonzerovalue. An example of a generalized permutation matrix is

26640 0 3 00 2 0 01 0 0 00 0 0 1

3775:

7.1 StructureAn invertible matrix A is a generalized permutation matrix if and only if it can be written as a product of an invertiblediagonal matrix D and an (implicitly invertible) permutation matrix P: i.e.,

A = DP:

7.1.1 Group structureThe set of nn generalized permutation matrices with entries in a eld F forms a subgroup of the general linear groupGL(n,F), in which the group of nonsingular diagonal matrices (n, F) forms a normal subgroup. Indeed, the gener-alized permutation matrices are the normalizer of the diagonal matrices, meaning that the generalized permutationmatrices are the largest subgroup of GL in which diagonal matrices are normal.The abstract group of generalized permutation matrices is the wreath product of F and Sn. Concretely, this meansthat it is the semidirect product of (n, F) by the symmetric group Sn:

(n, F) Sn,

where Sn acts by permuting coordinates and the diagonal matrices (n, F) are isomorphic to the n-fold product (F)n.To be precise, the generalized permutation matrices are a (faithful) linear representation of this abstract wreathproduct: a realization of the abstract group as a subgroup of matrices.

7.1.2 Subgroups The subgroup where all entries are 1 is exactly the permutation matrices, which is isomorphic to the symmetricgroup.

19

20 CHAPTER 7. GENERALIZED PERMUTATION MATRIX

The subgroup where all entries are 1 is the signed permutation matrices, which is the hyperoctahedral group. The subgroup where the entries are mth roots of unity m is isomorphic to a generalized symmetric group. The subgroup of diagonal matrices is abelian, normal, and a maximal abelian subgroup. The quotient group isthe symmetric group, and this construction is in fact the Weyl group of the general linear group: the diagonalmatrices are a maximal torus in the general linear group (and are their own centralizer), the generalized per-mutation matrices are the normalizer of this torus, and the quotient, N(T )/Z(T ) = N(T )/T = Sn is theWeyl group.

7.2 Properties If a nonsingular matrix and its inverse are both nonnegative matrices (i.e. matrices with nonnegative entries),then the matrix is a generalized permutation matrix.

7.3 GeneralizationsOne can generalize further by allowing the entries to lie in a ring, rather than in a eld. In that case if the non-zeroentries are required to be units in the ring (invertible), one again obtains a group. On the other hand, if the non-zeroentries are only required to be non-zero, but not necessarily invertible, this set of matrices forms a semigroup instead.One may also schematically allow the non-zero entries to lie in a groupG,with the understanding that matrix multipli-cation will only involve multiplying a single pair of group elements, not adding group elements. This is an abuse ofnotation, since element of matrices being multiplied must allow multiplication and addition, but is suggestive notionfor the (formally correct) abstract group G o Sn (the wreath product of the group G by the symmetric group).

7.4 Signed permutation groupFurther information: Hyperoctahedral group

A signed permutation matrix is a generalized permutation matrix whose nonzero entries are 1, and are the integergeneralized permutation matrices with integer inverse.

7.4.1 Properties It is the Coxeter group Bn , and has order 2nn! . It is the symmetry group of the hypercube and (dually) of the cross-polytope. Its index 2 subgroup of matrices with determinant 1 is the Coxeter groupDn and is the symmetry group of thedemihypercube.

It is a subgroup of the orthogonal group.

7.5 Applications

7.5.1 Monomial representationsMain article: Monomial representation

Monomial matrices occur in representation theory in the context of monomial representations. A monomial represen-tation of a group G is a linear representation : GGL(n, F) of G (here F is the dening eld of the representation)such that the image (G) is a subgroup of the group of monomial matrices.

7.6. REFERENCES 21

7.6 References Joyner, David (2008). Adventures in group theory. Rubiks cube, Merlins machine, and other mathematical toys(2nd updated and revised ed.). Baltimore, MD: Johns Hopkins University Press. ISBN 978-0-8018-9012-3.Zbl 1221.00013.

Chapter 8

Identity matrix

In linear algebra, the identitymatrix or unitmatrix of size n is the n n squarematrix with ones on themain diagonaland zeros elsewhere. It is denoted by In, or simply by I if the size is immaterial or can be trivially determined by thecontext. (In some elds, such as quantum mechanics, the identity matrix is denoted by a boldface one, 1; otherwiseit is identical to I.) Less frequently, some mathematics books use U or E to represent the identity matrix, meaningunit matrix[1] and the German word Einheitsmatrix,[2] respectively.

I1 =1; I2 =

1 00 1

; I3 =

241 0 00 1 00 0 1

35; ; In =26666641 0 0 00 1 0 00 0 1 0... ... ... . . . ...0 0 0 1

3777775When A is mn, it is a property of matrix multiplication that

ImA = AIn = A:

In particular, the identity matrix serves as the unit of the ring of all nn matrices, and as the identity element of thegeneral linear group GL(n) consisting of all invertible nnmatrices. (The identity matrix itself is invertible, being itsown inverse.)Where nn matrices are used to represent linear transformations from an n-dimensional vector space to itself, Inrepresents the identity function, regardless of the basis.The ith column of an identity matrix is the unit vector ei. It follows that the determinant of the identity matrix is 1and the trace is n.Using the notation that is sometimes used to concisely describe diagonal matrices, we can write:

In = diag(1; 1; :::; 1):It can also be written using the Kronecker delta notation:

(In)ij = ij :

The identity matrix also has the property that, when it is the product of two square matrices, the matrices can be saidto be the inverse of one another.The identity matrix of a given size is the only idempotent matrix of that size having full rank. That is, it is the onlymatrix such that (a) when multiplied by itself the result is itself, and (b) all of its rows, and all of its columns, arelinearly independent.The principal square root of an identity matrix is itself, and this is its only positive denite square root. However,every identity matrix with at least two rows and columns has an innitude of symmetric square roots.[3]

22

8.1. SEE ALSO 23

8.1 See also Binary matrix Zero matrix Unitary matrix Matrix of ones

8.2 Notes[1] Pipes, Louis Albert (1963), Matrix Methods for Engineering, Prentice-Hall International Series in Applied Mathematics,

Prentice-Hall, p. 91.

[2] Identity Matrix on MathWorld;

[3] Mitchell, Douglas W. Using Pythagorean triples to generate square roots of I2". The Mathematical Gazette 87, November2003, 499-500.

8.3 External links Identity matrix at PlanetMath.org.

Chapter 9

Nested dissection

In numerical analysis, nested dissection is a divide and conquer heuristic for the solution of sparse symmetric systemsof linear equations based on graph partitioning. Nested dissection was introduced by George (1973); the name wassuggested by Garrett Birkho.[1]

Nested dissection consists of the following steps:

Form an undirected graph in which the vertices represent rows and columns of the system of linear equations,and an edge represents a nonzero entry in the sparse matrix representing the system.

Recursively partition the graph into subgraphs using separators, small subsets of vertices the removal of whichallows the graph to be partitioned into subgraphs with at most a constant fraction of the number of vertices.

Perform Cholesky decomposition (a variant of Gaussian elimination for symmetric matrices), ordering theelimination of the variables by the recursive structure of the partition: each of the two subgraphs formed byremoving the separator is eliminated rst, and then the separator vertices are eliminated.

As a consequence of this algorithm, the ll-in (the set of nonzeromatrix entries created in the Cholesky decompositionthat are not part of the input matrix structure) is limited to at most the square of the separator size at each level ofthe recursive partition. In particular, for planar graphs (frequently arising in the solution of sparse linear systemsderived from two-dimensional nite element method meshes) the resulting matrix has O(n log n) nonzeros, due to theplanar separator theorem guaranteeing separators of size O(n).[2] For arbitrary graphs there is a nested dissectionthat guarantees ll-in within a logarithmic factor of optimal, but this method is not guaranteed to achieve optimalll-in and nding the optimal dissection is not a solved problem.[3]

9.1 See also

Cycle rank of a graph, or a symmetric Boolean matrix, measures the minimum parallel time needed to performCholesky decomposition

Vertex separator

9.2 Notes[1] George (1973).

[2] Lipton, Rose & Tarjan (1979); Gilbert & Tarjan (1986).

[3] Gilbert (1988).

24

9.3. REFERENCES 25

9.3 References George, J. Alan (1973), Nested dissection of a regular nite element mesh, SIAM Journal on NumericalAnalysis 10 (2): 345363, doi:10.1137/0710032, JSTOR 2156361.

Gilbert, John R. (1988), Some nested dissection order is nearly optimal, Information Processing Letters 26(6): 325328, doi:10.1016/0020-0190(88)90191-3.

Gilbert, John R.; Tarjan, Robert E. (1986), The analysis of a nested dissection algorithm, Numerische Math-ematik 50 (4): 377404, doi:10.1007/BF01396660.

Lipton, Richard J.; Rose, Donald J.; Tarjan, Robert E. (1979), Generalized nested dissection, SIAM Journalon Numerical Analysis 16 (2): 346358, doi:10.1137/0716027, JSTOR 2156840.

Chapter 10

Pentadiagonal matrix

In linear algebra, a pentadiagonal matrix is a matrix that is nearly diagonal; to be exact, it is a matrix in which theonly nonzero entries are on the main diagonal, and the rst two diagonals above and below it. So it is of the form

0BBBBBBBBBBBBB@

c1 d1 e1 0 0b1 c2 d2 e2

. . . ...a1 b2

. . . . . . . . . . . . ...0 a2

. . . . . . . . . en3 0... . . . . . . . . . . . . dn2 en2... . . . an3 bn2 cn1 dn10 0 an2 bn1 cn

1CCCCCCCCCCCCCA:

It follows that a pentadiagonal matrix has at most 5n 6 nonzero entries, where n is the size of the matrix. Hence,pentadiagonal matrices are sparse. This makes them useful in numerical analysis.

10.1 See also tridiagonal matrix

This article incorporates material from Pentadiagonal matrix on PlanetMath, which is licensed under the CreativeCommons Attribution/Share-Alike License.

26

Chapter 11

Permutation matrix

Matrices describing the permutations of 3 elementsThe product of two permutation matrices is a permutation matrix as well.These are the positions of the six matrices:(They are also permutation matrices.)

27

28 CHAPTER 11. PERMUTATION MATRIX

In mathematics, in matrix theory, a permutation matrix is a square binary matrix that has exactly one entry 1 ineach row and each column and 0s elsewhere. Each such matrix represents a specic permutation of m elements and,when used to multiply another matrix, can produce that permutation in the rows or columns of the other matrix.

11.1 DenitionGiven a permutation of m elements,

: f1; : : : ;mg ! f1; : : : ;mg

given in two-line form by

1 2 m

(1) (2) (m);

its permutation matrix is the m m matrix P whose entries are all 0 except that in row i, the entry (i) equals 1.We may write

P =

26664e(1)e(2)...

e(m)

37775;where ej denotes a row vector of length m with 1 in the jth position and 0 in every other position.[1]

11.2 PropertiesGiven two permutations and of m elements and the corresponding permutation matrices P and P

PP = P

This somewhat unfortunate rule is a consequence of the denitions of multiplication of permutations (composition ofbijections) and of matrices, and of the choice of using the vectors e(i) as rows of the permutation matrix; if one hadused columns instead then the product above would have been equal to P with the permutations in their originalorder.As permutation matrices are orthogonal matrices (i.e., PPT = I ), the inverse matrix exists and can be written as

P1 = P1 = PT :

Multiplying P times a column vector g will permute the rows of the vector:

Pg =

26664e(1)e(2)...

e(n)

3777526664g1g2...gn

37775 =26664g(1)g(2)...

g(n)

37775:Now applying P after applying P gives the same result as applying P directly, in accordance with the abovemultiplication rule: call Pg = g0 , in other words

11.3. NOTES 29

g0i = g(i)

for all i, then

P(P(g)) = P(g0) =

26664g0(1)g0(2)...

g0(n)

37775 =26664g((1))g((2))

...g((n))

37775:Multiplying a row vector h times P will permute the columns of the vector by the inverse of P :

hP =h1 h2 : : : hn

26664e(1)e(2)...

e(n)

37775 = h1(1) h1(2) : : : h1(n)

Again it can be checked that (hP)P = hP .

11.3 NotesLet Sn denote the symmetric group, or group of permutations, on {1,2,...,n}. Since there are n! permutations, thereare n! permutation matrices. By the formulas above, the n n permutation matrices form a group under matrixmultiplication with the identity matrix as the identity element.If (1) denotes the identity permutation, then P is the identity matrix.One can view the permutation matrix of a permutation as the permutation of the columns of the identity matrixI, or as the permutation 1 of the rows of I.A permutation matrix is a doubly stochastic matrix. The Birkhovon Neumann theorem says that every doublystochastic real matrix is a convex combination of permutationmatrices of the same order and the permutationmatricesare precisely the extreme points of the set of doubly stochastic matrices. That is, the Birkho polytope, the set ofdoubly stochastic matrices, is the convex hull of the set of permutation matrices.[2]

The product PM, premultiplying a matrixM by a permutation matrix P, permutes the rows ofM; row i moves to row(i). Likewise, MP permutes the columns of M.The map Sn A GL(n, Z2) is a faithful representation. Thus, |A| = n!.The trace of a permutation matrix is the number of xed points of the permutation. If the permutation has xedpoints, so it can be written in cycle form as = (a1)(a2)...(ak) where has no xed points, then ea1,ea2,...,eak areeigenvectors of the permutation matrix.From group theory we know that any permutation may be written as a product of transpositions. Therefore, anypermutation matrix P factors as a product of row-interchanging elementary matrices, each having determinant 1.Thus the determinant of a permutation matrix P is just the signature of the corresponding permutation.

11.4 Examples

11.4.1 Permutation of rows and columnsWhen a permutation matrix P is multiplied with a matrixM from the left PM it will permute the rows ofM (here theelements of a column vector),when P is multiplied withM from the rightMP it will permute the columns ofM (here the elements of a row vector):Permutations of rows and columns are for example reections (see below) and cyclic permutations (see cyclic per-mutation matrix).

30 CHAPTER 11. PERMUTATION MATRIX

11.4.2 Permutation of rows

The permutation matrix P corresponding to the permutation : =1 2 3 4 51 4 2 5 3

; is

P =

266664e(1)e(2)e(3)e(4)e(5)

377775 =266664e1e4e2e5e3

377775 =2666641 0 0 0 00 0 0 1 00 1 0 0 00 0 0 0 10 0 1 0 0

377775:Given a vector g,

Pg =

266664e(1)e(2)e(3)e(4)e(5)

377775266664g1g2g3g4g5

377775 =266664g1g4g2g5g3

377775:

11.5 ExplanationA permutation matrix will always be in the form

26664ea1ea2...eaj

37775where eai represents the ith basis vector (as a row) for Rj , and where

1 2 : : : ja1 a2 : : : aj

is the permutation form of the permutation matrix.Now, in performing matrix multiplication, one essentially forms the dot product of each row of the rst matrix witheach column of the second. In this instance, we will be forming the dot product of each row of this matrix with thevector of elements we want to permute. That is, for example, v= (g0,...,g5)T,

eaiv=gai

So, the product of the permutation matrix with the vector v above, will be a vector in the form (ga1, ga2, ..., gaj),and that this then is a permutation of v since we have said that the permutation form is

1 2 : : : ja1 a2 : : : aj

:

So, permutation matrices do indeed permute the order of elements in vectors multiplied with them.

11.6 See also Alternating sign matrix Generalized permutation matrix

11.7. REFERENCES 31

11.7 References[1] Brualdi (2006) p.2

[2] Brualdi (2006) p.19

Brualdi, Richard A. (2006). Combinatorial matrix classes. Encyclopedia of Mathematics and Its Applications108. Cambridge: Cambridge University Press. ISBN 0-521-86565-4. Zbl 1106.05001.

Chapter 12

Reverse Cuthill-McKee algorithm

Cuthill-McKee ordering of a matrix

In the mathematical subeld of matrix theory, the CuthillMcKee algorithm (CM), named for Elizabeth Cuthill

32

12.1. ALGORITHM 33

RCM ordering of the same matrix

and J. McKee ,[1] is an algorithm to permute a sparse matrix that has a symmetric sparsity pattern into a band matrixform with a small bandwidth. The reverse CuthillMcKee algorithm (RCM) due to Alan George is the samealgorithm but with the resulting index numbers reversed. In practice this generally results in less ll-in than the CMordering when Gaussian elimination is applied.[2]

The Cuthill McKee algorithm is a variant of the standard breadth-rst search algorithm used in graph algorithms. Itstarts with a peripheral node and then generates levels Ri for i = 1; 2; :: until all nodes are exhausted. The set Ri+1is created from set Ri by listing all vertices adjacent to all nodes in Ri . These nodes are listed in increasing degree.This last detail is the only dierence with the breadth-rst search algorithm.

12.1 Algorithm

Given a symmetric n n matrix we visualize the matrix as the adjacency matrix of a graph. The CuthillMcKeealgorithm is then a relabeling of the vertices of the graph to reduce the bandwidth of the adjacency matrix.The algorithm produces an ordered n-tuple R of vertices which is the new order of the vertices.

34 CHAPTER 12. REVERSE CUTHILL-MCKEE ALGORITHM

First we choose a peripheral vertex (the vertex with the lowest degree) x and set R := ({x}).Then for i = 1; 2; : : : we iterate the following steps while |R| < n

Construct the adjacency set Ai of Ri (with Ri the i-th component of R) and exclude the vertices we alreadyhave in R

Ai := Adj(Ri) nR

Sort Ai with ascending vertex order (vertex degree). Append Ai to the Result set R.

In other words, number the vertices according to a particular breadth-rst traversal where neighboring vertices arevisited in order from lowest to highest vertex order.

12.2 See also Graph bandwidth Sparse matrix

12.3 References[1] E. Cuthill and J. McKee. Reducing the bandwidth of sparse symmetric matrices In Proc. 24th Nat. Conf. ACM, pages

157172, 1969.

[2] J. A. George and J. W-H. Liu, Computer Solution of Large Sparse Positive Denite Systems, Prentice-Hall, 1981

CuthillMcKee documentation for the Boost C++ Libraries. A detailed description of the CuthillMcKee algorithm. symrcm MATLABs implementation of RCM.

Chapter 13

Shear matrix

In mathematics, a shear matrix or transvection is an elementary matrix that represents the addition of a multipleof one row or column to another. Such a matrix may be derived by taking the identity matrix and replacing one ofthe zero elements with a non-zero value.A typical shear matrix is shown below:

S =

0BBBB@1 0 0 00 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 1

1CCCCA:The name shear reects the fact that the matrix represents a shear transformation. Geometrically, such a transforma-tion takes pairs of points in a linear space, that are purely axially separated along the axis whose row in the matrixcontains the shear element, and eectively replaces those pairs by pairs whose separation is no longer purely axialbut has two vector components. Thus, the shear axis is always an eigenvector of S.A shear parallel to the x axis results in x' = x + y and y' = y. In matrix form:

x0

y0

=

1 0 1

xy

:

Similarly, a shear parallel to the y axis has x' = x and y' = y + x. In matrix form:

x0

y0

=

1 0 1

xy

:

Clearly the determinant will always be 1, as no matter where the shear element is placed, it will be a member of askew-diagonal that also contains zero elements (as all skew-diagonals have length at least two) hence its product willremain zero and won't contribute to the determinant. Thus every shear matrix has an inverse, and the inverse is simplya shear matrix with the shear element negated, representing a shear transformation in the opposite direction. In fact,this is part of an easily derived more general result: if S is a shear matrix with shear element , then Sn is a shearmatrix whose shear element is simply n . Hence, raising a shear matrix to a power nmultiplies its shear factor by n.

13.1 PropertiesIf S is an nn shear matrix, then:

35

36 CHAPTER 13. SHEAR MATRIX

S has rank n and therefore is invertible 1 is the only eigenvalue of S, so det S = 1 and trace S = n the eigenspace of S has n-1 dimensions. S is asymmetric S may be made into a block matrix by at most 1 column interchange and 1 row interchange operation the area, volume, or any higher order interior capacity of a polytope is invariant under the shear transformationof the polytopes vertices.

13.2 Applications Shear matrices are often used in computer graphics.[1]

13.3 See also Transformation matrix

13.4 Notes[1] Foley et al. (1991, pp. 207208,216217)

13.5 References Foley, James D.; van Dam, Andries; Feiner, Steven K.; Hughes, John F. (1991), Computer Graphics: Principlesand Practice (2nd ed.), Reading: Addison-Wesley, ISBN 0-201-12110-7

Chapter 14

Shift matrix

In mathematics, a shift matrix is a binary matrix with ones only on the superdiagonal or subdiagonal, and zeroeselsewhere. A shift matrix U with ones on the superdiagonal is an upper shift matrix. The alternative subdiagonalmatrix L is unsurprisingly known as a lower shift matrix. The (i,j):th component of U and L are

Uij = i+1;j ; Lij = i;j+1;

where ij is the Kronecker delta symbol.For example, the 55 shift matrices are

U5 =

0BBBB@0 1 0 0 00 0 1 0 00 0 0 1 00 0 0 0 10 0 0 0 0

1CCCCA L5 =0BBBB@0 0 0 0 01 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 0

1CCCCA:

Clearly, the transpose of a lower shift matrix is an upper shift matrix and vice versa.Premultiplying a matrix A by a lower shift matrix results in the elements of A being shifted downward by one posi-tion, with zeroes appearing in the top row. Postmultiplication by a lower shift matrix results in a shift left. Similaroperations involving an upper shift matrix result in the opposite shift.Clearly all shift matrices are nilpotent; an n by n shift matrix S becomes the null matrix when raised to the power ofits dimension n.

14.1 PropertiesLet L and U be the n by n lower and upper shift matrices, respectively. The following properties hold for both U andL. Let us therefore only list the properties for U:

det(U) = 0 trace(U) = 0 rank(U) = n1 The characteristic polynomials of U is

pU () = (1)nn:

Un = 0. This follows from the previous property by the CayleyHamilton theorem.

37

38 CHAPTER 14. SHIFT MATRIX

The permanent of U is 0.

The following properties show how U and L are related:

LT = U; UT = L

The null spaces of U and L are

N(U) = spanf(1; 0; : : : ; 0)T g;N(L) = spanf(0; : : : ; 0; 1)T g:

The spectrum of U and L is f0g . The algebraic multiplicity of 0 is n, and its geometric multiplicity is 1. Fromthe expressions for the null spaces, it follows that (up to a scaling) the only eigenvector for U is (1; 0; : : : ; 0)T, and the only eigenvector for L is (0; : : : ; 0; 1)T .

For LU and UL we have

UL = I diag(0; : : : ; 0; 1);LU = I diag(1; 0; : : : ; 0):These matrices are both idempotent, symmetric, and have the same rank as U and L

Ln-aUn-a + LaUa = Un-aLn-a + UaLa = I (the identity matrix), for any integer a between 0 and n inclusive.

14.2 Examples

S =

0BBBB@0 0 0 0 01 0 0 0 00 1 0 0 00 0 1 0 00 0 0 1 0

1CCCCA; A =0BBBB@1 1 1 1 11 2 2 2 11 2 3 2 11 2 2 2 11 1 1 1 1

1CCCCA:

Then SA =

0BBBB@0 0 0 0 01 1 1 1 11 2 2 2 11 2 3 2 11 2 2 2 1

1CCCCA; AS =0BBBB@1 1 1 1 02 2 2 1 02 3 2 1 02 2 2 1 01 1 1 1 0

1CCCCA:Clearly there are many possible permutations. For example, STAS is equal to the matrix A shifted up and left alongthe main diagonal.

STAS =

0BBBB@2 2 2 1 02 3 2 1 02 2 2 1 01 1 1 1 00 0 0 0 0

1CCCCA:

14.3 See also Nilpotent matrix

14.4 References Shift Matrix - entry in the Matrix Reference Manual

Chapter 15

Single-entry matrix

In mathematics a single-entry matrix is a matrix where a single matrix and the rest of the elements are zero,[1][2]e.g.,

J23 =

240 0 00 0 10 0 0

35 :It is a specic type of a sparse matrix. The single-entry matrix can be regarded a row-selector when it is multipliedon the left side of the matrix, e.g.:

J23A =

24 0 0 0a31 a32 a330 0 0

35Alternatively, a column-selector when multiplied on the right side:

AJ23 =

240 0 a120 0 a220 0 a32

35The name, single-entry matrix, is not common, but seen in a few works.[3]

15.1 References[1] Kaare Brandt Petersen & Michael Syskind Pedersen (2008-02-16). The Matrix Cookbook.

[2] Shohei Shimizu, Patrick O. Hoyer, Aapo Hyvrinen & Antti Kerminen (2006). A Linear Non-Gaussian Acyclic Modelfor Causal Discovery. Journal of Machine Learning Research 7: 20032030.

[3] Examples:

Distributed Gain Matrix Optimization in Non-Regenerative MIMO Relay Networks. Marcel Blattner. B-Rank: A top N Recommendation Algorithm.

39

Chapter 16

Skyline matrix

In scientic computing, skyline matrix storage, or SKS, or a variable band matrix storage, or envelope storagescheme[1] is a form of a sparse matrix storage format matrix that reduces the storage requirement of a matrix morethan banded storage. In banded storage, all entries within a xed distance from the diagonal (called half-bandwidth)are stored. In column-oriented skyline storage, only the entries from the rst nonzero entry to the last nonzero entryin each column are stored. There is also row oriented skyline storage, and, for symmetric matrices, only one triangleis usually stored.[2]

Skyline storage has become very popular in the nite element codes for structural mechanics, because the skyline ispreserved by Cholesky decomposition (a method of solving systems of linear equations with a symmetric, positive-denite matrix; all ll-in falls within the skyline), and systems of equations from nite elements have a relativelysmall skyline. In addition, the eort of coding skyline Cholesky[3] is about same as for Cholesky for banded matrices(available for banded matrices, e.g. in LAPACK; for a prototype skyline code, see [3]).Before storing a matrix in skyline format, the rows and columns are typically renumbered to reduce the size of theskyline (the number of nonzero entries stored) and to decrease the number of operations in the skyline Choleskyalgorithm. The same heuristic renumbering algorithm that reduce the bandwidth are also used to reduce the skyline.The basic and one of the earliest algorithms to do that is reverse CuthillMcKee algorithm.However, skyline storage is not as popular for very large systems (many millions of equations) because skylineCholesky is not so easily adapted for massively parallel computing, and general sparse methods,[4] which store onlythe nonzero entries of the matrix, become more ecient for very large problems due to much less ll-in.

16.1 See also Sparse matrix Banded matrix Frontal solver

16.2 References[1] Watkins, David S. (2002), Fundamentals of matrix computations (Second ed.), New York: John Wiley & Sons, Inc., p. 60,

ISBN 0-471-21394-2[2] Barrett, Richard; Berry; Chan; Demmel; Donato; Dongarra; Eijkout; Pozo; Romine; Van der Vorst (1994), Skyline

Storage (SKS)", Templates for the solution of linear systems, SIAM, ISBN 0-89871-328-5[3] George, Alan; Liu, Joseph W. H. (1981), Computer solution of large sparse positive denite systems, Prentice-Hall Inc.,

ISBN 0-13-165274-5. The book also contains the description and source code of simple sparse matrix routines, still usefuleven if long superseded.

[4] Du, Iain S.; Erisman, Albert M.; Reid, John K. (1986),Direct methods for sparse matrices, Oxford University Press, ISBN0-19-853408-6

40

Chapter 17

Sparse matrix

A sparse matrix obtained when solving a nite element problem in two dimensions. The non-zero elements are shown in black.

In numerical analysis, a sparse matrix is a matrix in which most of the elements are zero. By contrast, if most ofthe elements are nonzero, then the matrix is considered dense. The fraction of zero elements over the total numberof elements in a matrix is called the sparsity (density).

41

42 CHAPTER 17. SPARSE MATRIX

Conceptually, sparsity corresponds to systems which are loosely coupled. Consider a line of balls connected by springsfrom one to the next: this is a sparse system as only adjacent balls are coupled. By contrast, if the same line of ballshad springs connecting each ball to all other balls, the system would correspond to a dense matrix. The concept ofsparsity is useful in combinatorics and application areas such as network theory, which have a low density of signicantdata or connections.Large sparse matrices often appear in scientic or engineering applications when solving partial dierential equations.When storing and manipulating sparse matrices on a computer, it is benecial and often necessary to use specializedalgorithms and data structures that take advantage of the sparse structure of the matrix. Operations using standarddense-matrix structures and algorithms are slow and inecient when applied to large sparse matrices as processingand memory are wasted on the zeroes. Sparse data is by nature more easily compressed and thus require signicantlyless storage. Some very large sparse matrices are infeasible to manipulate using standard dense-matrix algorithms.

17.1 Storing a sparse matrix

A matrix is typically stored as a two-dimensional array. Each entry in the array represents an element ai,j of thematrix and is accessed by the two indices i and j. Conventionally, i is the row index, numbered from top to bottom,and j is the column index, numbered from left to right. For an m nmatrix, the amount of memory required to storethe matrix in this format is proportional to m n (disregarding the fact that the dimensions of the matrix also needto be stored).In the case of a sparse matrix, substantial memory requirement reductions can be realized by storing only the non-zero entries. Depending on the number and distribution of the non-zero entries, dierent data structures can beused and yield huge savings in memory when compared to the basic approach. The caveat is that accessing theindividual elements becomes more complex and additional structures are needed to be able to recover the originalmatrix unambiguously.Formats can be divided into two groups:

Those that support ecient modication, such as DOK (Dictionary of keys), LIL (List of lists), or COO(Coordinate list). These are typically used to construct the matrices.

Those that support ecient access and matrix operations, such as CSR (Compressed Sparse Row) or CSC(Compressed Sparse Column).

17.1.1 Dictionary of keys (DOK)

DOK consists of a dictionary that maps (row, column)-pairs to the value of the elements. Elements that are missingfrom the dictionary are taken to be zero. The format is good for incrementally constructing a sparse matrix in randomorder, but poor for iterating over non-zero values in lexicographical order. One typically constructs a matrix in thisformat and then converts to another more ecient format for processing.[1]

17.1.2 List of lists (LIL)

LIL stores one list per row, with each entry containing the column index and the value. Typically, these entries arekept sorted by column index for faster lookup. This is another format good for incremental matrix construction.[2]

17.1.3 Coordinate list (COO)

COO stores a list of (row, column, value) tuples. Ideally, the entries are sorted (by row index, then column index) toimprove random access times. This is another format which is good for incremental matrix construction.[3]

17.1. STORING A SPARSE MATRIX 43

17.1.4 YaleThe Yale sparse matrix format stores an initial sparse m n matrix, M, in row form using three (one-dimensional)arrays (A, IA, JA). Let NNZ denote the number of nonzero entries inM. (Note that zero-based indices shall be usedhere.)

The array A is of length NNZ and holds all the nonzero entries of M in left-to-right top-to-bottom (row-major) order.

The array IA is of length m + 1 and contains the index in A of the rst element in each row, followed by thetotal number of nonzero elements NNZ. IA[i] contains the index in A of the rst nonzero element of row i.Row i of the original matrix extends from A[IA[i]] to A[IA[i + 1] 1], i.e. from the start of one row to thelast index before the start of the next. The last entry, IA[m], must be the number of elements in A.[4]

The third array, JA, contains the column index inM of each element of A and hence is of length NNZ as well.

For example, the matrix

0BB@0 0 0 05 8 0 00 0 3 00 6 0 0

1CCAis a 4 4 matrix with 4 nonzero elements, henceA = [ 5 8 3 6 ] IA = [ 0 0 2 3 4 ] JA = [ 0 1 2 1 ]So, in array JA, the element 5 from A has column index 0, 8 and 6 have index 1, and element 3 has index 2.In this case the Yale representation contains 13 entries, compared to 16 in the original matrix. The Yale format saveson memory only when NNZ < (m (n 1) 1) / 2. Another example, the matrix

0BB@10 20 0 0 0 00 30 0 40 0 00 0 50 60 70 00 0 0 0 0 80

1CCAis a 4 6 matrix (24 entries) with 8 nonzero elements, soA = [ 10 20 30 40 50 60 70 80 ] IA = [ 0 2 4 7 8 ] JA = [ 0 1 1 3 2 3 4 5 ]The whole is stored as 21 entries.

IA splits the array A into rows: (10, 20) (30, 40) (50, 60, 70) (80);

JA aligns values in columns: (10, 20, ...) (0, 30, 0, 40, ...)(0, 0, 50, 60, 70, 0) (0, 0, 0, 0, 0, 80).

Note that in this format, the rst value of IA is always zero and the last is always NNZ, so they are in some senseredundant. However, they can make accessing and traversing the array easier for the programmer.

17.1.5 Compressed row Storage (CRS or CSR)CSR is eectively identical to the Yale Sparse Matrix format, except that the column array is normally stored ahead ofthe row index array. I.e. CSR is (val, col_ind, row_ptr), where val is an array of the (left-to-right, then top-to-bottom)non-zero values of the matrix; col_ind is the column indices corresponding to the values; and, row_ptr is the list ofvalue indexes where each row starts. The name is based on the fact that row index information is compressed relativeto the COO format. One typically uses another format (LIL, DOK, COO) for construction. This format is ecientfor arithmetic operations, row slicing, and matrix-vector products. See scipy.sparse.csr_matrix.

44 CHAPTER 17. SPARSE MATRIX

17.1.6 Compressed sparse column (CSC or CCS)CSC is similar to CSR except that values are read rst by column, a row index is stored for each value, and columnpointers are stored. I.e. CSC is (val, row_ind, col_ptr), where val is an array of the (top-to-bottom, then left-to-right)non-zero values of the matrix; row_ind is the row indices corresponding to the values; and, col_ptr is the list ofval indexes where each column starts. The name is based on the fact that column index information is compressedrelative to the COO format. One typically uses another format (LIL, DOK, COO) for construction. This format isecient for arithmetic operations, column slicing, and matrix-vector products. See scipy.sparse.csc_matrix. This isthe traditional format for specifying a sparse matrix in MATLAB (via the sparse function).

17.2 Special structure

17.2.1 BandedMain article: Band matrix

An important special type of sparse matrices is band matrix, dened as follows. The lower bandwidth of a matrix Ais the smallest number p such that the entry ai,j vanishes whenever i > j + p. Similarly, the upper bandwidth is thesmallest number p such that ai,j = 0 whenever i < j p (Golub & Van Loan 1996, 1.2.1). For example, a tridiagonalmatrix has lower bandwidth 1 and upper bandwidth 1. As another example, the following sparse matrix has lowerand upper bandwidth both equal to 3. Notice that zeros are represented with dots for clarity.

0B@X X X X X X X X X X X X X X X X X X X X X X X

1CAMatrices with reasonably small upper and lower bandwidth are known as band matrices and often lend themselvesto simpler algorithms than general sparse matrices; or one can sometimes apply dense matrix algorithms and gaineciency simply by looping over a reduced number of indices.By rearranging the rows and columns of a matrix A it may be possible to obtain a matrix A with a lower bandwidth.A number of algorithms are designed for bandwidth minimization.

17.2.2 DiagonalA very ecient structure for an extreme case of band matrices, the diagonal matrix, is to store just the entries in themain diagonal as a one-dimensional array, so a diagonal n n matrix requires only n entries.

17.2.3 SymmetricA symmetric sparse matrix arises as the adjacency matrix of an undirected graph; it can be stored eciently as anadjacency list.

17.3 Reducing ll-inThe ll-in of a matrix are those entries which change from an initial zero to a non-zero value during the execution ofan algorithm. To reduce the memory requirements and the number of arithmetic operations used during an algorithmit is useful to minimize the ll-in by switching rows and columns in the matrix. The symbolic Cholesky decompositioncan be used to calculate the worst possible ll-in before doing the actual Cholesky decomposition.There are other methods than the Cholesky decomposition in use. Orthogonalization methods (such as QR factor-ization) are common, for example, when solving problems by least squares methods. While the theoretical ll-in is

17.4. SOLVING SPARSE MATRIX EQUATIONS 45

still the same, in practical terms the false non-zeros can be dierent for dierent methods. And symbolic versionsof those algorithms can be used in the same manner as the symbolic Cholesky to compute worst case ll-in.

17.4 Solving sparse matrix equationsBoth iterative and direct methods exist for sparse matrix solving.Iterative methods, such as conjugate gradient method and GMRES utilize fast computations of matrix-vector productsAxi , where matrix A is sparse. The use of preconditioners can signicantly accelerate convergence of such iterativemethods.

17.5 See also

17.6 References Golub, Gene H.; Van Loan, Charles F. (1996). Matrix Computations (3rd ed.). Baltimore: Johns Hopkins.ISBN 978-0-8018-5414-9.

Stoer, Josef; Bulirsch, Roland (2002). Introduction to Numerical Analysis (3rd ed.). Berlin, New York:Springer-Verlag. ISBN 978-0-387-95452-3.

Tewarson, Reginald P. (May 1973). Sparse Matrices (Part of the Mathematics in Science & Engineering series).Academic Press Inc. (This book, by a professor at the State University of New York at Stony Book, was therst book exclusively dedicated to Sparse Matrices. Graduate courses using this as a textbook were oered atthat University in the early 1980s).

Bank, Randolph E.; Douglas, Craig C. Sparse Matrix Multiplication Package (PDF).

Pissanetzky, Sergio (1984). Sparse Matrix Technology. Academic Press.

Snay, Richard A. (1976). Reducing the prole of sparse symmetric matrices. Bulletin Godsique 50 (4):341. doi:10.1007/BF02521587. Also NOAA Technical Memorandum NOS NGS-4, National Geodetic Sur-vey, Rockville, MD.

[1] See scipy.org

[2] See scipy.org

[3] See scipy.org

[4] netlib.org

17.7 Further reading Gibbs, Norman E.; Poole, William G.; Stockmeyer, Paul K. (1976). A comparison of several bandwidth andprole reduction algorithms. ACMTransactions onMathematical Software 2 (4): 322330. doi:10.1145/355705.355707.

Gilbert, John R.; Moler, Cleve; Schreiber, Robert (1992). Sparse matrices in MATLAB: Design and Imple-mentation. SIAM Journal on Matrix Analysis and Applications 13 (1): 333356. doi:10.1137/0613024.

Sparse Matrix Algorithms Research at the University of Florida, containing the UF sparse matrix collection.

SMALL project A EU-funded project on sparse models, algorithms and dictionary learning for large-scaledata.

46 CHAPTER 17. SPARSE MATRIX

17.8 External links Equations Solver Online Oral history interviewwithHarryM.Markowitz, Charles Babbage Institute, University ofMinnesota. Markowitzdiscusses his development of portfolio theory (for which he received a Nobel Prize in Economics), sparse ma-trix methods, and his work at the RAND Corporation and elsewhere on simulation software development(including computer language SIMSCRIPT), modeling, and operations research.

Chapter 18

Sparse matrix-vector multiplication

Sparse matrix-vector multiplication (SpMV) of the form y = Ax is a widely used computational kernel existingin many scientic applications. The input matrix A is sparse. The input vector x and the output vector y are dense.In case of repeated y = Ax operation involving the same input matrix A but possibly changing numerical values ofits elements, A can be preprocessed to reduce both the parallel and sequential run time of the SpMV kernel.[1]

18.1 References[1] Hypergraph Partitioning Based Models and Methods for Exploiting Cache Locality in Sparse Matrix-Vector Multiplica-

tion. Retrieved 13 April 2014.

47

Chapter 19

Tridiagonal matrix

In linear algebra, a tridiagonal matrix is a matrix that has nonzero elements only on the main diagonal, the rstdiagonal below this, and the rst diagonal above the main diagonal.For example, the following matrix is tridiagonal:

0BB@1 4 0 03 4 1 00 2 3 40 0 1 3

1CCA:

The determinant of a tridiagonal matrix is given by the continuant of its elements.[1]

An orthogonal transformation of a symmetric (or Hermitian) matrix to tridiagonal form can be done with the Lanczosalgorithm.

19.1 Properties

A tridiagonal matrix is a matrix that is both upper and lower Hessenberg matrix.[2] In particular, a tridiagonal matrixis a direct sum of p 1-by-1 and q 2-by-2 matrices such that p + q/2 = n -- the dimension of the tridiagonal. Althougha general tridiagonal matrix is not necessarily symmetric or Hermitian, many of those that arise when solving linearalgebra problems have one of these properties. Furthermore, if a real tridiagonal matrix A satises ak,k ak,k > 0for all k, so that the signs of its entries are symmetric, then it is similar to a Hermitian matrix, by a diagonal change ofbasis matrix. Hence, its eigenvalues are real. If we replace the strict inequality by ak,k ak,k 0, then by continuity,the eigenvalues are still guaranteed to be real, but the matrix need no longer be similar to a Hermitian matrix.[3]

The set of all n n tridiagonal matrices forms a 3n-2 dimensional vector space.Many linear algebra algorithms require signicantly less computational eort when applied to diagonal matrices, andthis improvement often carries over to tridiagonal matrices as well.

19.1.1 Determinant

Main article: continuant (mathematics)

The determinant of a tridiagonal matrix A of order n can be computed from a three-term recurrence relation.[4] Writef1 = |a1| = a1 and

48

19.1. PROPERTIES 49

fn =

a1 b1c1 a2 b2

c2. . . . . .. . . . . . bn1

cn1 an

:

The sequence (fi) is called the continuant and satises the recurrence relation

fn = anfn1 cn1bn1fn2

with initial values f0 = 1 and f = 0. The cost of computing the determinant of a tridiagonal matrix using thisformula is linear in n, while the cost is cubic for a general matrix.

19.1.2 Inversion

The inverse of a non-singular tridiagonal matrix T

T =

0BBBBBB@a1 b1c1 a2 b2

c2. . . . . .. . . . . . bn1

cn1 an

1CCCCCCAis given by

(T1)ij =

((1)i+jbi bj1i1j+1/n if i j(1)i+jcj ci1j1i+1/n if i > j

where the i satisfy the recurrence relation

i = aii1 bi1ci1i2 for i = 2; 3; : : : ; n

with initial conditions 0 = 1, 1 = a1 and the i satisfy

i = aii+1 bicii+2 for i = n 1; : : : ; 1

with initial conditions n = 1 and n = an.[5][6]

Closed form solutions can be computed for special cases such as symmetric matrices with all o-diagonal elementsequal[7] or Toeplitz matrices[8] and for the general case as well.[9][10]

In general, the inverse of a tridiagonal matrix is a semiseparable matrix and vice versa.[11]

50 CHAPTER 19. TRIDIAGONAL MATRIX

19.1.3 Solution of linear systemMain article: tridiagonal matrix algorithm

A system of equationsA x = b for b2Rn can be solved by an ecient form of Gaussian elimination whenA is tridiagonalcalled tridiagonal matrix algorithm, requiring O(n) operations.[12]

19.1.4 EigenvaluesWhen a tridiagonal matrix is also Toeplitz, there is a simple closed-form solution for its eigenvalues, namely a +2pbc cos(k/(n+ 1)) , for k = 1; :::; n: [13][14]

19.2 Computer programmingA transformation that reduces a general matrix to Hessenberg form will reduce a Hermitian matrix to tridiagonalform. So, many eigenvalue algorithms, when applied to a Hermitian matrix, reduce the input Hermitian matrix totridiagonal form as a rst step.A tridiagonal matrix can also be stored more eciently than a general matrix by using a special storage scheme. Forinstance, the LAPACK Fortran package stores an unsymmetric tridiagonal matrix of order n in three one-dimensionalarrays, one of length n containing the diagonal elements, and two of length n 1 containing the subdiagonal andsuperdiagonal elements.

19.3 See also Pentadiagonal matrix

19.4 Notes[1] Thomas Muir (1960). A treatise on the theory of determinants