direct methods for sparse linear systems
DESCRIPTION
Direct Methods for Sparse Linear Systems. Lecture 4 Alessandra Nardi. Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy. Last lecture review. Solution of system of linear equations Existence and uniqueness review Gaussian elimination basics - PowerPoint PPT PresentationTRANSCRIPT
Direct Methods for Sparse Linear Systems
Lecture 4
Alessandra Nardi
Thanks to Prof. Jacob White, Suvranu De, Deepak Ramaswamy, Michal Rewienski, and Karen Veroy
Last lecture review
• Solution of system of linear equations
• Existence and uniqueness review
• Gaussian elimination basics– GE basics– LU factorization– Pivoting
Outline
• Error Mechanisms
• Sparse Matrices – Why are they nice– How do we store them– How can we exploit and preserve sparsity
Error Mechanisms
• Round-off error– Pivoting helps
• Ill conditioning (almost singular)– Bad luck: property of the matrix– Pivoting does not help
• Numerical Stability of Method
Ill-Conditioning : Norms
• Norms useful to discuss error in numerical problems
• Norm V:
Vyxyxyx
Vxxx
Vxxx
, if )3(
, if )2(
,0 if 0 )1(
L2 (Euclidean) norm :
n
iixx
1
2
2
L1 norm :
L norm :
n
iixx
11
ii
xx max
12x
Unit circle
Unit square
1
1 11x
1
x
Ill-Conditioning : Vector Norms
Vector induced norm : Axx
AxA
xx 1maxmax
n
iij
jAA
11
max
Induced norm of A is the maximum “magnification” of by
= max abs column sum
n
jij
iAA
1
max = max abs row sum
2A = (largest eigenvalue of ATA)1/2
x A
Ill-Conditioning : Matrix Norms
Ill-Conditioning : Matrix Norms
• More properties on the matrix norm:
• Condition Number:
-It can be shown that:-Large (A) means matrix is almost singular (ill-conditioned)
BAAB
I
1
)( 1 AAA
1)( A
What happens if we perturb b?
Ill-Conditioning: Perturbation Analysis
xxxbbb
b
bM
x
x
b
bMM
x
x
xMb
bM
b
bbMx
bMx
bMx
bxM
bbxMMx
bbxxM
)(
)(
1
11
1
1
(M) large is bad
What happens if we perturb M?
Ill-Conditioning: Perturbation Analysis
xxxMMM
M
MM
xx
x
M
MMM
xx
x
bxxMM
)(
))((
1
(M) large is bad
Bottom line:If matrix is ill-conditioned, round-off puts you in troubles
Geometric Approach is more intuitive
1 2 1 1 2 2[ ], Solving is finding M M M M x b x M x M b
2
2|| ||
M
M
1
1|| ||
M
M
2
2|| ||
M
M
1
1|| ||
M
M
When vectors are nearly aligned,Hard to decide how much of versus how much of 1M
2M
Vectors are orthogonal
1x
2x
1x
2x
Ill-Conditioning: Perturbation Analysis
Vectors are nearly aligned
Numerical Stability
• Rounding errors may accumulate and propagate unstably in a bad algorithm.
• Can be proven that for Gaussian elimination the accumulated error is bounded
Summary on Error Mechanisms for GE
• Rounding: due to machine finite precision we have an error in the solution even if the algorithm is perfect– Pivoting helps to reduce it
• Matrix conditioning– If matrix is “good”, then complete pivoting solves any round-
off problem– If matrix is “bad” (almost singular), then there is nothing to
do
• Numerical stability– How rounding errors accumulate– GE is stable
LU – Computational Complexity
• Computational Complexity– O(n3), where M: n x n
• We cannot afford this complexity
• Exploit natural sparsity that occurs in circuits equations– Sparsity: many zero elements– Matrix is sparse when it is advantageous to exploit
its sparsity
• Exploiting sparsity: O(n1.1) to O(n1.5)
LU – Goals of exploiting sparsity
(1) Avoid storing zero entries– Memory usage reduction– Decomposition is faster since you do need to access
them (but more complicated data structure)
(2) Avoid trivial operations– Multiplication by zero– Addition with zero
(3) Avoid losing sparsity
1 2 3 4 1m m
X X
X X X
X X X
X X X
X X X
X X X
X X X
X X
m
Sparse Matrices – Resistor Line
Tridiagonal Case
For i = 1 to n-1 { “For each Row”
For j = i+1 to n { “For each target Row below the source”
For k = i+1 to n { “For each Row element beyond Pivot”
} }}
Pivotjiji
ii
MM
M
Multiplier
jk jk ji ikM M M M
Order N Operations!
GE Algorithm – Tridiagonal Example
1R
5R
3R
4R
2R1V 2V
3V
1Si
1
1
R 2
1
R
2
1
R
2
1
R
2
1
R 3
1
R
4
1
R
4
1
R
4
1
R
4
1
R 5
1
R
SymmetricDiagonally Dominant
Nodal Matrix0
Sparse Matrices – Fill-in – Example 1
13
2
1
0
0
sie
e
e
X X X
X X 0
X 0 X
X X X
X X 0
X 0 X
X
X
X X
X= Non zero
Matrix Non zero structure Matrix after one LU step
X X
Sparse Matrices – Fill-in – Example 1
X X X X
X X 0 0
0 X X 0
X 0 00
Fill-ins Propagate
XX
X
X
X
X X
X
X X
Fill-ins from Step 1 result in Fill-ins in step 2
Sparse Matrices – Fill-in – Example 2
3V
Node Reordering Can Reduce Fill-in - Preserves Properties (Symmetry, Diagonal Dominance) - Equivalent to swapping rows and columns
1V 2V
0
x x x
x x x
x x x
Fill-ins
2V 1V
3V
0
x x 0
x x x
0 x x
No Fill-ins
Sparse Matrices – Fill-in & Reordering
Exploiting and maintaining sparsity• Criteria for exploiting sparsity:
– Minimum number of ops– Minimum number of fill-ins
• Pivoting to maintain sparsity: NP-complete problem heuristics are used– Markowitz, Berry, Hsieh and Ghausi, Nakhla and Singhal and
Vlach– Choice: Markowitz
• 5% more fill-ins but• Faster
• Pivoting for accuracy may conflict with pivoting for sparsity
Where can fill-in occur ?
x x x
x x x
x x x
Multipliers
Already Factored
Possible Fill-inLocations
Fill-in Estimate = (Non zeros in unfactored part of Row -1) (Non zeros in unfactored part of Col -1) Markowitz product
Sparse Matrices – Fill-in & Reordering
Markowitz Reordering (Diagonal Pivoting)
tor oF 1i n diagonal with min Markowitz PFind roductj i
rows and columns Swa p ij j i
the new row and determine fiFactor ll-insi
End
Greedy Algorithm (but close to optimal) !
Sparse Matrices – Fill-in & Reordering
Why only try diagonals ?
• Corresponds to node reordering in Nodal formulation
03
1 2
02
3 1
• Reduces search cost • Preserves Matrix Properties - Diagonal Dominance - Symmetry
Sparse Matrices – Fill-in & Reordering
Pattern of a Filled-in Matrix
Very Sparse
Very Sparse
Dense
Sparse Matrices – Fill-in & Reordering
Sparse Matrices – Fill-in & Reordering
Unfactored random Matrix
Sparse Matrices – Fill-in & Reordering
Factored random Matrix
Sparse Matrices – Data Structure
• Several ways of storing a sparse matrix in a compact form
• Trade-off– Storage amount– Cost of data accessing and update procedures
• Efficient data structure: linked list
Sparse Matrices – Data Structure 1
Orthogonal linked list
Val 11
Col 11
Val 12
Col 12
Val 1K
Col 1K
Val 21
Col 21
Val 22
Col 22
Val 2L
Col 2L
Val N1
Col N1
Val N2
Col N2
Val Nj
Col Nj
1
N
Arrays of Data in a RowVector of row pointers
Matrix entriesColumn index
Sparse Matrices – Data Structure 2
, 1 , 4 , 5 , 7 , 9 , 12 , 15
1 4 5 7 9 12 15i i i i i i i i i i i i i iM M M M M M M
i i i i i i i
, 1 , 7 , 15
1 7 15i i i i i iM M M
i i i
Eliminating Source Row i from Target row j
Row i
Row j
Must read all the row j entries to find the 3 that match row iEvery Miss is an unneeded memory reference (expensive!!!)
Could have more misses than ops!
Sparse Matrices – Data StructureProblem of Misses
, 1 , 4 , 5 , 7 , 9 , 12 , 15
1 4 5 7 9 12 15i i i i i i i i i i i i i iM M M M M M M
i i i i i i i
Row j
, 1i iM , 4i iM , 5i iM , 7i iM , 9i iM , 12i iM , 15i iM
1) Read all the elements in Row j, and scatter them in an n-length vector2) Access only the needed elements using array indexing!
Sparse Matrices – Data Structure
Scattering for Miss Avoidance
X
X
X
X
X
X
X X
X
XX
X
X
X X 1
2
4
3
5
• One Node Per Matrix Row
• One Edge Per Off-diagonal Pair
X
X
Structurally Symmetric Matrices and Graphs
Sparse Matrices – Graph Approach
X
X
X
X
X
X
X X
X
XX
X
X
X X
X
X 1
2
4
3
5
Markowitz Products = (Node Degree)2
2(degree 1) 92(deg ree 2) 42(deg ree 3) 9
2(degree 4) 42(degree 5) 4
Sparse Matrices – Graph ApproachMarkowitz Products
93311 M42222 M93333 M42244 M42255 M
X
X
X
X
X
X
X X
X
XX
X
X
X X 1
2
4
3
5
• Delete the node associated with pivot row
• “Tie together” the graph edges
X
X
X
X
X
One Step of LU Factorization
Sparse Matrices – Graph ApproachFactorization
x x x x
x x x
x x x x
x x x x
x x x x
1 2 3 4 5
Graph
Markowitz products( = Node degree)
Col Row
3 3 9
2 2 4
3 3 9
3 3 9
3 3 9
1
2
3
4
5
Sparse Matrices – Graph ApproachExample
x x x
x x x x x
x x x x x
x x x x
x x x x
1 3 4 5
Graph
Swap 2 with 1
Sparse Matrices – Graph ApproachExample
• Gaussian Elimination Error Mechanisms– Ill-conditioning
– Numerical Stability
• Gaussian Elimination for Sparse Matrices – Improved computational cost: factor in O(N1.5)
operations (dense is O(N3) )
– Example: Tridiagonal Matrix Factorization O(N)
– Data structure
– Markowitz Reordering to minimize fill-ins
– Graph Based Approach
Summary