Download - Cos323 s06 Lecture06 Linsys2
Solving Linear Systems:Solving Linear Systems:Iterative Methods and Sparse Iterative Methods and Sparse
SystemsSystems
COS 323COS 323
Direct vs. Iterative MethodsDirect vs. Iterative Methods
• So far, have looked at So far, have looked at direct methodsdirect methods forforsolving linear systemssolving linear systems– Predictable number of stepsPredictable number of steps
– No answer until the very endNo answer until the very end
• Alternative: Alternative: iterative methodsiterative methods– Start with approximate answerStart with approximate answer
– Each iteration improves accuracyEach iteration improves accuracy
– Stop once estimated error below toleranceStop once estimated error below tolerance
Benefits of Iterative AlgorithmsBenefits of Iterative Algorithms
• Some iterative algorithms designed for Some iterative algorithms designed for accuracy:accuracy:– Direct methods subject to roundoff errorDirect methods subject to roundoff error
– Iterate to reduce error to O(Iterate to reduce error to O( ))
• Some algorithms produce answer fasterSome algorithms produce answer faster– Most important class: Most important class: sparse matrix sparse matrix solverssolvers
– Speed depends on # of Speed depends on # of nonzerononzero elements, elements,not total # of elementsnot total # of elements
• Today: iterative improvement of accuracy,Today: iterative improvement of accuracy,solving sparse systems (not necessarily solving sparse systems (not necessarily iteratively)iteratively)
Iterative ImprovementIterative Improvement
• Suppose you’ve solved (or think you’ve Suppose you’ve solved (or think you’ve solved) some system Ax=bsolved) some system Ax=b
• Can check answer by computing Can check answer by computing residualresidual::
r = b – Ax r = b – Axcomputedcomputed
• If r is small (compared to b), x is If r is small (compared to b), x is accurateaccurate
• What if it’s not?What if it’s not?
Iterative ImprovementIterative Improvement
• Large residual caused by error in x:Large residual caused by error in x: e = x e = xcorrectcorrect – x – xcomputedcomputed
• If we knew the error, could try to improve x:If we knew the error, could try to improve x: x xcorrectcorrect = x = xcomputedcomputed + e + e
• Solve for error:Solve for error:AxAxcomputedcomputed = A(x = A(xcorrectcorrect – e) = b – r – e) = b – r
AxAxcorrectcorrect – Ae = b – r – Ae = b – r
Ae = r Ae = r
Iterative ImprovementIterative Improvement
• So, compute residual, solve for e,So, compute residual, solve for e,and apply correction to estimate of xand apply correction to estimate of x
• If original system solved using LU,If original system solved using LU,this is relatively fast (relative to O(nthis is relatively fast (relative to O(n33), ), that is):that is):– O(nO(n22) matrix/vector multiplication +) matrix/vector multiplication +
O(n) vector subtraction to solve for rO(n) vector subtraction to solve for r
– O(nO(n22) forward/backsubstitution to solve for e) forward/backsubstitution to solve for e
– O(n) vector addition to correct estimate of xO(n) vector addition to correct estimate of x
Sparse SystemsSparse Systems
• Many applications require solution ofMany applications require solution oflarge linear systems (n = thousands to large linear systems (n = thousands to millions)millions)– Local constraints or interactions: most entries Local constraints or interactions: most entries
are 0are 0
– Wasteful to store all nWasteful to store all n22 entries entries
– Difficult or impossible to use O(nDifficult or impossible to use O(n33) algorithms) algorithms
• Goal: solve system with:Goal: solve system with:– Storage proportional to # of Storage proportional to # of nonzerononzero elements elements
– Running time << nRunning time << n33
Special Case: Band DiagonalSpecial Case: Band Diagonal
• Last time: tridiagonal (or band diagonal) Last time: tridiagonal (or band diagonal) systemssystems– Storage O(n): only relevant diagonalsStorage O(n): only relevant diagonals
– Time O(n): Gauss-Jordan with bookkeepingTime O(n): Gauss-Jordan with bookkeeping
Cyclic Tridiagonal Cyclic Tridiagonal
• Interesting extension: cyclic tridiagonalInteresting extension: cyclic tridiagonal
• Could derive yet another special case Could derive yet another special case algorithm,algorithm,but there’s a better waybut there’s a better way
bx
aaa
aaa
aaa
aaa
aaa
aaa
666561
565554
454443
343332
232221
161211
bx
aaa
aaa
aaa
aaa
aaa
aaa
666561
565554
454443
343332
232221
161211
Updating InverseUpdating Inverse
• Suppose we have some fast way of Suppose we have some fast way of finding Afinding A-1-1
for some matrix Afor some matrix A
• Now A changes in a special way:Now A changes in a special way: A* = A + uv A* = A + uvTT
for some nfor some n1 vectors u and v1 vectors u and v
• Goal: find a fast way of computing (A*)Goal: find a fast way of computing (A*)-1-1
– Eventually, a fast way of solving (A*)Eventually, a fast way of solving (A*) x = bx = b
Sherman-Morrison FormulaSherman-Morrison Formula
xAAx
AAx
Ax
AAIA
AIAAA
T1T12
T1T12
T1
11T11*
T1T*
itCallScalar!
Let
)(
)(
uvvu
vuvuthatNote
uv
uv
uvuv
xAAx
AAx
Ax
AAIA
AIAAA
T1T12
T1T12
T1
11T11*
T1T*
itCallScalar!
Let
)(
)(
uvvu
vuvuthatNote
uv
uv
uvuv
Sherman-Morrison FormulaSherman-Morrison Formula
uv
uv1T
1T111*
1
2
1
1
1
1
01
1
A
AAAA
xIx
I
IxIx
I
IxIx
xI
xIx
x
xxIx
xx
uv
uv1T
1T111*
1
2
1
1
1
1
01
1
A
AAAA
xIx
I
IxIx
I
IxIx
xI
xIx
x
xxIx
xx
Sherman-Morrison FormulaSherman-Morrison Formula
zv
yvzyxuzby
bx
uv
bvubbx
T
T
*
1T
1T111*
1,,solve
,solvetoSo,
1
AA
A
A
AAAA
zv
yvzyxuzby
bx
uv
bvubbx
T
T
*
1T
1T111*
1,,solve
,solvetoSo,
1
AA
A
A
AAAA
Applying Sherman-MorrisonApplying Sherman-Morrison
• Let’s considerLet’s considercyclic tridiagonal again:cyclic tridiagonal again:
• TakeTake
bx
aaa
aaa
aaa
aaa
aaa
aaa
666561
565554
454443
343332
232221
161211
bx
aaa
aaa
aaa
aaa
aaa
aaa
666561
565554
454443
343332
232221
161211
166116616665
565554
454443
343332
232221
1211 1
,
1
,
1
a
v
a
u
aaaa
aaa
aaa
aaa
aaa
aa
A
166116616665
565554
454443
343332
232221
1211 1
,
1
,
1
a
v
a
u
aaaa
aaa
aaa
aaa
aaa
aa
A
Applying Sherman-MorrisonApplying Sherman-Morrison
• Solve Ay=b, Az=u using special fast Solve Ay=b, Az=u using special fast algorithmalgorithm
• Applying Sherman-Morrison takesApplying Sherman-Morrison takesa couple of dot productsa couple of dot products
• Total: O(n) timeTotal: O(n) time
• Generalization for several corrections: Generalization for several corrections: WoodburyWoodbury 1T1T111*
T*
AVUAVIUAAA
UVAA
1T1T111*
T*
AVUAVIUAAA
UVAA
More General Sparse MatricesMore General Sparse Matrices
• More generally, we can represent More generally, we can represent sparse matrices by noting which sparse matrices by noting which elements are nonzeroelements are nonzero
• Critical for Ax and ACritical for Ax and ATTx to be efficient:x to be efficient:proportional to # of nonzero elementsproportional to # of nonzero elements– We’ll see an algorithm for solving Ax=bWe’ll see an algorithm for solving Ax=b
using only these two operations!using only these two operations!
Compressed Sparse Row FormatCompressed Sparse Row Format
• Three arraysThree arrays– Values: actual numbers in the matrixValues: actual numbers in the matrix
– Cols: column of corresponding entry in Cols: column of corresponding entry in valuesvalues
– Rows: index of first entry in each rowRows: index of first entry in each row
– Example: (zero-based)Example: (zero-based)
3210
0000
5002
3230
3210
0000
5002
3230
values values 3 2 3 2 5 1 2 33 2 3 2 5 1 2 3colscols 1 2 3 0 3 1 2 31 2 3 0 3 1 2 3rowsrows 0 3 5 5 80 3 5 5 8
Compressed Sparse Row FormatCompressed Sparse Row Format
• Multiplying Ax:Multiplying Ax:
for (i = 0; i < n; i++) {for (i = 0; i < n; i++) {out[i] = 0;out[i] = 0;for (j = rows[i]; j < rows[i+1]; j++)for (j = rows[i]; j < rows[i+1]; j++)
out[i] += values[j] * x[ cols[j] ];out[i] += values[j] * x[ cols[j] ];}}
3210
0000
5002
3230
3210
0000
5002
3230values values 3 2 3 2 5 1 2 33 2 3 2 5 1 2 3colscols 1 2 3 0 3 1 2 31 2 3 0 3 1 2 3rowsrows 0 3 5 5 80 3 5 5 8
Solving Sparse SystemsSolving Sparse Systems
• Transform problem to a function Transform problem to a function minimization!minimization!
Solve Ax=bSolve Ax=b Minimize f(x) = x Minimize f(x) = xTTAx – 2bAx – 2bTTxx
• To motivate this, consider 1D:To motivate this, consider 1D:f(x) = axf(x) = ax22 – 2bx – 2bx
dfdf//dx dx = 2ax – 2b = 0= 2ax – 2b = 0
ax = b ax = b
Solving Sparse SystemsSolving Sparse Systems
• Preferred method: Preferred method: conjugate gradientsconjugate gradients
• Recall: plain gradient Recall: plain gradient descent has a descent has a problem…problem…
Solving Sparse SystemsSolving Sparse Systems
• … … that’s solved by that’s solved by conjugate gradientsconjugate gradients
• Walk along directionWalk along direction
• Polak and Ribiere Polak and Ribiere formula:formula:
kkkk dgd 11 kkkk dgd 11
kk
kkk gg
gggk
T
1T )(
1
kk
kkk gg
gggk
T
1T )(
1
Solving Sparse SystemsSolving Sparse Systems
• Easiest to think about A = symmetricEasiest to think about A = symmetric
• First ingredient: need to evaluate First ingredient: need to evaluate gradientgradient
• As advertised, this only involves A As advertised, this only involves A multipliedmultipliedby a vectorby a vector
bxxf
xbxxxf
A
A
2)(
2)( TT
bxxf
xbxxxf
A
A
2)(
2)( TT
Solving Sparse SystemsSolving Sparse Systems
• Second ingredient: given point xSecond ingredient: given point xii, ,
direction ddirection dii,,
minimize function in that directionminimize function in that direction
iii
ii
ii
iiiii
ii
iii
dtxx
dd
bxdt
ddtbxddt
tdm
tmdt
dtm
dtxftm
min1
T
T
min
wantTT 022
)(
0)(:)(Minimize
)()(Define
A
A
AA
iii
ii
ii
iiiii
ii
iii
dtxx
dd
bxdt
ddtbxddt
tdm
tmdt
dtm
dtxftm
min1
T
T
min
wantTT 022
)(
0)(:)(Minimize
)()(Define
A
A
AA
Solving Sparse SystemsSolving Sparse Systems
• So, each iteration just requires a fewSo, each iteration just requires a fewsparse matrix – vector multipliessparse matrix – vector multiplies(plus some dot products, etc.)(plus some dot products, etc.)
• If matrix is nIf matrix is nn and has m nonzero entries,n and has m nonzero entries,
each iteration is Oeach iteration is O((max(m,n)max(m,n)))
• Conjugate gradients may need n iterations Conjugate gradients may need n iterations forfor“perfect” convergence, but often get “perfect” convergence, but often get decent answer well before thendecent answer well before then