cos323 s06 lecture06 linsys2

Solving Linear Systems:Solving Linear Systems:Iterative Methods and Sparse Iterative Methods and Sparse

SystemsSystems

COS 323COS 323

Direct vs. Iterative MethodsDirect vs. Iterative Methods

• So far, have looked at So far, have looked at direct methodsdirect methods forforsolving linear systemssolving linear systems– Predictable number of stepsPredictable number of steps

– No answer until the very endNo answer until the very end

• Alternative: Alternative: iterative methodsiterative methods– Start with approximate answerStart with approximate answer

– Each iteration improves accuracyEach iteration improves accuracy

– Stop once estimated error below toleranceStop once estimated error below tolerance

Benefits of Iterative AlgorithmsBenefits of Iterative Algorithms

• Some iterative algorithms designed for Some iterative algorithms designed for accuracy:accuracy:– Direct methods subject to roundoff errorDirect methods subject to roundoff error

– Iterate to reduce error to O(Iterate to reduce error to O( ))

• Some algorithms produce answer fasterSome algorithms produce answer faster– Most important class: Most important class: sparse matrix sparse matrix solverssolvers

– Speed depends on # of Speed depends on # of nonzerononzero elements, elements,not total # of elementsnot total # of elements

• Today: iterative improvement of accuracy,Today: iterative improvement of accuracy,solving sparse systems (not necessarily solving sparse systems (not necessarily iteratively)iteratively)

Iterative ImprovementIterative Improvement

• Suppose you’ve solved (or think you’ve Suppose you’ve solved (or think you’ve solved) some system Ax=bsolved) some system Ax=b

• Can check answer by computing Can check answer by computing residualresidual::

r = b – Ax r = b – Axcomputedcomputed

• If r is small (compared to b), x is If r is small (compared to b), x is accurateaccurate

• What if it’s not?What if it’s not?


• Large residual caused by error in x:Large residual caused by error in x: e = x e = xcorrectcorrect – x – xcomputedcomputed

• If we knew the error, could try to improve x:If we knew the error, could try to improve x: x xcorrectcorrect = x = xcomputedcomputed + e + e

• Solve for error:Solve for error:AxAxcomputedcomputed = A(x = A(xcorrectcorrect – e) = b – r – e) = b – r

AxAxcorrectcorrect – Ae = b – r – Ae = b – r

Ae = r Ae = r


• So, compute residual, solve for e,So, compute residual, solve for e,and apply correction to estimate of xand apply correction to estimate of x

• If original system solved using LU,If original system solved using LU,this is relatively fast (relative to O(nthis is relatively fast (relative to O(n33), ), that is):that is):– O(nO(n22) matrix/vector multiplication +) matrix/vector multiplication +

O(n) vector subtraction to solve for rO(n) vector subtraction to solve for r

– O(nO(n22) forward/backsubstitution to solve for e) forward/backsubstitution to solve for e

– O(n) vector addition to correct estimate of xO(n) vector addition to correct estimate of x

Sparse SystemsSparse Systems

• Many applications require solution ofMany applications require solution oflarge linear systems (n = thousands to large linear systems (n = thousands to millions)millions)– Local constraints or interactions: most entries Local constraints or interactions: most entries

are 0are 0

– Wasteful to store all nWasteful to store all n22 entries entries

– Difficult or impossible to use O(nDifficult or impossible to use O(n33) algorithms) algorithms

• Goal: solve system with:Goal: solve system with:– Storage proportional to # of Storage proportional to # of nonzerononzero elements elements

– Running time << nRunning time << n33

Special Case: Band DiagonalSpecial Case: Band Diagonal

• Last time: tridiagonal (or band diagonal) Last time: tridiagonal (or band diagonal) systemssystems– Storage O(n): only relevant diagonalsStorage O(n): only relevant diagonals

– Time O(n): Gauss-Jordan with bookkeepingTime O(n): Gauss-Jordan with bookkeeping

Cyclic Tridiagonal Cyclic Tridiagonal

• Interesting extension: cyclic tridiagonalInteresting extension: cyclic tridiagonal

• Could derive yet another special case Could derive yet another special case algorithm,algorithm,but there’s a better waybut there’s a better way

bx

aaa

aaa

aaa

aaa

aaa

aaa

666561

565554

454443

343332

232221

161211

bx

aaa

aaa

aaa

aaa

aaa

aaa

666561

565554

454443

343332

232221

161211

Updating InverseUpdating Inverse

• Suppose we have some fast way of Suppose we have some fast way of finding Afinding A-1-1

for some matrix Afor some matrix A

• Now A changes in a special way:Now A changes in a special way: A* = A + uv A* = A + uvTT

for some nfor some n1 vectors u and v1 vectors u and v

• Goal: find a fast way of computing (A*)Goal: find a fast way of computing (A*)-1-1

– Eventually, a fast way of solving (A*)Eventually, a fast way of solving (A*) x = bx = b

Sherman-Morrison FormulaSherman-Morrison Formula

xAAx

AAx

Ax

AAIA

AIAAA

T1T12

T1T12

T1

11T11*

T1T*

itCallScalar!

Let

)(

)(

uvvu

vuvuthatNote

uv

uv

uvuv

xAAx

AAx

Ax

AAIA

AIAAA

T1T12

T1T12

T1

11T11*

T1T*

itCallScalar!

Let

)(

)(

uvvu

vuvuthatNote

uv

uv

uvuv


uv

uv1T

1T111*

1

2

1

1

1

1

01

1

A

AAAA

xIx

I

IxIx

I

IxIx

xI

xIx

x

xxIx

xx

uv

uv1T

1T111*

1

2

1

1

1

1

01

1

A

AAAA

xIx

I

IxIx

I

IxIx

xI

xIx

x

xxIx

xx


zv

yvzyxuzby

bx

uv

bvubbx

T

T

*

1T

1T111*

1,,solve

,solvetoSo,

1

AA

A

A

AAAA

zv

yvzyxuzby

bx

uv

bvubbx

T

T

*

1T

1T111*

1,,solve

,solvetoSo,

1

AA

A

A

AAAA

Applying Sherman-MorrisonApplying Sherman-Morrison

• Let’s considerLet’s considercyclic tridiagonal again:cyclic tridiagonal again:

• TakeTake

bx

aaa

aaa

aaa

aaa

aaa

aaa

666561

565554

454443

343332

232221

161211

bx

aaa

aaa

aaa

aaa

aaa

aaa

666561

565554

454443

343332

232221

161211

166116616665

565554

454443

343332

232221

1211 1

,

1

,

1

a

v

a

u

aaaa

aaa

aaa

aaa

aaa

aa

A

166116616665

565554

454443

343332

232221

1211 1

,

1

,

1

a

v

a

u

aaaa

aaa

aaa

aaa

aaa

aa

A

Applying Sherman-MorrisonApplying Sherman-Morrison

• Solve Ay=b, Az=u using special fast Solve Ay=b, Az=u using special fast algorithmalgorithm

• Applying Sherman-Morrison takesApplying Sherman-Morrison takesa couple of dot productsa couple of dot products

• Total: O(n) timeTotal: O(n) time

• Generalization for several corrections: Generalization for several corrections: WoodburyWoodbury 1T1T111*

T*

AVUAVIUAAA

UVAA

1T1T111*

T*

AVUAVIUAAA

UVAA

More General Sparse MatricesMore General Sparse Matrices

• More generally, we can represent More generally, we can represent sparse matrices by noting which sparse matrices by noting which elements are nonzeroelements are nonzero

• Critical for Ax and ACritical for Ax and ATTx to be efficient:x to be efficient:proportional to # of nonzero elementsproportional to # of nonzero elements– We’ll see an algorithm for solving Ax=bWe’ll see an algorithm for solving Ax=b

using only these two operations!using only these two operations!

Compressed Sparse Row FormatCompressed Sparse Row Format

• Three arraysThree arrays– Values: actual numbers in the matrixValues: actual numbers in the matrix

– Cols: column of corresponding entry in Cols: column of corresponding entry in valuesvalues

– Rows: index of first entry in each rowRows: index of first entry in each row

– Example: (zero-based)Example: (zero-based)

3210

0000

5002

3230

3210

0000

5002

3230

values values 3 2 3 2 5 1 2 33 2 3 2 5 1 2 3colscols 1 2 3 0 3 1 2 31 2 3 0 3 1 2 3rowsrows 0 3 5 5 80 3 5 5 8

Compressed Sparse Row FormatCompressed Sparse Row Format

• Multiplying Ax:Multiplying Ax:

for (i = 0; i < n; i++) {for (i = 0; i < n; i++) {out[i] = 0;out[i] = 0;for (j = rows[i]; j < rows[i+1]; j++)for (j = rows[i]; j < rows[i+1]; j++)

out[i] += values[j] * x[ cols[j] ];out[i] += values[j] * x[ cols[j] ];}}

3210

0000

5002

3230

3210

0000

5002

3230values values 3 2 3 2 5 1 2 33 2 3 2 5 1 2 3colscols 1 2 3 0 3 1 2 31 2 3 0 3 1 2 3rowsrows 0 3 5 5 80 3 5 5 8

Solving Sparse SystemsSolving Sparse Systems

• Transform problem to a function Transform problem to a function minimization!minimization!

Solve Ax=bSolve Ax=b Minimize f(x) = x Minimize f(x) = xTTAx – 2bAx – 2bTTxx

• To motivate this, consider 1D:To motivate this, consider 1D:f(x) = axf(x) = ax22 – 2bx – 2bx

dfdf//dx dx = 2ax – 2b = 0= 2ax – 2b = 0

ax = b ax = b


• Preferred method: Preferred method: conjugate gradientsconjugate gradients

• Recall: plain gradient Recall: plain gradient descent has a descent has a problem…problem…


• … … that’s solved by that’s solved by conjugate gradientsconjugate gradients

• Walk along directionWalk along direction

• Polak and Ribiere Polak and Ribiere formula:formula:

kkkk dgd 11 kkkk dgd 11

kk

kkk gg

gggk

T

1T )(

1

kk

kkk gg

gggk

T

1T )(

1


• Easiest to think about A = symmetricEasiest to think about A = symmetric

• First ingredient: need to evaluate First ingredient: need to evaluate gradientgradient

• As advertised, this only involves A As advertised, this only involves A multipliedmultipliedby a vectorby a vector

bxxf

xbxxxf

A

A

2)(

2)( TT

bxxf

xbxxxf

A

A

2)(

2)( TT


• Second ingredient: given point xSecond ingredient: given point xii, ,

direction ddirection dii,,

minimize function in that directionminimize function in that direction

iii

ii

ii

iiiii

ii

iii

dtxx

dd

bxdt

ddtbxddt

tdm

tmdt

dtm

dtxftm

min1

T

T

min

wantTT 022

)(

0)(:)(Minimize

)()(Define

A

A

AA

iii

ii

ii

iiiii

ii

iii

dtxx

dd

bxdt

ddtbxddt

tdm

tmdt

dtm

dtxftm

min1

T

T

min

wantTT 022

)(

0)(:)(Minimize

)()(Define

A

A

AA


• So, each iteration just requires a fewSo, each iteration just requires a fewsparse matrix – vector multipliessparse matrix – vector multiplies(plus some dot products, etc.)(plus some dot products, etc.)

• If matrix is nIf matrix is nn and has m nonzero entries,n and has m nonzero entries,

each iteration is Oeach iteration is O((max(m,n)max(m,n)))

• Conjugate gradients may need n iterations Conjugate gradients may need n iterations forfor“perfect” convergence, but often get “perfect” convergence, but often get decent answer well before thendecent answer well before then

cos323 s06 lecture06 linsys2

Documents

b r ae

sparse systemscos

iterative methodsstart

iterative methodsso

special way

b axcomputedif r

fast way of computing

b r axcorrect ae