![Page 1: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/1.jpg)
The Landscape of Ax=b SolversThe Landscape of Ax=b Solvers
Pivoting
LU
GMRES, QMR, …
Cholesky
Conjugate gradient
DirectA = LU
Iterativey’ = Ay
Non-symmetric
Symmetricpositivedefinite
More Robust Less Storage (if sparse)
More Robust
More General
D
![Page 2: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/2.jpg)
Conjugate gradient iterationConjugate gradient iteration
• One matrix-vector multiplication per iteration• Two vector dot products per iteration• Four n-vectors of working storage
x0 = 0, r0 = b, p0 = r0
for k = 1, 2, 3, . . .
αk = (rTk-1rk-1) / (pT
k-1Apk-1) step length
xk = xk-1 + αk pk-1 approx solution
rk = rk-1 – αk Apk-1 residual
βk = (rTk rk) / (rT
k-1rk-1) improvement
pk = rk + βk pk-1 search direction
![Page 3: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/3.jpg)
Conjugate gradient: Krylov subspacesConjugate gradient: Krylov subspaces
• Eigenvalues: Au = λu { λ1, λ2 , . . ., λn}
• Cayley-Hamilton theorem:
(A – λ1I)·(A – λ2I) · · · (A – λnI) = 0
Therefore Σ ciAi = 0 for some ci
so A-1 = Σ (–ci/c0) Ai–1
• Krylov subspace:
Therefore if Ax = b, then x = A-1 b and
x span (b, Ab, A2b, . . ., An-1b) = Kn (A, b)
0 i n
1 i n
![Page 4: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/4.jpg)
Conjugate gradient: Orthogonal sequencesConjugate gradient: Orthogonal sequences
• Krylov subspace: Ki (A, b) = span (b, Ab, A2b, . . ., Ai-1b)
• Conjugate gradient algorithm:for i = 1, 2, 3, . . .
find xi Ki (A, b)
such that ri = (Axi – b) Ki (A, b)
• Notice ri Ki+1 (A, b), so ri rj for all j < i
• Similarly, the “directions” are A-orthogonal:(xi – xi-1 )T·A· (xj – xj-1 ) = 0
• The magic: Short recurrences. . .A is symmetric => can get next residual and direction from the previous one, without saving them all.
![Page 5: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/5.jpg)
Conjugate gradient: ConvergenceConjugate gradient: Convergence
• In exact arithmetic, CG converges in n steps (completely unrealistic!!)
• Accuracy after k steps of CG is related to:• consider polynomials of degree k that are equal to 1 at 0.• how small can such a polynomial be at all the eigenvalues of A?
• Thus, eigenvalues close together are good.
• Condition number: κ(A) = ||A||2 ||A-1||2 = λmax(A) / λmin(A)
• Residual is reduced by a constant factor by O(κ1/2(A)) iterations of CG.
![Page 6: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/6.jpg)
(Matlab demo)(Matlab demo)
• CG on grid5(15) and bcsstk08• n steps of CG on bcsstk08
![Page 7: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/7.jpg)
• Lay out matrix and vectors by rows
• Hard part is matrix-vector product y = A*x
• AlgorithmEach processor j: Broadcast x(j) Compute y(j) = A(j,:)*x
• May send more of x than needed
• Partition / reorder matrix to reduce communication
x
y
P0
P1
P2
P3
P0 P1 P2 P3
Conjugate gradient: Parallel implementationConjugate gradient: Parallel implementation
![Page 8: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/8.jpg)
(Matlab demo)(Matlab demo)
• 2-way partition of eppstein mesh • 8-way dice of eppstein mesh
![Page 9: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/9.jpg)
PreconditionersPreconditioners
• Suppose you had a matrix B such that:
1. condition number κ(B-1A) is small2. By = z is easy to solve
• Then you could solve (B-1A)x = B-1b instead of Ax = b
• B = A is great for (1), not for (2)• B = I is great for (2), not for (1)• Domain-specific approximations sometimes work• B = diagonal of A sometimes works
• Or, bring back the direct methods technology. . .
![Page 10: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/10.jpg)
(Matlab demo)(Matlab demo)
• bcsstk08 with diagonal precond
![Page 11: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/11.jpg)
Incomplete Cholesky factorization (IC, ILU)Incomplete Cholesky factorization (IC, ILU)
• Compute factors of A by Gaussian elimination, but ignore fill
• Preconditioner B = RTR A, not formed explicitly
• Compute B-1z by triangular solves (in time nnz(A))
• Total storage is O(nnz(A)), static data structure
• Either symmetric (IC) or nonsymmetric (ILU)
x
A RT R
![Page 12: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/12.jpg)
(Matlab demo)(Matlab demo)
• bcsstk08 with ic precond
![Page 13: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/13.jpg)
Incomplete Cholesky and ILU: VariantsIncomplete Cholesky and ILU: Variants
• Allow one or more “levels of fill”• unpredictable storage requirements
• Allow fill whose magnitude exceeds a “drop tolerance”• may get better approximate factors than levels of fill• unpredictable storage requirements• choice of tolerance is ad hoc
• Partial pivoting (for nonsymmetric A)
• “Modified ILU” (MIC): Add dropped fill to diagonal of U or R• A and RTR have same row sums• good in some PDE contexts
2 3
1 4
2 3
1 4
![Page 14: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/14.jpg)
Incomplete Cholesky and ILU: IssuesIncomplete Cholesky and ILU: Issues
• Choice of parameters• good: smooth transition from iterative to direct methods• bad: very ad hoc, problem-dependent• tradeoff: time per iteration (more fill => more time)
vs # of iterations (more fill => fewer iters)
• Effectiveness• condition number usually improves (only) by constant factor
(except MIC for some problems from PDEs)• still, often good when tuned for a particular class of problems
• Parallelism• Triangular solves are not very parallel• Reordering for parallel triangular solve by graph coloring
![Page 15: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/15.jpg)
(Matlab demo)(Matlab demo)
• 2-coloring of grid5(15)
![Page 16: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/16.jpg)
Sparse approximate inversesSparse approximate inverses
• Compute B-1 A explicitly
• Minimize || B-1A – I ||F (in parallel, by columns)
• Variants: factored form of B-1, more fill, . .
• Good: very parallel
• Bad: effectiveness varies widely
A B-1
![Page 17: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/17.jpg)
Support graph preconditioners: example Support graph preconditioners: example [Vaidya]
• A is symmetric positive definite with negative off-diagonal nzs
• B is a maximum-weight spanning tree for A (with diagonal modified to preserve row sums)
• factor B in O(n) space and O(n) time
• applying the preconditioner costs O(n) time per iteration
G(A) G(B)
![Page 18: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/18.jpg)
• support each edge of A by a path in B
• dilation(A edge) = length of supporting path in B
• congestion(B edge) = # of supported A edges
• p = max congestion, q = max dilation
• condition number κ(B-1A) bounded by p·q (at most O(n2))
G(A) G(B)
Support graph preconditioners: example Support graph preconditioners: example
![Page 19: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/19.jpg)
• can improve congestion and dilation by adding a few
strategically chosen edges to B
• cost of factor+solve is O(n1.75), or O(n1.2) if A is planar
• in recent experiments [Chen & Toledo], often better than
drop-tolerance MIC for 2D problems, but not for 3D.
G(A) G(B)
Support graph preconditioners: example Support graph preconditioners: example
![Page 20: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/20.jpg)
Domain decomposition Domain decomposition (introduction)(introduction)
• Partition the problem (e.g. the mesh) into subdomains
• Use solvers for the subdomains B-1 and C-1 to precondition an iterative solver on the interface
• Interface system is the Schur complement:
S = G – ET B-1E – FT C-1F
• Parallelizes naturally by subdomains
B 0 E
0 C F
ET FT G
A =
![Page 21: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/21.jpg)
(Matlab demo)(Matlab demo)
• grid and matrix structure for overlapping 2-way partition of eppstein
![Page 22: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/22.jpg)
Multigrid Multigrid (introduction)(introduction)
• For a PDE on a fine mesh, precondition using a solution on a coarser mesh
• Use idea recursively on hierarchy of meshes• Solves the model problem (Poisson’s eqn) in linear time!• Often useful when hierarchy of meshes can be built• Hard to parallelize coarse meshes well
• This is just the intuition – lots of theory and technology
![Page 23: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/23.jpg)
Other Krylov subspace methodsOther Krylov subspace methods
• Nonsymmetric linear systems:• GMRES:
for i = 1, 2, 3, . . . find xi Ki (A, b) such that ri = (Axi – b) Ki (A, b)
But, no short recurrence => save old vectors => lots more space
• BiCGStab, QMR, etc.:Two spaces Ki (A, b) and Ki (AT, b) w/ mutually orthogonal bases
Short recurrences => O(n) space, but less robust
• Convergence and preconditioning more delicate than CG• Active area of current research
• Eigenvalues: Lanczos (symmetric), Arnoldi (nonsymmetric)
![Page 24: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/24.jpg)
The Landscape of Sparse Ax=b SolversThe Landscape of Sparse Ax=b Solvers
Pivoting
LU
GMRES, QMR, …
Cholesky
Conjugate gradient
DirectA = LU
Iterativey’ = Ay
Non-symmetric
Symmetricpositivedefinite
More Robust Less Storage
More Robust
More General
![Page 25: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/25.jpg)
Complexity of direct methodsComplexity of direct methods
n1/2 n1/3
2D 3D
Space (fill): O(n log n) O(n 4/3 )
Time (flops): O(n 3/2 ) O(n 2 )
Time and space to solve any problem on any well-shaped finite element mesh
![Page 26: The Landscape of Ax=b Solvers Direct A = LU Iterative y’ = Ay Non- symmetric Symmetric positive definite More RobustLess Storage (if sparse) More Robust](https://reader036.vdocuments.mx/reader036/viewer/2022062407/56649d555503460f94a331a7/html5/thumbnails/26.jpg)
Complexity of linear solversComplexity of linear solvers
2D 3DSparse Cholesky: O(n1.5 ) O(n2 )
CG, exact arithmetic: O(n2 ) O(n2 )
CG, no precond: O(n1.5 ) O(n1.33 )
CG, modified IC: O(n1.25 ) O(n1.17 )
CG, support trees: O(n1.20 ) O(n1.75 )
Multigrid: O(n) O(n)
n1/2 n1/3
Time to solve model problem (Poisson’s equation) on regular mesh