numerical methods rafał zdunek iterative...
TRANSCRIPT
![Page 1: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/1.jpg)
NumericalNumerical
MethodsMethods
Rafał ZdunekRafał Zdunek
Iterative MethodsIterative Methods
(4h.)(4h.)
![Page 2: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/2.jpg)
Introduction
• Stationary basic iterative methods, • Krylov subspace methods,• Nonnegative matrix factorization,• Multi-dimensional array decomposition
methods (tensor decompositions).
![Page 3: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/3.jpg)
Bibliography[1] A. Bjorck, Numerical Methods for Least Squares Problems, SIAM,
Philadelphia, 1996, [2] G. Golub, C. F. Van Loan, Matrix Computations, The John Hopkins
University Press, (Third Edition), 1996, [3] J. Stoer
R. Bulirsch, Introduction to Numerical Analysis (Second Edition),
Springer-Verlag, 1993, [4] C. D. Meyer, Matrix Analysis and Applied Linear Algebra, SIAM, 2000,[5] Ch. Zarowski, An Introduction to Numerical Analysis for Electrical and
Computer Engineers, Wiley, 2004,[6] Cichocki, R. Zdunek, A. H. Phan, S.-I. Amari, Nonnegative Matrix and
Tensor Factorization: Applications to Exploratory Multi-way Data Analysis and Blind Source Separation, Wiley and Sons, UK, 2009
![Page 4: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/4.jpg)
DefinitionIterative linear solvers attempt to iteratively approximate a solution [ ] N
jx ℑ∈=*x to a system of linear equations:
bAx = ,
where [ ] NMija ×ℑ∈=A is a coefficient matrix and [ ] M
ib ℑ∈=b is a data vector, using following updates:
( )( 1) ( ) , ,k kf+ =x x A b ,
where ( )kx is an approximation to the solution x in the k-th iterative step, and ( ), ,f ⋅ ⋅ ⋅ is an
update function determined by an underlying iterative method, where ( )( ) *lim , ,k
kf
→∞→x A b x .
![Page 5: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/5.jpg)
Stationary basic iterative methods
Let ,=Ax b ,n n×∈ℜAwhere (nonsingular) ,n∈ℜx n∈ℜb
,= −A S TAssume the splitting: thus 1 ,k k+ = +Sx Tx b (Basic Iterative Methods)
We have 1 ,k k+ = +x Gx c where 1 1 ,− −= = −G S T I S A 1−=c S b
Theorem: The iterations {xk } are convergent to x = A-1b for any starting guess x0 if and only if every eigenvalue λ
of S-1T satisfies | λ
| <1. Its convergence rate depends
on the maximum value of | λ
|, which is known as a spectral radius of S-1T:
( ) .max1iiλρ =− TS
.
.
![Page 6: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/6.jpg)
Stationary basic iterative methods
Proof: Let ek =xk -x* denote an error in the k-th iteration. Since Sxk+1 = Txk +b, we have S(xk+1 – x*) = T(xk – x*) +b, and the error in xk+1 is given by ek+1 =S-1T ek
= (S-1T)k+1 e0 . If ρ(S-1T)<1, (S-1T)k → 0.
The matrix S can be regarded as a preconditioner. There are several choices for splitting A. For example:
1. Jacobi method: S = diagonal part of A;
2. Gauss-Seidel method: S = lower triangular part of A;
3. Sucessive Over-Relaxation (SOR) method: combination of 1 and 2.
![Page 7: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/7.jpg)
Jacobi methodLet ,n n×∈ℜA
11
22
0 00
,0
0 0
n n
nn
aa
a
×
⎡ ⎤⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
S
,= −A S T
12 1
21
( 1)
1 ( 1)
00
.
0
n
n n
n n
n n n
a aa
aa a
×
−
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
T
Thus ( )11 ,k k
−+ = +x S Tx b or ( 1) ( )1 , 1, 2, , .
nk k
i i ij jj iii
x b a x i na
+
≠
⎛ ⎞= − =⎜ ⎟
⎝ ⎠∑ …
Remark: if ∃i: aii = 0, a permutation of rows or columns is necessary.
![Page 8: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/8.jpg)
Gauss-Seidel methodAssume
11
21 22
1 ( 1)
0 0
,0
n n
n n n nn
aa a
a a a
×
−
⎡ ⎤⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
S
12 1
( 1)
00 0
.
0 0 0
n
n n
n n
a a
a×
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
T
Thus ( )11 ,k k
−+ = +x S Tx b or
1( 1) ( 1) ( )
1 1
1 . i n
k k ki i ij j ij j
j j iii
x b a x a xa
−+ +
= = +
⎛ ⎞= − −⎜ ⎟
⎝ ⎠∑ ∑
Remark: Note that the Gauss-Seidel method is a cyclic method and uses also the elements xi
(k+1) that have been already updated in the current iterative cycle.However, unlike the Jacobi method, the computation for each element cannot be done in parallel.
![Page 9: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/9.jpg)
Successive Over-Relaxation (SOR)
Let ,n n×∈ℜA ,= + +A L D U
11
22
0 00
,0
0 0 nn
aa
a
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
D
12 1
( 1)
00 0
.
0 0 0
n
n n
a a
a −
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
U21
1 ( 1)
0 0 00
,00n n n
a
a a −
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
L
Assuming1 ,ω
= +S L D 1 .ωω−⎛ ⎞= − +⎜ ⎟
⎝ ⎠T U D ( )1
1 .k k−
+ = +x S Tx bUsing
we have ( ) ( )( )( )11 1 ,k kω ω ω ω−+ = + − + −x L D b U D x 0 < ω
< 2 - relaxation
parameter
![Page 10: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/10.jpg)
Successive Over-Relaxation (SOR)
Finally( )
1( 1) ( ) ( 1) ( )
1 11 .
i nk k k k
i i i ij j ij jj j iii
x x b a x a xaωω
−+ +
= = +
⎛ ⎞= − + − −⎜ ⎟
⎝ ⎠∑ ∑
For ω
=1, the SOR simplifies to the Gauss-Seidel method.
If :T n n×= ∈ℜA A (symmetric) ,T=U L11 ,
2
T
ω ω ω ω
−⎛ ⎞⎛ ⎞ ⎛ ⎞= + +⎜ ⎟⎜ ⎟ ⎜ ⎟− ⎝ ⎠⎝ ⎠ ⎝ ⎠
D D DP L L
( )11 ,k k kγ −+ = − −x x P Ax b (SSOR method)
![Page 11: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/11.jpg)
Stationary basic iterative methods for LS problems
Let ,m n×∈ℜA
,T = −A A S T
where ,m n≥
Splitting:
We have 1 ,k k+ = +x Gx c where 1 ,Tn
−= −G I S A A 1 T−=c S A b
and consider the normal equations of the first kind:
,T T=A Ax A b
In general: ( )1 ,k k k+ = + −x x B b Ax
The matrix B can take various forms, depending on the underlying method.
![Page 12: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/12.jpg)
Landweber iterations,T = −A A S TLet 1 ,nα
=S I 1 ,Tnα
= −T I A A for the splitting
and is a relaxation parameter.0α >
We have 1 ,k k+ = +x Gx c where ,Tn α= −G I A A .Tα=c A b
Finally ( )1 ,Tk k kα+ = + −x x A b Ax where .Tα=B A
The Landweber iterations are also known as the Richardson’s first-order method.
If (range of AT), then the Landweber iterations are convergent to x = A+b if
( )0 ,TR∈x A
( ) 1
max0 2 .Tα λ−
< < A A
![Page 13: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/13.jpg)
Jacobi method for LS problemsLet [ ]1 2, , , ,m n
n×= ∈ℜA a a a…
21 2
22 2
2
2
0 0
0,
0
0 0
n n
n
×
⎡ ⎤⎢ ⎥⎢ ⎥
= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a
aS
a
,T = −A A S T
1 2 1
2 1
1
1 1
00
.
0
T Tn
Tn n
Tn n
T Tn n n
×
−
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a a a aa a
Ta a
a a a a
From ( )11 ,k k k
−+ = + = +x S Tx b Gx c where
( )( 1) ( )2
2
, j 1, 2, , .Tjk k
j j k
j
x x n+ = + − =a
b Axa
…
1 ,Tn
−= −G I S A A 1 T−=c S A b
we have:
![Page 14: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/14.jpg)
Jacobi method for LS problemsIn the parallel mode:
( ) ( )11 ,T
k k k k k−
+ = + − = + −x x B b Ax x S A b Ax where 1 T−=B S A
and assuming that all columns in A are non-zero, we have : 0.jjj s∀ >
The Jacobi method is symmetrizable since:
( )1/2 1/2 1/2 1/2.Tn
− − −− =S I G S S A AS
![Page 15: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/15.jpg)
Gauss-Seidel method for LS problems
Assume 21 2
22 1 2 2
21 1 2
0 0
,0
Tn n
T Tn n n n
×
−
⎡ ⎤⎢ ⎥⎢ ⎥
= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a
a a aS
a a a a a
1 2 1
1
00 0
.
0 0 0
T Tn
n nTn n
×
−
⎡ ⎤⎢ ⎥⎢ ⎥= − ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
a a a a
Ta a
Thus ( )11 ,k k
−+ = +x S Tx b leads to
( )( )11 1 .T T
k k k k−
+ += + − + +x x S A b Tx S T x
![Page 16: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/16.jpg)
Kaczmarz algorithm
1 2
2
, Ti i kk k i
i
bα+
−= +
a xx x aa
In the sequential mode the Kaczmarz algorithm (devised by a Polish mathematician: Stefan Kaczmarz, and published in 1937) is given by:
Let1
,m n
m
×
⎡ ⎤⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎣ ⎦
aA
awhere is the i-th row vector of A.1 n
i×∈ℜa
mod 1i k m= −
In image reconstruction, this algorithm is known as the unconstrained Algebraic Reconstruction Technique (ART).
0 2α< < - relaxation
![Page 17: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/17.jpg)
Kaczmarz algorithmHyperplane defined by i-th equation:
The algorithm converges to:
1 2 mH H H∩ ∩ ∩…
![Page 18: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/18.jpg)
Kaczmarz algorithm
For consistent case:
Limit point:
where is an initial guess and G is a generalized inverse of A.
( ) ( )( ) LSNkkNP xAGbxxxx A +=+==
∞→)(lim 00
*
0x
(K. Tanabe, Numerische Mathematik, 1971)
( )R∈b A
(nullspace) (minimal norm LS solution)
![Page 19: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/19.jpg)
Kaczmarz algorithm
For inconsistent case:Limit point:
where
⇒
where:
( ) ( )( ) bGxxxx A~lim 00
* +==∞→ Nkk
P
,r δ= +b b b ,r nδ δ δ= +b b b ( ) ,r Rδ ∈b A ( ) ,Tn Nδ ∈b A
( ) ( )( )220
* ;, nrLSSd bGbGbGbAxx δδδ +==
( ) 2, .d = −u v u v
(C. Popa, R. Zdunek, Mathematics and Computers in Simulations, 2004)
( )R∉b A
![Page 20: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/20.jpg)
Kaczmarz algorithm (example from image reconstruction)
![Page 21: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/21.jpg)
Krylov subspace methods1k k kα+ = +x x r
k k= −r b Ax
The Richardson method: - solution update
- residual update
For α
= 1, we have ( )1 0 0 1 2 01
.k
ik k
i+
=
= + + + + + = − ⋅∑x x r r r r I A r…
Thus { } ( )11 0 0 0 0 0span , , , ; .k k
k K ++ − ∈ ⋅ ⋅ =x x r A r A r A r… - Krylov subspace
1
1 0 01
.k
ik i
iα
+
+=
= +∑r r A r
Assuming 1 1,k k+ += −r b Ax we have
11
1 0 01
,k
ik i
iα
+−
+=
= +∑x x A r
(Linear combinationcoefficients)
![Page 22: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/22.jpg)
Krylov subspace methodsA family of nonparametric polynomial iterative methods based on the general formula:
( )1 1 0p ,k k+ += ⋅r A r
where ( ) 11p ++ Π∈ kk A (space of polynomials with (k+1) degrees ) and ( ) 10p 1 =+k
(Residual polynomial)
( )1 0 0q ,k k+ − = ⋅x x A r
(Iterative polynomial)
( ) ( )1p 1 q .k k+ = − ⋅A A A
Assuming * :k k= −ε x x ( ) ( )1 10 0p p .k k k k
− −= = =ε A r A A r A ε
![Page 23: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/23.jpg)
Krylov subspace methodsThe classes of nonparametric polynomial iterative methods:
• Orthogonal Residual (OR) polynomials:
( ) ( ) 0, for p ,p
0, for OR ORk l
k lt t
k l> =⎧
= ⎨ ≠⎩where ( ) ( )p 0 p 0 1.OR OR
k l= =
Methods: CG, Lanczos, GENCG and FOM.
Minimal error criterion: ( ){ }* * 0 0min : , .OR kk K− = − ∈ +
AAx x x x x x A r
The Ritz-Galerkin condition: ( )0 , .OR kk K⊥r A r
• Minimal Residual (MR) polynomials: ( ) ( )p , p 0,MR MRk lt t t = .k l≠for
Minimal error criterion: ( ){ }0 0min : , .MR kk K− = − ∈ +b Ax b Ax x x A r
The Petrov-Galerkin condition: ( )0,rAAr kMRk K ⊥ Methods: CR, GMRES, MINRES
![Page 24: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/24.jpg)
Krylov subspace methods• Minimal Error (ME) polynomials: ( ) ( ) 0p,p =ttME
k for ( ) 1p ,kt −∀ ∈Π
( )p 0 1,MEk = ( ) ( )p 0 1.ME
k
′=where
( ){ }* * 0 0min : , .ME kk K− = − ∈ +x x x x x x A A rMinimal error criterion:
These methods satisfy the Ritz-Galerkin condition: ( )0 , .ME kk K⊥r A r
• Minimal Residual (MR) polynomials for normal equations: 2 ,=A x Ab
( ) ( )2 2 2p , p 0MR MRk l =A A A for .k l≠
Minimal error criterion: ( ){ }20 0min : , .LSQR k
k K− = − ∈ +b Ax b Ax x x A A r
Methods: LSQR
![Page 25: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/25.jpg)
Stieltjes algorithmThe OR polynomials satisfy the 3-term recurrences and can be computed by the Stieltjes algorithm:
( ) 0p 1 =− t( )0p 1,t =
( ) ( )( ) ( )( ) ( )( ) ( )
( )( ) ( ) ( ) ( )
1 1
1 1
1 2
2 2
1 2 1,2, ,
, p,
p , p
p ,p,
p , p
,
p p p ,
k kk
k k
k kk
k k
k k k
k k k k k k k K
t t tt t
t t tt t
t t t t
η
ξ
γ η ξ
γ η ξ
− −
− −
− −
− −
− − =
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪⎪ ⎪
=⎨ ⎬⎪ ⎪⎪ ⎪= − +⎪ ⎪⎪ ⎪= − −⎩ ⎭ …
( ) ( ) ( ){ }ttt kp,,p,p 10 …
![Page 26: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/26.jpg)
Conjugate Gradients (CG) algorithm
Let be an additional polynomial for determining a search direction for ( )tk 1φ − ( )p .k t
The search direction is tangential in the point to the hyperplane spanned by
( ){ }tk k 1p,1 −−
( ) ( ) ( ) ( ){ }0 1 2 1 p , p , p , p .k kt t t t− −… Thus
( ) ( ) ( ) ( )1 21 1 1
p pφ p .k k
k k k k
t tt t
tξ − −
− − −
⎛ ⎞−= + ∈Π⎜ ⎟⎝ ⎠
( ) ( )( ) ( )( ) ( )( ) ( )
( ) ( ) ( )( ) ( ) ( )
1 1
1 1
1 1
1 2 2
1 2 1
1 1 1,2, ,
p , p1 ,φ ,φ
p ,p,
p ,p
φ φ p ,
p p φ ,
k kk
k k k
k kkk
k k k
k k k k
k k k k k K
t tt t t
t tt t
t t t
t t t t
αγ
ξβγ
β
α
− −
− −
− −
− − −
− − −
− − =
⎧ ⎫= =⎪ ⎪
⎪ ⎪⎪ ⎪⎪ ⎪
= =⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= −⎩ ⎭ …
From the orthogonality of OR polynomials:
( )( ) ( )p p 0,T
l kt t = for kl ≠
( ) ( ) ( ) ( ){ }tttt kk 1101 p,,p,pspanφ −− ∈ …
( ) ( ) ( ){ }1 1φ span p , pk k kt t t t− −∈
![Page 27: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/27.jpg)
Conjugate Gradients (CG) algorithm
1 1
2 2
1 2 1
1 1
1 1
1 1
1 1
1,2,
,
,
,
,,
Tk k
k Tk k
k k k kTk k
k Tk k
k k k k
k k k k
k
β
β
α
αα
− −
− −
− − −
− −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪
= −⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
z z r
r rz Az
x x zr r Az
…
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e 0z =−1
- initial guess,0x 0 0= −r b Ax n n×∈ℜA- initial residual vector, where is a symmetric and positive-definite matrix.
Remark: ( )1 1p ,k kt− −r ( )1 1φk kt− −z
From the orthogonality of OR polynomials:
( )( ) ( )p p 0,T
l kt t = for we have:,l k≠ 0.Tl k =r r
From the t-orthogonality of the search direction polynomials:
for we have:,l k≠( )( ) ( )φ φ 0T
l kt t t = 0.Tl k =z Az
![Page 28: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/28.jpg)
Conjugate Gradients (CG) algorithm
Theorem:
For any the sequence generated by the conjugate direction algorithm converges to the solution x* of the linear system in at most n iterations.
0n∈ℜx { }kx
Convergence rate: ( )( )
* *0
1,
1
k
k
κ
κ
⎛ ⎞−⎜ ⎟− ≤ −⎜ ⎟+⎝ ⎠
A A
Ax x x x
A
where is a condition number of A. ( )κ A
![Page 29: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/29.jpg)
CG algorithm for normal equations
( )( )( )
( )
1
1 1
1
10,1,
,
,
,
,
,
Tk k
Tk k
k Tk k
k k k kTk k
k Tk k
k k k kk
β
β
α
α
−
− −
−
+=
⎧ ⎫= −⎪ ⎪⎪ ⎪
=⎪ ⎪⎪ ⎪⎪ ⎪= −⎨ ⎬⎪ ⎪⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎩ ⎭
q A b Ax
Aq Aw
Aw Aww q w
w qAw Aw
x x w…
( )1 0 ,T− = −w A b Ax
- initial guess,0x
1 0,k k− =q w
0Tl k =w A Aw
kzThe search direction is replaced with the correction vector that satisfies the following orthogonality conditions:
kw
1.
2. for .l k≠
![Page 30: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/30.jpg)
CGNR algorithmConjugate Gradient Normal Residual (CGNR) algorithm for normal equations:
( )
( )
1 1
2 2
1 2 1
1 1
1 1
1 1
1,2,
,
,
,
,
,
Tk k
k Tk k
k k k kTk k
k Tk k
k k k kT
k k
k
β
β
α
α
− −
− −
− − −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= −⎪ ⎪⎩ ⎭
r rr r
z z r
r rAz Az
x x z
r A y Ax…
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e 0z =−1
- initial guess,0x ( )0 0 ,T= −r A b Ax - initial residual vector,
![Page 31: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/31.jpg)
Preconditioning• Left preconditioning: 1 1 .− −=M Ax M b
• Right preconditioning: 1 ,− =AM y b 1 .−=x M y
In general, a good preconditioner M should meet the following requirements:• the preconditioned system should be easy to solve,• the inverse of the preconditioner should be cheap to construct and apply,• the condition number of the preconditioned matrix is considerably lower,• eigenvalues of of the preconditioned matrix are clustered,• the computational cost of calculating M-1 is much lower than for A-1.
Usually the incomplete Cholesky factorization or LU factorization are used to compute M from A.
![Page 32: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/32.jpg)
PCG algorithm
11 1
1 1
2 2
1 2 1
1 1
1 1
1 1
1 1 1,2,
ˆ ,ˆ
,ˆ
ˆ ,ˆ
,
,,
k kTk k
k Tk k
k k k kTk k
k Tk k
k k k k
k k k k k
β
β
α
αα
−− −
− −
− −
− − −
− −
− −
− −
− − =
⎧ ⎫=⎪ ⎪⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎨ ⎬⎪ ⎪=⎪ ⎪⎪ ⎪
= +⎪ ⎪⎪ ⎪= −⎩ ⎭
r M r
r rr r
z z r
r rz Az
x x zr r Az
…
( ){ }1 1* 0 0min : , .kK − −− ∈ +
Ax x x x M A M r
[ ]1, ,1 ,T n= ∈ℜe … 1 ,− =r e
1 ,− =z 0
- initial guess,0x 0 0= −r b Ax n n×∈ℜA- initial residual vector, where is a symmetric and positive-definite matrix.
M - preconditioner1ˆ .k k
−=r M rPreconditioning for the residual vector:
Minimal error criterion for the preconditioned vectors:
![Page 33: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/33.jpg)
Lanczos algorithmThe Lanczos algorithm generates the orthogonal vectors {v1 , v2 , ..., vK } that span the Krylov subspace for the symmetrix matrix A and the residual vector r0 :
( ) { } { }10 0 0 0 1 2, span , , , span , , , .K K
KK −= =A r r Ar A r v v v… …
( )1 1
1 1
11
1 1, ,
, ,
,
,
.
k k k
k k n k k k
k k
kk
k k K
α
α β
β
β
+ −
+ +
++
+ =
⎧ ⎫=⎪ ⎪
= − −⎪ ⎪⎪ ⎪⎨ ⎬=⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
Av v
v A I v v
vvv
…
0 ,=v 01 0 ,β = r 01
1
,β
=rv n n
n×∈ℜI
0 0= −r b AxAlgorithm:
Remark: Note that the Lanczos algorithm can be easily obtained from the 3-term recurrence Stieltjes algorithm by replacing t with A and the orthogonal polynomials
, ...., with the vectors:( )t0p ( )tkp 1, , .kv v…
![Page 34: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/34.jpg)
Lanczos algorithmThe vectors satisfy the recurrence formula:
11
,KK K T
K Kβ++
⎡ ⎤= ⎢ ⎥⋅⎣ ⎦
JAV V
e
1 2
2 2
0 0
.0 0
0 0
K
K
K K
α ββ α
ββ α
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
J
[ ]0 0 1 T KK = ∈ℜe …
[ ]KK vvvV ,,, 21 …=
where and the tridiagonal Jacobi matrix:
Exact solution can be obtained for K ≤
n.
![Page 35: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/35.jpg)
Lanczos algorithmAssuming the Ritz-Galerkin condition: T
k k =V r 0 for k = 0, 1, ..., K.
Considering and the orthogonality of the Lanczos vectors, we have0
01 r
rv =
0 1Tk k =V Ax r e where [ ]1 1,0, ,0 .T k= ∈ℜe …
From the minimal error criterion for the Ritz-Galerkin condition and
( ) { }0 1 2span , , , ,k k− ∈x x v v v… we have 0 ,k k k= +x x V ζ
where ξk is a vector of the coefficients of a linear combination of the Lanczos vectors. For one obtains0 0,=x
0 1.k k =J ζ r e
![Page 36: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/36.jpg)
Lanczos algorithm
( )1 1
1 1
11
1 1, ,
, ,
,
,
.
k k k
k k n k k k
k k
kk
k k K
α
α β
β
β
+ −
+ +
++
+ =
⎧ ⎫=⎪ ⎪
= − −⎪ ⎪⎪ ⎪⎨ ⎬=⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
Av v
v A I v v
vvv
…
0 ,=v 01 0 ,β = r 01
1
,β
=rv n n
n×∈ℜI0 0= −r b Ax
Lanczos algorithm for the system Ax = b:
[ ]1, , ,K K=V v v…
1 1
1 2
1
1
0 00
00 0
KK
K K
α ββ α
ββ α
−
−
⎡ ⎤⎢ ⎥⎢ ⎥=⎢ ⎥⎢ ⎥⎣ ⎦
J[ ] KT ℜ∈= 0,,0,11 …e1
0 1K K−=ζ J r e
0 K K= +x x V ζ
![Page 37: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/37.jpg)
Arnoldi algorithmThe Arnoldi algorithm computes orthogonal vectors for the Krylov subspace for the non-symmetrix matrix A and the residual vector r0 with the modified Gram-Schmidt orthogonalization.
,
, 1, ,
1,
11, 1, ,
,
,,
,
.
k
Tn k n
n k n n k
k k
kk k k K
hh
h
h
=
+
++ =
=⎧ ⎫⎪ ⎪⎧ ⎫=⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪← −⎪ ⎪⎪ ⎪⎩ ⎭⎨ ⎬
=⎪ ⎪⎪ ⎪⎪ ⎪=⎪ ⎪⎩ ⎭
w Av
v ww w v
wwv
…
…
0
01 r
rv =
[ ]KK vvV ,,1 …=The orthogonal Arnoldi vectors:
satisfy the formula: 1 1,K K K K+ +=AV V H
11 12 13 1,
21 22 23 2,( 1)
32 331,
,
( 1),( 1) ( 1),
0
0 0
K
KK K
K K
K K
K K K K
h h h hh h h h
h hh
h h
+ ×+
+ − +
⎡ ⎤⎢ ⎥⎢ ⎥⎢ ⎥= ∈ℜ⎢ ⎥⎢ ⎥⎢ ⎥⎣ ⎦
H
where HK+1,K is an upper Hessenberg matrix:
![Page 38: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/38.jpg)
GMRES algorithmIf A is a symmetric matrix, the Arnoldi algorithm simplifies to the Lanczos algorithm.
In GMRES algorithm, the Petrov-Galerkin condition is used, and from the Minimal Residual (MR) polynomial criterion, we have:
{ }02 2min min ,
KK K− = −
x ζb Ax r AV ζ
where 0 .K K= +x x V ζ
( )0 1 0 1 1, 0 1 1,2 2 2 22.K K K K K K K K K+ + +− = − = −r AV ζ V r e H ζ r e H ζ
From the relation 1 1, :K K K K+ +=AV V H
Finally0 1 1,2 2
min .K
K K K+−ζ
r e H ζ
![Page 39: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/39.jpg)
GMRES algorithmIn GMRES, the (K+1)-th row of the upper Hessenberg matrix H(K+1),K is neglected, and the square matrix HK,K is transformed to the upper triangular one using the QR factorization with the Givens rotations:
, .K K K K=H Q R
Since the upper triangular system can be easily solved with
the Gausian eliminations.2
1,K =Q
![Page 40: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/40.jpg)
GMRES algorithm1
01 1, ,
1 1 ,1
1k k
11 1
22
,,
,
, , , ,
, ,
If: : Elseif: : 11 , ,, ,11
Elseif:
Tn n n
kn n n n n k
Tnk n k k k k
k
k
kk k
k k k kk k k k
kk
hh
h rh
hh
h hc s cs c s
ρ
ρθ θρ
ρ ρθθ
θθ
−
− =
+ ++
+
+
+ +
⎧ ⎫=⎪ ⎪= ⎨ ⎬= −⎪ ⎪⎩ ⎭
= = = =
⎧ ⎧= =⎪ ⎪⎪ ⎪> ≤⎨ ⎨⎪ ⎪ = = −= = −⎪ ⎪ ++ ⎩⎩
v uu Av
u u v
uu v r Q h
…
{1
, , 1
, , , 1 ,
, 1 1, 1
1
1
,1
0,
1
0: 1, 0,,
, ,, ,
, ,
,If: : Elseif: :
,
k k k
k k k k k k k
k k k k k k k k k k
k k k k k k
k k k k k k
k
k n k nn
k k kk k
k k k k
h c sr r c h sq c q q s qq s q cf c s
rk m k mr
f
ϕ ϕ ϕ
+
+
+
+ + +
+
−
=
−
= = =
← −
← ←
← − ←
= =
⎧−⎪⎪ =< = = +⎨
⎪⎪ = +⎩
∑v ww x x V R
x x w
,k
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎩ ⎭
f
Initialization:
0 0= −r b Ax
1 0 ,h = r1
01 hr
v =
1,1 1,q = 11 h=φ
(Saad and Schultz, 1986)
![Page 41: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/41.jpg)
Bi-Conjugate GradientsFor a non-symmetrix matrix A, the 3-term Stieltjes recurrences are applied to generate 2 Krylov bases:
( ) { }0 0 1; , , ,kkK span −=A r r r…
( ) { }0 0 1; , , .k TkK span −=A r r r…
Tk k k=R R D
[ ]0 1, , .k k−=R r r…[ ]0 1, , ,k k−=R r r…
k kk
×∈ℜD
- the shifted Krylov base
Let
The bases are mutually orthogonal: - Petrov-Galerkin condition,
- diagonal matrix.
![Page 42: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/42.jpg)
BiCG algorithm1 1
2 2
1 2 1
1 2 1
1 1
1 1
1 1
1 1
1 1
1,2,
,
,,
,
,,
,
Tk k
k Tk k
k k k k
k k k kTk k
k Tk k
k k k k
k k k kT
k k k k
k
β
ββ
α
αα
α
− −
− −
− − −
− − −
− −
− −
− −
− −
− −
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪
= +⎪ ⎪⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= +⎪ ⎪
= −⎪ ⎪⎪ ⎪= −⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
z z rz z r
r rz Az
x x zr r Az
r r A z
…
Initialization:(Fletcher, 1975)
0 0= −r b Ax
0 0 ,=r r [ ]1, ,1 ,T n= ∈ℜe …
1 ,− =r e 1 ,− =r e
1 ,− =z 0 0z =−1~
A – non-singular (n x n) matrix
If the denominator equals to zero, a serious breakdown occurs.
![Page 43: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/43.jpg)
CGS algorithmIn the BiCG method, we have: ( ) 0p ,k k=r A r ( ) 0p .T
k k=r A r
From the orthogonality condition:
( ) ( ) ( ) ( ) ( )20 0 0 0 0 0, p , p , p p , p 0.T
k k k k k k k= = = =r r A r A r r A A r r A r
Equivalently, in the CGS (CG Square) method the residual vector is defined as follows:
( )20p .k k=r A r
Starting from the orthogonality condition:0 0 ,=r r 0 , 0.k =r r
![Page 44: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/44.jpg)
CGS algorithm
( )
( )( )
0 1
0 2
1 1
1 1
0 1
0
1
1
1,2,
,
,,
,
,,
.
Tk
k Tk
k k k k
k k k k k k
Tk
k Tk
k k k k
k k k k k
k k k k k
k
β
β
β β
α
αα
α
−
−
− −
− −
−
−
−
=
⎧ ⎫=⎪ ⎪
⎪ ⎪⎪ ⎪= +⎪ ⎪
= + +⎪ ⎪⎪ ⎪⎪ ⎪=⎨ ⎬⎪ ⎪⎪ ⎪= −⎪ ⎪
= + +⎪ ⎪⎪ ⎪
= − +⎪ ⎪⎪ ⎪⎩ ⎭
r rr r
u z rq u z q
r rr Aq
z u Aqx x u z
r r A u z
…
Initialization:
0 0 ,= −r b Ax [ ]1, ,1 ,T n= ∈ℜe …
1 ,− =r e 0 ,=z 0 0 0.=q r
A – non-singular (n x n) matrix,
(Sonneveld, 1989)
![Page 45: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/45.jpg)
BiCGSTAB algorithmThe residual vector is defined as follows: ( ) ( ) 0q pk k k=r A A r
where ( ) ( )( ) ( )1 2q 1 1 1 .k kω ω ω= − − −A A A A…
( )( )( )
( )( )
0 11
0
1
0
0 1
1 1 1,2,
, , ,
, ,
,
.
TTk kk
k k k k k kT Tk k k
k k k k k k k k k k
Tk k
k Tkk
k k k k k k k
α α ω
α ω ω
αβω
β ω
−−
−
−
− − =
⎧ ⎫= = − =⎪ ⎪
⎪ ⎪⎪ ⎪
= + + = −⎪ ⎪⎨ ⎬⎛ ⎞⎛ ⎞⎪ ⎪⎜ ⎟= ⎜ ⎟⎪ ⎪⎜ ⎟⎝ ⎠⎝ ⎠⎪ ⎪⎪ ⎪= + −⎩ ⎭
As sr r s r AzAr Az As As
x x z s r s As
r rAr Ar
z r z Az…
Initialization:
0 0 ,= −r b Ax
0 0.=z r
A – non-singular (n x n) matrix,
If ωk = 0, a serious breakdown
(Van der Vorst, 1992)
![Page 46: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/46.jpg)
QMR methodSimilarly to the GMRES algorithm, the Quasi-Minimal Residual (QMR) method uses the Petrov-Galerkin condition but the Minimal Residual (MR) polynomial criterion takes the form:
{ }0min mink
k k− = −x ζb Ax r AR ζ
where [ ]10 ,, −= kk rrR … contains the successive residual vectors , and
1 1, .k k k k+ +=AR R H
Thus the MR criterion can be simplified to 0 1 1,mink
k k k+−ζ
r e H ζ and
0 .k k= +x x R ζ
![Page 47: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/47.jpg)
QMR method
( )
1 1
1 1
1 1
1 1 1 1
21 1
221 1
1
, ,
,
, ,
,
, ,
1, , ,1
k kk k
k k
Tk k
k T Tk k
k k k k k k k k k k
T Tk k
k Tk k
Tk k k k k k k k
k k k kk k k
k k k kk
k k k k k
α
α α
β
β β
ε γϑ γ ε
γ β β γϑ
ε ϑ γ
− −
− −
− −
− − − −
− −
− −
−
= =
=
= − = −
=
= − = −
= = = −+
= +
v wv wv w
w vq A Ap
p v w p q w v q
q A Apw v
v Ap v w A q w
v v
d p ( )2 21 1 1
1 1 1,2,
, ,, .
k k k k k k k
k k k k k k k
ε ϑ γ− − −
− − =
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪= +⎪ ⎪⎪ ⎪= + = −⎩ ⎭
d s Ap sx x d r r s
…
Initialization:
0 0 ,= −r b AxA – non-singular (n x n) matrix,
(Freund and Nachtigal, 1991)
0 0 ,=v r 0 0 ,=w r
0 ,=p 0 0 ,=q 0 0 ,=d 0 0 .=s 0
Remark: numerically stable
![Page 48: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/48.jpg)
LSQR method
{ }
1
11 1
1 1
11 1 1
1 Bi diagonalization
2 21k 1 1 1 1 2 1
QR factorization
1 1k
1 1
,
, ,
,
, .
, , ,
c , ,
k k k k
kk k k
k
k k k k
kk k k
k
k k k k k k k k
k kk
k k
r c r r r s
r sr r
β
αα
α
ββ
α β α
β
+
++ +
+ +
++ + +
+ −
− + −
+
= −⎧ ⎫⎪ ⎪⎪ ⎪= =⎪ ⎪⎨ ⎬= −⎪ ⎪⎪ ⎪
= =⎪ ⎪⎩ ⎭
= = + =
= =
v Au vvv v
u Av uuu u
( )Givens rotations
2 11 1
1
1 1 Updates 1, ,
, ,
, ,
k k kk k k k k k
k
k k k k k kk K
rc
rs s
η
η η
−− −
− −=
⎧ ⎫⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎧ ⎫⎪ ⎪⎨ ⎬⎪ ⎪⎩ ⎭⎪ ⎪⎧ ⎫⎪ ⎪− ⋅
= = + ⋅ ⋅⎪ ⎪⎪ ⎪⎨ ⎬⎪ ⎪⎪ ⎪⎪ ⎪= − ⋅ = ⋅⎩ ⎭⎩ ⎭
v ww x x w
r r…
Initialization:
0 0 ,= −r b Ax
A – non-singular (m x n) matrix
1 0 ,=u r
1 1,=v Au 1 1 ,α = v
1 1,β = u11
1
,β
=uu
0 1,c = 0 0,s =
0 ,=w 0 0 1.η β=
(Paige and Saunders, 1982)
(normal equations),
![Page 49: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/49.jpg)
Nonnegative Matrix Factorization (NMF)
Nonnegative matrix factorization (NMF) solves the following problem: estimate (up to scale and permutation ambiguity) such nonnegative matrices and that
given only the assigned lower-rank J, and possibly the prior knowledge on the estimated factors or noise distribution.
I J×+∈ℜA T J×
+∈ℜB
,I T×+∈ℜX
T≅X AB
J I T≤Usually
![Page 50: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/50.jpg)
Rank-1 matrix factorization
1 1
J JT
j j j jj j= =
= + = +∑ ∑X a b V a b V
(Outer product)(Noise or error matrix)
![Page 51: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/51.jpg)
Feature extraction
Published in Nature, 401, (1999): D.D. Lee from Bell Lab, H.S. Seung from MIT
(X)
(A)
(BT)
![Page 52: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/52.jpg)
NMF for BSSy1 y2 y3
y4 y5 y6
y7 y8 y9
s1 s2
s3 s4
Original images Mixed images
![Page 53: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/53.jpg)
Alternating minimization• First step (Update for B): • Second step (Update for A):
Iterative algorithm:
T =AB XT T=BA X
For 1, 2,t = … doInitialize randomly: (0) (0),A B
( )( )( ) ( 1) ( 1)
0arg min , ,
Tt t tD − −
≥←
BB X A B
( )( )( ) ( 1) ( )
0arg min , ,
Tt t tD −
≥←
AA X A B
End
( )|| TD X AB - objective function
![Page 54: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/54.jpg)
Objective functions• Square Euclidean distance (square Frobenius norm):
• Generalized Kullback-Leibler divergence (I-divergence):
( )22
1 1
1 1( || ) || || [ ]2 2
I TT T T
F F it iti t
D x= =
= − = −∑∑X AB X AB AB
s t 0 0 .ij tja b i j t. . ≥ , ≥ ∀ , ,
( || ) log [ ][ ]
T TitKL it it itT
it it
xD x x⎛ ⎞⎜ ⎟⎜ ⎟⎜ ⎟⎜ ⎟⎝ ⎠
= + −∑X AB ABAB
11
s t 0 0 || || 1I
jtj ij iji
b a a=
. . ≥ , ≥ , = = .∑a
![Page 55: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/55.jpg)
Multiplicative algorithms• Euclidean distance ⇒ ISRA
• I-divergence ⇒ EMML
[ ] [ ],
[ ] [ ]
Tij tj
ij ij tj tjT Tij tj
a a b b← , ←XB X AAB B BA A
1 1
1 1
( [ ] ) ( [ ] )
I TT Tij it it tj it iti t
tj tj ij ijI Tqj pjq p
a x b xb b a a
a b= =
= =
/ /← , ← ,∑ ∑
∑ ∑AB AB
(Image Space Reconstruction Algorithm)
(Expectation-Maximization Maximum Likelihood)
![Page 56: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/56.jpg)
ALS algorithmEuclidean distance ⇒ ALS (Alternating Least-Squares)
( ) ( )|| 0,T
T T TFD∇ = − ≡
BX AB A AB X ( ) ( )|| 0,T T
FD∇ = − ≡A
X AB AB X B
Stationary points:
ALS algorithm:
( ) ( )1,
TT T − +← =B X A A A A X
( ) ( )1,
TT T − +← =A XB B B X B
( ) ,T
P +Ω⎡ ⎤← ⎢ ⎥⎣ ⎦
B A X
( ) ,T
P +Ω⎡ ⎤← ⎢ ⎥⎣ ⎦
A X B
Projected ALS :
( )22
1 1
1 1( || ) || || [ ]2 2
I TT T T
F F it iti t
D x= =
= − = −∑∑X AB X AB AB
![Page 57: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/57.jpg)
TensorsDefinition: Let
I1
, I2
, . . . , IN
∈ N denote index upper bounds. A tensor of
order N is an
N-way array where elements
are indexed
by
in
∈ {1, 2, . . . , In
} for 1 ≤ n ≤ N.
1 2 NI I I× × ×∈ℜX …1 2, , , ni i ix
Tensors are obviously generalizations of vectors and matrixes, for example, a third-order tensor (or 3-way array) has 3 modes (or indices or dimensions). A zero-order tensor is a scalar, a first-order tensor is a vector, a second-order tensor is a matrix, and tensors of order three and higher are called higher- order tensors.
![Page 58: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/58.jpg)
Tensor fibers
![Page 59: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/59.jpg)
Tensor slices
![Page 60: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/60.jpg)
Unfolding
Mode-1 unfolding
Mode-2 unfolding Mode-3 unfolding
![Page 61: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/61.jpg)
Unfolding
![Page 62: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/62.jpg)
PARAFAC
1
J
j j jj=
= +∑X a b c V
![Page 63: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/63.jpg)
PARAFAC
1 2 31
J
j j jj
I=
= + = × × × +∑X a b c V A B C V
![Page 64: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/64.jpg)
Harshman’s PARAFAC
( )1
J
j j j jjλ
=
= +∑X a b c V
![Page 65: Numerical Methods Rafał Zdunek Iterative Methodsue.pwr.wroc.pl/numerical_methods_lectures/NM_Iterative_methods.pdf · iiijj ijj ii jji. x b ax ax a − ++ ==+ ⎛⎞ =− −⎜⎟](https://reader030.vdocuments.mx/reader030/viewer/2022021606/5e180c78f363a969de103c3d/html5/thumbnails/65.jpg)
TUCKER decomposition
( )1 1 1
J R P
jrp j r pj r p
g= = =
= +∑∑∑X a b c V