randomized iterative methods for linear systems and ...€¦ · rmg and peter richtárik stochastic...
TRANSCRIPT
![Page 1: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/1.jpg)
Randomized iterative methods for linear systems and
inverting matrices
Robert Mansel Gower Joint work with Peter Richtárik
University of Edinburgh
Cambridge, January 2016
![Page 2: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/2.jpg)
RMG and Peter RichtárikStochastic Dual Ascent for Solving Linear SystemsPreprint arXiv:1512.06890, 2015
RMG and Peter RichtárikStochastic Iterative Matrix InversionIn progress, 2016
RMG and Peter RichtárikRandomized Iterative Methods for Linear SystemsSIAM. J. Matrix Anal. & Appl., 36(4), 1660–1690, 2015
![Page 3: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/3.jpg)
Linear Systems
![Page 4: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/4.jpg)
The Problem
We can also think of this as m linear equations, where the ith equation looks as follows:
Assumption: The system is consistent (i.e., has a solution)
![Page 5: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/5.jpg)
The Problem
B: Symmetric and positive definite
Insight: As there are possibly multiple solutions, we compute the solution with the least B-norm.
![Page 6: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/6.jpg)
Standard Randomized Methods
![Page 7: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/7.jpg)
The return of old methods
Old methods (Kaczmarz 1937, Guass-Seidel 1823) make a randomized return, why?
● Often suitable for Big Data problems (short recurrence, low iteration cost, low memory, block variants...etc)
● Easy to implement
● Easy to analyse, good complexity
● Often fits in parallel/distributed architecture
![Page 8: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/8.jpg)
Randomized Kaczmarz T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
G.N. Hounsfield. Computerized transverse axial scanning (tomography): Part I. description of the system. British Journal Radiology. 1973
Karczmarz, M. S. (1937). Angenaherte Auflosung von Systemen linearer Gleichungen. Bulletin International de l’Académie Polonaise Des Sciences et Des Lettres, 35, 355–357.
![Page 9: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/9.jpg)
Framework for Randomized Methods
![Page 10: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/10.jpg)
1. Relaxation Viewpoint“Sketch and Project”
B: Symmetric and positive definite
S: random matrix
![Page 11: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/11.jpg)
2. Optimization Viewpoint “Constrain and Approximate”
![Page 12: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/12.jpg)
3. Geometric Viewpoint “Random Intersect”
(2)
(1)
![Page 13: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/13.jpg)
4. Algebraic Viewpoint“Random Linear Solve”
Unknown: x Unknown: y
![Page 14: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/14.jpg)
5. Algebraic Viewpoint“Random Update”
Moore-Penrose pseudo inverse
Random Update Vector
Fact:
Small matrix
![Page 15: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/15.jpg)
6. Analytic Viewpoint“Random Fixed Point”
Random Iteration Matrix
![Page 16: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/16.jpg)
Theory
![Page 17: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/17.jpg)
Complexity / Convergence
Theorem [GR‘15]
1
2
![Page 18: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/18.jpg)
Proof of for A full column rank 1
![Page 19: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/19.jpg)
Case study of
Special Choice of Parameters
No zero rows in A is positive definite
Weak assumption
![Page 20: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/20.jpg)
The rate: lower and upper bounds
Theorem [RG‘15]
Insight: The method is a contraction (without any assumptions on S whatsoever). That is, things can not get worse.
Insight: The lower bound on the rate is better for A low rank and when the dimension of the search space in the “constrain and approximate” viewpoint grows.
![Page 21: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/21.jpg)
Special Case: Randomized
Kaczmarz Method
![Page 22: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/22.jpg)
Randomized Kaczmarz: derivation and rate
General Method
Special Choice of Parameters
Complexity Rate.
![Page 23: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/23.jpg)
Special Case: Randomized
Coordinate Descent
![Page 24: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/24.jpg)
Randomized Coordinate Descent: derivation and rateGeneral Method
Special Choice of Parameters
Complexity Rate
positve definite
![Page 25: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/25.jpg)
Theory recovers known and new convergence results
Method Convergence Rate
Randomized CDLeast square
B S
T. Strohmer and R. Vershynin. A Randomized Kaczmarz Algorithm with Exponential Convergence. Journal of Fourier Analysis and Applications 15(2), pp. 262–278, 2009
*Leventhal, D., & Lewis, A. S. (2010). Randomized Methods for Linear Constraints: Convergence Rates and Conditioning. Mathematics of Operations Research, 35(3), 641–654.
Gaussian psd
Gaussian Kaczmarz
![Page 26: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/26.jpg)
Convenient probability
Theorem [GR‘15]
![Page 27: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/27.jpg)
Conclusion for linear systems
● Unites many randomized methods under a single framework
● Improved convergence: New lower bound, less assumptions, RK convergence without full rank assumption.
● Design new methods: S = Guassian, count-sketch, Walsh-Hadamard ...etc
● Optimal Sampling: We can choose a sampling that optimizes the convergence rate.
![Page 28: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/28.jpg)
Inverting a Matrix
![Page 29: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/29.jpg)
The Problem
Assumption: The matrix A is nonsingular
Identity matrix
![Page 30: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/30.jpg)
Why iteratively invert a matrix?
● Needed to calculate a Schur complement or a projection operator
● Iterative methods are good when we can tolerate an error or have an initial guess
● Staging for randomized variable metric methods and randomized preconditioning
![Page 31: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/31.jpg)
Randomized Methods for Nonsymmetric Matrices
![Page 32: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/32.jpg)
Equivalence to solving linear systems
B: Symmetric and positive definite
S: random matrix
This method is equivalent to the sketch and project method for solving linear systems, but applied simultanously to the n equations defined by AX = I
![Page 33: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/33.jpg)
Randomized Methods for Symmetric Matrices
![Page 34: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/34.jpg)
Sketch and Project
Connection to quasi-Newton Methods: This is a randomized block extension of the quasi-Newton updates. In the quasi-Newton setting
and A is an unknown operator. However, we can sample its action Aδ and
is known as the secant equation
Goldfarb, D. (1970). A Family of Variable-Metric Methods Derived by Variational Means. Mathematics of Computation, 24(109), 23.
![Page 35: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/35.jpg)
Constrain and Approximate
Duality: This is dual problem of the sketch and project viewpoint, new insight into quasi-Newton methods.
![Page 36: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/36.jpg)
New viewpoint for BFGS
Duality: The BFGS minimizes a residual restricted to an affine space of symmetric matrices
Constrain and approximate
Sketch and project
![Page 37: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/37.jpg)
Random Update
Random Fixed PointLow rank 3 X τ update
![Page 38: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/38.jpg)
Complexity / Convergence
Theorem [GR‘16]
1
2
![Page 39: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/39.jpg)
Special Case:Randomized Block BFGS
![Page 40: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/40.jpg)
Randomized BFGS
Special Choice of Parameters
Complexity Rate.
positve definite
![Page 41: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/41.jpg)
Randomized Block BFGS
Special Choice of Parameters
Complexity Rate.
positve definite
Idea: To minimize condition number, choose S so that S is an approximate inverse of A1/2
![Page 42: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/42.jpg)
BFGS with Randomized Self-Conditioning (RASC)
Self conditioning sampling:
*Gratton, S., Sartenaer, A., & Ilunga, J. T. (2011). On a Class of Limited Memory Preconditioners for Large-Scale Nonlinear Least-Squares Problems. SIAM Journal on Optimization, 21(3), 912–935.
![Page 43: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/43.jpg)
Experiments
![Page 44: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/44.jpg)
Current state of the art
Symmetric Newton-Schulz
Self-conditioning Minimal Residual (MR)
![Page 45: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/45.jpg)
Synthetic ProblemsSynth
eti
c data
(randn, n = 1000)
![Page 46: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/46.jpg)
Synthetic ProblemsSynth
eti
c data
(randn, n = 5000)
![Page 47: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/47.jpg)
Ridge Regression HessianLI
BS
VM
data
(aloi, n = 128)
![Page 48: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/48.jpg)
Ridge Regression HessianLI
BS
VM
data
(aloi, n = 20,958)
![Page 49: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/49.jpg)
Sparse Matrices from Engineering
UF
colle
ctio
n
(Nasa-nasa, n = 4,705)
![Page 50: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/50.jpg)
Sparse Matrices from Engineering
UF
colle
ctio
n
(ND-nd6k, n = 18,000)
![Page 51: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/51.jpg)
Consequences and Future Work
![Page 52: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/52.jpg)
Smooth minimization
Cheap to calculate, costs τ X function evaluations
![Page 53: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/53.jpg)
Variable metric methods
Update metric with RASC update
![Page 54: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/54.jpg)
Preconditioning Sketched Newton
Sketch and project Newton system
Update metric with RASC update
![Page 55: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/55.jpg)
Conclusion for Inverting Matrices
● New randomized methods capable of inverting large-scale matrices
● Convergence rates which can form the basis of convergence of preconditioning or variable metric methods.
● Dual viewpoints of classic quasi-Newton methods
● Can be extended to calculating pseudo-inverse
![Page 56: Randomized iterative methods for linear systems and ...€¦ · RMG and Peter Richtárik Stochastic Dual Ascent for Solving Linear Systems Preprint arXiv:1512.06890, 2015 RMG and](https://reader034.vdocuments.mx/reader034/viewer/2022042812/5fac7f84f00593373f138480/html5/thumbnails/56.jpg)
Thank you,Questions?