ams526: numerical analysis i (numerical linear algebra for computational … · 2019-08-26 · ot...

AMS526: Numerical Analysis I(Numerical Linear Algebra for

Computational and Data Sciences)Lecture 1: Course Overview;

Matrix Multiplication

Xiangmin Jiao

Stony Brook University

Xiangmin Jiao Numerical Analysis I 1 / 22

Outline

1 Course Overview

2 Matrix Notation and Basic Operations (MC �1.1)

3 Range, Null Space and Rank (NLA �1)


Course Description

What is numerical linear algebra?

I Solving linear algebra problems using e�cient algorithms on computers

Topics: Direct and iterative methods for solving simultaneous linearequations, least squares problems, computation of eigenvalues andeigenvectors, and singular value decomposition

Required textbooks[NLA] L. N. Trefethen and D. Bau III, Numerical Linear Algebra,SIAM, 1997.[MC] G. H. Golub and C. F. Van Loan, Matrix Computations, 4thEdition, Johns Hopkins University Press, 2012. ISBN 978-1421407944.

Additional supplementary material will be provided as needed

Course webpage:http://www.ams.sunysb.edu/~jiao/teaching/ams526


https://www.amazon.com/Numerical-Linear-Algebra-Lloyd-Trefethen/dp/0898713617

https://www.amazon.com/dp/1421407949

http://www.ams.sunysb.edu/~jiao/teaching/ams526

Prerequisite

This MUST NOT be your �rst course in linear algebra, or you willget lost

Prerequisite/Co-requisite:

I AMS 510 (linear algebra portion) or equivalent undergraduate-levellinear algebra course. Familiarity with following concepts is assumed:Vector spaces, Gaussian elimination, Gram-Schmidt orthogonalization,and eigenvalues/eigenvectors

I AMS 595 (co-requisite for students without programming experience)

To review fundamental concepts of linear algebra, see textbook such as

I Gilbert Strang, Linear Algebra and Its Applications, 4th Edition, BrooksCole, 2006.


Why Learn Numerical Linear Algebra?Foundation of scienti�c computations and data sciencesMany problems ultimately reduce to linear algebra concepts oralgorithms, either analytical or computationalExamples: Finite-element analysis, data �tting, PageRank (Google)

Focus: Fundamental concepts, e�ciency and stability of algorithms,and programmingNew focus: relevance to computational and data sciences


Course OutlineFundamentals (matrix notation and basic operations; vector spaces;algorithmic considerations; norms and condition numbers;decomposition of matrices)Linear systems (triangular systems; Gaussian elimination; accuracyand stability; Cholesky factorization; sparse linear systems)QR factorization and least squares (Gram-Schmidtorthogonalization; QR factorization with Householder re�ection;updating QR factorization with Givens rotation; stability; least squaresproblems; rank-revealing QR; SVD and low-rank approximations)Eigenvalue problems (eigenvalues and invariant spaces; classicaleigenvalue methods; QR algorithms; two-stage methods; Arnoldi andLanczos iterations)Iterative Methods for linear systems (basic iterative methods;conjugate gradient methods; minimal residual style methods;bi-Lanczos iterations; preconditioners)Special topics (multigrid methods; under-determined systems, etc., iftime permits)


Course Policy

Assignments (written or programming)

I Assignments are due in class one to two weeks after assignedI You can discuss course materials and homework problems with others,

but you must write your answers completely independentlyI Do NOT copy solutions from any source. Do NOT share your solutions

with others

Exams and tests

I All exams are closed-bookI However, one-page cheat sheet is allowed

Grading

I Assignments: 30%I Two midterm exams: 40%I Final exam: 30%


Outline

1 Course Overview




Matrices and Vectors

Denote vector space of all m-by-n real matrices by Rm×n.

A ∈ Rm×n ⇔ A = (aij) =

a11 a12 . . . a1na21 a22 . . . a2n...

.... . .

...am1 am2 . . . amn

Denote vector space of all real n-vectors by Rn, or Rn×1

x ∈ Rn ⇔ x =

x1x2...xn

Transposition (Rm×n → Rn×m): C = AT ⇒ cij = aji

Row vectors are transpose of column vectors and are in R1×n


Matrix Operations

Addition and subtraction (Rm×n × Rm×n → Rm×n):C = A± B ⇒ cij = aij ± bij

Scalar-matrix multiplication or scaling (R× Rm×n → Rm×n):C = αA⇒ cij = αaij

Matrix-matrix multiplication/product (Rm×p × Rp×n → Rm×n):C = AB ⇒ cij =

∑pk=1 aikbkj (denoted by A ∗ B in MATLAB)

Each operation also applies to vectors. In particular,

I Inner product is row vector times column vector, i.e., c = xT y

(it is called dot product in vector calculus and denoted as x · y)I Outer product is column vector times row vector, i.e., C = xyT

(it is a special case of Kronecker product and denoted as x ⊗ y)

Element-wise multiplication and division (Rm×n × Rm×n → Rm×n)

I C = A. ∗ B ⇒ cij = aijbijI C = A./B ⇒ cij = aij/bij , where bij 6= 0

Matrix inversion (A−1) and division (A/B and A\B) to be de�ned later


Notation of Matrices and VectorsMatrix notation

I Capital letters (e.g., A, B, ∆, etc.) for matricesI Corresponding lower case with subscript ij (e.g., aij , bij , δij) for (i , j)

entry; sometimes with notation [A]ij or A(i , j)

Vector notationI Lowercase letters (e.g., x , y , etc.) for vectorsI Corresponding lower case with subscript i for ith entry (e.g., xi , yi )

Lower-case letters for scalars (e.g., c , s, α, β, etc.)Some books suggest using boldface lowercase (e.g., x) for vectors,regular lowercase (e.g. c) for scalars, and boldface uppercase formatricesA matrix is a collection of column vectors or row vectors

A ∈ Rm×n ⇔ A = [c1|c2| . . . |cn] , ck ∈ Rm

A ∈ Rm×n ⇔ A =

rT1...rTm

, rk ∈ Rn


Complex Matrices

Occasionally, complex matrices are involved

Vector space of m-by-n complex matrices is designated by Cm×n

I Scaling, addition, multiplication of complex matrices correspondexactly to real case

I If A = B + iC ∈ Cm×n, then Re(A) = B, Im(A) = C , and conjugate ofA is A = (aij)

I Conjugate transpose is de�ned as AH = BT − iCT , or

G = AH ⇒ gij = aji

(also called adjoint, and denoted by A∗; (AB)∗ = B∗A∗)

Vector space of complex n-vectors is designated by Cn

I Inner product of complex n-vectors x and y is s = xHyI Outer product of complex n-vectors x and y is S = xyH

We will primarily focus on real matrices


Matrix-Vector Product

Matrix-vector product b = Ax is special case of matrix-matrix product

bi =n∑

j=1

aijxj

For A ∈ Rm×n, Ax is a mapping x 7−→ Ax from Rn to Rm

This map is linear, which means that for any x , y ∈ Rn and any α ∈ R

A(x + y) = Ax + Ay

A(αx) = αAx


Linear Combination

Let aj denote jth column of matrix A

I Alternative notation is colon notation: A(:, j) or a:,jI Use A(i , :) or ai,: to denote ith row of A

b is a linear combination of column vectors of A, i.e.,

b = Ax =n∑

j=1

xjaj = x1

a11a21...

am1

+ x2

a12a22...

am2

+ · · ·+ xn

a1ma2n...

amn

In summary, two di�erent views of matrix-vector products:

1 Scalar operations: bi =∑n

j=1aijxj : A acts on x to produce b

2 Vector operations: b =∑n

j=1xjaj : x acts on A to produce b


Matrix-Matrix Multiplication

Computes C = AB , where A ∈ Rm×r , B ∈ Rr×n, and C ∈ Rm×n

Element-wise, each entry of C is

cij =r∑

k=1

aikbkj

Column-wise, each column of C is

cj = Abj =r∑

k=1

bkjak ;

in other words, jth column of C is A times jth column of B


Outline

1 Course Overview




Terminology of Vector Space

Vector space is closed under addition and scalar multiplication, withzero vector as a member

Vector space spanned by a set of vectors {aj} is

span{a1, . . . , an} =

n∑

j=1

βjaj | βj ∈ R

I Space spanned by n-vectors is a subspace of Rn

I If {a1, . . . , an} is linearly independent, then the aj are the basis ofS = span{a1, . . . , an}, dimension of S is dim(S) = n, and each b ∈ S isunique linear combination of the aj

If S1 and S2 are two subspaces, then S1 ∩ S2 is a subspace, so isS1 + S2 = {b1 + b2 | b1 ∈ S1, b2 ∈ S2}

I Note: S1 + S2 is di�erent from S1 ∪ S2; latter may not be a subspace

Two subspaces S1 and S2 of Rn are complementary subspaces of eachother if S1 + S2 = Rn and S1 ∩ S2 = {0}

I In other words, dim(S1) + dim(S2) = n and S1 ∩ S2 = {0}Xiangmin Jiao Numerical Analysis I 17 / 22

Range and Null Space

De�nition

The range of a matrix A, written as range(A) or ran(A), is the set ofvectors that can be expressed as Ax for some x{y ∈ Rm|y = Ax for some x ∈ Rn}.

Theorem

range(A) is the space spanned by the columns of A.

Therefore, range of A is also called the column space of A.

De�nition

The null space of A ∈ Rm×n, written as null(A), is the set of vectors x thatsatisfy Ax = 0.

Entries of x ∈ null(A) give coe�cient of∑

xiai = 0.Note: The null space of A is in general not a complementary subspace ofrange(A).


Relationship Between Null and Range Space

For real matrices A ∈ Rm×n

I null(A) and range(AT ) are complementary subspacesI For symmetric matrices (A = AT ), null(A) and range(A) are

complementary subspaces

For complex matrices A ∈ Cm×n

I null(A) and range(AH) are complementary subspacesI For Hermitian matrices (A = AH), null(A) and range(A) are

complementary subspaces


Rank

De�nition

The column rank of a matrix is the dimension of its column space. The rowrank is the dimension of the space spanned by its rows.

Question: Can the column rank and the row rank be di�erent?

Answer: No. We will give a proof in later lectures.

We therefore simply say the rank of a matrix.

Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?Answer: n.

A real matrix A is rank-1 if it can be written as A = uvT , where u andv are nonzero vectors


Rank

De�nition


Question: Can the column rank and the row rank be di�erent?Answer: No. We will give a proof in later lectures.


Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?

Answer: n.



Rank

De�nition


Question: Can the column rank and the row rank be di�erent?Answer: No. We will give a proof in later lectures.


Question: Given A ∈ Rm×n, what is dim(null(A)) + rank(A) equal to?Answer: n.



Full Rank

De�nition

A matrix has full rank if it has the maximal possible rank, i.e., min{m, n}.

Otherwise, it is called rank de�cient.

Theorem

A matrix A ∈ Rm×n with m ≥ n has full rank if and only if it maps no twodistinct vectors to the same vector.

In other word, the linear mapping de�ned by Ax for x ∈ Rn is one-to-one.

Proof.

(⇒) Column vectors of A forms a basis of range(A), so every b ∈ range(A)has a unique linear expansion in terms of the columns of A.(⇐) If A does not have full rank, then its column vectors are lineardependent, so its vectors do not have a unique linear combination.


Full Rank vs. Non-singularity

If A ∈ Rn×n and AX = I , then X is the inverse of A, denoted by A−1

I (AB)−1 = B−1A−1

I (A−1)T = (AT )−1 = A−T

I (A + UV T )−1 = A−1 − A−1U(I + V TA−1U)−1V TA−1

for U,V ∈ Rn×k (Sherman-Morrison-Woodbury formula)

If A−1 exists, A is nonsingular. A is square and has full rank.

In A is nonsingular, linear system Ax = b results in x = A−1b, and itis the inverse problem of matrix-vector multiplication

If A ∈ Rm×n where n 6= m and A has full rank, what is the inverseproblem of matrix-vector multiplication?

If A is rank de�cient, what is the inverse problem of matrix-vectormultiplication?

We will need progressively more advanced linear algebra concepts toanswer these questions in later part of this semester.


ams526: numerical analysis i (numerical linear algebra for computational … · 2019-08-26 · ot...

Documents