lecture 2: linear algebra revisited - university of...
TRANSCRIPT
![Page 1: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/1.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 1
Lecture 2: Linear Algebra Revisited
Overview • Vector spaces, Hilbert & Banach Spaces, Metrics & Norms
• Matrices, Eigenvalues, Orthogonal Transformations, Singular Values
• Operators, Operator Norms, Function Spaces
Note: We will need many of these concepts as basic tools to quantify and evaluate the
performance of machine learning algorithms and also to come up with more efficient
and effective solutions …
![Page 2: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/2.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 2
Vectors
Multiplication by scalar (ax).
Addition of vectors (x+y) – x and y have to be of same dimensions.
Linear combination.
u = ax+by (x and y have to be of same dimensions).
Angle between vectors.
Linear independence.
When one vector cannot be written as a linear combination of other, then the vectors are said to be linearly independent.
Usually denoted by lower case, bold letters, e.g. x, y
Operations :
wv
wv, cos
![Page 3: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/3.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 3
Metric
Definition 1 (Metric/ Distance)
Example 2 (Manhattan Distance)
Example 1 (Trivial Metric)
![Page 4: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/4.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 4
Vector Spaces
Definition 2 (Vector Spaces)
Definition 4 (Completeness)
Definition 3 (Cauchy Series)
![Page 5: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/5.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 5
Examples of Vector Spaces
Rational Numbers, Real Numbers, Polynomials are all Vector Spaces
![Page 6: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/6.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 6
Norm & Banach Spaces
Definition 5 (Norm / Length)
Definition 6 (Banach Space)
![Page 7: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/7.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 7
Examples of Banach Spaces
![Page 8: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/8.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 8
Dot Products & Hilbert Spaces
Definition 7 (Dot Product/ Inner Product)
Definition 8 (Hilbert Space)
543; 222
1
vv,v
4
3v
Example :
![Page 9: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/9.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 9
Examples of Hilbert Spaces
![Page 10: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/10.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 10
Matrices
m n
M
mv
v
1
v
nu
u
1
u
Review:
• Addition of Matrices
• Multiplication of matrices by scalars, vectors and matrices.
• Domain and Range of a Matrix
Range
Domain
u =Mv
![Page 11: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/11.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 11
Special matrices
Square and Diagonal Matrix A square matrix has equal number of rows and columns. A diagonal matrix has all off-diagonal elements zero.
Symmetric Matrix
Anti-symmetric Matrix
Orthogonal Matrix
(Often denoted as O(m) )
![Page 12: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/12.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 12
Matrix Concepts
Rank of a Matrix
Range, Domain and Null Space
Range of M, denoted by R(M), is the space of all vectors that can be obtained by the
operation of M on the vectors in the domain of M denoted by D(M).
Null Space of M, denoted by N(M), is a subspace of all the vectors in the domain of M
D(M) that map to the zero (null) vector in R(M) when operated upon by the matrix M.
vutsDuiffRv MMM ..)()(
0MM viffNv )(
![Page 13: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/13.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 13
Matrix Invariants
Trace
Properties of Trace :
)()(
)()()(
)()(
BAAB
BABA
AA
trtr
trtrtr
traatr
Determinant
Determinant can be written as the product of the eigenvalues :
Note: Trace and Determinant are invariant under orthogonal transformation
![Page 14: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/14.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 14
Matrix Norms
Frobeius Norm
Operator Norm
![Page 15: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/15.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 15
Eigensystems
Intuitive Explanation. A square matrix M is a mapping from n-dim to n-dim space.
Most vectors change both direction and length when undergoing this mapping transformation.
Those vectors which only change length (i.e., multiplying them by a matrix is similar to multiplying by a scalar) are called eigenvectors and the eigenvalue indicates how much they are shortened or lengthened.
Definition 9 (Eigenvalues/ Eigenvectors)
IMP: Defined only for square Matrices
![Page 16: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/16.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 16
Eigensystems II
o All eigenvalues of symmetric matrices are real
o Symmetric matrices are fully diagonalizable, i.e. we can find m eigenvectors
o All eigenvectors of symmetric matrices M with different eigenvalues are mutually orthogonal (Prove !!)
Eigenvectors/Eigenvalues of Symmetric Matrices
Decomposition of Symmetric Matrices
Example
![Page 17: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/17.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 17
Positive Matrices
Definition 10 (Positive Definite Matrices)
Induced Norm and Metrics
![Page 18: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/18.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 18
Mahalanobis Distance
10
01IM
Myxyx,yx,MM
Td )(
10
02IM
14.1
4.12M
= Euclidean Distance
![Page 19: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/19.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 19
Singular Value Decomposition
Note: Eigenvalue/Eigenvector decompositions are valid only for square matrices
Do we have some decomposition for rectangular matrices ??
Singular value Decomposition (SVD)
![Page 20: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/20.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 20
Matrix Inverse
m n
mv
v
1
v
nu
u
1
uRange
Domain M
1M
IMMIMM11 ;
Note: A regular inverse exists only for square matrices with linearly independent column vectors
Interpretation: We need a one-to-one mapping to uniquely go from one element of a space to another and back. Square matrices and linearly independent columns ensure this !!
![Page 21: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/21.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 21
Pseudoinverse
11 )()( TTTT or MMMMMMM
For rectangular matrices and square matrices with linearly dependent
columns, there exists the pseudoinverse or generalized inverse
which performs the inverse mapping. In general, these inverses are not
unique.
The above generalized inverse is called the Moore-Penrose
Psuedoinverse and is unique. Among the multiple inverse solutions, it
chooses the one with the minimum norm.
The Moore-Penrose Pseudoinverse (M+)
![Page 22: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/22.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 22
Operators
Linear Operators
Generalization of matrix – a mapping from one Banach space to another. Norms and
eigenvalues/eigenvectors are defined as for matrices; so are Range & Null Spaces.
Notation
A : FG denotes a linear operator A mapping from space F to space G.
Matrix Transpose Adjoint Operator
Symmetric Matrix Self Adjoint Operator
Orthogonal Matrix Isometry
A Matrix-Operator Correspondence
.,,, * GgFfallforgfgf AA
.,,, * GgFfallforgfgf AA
.,,, GgFfallforgfgf AA
TA
TAA
TAA 1
![Page 23: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/23.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 23
Linear Operators - Examples
Input transformation
Sampling
Sampling from a function to yield scalar outputs.
( We will see later why this is so !!! )
1x 2x 3x
f
Fourier Transform
![Page 24: Lecture 2: Linear Algebra Revisited - University of Edinburghwcms.inf.ed.ac.uk/ipab/rlsc/lecture-notes/RLSC-Lec2.pdf · 2013-01-13 · Lecture 2: MLSC - Prof. Sethu Vijayakumar 1](https://reader035.vdocuments.mx/reader035/viewer/2022070815/5f0f1ece7e708231d4429714/html5/thumbnails/24.jpg)
Lecture 2: MLSC - Prof. Sethu Vijayakumar 24
Range and Null Space of Operators
R(A) R(A*)
A N(A*)
N(A)
A*
Recollect: Definition of Range, Domain and Null space of a matrix