instructor: taylor berg-kirkpatrick slides: sanjoy dasgupta · dot products and angles dot product...
TRANSCRIPT
Linear algebra primer
Instructor: Taylor Berg-KirkpatrickSlides: Sanjoy Dasgupta
Data as vectors and matrices
0 1 2 3 4 5 6 0
1
2
3
4
5
6
Matrix-vector notation
Vector x ∈ Rd :
x =
x1x2x3...xd
Matrix M ∈ Rr×d :
M =
M11 M12 · · · M1d
M21 M22 · · · M2d...
.... . .
...Mr1 Mr2 · · · Mrd
Mij = entry at row i , column j
Matrix-vector notation
Vector x ∈ Rd :
x =
x1x2x3...xd
Matrix M ∈ Rr×d :
M =
M11 M12 · · · M1d
M21 M22 · · · M2d...
.... . .
...Mr1 Mr2 · · · Mrd
Mij = entry at row i , column j
Transpose of vectors and matrices
x =
1630
has transpose xT =
M =
1 2 0 43 9 1 68 7 0 2
has transpose MT = .
• (AT )ij = Aji
• (AT )T = A
Adding and subtracting vectors and matrices
Dot product of two vectors
Dot product of vectors x , y ∈ Rd :
x · y = x1y1 + x2y2 + · · ·+ xdyd .
What is the dot product between these two vectors?
0 1 2 3 4
1
2
3
4
-1 -2 -3 -4
x
y
Dot product of two vectors
Dot product of vectors x , y ∈ Rd :
x · y = x1y1 + x2y2 + · · ·+ xdyd .
What is the dot product between these two vectors?
0 1 2 3 4
1
2
3
4
-1 -2 -3 -4
x
y
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y
• What is x · x?
Dot products and angles
Dot product of vectors x , y ∈ Rd :x · y = x1y1 + x2y2 + · · ·+ xdyd .
Tells us the angle between x and y :
y
✓
x
cos θ =x · y‖x‖ ‖y‖
.
• x is orthogonal (at right angles) to y if and only if x · y = 0
• When x , y are unit vectors (length 1): cos θ = x · y• What is x · x?
Matrix-vector product
Product of matrix M ∈ Rr×p and vector x ∈ Rp:
Linear functions
Let M be any r × d matrix.The function x 7→ Mx sends Rd to Rr .
It is a linear function: M(x + x ′) = Mx + Mx ′.
E.g. M =
(2 0 −10 1 1
)
Write the linear function f (x1, x2) = 3x1 + 2x2 using vectornotation.
A linear function from R2 to R2 is given by M =
(1 12 2
).
As x varies, does Mx fill up all of R2?
The identity matrix
The d × d identity matrix Id sends each x ∈ Rd to itself.
Id =
1 0 0 · · · 00 1 0 · · · 00 0 1 · · · 0...
......
. . ....
0 0 0 · · · 1
Matrix-matrix product
Product of matrix A ∈ Rr×k and matrix B ∈ Rk×p:
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
Matrix products
If A ∈ Rr×k and B ∈ Rk×p, then AB is an r × p matrix with
(AB)ij = (dot product of ith row of A and jth column of B)
=k∑
`=1
Ai`B`j
• IkB = B and A Ik = A
• Can check: (AB)T = BTAT
• For two vectors u, v ∈ Rd , what is uT v?
Some special cases
For vector x ∈ Rd , what are xT x and xxT ?
Special case, generalized
For vector x ∈ Rd , we have xT x = ‖x‖2.
What about xTMx , for arbitrary d × d matrix M?
Quadratic functions
Let M be any d × d (square) matrix.For x ∈ Rd , the mapping x 7→ xTMx is a quadratic functionfrom Rd to R:
xTMx =d∑
i ,j=1
Mijxixj .
What is the quadratic function associated with M =
1 0 00 2 03 4 5
?
Write the quadratic function f (x1, x2) = x21 + 2x1x2 + 3x22 usingmatrices and vectors.
Special cases of square matrices
• Symmetric: M = MT1 2 32 4 53 5 6
,
1 2 31 2 43 4 6
• Diagonal: M = diag(m1,m2, . . . ,md)
diag(1, 4, 7) =
1 0 00 4 00 0 7
Determinant of a square matrix
Determinant of A =
(a bc d
)is |A| = ad − bc.
Example: A =
(3 11 2
)
Inverse of a square matrix
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
Example: if A =
(1 2−2 0
)then A−1 =
(0 −1/2
1/2 1/4
). Check!
Inverse of a square matrix
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
Example: if A =
(1 2−2 0
)then A−1 =
(0 −1/2
1/2 1/4
). Check!
Inverse of a square matrix, cont’d
The inverse of a d × d matrix A is a d × d matrix B for whichAB = BA = Id .Notation: A−1.
• Not all square matrices have an inverse
• Square matrix A is invertible if and only if |A| 6= 0
• What is the inverse of A = diag(a1, . . . , ad)?