dan witzner hansen email: [email protected]. groups? improvements – what is missing?
Post on 19-Dec-2015
226 views
TRANSCRIPT
![Page 2: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/2.jpg)
LAST WEEK?
mnm
n
aa
aaa
A
1
11111
h d g c
f b e a
h g
f e
d c
b a
h d g c
f b e a
h g
f e
d c
b a
dh cf dg ce
bh af bg ae
h g
f e
d c
b a
![Page 3: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/3.jpg)
MISC
Groups?
Improvements – what is missing?
![Page 4: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/4.jpg)
TODAY
The goal is to be able to solve linear equations
Continue with linear algebra
Linear mappings
Basis vectors & independence
Solving linear equations & Determinants
Inverse & Least squares
SVD
Lot’s of stuff. Don’t despair – you will be greatly rewarded in the future
![Page 5: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/5.jpg)
LINEAR SYSTEMS
![Page 6: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/6.jpg)
WHAT IS A LINEAR EQUATION?
A linear equation is an equation of the form,
anxn+ an-1xn-1+ . . . + a1x1 = b.
![Page 7: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/7.jpg)
WHAT IS A SYSTEM OF LINEAR EQUATIONS?
A system of linear equations is simply a set of linear equations. i.e.
a1,1x1+ a1,2x2+ . . . + a1,nxn = b1
a2,1x1+ a2,2x2+ . . . + a2,nxn = b2
. . .
am,1x1+ am,2x2+ . . . + am,nxn = bm
![Page 8: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/8.jpg)
MATRIX FORM OF LINEAR SYSTEM
Compact notation A x = b
![Page 9: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/9.jpg)
LINEAR MAPPINGS
A
Axy baxy
bAxy Affine mapping
![Page 10: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/10.jpg)
EXAMPLE
Species 1: eats 5 units of A and 3 of B. Species 2: eats 2 units of A and 4 of B. Everyday a total of 900 units of A and
960 units of B are eaten. How many animals of each species are there?
€
5x1 + 2x2 = 900
3x1 + 4x2 = 960
Species
![Page 11: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/11.jpg)
EXAMPLE:
€
5 2
3 4
⎡
⎣ ⎢
⎤
⎦ ⎥x1
x2
⎡
⎣ ⎢
⎤
⎦ ⎥=
900
960
⎡
⎣ ⎢
⎤
⎦ ⎥
Ax = b
€
5x1 + 2x2 = 900
3x1 + 4x2 = 960
![Page 12: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/12.jpg)
MATLAB CODE
A = [5 2; 3 4];b = [900 960];
x = linspace(0,150,100);y1 = (-A(1,1)*x+b(1))/A(1,2); %made for clarityy2 = (-A(2,1)*x+b(2))/A(2,2);
Plot(x,y1,'r-','LineWidth',3); hold onPlot(x,y2,'b-','LineWidth',3); hold offtitle('Linear equations and their solution')
![Page 13: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/13.jpg)
BASIS, INDEPENDENCE AND SUBSPACES
![Page 14: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/14.jpg)
AN NOW FOR SOME FORMALISM
Subspaces Independent vectors Basis vectors / Orthonomal basis
![Page 15: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/15.jpg)
SUBSPACES
A subspace is a vector space contained in another vector space
![Page 16: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/16.jpg)
INDEPENDENT VECTORS
Axy Can it happen that y=0 if x is nonzero?
If y is non-zero for all non-zero x, then the column vectors of A are said to be linear independent.
These vectors form a set of basis vectors
Orthonormal basis when the vectors are unit size and orthogonal.
![Page 17: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/17.jpg)
BASIS VECTORS - EXAMPLE
0
0
1
1e
0
1
0
2e
1
0
0
3e
4
3
2
1
0
0
4
0
1
0
3
0
0
1
2x
€
x =
2
3
4
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥=
1 0 0
0 1 0
0 0 1
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
2
3
4
⎡
⎣
⎢ ⎢ ⎢
⎤
⎦
⎥ ⎥ ⎥
Change of basis
![Page 18: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/18.jpg)
SHOW THAT IT IS AN ORTHONORMAL BASIS
€
b1 =cosv
sinv
⎡
⎣ ⎢
⎤
⎦ ⎥,b2 =
−sinv
cosv
⎡
⎣ ⎢
⎤
⎦ ⎥
vv
vvB
cossin
sincos
![Page 19: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/19.jpg)
WHAT HAPPENS WITH THIS ONE?
4123
682
241
A = [1 4 2;2 8 6; 3 12 4];[X,Y,Z] = meshgrid(-10:10,-10:10,-10:10);x = [X(:),Y(:),Z(:)]’;p = A*x;plot3(p(1,:),p(2,:),p(3,:),'rx')
![Page 20: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/20.jpg)
A CLOSER LOOK AT MATRIX MULTIPLICATION
nmnmm
nn
nn
xaxaxa
xaxaxa
xaxaxa
Ax
2211
2222121
1212111
mn
n
n
n
mm a
a
a
x
a
a
a
x
a
a
a
x
2
1
2
22
12
2
1
21
11
1
nnAxAxAx :,2:,21:,1
![Page 21: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/21.jpg)
2D EXAMPLE
€
Ax =a11 a12
a21 a22
⎡
⎣ ⎢
⎤
⎦ ⎥x1
x2
⎡
⎣ ⎢
⎤
⎦ ⎥
€
Ax =a11 a12
a21 a22
⎡
⎣ ⎢
⎤
⎦ ⎥x1
x2
⎡
⎣ ⎢
⎤
⎦ ⎥=x1
a11
a21
⎡
⎣ ⎢
⎤
⎦ ⎥+ x2
a12
a22
⎡
⎣ ⎢
⎤
⎦ ⎥
![Page 22: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/22.jpg)
WHAT HAPPENS WITH THIS ONE?
4123
682
241
4x =
![Page 23: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/23.jpg)
SOLVING LINEAR EQUATIONS
![Page 24: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/24.jpg)
SOLUTIONS OF LINEAR EQUATIONS
A solution to a system of equations is simply an assignment of values to the variables that satisfies (is a solution to) all of the equations in the system.
If a system of equations has at least one solution, we say it is consistent.
If a system does not have any solutions we say that it is inconsistent.
![Page 25: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/25.jpg)
RECALL
€
5 2
3 4
⎡
⎣ ⎢
⎤
⎦ ⎥x1
x2
⎡
⎣ ⎢
⎤
⎦ ⎥=
900
960
⎡
⎣ ⎢
⎤
⎦ ⎥
Ax = b€
5x1 + 2x2 = 900
3x1 + 4x2 = 960
Solution
![Page 26: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/26.jpg)
SOLVING SYSTEMS ALGEBRAICALLY
2x 3y z 5
y z 1
z 3Which solution(s)?
Can we always do this?
How many solutions are there?
![Page 27: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/27.jpg)
DETERMINANT
For A (2x2 matrix)
When det A ≠0 a unique solution exists (nonsingular) When det A =0 the matrix is singular (lines same
slope) and are therefore the columns are linear dependent Coincident (infinitely many solutions) Parallel (no solutions)
Determinant can be used when solving linear equations (Cramers’ rule), but not useful in practice
12212211det aaaaA >>det(A)
![Page 28: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/28.jpg)
WHAT IF?
What to do when the dimension and the number of data points is large?
How many data points are needed to solve for the unknown parameters in x?
![Page 29: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/29.jpg)
MATRIX INVERSE
Solve simple linear equation
Matrix inverse:
A (unique) inverse exist if det(A) ≠ 0 (NxN matrices)
Matlab: >>invA =inv(A)
€
ax = b
a−1ax = a−1bx
x = a−1b
0000
000
0010
0001
1
IAA
![Page 30: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/30.jpg)
SOLVING LINEAR SYSTEMS
If m = n (A is a square matrix & Det(A)!=0), then we can obtain the solution by simple inversion (:
If m > n, then the system is over-constrained and A is not invertible
If n>m then under constrained.
![Page 31: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/31.jpg)
MATRIX INVERSE EXAMPLE
Solve Ax=b (notice multiply from right):
6
4,
24
35
2
1,
54
32 1 bAA
bAx 1
![Page 32: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/32.jpg)
NOTICE: IMPLEMENTATION
Don’t use for solving the linear system. It is mostly meant for notational convenience.
It is faster and more accurate (numerically) to write (solve) x=A\b than inv(A)*b:
bA 1
1111)( ABCABC
TT AA )()( 11
![Page 33: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/33.jpg)
SIMPLE INVERSION OF (SOME) MATRICES Diagonal matrices
Orthogonal matrices
nnd
d
d
D
000
000
000
000
22
11
T
T
AA
IAA
1
,cossin
sincos,
cossin
sincos 1
TRRR
1
122
111
1
000
000
000
000
nnd
d
d
D
![Page 34: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/34.jpg)
FITTING LINES
A 2-D point x = (x, y) is on a line with
slope m and intercept b if and only if y =
mx + b Equivalently,
So the line defined by two points x1, x2 is
the solution to the following system of equations:€
x 1[ ]m
b
⎡
⎣ ⎢
⎤
⎦ ⎥= y
€
x1 1
x2 1
⎡
⎣ ⎢
⎤
⎦ ⎥m
b
⎡
⎣ ⎢
⎤
⎦ ⎥=y1
y2
⎡
⎣ ⎢
⎤
⎦ ⎥
![Page 35: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/35.jpg)
EXAMPLE: FITTING A LINE
Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)?????
3
3
2
1
18
17
15
12
b
m
![Page 36: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/36.jpg)
FITTING LINES
With more than two points, there is no guarantee that they will all be on the same line
courtesy ofVanderbilt U.
![Page 37: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/37.jpg)
LEAST SQUARESFxy
)(,minargˆ Fxyrrx Objective:
Find the vector Fx in the column range of F, which is closest to the right-hand side vector y.The residual r=y-Fx
![Page 38: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/38.jpg)
FITTING LINES
courtesy ofVanderbilt U.
Solution: Use the pseudoinverse
A+ = (ATA)-1AT to obtain least-
squares solution x = A+b
![Page 39: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/39.jpg)
EXAMPLE: FITTING A LINE
Suppose we have points (2, 1), (5, 2), (7, 3), and (8, 3)
Then????
and x = A+b = (0.3571, 0.2857)T
![Page 40: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/40.jpg)
EXAMPLE: FITTING A LINE(2, 1), (5, 2), (7, 3), and (8, 3)
![Page 41: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/41.jpg)
HOMOGENEOUS SYSTEMS OF EQUATIONS
Suppose we want to solve A x = 0 There is a trivial solution x = 0, but we
don’t want this. For what other values of x is A x close to 0?
This is satisfied by computing the singular value decomposition (SVD) A = UDVT (a non-negative diagonal matrix between two orthogonal matrices) and taking x as the last column of V In Matlab [U, D, V] = svd(A)
![Page 42: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/42.jpg)
SINGULAR VALUE DECOMPOSITION
Tnnnmmmnm VΣUA
IUU T
021 n
IVV T
nm
XXVT XVΣ T XVUΣ T
TTTnnn VUVUVUA 222111
000
00
00
00
Σ
2
1
n
![Page 43: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/43.jpg)
NULL SPACE & IMAGE SPACE
4123
682
241
![Page 44: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/44.jpg)
PROPERTIES OF SVD
When the columns of A =UDV are independent then all
Tells how close to singular A is. Inverse and pseudoinverse The columns of U corresponding to
nonzeros singular values span the range of A, the columns of V corresponding to zero singular values the nullspace.
€
dii > 0
0000
0000
000
000
Σ 2
1
TUVDA 11 TUVDA 1
0
![Page 45: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/45.jpg)
EXAMPLE: LINE-FITTING AS A HOMOGENEOUS SYSTEM
A 2-D homogeneous point x = (x, y, 1)T is on the line l = (a, b, c)T only when
ax + by + c = 0
We can write this equation with a dot product:
x .l = 0, and hence the following system is implied for
multiple points x1, x2, ..., xn:
![Page 46: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/46.jpg)
EXAMPLE: HOMOGENEOUS LINE-FITTING
Again we have 4 points, but now in homogeneous form:
(2, 1, 1), (5, 2, 1), (7, 3, 1), and (8, 3, 1)
The system of equations is:
Taking the SVD of A, we get:compare to x = (0.3571, 0.2857)T
![Page 47: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/47.jpg)
ROBUST METHODS
So what about outliers
• Other metrics such as other norms• More about this later
![Page 48: Dan Witzner Hansen Email: witzner@itu.dk. Groups? Improvements – what is missing?](https://reader036.vdocuments.mx/reader036/viewer/2022062714/56649d2d5503460f94a04b35/html5/thumbnails/48.jpg)
NEXT WEEK