least square equation solving
TRANSCRIPT
-
8/2/2019 Least Square Equation Solving
1/22
Math for CS Lecture 4 1
Linear Least Squares Problem
Consider an equation for a stretched beam:
Y = x1 + x2 T
Where x1 is the original length, T is the force applied and x2
is the inverse coefficient of stiffness.
Suppose that the following measurements where taken:T 10 15 20Y 11.60 11.85 12.25
Corresponding to the overcomplete system:11.60 = x1 + x2 1011.85 = x1 + x2 15 - can not be satisfied exactly12.25 = x1 + x2 20
-
8/2/2019 Least Square Equation Solving
2/22
-
8/2/2019 Least Square Equation Solving
3/22
Math for CS Lecture 4 3
Linear Least Squares
There are three different algorithms for computing the least
square minimum.
1. Normal Equations (Cheap, less Accurate).
2. QR decomposition.3. SVD (expensive, more reliable).
The first algorithm in the fastest and the least accurate among the
three. On the other hand SVD is the slowest and most accurate.
-
8/2/2019 Least Square Equation Solving
4/22
Math for CS Lecture 4 4
0LQLPL]HWKHVTXDUHG(XFOLGHDQQRUPRIWKHUHVLGXDOYHFWRU
7RPLQLPL]HZHWDNHWKHGHULYDWLYHZLWKUHVSHFWWR[DQG
VHWLWWR]HUR
:KLFKUHGXFHVWRDQQ[QOLQHDUV\VWHPFRPPRQO\
NQRZQDV1250$/(48$7,216
Normal Equations 1
-
8/2/2019 Least Square Equation Solving
5/22
Math for CS Lecture 4 5
Normal Equations 2
11.60 = x1 + x2 10
11.85 = x1 + x2 1512.25 = x1 + x2 20
A x b
min||Ax-b||2
-
8/2/2019 Least Square Equation Solving
6/22
Math for CS Lecture 4 6
Normal Equations 3
201
151
101
A
72545
453AAT
201510
111TA
25.12
85.11
60.11
B
650.0
925.10)(
1bAAAx TT
We must solve the system
For the following values
(ATA)-1AT is called a Pseudo-inverse
-
8/2/2019 Least Square Equation Solving
7/22
Math for CS Lecture 4 7
QR factorization 1 A matrix Q is said to be orthogonal if its columns are
orthonormal, i.e. QT
Q=I.
Orthogonal transformations preserve the Euclidean normsince
Orthogonal matrices can transform vectors in various
ways, such as rotation or reflections but they do notchange the Euclidean length of the vector. Hence, theypreserve the solution to a linear least squares problem.
-
8/2/2019 Least Square Equation Solving
8/22
Math for CS Lecture 4 8
QR factorization 2
Any matrix A(mn) can be represented as
A = QR
,where Q(mn) is orthonormal and R(nn) is upper triangular:
> @ > @
nn
n
nn
r
r
rrr
qqaa
000
00
0|...||...|
11
11211
11
!!
!!
!
-
8/2/2019 Least Square Equation Solving
9/22
Math for CS Lecture 4 9
QR factorization 2 Given A , let its QR decomposition be given as A=QR, where
Q is an (m x n) orthonormal matrix and R is upper triangular.
QR factorization transform the linear least square problem into atriangular least squares.
QRx = b
Rx = QT
bx=R-1QTb
Matlab Code:
-
8/2/2019 Least Square Equation Solving
10/22
Math for CS Lecture 4 10
Singular Value Decomposition
Normal equations and QR decomposition only workfor fully-ranked matrices (i.e. rank( A) = n). If A is
rank-deficient, that there are infinite number ofsolutions to the least squares problems and we can
use algorithms based on SVD's.
Given the SVD:U(m x m) , V(n x n) are orthogonal
is an (m x n) diagonal matrix (singular values of A)
The minimal solution corresponds to:
-
8/2/2019 Least Square Equation Solving
11/22
Math for CS Lecture 4 11
Singular Value DecompositionMatlab Code:
-
8/2/2019 Least Square Equation Solving
12/22
Math for CS Lecture 4 12
/LQHDUDOJHEUDUHYLHZ 69'
( )
TT
TT
T
UUAA
VVAA
vuVUA
IVV
IUUVU
AA
2
2
1
21
T
T
0),,...,,diag(
such that,,existthere
,)rank(,everyfor:Fact
=
=
==
>=
=
=
=
=
p
i
T
iii
ip
pppnpm
nm p
-
8/2/2019 Least Square Equation Solving
13/22
Math for CS Lecture 4 13
$SSUR[LPDWLRQE\DORZUDQNPDWUL
( )
( )
{ } ( )1222)(rank:
2
21
1
21
1
~~
min
:inion toapproximatrankbesttheIs
),,...,,diag(
~
,
~
~
Then
...,ofSVDthebelet
.rankhavelet:IIFact
+=
=
=
===
-
8/2/2019 Least Square Equation Solving
14/22
Math for CS Lecture 4 14
*HRPHWULF,QWHUSUHWDWLRQRIWKH
69' xAb nm u
The image of the unit sphere under any mxn matrix is a hyperellipse
v1
v2
v2
v1
-
8/2/2019 Least Square Equation Solving
15/22
Math for CS Lecture 4 15
/HIWDQG5LJKWVLQJXODUYHFWRUV
SAAS nm u
We can define the properties of A in terms of the shape of AS
v1
v2
u2
u1S AS
Singular values ofA are the lengths of principal axes of AS, usuallywritten in non-increasing order 1 2 n
n left singular vectors ofA are the unit vectors {u1,, un}, oriented in the
directions of the principal semiaxes of AS numbered in correspondance with { i}
n right singular vectors ofA are the unit vectors {v1,, vn}, of S, which are the
preimages of the principal semiaxes of AS: Avi= iui
-
8/2/2019 Least Square Equation Solving
16/22
-
8/2/2019 Least Square Equation Solving
17/22
Math for CS Lecture 4 17
0DWULFHVLQWKH'LDJRQDO)RUP
Every matrix is diagonal in appropriate basis:
Any vector b(m,1) can be expanded in the basis of left singular vectors of A {ui};
Any vector x(n,1) can be expanded in the basis of right singular vectors of A {vi};
Their coordinates in these new expansions are:
Then the relation b=Ax can be expressed in terms of b and x:
xAb nm u
xVxbUb ** ;
**** xbxVUUxAUbUxAb 66
-
8/2/2019 Least Square Equation Solving
18/22
Math for CS Lecture 4 18
5DQNRI$
Let p=min{m,n}, let rp denote the number of nonzero
singlular values of A,
Then:
The rank of A equals to r, the number of nonzero
singular valuesProof:
The rank of a diagonal matrix equals to the number of its
nonzero entries, and in the decomposition A=U V* ,U and V
are of full rank
-
8/2/2019 Least Square Equation Solving
19/22
Math for CS Lecture 4 19
'HWHUPLQDQWRI$
For A(m,m),
Proof:
The determinant of a product of square matrices is the
product of their determinants. The determinant of a Unitarymatrix is 1 in absolute value, since: U*U=I. Therefore,
m
i
iA1
|)det(| V
666m
i
iVUVUA1
** |)det(||)det(||)det(||)det(||)det(||)det(| V
-
8/2/2019 Least Square Equation Solving
20/22
Math for CS Lecture 4 20
For A(m,n), can be written as a sum of r rank-one matrices:
(1)
Proof:
If we write as a sum of i, where i=diag(0,.., i,..0), then
(1)
Follows from
(2)
$LQWHUPVRIVLQJXODUYHFWRUV
r
j
jjj vuA1
*V
*VUA 6
-
8/2/2019 Least Square Equation Solving
21/22
Math for CS Lecture 4 21
The L2 norm of the vector is defined as:
(1)
The L2 norm of the matrix is defined as:
Therefore
,where i are the eigenvalues
1RUPRIWKHPDWUL[
n
ii
T
xxxx1
2
2
2
2
2sup
x
xAA
)max(2 i
A O
iii xxA O
-
8/2/2019 Least Square Equation Solving
22/22
Math for CS Lecture 4 22
For any with 0 r, define
(1)
If =p=min{m,n}, define v+1=0. Then
0DWUL[$SSUR[LPDWLRQLQ69'EDVLV
v
j
jjj vuA1
*V
Q
12
)(
2inf
d
u Q
Q
QVBAAA
Brank
CB nm