review of similarity transformation and singular value ...โฌยฆย ยท 2 1 similarity transformation a...
TRANSCRIPT
Review of similarity transformation and Singular ValueDecomposition
Nasser M. Abbasi
Applied Mathematics Department, California State University, Fullerton. July 8 2007 Compiled on May 19, 2020 at
2:03am
Contents
1 Similarity transformation 21.1 Derivation of similarity transformation based on algebraic method . . . . . . . . . . 21.2 Derivation of similarity transformation based on geometric method . . . . . . . . . 3
1.2.1 Finding matrix representation of linear transformation ๐ . . . . . . . . . . . 51.2.2 Finding matrix representation of change of basis . . . . . . . . . . . . . . . . 61.2.3 Examples of similarity transformation . . . . . . . . . . . . . . . . . . . . . . 81.2.4 How to find the best basis to simplify the representation of ๐? . . . . . . . . 11
1.3 Summary of similarity transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2 Singular value decomposition (SVD) 122.1 What is right and left eigenvectors? . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122.2 SVD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3 Conclusion 14
4 References 15
1
2
1 Similarity transformation
A similarity transformation is๐ต = ๐โ1๐ด๐
Where ๐ต,๐ด,๐ are square matrices. The goal of similarity transformation is to find a ๐ต matrix whichhas a simpler form than ๐ด so that we can use ๐ต in place of ๐ด to ease some computational work.Lets set our goal in having ๐ต be a diagonal matrix (a general diagonal form is called block diagonalor Jordan form, but here we are just looking at the case of ๐ต being a diagonal matrix).
The question becomes: Given ๐ด, can we find ๐ such that ๐โ1๐ด๐ is diagonal?
The standard method to show the above is via an algebraic method, which we show first. Next weshow a geometric method that explains similarity transformation geometrically.
1.1 Derivation of similarity transformation based on algebraic method
Starting with๐ต = ๐โ1๐ด๐
Our goal is to find a real matrix ๐ต such that it is diagonal. From the above, by pre multiplyingeach side by ๐ we obtain
๐ด๐ = ๐๐ต
Now, since our goal is to make ๐ต diagonal, let us select the eigenvalues of ๐ด to be the diagonal of๐ต. Now we write the above in expanded form as follows
โกโขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐
โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ=
โกโขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐
โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโฃ
๐1 0 00 โฑ 00 0 ๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
The above can be written as ๐ separate equations by looking at the column view of matrix multi-plication โก
โขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโฃ
๐11
โฎ๐๐1
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ= ๐1
โกโขโขโขโขโขโขโขโขโขโฃ
๐11
โฎ๐๐1
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
And โกโขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโฃ
๐12
โฎ๐๐2
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ= ๐2
โกโขโขโขโขโขโขโขโขโขโฃ
๐12
โฎ๐๐2
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
All the way to the n๐กโ column of ๐โกโขโขโขโขโขโขโขโขโขโฃ
๐11 โฏ ๐1๐โฎ โฑ โฎ๐๐1 โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโฃ
๐1๐
โฎ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ= ๐๐
โกโขโขโขโขโขโขโขโขโขโฃ
๐1๐
โฎ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
Each of the above ๐ equations is in the form
๐ด๐ = ๐๐
and this is the key idea.
The above shows that if we take the diagonal elements of ๐ต to be the eigenvalues of ๐ด then the ๐matrix is the (right) eigenvectors of ๐ด.
Hence, if we want the ๐ต matrix to be diagonal and real, this means the ๐ด matrix itself must havesome conditions put on it. Specifically, ๐ด eigenvalues must be all distinct and real. This happenswhen ๐ด is real and symmetric. Therefore, we have the general result that given a matrix ๐ด whichhas distinct and real eigenvalues, only then can we find a similar matrix to it which is real anddiagonal, and these diagonal values are the eigenvalues of ๐ด.
3
1.2 Derivation of similarity transformation based on geometric method
It turns out that this derivation is more complicated than the algebraic approach. It involves achange of basis matrix (which will be our๐ matrix) and the representation of linear transformation๐ as a matrix under some basis (which will be our ๐ด matrix). Let us start from the beginning.
Given the vector ๐ in โ๐, it can have many di๏ฟฝerent representations (or coordinates) depending onwhich basis are used. There are an infinite number of basis that span โ๐, hence there are infiniterepresentations for the same vector. Consider for example โ2. The standard basis vectors areโงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ, but another basis are
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ, and yet another are
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ0โ1
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ, and so on.
basis v
basis v
basis vChange of basis
In โ๐ any ๐ linearly independent vectors can be used as basis. Let ๐๐ฃ be the vector representationin w.r.t. basis ๐ฃ, and let ๐๐ฃโฒ be the vector representation w.r.t. basis ๐ฃโฒ.
basis v
basisv
Change of basis
Same vector X has different representation depending on which basis are used
xv x,yxv x,y
Mv v
basis v
An important problem in linear algebra is to obtain ๐๐ฃ given ๐๐ฃโฒ and the change of basis matrix[๐]๐ฃโฒโ๐ฃ.
This requires finding a matrix representation for the change of basis. Once the [๐]๐ฃโฒโ๐ฃ matrix isfound, we can write
๐๐ฃ = [๐]๐ฃโฒโ๐ฃ ๐๐ฃโฒ (1)
Where [๐]๐ฃโฒโ๐ฃ is the change of basis matrix which when applied to ๐๐ฃโฒ returns the coordinates ofthe vector w.r.t. basis ๐ฃ.
From (1) we see that given ๐๐ฃ, then to obtain ๐๐ฃโฒ we write
๐๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ๐๐ฃ (1A)
Another important problem is that of linear transformation ๐, where now we apply some transfor-mation on the whole space and we want to find what happens to the space coordinates (all theposition vectors) due to this transformation.
Consider for example ๐ which is a rotation in โ2 by some angle ๐. Here, every position vector in thespace is a๏ฟฝected by this rotation but the basis remain ๏ฟฝxed, and we want to find the new coordinatesof each point in space.
Let ๐๐ฃ be the position vector before applying the linear transformation, and let ๐๐ฃ be the newposition vector. Notice that both vectors are written with respect to the same basis ๐ฃ. Similar tochange basis, we want a way to find ๐๐ฃ from ๐๐ฃ, hence we need a way to represent this lineartransformation by a matrix, say [๐]๐ฃ with respect to the basis ๐ฃ and so we could write the following
๐๐ฃ = [๐]๐ฃ ๐๐ฃ (2)
4
basis v
Linear Transformation T
Same basis v but T causes change in vector representation
basis vT
v
yv
xv xv
Assume the basis is changed to ๐ฃโฒ. Let the representation of the same linear transformation ๐ w.r.t.basis ๐ฃโฒ be called [๐]๐ฃโฒ, so we write
๐๐ฃโฒ = [๐]๐ฃโฒ ๐๐ฃโฒ (2A)
basis vChange basis v->vโ
Combining change of basis and linear transformation
xv
basis v
xv
basis v
xv
yv T
v xv
Linear Transformatio
n
Tv
Hence when the basis changes from ๐ฃ to ๐ฃโฒ, ๐ representation changes from [๐]๐ฃ to [๐]๐ฃโฒ .
Now assume we are given some linear transformation ๐ and its representation [๐]๐ฃ w.r.t. basis ๐ฃ,how could we find ๐ representation [๐]๐ฃโฒ w.r.t. to new basis ๐ฃโฒ?
From (2) we have๐๐ฃ = [๐]๐ฃ ๐๐ฃ
But from (1) ๐๐ฃ = [๐]๐ฃโฒโ๐ฃ ๐๐ฃโฒ, hence the above becomes
๐๐ฃ = [๐]๐ฃ ๏ฟฝ[๐]๐ฃโฒโ๐ฃ ๐๐ฃโฒ๏ฟฝ
Pre-multiplying from left the above equation by [๐]โ1๐ฃโฒโ๐ฃ we obtain
[๐]โ1๐ฃโฒโ๐ฃ๐๐ฃ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ๐๐ฃโฒ
But since [๐]โ1๐ฃโฒโ๐ฃ๐๐ฃ = ๐๐ฃโฒ, then above reduces to
๐๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ๐๐ฃโฒ
But from (2A) ๐๐ฃโฒ = [๐]๐ฃโฒ ๐๐ฃโฒ, hence the above becomes
[๐]๐ฃโฒ๐๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ๐๐ฃโฒ
Therefore
[๐]๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ (3)
Notice the di๏ฟฝerence between change of basis and linear transformation. In change of basis, thevector ๐ remained fixed in space, but the basis changed, and we want to find the coordinates of thevector w.r.t the new basis. With linear transformation, the vector itself is changed from ๐๐ฃ to ๐๐ฃ, butboth vectors are expressed w.r.t. the same basis.
Equation (3) allows us to obtain a new matrix representation of the linear transformation ๐ byexpressing the same ๐ w.r.t. to di๏ฟฝerent basis. Hence if we are given a representation of ๐ which isnot the most optimal representation, we can, by change of basis, obtain a di๏ฟฝerent representationof the same ๐ by using (3). The most optimal matrix representation for linear transformation is adiagonal matrix.
5
Equation (3) is called a similarity transformation. To make it easier to compare (3) above withwhat we wrote in the previous section when we derived the above using an algebraic approach, welet [๐]๐ฃโฒ = ๐ต, [๐]๐ฃ = ๐ด, hence (3) is
๐ต = ๐โ1๐ด๐
The matrix [๐]๐ฃโฒ is similar to [๐]๐ฃ. (i.e. ๐ต is similar to ๐ด). Both matrices represent the same lineartransformation applied on the space. We will show below some examples of how to find [๐]๐ฃโฒ given[๐]๐ฃ and [๐]. But first we show how to obtain matrix representation of ๐.
1.2.1 Finding matrix representation of linear transformation ๐
Given a vector ๐ โ โ๐ and some linear transformation (or a linear operator) ๐ that acts on thevector ๐ transforming this vector into another vector ๐ โ โ๐ according to some prescribed manner๐ โถ ๐ โ ๐. Examples of such linear transformation are rotation, elongation, compression, andreflection, and any combination of these operations, but it can not include the translation of thevector, since translation is not linear.
The question to answer here is how to write down a representation of this linear transformation? Itturns out that ๐ can be represented by a matrix of size ๐ ร ๐, and the actual numbers that go intothe matrix, or the shape of the matrix, will depend on which basis in โ๐ we choose to represent ๐.
We would like to pick some basis so that the final shape of the matrix is the most simple shape.
Let us pick a set of basis ๐ฃ = {๐1, ๐2,โฏ , ๐๐} that span โ๐ hence
๐๐ฃ = ๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐
Now
๐๐๐ฃ = ๐ (๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐)
And since ๐ is linear we can rewrite the above as
๐๐๐ฃ = ๐1๐๐1 + ๐2๐๐2 +โฏ+ ๐๐๐๐๐ (1)
We see from above that the new transformed vector ๐๐๐ฃ has the same coordinates as ๐๐ฃ if we view๐๐๐ as the new basis.
Now ๐๐๐ itself is an application of the linear transformation but now it is being done on each basisvector ๐๐. Hence it will cause each specific basis vector to transform in some manner to a new vector.Whatever this new vector is, we must be able to represent it in terms of the same basis vectors{๐1, ๐2,โฏ , ๐๐} , therefore, we write
๐๐1 = ๐11๐1 + ๐21๐2 +โฏ+ ๐๐1๐๐
Where ๐๐๐ is the ๐๐กโ coordinate of the vector ๐๐๐. And we do the same for ๐๐2
๐๐2 = ๐12๐1 + ๐22๐2 +โฏ+ ๐๐2๐๐
And so on until๐๐๐ = ๐1๐๐1 + ๐2๐๐2 +โฏ+ ๐๐๐๐๐
Or๐๐๐ = ๐๐๐๐๐
Now plug the above into (1) and obtain
๐๐๐ฃ = ๐1 (๐11๐1 + ๐21๐2 +โฏ+ ๐๐1๐๐)+ ๐2 (๐12๐1 + ๐22๐2 +โฏ+ ๐๐2๐๐)+โฏ+ ๐๐ (๐1๐๐1 + ๐2๐๐2 +โฏ+ ๐๐๐๐๐)
Hence, if we take the ๐๐ as common factors, we have
๐๐ = ๐1 (๐1๐11 + ๐2๐12 +โฏ+ ๐๐๐1๐)+ ๐2 (๐1๐21 + ๐2๐22 +โฏ+ ๐๐๐2๐)+โฏ+ ๐๐ (๐1๐๐1 + ๐2๐๐2 +โฏ+ ๐๐๐๐๐)
6
Hence, since ๐๐๐ฃ = ๐1๐๐1 +๐2๐๐2 +โฏ+๐๐๐๐๐ from (1), then by comparing coe๏ฟฝcients of each basisvector ๐๐ we obtain
๐1๐ = ๐1๐11 + ๐2๐12 +โฏ+ ๐๐๐1๐๐2๐ = ๐1๐21 + ๐2๐22 +โฏ+ ๐๐๐2๐
โฏ๐๐๐ = ๐1๐๐1 + ๐2๐๐2 +โฏ+ ๐๐๐๐๐
Or in matrix form
๐
โโโโโโโโโโโโโโโโ
๐1๐2โฎ๐๐
โโโโโโโโโโโโโโโโ
=
โโโโโโโโโโโโโโโโ
๐11 ๐12 โฏ ๐1๐๐21 ๐22 โฏ ๐2๐โฎ โฎ โฏ โฎ
๐๐1 ๐๐2 โฏ ๐๐๐
โโโโโโโโโโโโโโโโ
โโโโโโโโโโโโโโโโ
๐1๐2โฎ๐๐
โโโโโโโโโโโโโโโโ
Hence we finally obtain that
๐ โ [๐]๐ฃ =
โโโโโโโโโโโโโโโโ
๐11 ๐12 โฏ ๐1๐๐21 ๐22 โฏ ๐2๐โฎ โฎ โฏ โฎ
๐๐1 ๐๐2 โฏ ๐๐๐
โโโโโโโโโโโโโโโโ
Let us call the matrix that represents ๐ under basis ๐ฃ as [๐]๐ฃ .
We see from the above that the ๐๐กโ column of [๐]๐ฃ contain the coordinates of the vector ๐๐๐. Thisgives us a quick way to construct [๐]๐ฃ: Apply ๐ to each basis vector ๐๐, and take the resulting vectorand place it in the ๐๐กโ column of [๐]๐ฃ.
We now see that [๐]๐ฃ will have a di๏ฟฝerent numerical values if the basis used to span โ๐ are di๏ฟฝerentfrom the ones used to obtain the above [๐]๐ฃ .
Let use pick some new basis, say ๐ฃโฒ = ๏ฟฝ๐โฒ1, ๐โฒ2,โฏ , ๐โฒ๐๏ฟฝ. Let the new representation of ๐ now be thematrix [๐]๐ฃโฒ, then the ๐๐กโ column of [๐]๐ฃโฒ is found from applying ๐ on the new basis ๐โฒ๐
๐๐โฒ๐ = ๐๐๐๐โฒ๐
Where now ๐๐๐ is the ๐๐กโ coordinates of the the vector ๐๐โฒ๐ which will be di๏ฟฝerent from ๐๐๐ since in
one case we used the basis set ๐ฃโฒ = ๏ฟฝ๐โฒ๐๏ฟฝ and in the other case we used the basis set ๐ฃ = {๐๐}. Hencewe see that [๐]๐ฃโฒ will numerically be di๏ฟฝerent depending on the basis used to span the space eventhough the linear transformation ๐ itself did not change.
1.2.2 Finding matrix representation of change of basis
Now we show how to determine [๐]๐ฃโ๐ฃโฒ, the matrix which when applied to a vector ๐๐ฃ will resultin the vector ๐๐ฃโฒ.
Given a vector ๐, it is represented w.r.t. basis ๐ฃ = {๐1, ๐2,โฏ , ๐๐} as
๐๐ฃ = ๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐
and w.r.t. basis ๐ฃโฒ = ๏ฟฝ๐โฒ1, ๐โฒ2,โฏ , ๐โฒ๐๏ฟฝ as
๐๐ฃโฒ = ๐1๐โฒ1 + ๐2๐โฒ2 +โฏ+ ๐๐๐โฒ๐
basis v basis v
e1
e2
xv a1e1 a2e2
a1
a2
e1
e2
xv b1e1 b2e2
b1
b2
The same vector having different representation depending on basis used
7
But the vector itself is invariant under any change of basis, hence ๐๐ฃ = ๐๐ฃโฒ
๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐ = ๐1๐โฒ1 + ๐2๐โฒ2 +โฏ+ ๐๐๐โฒ๐ (1)
Now write each basis ๐โฒ๐ in terms of the basis ๐๐, hence we have
๐โฒ1 = ๐11๐1 + ๐12๐2 +โฏ+ ๐1๐๐๐๐โฒ2 = ๐21๐1 + ๐22๐2 +โฏ+ ๐2๐๐๐
โฎ๐โฒ๐ = ๐๐1๐1 + ๐๐2๐2 +โฏ+ ๐๐๐๐๐ (2)
Substitute (2) into RHS of (1) we obtain
๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐ = ๐1 (๐11๐1 + ๐12๐2 +โฏ+ ๐1๐๐๐) +๐2 (๐21๐1 + ๐22๐2 +โฏ+ ๐2๐๐๐) +โฏ+ ๐๐ (๐๐1๐1 + ๐๐2๐2 +โฏ+ ๐๐๐๐๐)
Factor out the basis ๐๐ from the RHS, we obtain
๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐ = (๐1๐11๐1 + ๐1๐12๐2 +โฏ+ ๐1๐1๐๐๐) +(๐2๐21๐1 + ๐2๐22๐2 +โฏ+ ๐2๐2๐๐๐) +โฏ+ (๐๐๐๐1๐1 + ๐๐๐๐2๐2 +โฏ+ ๐๐๐๐๐๐๐)
Hence
๐1๐1 + ๐2๐2 +โฏ+ ๐๐๐๐ = ๐1 (๐1๐11 + ๐2๐21 +โฏ+ ๐๐๐๐1) +๐2 (๐1๐12 + ๐2๐22 +โฏ+ ๐๐๐๐2) +โฏ+ ๐๐ (๐1๐1๐ + ๐2๐2๐ +โฏ+ ๐๐๐๐๐)
Now comparing coe๏ฟฝcients of each of the basis ๐๐ we obtain the following result
๐1 = ๐1๐11 + ๐2๐21 +โฏ+ ๐๐๐๐1๐2 = ๐1๐12 + ๐2๐22 +โฏ+ ๐๐๐๐2โฏ
๐๐ = ๐1๐1๐ + ๐2๐2๐ +โฏ+ ๐๐๐๐๐
Or in Matrix form, we writeโกโขโขโขโขโขโขโขโขโขโขโขโขโขโขโฃ
๐1๐2โฎ๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
=
โกโขโขโขโขโขโขโขโขโขโขโขโขโขโขโฃ
๐11 ๐21 ๐31 โฏ ๐๐1๐12 ๐22 ๐32 โฏ ๐๐2โฎ โฏ โฏ โฑ โฎ๐1๐ ๐2๐ ๐3๐ โฏ ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโขโขโขโขโขโขโขโขโขโขโฃ
๐1๐2โฎ๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
(3)
The above gives a relation between the coordinates of the vector ๐ w.r.t. basis ๐ฃ (these are the ๐๐coordinates) to the coordinates of the same vector w.r.t. to basis ๐ฃโฒ (these are the coordinates ๐๐).The mapping between these coordinates is the matrix shown above which we call the [๐] matrix.Since the above matrix returns the coordinates of the vector w.r.t. ๐ฃ when it is multiplied by thecoordinates of the vector w.r.t. basis ๐ฃโฒ, we write it as [๐]๐ฃโฒโ๐ฃ to make it clear from which basis towhich basis is the conversion taking place.
Looking at (2) and (3) above, we see that column ๐ in [๐]๐ฃโฒโ๐ฃ is the coordinates of the ๐โฒ๐ w.r.t. basis
๐ฃ.
Hence the above gives a method to generate the change of basis matrix [๐]๐ฃโฒโ๐ฃ. Simply representeach basis in ๐ฃโฒ w.r.t. to basis ๐ฃ. Take the result of each such representation and write it as a columnin [๐]๐ฃโฒโ๐ฃ. We now show few examples to illustrate this.
Example showing how to ๏ฟฝnd change of basis matrix
Given the โ2 basis ๐ฃ = {๐1, ๐2} =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ, and new basis ๐ฃโฒ = ๏ฟฝ๐โฒ1, ๐โฒ2๏ฟฝ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃโ11
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ, find the change
of basis matrix [๐]๐ฃโ๐ฃโฒ
8
Column 1 of [๐]๐ฃโ๐ฃโฒ is the coordinates of ๐1 w.r.t. basis ๐ฃโฒ. i.e.
๐1 = ๐11๐โฒ1 + ๐21๐โฒ2 (4)
and column 2 of [๐]๐ฃโ๐ฃโฒ is the coordinates of ๐2 w.r.t. basis ๐ฃโฒ. i.e.
๐2 = ๐12๐โฒ1 + ๐22๐โฒ2 (5)
But (4) is โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ = ๐11
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ + ๐21
โกโขโขโขโขโฃโ11
โคโฅโฅโฅโฅโฆ
Hence 1 = ๐11 โ ๐21 and 0 = ๐11 + ๐21, solving, we obtain ๐11 =12 , ๐21 =
โ12 and (5) is
โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ = ๐12
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ + ๐22
โกโขโขโขโขโฃโ11
โคโฅโฅโฅโฅโฆ
Hence 0 = ๐12 โ ๐22 and 1 = ๐12 + ๐22, solving we obtain ๐12 =12 , ๐22 =
12 , hence
[๐]๐ฃโ๐ฃโฒ =โกโขโขโขโขโฃ12
12
โ12
12
โคโฅโฅโฅโฅโฆ
Now we can use the above change of basis matrix to find coordinates of the vector under ๐ฃโฒ given
its coordinates under ๐ฃ. For example, given ๐๐ฃ =โกโขโขโขโขโฃ105
โคโฅโฅโฅโฅโฆ, then
๐๐ฃโฒ =โกโขโขโขโขโฃ12
12
โ12
12
โคโฅโฅโฅโฅโฆ
โกโขโขโขโขโฃ105
โคโฅโฅโฅโฅโฆ =
โกโขโขโขโขโฃ7.5โ2.5
โคโฅโฅโฅโฅโฆ
1.2.3 Examples of similarity transformation
Example 1 This example from Strang book (Linear Algebra and its applications, 4th edition),but shown here with little more details.
Consider โ2, and let ๐ be the projection onto a line at angle ๐ = 1350. Let the first basis ๐ฃ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ
and let the second basis ๐ฃโฒ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ1โ1
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ.
basis v basis v
e1
e2
e1
e2
1
0
0
1
Te1 0.5
0.5
1350
Te2 0.5
0.5
Tv
0.5 0.5
0.5 0.5
Tv
1 0
0 0
Change of basis
The first step is to find [๐]๐ฃโฒโ๐ฃ. The first column of [๐]๐ฃโฒโ๐ฃ is found by writing ๐โฒ1w.r.t. basis ๐ฃ, andthe second column of [๐]๐ฃโฒโ๐ฃ is found by writing ๐โฒ2w.r.t. basis ๐ฃ, Hence
๐โฒ1 = ๐11๐1 + ๐21๐2โกโขโขโขโขโฃ1โ1
โคโฅโฅโฅโฅโฆ = ๐11
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ + ๐21
โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
9
Or 1 = ๐11 and ๐21 = โ1. And
๐โฒ2 = ๐12๐1 + ๐22๐2โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ = ๐12
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ + ๐22
โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
Hence ๐12 = 1 and ๐22 = 1. Hence
[๐]๐ฃโฒโ๐ฃ =โกโขโขโขโขโฃ1 1โ1 1
โคโฅโฅโฅโฅโฆ
Now we need to find [๐]๐ฃ the representation of ๐ w.r.t. basis ๐ฃ. The first column of [๐]๐ฃ is the newcoordinates of the basis ๐ฃ1 after applying ๐ onto it. but
๐๐1 = ๐โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
=โกโขโขโขโขโฃ0.5โ0.5
โคโฅโฅโฅโฅโฆ
and the second column of [๐]๐ฃ is the new coordinates of the basis ๐ฃ2 after applying ๐ onto it. but
๐๐2 = ๐โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ
=โกโขโขโขโขโฃโ0.50.5
โคโฅโฅโฅโฅโฆ
Hence
[๐]๐ฃ =โกโขโขโขโขโฃ0.5 โ0.5โ0.5 0.5
โคโฅโฅโฅโฅโฆ
Therefore
[๐]๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ
=โกโขโขโขโขโฃ1 1โ1 1
โคโฅโฅโฅโฅโฆ
โ1 โกโขโขโขโขโฃ0.5 โ0.5โ0.5 0.5
โคโฅโฅโฅโฅโฆ
โกโขโขโขโขโฃ1 1โ1 1
โคโฅโฅโฅโฅโฆ
[๐]๐ฃโฒ =โกโขโขโขโขโฃ1 00 0
โคโฅโฅโฅโฅโฆ
Hence we see that the linear transformation ๐ is represented as
โกโขโขโขโขโฃ1 00 0
โคโฅโฅโฅโฅโฆ w.r.t. basis ๐ฃโฒ, while it is
represented as
โกโขโขโขโขโฃ0.5 โ0.5โ0.5 0.5
โคโฅโฅโฅโฅโฆw.r.t. basis ๐ฃโฒ. Therefore, it will be better to perform all calculations
involving this linear transformation under basis ๐ฃโฒ instead of basis ๐ฃ
Example 2
10
basis v
basis v
e1
e2
e1
e2
1
0
0
1
Change of basis
4
Te1 cos
sinTe2
sin
cos
Tv
cos sin
sin cos
Tv
1
2 1
2
1
2
1
2
1
1
1
1
Tv
1
2 1
2
1
2
1
2
Change of basis selected did not result in simplification of representation of T
Consider โ2, and let ๐ be a rotation of space by ๐ = ๐4 . Let the first basis ๐ฃ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญand let the
second basis be ๐ฃโฒ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃโ11
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญ.
The first step is to find [๐]๐ฃโฒโ๐ฃ. The first column of [๐]๐ฃโฒโ๐ฃ is found by writing ๐โฒ1w.r.t. basis ๐ฃ, andthe second column of [๐]๐ฃโฒโ๐ฃ is found by writing ๐โฒ2w.r.t. basis ๐ฃ, Hence
๐โฒ1 = ๐11๐1 + ๐21๐2โกโขโขโขโขโฃ11
โคโฅโฅโฅโฅโฆ = ๐11
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ + ๐21
โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
Or 1 = ๐11 and ๐21 = 1 and
๐โฒ2 = ๐12๐1 + ๐22๐2โกโขโขโขโขโฃโ11
โคโฅโฅโฅโฅโฆ = ๐12
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ + ๐22
โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
Hence ๐12 = โ1 and ๐22 = 1. Hence
[๐]๐ฃโฒโ๐ฃ =โกโขโขโขโขโฃ1 โ11 1
โคโฅโฅโฅโฅโฆ
Now we need to find [๐]๐ฃ the representation of ๐ w.r.t. basis ๐ฃ. The first column of [๐]๐ฃ is the newcoordinates of the basis ๐ฃ1 after applying ๐ onto it. but
๐๐1 = ๐โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
=โกโขโขโขโขโฃcos๐sin๐
โคโฅโฅโฅโฅโฆ
and the second column of [๐]๐ฃ is the new coordinates of the basis ๐ฃ2 after applying ๐ onto it. but
๐๐2 = ๐โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ
๐๐2 =โกโขโขโขโขโฃโ sin๐cos๐
โคโฅโฅโฅโฅโฆ
11
Hence
[๐]๐ฃ =โกโขโขโขโขโฃcos๐ โ sin๐sin๐ cos๐
โคโฅโฅโฅโฅโฆ
=โกโขโขโขโขโฃcos ๐
4 โ sin ๐4
sin ๐4 cos ๐
4
โคโฅโฅโฅโฅโฆ
[๐]๐ฃ =
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ
Therefore
[๐]๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ
=โกโขโขโขโขโฃ1 โ11 1
โคโฅโฅโฅโฅโฆ
โ1 โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโฃ1 โ11 1
โคโฅโฅโฅโฅโฆ
=โกโขโขโขโขโฃ
12
12
โ 12
12
โคโฅโฅโฅโฅโฆ
โกโขโขโขโขโฃ0 โโ2โ2 0
โคโฅโฅโฅโฅโฆ
=
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ
Hence we see that the linear transformation ๐ is represented as
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ w.r.t. basis ๐ฃ
โฒ, while it is
represented as
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆw.r.t. basis ๐ฃ
โฒ. Therefore the change of basis selected above did not result
in making the representation of ๐ any simpler (in fact there was no change in the representation).This means we need to find a more direct method of finding the basis under which ๐ has thesimplest representation. Clearly we canโt just keep trying di๏ฟฝerent basis to find if ๐ has simplerrepresentation under the new basis.
1.2.4 How to ๏ฟฝnd the best basis to simplify the representation of ๐?
Our goal of simpler representation of ๐ is that of a diagonal matrix or as close as possible to beingdiagonal matrix (i.e. Jordan form). Given [๐]๐ฃ and an infinite number of basis ๐ฃโฒ that we couldselect to represent ๐ under in the hope we can find a new representation [๐]๐ฃโฒ such that it is simplerthan [๐]๐ฃ, we now ask, how does one go about finding such basis?
It turns out the if we select the eigenvectors of [๐]๐ฃ as the columns of [๐]๐ฃโฒโ๐ฃ, this will result in[๐]๐ฃโฒ being diagonal or as close to being diagonal as possible (block diagonal or Jordan form).
Let us apply this to the second example in the previous section. In that example, we had
[๐]๐ฃ =
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ
The eigenvectors of [๐]๐ฃ areโกโขโขโขโขโฃโ๐1
โคโฅโฅโฅโฅโฆ and
โกโขโขโขโขโฃ๐1
โคโฅโฅโฅโฅโฆ, [๐]๐ฃโฒโ๐ฃ =
โกโขโขโขโขโฃโ๐ ๐1 1
โคโฅโฅโฅโฅโฆ. Therefore
[๐]๐ฃโฒ = [๐]โ1๐ฃโฒโ๐ฃ[๐]๐ฃ[๐]๐ฃโฒโ๐ฃ
=โกโขโขโขโขโฃโ๐ ๐1 1
โคโฅโฅโฅโฅโฆ
โ1 โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ
โกโขโขโขโขโฃโ๐ ๐1 1
โคโฅโฅโฅโฅโฆ
[๐]๐ฃโฒ =
โกโขโขโขโขโขโขโขโฃ
๏ฟฝ 12 โ
12 ๐๏ฟฝโ2 0
0 ๏ฟฝ 12 +
12 ๐๏ฟฝโ2
โคโฅโฅโฅโฅโฅโฅโฅโฆ
Which is diagonal representation. Hence we see that the linear transformation ๐ is represented asโกโขโขโขโขโขโขโขโฃ
๏ฟฝ 12 โ
12 ๐๏ฟฝโ2 0
0 ๏ฟฝ 12 +
12 ๐๏ฟฝโ2
โคโฅโฅโฅโฅโฅโฅโฅโฆw.r.t. basis ๐ฃโฒ
12
1.3 Summary of similarity transformation
To obtain ๐ต = ๐โ1๐ด๐ such that ๐ต is real and diagonal requires that ๐ด be real and symmetric. Theeigenvalues of ๐ด goes into the diagonal of ๐ต and the eigenvectors of ๐ด go into the columns of ๐.This is an algebraic view.
Geometrically,๐ด is viewed as the matrix representation under some basis ๐ฃ of a linear transformation๐. And ๐ต is the matrix representation of the same linear transformation ๐ but now under a newbasis ๐ฃโฒ and ๐ is the matrix that represents the change of basis from ๐ฃโฒ to ๐ฃ.
The question then immediately arise: If ๐ดmust be real and symmetric (for ๐ต to be real and diagonal),what does this mean in terms of the linear transformation ๐ and change of basis matrix ๐? Thisclearly mean that, under some basis ๐ฃ, not every linear transformation represented by ๐ด can have asimilar matrix ๐ต which is real and diagonal. Only those linear transformations which result in realand symmetric representation can have a similar matrix which is real and diagonal. This is shownin the previous examples, where we saw that ๐ defined as rotation by 450 under the standard basis
๐ฃ =
โงโชโชโจโชโชโฉ
โกโขโขโขโขโฃ10
โคโฅโฅโฅโฅโฆ ,โกโขโขโขโขโฃ01
โคโฅโฅโฅโฅโฆ
โซโชโชโฌโชโชโญresulted in ๐ด =
โกโขโขโขโขโขโขโฃ
1
โ2โ 1
โ21
โ21
โ2
โคโฅโฅโฅโฅโฅโฅโฆ and since this is not symmetric, hence we will not be able
to factor this matrix using similarity transformation to a diagonal matrix, no matter which changeof basis we try to represent ๐ under. The question then, under a given basis ๐ฃ what is the class oflinear transformation which leads to a matrix representation that is symmetric? One such examplewas shown above which is the projection into the line at 1350. This question needs more time tolook into.
We now write ฮ to represent a diagonal matrix, hence the similarity transformation above can bewritten as
ฮ = ๐โ1๐ด๐
Or๐ด = ๐ฮ๐โ1
A M M1
Eigenvalues of A go into the diagonal
Eigenvectors of A go into the columns
A is real and symmetric
2 Singular value decomposition (SVD)
Using similarity transformation, we found that for a real and symmetric matrix ๐ด we are able todecompose it as ๐ด = ๐ฮ๐โ1 where ฮ is diagonal and contains the eigenvalues of ๐ด on the diagonal,and ๐ contains the right eigenvectors of ๐ด in its columns.
2.1 What is right and left eigenvectors?
Right eigenvectors are the standard eigenvectors we have been talking about all the time. When ๐is an eigenvalues of ๐ด then ๐ is a right eigenvector when we write
๐ด๐ = ๐๐
However, ๐ is a left eigenvector of ๐ด when we have
๐๐ป๐ด = ๐๐๐ป
where ๐๐ป is the Hermitian of ๐
2.2 SVD
Given any arbitrary matrix ๐ด๐ร๐ it can be factored into 3 matrices as follows ๐ด๐ร๐ = ๐๐ร๐๐ท๐ร๐๐๐ร๐where ๐ is a unitary matrix (๐๐ป = ๐โ1 or ๐๐ป๐ = ๐ผ), and ๐ is also unitary matrix.
These are the steps to do SVD
1. Find the rank of ๐ด, say ๐
13
2. Let ๐ต = ๐ด๐ป๐ร๐๐ด๐ร๐, hence ๐ต๐ร๐ is a square matrix, and it is semi positive definite, hence its
eigenvalues will all be โฅ 0. Find the eigenvalues of ๐ต, call these ๐2๐ . There will be ๐ sucheigenvalues since ๐ต is of order ๐ร๐. But only ๐ of these will be positive, and ๐โ ๐ will be zero.Arrange these eigenvalues such that the first ๐ non-zero eigenvalues come first, followed by
the zero eigenvalues:
๐ ๐๐๐๐๐๐ฃ๐๐๐ข๐๐
๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐21, ๐22,โฏ , ๐2๐ , 0, 0,โฏ , 0
3. Initialize matrix ๐ท๐ร๐ to be all zeros. Take the the first ๐ eigenvalues from above (non-zero
ones), and take the square root of each, hence we get๐ singular values
๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐1, ๐2,โฏ , ๐๐ ,and write these down thediagonal of ๐ท starting at ๐ท (1, 1), i.e. ๐ท (1, 1) = ๐1, ๐ท (2, 2) = ๐2,โฏ ,๐ท (๐, ๐) = ๐๐. Notice that thematrix ๐ท need not square matrix. Hence we can obtain an arrangement such as the followingfor ๐ = 2
๐ท =
โโโโโโโโโโโ
๐1 0 0 00 ๐2 0 00 0 0 0
โโโโโโโโโโโ
where the matrix ๐ด was 3 ร 4 for example.
4. Now find each eigenvalue of ๐ด๐ป๐ด. For each eigenvalue, find the corresponding eigenvector.Call these eigenvectors ๐1, ๐2,โฏ , ๐๐.
5. Normalize the eigenvectors found in the above step. Now ๐1, ๐2,โฏ , ๐๐ eigenvector will bean orthonormal set of vectors. Take the Hermitian of each eigenvector ๐๐ป1 , ๐๐ป2 ,โฏ , ๐๐ป๐ andmake one of these vectors (now in row format instead of column format) go into a rowin the matrix ๐.i.e. the first row of ๐ will be ๐๐ป1 , the second row of ๐ will be ๐๐ป2 , etc...
๏ฟฝ๐ด๐ป๐ด๏ฟฝ๐ร๐
find eigenvectors and normalizeโ
โงโชโชโจโชโชโฉ๐1, ๐2,โฏ , ๐๐,
ortho basis (n-r) for N(A)
๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐๐+1,โฏ , ๐๐
โซโชโชโฌโชโชโญโ ๐๐ร๐ =
โกโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโขโฃ
๐๐1๐๐2โฎ๐๐๐๐๐๐+1โฎ๐๐๐
โคโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฅโฆ
6. Now we need to find a set of ๐ orthonormal vectors, these will be the columns of the matrix๐: find a set of ๐ orthonormal eigenvector ๐๐ =
1๐๐๐ด๐๐, for ๐ = 1โฏ๐. Notice that here we only
use the first ๐ vectors found in step 5. Take each one of these ๐๐ vectors and make them intocolumns of ๐. But since we need ๐ columns in ๐ not just ๐, we need to come up with ๐ โ ๐more basis vectors such that all the ๐ vectors form an orthonormal set of basis vectors forthe row space of ๐ด, i.e. โ๐. If doing this by hand, it is easy to find the this ๐โ ๐ by inspection.In a program, we could use the same process we used with Gram-Schmidt, where we learnedhow find a new vector which is orthonormal to a an existing set of other vectors. Anotherway to find the matrix ๐ is to construct the matrix ๐ด๐ด๐ป and find its eigenvectors, those gointo the columns of the matrix ๐ as follows
๏ฟฝ๐ด๐ด๐ป๏ฟฝ๐ร๐
find eigenvectors and normalizeโ
โงโชโชโจโชโชโฉ
orthonormal basis for range A
๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๏ฟฝ๐1, ๐2,โฏ , ๐๐ , ๐๐+1,โฏ , ๐๐
โซโชโชโฌโชโชโญโ ๐๐ร๐ = ๏ฟฝ๐1 ๐2 โฏ ๐๐ ๐๐+1 โฏ ๐๐๏ฟฝ
7. This completes the factorization, now we have ๐ด = ๐ ๐ท ๐
The following diagram help illustrate the SVD process described above.
14
For these n r vectors
Aui 0
Hence basis for NA
Row space(A)
Null space(A)
A(MxN) matrix, Rank=r
Rn
Dimension=r
Dimension= n-r
Column space(A)
(or range of A)
Left Null space(A)
Dimension=r
Dimension= m-r
Rm
A
a11 a12 a1n
am1 am2 amn
A PDQ
SVD factorization
Orthonormal Basis for Null Space of A
AHAnn
f ind eigenvectors
u1,u2,,ur,
ortho basis for N(A)
ur1,,un Qnn
u1T
u2T
urT
ur1T
unT
AAH mm
f ind eigenvectors
r ortho basis for range A
v1,v2,,vr ,vr1,,vm Pmm v1 v2 vr vr1 vm
Amn PmmDmnQnn
D1.vsd
By Nasser M. Abbasi
July 3,2007
From these r vectors we obtain
vi Aui
i
Hence r basis for Range A
And normalize
And normalize
n eigenvalues
12,2
2,,r
2, 0,0,, 0
3 Conclusion
As a conclusion, the following diagram shows the di๏ฟฝerence between similarity transformationfactorization of a matrix and SVD.
15
A M M1
Eigenvalues of A go into the diagonal
Eigenvectors of A go into the columns
A is real, sqaure and symmetric
Similarity Transformation
nxn nxn nxn nxn
Diagonal matrix
A P D QA is arbitrary mxn matrix
AAH
Normalized Eigenvectors of AAโ go into the columns of P
AHANormalized Eigenvectors of AโA go into the rows of Q
The square root of the nonzero eigenvalues of AAโ (or AโA) go into the diagonal of matrix D. These are called the singular values of A
mxnmxm mxn nxn
Nasser M. Abbasi
diag10.vsdx
Trying to understand SVD from geometrical point view requires more time, and this is left forfuture research into this subject.
4 References
1. Linear Algebra and its applications 4th edition by Gilbert Strang
2. Course notes, Linear Algebra, Math 307, CSUF mathematics department, spring 2007 by DrAngel Pineda.
3. Principles and techniques of applied Mathematics by Bernard Friedman, Spectral theory ofoperators chapter.
4. Matrix computation, 3rd edition by Gene Golub and Charles Van Loan.
5. Numerical analysis: Mathematics of scientific computing, by David Kincaid and Ward Cheney.