l.vandenberghe ece133a(fall2019) 5.orthogonalmatricesvandenbe/133a/lectures/...matrix-vectorproduct...

L. Vandenberghe ECE133A (Spring 2021)

5. Orthogonal matrices

• matrices with orthonormal columns

• orthogonal matrices

• tall matrices with orthonormal columns

• complex matrices with orthonormal columns

5.1

Orthonormal vectors

a collection of real <-vectors 01, 02, . . . , 0= is orthonormal if

• the vectors have unit norm: ‖08‖ = 1

• they are mutually orthogonal: 0)80 9 = 0 if 8 ≠ 9

Example

00−1

,1√2

110

,1√2

1−1

0

Orthogonal matrices 5.2

Matrix with orthonormal columns

� ∈ R<×= has orthonormal columns if its Gram matrix is the identity matrix:

�)� =[01 02 · · · 0=

]) [01 02 · · · 0=

]=

0)101 0)102 · · · 0)10=0)201 0)202 · · · 0)20=... ... . . . ...

0)=01 0)=02 · · · 0)=0=

=

1 0 · · · 00 1 · · · 0... ... . . . ...

0 0 · · · 1

there is no standard short name for “matrix with orthonormal columns”


Matrix-vector product

if � ∈ R<×= has orthonormal columns, then the linear function 5 (G) = �G

• preserves inner products:

(�G)) (�H) = G)�)�H = G) H

• preserves norms:

‖�G‖ =((�G)) (�G)

)1/2= (G)G)1/2 = ‖G‖

• preserves distances: ‖�G − �H‖ = ‖G − H‖

• preserves angles:

∠(�G, �H) = arccos( (�G)) (�H)‖�G‖‖�H‖

)= arccos

(G) H

‖G‖‖H‖

)= ∠(G, H)


Left-invertibility

if � ∈ R<×= has orthonormal columns, then

• � is left-invertible with left inverse �) : by definition

�)� = �

• � has linearly independent columns (from page 4.24 or page 5.2):

�G = 0 =⇒ �)�G = G = 0

• � is tall or square: < ≥ = (see page 4.13)


Outline





Orthogonal matrix

Orthogonal matrix

a square real matrix with orthonormal columns is called orthogonal

Nonsingularity (from equivalences on page 4.14): if � is orthogonal, then

• � is invertible, with inverse �) :

�)� = �

� is square

}=⇒ ��) = �

• �) is also an orthogonal matrix

• rows of � are orthonormal (have norm one and are mutually orthogonal)

Note: if � ∈ R<×= has orthonormal columns and < > =, then ��) ≠ �


Permutation matrix

• let c = (c1, c2, . . . , c=) be a permutation (reordering) of (1, 2, . . . , =)• we associate with c the = × = permutation matrix �

�8c8 = 1, �8 9 = 0 if 9 ≠ c8

• �G is a permutation of the elements of G: �G = (Gc1, Gc2, . . . , Gc=)• � has exactly one element equal to 1 in each row and each column

Orthogonality: permutation matrices are orthogonal

• �)� = � because � has exactly one element equal to one in each row

(�)�)8 9 ==∑:=1

�:8�: 9 =

{1 8 = 9

0 otherwise

• �) = �−1 is the inverse permutation matrix


Example

• permutation on {1, 2, 3, 4}

(c1, c2, c3, c4) = (2, 4, 1, 3)

• corresponding permutation matrix and its inverse

� =

0 1 0 00 0 0 11 0 0 00 0 1 0

, �−1 = �) =

0 0 1 01 0 0 00 0 0 10 1 0 0

• �) is permutation matrix associated with the permutation

(c1, c2, c3, c4) = (3, 1, 4, 2)


Plane rotation

Rotation in a plane

� =

[cos \ − sin \sin \ cos \

]x

Ax

θ

Rotation in a coordinate plane in R=: for example,

� =

cos \ 0 − sin \

0 1 0sin \ 0 cos \

describes a rotation in the (G1, G3) plane in R3


Reflector

Reflector: a matrix of the form

� = � − 200)

with 0 a unit-norm vector (‖0‖ = 1)

Properties

• a reflector matrix is symmetric

• a reflector matrix is orthogonal

�)� = (� − 200)) (� − 200)) = � − 400) + 400)00) = �


Geometrical interpretation of reflector

z = Ax = (I − 2aaT)x

H

line through a and origin

x

y = (I − aaT)x0

• � = {D | 0)D = 0} is the (hyper-)plane of vectors orthogonal to 0

• if ‖0‖ = 1, the projection of G on � is given by

H = G − (0)G)0 = G − 0(0)G) = (� − 00))G

(see next page)

• reflection of G through the hyperplane is given by product with reflector:

I = H + (H − G) = (� − 200))G


Exercise

suppose ‖0‖ = 1; show that the projection of G on � = {D | 0)D = 0} is

H = G − (0)G)0

• we verify that H ∈ �:

0) H = 0) (G − 0(0)G)) = 0)G − (0)0) (0)G) = 0)G − 0)G = 0

• now consider any I ∈ � with I ≠ H and show that ‖G − I‖ > ‖G − H‖:

‖G − I‖2 = ‖G − H + H − I‖2= ‖G − H‖2 + 2(G − H)) (H − I) + ‖H − I‖2= ‖G − H‖2 + 2(0)G)0) (H − I) + ‖H − I‖2= ‖G − H‖2 + ‖H − I‖2 (because 0) H = 0) I = 0)

> ‖G − H‖2


Product of orthogonal matrices

if �1, . . . , �: are orthogonal matrices and of equal size, then the product

� = �1�2 · · · �:

is orthogonal:

�)� = (�1�2 · · · �:)) (�1�2 · · · �:)= �): · · · �)2 �)1 �1�2 · · · �:= �


Linear equation with orthogonal matrix

linear equation with orthogonal coefficient matrix � of size = × =

�G = 1

solution isG = �−11 = �)1

• can be computed in 2=2 flops by matrix-vector multiplication

• cost is less than order =2 if � has special properties; for example,

permutation matrix: 0 flopsreflector (given 0): order = flopsplane rotation: order 1 flops


Outline





Tall matrix with orthonormal columns

suppose � ∈ R<×= is tall (< > =) and has orthonormal columns

• �) is a left inverse of �:�)� = �

• � has no right inverse; in particular

��) ≠ �

on the next pages, we give a geometric interpretation to the matrix ��)


Range

• the span of a collection of vectors is the set of all their linear combinations:

span(01, 02, . . . , 0=) = {G101 + G202 + · · · + G=0= | G ∈ R=}

• the range of a matrix � ∈ R<×= is the span of its column vectors:

range(�) = {�G | G ∈ R=}

Example

range(

1 01 20 −1

) =

G1G1 + 2G2−G2

| G1, G2 ∈ R


Projection on range of matrix with orthonormal columns

suppose � ∈ R<×= has orthonormal columns; we show that the vector

��)1

is the orthogonal projection of an <-vector 1 on range(�)

range(A)

b

AAT b

• G = �)1 satisfies ‖�G − 1‖ < ‖�G − 1‖ for all G ≠ G• this extends the result on page 2.12 (where � = (1/‖0‖)0)


Proof

the squared distance of 1 to an arbitrary point �G in range(�) is

‖�G − 1‖2 = ‖�(G − G) + �G − 1‖2 (where G = �)1)

= ‖�(G − G)‖2 + ‖�G − 1‖2 + 2(G − G))�) (�G − 1)= ‖�(G − G)‖2 + ‖�G − 1‖2= ‖G − G‖2 + ‖�G − 1‖2≥ ‖�G − 1‖2

with equality only if G = G

• line 3 follows because �) (�G − 1) = G − �)1 = 0

• line 4 follows from �)� = �


Outline





Gram matrix

� ∈ C<×= has orthonormal columns if its Gram matrix is the identity matrix:

�� =[01 02 · · · 0=

]� [01 02 · · · 0=

]=

0�1 01 0�1 02 · · · 0�1 0=0�2 01 0�2 02 · · · 0�2 0=... ... ...

0�= 01 0�= 02 · · · 0�= 0=

=

1 0 · · · 00 1 · · · 0... ... . . . ...

0 0 · · · 1

• columns have unit norm: ‖08‖2 = 0�

808 = 1

• columns are mutually orthogonal: 0�80 9 = 0 for 8 ≠ 9


Unitary matrix

Unitary matrix

a square complex matrix with orthonormal columns is called unitary

Inverse

�� = �

� is square

}=⇒ �� = �

• a unitary matrix is nonsingular with inverse ��

• if � is unitary, then �� is unitary


Discrete Fourier transform matrix

recall definition from page 3.37 (with l = 42cj/= and j =√−1)

, =

1 1 1 · · · 11 l−1 l−2 · · · l−(=−1)

1 l−2 l−4 · · · l−2(=−1)... ... ... ...

1 l−(=−1) l−2(=−1) · · · l−(=−1) (=−1)

the matrix (1/√=), is unitary (proof on next page):

1=,�, =

1=,,� = �

• inverse of, is,−1 = (1/=),�

• inverse discrete Fourier transform of =-vector G is,−1G = (1/=),�G


Gram matrix of DFT matrix

we show that,�, = =�

• conjugate transpose of, is

,� =

1 1 1 · · · 11 l l2 · · · l=−1

1 l2 l4 · · · l2(=−1)... ... ... ...

1 l=−1 l2(=−1) · · · l(=−1) (=−1)

• 8, 9 element of Gram matrix is

(,�,)8 9 = 1 + l8− 9 + l2(8− 9) + · · · + l(=−1) (8− 9)

(,�,)88 = =, (,�,)8 9 = l=(8− 9) − 1l8− 9 − 1

= 0 if 8 ≠ 9

(last step follows from l= = 1)


l.vandenberghe ece133a(fall2019) 5.orthogonalmatricesvandenbe/133a/lectures/...matrix-vectorproduct...

Documents