orthogonal bases

Lesson 13Orthonormal Bases and Orthogonal Projections

Math 21b

March 9, 2007

Announcements

I Graded midterms will be returned after class. µ = 77, σ = 10

I Solutions to the first midterm are on the course website.

I Homework for March 12: 5.1: 6,16,18,20,28,38*,14*

I Problem Session Wednesdays, 7-8 PM in SC 101b

I Office hours: Monday 2-4, Tuesday 3-5 in SC 323 (resumingMonday)

Here’s Felix!

Our friend the dot product

DefinitionIf ~u and ~v are two vectors in Rn, we define their dot product to be

~u · ~v = u1v1 + · · ·+ unvn.

In Math 21a1 the dot product in R2 and R3 is used for manydifferent geometric operations:

I projecting a vector onto a line

I Find the equation of a plane

I measuring the angle between two vectors

We want to generalize these constructions to several dimensions.

1Read Appendix A if you did not take Math 21a.

Definition

a. Two vectors ~v and ~w in Rn are called perpendicular ororthogonal if ~v · ~w = 0.

b. The length or magnitude or norm of a vector ~v in Rn is thescalar ‖~v‖ =

√~v · ~v .

c. A vector ~v in Rn is called a unit vector if its length is 1, i.e.,‖~v‖ = 1 or ~v · ~v = 1.

Example

Find two unit vectors in R2 which are orthogonal.

SolutionOne pair is (~e1,~e2). Can you find more?

Example

Find two unit vectors in R2 which are orthogonal.

SolutionOne pair is (~e1,~e2). Can you find more?

Definition

I A set (~u1, ~u2, . . . , ~um) of vectors in Rn is called orthogonal ifeach is orthogonal to every other.

I The set is called orthonormal if additionally every vector is aunit vector.

Thus a set (~u1, ~u2, . . . , ~um) of vectors in Rn is orthonormal if andonly if

~ui · ~uj =

{0 if i 6= j

1 if i = j.

The physicists have a slick way to write this: ~ui · ~uj = δij .

Definition

~ui · ~uj =

{0 if i 6= j

1 if i = j.

Definition

~ui · ~uj =

{0 if i 6= j

1 if i = j.

Example

In R2, the vectors

[cos θsin θ

[− sin θcos θ

]are orthonormal.

Given a general set of vectors (~v1, ~v2, . . . , ~vm) of vectors in Rn, itwas a drag to decide if they were linearly independent. Not soorthonormal sets:

a. Orthonormal sets are linearly independent.

b. An orthonormal set of n vectors in Rn is a basis for Rn.

Proof of (b).

A set of n orthonormal vectors in Rn spans an n-dimensionalsubspace of Rn, of which there is only one, Rn itself.

Given a general set of vectors (~v1, ~v2, . . . , ~vm) of vectors in Rn, itwas a drag to decide if they were linearly independent. Not soorthonormal sets:

a. Orthonormal sets are linearly independent.

b. An orthonormal set of n vectors in Rn is a basis for Rn.

Proof of (b).

A set of n orthonormal vectors in Rn spans an n-dimensionalsubspace of Rn, of which there is only one, Rn itself.

Proof of (a).

Suppose (~u1, ~u2, . . . , ~um) is an orthonormal set of vectors in Rn,and suppose there are scalars c1, c2, . . . , cm such that

~0 = c1~u1 + c2~u2 + · · ·+ cm~um.

We will show all ci must be zero.

To do this, pick one of the ~ui

and dot the above equation with it. The above implies

0 = ~0 · ~ui = c1(~u1 · ~ui ) + c2(~u2 · ~ui ) + · · ·+ cm(~um · ~ui )

Of the dot products on the right-hand side above, most of themare zero except for the ith one, which is one. Hence

0 = ci · 1.

But i was arbitrary, so all c1 = c2 = · · · = cm = 0.

Proof of (a).

~0 = c1~u1 + c2~u2 + · · ·+ cm~um.

We will show all ci must be zero. To do this, pick one of the ~ui

0 = ci · 1.

Proof of (a).

~0 = c1~u1 + c2~u2 + · · ·+ cm~um.

0 = ci · 1.

Proof of (a).

~0 = c1~u1 + c2~u2 + · · ·+ cm~um.

0 = ci · 1.

Coordinates with respect to an orthonormal basis

Example

Let V be the subspace of R3 consisting of those vectors[x1 x2 x3

]Tsuch that x1 + x2 + x3 = 0. The set

A = (~u1, ~u2) =

1−10

,1√6

11−2

is an orthonormal basis for V . Find the coordinates of~v =

[3 4 −7

]Tin this basis.

Solution (Old way)

Form an augmented matrix and row reduce: 1/√

2 1/√

6 3−1/

√2 1/

√6 4

0 −2/√

6 −7

1 0 −1/

0 1 7√

Coordinates with respect to an orthonormal basis

Example

Let V be the subspace of R3 consisting of those vectors[x1 x2 x3

]Tsuch that x1 + x2 + x3 = 0. The set

A = (~u1, ~u2) =

1−10

,1√6

11−2

is an orthonormal basis for V . Find the coordinates of~v =

[3 4 −7

]Tin this basis.

Solution (Old way)

Form an augmented matrix and row reduce: 1/√

2 1/√

6 3−1/

√2 1/

√6 4

0 −2/√

6 −7

1 0 −1/

0 1 7√

Solution (New way)

Since A is a basis, we know

~v = c1~u1 + c2~u2

So dot with ~u1:

~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)

− 1√2

= c1(1) + c2(0) = c1.

Likewise

c2 = ~v · ~u2 =21√

Solution (New way)

~v = c1~u1 + c2~u2

So dot with ~u1:

~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)

− 1√2

= c1(1) + c2(0) = c1.

Likewise

c2 = ~v · ~u2 =21√

Solution (New way)

~v = c1~u1 + c2~u2

So dot with ~u1:

~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)

− 1√2

= c1(1) + c2(0) = c1.

Likewise

c2 = ~v · ~u2 =21√

Solution (New way)

~v = c1~u1 + c2~u2

So dot with ~u1:

~v · ~u1 = c1(~u1 · ~u1) + c2(~u2 · ~u1)

− 1√2

= c1(1) + c2(0) = c1.

Likewise

c2 = ~v · ~u2 =21√

Summary

FactSuppose A = (~u1, ~u2, . . . , ~um) is an orthonormal basis for asubspace V of Rn, and ~v is in V . Then

[~v ]A =

~v · ~u1

~v · ~u2...

~v · ~um

The orthogonal complement

DefinitionLet V be a subspace of Rn. The set

V⊥ = { ~w ∈ V | ~w · ~v = 0 for all v ∈ V }

is called the orthogonal complement to V .

FactV⊥ is a subspace of Rn.

FactLet (~u1, . . . , ~um) be a basis for V . A vector ~w is in V⊥ if and onlyif ~w · ~vi = 0 for each i .

V⊥ = { ~w ∈ V | ~w · ~v = 0 for all v ∈ V }

Finding a basis for V⊥

Example

The orthonormal vectors ~u1 = 1√30

[1 2 3 4

~u2 = 1√12

[−3 1 −1 1

]Tspan a subspace V of R4. Find a

basis for V⊥.

SolutionA vector w is in V⊥ if and only if ~w · ~u1 = ~w · ~u2 = 0. This is trueif and only if ~w is in the kernel of the matrix[

1 2 3 4−3 1 −1 1

[1 0 5/7 2/7

0 1 8/7 13/7

]The kernel is spanned by

[−5/7 −8/7 1 0

]Tand[

−2/7 −13/7 0 1]T

Finding a basis for V⊥

Example

[1 2 3 4

~u2 = 1√12

[−3 1 −1 1

]Tspan a subspace V of R4. Find a

basis for V⊥.

SolutionA vector w is in V⊥ if and only if ~w · ~u1 = ~w · ~u2 = 0. This is trueif and only if ~w is in the kernel of the matrix[

1 2 3 4−3 1 −1 1

[1 0 5/7 2/7

0 1 8/7 13/7

]The kernel is spanned by

[−5/7 −8/7 1 0

]Tand[

−2/7 −13/7 0 1]T

Warning!

Although this process produces a basis for V⊥ no matter whatbasis for V you use (orthonormal or not), it will not necessarilyproduce an orthogonal basis for V⊥.

FactLet S be an n ×m matrix. Then

(image S)⊥ = ker(ST ).

Note that both sides of the equation are subspaces of Rn.

Proof.Let S =

[~v1 · · · ~vm

]. Let ~x be in Rn. Then

ST~x =

~v1 · ~x...

~vm · ~x

ST~x = ~0 ⇐⇒ ~vi · ~x = 0 for all i ⇐⇒ ~x ∈ (image S)⊥

Proof.Let S =

[~v1 · · · ~vm

ST~x =

~v1 · ~x...

~vm · ~x

Proof.Let S =

[~v1 · · · ~vm

ST~x =

~v1 · ~x...

~vm · ~x

Orthogonal projections

FactLet L be any line and ~x any vector. Then ~x can be decomposed as

~x = ~x‖ + ~x⊥

where ~x‖ is parallel to L and ~x⊥ is perpendicular to L.

~x = ~x‖ + ~x⊥

Proof.Let ~u be any unit vector spanning L. Then ~x‖ = k~u for some k,and ~x⊥ · ~u = 0. So

~x · ~u = (~x‖ + ~x⊥) · ~u = ~x‖ · ~u + ~x⊥ · ~u = k + 0

so~x‖ = (~x · ~u)~u, ~x⊥ = ~x − ~x‖.

How do we generalize this?

Proof.Let ~u be any unit vector spanning L. Then ~x‖ = k~u for some k,and ~x⊥ · ~u = 0. So

~x · ~u = (~x‖ + ~x⊥) · ~u = ~x‖ · ~u + ~x⊥ · ~u = k + 0

so~x‖ = (~x · ~u)~u, ~x⊥ = ~x − ~x‖.

How do we generalize this?

FactConsider a subspace V of Rn and a vector ~x in Rn. Then we canwrite

~x = ~x‖ + ~x⊥,

where ~x‖ is in V and ~x⊥ is in V⊥, and this representation isunique.

Proof.Choose an orthonormal basis ~u1, . . . , ~um of V . Supposing thedecomposition ~x = ~x‖ + ~x⊥ does exist, we know

~x‖ = c1~u1 + · · ·+ cm~um,

and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0. But

(~x − ~x‖) · ~ui = ~x · ~ui − ~x‖ · ~ui = ~x · ~ui − ci

so ci = ~x · ~ui .

~x = ~x‖ + ~x⊥,

~x‖ = c1~u1 + · · ·+ cm~um,

and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0.

so ci = ~x · ~ui .

~x = ~x‖ + ~x⊥,

~x‖ = c1~u1 + · · ·+ cm~um,

and ~x − ~x‖ is in V⊥. In particular, ~x − ~x‖ is orthogonal to ~ui , so(~x − ~x‖) · ~ui = 0. But

so ci = ~x · ~ui .

FactIf V is a subspace of Rn with orthonormal basis ~u1, . . . , ~um, then

projV (~x) = ~x‖ = (~u1 · ~x)~u1 + (~u2 · ~x)~u2 + · · ·+ (~um · ~x)~um

Example

[1 2 3 4

~u2 = 1√12

[−3 1 −1 1

]Tspan a subspace V of R4. Find the

projection of[1 −1 1 −1

]Tonto V .

Solution

projV (~x) = (~u1 · ~x)~u1 + (~u2 · ~x)~u2

= − 2√30

1√30

− 6√12

1√12

−31−11

−19/30

−23/30

Example

[1 2 3 4

~u2 = 1√12

[−3 1 −1 1

]Tspan a subspace V of R4. Find the

projection of[1 −1 1 −1

]Tonto V .

Solution

projV (~x) = (~u1 · ~x)~u1 + (~u2 · ~x)~u2

= − 2√30

1√30

− 6√12

1√12

−31−11

−19/30

−23/30

Geometric consequences

FactGiven two vectors ~x and ~y in Rn, we have

‖~x + ~y‖2 = ‖~x‖2 + ‖~y‖2

if and only if ~x · ~y = 0.

Proof.FOIL it out!

‖~x + ~y‖2 = (~x + ~y) · (~x + ~y) = ~x · ~x + ~x · ~y + ~y · ~x + ~y · ~y= ‖~x‖2 + 2(~x · ~y) + ‖~y‖2

Geometric consequences

FactGiven two vectors ~x and ~y in Rn, we have

‖~x + ~y‖2 = ‖~x‖2 + ‖~y‖2

if and only if ~x · ~y = 0.

Proof.FOIL it out!

‖~x + ~y‖2 = (~x + ~y) · (~x + ~y) = ~x · ~x + ~x · ~y + ~y · ~x + ~y · ~y= ‖~x‖2 + 2(~x · ~y) + ‖~y‖2

FactLet V be a subspace of Rn and ~x in Rn. Then

‖projV ~x‖ ≤ ‖~x‖

and equality holds if and only if ~x is in V .

Proof.Use the Pythagorean theorem:

‖~x‖2 =∥∥∥~x‖ + ~x⊥

∥∥∥2=

∥∥∥~x‖∥∥∥2

+∥∥∥~x⊥

∥∥∥2≥

∥∥∥~x‖∥∥∥2

FactLet V be a subspace of Rn and ~x in Rn. Then

‖projV ~x‖ ≤ ‖~x‖

and equality holds if and only if ~x is in V .

Proof.Use the Pythagorean theorem:

‖~x‖2 =∥∥∥~x‖ + ~x⊥

∥∥∥2=

∥∥∥~x‖∥∥∥2

+∥∥∥~x⊥

∥∥∥2≥

∥∥∥~x‖∥∥∥2

Let ~x and ~y be vectors and project ~x onto the subspace V spannedby ~u = ~y

‖y‖ . Then the previous fact applied to ~x and V gives

‖~x‖ ≥ ‖projV ~x‖ = ‖(~x · ~u)~u‖ =

∥∥∥∥(~x ·

‖~y‖

∥∥∥∥ =|~x · ~y |‖~y‖

Fact (The Cauchy-Schwarz inequality)

If ~x and ~y are vectors in Rn, then

|~x · ~y | ≤ ‖~x‖ ‖~y‖

with equality if and only if the vectors are parallel.

Let ~x and ~y be vectors and project ~x onto the subspace V spannedby ~u = ~y

‖y‖ . Then the previous fact applied to ~x and V gives

‖~x‖ ≥ ‖projV ~x‖ = ‖(~x · ~u)~u‖ =

∥∥∥∥(~x ·

‖~y‖

∥∥∥∥ =|~x · ~y |‖~y‖

Fact (The Cauchy-Schwarz inequality)

If ~x and ~y are vectors in Rn, then

|~x · ~y | ≤ ‖~x‖ ‖~y‖

with equality if and only if the vectors are parallel.

The angle between two vectors

In Math 21a we learned that in R2

~x · ~y = ‖~x‖ ‖~y‖ cos θ,

where θ is the angle between ~x and ~y .We can now define for a pair of vectors in any Rn the anglebetween them, by the formula

cos θ =~x · ~y‖~x‖ ‖y‖

The Cauchy-Schwarz inequality says that θ exists.

The angle between two vectors

In Math 21a we learned that in R2

~x · ~y = ‖~x‖ ‖~y‖ cos θ,

where θ is the angle between ~x and ~y .We can now define for a pair of vectors in any Rn the anglebetween them, by the formula

cos θ =~x · ~y‖~x‖ ‖y‖

The Cauchy-Schwarz inequality says that θ exists.

orthogonal bases

Documents